- If enough care about this that can and will do something about it (making formalization easier for the average author), that happens over time. Today there's a gap, and in the figurative tomorrow, said gap shrinks. Who knows what the future holds? I'm not discounting that the situation might change.
- Learn what? I don't agree and you haven't given reasons. I don't write for your personal satisfaction.
- Alternatively: some people are just better at / more comfortable thinking in auditory mode than visual mode & vice versa.
In principle I don't see why they should have different amounts of thought. That'd be bounded by how much time it takes to produce the message, I think. Typing permits backtracking via editing, but speaking permits 'semantic backtracking' which isn't equivalent but definitely can do similar things. Language is powerful.
And importantly, to backtrack in visual media I tend to need to re-saccade through the text with physical eye motions, whereas with audio my brain just has an internal buffer I know at the speed of thought.
Typed messages might have higher _density_ of thought per token, though how valuable is that really, in LLM contexts? There are diminishing returns on how perfect you can get a prompt.
Also, audio permits a higher bandwidth mode: one can scan and speak at the same time.
- Unavoidable: expecting someone else to do the connection isn't a viable strategy in semi-adversarial conditions so it has to be bound into the local context, which costs clarity:
- Escaping death doesn't become more tractable because you don't want to die.
This is trivially 'willfully misunderstood', whereas my original framing is more difficult -- you'd need to ignore the parallel with the root level comment, the parallel with the conversation structure thus far, etc. Less clear, but more defensible. It's harder to plausibly say it is something it is not, and harder to plausibly take it to mean a position I don't hold (as I do basically think that requiring formalized proofs is a _practically_ impossible ask).
By your own reckoning, you understood it regardless. It did the job.
It does demonstrate my original original point though, which is that messages under optimization reflect environmental pressures in addition to their content.
- I don't think it matters, to be quite honest. Absolute tractability isn't relevant to what the analogy illustrates (that reality doesn't bend to whims). Consider:
- Locating water doesn't become more tractable because you are thirsty.
- Popping a balloon doesn't become more tractable because you like the sound.
- Readjusting my seat height doesn't become more tractable because it's uncomfortable.
The specific example I chose was for the purpose of being evocative, but is still precisely correct in providing an example of: presenting a wish for X as evidence of tractability of X is silly.
I object to any argument of the form: "Oh, but this wish is a medium wish and you're talking about a large wish. Totally different."
I hold that my position holds in the presence of small, medium, _or_ large wishes. For any kind of wish you'd like!
- If wishes were fishes, as they say.
To demonstrate with another example: "Gee, dying sucks. It's 2025, have you considered just living forever?"
To this, one might attempt to justify: "Isn't it sufficient that dying sucks a lot? Is it so hard to understand that having seen people die, I really don't want to do that? It really really sucks!", to which could be replied: "It doesn't matter that it sucks, because that doesn't make it any easier to avoid."
- Do selection dynamics require awareness of incentives? I would think that the incentives merely have to exist, not be known.
On HN, that might be as simple as display sort order -- highly engaging comments bubble up to the top, and being at the top, receive more attention in turn.
The highly fit extremes are -- I think -- always going to be hyper-specialized to exploit the environment. In a way, they tell you more about the environment than whatever their content ostensibly is.
- It just overlaid a typical ATX pattern across the motherboard-like parts of the image, even if that's not really what the image is showing. I don't think it's worthwhile to consider this a 'local recognition failure', as if it just happened to mistake CMOS for RAM slots.
Imagine it as a markdown response:
# Why this is an ATX layout motherboard (Honest assessment, straight to the point, *NO* hallucinations)
1. *RAM* as you can clearly see, the RAM slots are to the right of the CPU, so it's obviously ATX
2. *PCIE* the clearly visible PCIE slots are right there at the bottom of the image, so this definitely cannot be anything except an ATX motherboard
3. ... etc more stuff that is supported only by force of preconception
--
It's just meta signaling gone off the rails. Something in their post-training pipeline is obviously vulnerable given how absolutely saturated with it their model outputs are.
Troubling that the behavior generalizes to image labeling, but not particularly surprising. This has been a visible problem at least since o1, and the lack of change tells me they do not have a real solution.
- >> But we don't usually believe in both the bullshit and in the fact the the BS is actually BS.
> I can't parse what you mean by this.
The point is that humans care about the state of a distributed shared world model and use language to perform partial updates to it according to their preferences about that state.
Humans who prefer one state (the earth is flat) do not -- as a rule -- use language to undermine it. Flat earthers don't tell you all the reasons the earth cannot be flat.
But even further than this, humans also have complex meta-preferences of the state, and their use of language reflects those too. Your example is relevant here:
> My dad absolutely insisted that the water draining in toilets or sinks are meaningfully influenced by the Coriolis effect [...]
> [...] should have been able to figure out from first principles why the Coriolis effect is exactly zero on the equator itself, didn't.
This is an exemplar of human behavior. Humans act like this. LLMs don't. If your dad did figure out from first principles and expressed it and continued insisting the position, I would suspect them of being an LLM, because that's how LLMs 'communicate'.
Now that the what is clear -- why? Humans experience social missteps like that as part of the loss surface. Being caught in a lie sucks, so people learn to not lie or be better at it. That and a million other tiny aspects of how humans use language in an overarching social context.
The loss surface that LLMs see doesn't have that feedback except in the long tail of doing Regularized General Document Corpora prediction perfectly. But it's so far away compared to just training on the social signal, where honesty is immediately available as a solution and is established very early in training instead of at the limit of low loss.
How humans learn (embedded in a social context from day one) is very effective at teaching foundational abilities fast. Natural selection cooked hard. LLM training recipes do not compare, they're just worse in so many different ways.
- The concrete itself can be damaged further over time by expanding root networks / growth.
- > if the body is experiencing adverse reactions to certain foods, the cause should be a biochemical one
No. I can look at a picture of gross food, or imagine it, and be nauseated. Restricting potential causes this tightly is wrong. UPFs have a huge causal surface--how one ingredient breaks down into nutrients is basically not optimized at all compared to all the other stuff. Why only look at the unoptimized part as causal origin, when the optimization is the common factor making it bad?
To put it another way, we know all the stuff that has had a ton of work put into making it something lots of people will buy, is just generally bad for unclear reasons. Examining the aspect which has had very little work put into it is clearly not the way to go.
It could be something as weird as people's built-in heuristics for which food to eat (cravings) being actually kinda important to hit specific nutrient breakpoints at different times. By subsuming those cravings using UPF technology, that stops working and general health suffers.
It could be that the baseline palatability of the output of the mechanical process is low, and so the product is universally combined with an additive that recovers palatability, but has some health drawback.
The overarching pattern here is that optimization geared toward overcoming people's heuristics of what to consume makes these kinds of decisions all the time. Doing one thing to make it cheaper to produce makes it worse at getting people to buy it, but we can just cheat back the ability to get people to buy it by turning the dial on another thing that reliably makes people want to buy it, at the cost of being horrible for them.
- 127.0.0.2
- There are different varieties of attention, which just amounts to some kind of learned mixing function between tokens in a sequence.
For an input of length N (tokens), the standard kind of attention requires N squared operations (hence, quadratic - it scales with the square of input length). You have to check how every token attends to every other token.
There are a bunch of alternative mixing functions which are instead linear with respect to N. Every additional token costs the same amount of work. The typical method is to have a constant size state manipulated recurrently, which necessarily implies some level of lossy compression in the state (quadratic attention doesn't really have state in this sense - it computes and checks every possible relation always).
Linear attentions kind of suck in comparison to quadratic attention but the efficiency is very attractive, especially at inference time where you don't need more VRAM to store more context.
TLDR; conventional attentions scale N^2 time, N space (kv cache), and are exact. linear attentions scale N time, constant space (recurrent state), and are lossy.
- In the spirit of the library, which contains both your comment and mine:
> The hypothetical "library of all possible books" isn't useful to anyone.
That's not an archive, and has no uses even for researchers, especially not for historians.
- Why wouldn't it? That's downright in-distribution. Plenty of it in the pretrain corpus.
- It doesn't matter / is not relevant. The harm is not caused by intent, but by action. Sending language at human beings in a way they can read has side effects. It doesn't matter if the language was generated by stochastic process or by conscious thinking entity, those side effects do actually exist. That's kind of the whole point of language.
The danger is that this class of generators generates language that seems to cause people to fall into psychoses. They act as a 'professed belief' valence amplifier[0], and seem to do so generally, and the cause is fairly obvious if you think about how these things actually work (language models generating most likely continuations for existing text that also by secondary optimization objective are 'pleasing' or highly RLHF positive).
To some degree, I agree that understanding how they work attenuates the danger, but not entirely. I also think it is absurd to expect the general public to thoroughly understand the mechanism by which these models work before interacting with them. That is such an extremely high bar to clear for a general consumer product. People use these things specifically to avoid having to understand things and offload their cognitive burdens (not all, but many).
No, "they're just stochastic parrots outputting whatever garbage is statistically likely" is not enough understanding to actually guard against the inherent danger. As I stated before, that's not the dangerous part - you'd need to understand the shape of the 'human psychosis attractor', much like the claude bliss attractor[0] but without the obvious solution of just looking at the training objective. We don't know the training objective for humans, in general. The danger is in the meta structure of the language emitted, not the ontological category of the language generator.
- Model weights are significantly larger than cache in almost all cases. Even an 8B parameter model is ~16G in half precision. The caches are not large enough to actually cache that.
Every weight has to be touched for every forward pass, meaning you have to wait for 16G to transfer from VRAM -> SRAM -> registers. That's not even close to 100ns: on a 4090 with ~1TB/s memory bandwidth that's 16 milliseconds. PCIe latency to launch kernels or move 20 integers or whatever is functionally irrelevant on this scale.
The real reason for batching is it lets you re-use that gigantic VRAM->SRAM transfer across the batch & sequence dimensions. Instead of paying a 16ms memory tax for each token, you pay it once for the whole batched forward pass.
- What would you consider to be a non memory safety critical section? I tried to answer this and ended up in a chain of 'but wait, actually memory issues here would be similarly bad...', mainly because UB and friends tend to propagate and make local problems very non-local.
- I think that discussing this subject in the abstract, with some ideal notion of a tool that generates perpetually enjoyable stories misses the thrust of the general objection, which is actually mechanistic, and not social. LLMs are not this tool, for many (I would say most, but...). LLMs recycle the same ideas over and over and over with trite stylistic variation. Once you have read enough LLM generated/adapted works they're all the same and they lose all value as entertainment.
There is a moment I come to over and again when reading any longer form work informed by AI. At first, I don't notice (if the author used it 'well'). But once far enough in, there is a moment where everything aligns and I see the structure of it and it is something I have seen a thousand times before. I have seen it in emails and stories and blog posts and articles and comments and SEO spam and novels passed off as human work. In that moment, I stop caring. In that moment, my brain goes, "Ah, I know this." And I feel as if I have already finished reading its entirety.
There is some amount of detail I obviously do not 'recall in advance of reading it'. The sum total of this is that which the author supplied. The rest is noise. There is no structure beyond that ever present skein patterned out by every single LLM in the same forms, and that skein I am bored of. It's always the same. I am tired of reading it again and again. I am tired of knowing exactly how what is coming up will come, if not the precise details of it, and the way every reaction will occur, and how every pattern of interaction will develop. I am tired of how LLMs tessellate the same shapes onto every conceptual seam.
I return now to my objection to your dismissal of the value of insight into the author's mind. The chief value, as I see it, is merely that it is always different. Every person has their own experiences and that means when I read them I will never have a moment where I know them (and consequently, the work) in advance, as I do the ghost-writing LLMs, which all share a corpus of experience.
Further, I would argue that the more apt notion of insight into the work is the sole value of said work (for entertainment), and that insight is one time use (or strongly frequency dependent, for entertainment value). Humans actively generate 'things to be insightful of' through lived experience, which enriches their outputs, while LLMs have an approximately finite quantity of such due to their nature as frozen checkpoints, which leads you to "oh, I have already consumed this insight; I have known this" situations.
If you have a magic tool that always produces a magically enjoyable work, by all means, enjoy. If you do not, which I suspect, farming insight from a constantly varying set of complex beings living rich real life experiences is the mechanical process through which a steady supply of enjoyable, fresh, and interesting works can be acquired.
Being unaware of this process does not negate its efficacy.
TLDR; from the perspective of consumption, generated works are predominantly toothless as reading any AI work depletes from a finite, shared pool of entertaining-insight that runs dry too quickly
That is to say, they are equally likely if you don't do next token prediction at all and instead do text diffusion or something. Architecture has nothing to do with it. They arise because they are early partial solutions to the reconstruction task on 'all the text ever made'. Reconstruction task doesn't care much about truthiness until way late in the loss curve (where we probably will never reach), so hallucinations are almost as good for a very long time.
RL as is typical in post-training _does not share those early solutions_, and so does not share the fundamental problems. RL (in this context) has its own share of problems which are different, such as reward hacks like: reliance on meta signaling (# Why X is the correct solution, the honest answer ...), lying (commenting out tests), manipulation (You're absolutely right!), etc. Anything to make the human press the upvote button or make the test suite pass at any cost or whatever.
With that said, RL post-trained models _inherit_ the problems of non-optimal large corpora reconstruction solutions, but they don't introduce more or make them worse in a directed manner or anything like that. There's no reason to think them inevitable, and in principle you can cut away the garbage with the right RL target.
Thinking about architecture at all (autoregressive CE, RL, transformers, etc) is the wrong level of abstraction for understanding model behavior: instead, think about loss surfaces (large corpora reconstruction, human agreement, test suites passing, etc) and what solutions exist early and late in training for them.