"tons what they" autocorrected from "to do what they do."
"Paucity of the stimulus" is the term for what I'm talking about with the brain needing much less data, but beyond just more parameters we may have innate language processing that isn't there in other animals; Chomsky has been kind of relegated away now after LLMs but he may still have been right if it isn't just parameter count and or the innate thing different from animals isn't something like transformers. If you look at the modern language program in Chomsky's later years, it does have some remarkably similar things to transformers: permutation independent internal representation, and the merge operation being very similar to transformer's soft max. It's kind of describing something very like single head attention.
We know animals have rich innate neural abilities beyond just beating the heart and breathing etc.: a baby horse can be blind folded from birth, several days later blind fold taken off and it can immediately walk and navigate. Further development goes on, but other animals like cats have a visual system that doesn't seem to develop at all if it doesn't get natural stimulus in a critical early period. Something like that may apply to human language, it may be multiple systems missing from other apes and early hominids, but whatever it is we don't think it had many generations to evolve. Researchers have identified circuits in songbird brains that are also in humans but not apes, and something like that may be a piece of it for tracking sequences.
"Paucity of the stimulus" is the term for what I'm talking about with the brain needing much less data, but beyond just more parameters we may have innate language processing that isn't there in other animals; Chomsky has been kind of relegated away now after LLMs but he may still have been right if it isn't just parameter count and or the innate thing different from animals isn't something like transformers. If you look at the modern language program in Chomsky's later years, it does have some remarkably similar things to transformers: permutation independent internal representation, and the merge operation being very similar to transformer's soft max. It's kind of describing something very like single head attention.
We know animals have rich innate neural abilities beyond just beating the heart and breathing etc.: a baby horse can be blind folded from birth, several days later blind fold taken off and it can immediately walk and navigate. Further development goes on, but other animals like cats have a visual system that doesn't seem to develop at all if it doesn't get natural stimulus in a critical early period. Something like that may apply to human language, it may be multiple systems missing from other apes and early hominids, but whatever it is we don't think it had many generations to evolve. Researchers have identified circuits in songbird brains that are also in humans but not apes, and something like that may be a piece of it for tracking sequences.