I'm not sure what authority linguists are supposed to have here. They have gotten approximately nowhere in the last 50 years. "Every time I fire a linguist, the performance of the speech recognizer goes up".
>Grammars are sets of rules on symbols and any form of encoding is very restrictive
But these rules can be arbitrarily complex. Hand-coded rules have a pretty severe complexity bounds. But LLMs show these are not in principle limitations. I'm not saying theory has nothing to add, but perhaps we should consider the track record when placing our bets.
We're yet to find any process at all that can't be computed with a Turing machine.
Why do you expect that "intelligence" is a sudden outlier? Do you have an actual reason to expect that?
Life. Consciousness. A soul. Imagination. Reflection. Emotions.
I can't help but perceive this as pseudo-profound bullshit. "Real soul and real imagination cannot run on a computer" is a canned "profound" statement with no substance to it whatsoever.
If a hunk of wet meat the size of a melon can do it, then why not a server rack full of nanofabricated silicon?
"Language" is an input/output interface. It doesn't define the internals that produce those inputs and outputs. And between those inputs and outputs sits a massive computational process that doesn't operate on symbols or words internally.
And, what "clearer milestones" do you want exactly?
To me, LLMs crushing NLU and CSR was the milestone. It was the "oh fuck" moment, the clear signal that old bets are off and AGI timelines are now compressed.
Humans create new words and grammatical constructs all the time in the process of building/discovering new things. This is true even in math, where new operators are created to express new operations. Are LLMs even capable of this kind of novelty?
There's also the problem that parts of human experience are inexpressible in language. A very basic example is navigating 3D space. This is not something that had to be explained to you as a baby, your brain just learned how to do it. But this problem goes deeper. For instance, intuition about the motion of objects in space. Even before Newton described gravitation every 3 year old still knew that an object that's dropped would fall to the ground a certain way. Formalizing this basic intuition using language took thousands of years of human development and spurred the creation of calculus. An AI does not have these fundamental intuitions nor any way to obtain them. Its conception of the world is only as good as the models and language (both mathematical and spoken) we have to express it.
Which is pretty damn good, all things considered.
And sure, training set text doesn't contain everything - but modern AIs aren't limited to just the training set text. Even in training stage, things like multimodal inputs and RLVR have joined the fray.
I don't think "create novel concepts" is a real limitation at all. Nothing prevents an AI from inventing new notations. GPT-4o would often do that when talking to AI psychosis victims.
Imagine trying to write apps without thinking about the limitations of the APIs you use. In fact we just recently escaped that same stupidity in the SaaS era! That's how silly LLMs will seem in the near future. They will stick around as the smarter chatbots we've wanted for so long, but they are so very far away from AGI.
That's how a lot of bleeding edge multimodals work already. They can take and emit images, sound, actions and more.
Yes, and most with a background in linguistics or computer science have been saying the same since the inception of their disciplines. Grammars are sets of rules on symbols and any form of encoding is very restrictive. We haven't come up with anything better yet.
The tunnel vision on this topic is so strong that many don't even question language itself first. If we were truly approaching AGI anytime soon, wouldn't there be clearer milestones beforehand? Why must I peck this message out, and why must you scan it with your eyes only for it to become something else entirely once consumed? How is it that I had this message entirely crystalized instantly in my mind, yet it took me several minutes of deliberate attention to serialize it into this form?
Clearly, we have an efficiency problem to attack first.