- > Game theory is inevitable. Because game theory is just math, the study of how independent actors react to incentives.
That's not how mathematics works. "it's just math therefore it's a true theory of everything" is silly.
We cannot forget that mathematics is all about models, models which, by definition, do not account for even remotely close to all the information involved in predicting what will actually occur in reality. Game Theory is a theory about a particular class of mathematical structures. You cannot reduce all of existence to just this class of structures, and if you think you can, you'd better be ready to write a thesis on it.
Couple that with the inherent unpredictability of human beings, and I'm sorry but your Laplacean dreams will be crushed.
The idea that "it's math so it's inevitable" is a fallacy. Even if you are a hardcore mathematical Platonist you should still recognize that mathematics is a kind of incomplete picture of the real, not its essence.
In fact, the various incompleteness theorems illustrate directly, in Mathematic's own terms, that the idea that a mathematical perspective or any logical system could perfectly account for all of reality is doomed from the start.
- Confusion of effect for cause. Unconscious or effortless processing by the brain is usually way more accurate and reliable than conscious processing, but outside of being "gifted" you only get to consistent unconscious processing after years of training and conscious practice that ingrain muscle memory etc.
- Yes. And the article is a perfect example of the dangerous sort of automation bias that people will increasingly slide into when it comes to LLMs. I realize Karpathy is sort of incentivized toward this bias given his career, but he doesn't even spend a single sentence even so much as suggesting that the results would need further inspection, or that they might be inaccurate.
The LLM is consulted like a perfect oracle, flawless in its ability to perform a task, and it's left at that. Its results are presented totally uncritically.
For this project, of course, the stakes are nil. But how long until this unfounded trust in LLMs works its way into high stakes problems? The reign of deterministic machines for the past few centuries has ingrained a trust in the reliability of machines in us that should be suspended when dealing with an inherently stochastic device like an LLM.
- This essay completely misunderstands how the notion of emergence gained prominence and how people tend to actually use it. It's a straw man that itself devolves into a circular argument "embrace a reductionist epistemology because you should embrace a reductionist epistemology".
It doesn't even meaningfully engage with the historical literature that established the term, etc. If you want to actually understand why the term gained prominence, check out the work of Edgar Morin.
- Thanks for the links! I'll have to dig into this more for sure. Looking at the bulleted summary, I'm not sure your argument is sufficiently nuanced or being made in good faith.
The article argues that the brain "predicts" acts of perception in order to minimize surprise. First of all, very few people mean to talk about these unconscious operations of the brain when they claim they are "thinking". Most people have not read enough neuroscience literature to have such a definition. Instead, they tend to mean "self-conscious activity" when they say "thinking". Thinking, the way the term is used in the vernacular, usually implies some amount of self-reflexivity. This is why we have the term "intuition" as opposed to thinking after all. From a neuronal perspective, intuition is still thinking, but most people don't think (ha) of the word thinking to encompass this, and companies know that.
It is clear to me, as it is to everyone one the planet, that when OpenAI for example claims that ChatGPT "thinks" they want consumers to make the leap to cognitive equivalence at the level of self-conscious thought, abstract logical reasoning, long-term learning, and autonomy. These machines are designed such that they do not even learn and retain/embed new information past their training date. That already disqualifies them from strong equivalence to human beings, who are able to rework their own tendencies toward prediction in a meta cognitive fashion by incorporating new information.
- I completely agree that we don't know enough, but I suggest that that entails that the critics and those who want to be cautious are correct.
The harms engendered by underestimating LLM capabilities are largely that people won't use the LLMs.
The harms engendered by overestimating their capabilities can be as severe as psychological delusion, of which we have an increasing number of cases.
Given we don't actually have a good definition of "thinking" what tack do you consider more responsible?
- When you have a thought, are you "predicting the next thing"—can you confidently classify all mental activity that you experience as "predicting the next thing"?
Language and society constrains the way we use words, but when you speak, are you "predicting"? Science allows human beings to predict various outcomes with varying degrees of success, but much of our experience of the world does not entail predicting things.
How confident are you that the abstractions "search" and "thinking" as applied to the neurological biological machine called the human brain, nervous system, and sensorium and the machine called an LLM are really equatable? On what do you base your confidence in their equivalence?
Does an equivalence of observable behavior imply an ontological equivalence? How does Heisenberg's famous principle complicate this when we consider the role observer's play in founding their own observations? How much of your confidence is based on biased notions rather than direct evidence?
The critics are right to raise these arguments. Companies with a tremendous amount of power are claiming these tools do more than they are actually capable of and they actively mislead consumers in this manner.
- The defenders and the critics around LLM anthropomorphism are both wrong.
The defenders are right insofar as the (very loose) anthropomorphizing language used around LLMs is justifiable to the extent that human beings also rely on disorder and stochastic processes for creativity. The critics are right insofar as equating these machines to humans is preposterous and mostly relies on significantly diminishing our notion of what "human" means.
Both sides fail to meet the reality that LLMs are their own thing, with their own peculiar behaviors and place in the world. They are not human and they are somewhat more than previous software and the way we engage with it.
However, the defenders are less defensible insofar as their take is mostly used to dissimulate in efforts to make the tech sound more impressive than it actually is. The critics at least have the interests of consumers and their full education in mind—their position is one that properly equips consumers to use these tools with an appropriate amount of caution and scrutiny. The defenders generally want to defend an overreaching use of metaphor to help drive sales.
- Luckily for us, technologies like SQL made similar promises (for more limited domains) and C suites couldn't be bothered to learn that stuff either.
Ultimately they are mostly just clueless, so we will either end up with legions of way shittier companies than we have today (because we let them get away with offloading a bunch of work to tools they rms int understand and accepting low quality output) or we will eventually realize the continued importance of human expertise.
- Coding Contest != Software Engineering
Or even solving problems that business need to solve, generally speaking.
This complete misunderstand of what software engineering even is is the major reason so many engineers are fed up with the clueless leaders foisting AI tools upon their orgs because they apparently lack the critical reasoning skills to be able to distinguish marketing speak from reality.
- Yeah, unfortunately Marx was right about people not realizing the problem, too. The proletariat drowns in false consciousness :(
In reality, the US is finally waking up to the fact that the "golden age" of capitalism in the US was built upon the lite socialism of the New Deal, and that all the bs economic opinions the average american has subscribed to over the past few decades was completely just propaganda and anyone with half a brain cell could see from miles away that since reagonomics we've had nothing but a system that leads to gross accumulation to the top and to the top alone and this is a sure fire way (variable maximization) in any complex system to produce instability and eventual collapse.
- Many mathematicians do in fact teach the rules of the game in numerous introductory texts. However, you don't expect to have to explain the rules every time you play the game with people who you've established know the game. Any session would take ages if so, and in many cases the game only become more fun the more fluent the players are.
I'm not fully convinced the article makes the claim that jargon, per se, is what needs to change nor that the use of jargon causes gatekeeping. I read more about being about the inherent challenges of presenting more complicated ideas, with or without jargon and the pursuit of better methods, which themselves might actually depend on more jargon in some cases (to abstract away and offload the cognitive costs of constantly spelling out definitions). Giving a good name to something is often a really powerful way to lower the cognitive costs of arguments employing the names concept. Theoretics in large part is the hunt for good names for things and the relationships between them.
You'd be hard pressed to find a single human endeavor that does not employ jargon in some fashion. Half the point of my example was to show that you cannot escape jargon and "gatekeeping" even in something as silly and fun as a card game.
- Precisely. Think of mathematics like a game.
Players of magic the gathering will say a creature "has flying" by which they mean "it can only be blocked by other creatures with reach or flying".
Newcomers obviously need to learn this jargon, but once they do, communication is greatly facilitated by not having to spell out the definition.
Just like games, the definitions in mathematics are ethereal and purely formal as well, and it would be a pain to spell them out on every occasion. It stems more from efficient communication needs then from gatekeeping.
You expect the players of the game to learn the rules before they play.
- As someone who has always struggled with mathematics at the calculational level, but who really enjoys theorems and proofs (abstract mathematics), here are some things that help me.
1. Study predicate logic, then study it again, and again, and again. The better and more ingrained predicate logic becomes in your brain the easier mathematics becomes.
2. Once you become comfortable with predicate logic, look into set theory and model theory and understand both of these well. Understand the precise definition of "theory" wrt to model theory. If you do this, you'll have learned the rules that unify nearly all of mathematics and you'll also understand how to "plug" models into theories to try and better understand them.
3. Close reading. If you've ever played magic the gathering, mathematics is the same thing--words are defined and used in the same way in which they are in games. You need to suspend all the temptation to read in meanings that aren't there. You need to read slowly. I've often only come upon a key insight about a particular object and an accurate understanding only after rereading a passage like 50 times. If the author didn't make a certain statement, they didn't make that statement, even if it seems "obvious" you need to follow the logical chain of reasoning to make sure.
4. Translate into natural english. A lot of math books will have whole sections of proofs and /or exercises with little to no corresponding natural language "explainer" of the symbolic statements. One thing that helps me tremendously is to try and frame any proof or theorem or collection of these in terms of the linguistic names for various definitions etc. and to try and summarize a body of proofs into helpful statements. For example "groups are all about inverses and how they allow us to "reverse" compositions of (associative) operations--this is the essence of "solvability"". This summary statement about groups helps set up a framing for me whenever I go and read a proof involving groups. The framing helps tremendously because it can serve as a foil too—i.e. if some surprising theorem contravene's the summary "oh, maybe groups aren't just about inversions" that allows for an intellectual development and expansion that I find more intuitive. I sometimes think of myself as a scientist examining a world of abstract creatures (the various models (individuals) of a particular theory (species))
5. Contextualize. Nearly all of mathematics grew out of certain lines of investigation, and often out of concrete technical needs. Understanding this history is a surprisingly effective way to make many initially mysterious aspects of a theory more obvious, more concrete, and more related to other bits of knowledge about the world, which really helps bolster understanding.
- > Imagine each Substack owner can make their own font to highlight the essence of their writing.
No please, for the love of all that is good and holy, please no
- I think most people are in complete agreement.
What people don't like about LLM PRs is typically:
a. The person proposing the PR usually lacks adequate context and so it makes communication and feedback, which are essential, difficult if not impossible. They cannot even explain the reasoning behind the changes they are proposing, b. The volume/scale is often unreasonable for human reviewed to contend with. c. The PR may not be in response to an issue but just the realization of some "idea" the author or LLM had, making it even harder to contextualize. d. The cost asymmetry, generally speaking is highly unfavorable to the maintainers.
At the moment, it's just that LLM driven PRs have these qualities so frequently that people use LLM bans as a shorthand since writing out a lengthy policy redescrbiing the basic tenets of participation in software development is tedious and shouldn't be necessary, but here we are, in 2025 when everyone has seemingly decided to abandon those principles in favor of lazyily generating endless reams of pointless code just because they can.
- > But they'll often be close enough or even equivalent mathematically
Who is babbling? The number of concepts in human language that have no mathematical formalization far outnumber the ones that do, lol.
Yes, we can, obviously, come up with shared, mathematically precise definitions for certain concepts. Keep in mind that:
A. These formal or scientific definitions are not the full exhaustion of the concept. Linguistic usage is varied and wide. Anyone who has bothered to open an introductory linguistics textbook understands this.
B. The scientific and mathematical definitions still change over time and can also change across cultures and contexts.
I can assure you that someone who has scored very high on an IQ test would not be considered "intelligent" in a group of film snobs if they were not aware of the history of film, up to date on the latest greats, etc. etc. These people would probably use the word intelligent to describe what they mean (knowledge of film) and not the precise technical definition we've come up with, if any, whether you like it or not.
My point is not that it is impossible to come up with definitions, my point is that for socially fluid concepts like intelligence, which are highly dependent on the needs and circumstances of the people employing the word, we will likely never pin it down. There is an asterisk on every use of the word. This is the case with basically every word to more or lesser degree, that's why language and ideas evolve in the first place.
My whole point is that people that don't realize this and put faith in IQ as though it is some absolute, or final, indicator of intelligence are dumb and probably just egotists who are uncomfortable with uncertainty and want reassurance that they are smart so that they can tell other people they are "babbling" and feel good about themselves and their intellectual superiority complex (read: self justified pride in being an asshole).
My claim is that this high variability and contextual sensitivity is a core part of this word and the way we use it. That's what I mean when I say I don't think we'll ever have a good definition.
EDIT: Or, to make it a little easier to understand. We will never have a universal definition of "moral good" because it is dependent on value claims, people will argue morality forever. My position is that "intelligence" is equally dependent on value claims which I think anyone who has spent more than five minutes with people not like themselves or trained in different forms of knowledge intuitively understands this.
- Exactly. We don't have a good definition of intelligence and I don't think we ever will. Like all social concepts, it is highly dependent on the needs, goals, and values of the human societies that define it, and so it is impossible to come up with a universal definition. If your needs don't align with the needs an AI has been trained to meet, you are not going to find it very intelligent of helpful for meeting those needs.
- The gauge I use for intelligence is how much stock a person puts into an IQ test.
In my view, people who are able to question the legitimacy or applicability of IQ as a general measure of "intelligence", an idea that is highly contextual, are probably intelligent. They are at least smart enough to question social conceptions and to recognize the contingent nature of such conceptions. People who uncritically view IQ as some kind of unassailable proof of "intelligence" may be good at solving certain classes of known problems but, I really am not surprised that they may lack the imagination to contribute meaningful things to society, as a blind faith in a measure developed by fallible human beings is indicative of limited thinking /creativity.
Obviously someone can score well on an IQ test and question its validity as a signifier of intelligence, just as one can score poorly and place a strong degree of faith in it—but the way someone approaches it, in either case, is a very telling indicator of their own intellectual biases and limitations.
If you start studying basically any field that isn't computer science you will in fact discover that the world is rife with randomness, and that the dreams of a Laplace or Bentham are probably unrealizable, even if we can get extremely close (but of course, if you constrain behavior in advance through laws and restraints, you've already made the job significantly easier).
Thinking that reality runs like a clock is literally a centuries outdated view of reality.