I think that you seem to be under the impression that Karpathy somehow alluded to or hinted at that in his talk, which indicates you haven't actually watched the talk, which makes your first point kind of weird.
I feel like one of the stronger points he made, was that you cannot treat the LLMs as something they're explicitly not, so why would anyone expect deterministic outcomes from them?
He's making the case for coding with LLMs, not letting the LLMs go by themselves writing code ("vibe coding"), and understanding how they work before attempting to do so.
The disclaimer you mention was indeed mentioned, although it's "in one ear, out the other" with most of his audience.
If I give you a glazed donut with a brief asterisk about how sugar can cause diabetes will it stop you from eating the donut?
You also expect deterministic outcomes when making analogies with power plants and fabs.
> maybe you've seen a lot of GitHub code is not just like code anymore there's a bunch of like English interspersed with code and so I think kind of there's a growing category of new kind of code so not only is it a new programming paradigm it's also remarkable to me that it's in our native language of English and so when this blew my mind a few uh I guess years ago now I tweeted this and um I think it captured the attention of a lot of people and this is my currently pinned tweet uh is that remarkably we're now programming computers in English now
I agree that it's remarkable that you can tell a computer "What is the biggest city in Maresme?" and it tries to answer that question. I don't think he's saying "English is the best language to make complicated systems uncomplicated with", or anything to that effect. Just like I still think "Wow, this thing is fucking flying" every time I sit onboard a airplane, LLMs are kind of incredible in some ways, yet so "dumb" in some other ways. It sounds to me like he's sharing a similar sentiment but about LLMs.
> although it's "in one ear, out the other" with most of his audience.
Did you talk with them? Otherwise this is just creating an imaginary argument against some people you just assume they didn't listen.
> If I give you a glazed donut with a brief asterisk about how sugar can cause diabetes will it stop you from eating the donut?
If I wanted to eat a donut at that point, I guess I'd eat it anyways? But my aversion to risk (or rather the lack of it) tend to be non-typical.
What does my answer mean in the context of LLMs and non-determinism?
> You also expect deterministic outcomes when making analogies with power plants and fabs.
Are you saying that the analogy should be deterministic or that power plants and fabs are deterministic? Because I don't understand if the former, and the latter really isn't deterministic by any definition I recognize that word by.
hehe, I wish.
The topics in the talk are not new. They have been explored and pondered up for quite a while now.
As for the outcome of the donut experiment, I don't know. You tell me. Apply it repeatedly at a big scale and see if you should alter the initial offer for best outcomes (as relative as "best" might be).
Sure, but your initial dismissal ("95% X, 5% Y") is literally about this talk no? And when you say 'it's "in one ear, out the other" with most of his audience' that's based on some previous experience, rather than the talk itself? I guess I got confused what applied to what event.
> As for the outcome of the donut experiment, I don't know. You tell me. Apply it repeatedly at a big scale and see if you should alter the initial offer for best outcomes (as relative as "best" might be).
Maybe I'm extra slow today, how does this tie into our conversation so far? Does it have anything to do with determinism or what was the idea behind bringing it up? I'm afraid you're gonna have to spell it out for me, sorry about that :)
I have, unfortunately. Start-up founders, managers, investors who taunt the need for engineers because "AI can fix it".
Don't get me wrong, there are plenty of "stochastic parrot" engineers even without AI, but still, not enough to make blanket statements.
Still, what's the outcome of our "glazed donut" argument, you got me curious what that would lead to. Did I die of diabetes?
Chip fabs are defo far into said depths.
Must we apply this at more shallow levels too?
As is, I don't quite understand what you're getting at here. Please just think that through and tell us what happens to the yield ratio when the software running on all those photolithography machines wouldn't be deterministic.
Non-determinism is not the problem, it's the quality of the software that matters. You can repeatedly ask me to solve a particular leetcode puzzle, and every time I might output a slightly different version. That's fine as long as the code solves the problem.
The software running on the machines (or anywhere) just needs to be better (choose your metric here) than the software written by humans. Software written by GPT-4 is better than software written by GPT-3.5, and the software written by o3 is better than software written by GPT-4. That's just the improvement from the last 3 years, and there's a massive, trillion-dollar effort worldwide to continue the progress.
Particularly not a 40min video.
Maybe it is tongue-in-cheek, maybe I am serious. I am not sure myself. But sometimes the interesting discussions comes from what is on top of the posters mind when viewing the title. Is that bad?
It doesn't have to be. But it does get somewhat boring and trite after a while when you start noticing that certain subjects on HN tend to attract general and/or samey comments about $thing, rather than the submission topic within $thing, and I do think that is against the guidelines.
> Please don't post shallow dismissals [...] Avoid generic tangents. Omit internet tropes. [...]
The specific part of:
> English is a terrible language for deterministic outcomes
Strikes me as both as a generic tangent about LLMs, and the comment as a whole feels like a shallow dismissal of the entire talk, as Karpathy never claims English is a good language for deterministic outcomes, nor have I heard anyone else make that claim.
There is a community expectation that people will know what they're talking about before posting, and in most cases that means having read the article. At the same time, I suspect that in many cases a lot of people commenting have not actually read the thing they're nominally commenting on, and they get away with it because the people upvoting them haven't either.
However, I think it's a good idea to do so, at least to make a top-level comment on an article. If you're just responding to someone else's comment, I don't think it's as necessary. But to stand up and make a statement about something you know nothing about seems buffoonish and would not, in general, elevate the level of discussion.
Someone here shared this ancient article by Dijkstra about this exact thing a few weeks ago: https://www.cs.utexas.edu/~EWD/transcriptions/EWD06xx/EWD667...
- training data - approximation of the desired outcome
Neither support a good direction for the complexity of some of the system around us, most of which require dedicated language. Imagine doing calculus or quantum physics in English. Novels of words would barely suffice.
So a context window as big as the training data itself?
What if the training data is faulty?
I'm confident you understand that working code or not doesn't matter in this analogy. Neither does LLMs reaching out for the right tool.
LLMs has its merits. Replacing concrete systems that require a formal language and grammar is not.
`1 + 1 = 2` because that's how maths works, not because of deja vú.
Context window will solve a class of problems, but will not solve all problems with AI.
I am a real user and I am on a general purpose e-commerce site and my ask is "I want a TV that is not that expensive", then by definition the user request is barely deterministic. User requests are normally like this for any application. High level and vague at best. Then developers spend all their time on edge cases, user QA, in the weeds junk that the User does not care about at all. People dont want to click filters and fill out forms for your app. They want it to be easy.
I've been coming to the realization that working with LLMs offer a different set of considerations than working on your own. Notably i find that i often obsess about design, code location, etc because if i get it wrong then my precious after-work time and energy are wasted on refactoring. The larger the code base, the more crippling this becomes for me.
However refactoring is almost not an issue with LLMs. They do it very quickly and aggressively. So the areas i'm not vibing on is just reviewing, and ensuring it isn't committing any insane sins. .. because it definitely will. But the structure i'm accepting is far from what i'd make myself. We'll see how this pans out long term for me, but it's a strategy that i'm exploring.
On the downside, my biggest difficulty with LLMs is getting them to just.. not. To produce less. Choosing too large of tasks is very easy and the code can snowball before you have a chance to pump the breaks and course correct.
Still, it's been a positive experience so far. I still consider it vibing though because i'm accepting far less quality work than what i'd normally produce. In areas where it matters though, i enforce correctness, and have to review everything as a result.
First, instruct a friend/colleague of how to multiply two 2 digit numbers in plain English.
Secondly (ideally with a different friend, to not contaminate tests), explain the same but using only maths formulas.
Where does the prompting process start and where does it end? Is it a one-off? Is the prompt clear enough? Do all the parties involved communicate within same domain objects?
Hopefully my example is not too contrived.
This is what an agent can do with an LLM. LLMs can help take English and generate some sort of an algorithm. The agent stores algorithm not the prompt. I do not know what current commercially available agents do but this was always clear to me.
He explicitly says that both LLMs and traditional software have very important roles to play.
LLMs though are incredibly useful when encoding the behavior of the system deterministically is impossible. Previously this fell under the umbrella of problems solved with ML. This would take a giant time investment and a highly competent team to pull off.
Now anyone can solve many of these same problems with a single API call. It’s easy to wave this off, but this a total paradigm shift.
English is a terrible language for deterministic outcomes in complex/complicated systems. Vibe coders won't understand this until they are 2 years into building the thing.
LLMs have their merits and he sometimes aludes to them, although it almost feels accidental.
Also, you don't spend years studying computer science to learn the language/syntax, but rather the concepts and systems, which don't magically disappear with vibe coding.
This whole direction is a cheeky Trojan horse. A dramatic problem, hidden in a flashy solution, to which a fix will be upsold 3 years from now.
I'm excited to come back to this comment in 3 years.