- inimino parentNot for long.
- I both agree and disagree. Yes, AI will democratize access to formal methods and will probably increase the adoption of them in areas where they make sense (e.g. safety-critical systems), but no, it won't increase the areas where formal methods are appropriate (probably < 1% of software).
What will happen instead is a more general application of AI systems to verifying software correctness, which should lead to more reliable software. The bottleneck in software quality is in specifying what the behavior needs to be, not in validating conformance to a known specification.
- > does this include the training data and fine-tune content?
No, I'm excluding that because I'm responding to the post which starts out with the example of: [prompt containing obvious exploit] -> [code containing obvious exploit] and proceeds immediately to the conclusion that local LLMS are less secure. In my opinion, if you're relying on the LLM to reject a prompt because it contains an exploit, instead of building a system that does not feed exploits into the LLM in the first place, security exploits are probably the least of your concerns.
There actually are legitimate concerns with poisoned training sets, and stuxnet-level attacks could plausibly achieve something along these lines, but the post wasn't about that.
There's a common thread among a lot of "LLM security theatre" posts that starts from implausible or brain-dead scenarios and then asserts that big AI providers adding magical guard rails to their products is the solution.
The solution is sanity in the systems that use LLMs, not pointing the gun at your foot and firing and hoping the LLM will deflect the bullet.
- The most "shocking" thing to me in the article is that people (apparently) think it's acceptable to run a system where content you've never seen can be fed into the LLM when it's generating code that you're putting in production. In my opinion, if you're doing that, your whole system is already compromised and you need to literally throw away what you're doing and start over.
Generally I hate these "defense in depth" strategies that start out with doing something totally brain-dead and insecure, and then trying to paper over it with sandboxes and policies. Maybe just don't do the idiotic thing in the first place?
- People knock "English as a programming language", but in my opinion this is the whole value of AI programming: by the time you've expressed your design and constraints well enough that an LLM can understand it, then anyone can understand it, and you end up with a codebase that's way more maintainable than what we're used to.
The problem of course is when people throw away the prompt and keep the code, like the code is somehow valuable. This would be like everyone checking in their binaries and throwing away their source code every time, while arguments rage on HN about whether compilers are useful. (Meanwhile, compiler vendors compete on their ability to disassemble and alter binaries in response to partial code snippets.)
The right way to do AI programming is: English defines the program, generated code is exactly as valuable as compiler output is, i.e. it's the actual artifact that does the thing, so in one sense it's the whole point, but iterating on it or studying it in detail is a waste of time, except occasionally when debugging. It's going to take a while, but eventually this will be the only way anybody writes code. (Note: I may be biased, as I've built an AI programming tool.)
If you can explain what needs to be done to a junior programmer in less time than it takes to do it yourself, you can benefit from AI. But, it does require totally rethinking the programming workflow and tooling.
- That's a potential outcome of any increase in training efficiency.
Which we should expect, even from prior experience with any other AI breakthrough, where first we learn to do it and then we learn to do it efficiently.
E.g. Deep Blue in 1997 was IBM showing off a supercomputer, more than it was any kind of reasonably efficient algorithm, but those came over the next 20-30 years.
- You can't but the transistor-based approach is the inefficient one, and transistors are pretty good at efficiently doing logic, so either there's no possible efficient solution based on deterministic computation, or there's tremendous headroom.
I believe human and machine learning unify into a pretty straightforward model and this shows that what we're doing that ML doesn't can be copied across, and I don't think the substrate is that significant.
- I'll have to refer you to my forthcoming paper for the full argument, but basically, humans (and all animals) experience surprise and then we attribute that surprise to a cause, and then we update (learn).
In ANNs we backprop uniformly, so the error correction is distributed over the whole network. This is why LLM training is inefficient.
- This reminds me of a rule I have for naming things in code (functions, variables, etc).
Say you add a function, and then the first time you call that function, you call it by a different name. Don't fix the function call to match the original name, but instead go back and change the name to match how you tried to call it. The state of mind you are in when you called the function is a better guide to naming than the state of mind you were in when you implemented it.