I did walked you through how I would do it. Would you change your response if I said I work in this space? It seems like an irrelevant point in this discussion.
You don't need to explain anything. This is on a flagged thread, obscure and unseen. I'm actually surprised by how invested you are in this apparently irrelevant matter.
In summary: the existing program analysis tooling in this space has been ineffective for decades, despite hundreds of millions of dollars invested in the tooling. If it is effective now, that strongly indicates that the LLM component of it isn't irrelevant; nothing else in the field has changed.
Note that everybody in this story concedes the LLM involvement. The only person who isn't is you, and you're not actually involved. (I'm not either, but I'm agreeing with --- checks again --- everybody involved in the story).
If it is a central role, then there is nothing to loose from describing it better. That's why this feels so strange. You disagree with me, but you don't present an arrangement in which the LLM plays a role different to what I described. In fact, no one here did. It's like you're not disagreeing with me, but trying to make me stop describing how to achieve a similar quality system out of free pieces.
Also, somehow, you keep coming back to this uninteresting conversation where no one offers you anything new.
I recommending being kinder to people who offer their time. Even when we disagree or are having a rollicking discussion, there's a fundamental respect we should have for each other, if begrudging.
[1] Where you are: "[it seems you are] trying to make me stop describing how to achieve a similar quality system out of free pieces."
Where you started: "Do you believe AI is at the core of these security analyzers? If so, why the personal story blogpost? You can just explain me in technical terms why is that so.
Claiming to work for Google does not work as an authority card for me, you still have to deliver a solid argument.
Look, AI is great for many things, but to me these products sounds like chocolate that is actually just 1% real chocolate. Delicious, but 99% not chocolate."
Allegedly (it's proprietary, we don't know). Maybe it's the triage approach, and there are undiscovered non-LLM triage techniques that would surpass it.
If I were to guess, the AI naming was for marketing purposes (to ride on the hype train), not because it accurately describes the product (even though it might accurately describe the product).
Most importantly, how is that it's so effective? I want to know. Perhaps you and some others just want to celebrate an LLM win. That's fine, but I want to know how it works.
I'd say my guess is fair, and it's a viable approach for someone trying to create a similar tool. If I were to try and replicate this, I would definitely start with an existing static analyzer. For example, I would do it with phpstan (just because I know it a little bit better).
I would extend it so it becomes more verbose than what it currently is (something humans don't want, but machines might benefit from it). Perhaps I would introduce some rules that make it report things that aren't even issues, but just information I can gauge from the AST (like, does this controller has a middleware? If so, emit something in the report). Then I would attempt to use that enriched report as the input for a coding model, and experiment with different prompts and different granularity units on the input.
It sounds reasonable, doesn't it? I could describe that approach as "LLM right in the core of the solution", but I know by heart that in that arrangement, the quality of the final product is still capped by the static analyzer and what it can detect and describe. It doesn't matter that the LLM is what makes it better. My wheat farm is still about wheat, not the fancy sift I recently bought to separate it from the chaff.
I don't understand why this sounds so offensive to some of the readers here. I was just thinking "how would I use AI in such a product" and the only way I can come up with is this way in which is not the main show.
I mean, my experience with LLMs also confirm that. Prompting "find me bugs" or stuff like that almost never works. It works better if I get an error and ask it to explain it to me, giving the application context. The static analyzer is there to give this initial kick, to create these nucleation sites in which the LLM will crystalize answers upon.
This sounds like the most viable, easy to make product that can find bugs with LLMs. It's only offensive if that's actually what these products are doing, it's not supposed to be known and I struck a nerve or something.