This is a good take. What models seem to be poor at is undoing their own thinking down a path even when they can test.

If you let a model write code, test it, identify bugs and fix them, you get an increasingly obtuse and complex code base where errors happen more. The more it iterate the worse it gets.

At the end of the day, written human language is a poor way of describing software. Even to a model. The code is the description.

At the moment we describe solutions we want to see to the models and they aren't that smart about translating that to an unambiguous form.

We are a long was off describing the problems and asking for a solution. Even when the model can test and iterate.