I’ve seen it solve a complex domain specific problem and build a basis of code in 10 minutes what took a year for a human to do. And it did it better.
I’ve also seen it fuck up in the same way you describe. So do I weigh and balance these two pieces of contrasting evidence to form a logical conclusion? Or do I pick and choose one of pieces of evidence that is convenient to my world view? What should I do? Actually why don’t you tell me what you ended up doing?
It will just spew over-confident sycophantic vomit. There is no attempt to reason. It’s all worthless nonsense.
It’s a fancy regurgitation machine and that will go completely off the rails when it steps outside of it’s training area. That’s it.