And that is, in a nutshell, my point. An AGI has to be autonomous. It cannot "go nuts" without handholding, same as a human needs to be able to (under normal operating conditions) remain coherent, even if left to their own devices.
> the environment deserves just as much thought in any analysis of an AI-based system.
Couldn't agree more, and since I know how much work these environments are to build, the people doing so well, have at least as much of my respect as the ones who devise the models.
But again, and I'm sorry I am pulling the "definition and meaning" card again: We cannot devise a system that requires a tight corset of an execution environment keeping tabs on it all the time lest it goes bananas, and still call it an AGI. Humans don't work that way, and no matter how we define "AGI", in the end I think we can agree that "something like how we do thinking" is pretty close to any valid definition, no?
If I need to lock something in 10 days to sunday to prevent it from going off the rails, I cannot really call it an AGI.
> An AGI has to be autonomous. It cannot "go nuts" without handholding [...]
So I think this is where I get off your bus - regardless of what you call it, I think current agentic systems like claude code are already there. They can construct their own handholds as they go. I have a section in all my CLAUDE.md files that tells them to always develop within a feedback loop like a test, and to set it up themselves if necessary, for instance. It works remarkably well!
There are lots of aspects of human cognition they don't seem to share... like curiousity or a drive for survival (hopefully lol). And creativity is very bad right now - although even there I think there's evidence it has some ability to be creative. So if you want that in your AGI, yeah, it's got a ways to go.
Situation seems very murky for an impossibility theorem though (to me).
> in the end I think we can agree that "something like how we do thinking" is pretty close to any valid definition, no?
I agree, we aren't even close to human-level ability here. I just think that people get hung up on looking at a bunch of tensors, but to me at least the real complexity is when these things embed in an environment.
All these arguments considering pure Turing machines miss this, I think. You don't study ecology by taking organisms out individully and cutting them up. There's value in that, of course, but the interactions are where the really interesting stuff happens.
> And that observable behavior includes hallucinations, a tendency to be repettive, falling for leading questions [...]
I agree with you, obviously, these are common behaviours. You can improve the outcomes a lot with tight feedback loops for development workflows (like fast-running tests and linting/formatting for the agent to code against). In a vacuum these things go totally nuts - part of the reason I think the environment deserves just as much thought in any analysis of an AI-based system!
> Go run an agentic workflow using RAG on a local model. Do an md5 checksum of the model before and after usage. The result will be the same.
As I said in my last comment, I agree with you. The md5 checksum of the tensors won't change. If your workflow accomplished anything at all, however, there will be many changes elsewhere in the system and it's environment (like your codebase). And those changes will in turn affect the future execution of workflows. Nothing controversial here.