Hacker Neue

Preferences
usrbinbash parent
> In a vacuum these things go totally nuts

And that is, in a nutshell, my point. An AGI has to be autonomous. It cannot "go nuts" without handholding, same as a human needs to be able to (under normal operating conditions) remain coherent, even if left to their own devices.

> the environment deserves just as much thought in any analysis of an AI-based system.

Couldn't agree more, and since I know how much work these environments are to build, the people doing so well, have at least as much of my respect as the ones who devise the models.

But again, and I'm sorry I am pulling the "definition and meaning" card again: We cannot devise a system that requires a tight corset of an execution environment keeping tabs on it all the time lest it goes bananas, and still call it an AGI. Humans don't work that way, and no matter how we define "AGI", in the end I think we can agree that "something like how we do thinking" is pretty close to any valid definition, no?

If I need to lock something in 10 days to sunday to prevent it from going off the rails, I cannot really call it an AGI.

galangalalgol
Are we sure people don't work that way? Almost all of us operate on instincts almost all of the time. We have guardrails, people who operate them are often committed to institutions. When we choose to do things, it is based on that static hardwiring. Our meta model later comes up with reasons why we did the things. Sometimes, but rarely, it is correct. The human brain is extremely heterogenous, modular even. Some of our modules function remarkably like a memory store fed back into a context window. Adding a meta model to an llm that is updated autonomously by an additional model that analyzes outcomes to upde this predictive meta model would quite likely result in the agent's models mistaking the meta model for a self. Much like we do.
bubblyworld
Haha, this is the weird thing about definition debates, you often don't disagree about anything substantial =P thanks for the measured response.

> An AGI has to be autonomous. It cannot "go nuts" without handholding [...]

So I think this is where I get off your bus - regardless of what you call it, I think current agentic systems like claude code are already there. They can construct their own handholds as they go. I have a section in all my CLAUDE.md files that tells them to always develop within a feedback loop like a test, and to set it up themselves if necessary, for instance. It works remarkably well!

There are lots of aspects of human cognition they don't seem to share... like curiousity or a drive for survival (hopefully lol). And creativity is very bad right now - although even there I think there's evidence it has some ability to be creative. So if you want that in your AGI, yeah, it's got a ways to go.

Situation seems very murky for an impossibility theorem though (to me).

> in the end I think we can agree that "something like how we do thinking" is pretty close to any valid definition, no?

I agree, we aren't even close to human-level ability here. I just think that people get hung up on looking at a bunch of tensors, but to me at least the real complexity is when these things embed in an environment.

All these arguments considering pure Turing machines miss this, I think. You don't study ecology by taking organisms out individully and cutting them up. There's value in that, of course, but the interactions are where the really interesting stuff happens.

This item has no comments currently.