The wild thing is that they'll probably find entirely novel versions of this:
> they start spicing the soup with lead to give it cinnamon flavor or whatever
Like, we all know lead is bad, and we all know that humans are unscrupulous, but at least the human putting lead in the soup knows they're being unscrupulous at that time (probably). For an AI it would just be an entirely favorable optimization.
We're going to find out how far we can trust them, and the limits of that trust will determine where the people need to be.
> they start spicing the soup with lead to give it cinnamon flavor or whatever
Like, we all know lead is bad, and we all know that humans are unscrupulous, but at least the human putting lead in the soup knows they're being unscrupulous at that time (probably). For an AI it would just be an entirely favorable optimization.
We're going to find out how far we can trust them, and the limits of that trust will determine where the people need to be.