> they start spicing the soup with lead to give it cinnamon flavor or whatever
Like, we all know lead is bad, and we all know that humans are unscrupulous, but at least the human putting lead in the soup knows they're being unscrupulous at that time (probably). For an AI it would just be an entirely favorable optimization.
We're going to find out how far we can trust them, and the limits of that trust will determine where the people need to be.
- consolidation, such that there are only a few different choices of soup
- a race to the bottom in quality
- poisoning
These are all possibilities under our current system, and we have mechanisms (laws and market competition) which limit the extent to which they occur.
What is it about extreme automation technology that you think will increase these prevalence of these issues? By what mechanisms will these issues occur more frequently (rather than less frequently), as production technology becomes more capable?
A lot of people think wealth inequality isn't a big deal, but I disagree. The more proportion of money a select few have in comparison to everyone else, the higher the likelihood those select few can mold society to their whim. Corruption thrives off of wealth inequality. Without it, it cannot exist.
This is a decent point, but you're describing the world we already live in, no? I mean, we already have significant automation, significant wealth inequality, significant ability for control of money to affect legislation and culture.
But we (in the US at least) generally have abundant access to food which is safe to eat.
> Corruption thrives off of wealth inequality. Without it, it cannot exist.
Corruption can exist without wealth inequality.
Consider a city where teachers and politicians earn the exact same salary (and the same ability to build wealth over time). You might think this eliminates the potential for corruption, but imagine a scenario where politicians heavily rely on teachers' unions for their election campaigns. In this case, politicians might make decisions that benefit the unions (e.g. no performance standards, lifetime tenure, long holidays, keeping underenrolled schools open) in exchange for their support, even if those decisions don't optimize for student achievement (or whatever else taxpayers want schools to promote).
There will not be a "quality" dial that you get to tweak to decide on your perfect quality of soup. There will be graduations, and you will be stuck with whatever the store provides. If you want medium quality soup, but the store only carries 3 brands of soup (because unlike in your utopia somebody actually has to maintain an inventory and relationships with their supply chain) and your favourite brand decides to bottom out their quality. It's not "good actually" because of economic whatever. Your soup just sucks now.
Oh but "the market will eventually provide a new brand" is a terrible strategy when they start spicing the soup with lead to give it cinnamon flavor or whatever.
I'm not an ethereal being. I'm a human, I need it to be good now. Not in theory land.