If an AI is advanced enough to design mass production ready fusion reactors, I can't help but feel that the "human in charge of that AI" quickly becomes fully redundant. There is nothing that human can do that the AI itself can't do faster, better and cheaper.
This idea of "AI taking control" reminds me a lot what Karl Marx would say about "capital".
He wrote his things during the blooming of the second industrial revolution, a time when machines were replacing humans and forcing forward new economic, social, political, cultural and labour relations. And a key issue he stressed a lot is that this diffusion of machines and capital reshaping society was brought forward by a class of people that he called the bourgeoisie. He stressed a lot that it was a power struggle within society.
We're going through something similar today with the information technologies reshaping social relations. And the Bezos/Zuckerberg/Altman/Ellison of today are similar to the industrial barons from the Gilded Age. But, the same way that people reacted against the full-blown wild capitalism from 19th century's second half, we might also see some reactions against the advance of this techno-plutocracy.
In particular, I am optimistic about how the EU and some 3rd World countries (e.g. India, Brazil) are placing restrictions on social networks and techno-cartels.
> What is the value of you in this system?
So, to answer your question: individually I can't go beyond much more than careful choices (avoid cookies, stay out of Facebook, etc). Collectively we can make political choices. Ultimately, the most consequential political choice is move away from countries that give all power to the techno barons.
But if I can't understand how and why the fusion reactor is safe, small and cheap, I wouldn't consider it safe.
Very much the same way that I don't take Claude Code's changes to my code without understanding what it does.
Augmenting my intelligence is non-negotiable. I want to be in control.
It is the classic "trust but verify". And I need my own intelligence to verify.