Imagine there's some kind of problem. Say, an enterprising journalist shows that the system has somehow broken; 1/3 of its decisions are essentially random, rendering thousands of deserving workers homeless. What does Nevada do? Everyone who knew how to do reviews manually has gone on to other work or retirement.
That institutional knowledge isn't coming back. Before you can turn the AI off, you need to build a whole new human-driven review process from whatever legible information you have archived somewhere.
For a single process which is catastrophic only for improperly rejected individuals, this process is expensive but doable. But unemployment review is not the end; AI is not going to stop at doing the vast majority of the decision-making for only one single, non-vital process. Add in a few more--say, 911 dispatching, payroll processing, a few utilities--and putting them back under the control of competent humans will quickly grow beyond the capacity of a the state.
If anything, it reinforces my impression that AI’s true application is allowing institutions to kick the can down the road further, just with a novel excuse.
Isn’t the root problem here municipal debt, and/or societal debt?