This is Palo Alto and his neighbors for the most part aren't "regular people". They all own $5M+ homes.
I have close relatives in that same neighborhood. They are regular people. Yes their house is now worth millions but it wasn’t when they bought it and they are not wealthy (unless they sell and move).
A $5M home in Palo Alto is hardly a mansion.
it is easier and safer to have illegal school and other unpermitted things and all the noise and street blocking and all the other disruptions where regular people live than to piss off a billionaire neighbor.