Why do you believe that this will happen?
The usual response for people whose ideologies have lead to problems, is to double down on their ideologies. After which cognitive dissonance makes it easy to reject all evidence of their own mistakes.
But your point is that most people are not zealots. True. But we live in a country which, thanks to gerrymandering, mostly sees politicians lose re-election in the primary. And since primaries bring out the most motivated people in your own party, that means that you are most likely to be replaced by someone who is more to the taste of zealots. And the sight of the other party becoming more extreme, breeds future zealots to continue the process.
This dynamic is one of the key reasons why politicians have become more extreme over the last couple of decades. With the result that, despite the fact that most of us would gladly vote for a more moderate person, in an election we're forced to choose between extremes. And for over 80% of us, our choice truly doesn't matter. We live in districts that have a sufficient partisan lean that the outcome is essentially predetermined.
It doesn't matter how polarized "both sides" are. Both sides is a subset of the whole society. As they become more and more insane in their polarization, more and more of society realizes that they are insane, and becomes repulsed. The pendulum is restored not by the other side becoming dominant. The pendulum swings back by the middle looking at the extreme and saying "No, you're insane, we aren't going to walk down your road."