It seems there's always a line able to be drawn from a view one disagrees with to the harm that is caused by not contradicting them, and in my experience this does not correlate with how harmful contradiction of the initial view actually is.

So with regard to the paradox of tolerance, I would say one needs to establish that their opponent's views are actually sufficiently intolerant to intrinsically be harmful - and this is not something one can always trust to be done without exaggeration.