cat could tell but it probably wouldn't be a good design. Calling isatty is easy, and I think GNU cat already does so and buffers differently. But how can it tell what raw binary data did to your terminal? First you'd need to program all the different terminal types into cat (probably by using termcap) which isn't great simple "Unix" design to begin with. Then you can't just reset the terminal after running cat. You need to see if the binary data screwed up the terminal. That isn't in termcap and is probably intractable in the general case.

While catting actual random data is contrived, using ssh is not. What if the connection is broken while you're in vi. Vi switches to the alternate screen and then scrolling stops working. If you don't know what happened it's not clear why. How can ssh fix this other than acting like tmux and parsing every terminal command sequence? I'm not convinced there is a simple fix.

This doesn't even get into encoding issues. Some UTF-8 sequences are terminal commands. The UTF-8 encoding for U+00DF (the German eszett) contains a sequence which will cause some terminals to wait for a termination byte you wont ever send.

Plan 9 had the right idea: do away with terminal command sequences altogether. But that only works there because you can take your "terminal" and turn it into a GUI window.