And like I said, we've been where you want to be: .xls, in particular, is actually a very simple format at its core and easily inspectable with a hex dump or whatnot. Likewise Wordperfect's binary format back in the day was straightforward for humans to use, yet still binary. It didn't help. They still sucked.
To wit: you can't solve this problem your way. All you'll do is create another messy format, c.f. XKCD 927.
XLS as a format sucks because:
a) It was undocumented for most of its life
b) Undocumented even inside Microsoft because it consisted partly of memory dumps from the app
c) Was heavily optimised for fast loading and saving on very slow machines
100% of uses of JSON, translated to binary, would not suffer those issues. They'd have documented schemas, at least internally, they wouldn't be created by memcpy to disk, and they wouldn't be stuffed with app-specific loading optimisations.
I'm saying the idea that we as humans should view/consume the exact same thing raw and byte-by-byte without any transformations in between as computers do creates a problematic trade-off because that representation now has two very different masters, humans and machines.
If you want to make that representation pretty and friendly for humans it becomes hard to parse and consume for machines.
If you try to make it friendly for machines it becomes inaccessible, noisy and annoying for humans.
So you have to settle somewhere in between that is tolerable for both masters but not perfect for either one.
With some exaggeration, it's as if we required the binary representation of an image to be "raw viewable/consumable" by humans in a text editor.