You're using a file format designed three decades ago to make arguments about what we should be doing today.
XLS as a format sucks because:
a) It was undocumented for most of its life
b) Undocumented even inside Microsoft because it consisted partly of memory dumps from the app
c) Was heavily optimised for fast loading and saving on very slow machines
100% of uses of JSON, translated to binary, would not suffer those issues. They'd have documented schemas, at least internally, they wouldn't be created by memcpy to disk, and they wouldn't be stuffed with app-specific loading optimisations.
And like I said, we've been where you want to be: .xls, in particular, is actually a very simple format at its core and easily inspectable with a hex dump or whatnot. Likewise Wordperfect's binary format back in the day was straightforward for humans to use, yet still binary. It didn't help. They still sucked.
To wit: you can't solve this problem your way. All you'll do is create another messy format, c.f. XKCD 927.