Would spaces magically solve this problem? I guess it would solve some things -- you'd no longer have to guess where to terminate the prefix search, and I think you're right about word shape -- but it would definitely not provide the additional semantic context you get from having the high-bit-density characters in the mix. This makes reading faster.
I suspect that one could make a kana-only writing system that would be functional enough, but it would still be slower to read than mixed script. Also, the Korean comparison isn't exactly valid -- Korean has more sounds than Japanese. It seems minor, but Japanese has a ton of homonyms because of the tiny phoneme. Expanding that, even slightly, would be a benefit to reading.
It's somewhat hard to believe that Japanese sits in some magic spot where a phonetic script wouldn't work just fine when Korean does it fine, and on the Sinitic side people write books in pinyin, Vietnamese is phonetic, and the Dungan people write their 3-tone Mandarin dialect with cyrillic alphabet without even notating tones.
> Maybe you're right that it's all just hard-headed stubbornness from fluent people.
It's not just hard-headed stubbornness - reading kana really is more difficult to proficient readers of today's Japanese, and change is work.
Every competent reader of Japanese is first and foremost used to the kanji-hiragana mixed script, and has shortcuts for the kanji forms of words and the sounds of those words. The hiragana only forms? Not so much. So when they complain about hiragana only being hard to read, they're not lying. It really is harder. But it's not harder due to any inherent defect in a hiragana-only script, it's just about a lack of exposure to form those shortcuts that make reading feel easy.
Similar arguments used to be made in Korea, yet if you look at Koreans today they have no difficulty in reading hangeul - they have spaces so the words have form, and they have mass exposure to the hangeul forms of words. Ergo, their shortcuts are for the phonetic forms and those are what feels natural, solid, meaningful and easy to read. Same as both of us in English or me in say, Finnish or Swedish or French.