I once told one of their marketing guys "Quit worrying about being pirated and worry more about being ignored."
I wonder if a business model like Epic's would have worked. Unreal Engine is free until you hit US$1 million in revenue, and then they want a cut of revenue. This works in games because any game with significant revenue is publicly visible. Less so in operating systems. You could have QNX inside a million traffic lights without anybody knowing.
The main application we wrote (and of which I rewrote significant chunks) processed a record number of messages on every Monday morning, week after week, year after year. It was a massive cash cow for the company that had commissioned it (KVSA, https://www.kvsa.nl/en/data-and-intelligence/bts-broker-tool... , it is still alive but it no longer runs on QNX) and if not for QNX I doubt it would have gotten as far as it did. Especially in those first few years when it was mostly my old boss and one or two helpers building it. The OS pretty much enforced clean abstraction layers by showing at the OS level itself how this was done. Extending that was a natural path and I suspect that many QNX shops did this simply because it felt right and made it easy. They likely did not realize they were building what we would call a 'service oriented architecture' many years later and in a completely different context.
This experience still leaves me looking at many of the systems we use today as unnecessarily bloated and frankly, quite ugly. QNX was light weight, but not a lightweight. The only other eco-systems that give me that same feeling are Plan9 and Erlang.