Not only that, but you should not get suckered down into overcomplicating things by chasing complex novelties, e. g. integrated slider- or clamshell-implemented keyboards, silly and outdated form factors (clamshell UMPCs, OQO already showed the way), etc.
You want a good, small keyboard? Design it to be attachable. This is possible in a variety of ways and can be adapted to your manufacturing expertise. It also leaves open third-party hardware support for your device. Not to mention maintainability/repairability. It's utterly puzzling to me how many hardware start-ups already fuck up the basics.
And never forget: In a satured market, even catering to a niche, means you should go for a somewhat unique feature set. How many ultramobile devices are out there that are truly accessible and usable? That goes beyond just safety or repairability.
OLED screen? I'd rather prefer something PWM-free. Precision control? Digitizer/stylus support. You don't even need to house the stylus in the device. But it would be very useful to have at least one. Audio? Yeah, 3.5 mm is a must. Dedicated, easy-access mSD (Express) card slot? Yes, please. Exchangeable batteries? Good idea, as long as it's a standard design in good supply. Kill switches. Maybe a modular camera set up like those Chinese flagships that are otherwise rather useless. Full-feature connectivity (1-2 x USB 4). Etc.
Get one of the BB + USB-C keyboards available.
But then again, experience shows I'm wrong.
If you do decide to buy one, the hardware (of the Pocket 2 at least) is OK. Easy to open and such. You can even buy replacement battery. Buy one from Amazon in EU if you can, they have to provide 2 years warranty, at least.
You need to operate this as a business first, with the freedom part being a nice bonus. Nobody cares how free your thing is if it's dead on arrival and gets beaten by an entry-level smartphone.
Make a competitive product. Nowadays that could very well just mean Android with manufacturer-sanctioned root access and preinstalled terminal & X/wayland server for those who want to run desktop apps.
The Jolla phone someone linked below actually looks like a decent product. The Android app support means it's actually usable in the modern world, and the specs look competitive.