I dunno: I have never ever wanted to make a compositor -- which, to me, feels like a really boring piece of graphics infrastructure -- and yet I have used multiple window managers over the years and have absolutely wanted to make my own window manager? In X, making your own window manager was so popular of an activity that it honestly felt kind of unreasonable just how many window managers existed, and yet everyone used one of a handful of compositors and I'm honestly not sure why anyone in their right mind would bother making their own display server?
for instance as a X11 user I don't want a compositor at all
(Same... I know people use them to get some pretty effects; but, they add a frame of latency I do not want and require lots of memory and assume acceleration I don't need.)
There is no way to avoid a frame of latency without "racing the beam", which AFAIK quite complicated and not compatible with most GUI frameworks. That is, if you don't want tearing.
But I may be wrong here
One frame of latency and adding a frame of latency are different things. The first is required (without tearing) the second should be avoided at all cost (athough high display refresh rates reduce the problem of "long" swapchains quite a bit).
Why not just make a display server (which handles everything rendering related, compositing included), and then add a window manager as a plugin/extension on top? Window managers are not that complicated.