X originally was created on/ran on a graphics terminal - the DEC VAXstation 100. The VS100 was quite different to the later X thin client terminals: it required an adapter card to be installed in a host system, and the software which ran on the VS100 could directly access a chunk of shared memory on the host.

Ports to workstations with inbuilt graphics hardware came later.

References:

[1] https://www.youtube.com/watch?v=cj02_UeUnGQ

[2] https://en.wikipedia.org/wiki/VAXstation#VAXstation_100