What would a clean-slate implementation of a serial terminal look like today?
I work in embedded, and a serial terminal is practically a must in any system, if only for debug. A way to provide input and receive output.
You'll definitely want a FIFOs for RX/TX. You also likely want control chars and a way to shortcut the FIFO, if only to interrupt output. The FIFO means you can abstract away the exact physical communication medium.
TUI interfaces mean you also want a way to tell the program what the dimensions of your screen are, through the control chars.
What else?
R0b0t1
I've given this some thought. The way to improve the TTY is to give it channels, like SSH or RDP. This way you can transit unrelated data over the same connection. And, if you think about it, most of the complexity of a serial interface is handling mode settings and state; streams abstract that away for you. Streams get rid of the need for control characters and in band signalling. Streams could be given priority and interrupt levels.
gonzus
A big fat YES from me. I have been looking into TUIs lately, and the state of terminal handling is ridiculously bogged down by years of cruft that, honestly, I don't give a shit about anymore. To put it succinctly: I want Alt-Home and Ctrl-Click to work on a TUI. Period.
If I had superpowers, I would layer the system as a very low-level library that recognizes keyboard / mouse / tablet / microphone input hardware events and encodes them somehow (so Alt is a separate key, pressing multiple keys is handled, clicking + moving the mouse is handled, etc); then a protocol that can give you this event stream, either from a local or remote source; then a layer that maps this into higher-level events (so Alt and Home pressed together is interpreted as Alt-Home). The idea would be to have the ability to detect, even over a serial connection, any of these input events, without resorting to escape codes.
I don't even know if this all makes sense; I just hate the loops you have to go through today to even try to make sense of key combinations.
/rant
ksherlock
If you add a header with the channel number and data length and a CRC, it starts looking like UDP.
tyingq
RS-485 is essentially RS-232 with channels. Never used it, but Linux supports it at some level.
nrclark
This might not be accurate. RS-485 is an electrical signaling standard for carrying arbitrary binary signals. It's usually used for carrying UART data, as a longer-range / higher-speed replacement for RS-232. It has no opinion about what is transmitted, and has no concept of channels or even any particular encoding format.
tyingq
It could be the basis for a tty with multiple channels, fairly easily.
RS-485 in itself really isn't. It has no concept of "channels" although it can be operated as multipoint.
You may be mixing this with Modbus which is typically run on RS-485.
zamadatix
Clean slate definitely object/packet based - just because we need to find the dimensions of the screen doesn't mean it needs to be done via control characters.
wanderingjew
Fairly easy, a few thousand lines of C. Bit of work, though. Parser is the hardest, a lot of details to get right. You don't really need a buffer for the output, because you can't type that fast.
Of course, driving the display is a bit harder. I'm using a parallel tft, wrote the driver, too. Keyboard code is kludgy because I'm not using diodes for cost reduction on the bom.
You can see it here: violence.works yes that's a url.
girvo
I'm both amazed at what you've accomplished and a little miffed that someone beat me to it: I've been working on-and-off on a similar project for a couple of years now haha. Awesome work
What display did you end up settling on, where'd you source it, and how many columns do you end up fitting? That was the biggest sticking point for me.
kragen
Are you talking about a physical machine or a protocol design? I don't think you can abstract away the exact physical communication medium from a physical machine (or your cable won't fit in the hole, or it'll burn out your circuitry), and the protocol design probably doesn't need a FIFO.
Protocol design: maybe typewriter text and shitty VT100 escape sequences were a bad idea? Let's use HTML-style tags, but in an endless tag soup similar to VT100 semantics, not a tree.
AceJohnny2
I was thinking mostly protocol design, I suppose, with the implicit requirement that it'd have to work over various physical lines.
kragen
I have lots of ideas.
But we have 55 years of experience with the advantages and disadvantages of the ADM3A, VT100, and similar terminals, and we've gradually improved it over that time. Now it's ten thousand times faster than a physical VT100, supports 24-bit color, auto-highlights URLs, has enormous scrollback, supports copy and paste, allows applications to handle mouse events, can be recorded with script or ttyrec or asciinema, can be detached and reattached later with screen or tmux or mosh, can get local echo with mosh, can share a screen and keyboard between people in different locations with screen or tmux, and has hundreds of thousands of programs already written for it — aside from backward-compatibility with teletype software like ex, ls, sh, grep, ps, ping, vmstat, and so on, you have user interfaces designed for character-cell terminals like Emacs, Vim, links, mtr, dstat, top, htop, dialog/whiptail, less, mutt, irssi, ircII, weechat, nethack, sl, robotfindskitten, command-line editing in things like tcsh and bash, progress bars in things like apt, nano, and of course the aa demo bb (if run with env -u DISPLAY).
There are a lot of things that can't really be done without breaking compatibility with that ecosystem. Proportional fonts of continuously varying sizes, or padding? Now screen positions are no longer integer multiples of a character size, and you can't get things to vertically align by inserting space characters. (This is already a problem with Unicode, where different monospace fonts are monospace over different amounts of the character set; see http://canonical.org/~kragen/sw/dev3/randomlayout.py for some attempts to test the limits.) Pixel graphics? Are those going to pass through tmux unharmed? (Some recent attempts to jimmy pixel graphics into terminals won't even work over ssh.) How will copy and paste work with them? Will they send another megabyte over my cellphone connection every time I press ^L to redraw the screen?
We can definitely do better. But doing better is going to require a lot of experimentation to be a Pareto improvement over 55 years of tweaks to the VT100. Most of the ideas we come up with are going to turn out not to work as well in practice as we hoped they would in theory.
And not just the terminal. Why on Earth people thought that the regular PC keyboard and the mouse required anything more complex than the good old RS-232.
gumby
This is a pretty good article. A few comments:
> Teletypes were once connected across the world in a large network, called Telex, which was used for transferring commercial telegrams, but the teletypes weren't connected to any computers yet.
Telex was a private system disjoint from the telegram system which was, in most countries, run by the PTT (government agency called Post, Telegraph and Telephone). In the US and some other countries it was private, of course, where Western Union was the dominant player.
I remember staying in a hotel in India the early 80s that had four different systems (with its own model of printing terminals, each one a clunky electromechanical beast). One was telex and one some iron curtain alternative, I remember, but I don’t remember the other two.
> ... a backspace key is often useful. This could of course be implemented by the applications themselves, but in accordance with the UNIX design philosophy, applications should be kept as simple as possible. So as a convenience, the operating system provides an editing buffer and some rudimentary editing commands
This approach far predates Unix, and Unix got it from Multics. Mainframes implemented it in their channel controllers to save time on the CPU (it was central, after all), while minicomputers typically did it the way Unix did.
Non-“glass tty” video terminals didn’t have frame buffers in the sense today, but just character storage. Characters were rendered by looking up pixel data in a ROM (before that, hardwared!) which might only have been a rectangle of 32 pixels or less.
> In a similar way, it is possible to configure the TTY to react on a stop flow byte and a start flow byte. These are typically ^S (ASCII code 19) and ^Q (ASCII code 17) respectively. Old hardware terminals transmit these bytes automatically, and expect the operating system to regulate its flow of data accordingly. This is called flow control, and it's the reason why your xterm sometimes appears to lock up when you accidentally press ^S.
I don’t remember any terminals that recognised soft (inline) flow control though they may have existed. Instead, they signaled it through the transmission cable itself. If you only used three wires in your serial cable this was impossible, but check out the full pin out of a DB-25 or DE-9 (sometimes ignorantly referred to as “DB9”) connector.
fuckf4ce
> I don’t remember any terminals that recognised soft (inline) flow control
XON/XOFF. The VT100, as well as many other contemporary terminals of the time did. Not uncommon at all.
This gets posted here every so often but to be honest it still isn’t demystified for me even on the third read.
galacticaactual
I really, really enjoy coming to HN for articles like this.
sn41
Oh yeah, I remember that this is where I really understood line discipline. For example, why backspace/Ctrl-H worked even if you are typing in things directly to a terminal (e.g. cat >> output file) and it did not need an editor to "interpret" the key, whereas things like Ctrl+A would need an editor. I know it's pretty basic, but this is the article that really helped me understand it.
timonoko
Sorry, I am still mystified. Especially with software flow control. Sometimes using emacs "search-forward" everything just stops and it takes time to figure out what just happened. But this phenomenon has becomen increasingly rare in recent years. It is truly mystifying.
u678u
Its amazing to me that despite our enormously powerful PCs and wonderful modern software tools we still rely on foundations built in a different time tty, awk, even bash scripts.
Half the time its great to work with such finely tuned Darwinian survivors, the other half I want to start again. Doing helm k8s yaml templates makes me wonder if its possible for us to write something simple and clean that could replace those old building blocks.
gpanders
I won't defend TTY or bash, but I will say that awk has stood the test of time because it's an amazingly potent tool for doing ad-hoc processing of textual information, which is still a frequent task even in the era of Kubernetes and yaml templates. Why does it need replacing?
When I learned awk I felt like I unlocked a superpower.
saurik
Yeah: it's like, the problem we are trying to solve largely hasn't changed, so the solutions are maybe a bit lacking in elegance sometimes, but they are generally fine... and the concern would be that someone trying to do it today would be like "the correct way to do this is to have some ridiculously complex stack built on top of 70 pseudo-standards from Google with a single bespoke implementation in JavaScript that no one could ever reimplement".
anthk
Windows 10 is still using the CP/M unit labeling scheme.
kjeetgill
I definitely found jq to be my modern addition to the pantheon alongside awk, sed, and grep.
I work in embedded, and a serial terminal is practically a must in any system, if only for debug. A way to provide input and receive output.
You'll definitely want a FIFOs for RX/TX. You also likely want control chars and a way to shortcut the FIFO, if only to interrupt output. The FIFO means you can abstract away the exact physical communication medium.
TUI interfaces mean you also want a way to tell the program what the dimensions of your screen are, through the control chars.
What else?
If I had superpowers, I would layer the system as a very low-level library that recognizes keyboard / mouse / tablet / microphone input hardware events and encodes them somehow (so Alt is a separate key, pressing multiple keys is handled, clicking + moving the mouse is handled, etc); then a protocol that can give you this event stream, either from a local or remote source; then a layer that maps this into higher-level events (so Alt and Home pressed together is interpreted as Alt-Home). The idea would be to have the ability to detect, even over a serial connection, any of these input events, without resorting to escape codes.
I don't even know if this all makes sense; I just hate the loops you have to go through today to even try to make sense of key combinations.
/rant
You may be mixing this with Modbus which is typically run on RS-485.
Of course, driving the display is a bit harder. I'm using a parallel tft, wrote the driver, too. Keyboard code is kludgy because I'm not using diodes for cost reduction on the bom.
You can see it here: violence.works yes that's a url.
What display did you end up settling on, where'd you source it, and how many columns do you end up fitting? That was the biggest sticking point for me.
Protocol design: maybe typewriter text and shitty VT100 escape sequences were a bad idea? Let's use HTML-style tags, but in an endless tag soup similar to VT100 semantics, not a tree.
But we have 55 years of experience with the advantages and disadvantages of the ADM3A, VT100, and similar terminals, and we've gradually improved it over that time. Now it's ten thousand times faster than a physical VT100, supports 24-bit color, auto-highlights URLs, has enormous scrollback, supports copy and paste, allows applications to handle mouse events, can be recorded with script or ttyrec or asciinema, can be detached and reattached later with screen or tmux or mosh, can get local echo with mosh, can share a screen and keyboard between people in different locations with screen or tmux, and has hundreds of thousands of programs already written for it — aside from backward-compatibility with teletype software like ex, ls, sh, grep, ps, ping, vmstat, and so on, you have user interfaces designed for character-cell terminals like Emacs, Vim, links, mtr, dstat, top, htop, dialog/whiptail, less, mutt, irssi, ircII, weechat, nethack, sl, robotfindskitten, command-line editing in things like tcsh and bash, progress bars in things like apt, nano, and of course the aa demo bb (if run with env -u DISPLAY).
There are a lot of things that can't really be done without breaking compatibility with that ecosystem. Proportional fonts of continuously varying sizes, or padding? Now screen positions are no longer integer multiples of a character size, and you can't get things to vertically align by inserting space characters. (This is already a problem with Unicode, where different monospace fonts are monospace over different amounts of the character set; see http://canonical.org/~kragen/sw/dev3/randomlayout.py for some attempts to test the limits.) Pixel graphics? Are those going to pass through tmux unharmed? (Some recent attempts to jimmy pixel graphics into terminals won't even work over ssh.) How will copy and paste work with them? Will they send another megabyte over my cellphone connection every time I press ^L to redraw the screen?
We can definitely do better. But doing better is going to require a lot of experimentation to be a Pareto improvement over 55 years of tweaks to the VT100. Most of the ideas we come up with are going to turn out not to work as well in practice as we hoped they would in theory.
But if you're interested in reading them anyway, some of my notes are in https://dercuano.github.io/topics/terminals.html.
> Teletypes were once connected across the world in a large network, called Telex, which was used for transferring commercial telegrams, but the teletypes weren't connected to any computers yet.
Telex was a private system disjoint from the telegram system which was, in most countries, run by the PTT (government agency called Post, Telegraph and Telephone). In the US and some other countries it was private, of course, where Western Union was the dominant player.
I remember staying in a hotel in India the early 80s that had four different systems (with its own model of printing terminals, each one a clunky electromechanical beast). One was telex and one some iron curtain alternative, I remember, but I don’t remember the other two.
> ... a backspace key is often useful. This could of course be implemented by the applications themselves, but in accordance with the UNIX design philosophy, applications should be kept as simple as possible. So as a convenience, the operating system provides an editing buffer and some rudimentary editing commands
This approach far predates Unix, and Unix got it from Multics. Mainframes implemented it in their channel controllers to save time on the CPU (it was central, after all), while minicomputers typically did it the way Unix did.
Non-“glass tty” video terminals didn’t have frame buffers in the sense today, but just character storage. Characters were rendered by looking up pixel data in a ROM (before that, hardwared!) which might only have been a rectangle of 32 pixels or less.
> In a similar way, it is possible to configure the TTY to react on a stop flow byte and a start flow byte. These are typically ^S (ASCII code 19) and ^Q (ASCII code 17) respectively. Old hardware terminals transmit these bytes automatically, and expect the operating system to regulate its flow of data accordingly. This is called flow control, and it's the reason why your xterm sometimes appears to lock up when you accidentally press ^S.
I don’t remember any terminals that recognised soft (inline) flow control though they may have existed. Instead, they signaled it through the transmission cable itself. If you only used three wires in your serial cable this was impossible, but check out the full pin out of a DB-25 or DE-9 (sometimes ignorantly referred to as “DB9”) connector.
XON/XOFF. The VT100, as well as many other contemporary terminals of the time did. Not uncommon at all.
http://bitsavers.trailing-edge.com/pdf/dec/terminal/vt100/EK...
Previous discussions: https://hn.algolia.com/?q=TTY+Demystified
The TTY demystified - https://www.hackerneue.com/item?id=8094186 - July 2014 (21 comments)
The TTY demystified - https://www.hackerneue.com/item?id=4062981 - June 2012 (17 comments)
The TTY demystified - https://www.hackerneue.com/item?id=658155 - June 2009 (13 comments)
Half the time its great to work with such finely tuned Darwinian survivors, the other half I want to start again. Doing helm k8s yaml templates makes me wonder if its possible for us to write something simple and clean that could replace those old building blocks.
When I learned awk I felt like I unlocked a superpower.