Preferences

There is something wrong with the description of the laptop use (besides calling DOS a language) - on one hand they are running windows 10 with Win98 virtualized. Then they mention Thinkpads circa 2000. I'm pretty sure Win10 won't run on those machines, so I guess the old laptops are for the interface to the train computers - RS-232 is my guess (could be the parallel port). If all they are using the DOS software for is to dump a log from the train, then it sound to me, that sniffing the protocol would be fairly simple. Calling people from bay-area hacker spaces to give them a hand in creating something more modern.

> that sniffing the protocol would be fairly simple

ah yes, "fairly simple". The protocol is not the issue, the data within is. There are hundreds of datapoints, various switching logic, and tons more of 50 years of accumulated cruft. At least. All in a well tested, stable, and understood package. The documentation is probably pretty good but likely has small (but crucial) errors.

It's easy to assume that this would be a simple rewrite, but real engineering (not the kind we do on hacker news! lol) requires tons of planning, testing, and phasing in. This is dealing directly with people's safety after all! It's interfacing with a train control system, not a webapp that turns markdown into memes.

also

> RS-232 is my guess (could be the parallel port)

RS-232 is inherently a serial protocol. If you're using RS-232 since basically ever, it's either a built in serial port, some sort of serial headers, or some sort of usb-to-serial adapter. The parallel port has never been directly used for RS-232 - they are fundamentally different technologies.

In 1990, RS-232 used 25-pin ports (DB25), and the DB9 ports were "new". Why they used 25-pin ports for RS-232 I'm not sure, since only a fraction of the lines (less than 9) were actually used, probably some silly historical reason. 25-to-9-pin adapters were very common then.

The DB25 connector was also used for parallel ports as you said, which caused confusion sometimes. But on a standard PC, the printer port (parallel) was a 25-pin female port, whereas the serial ports (25 or 9 pin) were male.

I guess around 1995 then? Whenever they switched to DB25 to DE9 for serial.

> probably some silly historical reason

original rs232 spec had a lot more signals and capabilities that very few consumer products implemented. So you were expected to have the full connector, even if the lines weren't hooked up.

>I guess around 1995 then? Whenever they switched to DB25 to DE9 for serial.

There was a transition during the 90s where the DB25s were phased out and replaced with DE9s. Computers became more commonplace and popular, and they also became smaller, so the bulky 25-pin port fell out of favor.

The bulky 25 pin fell out of favor because some of its pins function was implemented in SW (new SW appeared which used SW flow control).
Hardware flow control was retained with DE-9. The lines that were dropped included an entire secondary communications channel and timing signals, the sorts of features that I doubt saw much use in personal computers in the 1980's (when it was mostly used for things like modems and printers) and would see even less use today (when it seems to be mostly used for diagnostics ports).
> I guess around 1995 then? Whenever they switched to DB25 to DE9 for serial.

1984.

I own a wacky serial cable that has both DB25 and DB9 connectors on both ends and it has saved my ass more than once in the last decade or so. I'm never giving it up.
DE9 ports - get the terminology right! - were introduced on the IBM PC/AT in 1984 so they weren't especially new in 1990.

Indeed, by 1990 DB25 serial had pretty much vanished from PCs, and only existed as a thing that was a pain in the arse to connect to over existing cabling.

> DE9 ports - get the terminology right!

Warms my heart to see that I'm not the only person who's a stickler for this :-)

As far as I understand there were actually lots of different "RS232" variations. Nowadays I believe everything is asynchronous, which makes many of the pins that used to be connected redundant; lines that carried things like clock or additional flow control related signals. I'm not an expert, I just noticed this looking at old NEC computers.
RS232 itself is a well-defined spec. But you're right. An lot of equipment existed that was only vaguely RS232 compatible. Some of it even predates RS232. For example, the old electromechanical teletype machines send and receive something close enough to RS232 line signalling that you can connect them to a modern computer's RS232 port, if you wire up a tiny analog circuit to convert from telegraph-style current impulses to RS232-style voltage levels. Just set the UART to 50 bps with 5 bit data and 2 stop bits. And don't send anything too soon after a carriage return! Not all UARTs can handle those settings, but that's why such options were in terminal programs back in the day.
One particular thing to note is that on anything IBM PC related the DB25 RS-232 connector has the wrong gender (apparently so that it can be connected directly to modem with DB25 extension cable instead of 1:1 M-M cable as would be expected for RS-232).
DB25 ports were also used for different (although related) standards like RS-422.
> DB9 port

trivia: believe it or not, it's actually called DE-9

Yes, but usage of the incorrect “DB-9” terminology was very widespread, almost to the extent that it could be considered an alternate term.
Well said!

Kids nowadays don't understand what it takes to build, battle-test and maintain complex systems over decades in operation. It is a marvel of Engineering (both HW and SW) and the experts are worth their weight in gold. There is much to learn from them.

Do you have any links to where we can read about its architecture and system as a whole?

They were using a ThinkPad from 2000, therefore I was guessing RS232 and not RS-424 or other industrial ports. A laptop for that age would also have a parallel port, so in theory it could be some bespoke parallel protocol, but pretty unlikely.

> It's easy to assume that this would be a simple rewrite, but real engineering

I said - "If all they are using the DOS software for, is to get a log-file from the train, then its probably easy".

The system is from the 70'ies so its very possible that they originally hooked up dumb terminals. Those are even harder to come by today and had no real way of getting data out of (hard-copy of what was on the screen to line-printer perhaps). Old terminal are also very clunky to haul around compared to even an old laptop. So the DOS program could very well just be a VT05 emulator, without any actual program logic built into it.

So no I was not suggesting a rewrite but rather hack something together like some of us old greybeards here on hackernews like to do.

A lot of old industrial equipment used PCs with parallel ports in place of custom controllers. Parallel ports that were on the motherboard and hooked into the southbridge chipset could be accessed by DMA and driven in hard real time which is otherwise practically impossible to achieve on a speculative processor running Windows. The drivers could convert some gcode to digital signals in a ring buffer driven by an interrupt and just let the parallel port output the 5V logic signals to amplifiers, motor drivers, and so on. This was especially important for a lot of machinery because only mass market CPUs were fast enough to i.e. run computer vision to calibrate and then interpolate tool paths for a pick and place machine that placed thousands of components a minute.

MachCNC was a popular CNC controller among hobbyists that exploited this feature, for example. It wasn’t until the explosion in the popularity of 3D printers that open source CNC controllers that could run on a cheap MCU became widely available (Klipper, Marlin, etc)

The manufacturers saved a lot in NRE and manufacturing costs because a driver is a lot cheaper to write than a low volume controller design - with the parallel port they just had to design a board with a parallel port input that fanned out to a bunch of connectors going to off the shelf motor drivers and stuff. It also meant a much better user experience for users and devs alike but starting with Windows 8 drivers could no longer access LPT via DMA for security reasons (everything on the southbridge was exposed including USB iirc).

I suspect theres some proprietary component for which the driver source code was lost so they can’t adapt it to run off a PCI parallel port card (my memory is fuzzy but DMA to southbridge works differently enough from PCI DMA that it’s not as simple as changing the port number) and USB adapters can’t provide the timing guarantees. They could use an off the shelf PCI FPGA card but parallel ports were extremely common all over industry especially with offices printers so it’s very battle tested hardware. Replacing that is just too risky because redundant safety systems are far better at covering for failures than making sure new bugs don’t slip through.

I once used libparport -- a Linux library with hooks into kernel modules -- to obtain sub microsecond timing on custom hardware on a ~2008 Xeon pc. The parallel port has, by modern standards, terrible bandwidth but excellent latency: I waited for an interrupt and then "did a thing" once it was sent. I remember testing the timing with a 500 MHz scope and realising that I couldn't actually measure the timing error or delay accurately once I'd calibrated the whole system.

The parallel port was basically wired as a general purpose ADC and DAC accessible by DMA, as the parent here says. Amazing kit at the time, and free in the context of expensive ADCs requiring custom hardware and difficulty interfacing in comparison.

Would agree with all of this.

The problem is that BART doesn't employ talented reverse engineers. There is now way they could afford them in that area.

Also: Talented reverse engineers are hard to find on a good day in any given area. I used to think that my mentors from the start of my tech career were just the baseline level of sophistication you'd expect of any decent software engineer. 20 years later I have seen how few people have the mindset to get into that sort of thing. (And it is a mindset much more than a level of intellect, IMO)

>Also: Talented reverse engineers are hard to find on a good day in any given area

Not true. Have you looked in Europe? It's full of them thanks to the tinker culture and the presence all big anti-malware companies (Esset, Avast, Avira, Bit-Defender, Kaspersky, etc). Plus every single cybersecurity consulting company out there in the world, big and small, has guys who can do reverse engineering for you. There's thousands of these companies, just look them up.

The reason you're probably not finding them is because you're probably not looking for them or that many juniors who don't have the "RE bug" don't stay in that gig professionally too long because, unless you're a zero-day finding God, cashing out six figure CVE bounties and giving Defcon talks, the pay is worse than a webdev doing CRUD since the demand for reverse engineering work is very low and not valued by customers as it's not the kind of job that impacts their sales or share price so they just look to offshore it to the lowest bidder.

I also left the field and moving to cloud engineering since the demand, pay, benefits, respect is so much better.

Not true. Have you looked in Europe?

I wonder to what extent that is still the case. When I graduated 15ish years ago it was definitely true that many of my classmates had those low level skills, and serval of them had written their Masters thesis on something related. However looking at people graduating from the same program today, I hardly see anybody doing that sort of work. Everything is either some sort of high level web stuff or machine learning.

Even the people I knew who had those skills haven't used them professionally in over decade since, as you say, the pay sucks.

>I wonder to what extent that is still the case.

It is. The university in my area has a famous master's program for cybersecurity and many grads there know reverse engineering to a degree. And there a re many such universities in Europe, plus all the Eastern European gray beards who work in the industry since they cut their teeth breaking DRM to pirate games in their youth.

I personally know at least 5 people who can do this work professionally at a consulting company.

I don't ride Bart anymore, but plenty of us would have done this work for free.. just to see a small chance at improving the system.

The dysfunction is broader than them being unable to afford engineering talent.

> plenty of us would have done this work for free

This is the interface to the train control system - a primary safety system - responsible for the lives of literally a million people on an average month. Are you prepared to spend months of your lives rigorously testing and phasing this system in? Are you prepared for the intense legal liability if something goes wrong?

If an individual engineer is put in a position where they can endanger lives by accident, there is something wrong with the management and systems in place around that person.

This is a team game. The people figuring out how the comms works and creating tests for those communications then hand their work off to the "rigor" team.

Or do like what the dutch railways are doing: run the old and new systems in parallel on real inputs and check if the outputs match. Then proceed to only use the output for the old system until you're really really sure there's no bugs.
>responsible for the lives of literally a million people on an average month

Even in 2019, BART never cracked more than 500K station exits per month, and now it is well below half that number[0]. And of those monthly rides, surely a majority of them were the same people taking multiple rides.

Additionally, even a fatal flaw in the software would most likely not result in every one on every train simultaneously dying or even being injured.

[0]https://mtc.ca.gov/tools-resources/data-tools/monthly-transp...

100,000s of thousands of people is still a lot of people. my bad

> Additionally, even a fatal flaw in the software would most likely not result in every one on every train simultaneously dying or even being injured.

I'm glad you're so willing to play fast and loose with safety, but I'm not. Not to mention the impact due to lack of trust when an accident happens! I'll leave train systems to engineers who are rigorously testing their implementations, not you, thanks :)

Sorry, I should have been clearer. "The work" here referred to assisting with the reverse engineering aspect of this.

I, and almost everyone else here, lack the domain expertise to be of any use in that.

  The problem is that BART doesn't employ talented reverse engineers. There is now way they could afford them in that area.
Well, no. Money is not the problem and BART doesn't need reverse engineers. They've already thrown away tens of millions of dollars on a replacement train control system (AATCS). Everyone's favorite defense contractor Raytheon was tasked with the project until they realized they couldn't make it work. Eventually Raytheon pawned the project off on Harmon, which was later bought by General Electric. GE shut the whole thing down because it was going nowhere.

I think most folks here are drastically underestimating the complexity involved in getting a train control system right (and how badly BART botched the original system). Considering that BART dates back to the 70s, DOS and Windows 98 were upgrades. Meanwhile Muni soldiered on with OS/2 for nearly a couple decades.

> I think most folks here are drastically underestimating the complexity involved in getting a train control system right

The phrase "butt-puckering" comes to mind. Ya some whizkid could come along and put something together but they'd be foolish to attempt, there is 1000 atm of pressure to not get one damn thing wrong. All the things currently wrong with the system are grandfathered. One single wrong thing in your new system and you are toasted.

Sure there are safety considerations (and plenty of dramatic safety events from BART), but I'd venture to guess even there it's rarely quite so black and white.

I was actually thinking more of day-to-day nuisance problems than anything else. Things like getting stuck at the station while the train's computer gizmos reboot, stop annunciators getting confused (or out of sync with each other), having to position the train manually within a station. Or even software failures as hardware wears out – shit like that almost always fails in a safe manner. Unfortunately even a safe failure can have massive consequences which raises the cost of failing fast and breaking thing often.

If you read to the end of the article you see:

"Finding replacement parts won’t be a problem and DOS is no longer necessary for Allen and his crew, but it will still be bittersweet to say goodbye to trains that they have lovingly kept alive."

So they don't actually use DOS anymore, maybe, or possibly this just means they don't care because the new trains will soon replace the old ones and then DOS will be gone too. It's a poorly written article even allowing for the usual tech mistakes but either interpretation would be reasonable.

The big problem it seems like is Windows Vista dropped legacy support drivers and 16-bit support fifteen years ago, and Windows 10 dropped even more legacy support drivers. So they keep the ThinkPads around because they can't find the drivers for some things anymore. I've run into that a few times, where old drivers were made specifically for DOS 7/8 or even DOS 6 to interface with custom fabricated CNC machines or injection mould machines. In the end the company that originally created those few hundred units no longer exists, and nobody has the floppies that held the original drivers or software. So you keep cloning the drives for when they fail and hoping there's enough near-spec hardware spares to hold out until a total system replacement.
>Calling people from bay-area hacker spaces to give them a hand in creating something more modern.

Tragically, from the small amount of time I've spent in SF, my understanding is this is unlikely to be their first instinct, even if it's a great idea. Doesn't the SF local government have a testy, if not outright hostile relationship with the tech worker/hacker/startup scene?

I've met people from either side of the discussion who pretty much entirely blame the opposite group (hackers or city workers) for why SF is so different now than 'back in the day.'

This is perhaps more specious than the "cops don't want to do their jobs because people are just too mean to them" song and dance. At the risk of putting too fine a point on it BART is not run as part of any city government, San Francisco or otherwise. You're pretty much tilting at windmills here.

Besides this sort of thing is well beyond the scope of folks creating subscription based fruit juicers.

Gotcha. BART being funded by the Bay Area Rapid Transit District is something I'd forgotten, and does make my point a lot less salient. It's been a long time since I was in SF, so it's possible my assessments here are way off base.

The main observation I was trying to make is that the public/private sector has always seemed to have a harder time working hand-in-hand in the Bay Area than it does in other cities I've lived in, whatever that reason may be.

This item has no comments currently.

Keyboard Shortcuts

Story Lists

j
Next story
k
Previous story
Shift+j
Last story
Shift+k
First story
o Enter
Go to story URL
c
Go to comments
u
Go to author

Navigation

Shift+t
Go to top stories
Shift+n
Go to new stories
Shift+b
Go to best stories
Shift+a
Go to Ask HN
Shift+s
Go to Show HN

Miscellaneous

?
Show this modal