- In a past life I had a Wall of Shame of headlines on firmware update fails.
The lesson was you built firmware updates upfront and right into your development process so it became a non-event. You put in lots of tests, including automatic verification and rollback recovery. You made it so everyone was 100% comfortable pushing out updates, like every hour. It wasn't this big, scary release thing.
You did binary deltas so each update was small, and trickle download during down-time. You did A/B partitions, or if you had flash space, A/B/C updates (current firmware, new update, last known good one). Bricking devices and recalls are expensive and cause reputational damage. Adding OTA requires WiFi, BLE, or cell, which increases BOM cost and backend support. Trade-off is manual updates requiring dealership visits or on-site tech support calls with USB keys. It doesn't scale well. For consumer devices, it leads to lots of unpatched, out-of-date devices, increasing support costs and legal risk. OTA also lets you push out in stages and do blue-green deployment testing.
For security, you had on-device asymmetric encryption keys and signed each update, then rolled the keys so if someone reverse-engineered the firmware, it wouldn't be a total loss. Ideally add a TPM to the BOM with multiple key slots and a HW encryption engine. Anyone thinking about shipping unencrypted firmware, or baking symmetric encryption keys into firmware should be publicly flogged.
You also needed a data migration system so user-customizations aren't wiped out. My newish car, to this day, resets most user settings when it gets an OTA. No wonder people turn off automatic updates.
The really good systems also used realistic device simulators to measure impact before even pushing things out. And you definitely tested for communication failures and interruptions. Like, yoink out a power-line mid-update and then watch what happens after power is back on. Yes, it's costly and time-consuming, but consider the alternatives.
The ones that failed the most were when they spent months or years developing the basic system, then tacked on update at the end as part of deployment. Since firmware update wasn't as sexy as developing cool new tech, this was doled out to lower-tier devs who didn't know what they were doing. Also, doing it at the end of the project meant it was often the least-tested feature.
The other sin was waiting months before rolling out updates, so there were lots of changes packed into one update, which made a small failure have a huge blast radius.
These were all technical management failures. Designing a robust update system should be right up-front in the project plan, built by your best engineers, then including it in the CI/CD pipeline.
Just for context, the worst headline I had was for update failure in a line of hospital infant incubators.
- One of the main points of using a UUID as a record identifier was to make it so people couldn't guess adjacent records by incrementing or decrementing the ID. If you add sequential ordering, won't that defeat the purpose?
Seems like it would be wise to add caveats around using this form in external facing applications or APIs.
- Was going to ask how much all this cost, but this sort of answers it:
> "Managing Cost and Usage Limits: Chaining agents, especially in a loop, will increase your token usage significantly. This means you’ll hit the usage caps on plans like Claude Pro/Max much faster. You need to be cognizant of this and decide if the trade-off—dramatically increased output and velocity at the cost of higher usage—is worth it."
- Any post about IoT security that doesn't mention or link to Shodan (https://www.shodan.io) is missing a lot of context. It's way worse than you think.
Also, with tools like Chip Whisperer (https://www.newae.com/chipwhisperer) the physical security of the hardware root of trust needs to be reevaluated.
- Relevant Calvin and Hobbes: https://www.gocomics.com/calvinandhobbes/1991/11/29
- Costco people recommended showing up early. Went there right at opening in the morning and there was plenty. To their credit, they haven't raised their prices.
Also, shocked to see people hoarding multiple boxes of 50 eggs. Hoping it was for a commercial purpose and not someone panic-buying like they were for TP during the pandemic.
- Lattner left Apple a long while ago. He's been working on Mojo, a different (Pythonic) language and runtime: https://www.modular.com
- I didn't mean to send you down a rabbit-hole since you've already built an awesome-looking platform. Just googling, came across a devkit: https://ameridroid.com/products/pinetime-smart-watch?variant...
The DevKit is mainly there to allow easy access the SWD pins. It can be flashed via a Pi: https://wiki.pine64.org/wiki/PineTime_Devkit_Wiring
Aaron Christophel (go-to guy for hacking wearables) has a video on flashing them via an ESP32: https://www.youtube.com/watch?v=Iu6RoXRZxOk
For OTA flash, some folks have had good luck with https://gadgetbridge.org
For those interested in going the cheap-o activity tracker route, here's another old thread on Hackaday, featuring Aaron. More good info in the comments: https://hackaday.com/2019/02/20/custom-firmware-for-cheap-fi...
Also, one last tidbit: I know the idea was to stay away from the more expensive devices, but many Apple and Android watches have built-in BLE central capability so they can scan and connect to peripherals without having to go through the phone. In later models, they can even scan in the background and update complications.
Again, not remotely as cool as building one's own device!
- Awesome work building it all from scratch!
Hate to be that guy, but was using something hackable like a PineTime (https://pine64.org/devices/pinetime/) and doing a direct BLE connection to the device using the Blood Glucose Profile (https://www.bluetooth.com/specifications/specs/glucose-profi...) not an option?
Another approach could be to hack one of those cheap fitness trackers (https://rbaron.net/blog/2018/05/27/Hacking-a-cheap-fitness-t...). I got a few off AliExpress a few years ago for $9 each.
Again, not to take away anything from the amazing accomplishment to build a device from scratch, but if anyone else wants to try making one, maybe they could start with less of a steep learning curve.
- I remember talking to techie friends about this a couple years ago. With the advent of AI screeners, it would only be a matter of time before candidates figured out how to craft and rewrite not just their cover letter, but their whole resume to semantically best-fit a job listing. It could even A/B test for the best response.
Everyone laughed and said it was too much work. I predicted it would be a YC company before long.
Only a matter of time before AIs will be talking to AIs to have a technical interview and negotiate salary.
- On semi-related news...
Santa Cruz Wharf’s fallen restroom becomes an unlikely tourist attraction: https://archive.ph/k1lwt
- Back when I was consulting, I always made sure to stay in touch separately with three people:
a) The internal project owner. Usually a VP or Director.
b) The owner's assistant/gatekeeper, who made sure they saw and signed documents.
c) The accounts payable person who actually pushed the button that cut the checks <- The most important one if you want to get paid on time.
I also never did more than Net-30 (and often Net-7). Things went smoothly with the larger companies (once everything was signed). Startups and medium-size ones, though, had no problem stiffing you on a bill if their funding was wobbly. Those were often payable on invoice, or upfront retainer.
Still got screwed multiple times. My own damn fault for taking on projects just because they sounded cool.
- Relatives were asking for a basic explainer. Here's a good one by Hannah Fry: https://youtu.be/1_gJp2uAjO0
- I'm a little confused. The baked-data model is so you DON'T have to generate a thousand static pages. But this solution does exactly that.
Not complaining, mind you. My kid is trying to learn HTML/CSS/JS and wants to put together a read-only website with a database backend. I'll be pointing him this way as an ootion once he's far enough.
But it's still puzzling to link it to baked-data. Maybe I'm missing something.
- When you went from Jekyll to Kubernetes, it was must have been like when Ted Kaczynski first learned about battery-operated timed fuses.
Enjoyed the read.
Looking forward to when you chuck the whole thing into the bin and run Wordpress on your linux home server, fronted by a free Cloudflare zero trust tunnel.
- Built an app when the iPhone first came out. Spent 2 months building the core app and another 3 months working to reduce the number of taps and remove road-bumps in the UI/UX flow.
Totally paid off.
Working on another app now. Sweating the details on the 'watercourse way.' That first experience is critical.
- RabbitMQ can also act as a native MQTT broker. For edge applications/devices, you can also use MQTT over websockets: https://www.rabbitmq.com/docs/web-mqtt
Edit: FWIW, NATS also supports MQTT: https://docs.nats.io/running-a-nats-service/configuration/mq...
- The amount of effort put into all the porting and reverse-engineering is impressive. Especially, getting the toolchain working. That's usually where most of the hair-pulling comes in.
For those wanting to hack their own tracker, for slightly more money ($5), you can get one with a Nordic nrf52832 which has lots more documentation and support, including Micropython:
- For mobile, this looks promising: https://skip.tools
Transpiles SwiftUI over to Kotlin/JetPack Compose. Native all the way down. Haven't used it in production, but the demos work and are native after going through XCode and Android Studio.
Would have been great if it had been open-sourced, but they paid for all the development and owned the codebase. They wanted to use it to dynamically generate content for every page for every unique device and client that hit their server. They had the infrastructure to do that for millions of users. The processing could be done on the server for plain web browsers or embedded inside a client binary app, so live rendering to native could be done on-device.
Back then, it was trivial to generate XML on-the-fly from a SQL-based database, then send that back, or render it to XHTML or any other custom presentation format via XSLT. Through XSD schema, the format was self-documenting and could be validated. XSLT also helped push the standardizing on XHTML and harness the chaos of mis-matched HTML versions in each browser. It was also a great way to inject semantic web tags into the output.
But I always thought it got dragged down with the overloaded weight of SOAP. Once REST and Node showed up, everyone headed for new pastures. Then JS in browsers begat SPAs, so rendering could be done in the front-end. Schema validation moved to ad-hoc tools like Swagger/OpenAPI. Sadly, we don't really have a semantic web alternative now and have to rely on best guesses via LLMs.
For a brief moment, it looked like the dream of a hyper-linked, interconnected, end-to-end structured, realtime semantic web might be realizable. Aaand, then it all went poof.
TL;DR: The XML/XSLT stack nailed a lot of the requirements. It just got too heavy and lost out to lighter-weight options.