Today we want to share our brand new Python library providing ultra-lightweight durable execution.
https://github.com/dbos-inc/dbos-transact-py
Durable execution means your program is resilient to any failure. If it is ever interrupted or crashes, all your workflows will automatically resume from the last completed step. If you want to see durable execution in action, check out this demo app:
https://demo-widget-store.cloud.dbos.dev/
Or if you’re like me and want to skip straight to the Python decorators in action, here’s the demo app’s backend – an online store with reliability and correctness in just 200 LOC:
https://github.com/dbos-inc/dbos-demo-apps/blob/main/python/...
Don't want to keep reading and just try it out:
https://console.dbos.dev/launch
No matter how many times you try to crash it, it always resumes from exactly where it left off! And yes, that button really does crash the app.
Under the hood, this works by storing your program's execution state (which workflows are currently executing and which steps they've completed) in a Postgres database. So all you need to use it is a Postgres database to connect to—there's no need for a "workflow server." This approach is also incredibly fast, for example 25x faster than AWS Step Functions.
Some more cool features include:
* Scheduled jobs—run your workflows exactly-once per time interval, no more need for cron.
* Exactly-once event processing—use workflows to process incoming events (for example, from a Kafka topic) exactly-once. No more need for complex code to avoid repeated processing
* Observability—all workflows automatically emit OpenTelemetry traces.
Docs: https://docs.dbos.dev/
Examples: https://docs.dbos.dev/examples
We also have a webinar on Thursday where we will walk through the new library, you can sign up here: https://www.dbos.dev/webcast/dbos-transact-python
We'd love to hear what you think! We’ll be in the comments for the rest of the day to answer any questions you may have.
Ask me anything!
I do a lot of consulting on Kafka-related architectures and really like the concept of DBOS.
Customers tend to hit a wall of complexity when they want to actually use their streaming data (as distinct from simply piping it into a DWH).. being able to delegate a lot of that complexity to the lower layers is very appealing.
Would DBOS align with / complement these types of Kafka streaming pipelines or are you addressing a different need?
In fact, one of our first customers used DBOS to build an event processing pipeline from Kafka. They hit the "wall of complexity" you described trying to persist events from Kafka to multiple backend data stores and services. DBOS made it much simpler because they could just write (and serverlessly deploy) durable workflows that ran exactly-once per Kafka message.
How would you compare DBOS with that ?
There isn't a lot of public information about how they are built, but from what I can tell you're right -- their architecture is more oriented for small projects.
It looks like they store the entire JS heap in a SQLLite database. We store schematized state checkpoints in Postgres compatible database, which makes it so that we can scale up and allow interesting things like querying the previous states and time travel debugging, where you can actually step though previously run workflows.
From what I can tell, the programming model seems to be pretty similar but DBOS doesn't require a centralized workflow server, just serverless functions?
Great question! Yeah, the biggest difference is that DBOS doesn't require a centralized workflow server, but does all orchestration directly in your functions (through the decorators), storing your program's execution state in Postgres. Implications are:
1. Performance. A state transition in DBOS requires only a database write (~1 ms) whereas in Temporal it requires a roundtrip and dispatch from the workflow servers (tens of ms -- https://community.temporal.io/t/low-latency-stateless-workfl...). 2. Simplicity. All you need to run DBOS is Postgres. You can run locally or serverlessly deploy your app to our hosted cloud offering.
definitely a fan of what these types of systems can do in replay/recovering and retying steps etc. as well as centralizing a lot of didferent workloads to a common execution engine.
More info here: https://docs.dbos.dev/explanations/how-workflows-work
https://github.com/electric-sql/pglite
And how the heck are you maintaining Typescript and Python copies? lol
https://docs.dbos.dev/cloud-tutorials/timetravel-debugging
One neat thing about starting a child workflow is you can assign an idemopotency ID, which might be intentionally calculated in a way such that multiple parents will only start one run of the child workflow.
Then the DBOS cloud platform optimizes those interactions between the database and code so that you get a superior experience to running locally.
We know of users running puppeeter to scrap data.