18
points
kodablah
Joined 9,674 karma
https://github.com/cretz
- > that your entire workflow still needs to be idempotent
If just meaning workflow logic, as the article mentions it has to be deterministic, which implies idempotency but that is fine because workflow logic doesn't have side effects. But the side-effecting functions invoked from a workflow (what Temporal dubs "activities") of course _should_ be idempotent so they can be retried upon failure, as is the case for all retryable code, but this is not a requirement. These side effecting functions can be configured at the callsite to have at-most-once semantics.
In addition to many other things like observability, the value of durable execution is persisted advanced logic like loops, try/catch, concurrent async ops, sleeping, etc and making all of that crash proof (i.e. resumes from where it left off on another machine).
- At Temporal, for Java we did a hybrid approach of what you have. Specifically, we do the java.lang.reflect.Proxy approach, but the user has to make a call instantiating it from the implementation. This allows users to provide those options at proxy creation time and not require they configure a build step. I can't speak for all JVM people, but I get nervous if I have to use a library that requires an agent or annotation processor.
Also, since Temporal activity invocations are (often) remote, many times a user may only have the definition/contract of the "step" (aka activity in Temporal parlance) without a body. Finally, many times users _start_ the "step", not just _execute_ it, which means it needs to return a promise/future/task. Sure this can be wrapped in a suspended virtual thread, but it makes reasoning about things like cancellation harder, and from a client-not-workflow POV, it makes it harder to reattach to an invocation in a type-safe way to, say, wait for the result of something started elsewhere.
We did the same proxying approach for TypeScript, but we saw as we got to Python, .NET, and Ruby that being able to _reference_ a "step" while also providing options and having many overloads/approaches of invoking that step has benefits.
- > The author's point about the friction from explicit step wrappers is fair, as we don't use bytecode generation today, but we're actively exploring it to improve DX.
There is value in such a wrapper/call at invocation time instead of using the proxy pattern. Specifically, it makes it very clear to both the code author and code reader that this is not a normal method invocation. This is important because it is very common to perform normal method invocations and the caller needs to author code knowing the difference. Java developers, perhaps more than most, likely prefer such invocation explicitness over a JVM agent doing byte code manip.
There is also another reason for preferring a wrapped-like approach - providing options. If you need to provide options (say timeout info) from the call site, it is hard to do if your call is limited to the signature of the implementation and options will have to be provided in a different place.
- > is it the dev ergonomics that's cool here?
Yup. Being able to write imperative code that automatically resumes where it left off is very valuable. It's best to represent durable turing completeness using modern approaches of authoring such logic - programming languages. Being able to loop, try/catch, apply advanced conditional logic, etc in a crash-proof algorithm that can run for weeks/months/years and is introspectable has a lot of value over just using queues.
Durable execution is all just queues and task processing and event sourcing under the hood though.
- I think correlating "pushes per repository" to certain languages is interesting. The top "pushes per repository" are C++, TeX, Rust, C, and CSS. I guess it's no surprise many would also consider those the most guess-and-check or hard-to-get-right-upfront-without-tooling languages too.
- Deterministic output is needed when LLMs are used for validations. This can be anything from input validation at runtime to a CI check leveraging LLMs. It can be argued this is not an acceptable use of AI, but it will become increasingly common and it will need to be tweaked/tested. You cannot tweak/test a response you don't know you're going to get.
- As someone that has had to build libraries for the nuances of coroutine vs thread async in several languages (Python, .NET, Java, Ruby, etc), I believe how Ruby did fibers to be the best.
Ruby's standard library was not littered with too many sync helpers, so making them fiber capable without much standard library effect is a big win. In Python, explicit coloring is required and it's easy to block your asyncio coroutines with sync calls. In .NET, it is nice that tasks can be blocking or not, but there is one fixed global static thread pool for all tasks and so one is tacitly encouraged to do CPU bound work in a task (granted CPU bound fibers are an issue in Ruby too), not to mention issues with changing the default scheduler. In Java, virtual threads take a Ruby-esque approach of letting most code work unchanged, but the Java concurrency standard library is large with slight potential incompatibilities.
Ruby is both 1) lucky it did not have a large standard library of thread primitives to adapt, and 2) smart in that they can recognize when they are in a fiber-scheduler-enabled environment or not.
Granted that lack of primitives sure does hurt if you want to use concurrency utilities like combinators. And at that point, you reach for a third party and you're back in the situation of not being as portable/obvious.
- I use RBS/steep with great success to catch plenty of nil issues early, but it's similarly not great from a dev POV to have to maintain a completely separate set of rbs files (or special comments with rbs-inline). Also in my experience, modern editors don't leverage it for typing/intellisense.
- > Agents, on the other hand, are often given a set of tools and a prompt. They are much more free-form.
This defines how workflows are used with modern systems in my experience. Workflows are often not predictable, they often execute one of a set of tools based on a response from a previous invocation (e.g. an LLM call).
- I believe the definition of workflows in this article is inaccurate. Workflows in modern engines do not take predefined code paths, and agents are effectively the same as workflows in these cases. The redefinition of workflows seems to be an attempt to differentiate, but for the most part an agent is nothing more than a workflow that is a loop that dynamically invokes things based on LLM responses. Modern workflow engines are very dynamic.
- > I hate this idea that Ruby needs to be more like Python or Typescript
It's not be more like those, it's be more like helpful, author-friendly programming which is very much Ruby's ethos.
Every time I think about ripping out all of the RBS sig files in my project because I'm tired of maintaining them (I can't use Sorbet for a few reasons), Steep catches a `nil` error ahead of time. Sure we can all say "why didn't you have test coverage?" but ideally you want all help you can get.
- Fingers crossed this is/becomes extensible. Pyright and MyPy both suffer from lack of extensibility IMO (Pyright doesn't consider the use case and MyPy plugins come across as an afterthought with limited capabilities). There are many things that can be built on the back of type-checked AST.
- Not true, you can compete with the quality of the deployed service _separate_ from the development of the software. The quality of the service can include internal, at-scale optimizations that don't affect user-facing parity with the open-source software.
Open source companies with SaaS offerings need to have plans to differentiate themselves on hosting quality, not features. Yes, you can do better at hosting your own product than Amazon in many cases with customized, closed-source optimizations (that are unrelated to feature parity and does not intentionally limit the open-source/self-hosted form), support, etc.
- There just aren't good Python sandboxing approaches. There are subinterpreters but they can slow to start from scratch. There are higher-level sandboxing approaches like microvms, but they have setup overhead and are not easy to use from inside Python.
At Temporal, we required a sandbox but didn't have any security requirement, so we wrote it from scratch with eval/exec and a custom importer [0]. It is not a foolproof sandbox, but it does a good job at isolating state, intercepting and preventing illegal calls we don't like, and allowing some imports to "pass through" the outside instead of being reloaded for performance reasons.
0 - https://github.com/temporalio/sdk-python?tab=readme-ov-file#...
- > Workflows are systems where LLMs and tools are orchestrated through predefined code paths
This definition keeps coming up, but the definition isn't accurate for workflows. Modern workflow systems are very dynamic in nature and direct their own process and tool usage (e.g. like Temporal, disclaimer: my employer). You can even write workflows that eval code if you want though for most that's a step of flexibility too far to give to an LLM. Many workflows have LLMs tell them what to do next, sometimes via a bounded tool list, or sometimes open ended e.g. process execution or code eval. There is no limit here. A better definition of a workflow is that it durably orchestrates things, not that the sequence or code is predefined.
So by a more accepted/modern definition of "workflow", agents are workflows that just happen to be more dynamic than some more rigid workflows.
- We wanted to use Ractors in our latest Ruby project (which is Rust based using rb-sys/magnus for some parts), but since our users use Google Protobuf inside the code that may run in Ractors, we could not because the library is not Ractor safe[0] (and it's unreasonable for us to ask our users to not be able to use the official Protobuf library).
0 - https://github.com/protocolbuffers/protobuf/issues/19321
- My biased answer, because I work at Temporal[0], is to use an existing workflow solution that solves all of these problems instead of reaching for a solution that doesn't help with any of these but happens to be AI specific. Most agentic AI workflows are really just microservice orchestrations, the only "AI" involved is prompting an HTTP API that uses AI on its end. So use a good solution for "agentic X" whether that X is AI or any other orchestration needs.
- I think Anthropic's definition of workflows is inaccurate for modern definitions of the term. Temporal for instance (disclaimer, my employer) allows completely dynamic logic in agentic workflows to let the LLM choose what to do next. It can even be very dynamic (e.g. eval some code) though you may want it to operate on a limited set of "tools" you make available.
The problem with all of these AI specific workflow engines is they are not durable, so they are process local, suffer crashes, cannot resume, don't have good visibility or distribution, etc. They often only allow limited orchestration instead of code freedom, only one language, etc
- > Just like the external-distribution model, arbitrary-location architectures often come with a performance cost. Durable execution systems typically snapshot their state to a persistent store between every step.
This is not true by most definitions of "snapshot". Most (all?) durable execution systems use event sourcing and therefore it's effectively an immutable event log. And it's only events that have external side effects enough to rebuild the state, not all state. While technically this is not free, it's much more optimal than the traditional definition of capturing and storing a "snapshot".
> But this simplicity comes at a significant cost: control. By letting the runtime decide how the code is distributed [...] we don’t want to give up: Explicit control over placement of logic on machines, with the ability to perform local, atomic computations
Not all durable execution systems require you to give this up completely. Temporal (disclaimer: my employer) allows grouping of logical work by task queue which many users use to pick locations of work, even so far as a task queue per physical resource which is very common for those wanting that explicit control. Also there are primitives for executing short, local operations within workflows assuming that's what is meant there.
- Replied to another in this thread, but basically https://temporal.io/in-use lists many, AI and not.
- I fear I'll come off as a shill, but I've seen dozens of company uses of AI in workflows in the real world. Agents are just orchestrating multiple AI steps basically (granted not all of them are using AI to _pick_ the step to take which is often what "agentic" is seen as). Some are listed at https://temporal.io/in-use alongside the many non-AI things, e.g. https://temporal.io/resources/case-studies/bugcrowd, https://temporal.io/resources/on-demand/arc-xp-washington-po..., https://temporal.io/resources/on-demand/practical-tactical-a..., and more and more. All those companies use AI workflows in real world cases, and there are many more. I only showed the tutorial to agree with OP that it is just workflows with fuzzier steps and that's ok.
> The real golden ticket I think is to make readable intuitive abstractions around durability, not hide it behind normal-looking code.
It's a tradeoff. People tend to want to use languages they are familiar with, even at the cost of being constrained within them. A naive DSL would not be expressive enough for the turing completeness one needs, so effectively you'd need a new language/runtime. It's far easier to constrain an existing language than write a new one of course.
Some languages/runtimes are easier to apply durable/deterministic constraints too (e.g. WASM which is deterministic by design and JS which has a tiny stdlib that just needs a few things like time and rand replaced), but they still don't take the ideal step you mention - put the durable primitives and their benefits/constraints in front of the dev clearly.