Preferences

How is this different from pg-boss[1]? Other than the distributed part it also seems to use skip locked.

[1] https://github.com/timgit/pg-boss


I haven't used pg-boss, and feature-wise it looks very similar and is an impressive project.

The core difference is that pg-boss is a library while Hatchet is a separate service which runs independently of your workers. This service also provides a UI and API for interacting with Hatchet - I don't think pg-boss has those things, so you'd probably have to build out observability yourself.

This doesn't make a huge difference when you're at 1 worker, but having each worker poll your database can lead to DB issues if you're not careful - I've seen some pretty low-throughput setups for very long-running jobs using a database with 60 CPUs because of polling workers. Hatchet distributes in two layers - the "engine" and the "worker" layer. Each engine polls the database and fans out to the workers over a long-lived gRPC connection. This reduces pressure on the DB and lets us manage which workers to assign tasks to based on things like max concurrent runs on each worker or worker health.

This item has no comments currently.

Keyboard Shortcuts

Story Lists

j
Next story
k
Previous story
Shift+j
Last story
Shift+k
First story
o Enter
Go to story URL
c
Go to comments
u
Go to author

Navigation

Shift+t
Go to top stories
Shift+n
Go to new stories
Shift+b
Go to best stories
Shift+a
Go to Ask HN
Shift+s
Go to Show HN

Miscellaneous

?
Show this modal