Hacker Neue

Preferences
nijave parent
Really if you could just disable fsync at the OS level. A bunch of other common package managers and tools also do. Docker is a big culprit

If you corrupt a CI node, whatever. Just rerun the step

wtallis
CI containers should probably run entirely from tmpfs.
nijave OP
Can tmpfs be backed by persistent storage? Most of the recent stuff I've worked on is a little too big to fit in memory handily. Ideally about 20GiB of scratch space for 4-8GiB of working memory would be ideal.

I've had good success with machines that have NVMe storage (especially on cloud providers) but you still are paying the cost of fsync there even if it's a lot faster

wtallis
tmpfs is backed by swap space, in the sense that it will overflow to use swap capacity but will not become persistent (since the lack of persistence is a feature).
candiddevmike
We built EtchaOS for this use case--small, immutable, in memory variants of Fedora, Debian, Ubuntu, etc bundled with Docker. It makes a great CI runner for GitHub Actions, and plays nicely with caching:

https://etcha.dev/etchaos/

jacobwg
We're having some success with doing this at the block level (e.g. in-memory writeback cache).
yjftsjthsd-h
Why do it at the block level (instead of tmpfs)? Or do you mean that you're doing actual real persistent disks that just have a lot of cache sitting in front of them?
jacobwg
The block level has two advantages: (1) you can accelerate access to everything on the whole disk (like even OS packages) and (2) everything appears as one device to the OS, meaning that build tools that want to do things like hardlink files in global caches still work without any issue.
formerly_proven
You can probably use a BPF return override on fsync and fdatasync and sync_file_range, considering that the main use case of that feature is syscall-level error injection.

edit: Or, even easier, just use the pre-built fail_function infrastructure (with retval = 0 instead of an error): https://docs.kernel.org/fault-injection/fault-injection.html

jacobwg
I'd love to experiment with that and/or flags like `noatime`, especially when CI nodes are single-use and ephemeral.
formerly_proven
noatime is irrelevant because everyone has been using relatime for ages, and updating the atime field with relatime means you're writing that block to disk anyway, since you're updating the mtime field. So no I/O saved.
3np
atime is so exotic you shouldn't need to consider disabling it experimental. I consider it legacy at this point.
chippiewill
> Docker is a big culprit

Actually in my experience with pulling very large images to run with docker it turns out that Docker doesn't really do any fsync-ing itself. The sync happens when it creates an overlayfs mount while creating a container because the overlayfs driver in the kernel does it.

A volatile flag to the kernel driver was added a while back, but I don't think Docker uses it yet https://www.redhat.com/en/blog/container-volatile-overlay-mo...

nijave OP
Well yeah, but indirectly through the usage of Docker, I mean.

Unpacking the Docker image tarballs can be a bit expensive--especially with things like nodejs where you have tons of tiny files

Tearing down overlayfs is a huge issue, though

kylegalbraith
This is a neat idea that we should try. We've tried the `eatmydata` thing to speed up dpkg, but the slow part wasn't the fsync portion but rather the dpkg database.

This item has no comments currently.

Keyboard Shortcuts

Story Lists

j
Next story
k
Previous story
Shift+j
Last story
Shift+k
First story
o Enter
Go to story URL
c
Go to comments
u
Go to author

Navigation

Shift+t
Go to top stories
Shift+n
Go to new stories
Shift+b
Go to best stories
Shift+a
Go to Ask HN
Shift+s
Go to Show HN

Miscellaneous

?
Show this modal