My own project
A five-node k3s build fleet for CI
Build where the cache already is, and never run the same validation twice.
- Build workers
- 5 BuildKit workers, one each on five nodes
- Same 38 jobs
- 51m01.5s → 17m29.1s (2.918×, four workers)
- Gate pods
- 32 or 64 CPUs, 90-minute deadline
- Postgres suite
- 752 s → 48 s
- Push with a warm receipt
- 0.31 s instead of 76.5 s
- k3s
- BuildKit
- Node.js
- Python
- PostgreSQL
- Git hooks
I run a product split across five repositories, and every change to it passes the same validation gates before it can merge. For a long time those gates and the CI builds ran on whichever machine started them. Now they run on my bare-metal k3s cluster, with a BuildKit worker on five of its nodes.
The same 38 CI jobs that took 51 minutes one after another finished in about 17 on four workers. Most of that is parallelism, helped by caches an interrupted earlier attempt had warmed; neither run reused a previous test pass. The rest of this page is about what keeps later runs fast: sending builds to the node that already has their cache, and keeping receipts so a validation that already passed doesn't run again.
The setup
Each validation gate runs in its own temporary pod on nodes labelled as burst-capable. A fast gate gets up to 32 CPUs and a full gate up to 64, with a 90-minute deadline. The pods are offline and credential-free. Validators get a scrubbed environment, no secrets and no network to fetch anything from.
That works because nothing needs fetching. The toolchains and each verified dependency bundle are mirrored once per content key into a cache on each gate node, then hardlinked read-only into each work tree. A gate pod starts with everything it's allowed to use already on disk.
CI builds go to BuildKit: five workers, one per node, and a run uses every Ready worker by default.
Placement: long jobs first, and builds go home
Two rules decide where work lands. The longest job gets the fastest free pod, so the critical path starts first instead of queueing behind short jobs. And a build of a given Git tree prefers the node that last built that tree, because that node's BuildKit already holds its layers.
The same validation build took 83.7 seconds on its recorded node, with 19 of 30 layers cached, and 129.4 seconds on a different node, with 3 of 28 cached.
The node record is only a hint. It chooses among Ready pods and never changes what gets built or how it's verified. With no usable record, the build goes to the Ready pod with the most spare CPU.
Receipts: never validate the same thing twice
A passing gate writes a receipt. The receipt is bound to the exact Git tree, the validators, the dependency bundle, the runtime files, the runner's own bytes and the executor, and it's valid for one hour. When all of that matches, the next gate reuses the pass instead of running again. Exact-content reuse took 0.158 seconds against 87.001 seconds for a fresh run.
Binding the executor is deliberate. A pass inside a k3s pod never satisfies a check that asked for the host, and a host pass never satisfies a k3s check.
Getting the key right took a bug fix. Git changes PATH when it runs a hook, PATH was part of the key, and so nearly every hook run missed. After the fix, a push whose full gate was prewarmed takes 0.31 seconds instead of 76.5. That bug has its own post.
Sharding the slow suite
The biggest single cost in the gates was a PostgreSQL test suite. Split across workers, it takes 48 seconds instead of 752 on the host. Inside the BuildKit step that runs the same gates during an image build, it takes 52 seconds instead of 309.5.
The numbers
| Measurement | Before | After |
|---|---|---|
| Same 38 jobs, serial vs four workers | 51m01.5s | 17m29.1s (2.918×) |
| Same 49-job selection, 2026-09-24, one worker per node | 857.2 s | 649.7 s |
| Fresh full hook gate, one repository | 126 s | 35–44 s |
| Fresh full hook gate, another repository | 156 s | 72–109 s |
| Push after a prewarmed full gate | 76.5 s | 0.31 s |
A larger acceptance run of 27 entries and 49 jobs passed in 16m39.6s. With five workers, a full run sits near a critical-path floor of about 470 seconds: the longest chain of jobs that depend on each other, which more workers can't shorten. So I don't raise concurrency without measured headroom.
Caveats
The 51-to-17 comparison isn't a clean cold-build experiment. The parallel run had warmer caches, and the cluster was carrying other load at the time, so I treat it as an observation, not a promised saving. The 2026-09-24 pair is cleaner: identical revisions, with the cluster dedicated to CI.
k3s isn't free either. On an idle host, the small fast and full gates for one repository ran in 24.5 and 31.0 seconds. On k3s they took 35.7 and 38.1, because the host's cores are faster one thread at a time than the cluster's EPYC nodes. Offloading pays when the host is busy or the work is heavy. For tiny checks on a quiet machine it doesn't, and those stay local.
Receipts before nodes
I'd build the receipts and the cache-aware placement before adding any nodes. Extra workers did nothing for a key that missed 97% of the time, and none of them can shorten the 470-second critical path.
I'd also record layer hits per job from the first day. Those counts are what explained the 83.7-second and 129.4-second runs of the same build, and without them, placement decisions are guesses.