Spreading CI across four k3s workers took a 51-minute run to 17
Parallelism bought the headline number. Putting jobs where their cache lives and never repeating a passed validation is what keeps runs fast after that.
I run CI for a product of mine, split across five repositories, on my bare-metal k3s cluster, where five nodes each run one BuildKit worker. On 2026-09-23 I ran the same frozen set of 38 validation jobs twice: once serially, once spread across four workers. Serial took 51m01.5s. Parallel took 17m29.1s, a 2.918× speedup.
Most of that speedup is plain parallelism. The harder work was deciding where each job lands and what it's allowed to skip, which is what keeps later runs fast, and finding the one chain of jobs that sets a floor more workers can't get under. The comparison has holes, which I'll get to.
What's in the fleet
The fleet has four parts:
- A bare-metal k3s cluster, with BuildKit on five of its nodes.
- One BuildKit worker on each of those nodes, with its layer cache on that node's disk. A warm cache on one node does nothing for a job running on another.
- Temporary gate pods for the Git hooks, offline and without credentials, on nodes labelled as burst-capable. A fast gate gets a 32-CPU budget, a full gate 64, and both have a 90-minute deadline.
- A receipt store that records what passed, keyed on exactly what was validated.
Placement is a cache decision
The scheduler follows two rules. First, the longest job goes to the fastest free pod. A run's wall time is bounded by its longest chain, so starting that chain first on the best hardware matters more than anything clever I could do with the short jobs.
Second, builds have affinity to the node that built the same tree before. After a successful build, CI writes a small record naming the pod and node that built that exact source tree. The next build of the same tree reads it and goes back to that node if its worker is Ready. If it isn't, the build falls back to the worker with the most spare CPU.
On the same day, the same validation build of the same tree took 83.7 s with 19 of 30 layer hits on its recorded node and 129.4 s with 3 of 28 hits on a different one. Only the node changed.
The record is only a hint. It can choose among Ready workers in the same fleet and nothing else, so a stale or missing entry costs speed, never correctness.
Never run the same validation twice
Before receipts, every gate reran its full test set, even on content that had already passed. Now every gate writes a receipt bound to the exact tree, the validator commands, the dependency bundle, the runtime files, the runner's own bytes and the executor, and it's good for one hour. A commit or push that finds a matching receipt reuses it.
On one repository the full offline gate took 87.001 s fresh and 0.158 s when the receipt matched. A push after a prewarmed full gate now takes 0.31 s instead of 76.5 s. Getting that last number took fixing a bug where Git's hook-time PATH changed the key, and that story has its own post.
Shard the suite that dominates
One PostgreSQL test suite was the slowest thing in the product gates. Run serially on the host, it took 752 s. Split across workers, it took 48 s. The same suite inside the BuildKit CI step went from 309.5 s to 52 s.
It split well because its files didn't depend on each other. A suite with long serial dependencies wouldn't, and I'd profile before assuming sharding will help anywhere else.
The numbers, with their conditions
| Measurement | Before | After | Conditions |
|---|---|---|---|
| Same 38 jobs | 51m01.5s serial | 17m29.1s | four workers, 2026-09-23 |
| Expanded set, 27 entries and 49 jobs | 16m39.6s | acceptance run | |
| Same 49 jobs at identical revisions | 857.2 s | 649.7 s | one worker per node, 2026-09-24 |
| Fresh full hook gate, one repository | 126 s | 35–44 s | gate pods on k3s |
| Fresh full hook gate, another repository | 156 s | 72–109 s | |
| PostgreSQL suite on the host | 752 s | 48 s | sharded |
| PostgreSQL step in BuildKit | 309.5 s | 52 s | sharded |
| Push after a prewarmed gate | 76.5 s | 0.31 s | receipt reuse |
The 857.2 s and 649.7 s pair is the cleanest comparison I have. The revisions were identical, the cluster was dedicated to CI, and the only change was how the slowest stage ran its test groups inside its job.
It also showed the limit. A full run now sits near its critical-path floor of about 470 s at five workers, so adding workers or raising concurrency won't buy much. The next gain has to come from shortening the longest chain.
Where the comparison is weak
The 51-to-17 pair is an observation, not a controlled experiment, and two things favored the parallel run. An interrupted parallel attempt had warmed caches before the accepted one. Other work on the cluster wasn't isolated either. Runtime and dependency caches were on in both phases; only the test stages were forced to run. So I don't credit the whole 2.918× to scheduling, and I wouldn't extrapolate it to the larger catalog.
The fleet isn't always faster than one machine. On an idle host, one repository's fast and full gates took 24.5 s and 31.0 s. On k3s the same gates took 35.7 s and 38.1 s, because the host's cores are quicker one thread at a time. I run gates on k3s anyway because they use no host CPU and don't slow each other down, so several agents in my agent fleet can push at once without queueing behind a lock. For one person running one small check on a quiet machine, local wins, and the tooling allows it.
Cold numbers come first next time
I'd run a cold-cache benchmark on an otherwise idle cluster before quoting any speedup, and publish it next to the warm number. The warm number is what you'll see day to day. The cold one is what you'll see after a node is replaced, and I don't have it.
I'd also write the affinity record from the first day. It's a small file per build, and it decides whether a build takes 83.7 s or 129.4 s.