One ARG in the wrong place cost 226 seconds a build

In June, the worker image for my coding-agent benchmarks took 239 seconds to build in one pipeline run and 13 seconds in another. The slow run had to rebuild the apt and uv layer, and in that Dockerfile one line made that happen on every agent update. An ARG holding the agent's version sat above the apt packages and the uv setup, so every agent version bump threw away the most expensive layer in the image and reinstalled the OS packages.

The worker image is what every benchmark task runs in; the operator case study has the context. This post covers the layer-order fix, a second fix for task images that were building one at a time, and the timings I still don't have.

Docker's rule about ARG

The Dockerfile reference is explicit. If an ARG has a different value from the previous build, the cache miss happens at its first use, not where it's declared. And every RUN instruction after an ARG uses that variable implicitly, as an environment variable, so each of them can miss even if it never mentions the variable.

Put those two together and an ARG near the top of a Dockerfile busts the cache for every RUN below it. Change its value and they all rebuild, including the ones that have nothing to do with it.

That stacks with the general rule in Docker's build cache documentation: once a layer changes, every layer after it has to rebuild as well. It's also easy to miss in review. A diff that adds or moves an ARG line reads like a harmless declaration, not a cache boundary, and the build log shows which steps weren't cached without saying why.

Before and after

This is the worker Dockerfile, trimmed to the parts that matter. The agent version at the time was @oh-my-pi/pi-coding-agent@16.0.4.

# Before: the version ARG sits above the slow, stable layer
FROM ${BASE_IMAGE} AS runtime-base
ARG PI_PACKAGE="@oh-my-pi/pi-coding-agent@16.0.4"

# apt packages and uv: slow, and they almost never change
RUN apt-get update \
    && apt-get install -y --no-install-recommends python3 bash curl git jq ripgrep ca-certificates tar \
    && rm -rf /var/lib/apt/lists/* \
    && curl -LsSf https://astral.sh/uv/install.sh | sh

# Node, Bun and the agent: changes with every version bump
RUN bash -lc 'nvm install 22 && npm install -g bun "${PI_PACKAGE}"'

COPY entrypoint.sh /usr/local/bin/entrypoint.sh

# After: the ARG moves down to the only layer that uses it
FROM ${BASE_IMAGE} AS runtime-base

RUN apt-get update \
    && apt-get install -y --no-install-recommends python3 bash curl git jq ripgrep ca-certificates tar \
    && rm -rf /var/lib/apt/lists/* \
    && curl -LsSf https://astral.sh/uv/install.sh | sh

ARG PI_PACKAGE="@oh-my-pi/pi-coding-agent@16.0.4"
RUN bash -lc 'nvm install 22 && npm install -g bun "${PI_PACKAGE}"'

COPY entrypoint.sh /usr/local/bin/entrypoint.sh

The real file also installs nvm in that last RUN and packs the Node runtime into a tarball for the tasks; I've cut that. The general rule is to order layers from least to most frequently changed, and to declare each ARG directly above the first instruction that needs it.

A quick audit works on any Dockerfile. For each ARG whose value changes often, find the first instruction that reads it. If an expensive RUN sits between the two, move the ARG down.

What 239 and 13 seconds compare

Both numbers come from runs of the Woodpecker CI pipeline in June:

That's where the 226 seconds in the title comes from: 239 minus 13. It's a fair measure of what a miss on that layer costs. It isn't a before-and-after measurement of the fix.

Pipeline 391 is a build where nothing invalidated the expensive layer. After the fix, a version bump still rebuilds the Node and agent layer, because that layer did change. So a post-fix bump should land somewhere above 13 seconds and well below 239, and I haven't timed one.

There's also a delay before the fix shows up anywhere. The new worker image only matters once it's built, pushed and referenced by a benchmark profile. Runs that already exist keep the image they started with.

Task images were building one at a time

The worker image wasn't the slowest step. Each benchmark task has its own environment image, and one pipeline spent 3,293 seconds in the step that built them, one task after another, whether or not a task had changed.

The new build script works differently. It only builds tasks whose directories changed in git diff HEAD~1..HEAD. It runs those builds in parallel batches, capped by FULL88_TASK_IMAGE_BUILD_JOBS, which has to be a positive integer. And it records which background buildctl process belongs to which task, so a failed build fails the step and names the task.

That last part is easy to get wrong. A shell script that starts builds with & and finishes with a bare wait returns 0 even when a child failed. The script waits on each PID and checks its exit status, so a failed build turns the pipeline red instead of leaving one image missing.

Testing a build script without building anything

I didn't want to push real images to prove the control flow, so I tested the script in a temporary repository against a fake buildctl:

• One committed change in one task directory produced exactly one buildctl call, for that task.
• Three changed tasks with FULL88_TASK_IMAGE_BUILD_JOBS=2 built all three, and the observed concurrency peaked at exactly 2.
• A forced failure for one task made the script exit nonzero with Full88 task image build failed for task 'chess-best-move'.

The test didn't dirty a single task Dockerfile in the real repository. It proves the selection, the cap and the failure path. It says nothing about Harbor push latency or how the real BuildKit behaves with several builds at once.

What I haven't measured

I don't have timings for the parallel task builds after the change. The script's behavior is verified; its speed in the real pipeline isn't.

Two more limits I know about. HEAD~1..HEAD only sees the latest commit, so a push with several commits could skip a task that changed in an earlier one. Diffing against the last commit that built successfully would close that gap.

And the BuildKit cache is a 24 GiB emptyDir with a 20 GiB garbage-collection cap. An emptyDir lives as long as its pod, so the cache disappears whenever the BuildKit pod is rescheduled. A persistent cache, managed through GitOps like the rest of the cluster, is the next change I'd make before counting on warm task-image builds.