A Kubernetes operator for coding-agent benchmarks

I benchmark coding agents on Terminal-Bench: 88 tasks where an agent works inside a container until a verifier checks its result. A profile here is one agent with one model at one reasoning effort. At 54 profiles that's 4,752 cells, and plenty of cells take more than one attempt. By June 23, 2026, the database held 38,150 attempts.

I started with shell scripts and Python loops on one host. This page is about the Kubernetes operator that replaced them, and about the parts that ended up mattering more than running tasks.

Why an operator

The old runner started Docker containers on one host through its socket and wrote logs to local files. That was fine for a few profiles. At dozens, I needed a queue that survives restarts, placement that knows how busy each node is, and one place to ask what happened to a given attempt.

Kubernetes already does most of that, so the operator is a thin layer on top. It's written in Go with kubebuilder and controller-runtime, and two of its custom resources do most of the work. A BenchmarkProfile is the blueprint: model, agent, task list, worker image, concurrency and resource requests. A BenchmarkRun is one requested run set against a profile. The controller accepts a run, writes its queue rows to Postgres, and turns runnable rows into Kubernetes Jobs as capacity allows.

How one task runs

Each task becomes a single-pod Job with two containers. The worker container runs the agent and the evaluation. A privileged Docker-in-Docker sidecar runs a Docker daemon inside the pod, and the two share /var/run through an emptyDir, so every docker command the worker runs goes to the sidecar's socket.

Terminal-Bench tasks are Docker Compose projects. The worker runs docker compose up -d --wait, which blocks until every container's health check passes, and then starts the agent. Task images come from a Harbor mirror inside the cluster, so pulls stay on the local network.

When the task ends, the worker uploads stdout.log, stderr.log and attempt.json to MinIO, and the pod's output goes to Loki. Postgres stores the artifact URI, checksums and the Loki selector. It never stores the log bodies.

Scheduling by headroom

Requests and limits say what a pod asked for, not what a node is doing. The scheduler also reads live node usage from metrics.k8s.io and leaves a queue row waiting when the cluster is out of headroom. The point is to avoid evictions and OOM kills mid-attempt, which otherwise look exactly like agent failures.

Concurrency is set per profile too. Rate-limited providers run at 1–2 tasks at a time, because going higher doesn't finish sooner. It just produces rate-limit errors.

Results you can audit

Postgres holds normalized run sets, queue items, attempts, final task results and leaderboard entries. When a Job finishes, the controller updates the attempt, then upserts the task result, then marks the queue item done, always in that order. If Job creation fails after a row was claimed, the row goes back to pending instead of disappearing.

Each configuration gets a hash that covers the agent, model, thinking mode, variant, runner image, resource policy and timeout policy. A fix to the worker image therefore starts a new configuration, and runs from before and after the fix can't merge into one leaderboard row. A run set becomes a leaderboard candidate only when its final results cover all 88 tasks under one hash.

Infrastructure failures are their own category

An agent whose task image never arrived didn't fail the task. Counting that as a failure blames the model for my cluster, so every outcome gets a class and infrastructure failures stay out of the score.

In the June 23 audit, 744 cells had attempts but no verdict yet: 578 infrastructure failures, 76 attempts with missing artifacts, 68 blocked on auth and 22 configuration failures.

The auth number hid a registry problem. Under multi-gigabyte layer pulls, Harbor got overloaded and dropped connections. The docker CLI exited with code 18, and the operator mapped that code to blocked auth. So part of that bucket was a registry under load, labelled as credentials.

Repair instead of rerun

The same audit found 1,883 cells that didn't need another run. 1,791 of them, across 36 profiles, had run under a legacy suite name, and repairing their source identity linked them to the official 88-task suite. The other 92, across 3 profiles, were encoded variant names that needed mapping to their base model.

Rerunning those cells would have meant real model spend and cluster time for results I already had. The repair was a metadata change.

The audit also caught my checkouts disagreeing about the denominator. The root checkout's expected matrix listed 52 profiles, or 4,576 cells. The active controller and the database listed 54, or 4,752. The database won, because it's what the controller schedules against.

Image builds were the slow step

One CI pipeline spent 3,293 seconds building task images one after another. Now only tasks changed in the latest commit rebuild, in parallel batches with a cap, and a failed build names its task. The layer-order post covers that change and a Dockerfile fix for the worker image.

The tradeoff I accepted

The DinD sidecar runs privileged. That's what lets task environments use Docker Compose inside a pod, and it's a real weakness: an escape from a privileged container lands on the node. I accept it because these jobs run on their own nodes, never the one that hosts the database, and the agents are benchmark subjects rather than anything with production access.

If I needed stronger isolation, I'd try gVisor or Kata Containers next. I'd also store the raw docker exit code beside every outcome class from the start. Code 18 turned out to cover an overloaded registry as well as bad credentials, so the raw code belongs next to the class.