Running 38,150 coding-agent benchmark attempts on Kubernetes

By June 23, 2026, my benchmark cluster had stored 38,150 coding-agent attempts on Terminal-Bench. The matrix is 54 model profiles by 88 tasks, so 4,752 cells, where a cell is one agent-and-model setting on one task.

The case study describes the operator that runs them. This post is what I'd tell someone about to build the same thing.

The shape of the system, briefly

A Go operator, built with kubebuilder and controller-runtime, turns queue rows in Postgres into Kubernetes Jobs. Each Job is one pod: a worker container that drives the agent, plus a privileged Docker-in-Docker sidecar, because every Terminal-Bench task is a Docker Compose project. Images come from an in-cluster Harbor mirror. Logs go to Loki, artifacts go to MinIO, and Postgres keeps the structure: run sets, queue items, attempts, task results and leaderboard entries.

The denominator lives in one table

A pass rate is passes over cells, and for a while I had two answers for the number of cells. The expected-matrix file in my root checkout listed 52 profiles, which is 4,576 cells. The active controller's copy and the database table listed 54, which is 4,752.

The root copy was missing five Grok profiles and one high-speed Kimi variant. It still listed four GLM-5.2 effort levels that the active matrix had dropped. Nothing crashed. Every report built from that checkout just divided by the wrong number.

The fix: the database table that the controller schedules against is the denominator, and anything else that needs the number reads it from there.

Every failure needs a cause before it counts

Early on, a failed attempt was a failed attempt. That's wrong in a way that moves rankings at random, because an agent whose task image never arrived didn't fail the task. My cluster did.

Every outcome now gets a class: a clean pass, a verifier failure, an infrastructure failure and a few others. In the June audit, 744 cells had attempts but no verdict:

None of those say anything about the model. They're cells I still owe a real attempt.

Classes need the same suspicion as any other derived data. When Harbor was overloaded during multi-gigabyte layer pulls, it dropped connections. The docker CLI exited with code 18, and the operator mapped code 18 to blocked auth. So part of the auth bucket was a busy registry. If I built this again, every attempt would keep its raw exit code beside its class, so a bad mapping can be fixed after the fact.

1,883 results were already in the database

The most expensive mistake on offer is paying again for results you already have. The same audit found two versions of it.

First, 1,883 cells could be filled by repairing metadata. 1,791 of them, across 36 profiles, had run under a legacy suite name, and linking their source identity to the official 88-task suite brought those profiles to full coverage. The other 92, across 3 profiles, used encoded variant names, such as a high-speed variant that should have counted as its base model.

Second, 1,461 cells already had a terminal result in the database that the live leaderboard view didn't link: 720 passes, 738 verifier failures and 3 verifier exceptions. Those needed a fix to the view's joins, not new runs.

Both come from the same root. Results were found by name, and names drift. Keying everything by a stable suite identity, and checking it when a result is written rather than when a report reads it, would have prevented both.

The scheduler reads live headroom

Requests and limits describe what a pod asked for. Agent tasks don't respect that: an agent compiling code, running a test suite or starting a database inside its Compose project can use far more than its baseline. The scheduler now reads live node usage from metrics.k8s.io next to requests and limits, and leaves a queue row pending when a node is out of headroom.

The goal is to avoid evictions and OOM kills in the middle of an attempt. Those look like agent failures in the results unless something tells you otherwise, which is the previous lesson again.

Model providers need their own limit. Rate-limited providers run at a concurrency of 1–2 per profile. Going higher doesn't finish a run any sooner. It fills the results with rate-limit errors that then need classifying.

Logs stay out of Postgres

Postgres stores the shape of the data. It doesn't store logs. Each attempt row holds an artifact URI in MinIO, checksums, the Loki selector for its logs and its Prometheus labels. Reports can join across all of it, and the database doesn't grow with every line an agent prints.

Writes happen in a fixed order. When a Job ends, the controller updates the attempt, then upserts the task result, then marks the queue item done. If creating a Job fails after a row was claimed, the row goes back to pending.

Configurations are immutable

A configuration hash covers the agent, model, thinking mode, variant, runner image, resource policy and timeout policy. Fixing the worker image therefore creates a new configuration, and results from before and after the fix can't merge into one leaderboard row.

A run set only becomes a leaderboard candidate when its final results cover all 88 tasks under one hash. Anything less is stored as not comparable, with the reason written down. That rule throws away some partial runs that were probably fine, and I'm comfortable with it.

The backups needed an audit too

The audit that found the 1,883 recoverable cells also looked at the cluster's Postgres backups. One database's dev dumps had been zero bytes for four days, on schedule, without an error anywhere. That gets its own post.

The bookkeeping I'd start with

I'd start with the normalized schema and the configuration hash on day one instead of migrating to them. I'd key every result by suite identity from the first write. And I'd track the infrastructure-failure share of each run as a number on a dashboard, so a struggling registry pauses a run instead of filling it with cells that need running again.