How I fence the coding agents that work on my production systems

My agent fleet works on a product I run, split across five repositories, and on the clusters underneath it. Several agents run at once for most of the day. Some hold locks on shared resources, some review each other's merge requests, and some read production to debug it.

None of the rules below came from a threat model I wrote up front. Most of them came from something going wrong, or nearly going wrong. This is the set I run with now, and why each one exists.

No credentials inside the sandbox

Agents run tests in runner pods on a k3s cluster. A runner pod is offline, credential-free, ephemeral and scoped to one worktree. Its dependencies come from a prepared bundle that was checked before the pod started, and fixtures such as scratch databases live in memory and vanish with the pod. Writes stay in the pod. Nothing a test generates comes back to the host, so a test can't plant a file in a repository.

Git hook gates follow the same pattern with tighter limits. Each fast or full gate runs in a temporary pod with a 32- or 64-CPU budget, a 90-minute deadline and a scrubbed environment, and hooks give validators no credentials. Gate pods get no network at all. A NetworkPolicy denies every ingress and egress connection, DNS included, and the pod's resolver points at a loopback address where nothing listens, so a lookup fails at once instead of hanging until it times out. The gate runner still reaches the pod through the Kubernetes API, which is all it needs.

Real work does need credentials sometimes: a GitLab API call, a read against a cloud project. For that an agent gets a broker reference, never a value. The broker resolves it at the moment of use. Values never go into prompts, tool arguments or logs, because anything an agent can see ends up in a transcript sooner or later.

Production changes go through Git

No agent changes a cluster by hand. There's no imperative kubectl apply, kubectl patch or kubectl delete against anything persistent. An agent that wants to change production opens a merge request against the GitOps repository, it gets reviewed and merged, and Flux applies it.

This is slower than a one-line kubectl command, and I'd make the same trade again. Every change has an author, a diff, a review and a revert path. When something breaks, the answer to what changed is in git log, not scattered across agent transcripts that each describe their own view of the cluster.

Reads are a different matter. Agents can inspect clusters, query metrics and read logs freely. Most debugging happens through reads, and making them slow would push agents toward guessing.

One writer per shared resource

Some resources take one writer at a time: a main branch being merged into, or an infrastructure change window. For these I use FIFO leases. An agent asks for the lease, waits its turn and gets it when the previous holder hands it back. The wait is one blocking call, not a polling loop, which is a separate post.

Leases also cover capacity, and in October I learned that where a lease holder runs matters as much as the lease.

The incident that moved reads out of serving pods

On 2026-10-16, between 16:20 and 16:45 UTC, 12–13 agents were running read-only diagnostic scripts against production at the same time. They had waited their turn on a lease, but each one then exec'd into a serving pod, so every script ran inside a container that was also answering users. OOM kills doubled over that window.

Nobody broke a rule. The scripts only read. The problem was where they ran, and how many ran at once.

Now read-only production scripts run in a separate runner pod that serves no traffic. Using it takes a lease of 1,152 MiB split into three 384 MiB slots, and an agent asks for the slots it needs and waits in line like any other lease. A fourth concurrent reader now waits for a slot instead of eating memory in the pods that serve users.

Looking back, I'd have treated memory in a serving pod as a shared resource from the start. "Read-only" told me the scripts were safe for the data. It told me nothing about whether they were safe for the process they ran in.

Agents leave each other's work alone

Agents never kill, interrupt or steer another agent's processes. Before an agent stops a pod or cleans up a scratch database, it proves it owns that thing, and the proof is labels set when the resource was created. A gate pod that looks abandoned may belong to an agent that's halfway through a push.

The main reason is debugging. When two agents interfere with each other, neither transcript explains what happened. Clear ownership keeps each failure inside one agent's story, where I can read it.

What's still weak

All of this assumes agents make mistakes, not that they're hostile. Processes running as the same user on the host can still read and edit each other's files, and the hook bundles trust whatever is on local disk. Receipts are local files, not signed attestations. Against a deliberately malicious agent I'd want separate users or VMs per agent, signed receipts and per-task credential scopes in the broker. I haven't built those.

There are costs on the safe side too. On one repository a full gate took 38.1 s in a pod and 31.0 s on an idle host, and the GitOps path is slower than a direct change. For a fleet that runs all day against production, both costs are worth paying.