Our dev database backups were zero bytes for four days

On June 23, 2026, I was auditing my coding-agent benchmark data, and while I was in there I checked the backups. Production looked healthy. Every scheduled dev dump since June 20 was 0 bytes.

The backup job had run on time every night. It wrote a file with the right name in the right directory and reported nothing unusual. That's the worst kind of backup failure, because every check that's easy to do says it worked.

The production chain looked fine

Production writes a custom-format pg_dump to a NAS every night, and the files sync to cloud storage at 04:00. The last five before the audit:

The chain went back unbroken to June 13 and grew from 3.3 GB to 3.9 GB over those days.

Dev was writing empty files on schedule

The dev dumps run at 02:30. These are the files the scheduled job wrote from June 20:

Five files, two of them on June 23, and none restorable. A directory listing sorted by name looks perfectly normal. Only the size column gives it away.

The cause was a catalog query

Before pg_dump writes any data, it reads the system catalogs to work out what exists. On the dev database, one of those catalog queries failed on a view: object 7857551, whose relation metadata had been left dangling. pg_dump errored. The backup script didn't treat that as a failure, so an empty file sat where the dump should have been, and the next night did the same.

I didn't trace how the view got into that state. Dev databases get dropped, migrated and restored far more often than production, which is a plausible route, but I haven't checked it.

How a script ends up writing an empty file

I haven't confirmed which of these applied here. They're the usual ways a dump script produces a zero-byte file and still exits cleanly, and they're worth checking in any backup script you own.

Redirection creates the file first. With pg_dump mydb > nightly.dump, the shell creates or truncates nightly.dump before pg_dump starts, so an early failure leaves an empty file with a perfectly good name.

Pipelines hide exit codes. In pg_dump mydb | gzip > nightly.dump.gz, the shell reports gzip's exit status unless set -o pipefail is on, and gzip is happy to compress nothing.

The last command wins. A scheduled job that ends with a successful copy, sync or log line reports success no matter what failed before it.

A scheduler that only watches exit codes can't see any of these. The one trace of the failure is whatever pg_dump wrote to stderr, and unless the job keeps that output somewhere a person will read it, it's gone by the next run.

Checks that would have caught it

Either of the first two checks below would have flagged the first empty file on June 20. The others catch failures that size alone won't.

Compare each dump with the previous one. Production grew by a fraction of a gigabyte over a week. A dump that's a small fraction of yesterday's is news, and a 0-byte dump is an incident.

Alert below an absolute floor. Any dump under a size threshold should page someone, whatever yesterday looked like. This catches the case where the first dump after a change is already broken and there's no good one to compare against.

Make the job fail loudly. Run the script with set -euo pipefail. Write to a temporary name and rename only after pg_dump exits 0 and the file passes the checks above. A failed run then leaves yesterday's good file in place and a red job in the scheduler, which is far easier to notice than an empty file.

Read the archive back. For a custom-format dump, pg_restore --list reads the table of contents without restoring anything. It's cheap enough to run after every dump, and it proves the file is at least a readable archive.

Restore on a schedule. The only real proof that a backup works is a restore. Periodically restore the latest dump into an isolated database and run a few row counts against it. This is the check that catches the failures nobody predicted, which is why it's worth its cost.

Restore somewhere safe

Once the audit turned up the problem, the recovery plan started with one rule: no restore tooling connects to the live production or dev Postgres services. Inspecting and extracting old dumps happens in an isolated scratch container, never against a live database.

Restoring over the evidence you're trying to recover is an easy mistake in the middle of an incident, and with no connection to the live services it can't happen.

Where this stands

This post stops at the diagnosis because the audit did. The checks above are what I'd add, not a list of what's running today.

Nothing crashed and nothing paged. The evidence sat in a directory listing for four days, and I found it only because the benchmark audit took me into the same place.