120 partner feeds that can be repaired and replayed

The client's 12 analysts built operating reports for commercial and service teams out of 120 partner feeds. Partners sent data through APIs, scheduled files and the occasional correction to old records. Each analyst spent about 6 hours a week reconciling totals by hand, because the dashboards couldn't say which version of which source had produced a given number.

We rebuilt the pipeline underneath the dashboards until every published number traced back to the partner bytes it came from.

Downloads aren't refreshes

The old pipeline treated a successful download as a successful refresh. It didn't notice when a field changed meaning, when a page of results went missing, or when a partner republished yesterday's records. Endpoint checks were green while the data was stale or wrong, and nobody could tell which.

Bounded scheduling

We started with a feed registry recording an owner, an allowed collection interval and partner-specific limits for every feed. Each collection got a deterministic key built from the source and the collection window, so overlapping scheduler runs couldn't double the work. Every job carried a cursor and a retry budget. One slow partner's backlog couldn't take every worker or flood a healthy partner with catch-up requests.

Fetch adapters handled authentication, pagination and retry-after headers, separately from any business logic. An expired credential raised an alert to the feed's owner instead of retrying forever. Adapters also kept response metadata and the partner's own timestamps, because collection time alone can't tell you whether the partner sent anything new.

Raw first

Every payload landed in object storage before anything parsed it, with a checksum, the collection key and the adapter version. A payload counted as committed only once both the object and its manifest were durable. Later stages passed references instead of copying large responses through every queue.

Keeping raw data turns a parser bug into a reprocessing job. You can investigate a bad deploy against the exact bytes it saw, then replay those bytes through the fixed parser and get a new transformation version with the original collection identity intact. Retention was still bounded by contract and privacy rules, replay or not.

Validate, quarantine, commit once

Validation checked schema, required fields, units and business rules before anything reached the store. It told an absent value apart from a parser failure, and it never guessed a unit. Valid records moved on. Rejected ones went to quarantine with a reason code and a pointer to the raw payload.

Commits were idempotent upserts keyed on partner, entity and source version. A retry could run twice and still produce one committed version, without needing exactly-once delivery anywhere in the chain. Deletes, late corrections and out-of-order records had explicit version rules, so replaying yesterday couldn't overwrite a newer fact.

Publish whole versions

A dataset version went out only when its required partitions met the completeness rule, and publishing meant flipping a version pointer atomically. The API and the prebuilt static snapshots read the same version. Both showed source freshness separately from publication time.

If a required source was incomplete, we kept the previous version or published a clearly marked degraded one. We never mixed new and old partitions under a fresh timestamp. Monitoring tracked source age, missing partitions and quarantine age separately from HTTP success, because the old pipeline's endpoint checks had stayed green over stale data.

Exceptions have owners, and replay is proven before cutover

Quarantine was a working queue, grouped by source and reason, with an owner and an age on every group. A reviewed parser or mapping fix authorized replay of a bounded range of raw data, and the resulting counts and totals had to reconcile before the new version was published.

We ran the old and new pipelines side by side across representative feed cycles and compared coverage, entity counts, corrections and key aggregates. When they disagreed, we investigated rather than forcing the totals to match. We also forced duplicate deliveries, partial writes and schema drift on purpose to see what broke. Consumers moved over one dataset version at a time, and rollback meant repointing them at an earlier compatible version without deleting the evidence needed for repair.

Results

Across 12 analysts that's 48 hours a week back, or 207.84 hours a month using 4.33 weeks. At $70 an hour, less the $3,500 a month the new pipeline costs to run, it comes to $11,048.80 a month of net capacity.

The analysts still review exceptions, and they should. Quarantine puts the ambiguous records in front of them instead of every routine row. The numbers above don't price any improvement in the decisions built on the data; I don't have evidence for that, so it isn't counted.

Earlier on, I'd show freshness and completeness to consumers, before any of the deeper pipeline work. A lot of hand reconciliation is people checking things the system could tell them, and a dataset's age and gaps printed next to the number answer some of those questions on their own.

Client name and identifying details are left out.