The Audit Became a Build Step

The Audit Became a Build Step

Tópico: The Audit Became a Build Step
Categoria: Tutoriais | Programação & Tecnologia
Idioma Principal: Português (Conteúdo de Tecnologia)

Descrição do Conteúdo / Informações:
-------------------------------------------------------------------------
I have audited my travel site's legal-status dataset twice. The first audit found the same fact stored in four places, drifting independently. The second found the drift had a fifth home — the prose — and ended with me writing that a half-derived page contradicting itself "is not a risk, it's a schedule."

Six weeks later the schedule delivered again. A country's status was wrong in one file and right in another. A different country sat in the wrong continent in the database, which quietly broke the links a component built from it — two dead links on a live page, generated from data everyone had already audited.

Both audits were real work and both were correct the day they shipped. That's the problem. An audit certifies a moment. The dataset has four writers — an admin panel, an agent editing files, me in a code editor, the occasional migration — and nothing in the system held the agreement after the auditor left. Every writer produces output that is valid on its own. The typechecker is happy, the build is green, and the site contradicts itself, because validity is checked per file and truth lives across files.

So the third audit is not a session. It's a script, and it runs before anything else in the build.

What the check actually does

The site's legal facts live in a Postgres database and four static TypeScript files — one for US states, one feeding the country detail pages, one feeding the maps, one feeding search autocomplete. The check pulls the live tables over the public read-only API, loads the four static files, and compares every value that appears in more than one place.

Two design decisions matter more than the diffing.

It loads the app's real modules, not a regex scrape of them. The check bundles the actual TypeScript files and imports them, then reuses the app's own slug transforms, name normalization, and merge logic. This sounds like overkill until you try the alternative: a scraper with its own parsing and its own slug rules validates a site that doesn't exist. Early on, my ad-hoc name normalizer stripped a cedilla differently than the app's did, and a country appeared "missing from the database" when it was there all along. The only comparison worth automating is the one the rendering code would make.

Not every mismatch is a failure. A contradiction between two sources a reader can reach — the database says one status, the search dropdown says another — is an error and kills the build. A country that exists in the static files but has no database row is a warning: nothing disagrees yet, but that page is running unmanaged, and edits made in the admin panel will never reach it. Errors are lies; warnings are blind spots. Conflating them either makes the check ignorable or makes it a liar itself.

Two smaller rules earned their place fast:

•
Failing to run is a failure. If the database is unreachable, the check exits nonzero and prints "this is not a pass." A verification step that silently passes when it can't verify is worse than no step — it launders outages into green builds.

•
The baseline must shrink. Day one produced 28 known conflicts, and blocking every deploy until all were resolved would have gotten the check deleted within a week. So known differences live in a baseline file — but every baseline entry that stops matching reality gets reported too. A baseline that only suppresses becomes a second copy of the lie. Mine went 28 → 26 → 0 in three days, and an empty baseline is the contract: any future disagreement, anywhere, fails the build.

The check audited my beliefs, not just my data

Building it forced me to read the actual data flow, and the data flow was not what I believed. For months I'd treated the database as admin-only bookkeeping — I'd even written it down as fact. Wrong. A merge function I'd forgotten about overrides the static file with the database row on every country detail page. The database had been winning on the most-read pages of the site the whole time, while the maps and search kept rendering the static values.

That's the four-way split in one sentence: the same fact had different authorities on different pages of the same site. No amount of auditing the values fixes that; I had to know which copy wins where before "disagree" even meant anything. The check now encodes that knowledge — it's the only place the whole routing of truth is written down and executed.

26 conflicts, one actual disagreement

The day-one errors split cleanly. Two were the dead links — wrong continent values in the database, five-minute fix. The other 26 looked like 26 separate factual disputes about 26 countries, and they were nothing of the sort. One store said "Medical" 51 times where the other said illegal or decriminalized, because the two stores meant different things by the word. One counted any statute with the word "medical" in it — including countries whose entire program is licensing crops for export, where no resident can be prescribed anything. The other described what a person there can actually do.

Arbitrating that country-by-country would have produced 26 judgment calls to re-litigate forever. Instead the fix was a definition, written at the top of the data files where the next editor — human or agent — can't miss it:

// STATUS FIELD — what the value means (settled, do not re-litigate)
//
// `status` describes WHAT AN ORDINARY VISITOR FACES, not what the
// statute book contains. Export-only licensing is NOT "Medical".
// "Decriminalized" means no CUSTODIAL penalty. When two statuses
// both apply, the one describing possession by an ordinary
// person wins.

One definition resolved all 26. And it settled which side to trust with evidence rather than vibes: the only two countries that disagreed in the opposite direction had both changed status in the real world years ago — the database had tracked the changes, the static file hadn't. The store that was right about the hard cases got believed about the rest.

The definition keeps paying. Later, seeding new rows, I hit two countries with nearly identical laws on paper — small-amount possession punished without prison in one, punished "administratively" in the other. The first is a fine or community service: decriminalized. The second allows fifteen days of detention: illegal, because fifteen days in a cell is custodial no matter what the statute calls it. Without the written rule, those two rows would have been decided by whichever mood I was in that day.

Closing the blind spot the check can't see

A comparison check has a structural limit: it can only check facts that exist in both stores. Sixty-three countries had pages but no database row — no contradiction possible, no protection either, and any admin edit to them silently went nowhere. So the last phase was seeding every missing row, region by region, each value verified against current primary sources before it went in.

That seeding was itself an audit — the last manual one — and it caught three statuses the site had carried wrong for months: a Balkan country whose medical program took effect this January (the top search result "contradicting" this was dated 2021 — it didn't refute the change, it predated it), an Indian Ocean island that quietly amended its drug law in 2022, and a South American country that replaced jail with counselling for small amounts back in 2022. Three more corrections that no diff between my own sources could ever have surfaced, because every internal copy agreed on the stale value.

The blind spot is now closed: every country on the site has a row, every row is compared on every build, and the check's first fully green run reported both stores in complete agreement for the first time in the site's history.

What I'd carry to any system with more than one copy of the truth

•
An audit is a snapshot; a build gate is a contract. If a fact has multiple writers, the agreement between copies needs an enforcer that outlives the session that created it.

•
Compare with the app's own logic or don't bother. A checker with its own parsing validates an imaginary site.

•
Errors are contradictions; warnings are blind spots. A reader-reachable disagreement fails the build. A gap gets counted, loudly, until it's closed.

•
A check that can't run must fail. "Couldn't verify" reported as success is the most expensive bug a verification step can have.

•
Baselines must shrink. Report stale exceptions as hard as new failures, and drive the file to zero. An empty baseline is the only version that means anything.

•
When two copies chronically disagree, look for a missing definition before arbitrating cases. Twenty-six disputes that share a shape are one dispute. Write the rule down where the data lives, and the next hundred edits inherit it.

Two audits taught me where the copies were. The check is what finally made them agree — and the next time they don't, I won't find out in an audit. I'll find out in a red build, minutes after it happens.