I audited my own AI-generated refactor and found 46 bugs. Here's what that taught me.

I audited my own AI-generated refactor and found 46 bugs. Here's what that taught me.



Tópico: I audited my own AI-generated refactor and found 46 bugs. Here's what that taught me.
Categoria: Tutoriais | Programação & Tecnologia
Idioma Principal: Português (Conteúdo de Tecnologia)

Descrição do Conteúdo / Informações:
-------------------------------------------------------------------------
How a 1,920-line file turned into 410 lines, why "it works" was a lie twice, and why the fix wasn't a smarter model — it was a gate that doesn't trust anyone, including me.



The setup


I maintain YouMindAG, an open-source CLI (npx youmindag) that injects project context — architecture, dependencies, rules, DB schema — into AI coding tools like Claude Code, Cursor, opencode, and Copilot, so they don't have to rediscover your codebase from scratch every session.

By v2.7.0, the CLI's entrypoint, bin/run.mjs, had grown to 1,920 lines. One file. Install logic, upgrade logic, seventeen subcommands, vault population, dev-server wrapping, AST-based tracing — all of it, flat.

I did what you're supposed to do: I modularized it. With an AI coding agent doing the heavy lifting and me reviewing, I split it into lib/ — one file per concern, one file per command family. By v2.9.0, bin/run.mjs was down to 410 lines, functioning only as an orchestrator.

Clean diff. Tests green. I shipped it.

It was broken in seven different ways.



Round one: three ReferenceErrors, found by hand


A few smoke tests in, three commands started throwing ReferenceError: RESET is not defined and similar. I traced it manually:

•
lib/graphify.mjs used a console-color constant RESET in 11 places. Zero imports of it.

•
lib/populate.mjs had the same missing import, silently breaking the summary output of the vault auto-population step.

•
lib/commands/misc.mjs referenced a VERSION constant that had never been passed in — it was a global in the old monolith, and extraction turned it into nothing.

I fixed the three, and — this is the part I want to be honest about — I did not go looking for a fourth. I wrote eight regression tests for the three I'd found, ran the suite, saw 58/58 green, and shipped v2.9.1.

Eight tests, three real bugs found by reading error messages one at a time. That's not an audit. That's whack-a-mole with good intentions.



Round two: I finally ran the tool made for exactly this


A couple of releases later I asked myself an obvious question I'd skipped: if extraction can silently drop an import once, what's stopping it from doing that seven more times in files I haven't touched yet?

So I added ESLint with a single rule that matters here — no-undef — and pointed it at everything under bin/ and lib/:

// eslint.config.mjs
{
rules: {
'no-undef': 'error',
},
files: ['bin/**/*.mjs', 'lib/**/*.mjs'],
}

Then I ran it.

46 undefined references. Spread across 7 files. 7 of 14 CLI commands broken or silently failing.

lib/fs-helpers.mjs      +readFileSync
lib/commands/db.mjs     +createInterface, parseEnvFile, hasPostgres  (+ a CWD→cwd typo bug)
lib/commands/dev.mjs    +mkdirSync, openSync, closeSync, appendFileSync,
dirname, execSync, spawn, writeYoumindagData
lib/commands/misc.mjs   +rmSync, relative, extname, execSync,
createInterface, findProjectFiles
lib/commands/sync.mjs   +mkdirSync, dirname, execSync, extname
lib/commands/trace.mjs  +join, existsSync, execSync
lib/commands/watch.mjs  +watch, statSync, populateVaultFiles
lib/vault.mjs           +writeYoumindagData  (lost during extraction entirely)

Some of these were commands that would crash loudly on first use — annoying, but at least visible. Others were worse: youmindag dev --wrap and youmindag sync were failing in code paths that only execute on specific flags or specific project states, meaning they could sit unnoticed for weeks in a project that never happened to hit that branch.

I fixed all 46, then verified every one of the 14 CLI commands manually against a real project — not just "does the test suite pass," but "does youmindag db, youmindag dev --status, youmindag trace --client X each actually do the thing" — one at a time.



The part that actually matters


The bug count isn't the interesting part. Missing imports after a mechanical extraction are a known failure mode; anyone who's done a big refactor by hand has hit this. The interesting part is why I missed 46 of them the first time and only 3 the second, and why "read the diff carefully" was never going to close that gap.

When an AI agent extracts a function from a 1,920-line file into a new module, the failure mode isn't wrong logic — the logic is usually copied verbatim. The failure mode is forgetting what the extracted code silently depended on. RESET used to be defined once, at the top of the monolith, and every function in that file could reach it without ever importing anything. Once you cut that function out into its own file, that implicit access disappears — and nothing about the extracted code looks wrong. It reads perfectly. It's only wrong at runtime, on the exact line that reaches for something that isn't there anymore.

And here's the uncomfortable bit: I was reviewing that same code with the same kind of attention an LLM has when it writes it — I was reading it for whether it looked correct, not tracing every identifier back to a binding. That's a slow, mechanical, unglamorous check. It's exactly the kind of check a human reviewer skips under time pressure, and exactly the kind of check a static analysis tool never skips, because it doesn't get tired and it doesn't trust that something "looks fine."

no-undef isn't a smart rule. It doesn't understand your architecture, doesn't know what a good abstraction looks like, doesn't have opinions. That's precisely why it caught what I didn't: it isn't pattern-matching on "does this look like working code," it's mechanically checking "is every identifier bound to something." My eyes are good at the first question and bad at the second. A linter is the opposite, and after this I stopped assuming my eyes were enough.



The fix wasn't a smarter model — it was a gate


The real fix in v2.9.3 wasn't "be more careful next time." It was making carelessness structurally impossible to ship:

"scripts": {
"test": "node --test",
"lint": "eslint bin/run.mjs lib/*.mjs lib/commands/*.mjs",
"verify": "npm run lint && npm test",
"prepublishOnly": "npm run verify"
}

prepublishOnly runs automatically on every npm publish. If lint or tests fail, the publish fails. I tested this by deliberately breaking a file and confirming the publish got blocked before it reached npm.

This means the class of bug that shipped in v2.9.0–v2.9.2 — a missing import slipping through review — cannot ship again without someone deliberately bypassing the gate. Not "probably won't." Cannot, structurally, by default.



What I'd tell someone auditing their own AI-generated refactor


•
A green test suite tells you the paths you tested work. It says nothing about the paths you didn't think to test. 58/58 passing coexisted with 7 broken commands, because the tests covered logic, not every command's happy path against a real project.

•
"I reviewed the diff" and "I ran a tool that checks a specific invariant" are not the same kind of confidence. The first catches what looks wrong. The second catches what is wrong regardless of how it looks.

•
If your refactor involves extraction — pulling code out of one file into several — assume implicit dependencies broke until a tool proves otherwise. Don't go looking for them by reading; that's how I caught 3 out of 46 the first time.

•
The fix that survives is the one that doesn't depend on remembering to do it again. I didn't add ESLint because I got smarter. I added a gate because I don't trust future-me — running low on time, mid-refactor, eager to ship — any more than I trust the AI that wrote the extraction in the first place.

YouMindAG is at v2.9.3 now, MIT-licensed, on GitHub. The commit history above is real and public if you want to check my math — git log doesn't round up.