Last Green Tag First: Bisect an OSS Regression Before the Patch Review

Last Green Tag First: Bisect an OSS Regression Before the Patch Review

Tópico: Last Green Tag First: Bisect an OSS Regression Before the Patch Review
Categoria: Tutoriais | Programação & Tecnologia
Idioma Principal: Português (Conteúdo de Tecnologia)

Descrição do Conteúdo / Informações:
-------------------------------------------------------------------------
A mid-size Python CLI received an issue that empty HTTP_PROXY values were now being sent to child processes as blank strings. A well-meaning contributor asked a coding model for a patch and received a thirty-line rewrite of the process launcher. The pull request looked clean in the diffstat, yet it also changed timeout handling that had been stable since the v3.2.0 tag. Maintainers closed the PR after a reviewer traced the real regression to a one-line defaulting change in a logging helper.

That pattern is now common in open-source inboxes, because models optimize for the symptom in the issue title. They rarely search for the first commit that changed documented behavior on a tagged release. A cheaper sequence is mechanical: prove the last release green, prove HEAD red, then bisect to one commit. Only after that pin should any model be invited to propose or review a patch.

Treat the last release tag as the control group

Issue templates rarely state which release last worked for the reporter, so contributors patch HEAD as if the entire tree were guilty. Maintainers then pay for that guesswork during review, because unrelated helpers get rewritten in the name of safety. A tagged control group makes the claim falsifiable for anyone who can install the test extra. If the last tag already fails the oracle, the report is not a regression and the patch needs a different contract.

The control group also keeps a later model review inside a small packet of evidence. A coding model that never sees the blamed commit will invent architecture around the issue title. The bisect log is a short, citable artifact that can travel with the pull request. Reviewers can replay it without inheriting a dirty local worktree or a half-applied virtualenv.

Worked example: empty proxy values

The files below are a labeled worked example, not a claim about any production codebase. The documented CLI contract treats a missing or empty HTTP_PROXY as unset for child processes. After a commit that "simplified defaults," empty strings started leaking into subprocess environments. An oracle that fails for that reason alone keeps the later patch inside one function.

# proj/envutil.py
from __future__ import annotations

import os
from typing import Mapping

def child_env(overrides: Mapping[str, str] | None = None) -> dict[str, str]:
"""Build an env dict for child processes.

Empty values are treated as unset, matching the v3.2 CLI contract.
"""
env = {k: v for k, v in os.environ.items() if v != ""}
if overrides:
for key, value in overrides.items():
if value == "":
env.pop(key, None)
else:
env[key] = value
return env

# tests/oracle_empty_proxy.py
from __future__ import annotations

from proj.envutil import child_env

def test_empty_http_proxy_is_unset(monkeypatch):
monkeypatch.setenv("HTTP_PROXY", "")
env = child_env()
assert "HTTP_PROXY" not in env

def test_missing_http_proxy_stays_missing(monkeypatch):
monkeypatch.delenv("HTTP_PROXY", raising=False)
env = child_env()
assert "HTTP_PROXY" not in env

A contributor who starts from the issue text might wrap the whole launcher in extra filtering. The oracle above fails for one contract only, which is the point of the control group. Neighboring timeout helpers stay out of the diff until bisect says they belong there. Maintainers can read the oracle in a minute and know what "fixed" is supposed to mean.

A maintainer-runnable bisect script

git bisect run needs a deterministic exit code from a script that does not care about chat history. Exit 0 means the bug is absent, exit 1 means the bug is present, and exit 125 means the tree cannot be tested. Any other code aborts the search and leaves reviewers with a half-finished log. The script below is meant to live on the topic branch, or to be pasted into the pull request body.

#!/usr/bin/env bash
# scripts/bisect-repro.sh
# Worked example: replayable oracle for an empty-proxy regression.
set -euo pipefail

ROOT="$(cd "$(dirname "$0")/.." && pwd)"
cd "$ROOT"

if [[ ! -f pyproject.toml && ! -f setup.cfg && ! -f setup.py ]]; then
echo "skip: Python project metadata missing in this commit" >&2
exit 125
fi

python -m pip install -e ".[dev]" -q || exit 125

# Isolate the oracle from the rest of a possibly red suite.
if python -m pytest -q tests/oracle_empty_proxy.py; then
exit 0
fi
exit 1

# Record the control group, then search toward HEAD.
git fetch --tags --prune
git checkout v3.2.0
bash scripts/bisect-repro.sh # must exit 0; otherwise this is not a regression

git checkout main
bash scripts/bisect-repro.sh # must exit 1; otherwise the bug is not on HEAD

git bisect start
git bisect bad HEAD
git bisect good v3.2.0
git bisect run bash scripts/bisect-repro.sh
git bisect log > bisect.log
git bisect reset

The 125 skip path matters on real repositories, where merge commits and missing extras would otherwise be labeled good. A skipped commit keeps the search honest instead of poisoning the good set with an unbuildable tree. After the pin, git show --stat on the blamed SHA and git tag --contains tell reviewers which releases already shipped the fault. Those two commands belong in the pull request body beside bisect.log.

Decision table for when a model may review

The table is the workflow gate, not a style preference for commit messages. If a row says the model stays out, the contributor keeps gathering evidence instead of drafting a rewrite. Tokens are still wasted when the blamed commit is unknown and the oracle is not replayable. Maintainers can apply the same rows without reading any chat transcript.

Observation after the oracle
What to do next
Model review allowed

Last tag fails the oracle
Not a regression; write a characterization test instead
No

HEAD passes the oracle
Cannot reproduce; freeze locale, cwd, and fixture paths
No

Oracle is flaky on the same commit
Fix seed, time, and network before any search
No

Bisect lands on a merge commit
Bisect the merged topic branch by parent
No

Pin is one function in one file
Draft a minimal patch against that hunk
Yes, scoped packet only

Pin touches generated code or lockfiles
Re-run codegen; do not hand-edit the artifact
No

Pin is a dependency bump
Read the upstream changelog; consider pinning
Changelog only

Most rejected AI patches fail the first three rows and still look productive in a chat window. The pull request then arrives without a replayable pin, so reviewers repeat the archaeology. The boring gate is the feature: it names the moment when a model is allowed to speak. Until that row is reached, the only artifacts that matter are the oracle and the bisect log.

Draft the patch against the blamed commit

Once bisect names a commit, the patch should restore the documented contract with the smallest hunk that makes the oracle pass. Drive-by refactors in neighboring helpers should stay out of that commit, even when a model offers them as cleanup. git range-diff helps when two drafts rewrite the same idea, because it shows whether the second draft still targets the blamed lines. The topic branch can start from main while the PR body still cites the pin.

git checkout -b fix/empty-http-proxy main

# After editing proj/envutil.py, keep the oracle green and the suite green.
python -m pytest -q tests/oracle_empty_proxy.py
python -m pytest -q

git add proj/envutil.py tests/oracle_empty_proxy.py scripts/bisect-repro.sh
git commit -m "fix: treat empty HTTP_PROXY as unset in child env

Regression introduced in <bisect-sha>.
Oracle: tests/oracle_empty_proxy.py
Replay: bash scripts/bisect-repro.sh"

# Optional: compare two drafts without losing the pin.
git range-diff main fix/empty-http-proxy-v1 fix/empty-http-proxy-v2

The commit message is part of the artifact, because maintainers should not reconstruct the bisect from a chat export. Citing the SHA, the oracle path, and the replay command is enough for a reviewer who never saw the original thread. A second draft that drops that citation is a weaker patch even if the diffstat looks smaller. Range-diff is then used to recover the pin rather than to debate formatting.

A scoped review packet, not a whole-tree dump

After the pin, a coding model is useful as a second reader, not as an archaeologist for the whole repository. The packet should contain the blamed commit, the oracle, the proposed diff, and the documented contract. It should not contain secrets, .env files, private traces, or unrelated application code. If any of those files are required to reproduce, the review stays on a private runner.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access is enough for this step when the packet stays small and the three checks below are explicit. The operator-supplied free server option is relevant when a laptop cannot install the project's native extras, because bisect must run against a clean clone rather than a dirty local venv. The model is asked only whether the diff restores the oracle, whether it widens public surface, and whether the commit message cites the bisect SHA.

A review prompt that stays inside that scope looks like the following template. It is not a recorded production session, and it is allowed to answer that evidence is missing.

You are reviewing an OSS regression patch. Do not propose a rewrite.

Inputs:
1) git show --stat <bisect-sha>
2) git show <bisect-sha> -- proj/envutil.py
3) tests/oracle_empty_proxy.py
4) git diff main...HEAD

Checks:
- Does the diff make the oracle pass for empty and missing HTTP_PROXY?
- Does the diff change any function that the blamed commit did not touch?
- Does the commit message cite the bisect SHA and the replay script?
- List any remaining public-API risk in one short bullet list.

If evidence is missing, say "insufficient evidence" instead of guessing.

That last instruction is the reason bisect happens before review, because models fill gaps with plausible architecture. Maintainers then spend the review cycle undoing those invented helpers and restored "simplifications." A packet that may answer "insufficient evidence" is cheaper than a confident patch against the wrong layer. The bisect log remains the source of truth if the model and the contributor disagree.

Limitations

This workflow assumes a deterministic oracle and annotated tags that can serve as a control group. Projects that never tag releases cannot form that group without an agreed good SHA written into the issue. Flaky suites poison git bisect run, because a random red on a good commit sends the search the wrong way. Native extensions, network fixtures, and GPU tests often need 125 skips, and those skips can leave a wide blame range instead of a single commit.

Remote review has a second constraint that no free-tier workspace removes for the contributor. Embargoed security issues, private customer traces, and credentials in fixtures should not leave the maintainer's machine. The bisect script also does not replace project CI; it only names the first bad commit before CI is asked to judge a patch. Generated code and lockfile bumps still need their own toolchain, not a model-authored edit.

Who should not use this approach

First-time contributors who have not yet run the project's test extra should not start with a model review of a guessed diff. Feature requests that add behavior, rather than restore a tagged contract, need a design thread instead of a bisect log. Documentation-only changes and typo fixes do not benefit from this ceremony and will annoy maintainers if the PR is padded with scripts. Teams under a disclosure embargo should keep both the oracle and the patch on private runners.

The durable output is a replayable script, an oracle that fails for one reason, and a commit that cites the first bad SHA. Maintainers can ignore the model entirely and still trust the pin on a clean checkout. Contributors who already reproduce locally can attach bisect.log and the replay command so the next reviewer does not repeat the search.