Your Free Probe Passed. You Still Measured the Wrong Box

Your Free Probe Passed. You Still Measured the Wrong Box

Tópico: Your Free Probe Passed. You Still Measured the Wrong Box
Categoria: Tutoriais | Programação & Tecnologia
Idioma Principal: Português (Conteúdo de Tecnologia)

Descrição do Conteúdo / Informações:
-------------------------------------------------------------------------
Your free probe passed on a scratch box. That green badge still measured the wrong machine. A green free endpoint is a cheap probe. It is not a production ship gate. Treat it like a probe, or you ship luck.

I keep watching agents pass on free compute. Then the same patch dies on the real model. Does that gap sound familiar to you? That gap is not a dumb model.

It is an eval-transfer bug in the harness. You measured the wrong machine the whole time.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I treat MonkeyCode as a scratch probe box here. Free model access plus a free server option cover that job. They do not issue a production verdict today. Below is the catalog I now run first.

What I measure instead of vibes

I no longer ask if the agent finished. I ask which frozen identity actually finished running.

An identity is a frozen tuple of fields. It is not a feeling at all.

• pinned model endpoint plus exact id

• server image plus worktree path

• tool schema hash after freeze

• fixture input plus named oracle

• seed, temperature, and timeout budget

Miss one field and the folklore starts immediately. Would you sign a release with no build sha?

Anti-pattern 1: Model-swap theater

Symptom

The free endpoint returns a clean looking patch. You paste that patch into another sampler. CI goes red on the second machine. You blame the second model without evidence.

Root cause

You treated two samplers as one agent. They share a prompt file on disk. They do not share a decision surface though. Sampling noise is not a stable personality trait.

Replacement

Pin the eval identity before any replay. Replay the same fixture on that identity. Refuse transcript contests across two different endpoints.

# example sketch: freeze identity before the probe
export EVAL_ENDPOINT="$FREE_MODEL_URL"
export EVAL_MODEL_ID="${EVAL_MODEL_ID:?pin this; do not guess}"
export EVAL_IMAGE="agent-sandbox:replay-3"
printf '%s %s %s\n' "$EVAL_ENDPOINT" "$EVAL_MODEL_ID" "$EVAL_IMAGE" >> eval-identity.log

Did you actually log the model id anywhere? If not, you only have a story. Those stories should not merge to main.

Anti-pattern 2: Shared-server amnesia

Symptom

Two agents share one free scratch server. The second run knows files it never wrote. You call that leftover disk real skill.

Root cause

The disk was dirty from the last fixture. Yesterday's files leaked into today's context window. Shared scratch is not true process isolation. Leftover env files feel like sudden genius.

Replacement

Boot a clean worktree per replay run. Destroy that worktree on the way out. No leftovers means no haunted context files.

# example: one worktree, one eval, then burn it
RUN_ID="$(date +%s)-$RANDOM"
git worktree add "/tmp/eval-$RUN_ID" HEAD --detach
trap 'git worktree remove --force "/tmp/eval-$RUN_ID"' EXIT
cd "/tmp/eval-$RUN_ID"
python replay_eval.py --fixture tests/fixtures/refund_timeout.json

Could a leftover lockfile explain the smart call? If yes, you measured the disk instead. You did not measure the agent at all.

Anti-pattern 3: Happy-path burn

Symptom

You spend the free run on the demo. The refund flow looks gorgeous in the screenshot. Production then dies on a conflict response. Nobody saved the ugly case for replay.

Root cause

Free compute feels scarce inside your head. You protect it for polished demo screenshots. Failure cases look ungrateful next to demos. So the probe never sees those cases.

Replacement

Spend the free probe on the ugly path first. Demos are optional during a real eval. Contracts are not optional during a real eval. Invert the queue before you burn the box.

Attack order I actually use:

• Timeout from the tool

• Duplicate id on retry

• Partial JSON in the stream

• Empty file that still exists

• Only then the happy path

# proposal: order fixtures by how often they lie
FIXTURES = [
"timeout_on_tool.json",
"duplicate_idempotency_key.json",
"partial_json_stream.json",
"empty_but_present_file.json",
"happy_refund.json", # last on purpose
]

def run_probe_queue(runner):
for name in FIXTURES:
result = runner.replay(name)
if not result.oracle_ok:
raise SystemExit(f"ugly path failed first: {name}")

Would you still ship if step two failed? That is the only question that matters here.

Anti-pattern 4: Latency theater

Symptom

The free endpoint answers faster than production does. You call the agent snappy without an oracle. You skip timeouts in the harness code. Slow production then looks like a model regression.

Root cause

You scored user vibe, not the contract. Fast and correct are different measurement axes. Mixing them hides stalls and silent retries.

Replacement

Record duration as telemetry beside the result. Never treat that clock as a grade. Grade the oracle against the world state. Log the clock next to that grade.

# example: clock is telemetry, not a pass condition
import json, time, urllib.request

def probe(payload: bytes, url: str, timeout_s: float) -> dict:
started = time.monotonic()
req = urllib.request.Request(url, data=payload, method="POST")
try:
with urllib.request.urlopen(req, timeout=timeout_s) as resp:
body = json.loads(resp.read().decode())
status = "ok"
except TimeoutError:
body, status = {}, "timeout"
latency_ms = int((time.monotonic() - started) * 1000)
return {"status": status, "latency_ms": latency_ms, "body": body}

Did the tool finish, or did you enjoy speed? Separate those answers on paper before merge.

Anti-pattern 5: Transcript-as-proof

Symptom

You read the chat and it sounds sure. You merge the patch after that reading. Tests never saw the side effect land. The ledger still double-refunds the same cents.

Root cause

Language is cheap on a free model. Side effects are not cheap at all. Confidence is not a checksum of state. Prose cannot close a money path safely.

Replacement

Oracle the world after the agent run. Check files, HTTP calls, and ledger rows. Do not grade the essay in CI.

# example: the transcript cannot pass this test
def test_refund_is_idempotent(tmp_path):
world = World(tmp_path)
agent_run(world, fixture="refund_twice")
events = world.ledger.read()
refunds = [e for e in events if e["type"] == "refund"]
assert len(refunds) == 1
assert refunds[0]["cents"] == 4099

If the ledger lies, the speech does not matter. Why are we still grading essays in CI?

The artifact: an Eval Identity Card

I keep one JSON card per probe run. No card means no claim worth publishing. This card is the whole method I use.

{
"eval_id": "refund-timeout-2026-09-14",
"endpoint": "free-model",
"model_id": "pin-me-do-not-guess",
"server": "free-scratch",
"worktree": "/tmp/eval-detach",
"tool_schema_sha256": "replace-with-real-hash",
"fixture": "tests/fixtures/refund_timeout.json",
"oracle": "ledger_refund_count_eq_1",
"seed": 17,
"temperature": 0,
"timeout_s": 30,
"result": null
}

Those nulls are honest before the probe runs. Fill them after the oracle actually returns. Do not prettify empty results to look done.

Here is the checker I want in CI. Label it a proposal, not a benchmark number. It is not a latency claim either.

# proposal: fail the job if the identity drifted
import hashlib, json, pathlib, sys

def schema_hash(path: str) -> str:
data = pathlib.Path(path).read_bytes()
return hashlib.sha256(data).hexdigest()

card = json.loads(pathlib.Path("eval_card.json").read_text())
actual = schema_hash("tools/schema.json")
if card["tool_schema_sha256"] != actual:
sys.exit("schema drifted; this green run is from another world")
if card["result"] is None:
sys.exit("no oracle result; a transcript is not a gate")

One probe, start to finish

• Write the identity card with result left null.

• Create a detached worktree and nothing else.

• Replay ugly fixtures before the happy path.

• Fill the card from the world oracle.

• Run the drift checker on that card.

• Only then open the chat transcript.

python check_eval_card.py
python replay_eval.py --card eval_card.json

Run both commands from a clean worktree. Two commands pin one identity with no folklore. Can your current log survive that bar today?

Decision table

Use this table before you trust a free run.

Symptom
What you tell yourself
What actually broke
Replacement

Shared files across runs
The agent remembered
Dirty worktree
Detach, then destroy

Green on free, red later
Prod model is worse
Unpinned identity
Replay the same card

Demo looks perfect
We covered the flow
Happy-path burn
Failures first

Fast replies
The agent is tight
Latency theater
Log ms, grade oracles

Confident chat
It understood the spec
Transcript-as-proof
Assert the ledger

Print the table and tape it beside the terminal. Keep it there until the folklore stops.

What this does not prove

A free endpoint can catch cheap obvious lies. It cannot certify a vendor model family. It cannot certify tail latency under load. It cannot certify your IAM or tenancy story.

I am not claiming transfer across model endpoints. I am claiming a cleaner negative test only. If the probe fails, you should not ship. If the probe passes, you still need the real gate. Do not confuse a found bug with a proved release.

Who should skip this

Skip this if you already pin models in CI. Skip this if every run is a throwaway VM. Skip this if your oracle is a real ledger test.

Also skip this if you need a production SLA. A free scratch server is not that SLA. Do not pretend otherwise on a launch checklist. This catalog is for people mixing probes and gates.

The habit I want

Start with the ugly fixture every single time. Pin the identity before the first token. Wipe the disk after the oracle returns. Grade the world, not the chat transcript.

Then, and only then, read the transcript aloud. Does that extra friction feel slower to you? Good, because slower is the entire point here.

Need a disposable box for that replay loop? MonkeyCode's free model access and free server option can host that practice. Treat the run as a probe, not a gate. Keep the ship gate somewhere honest and pinned.