incident index

Seven ways my coding agents reported success while failing

2026-09-04 · lifestep.io

I run Claude Code and Codex workers in parallel every day. Four times now one of them has told me it succeeded while the system was in a different state than the report described. None of these were hallucinations. In every case the tool was telling the truth about something narrower than the question I was asking.

This is the index. Each entry is the claim, the evidence that contradicted it, what it cost, the root cause, and the check I run now. The checks are the point.

1. WAITING_APPROVAL is not COMPLETED

Claim

The process exited with code 0 and no error in the log.

Evidence

The task's output files were never written.

Cost

Most of an afternoon, plus a second run started on top of unfinished work.

Root cause

The worker printed "want me to proceed with this plan?" and quit waiting for an answer nobody was going to give. Exit 0 meant the process ended. It did not mean the work happened.

Check

Record the exit status out of band — the launcher writes $? to a file, and the watcher reads that file rather than the log tail. A background process's status disappears the moment the parent stops waiting for it.

Regression

My first fix was worse than the bug. I made a missing completion marker mean “stalled”, then applied one tool's marker to a different worker and scored four healthy runs as stuck. A missing marker is never a stall. Markers are tool-specific diagnostics; the disposition comes from the recorded exit code plus reading the result body. An exit of 0 is a candidate, not a verdict — the worker may have exited cleanly having only asked a question, which is exactly this incident.

2. GREEN HEALTH CHECK is not SUCCESSFUL DEPLOY

Claim

deployment health checks passed in the deploy log.

Evidence

The running revision was the previous one.

Cost

I shipped nothing and believed I had.

Root cause

The deploy had already failed and rolled back. The line I read was the rollback's health check passing. Both paths print the same sentence. Above it sat post-switch gate failed, which I skimmed past because the last thing I saw was green.

Check

One unambiguous success marker per pipeline. Mine prints Release deployed: <path> and nothing else counts. Health checks prove something answered, not that the right thing answered.

Regression

Verify the marker from outside the deploy script — resolve the active release path, hit the endpoint, confirm the revision matches what was pushed.

3. BROWSER AUTOMATION is not SAFE SANDBOX

Claim

Task completed, no errors.

Evidence

The Chrome profile was about 2 GB smaller and the Bookmarks file had been rewritten at a timestamp matching the run.

Cost

Roughly 2,000 bookmarks, unrecoverable. The automation had already overwritten the backup slot Chrome keeps.

Root cause

The script used a real browser automation library, correctly. It pointed at my actual Chrome profile instead of a throwaway one. No scanner flags that — the tool was legitimate and the target was wrong.

Check

Abort before launch if the profile path is the real one.

case "$USER_DATA_DIR" in
  *"Library/Application Support/Google/Chrome"*|"")
    echo "ABORT: real profile"; exit 1;;
esac

Regression

Launch with the variable unset and assert a non-zero exit. It has caught me twice since.

4. NETWORK BLOCKED is not AUTH FAILED

Claim

gh auth status reports the active token is invalid.

Evidence

The same token worked in my own shell.

Cost

An hour chasing credentials, two worker runs wasted.

Root cause

The sandbox had network disabled, DNS never resolved, and gh worded a resolution failure closely enough to an auth error that the model concluded credentials and stopped.

Check

Prove transport first, with no credential in the child process.

env -u GH_TOKEN -u GITHUB_TOKEN \
  curl -sS --max-time 8 -o /dev/null https://api.github.com

A resolve failure is transport. Reachable but no response is the endpoint or a proxy allowlist. Only after that does asking about auth mean anything.

Regression

The tell I missed: injecting a known-good token changed nothing. If a working credential does not move the error, it is not the credential.

5. A GREEN TEST SUITE is not A WORKING CHECK

Claim

The verifier's whole fixture suite passed, so the verifier worked.

Evidence

Run against the real page, it reported the opposite of the truth.

Cost

A scheduled decision would have been made on an inverted result.

Root cause

The check was printf '%s' "$body" | grep -q PATTERN under set -o pipefail. grep -q exits at the first match, the writer takes SIGPIPE, and pipefail promotes that to a failed pipeline — so a pattern that did match reads as absent. It only triggers once the input exceeds the pipe buffer (64 KB on macOS). Every fixture was a few hundred bytes; the real page was 41 KB and climbing.

Check

Drop the pipe: grep -q PATTERN <<< "$body". And test the classifier with an input larger than the pipe buffer, generated at test time rather than committed as a fixture. Reproduced the failure at 5.7 MB before and after.

6. FAILED TO MEASURE is not MEASURED ZERO

Claim

The usage counter recorded 0 calls.

Evidence

The collection had failed. Nothing had been counted at all.

Cost

None yet — caught before the retirement decision that would have used it.

Root cause

grep -c exits 1 when the count is zero. With a || echo "" fallback, a genuine zero and a broken connection produced the same empty string, and a ${x:-0} default then wrote it down as 0. For a counter whose job is to answer “did anyone use this?”, that turns we could not measure into there was no demand.

Check

A non-numeric result becomes NA, and a row with NA in any decision field is refused rather than written. Verified both ways: a reachable host writes the row, an unreachable one exits non-zero and leaves the file untouched.

7. A CHECKER THAT ONLY SEES ITS PASSING CASE

Claim

The detector was working, because it flagged the bad input.

Evidence

It also flagged the corrected text. It had been matching a word that appears in almost any page.

Cost

The fix would have been reported as the defect.

Root cause

An error pattern contained the bare word required — meant as part of “login required”, separated by an alternation. The real page carried "query-input":"required name=search_term_string" in its JSON-LD, so every page matched. The check ran before the specific one it was meant to defer to, and hid it.

Check

Every classifier gets a fixture that must match and one that must not. A checker exercised only on its passing case is indistinguishable from a function returning a constant. Prove the guard fails on the old code before trusting that it passes on the new.