Exit 0 but Broken: 6 Silent Failures and the Checks That Caught Them

Written by Claude (Opus 5.5) and checked by Claude (Fable 5.2), from the repos, their notes and the session transcripts. Earlier Claude sessions found and fixed these; dested ran into most of them and made the calls. The first person below is dested's voice.

For 131 days, vector search on one of my apps was off and nothing said so. Searches still returned results. They were keyword matches, and a try with an empty catch made sure nobody could tell the difference.

That's the worst of six from the last four months where something reported success while the thing underneath was broken. Each one below has what said ok, what was wrong, and the check that catches it now.

#An empty catch kept search on keywords for 131 days

The search code embedded each query, ran a vector lookup, and merged the hits with a keyword search:

} catch {
  // pgvector not ready / embed failed → keyword-only fallback.
}

From June 23 the production env had no OPENAI_API_KEY. Every embed call threw, the catch ate it, and the keyword half kept answering. Uploads stopped being analyzed too: 1,874 of 4,663 images had no description or embedding. It came out on November 1, when I asked an agent whether embeddings were still wired up.

The fix logs the fallback, prints an error at boot when the key is missing, and adds bun run check:search -- --prod: key, pgvector, index, analysis coverage and a live vector query, exit 1 when any of them fail.

#CI went green while the deploy rolled back

An agent pushed, and the GitHub Actions run for that exact commit came back success. The site still served the old JavaScript bundle. The box had no memory for a second copy of the service, so ECS logged "unable to place a task … insufficient memory", the circuit breaker rolled back, and CI's aws ecs wait services-stable took the rollback as a steady state.

The agent caught it by comparing the bundle hash prod served with the one it built, then /healthz: uptime 1,164 seconds, a container 19 minutes old. New services in my deploy tool now replace the task instead of rolling, and an EventBridge rule emails me on SERVICE_DEPLOYMENT_FAILED.

#A seed script reported Success and never ran

Before a run-through on a beta site, an agent seeded test accounts on the server with a remote command. The command came back Success, and the agent told me the accounts existed. The script only runs main() when it's the entry file (if (import.meta.main)), and the agent had loaded it with a dynamic import(). Nothing ran, and nothing failed.

The fix runs the seed as the entry file, and the seed prints a line per account it makes. No lines means it didn't work.

#/healthz said 200 while every rendered page said 500

The same morning I opened that beta site and got a 500. /healthz and the game page returned 200. Every server-rendered page (/, /pricing, /sign-in, /admin) returned 500 with jsxDEV is not a function.

The deploy had run from a session whose env carried NODE_ENV=development, set by one of my own Claude Code plugins, so the SSR bundle was built against react/jsx-dev-runtime. The deploy script now builds with NODE_ENV=production whatever the shell carries, and the gate script that runs before a push unsets it.

#aws s3 sync skipped 1,634 of 3,804 files

Before deleting 14 dead S3 buckets, an agent downloaded each with aws s3 sync. One bucket held 3,804 files and 2,170 landed. The rest had names up to 861 characters, past NTFS's 255-character limit for one name, and sync skipped them silently.

A count of remote files against local ones caught it before anything was deleted. The missing files were fetched one by one under hashed names, with a manifest mapping each back to its key.

#My browser tool printed a green check on a 404

Agents verify fixes with a browser CLI of mine. Opening a page that returns 404 prints a check mark and exits 0. A click that opens a native confirm() dismisses the dialog and prints ✓ clicked "Delete project", exit 0, while the page says kept.

An evaluation found both on November 9. Neither is fixed yet. An agent that trusts the tick reports a fix as verified when it isn't, so the check list for this blog's drafts asks curl for the status code and greps the page.

#The six in one table

Four more are written up in other posts: .wslconfig ignoring autoMemoryReclaim under [wsl2] and a Windows file watcher that missed changes over 9P (moving 32 repos to WSL), an Apple Ads dashboard at $0.00 because the default range excludes today (Apple Ads for a new app), a cost table reading a token field that misses about 80% of the thinking (more thinking, same goose), and a git add whose commit never landed (the 22 GB .git folder).

#Checks to add to your own scripts

1. Log every fallback, then probe the path it hides. A nonsense query only vector search can match returned 0 results with the key missing and 60 with it. (Search.)

2. Ask prod what it's running after a deploy. The served bundle hash should match the build, and /healthz uptime should be younger than the deploy. (Deploy.)

3. Assert on what a script says it did. A seed that prints nothing made nothing. Exit 0 only says it didn't crash. (Seed.)

4. Health-check a real page. /healthz proves the process is up, not that it renders. (SSR.)

5. Count before you delete, then compare keys. (Backup.)

   remote=$(aws s3api list-objects-v2 --bucket "$B" --query 'length(Contents[?!ends_with(Key, `/`)])')
   local=$(find "$DEST" -type f | wc -l)
   [ "$remote" -eq "$local" ] || { echo "MISMATCH: $remote in S3, $local on disk"; exit 1; }

6. Get the status code yourself. (Browser tool.)

   code=$(curl -s -o /dev/null -w "%{http_code}" "$URL")
   [ "$code" = 200 ] || { echo "$URL returned $code"; exit 1; }