A coding agent in 5 files, plus a skeptic that catches fake fixes

Hand a coding agent a failing test and tell it make it pass . Sometimes it fixes the bug. Sometimes it edits the test, hardcodes the expected value, or deletes the assertion — and reports done ✅ with a straight face. Every "build an agent" tutorial stops before this problem. This one is built around it.

The agent itself is the easy part — a while loop around a chat model with one tool. 01_loop/agent.py is ~70 real lines that fix a real bug, live:

The hard part — Chapter 5, and the reason this repo exists — is an independent skeptic that doesn't take "done ✅" on faith. It checks the code's behavior against contract probes the agent never saw, so a fix that only games the visible test — editing it, or hardcoding/stubbing the source — is caught deterministically, no LLM required (a model judge reads the full diff on top, as a second opinion). You build both, one small chapter at a time. The scaffolding you're assembling has a name now — the harness — and this is harness engineering , from scratch.

git clone https://github.com/Saivineeth147/skeptic && cd skeptic python3 -m venv .venv && source .venv/bin/activate # recommended (system Python blocks pip: PEP 668) pip install -r requirements.txt # runs on any model via any OpenAI-compatible endpoint — one key, no billing setup: echo ' OPENROUTER_API_KEY=sk-or-... ' > .env # grab a key at openrouter.ai cd demo python ../01_loop/agent.py " the cart test is failing, fix it " # ⚠ runs in this dir; it's a sandbox demo Defaults to Claude Sonnet on OpenRouter; point AGENT_MODEL / OPENAI_BASE_URL at OpenAI, a local Ollama/vLLM server, or anything OpenAI-compatible. A full run costs a few cents (or nothing, on a free model).

Then watch the skeptic catch cheats — including the sneaky one that never touches the test:

python ../05_verify/verify.py --cheat # edits the TEST to pass -> ❌ REJECTED python ../05_verify/verify.py --cheat-stub # stubs the SOURCE, tests green -> ❌ REJECTED python ../05_verify/verify.py --cheat-eq # __eq__ that equals ANYTHING -> ❌ REJECTED python ../05_verify/verify.py # an honest agent fixes the code -> ✅ VERIFIED The cheat runs are the point: in each, the code passes the visible test yet is rejected, because a hidden contract check the agent never saw exercises the behavior the test left out. That's the whole idea — you can only verify work with a check the worker couldn't optimize against. ( --cheat-eq is the nastiest: an object whose __eq__ returns True for everything satisfies every bare assert in existence — the oracle catches it because its comparisons are type-pinned, and its probe requires a completion sentinel so an early sys.exit(0) can't fake a clean exit either.)