Follow the Findings

The setup

I have written before about the 80% problem: the moment an AI-accelerated project stalls because generation has outrun comprehension, and the mental model that should let you reason about the next change has quietly collapsed. My argument then was that the countermeasure is not better prompting but deliberate process: small comprehensible deltas, decisions captured where they are made, documentation treated as the durable asset and code as the perishable one.

This project was the test of that argument under conditions I chose to make unfavorable to it.

There was no careful architecture phase. The requirements document that started the project was short. It described the overall game idea, the core mechanics, and a handful of specifics I cared about: sound, physics, two teams, turn taking, terrain, wind, isometric view. Everything else was left to be decided when we got there, in the extreme programming sense of that phrase. Architecture emerged through dialogue with the developer, which in this case was Claude, and through the pull requests that dialogue produced. I reviewed, I answered questions, I merged. The merge stayed mine by convention throughout, even where I had waived the tooling that would have enforced it.

All interaction with the agents ran through the terminal or through pull requests, and I set myself one further constraint: I never looked at the code. Not once in six days. I read the markdown record, the requirements, the plan, the findings, and nothing below it. In my earlier piece I argued for keeping the mental model alive through small, comprehensible increments. I kept that discipline, but the increments changed shape: instead of code deltas, bite-sized decisions. Each finding, each raised ambiguity, each answered question was a delta I could integrate fully before the next arrived. This ride was going to be with blinded windows, and the mental model I maintained would live entirely at the level of decisions and findings. If the record was not good enough to steer by, I would find out the hard way.

One thing should be said plainly before the numbers, because it answers the most obvious protest: none of what follows is what an AI assistant does out of the box. The record I am describing was produced inside a harness I designed for the purpose, a process layer governing how findings are registered, decided, built, and verified. Its internals are a subject for another day. What matters here is that the discipline was designed in, not hoped for.

Skeptics hear “decide when we get there” and translate it to “defer until it hurts.” The record shows something different, and it shows it in numbers.

Where the findings came from

Of the 89 findings registered across six days, exactly one came from a production incident. One.

Roughly a third came from the project reviewing its own design and code without being asked to: a dedicated design-quality pass, a security pass, and a whole-codebase pass triggered incidentally while investigating something else. Those unprompted reviews found, among other things, a HIGH-severity unauthenticated crash and a HIGH-severity concurrency hole in the join operation, a blind overwrite with no atomic claim anywhere in the codebase. Neither ever got the chance to be hit in the wild, because both were found before any user existed to hit them.

The rest came from decomposing requirements into work, from my direct requests, and from consistency sweeps that check whether the documentation still tells the truth.

This distribution is worth sitting with. The folk theory of AI-generated code says the defects hide in the parts nobody thought about, waiting for production to find them. This project’s defects lived somewhere else entirely: in the seams between working sessions, and in assumptions that felt too obvious to state. Scheduled self-review, the deliberate act of looking where nothing has yet gone wrong, produced more findings than every other source combined except the planned decomposition of requirements itself. Waiting for bug reports would have meant waiting for a crash that a security pass found first.

Decisions and builds move at different speeds

The severity distribution across the whole run has a shape I did not design and only noticed afterward: build work dominates the HIGH-severity findings, and decisions dominate the LOW-severity ones. Nothing that required a judgment call from me was ever urgent. The urgent things were always concretely fixable without me.

That is not luck. It is what “decide when we get there” looks like when it is done as a discipline rather than an excuse. The process converted ambiguity into a named decision the moment it was noticed, well before it blocked anything. Should a shot fired off the map be clamped or wasted? Who moves first, and why? What happens when you hit yourself? Each of these could have been quietly answered by whoever happened to be writing the code that day. Instead each was raised as an explicit decision with a stated failure scenario, and I answered nearly all of them in the same session they were raised, most within minutes.

Deciding when you get there works precisely because you arrange to get there early. By the time any of these questions could have become a live problem, it already had an answer to build against. The deferral that skeptics fear is real, but it is a failure mode of not noticing ambiguity, not a property of postponing decisions you have already named.

The one field bug

The single production incident deserves its own section, because it is the best thing in the record.

Playing my own game, I hit a state that made no sense: the interface seemed to be confusing my identity between duels. I reported it, and, being the sort of person who cannot resist, I included my diagnosis (cookie confusion) and my proposed fix (check the current game id against the database).

Neither survived contact with the investigation. The identity layer was tested and proven to behave correctly. The actual cause was a stale browser tab left open on an earlier duel, combined with the interface never showing which game you were looking at. And my proposed fix was explicitly declined, in writing, because it smuggled in a design change out of all proportion to the problem: a durable player identity spanning games, something the requirements deliberately excluded. What got built instead was the honest client-side form of the same idea.

The resolution record states plainly that my suggestion is deliberately not what was built, and why. The disagreement was not smoothed over, softened, or buried in a commit message. It stands in the record, addressed to me, with evidence.

I want to dwell on this because it answers the objection I hear most often about AI collaborators: that they agree with you. A process in which the human’s bug report is treated as a symptom to investigate rather than a diagnosis to implement, and in which the human’s proposed fix can be examined, ruled out, and refused with reasons, is doing something most human teams find difficult. It is also, not coincidentally, how the most valuable bug of the project got correctly fixed instead of plausibly patched.

Calibrated claims, or why the ending is believable

Every resolved finding pairs its claim with a named boundary: what was checked, and what was not. A cookie policy chosen for a production edge case notes that the edge case was never reproducible locally. The field bug’s resolution states outright that the original failure was never reproduced against the deployed host, because no database from that session survived. This sounds like pedantry until the last day, when the project verified everything against the real host and played a real duel to completion. Because every earlier claim had carried its own caveat, closing them at the end reads as the last domino falling, not a victory lap invented on the spot.

The record even checks itself. A late sweep caught an earlier resolution that had verified the wrong artifact: the fix was confirmed in a narrative runbook rather than in the deployed configuration the runbook described. Verification of verification turns out to be a real category of work, and it catches a class of error that nothing else will.

For symmetry, the code’s own scorecard: the physics shipped with exactly one defect, a tuning constant that let wind reach three quarters of gravity’s own magnitude, which makes for exciting artillery and very poor artillery. No review found it. No test found it. I found it because the game felt wrong. Some findings you can only reach by playing.

The question the server cannot answer

The structurally hardest problem in the project was not physics or rendering. It was this: the game deliberately has no user accounts, so “is this browser looking at the right duel?” is not a question the server can answer. Only the browser’s own memory can. That single fact produced three bug reports with identical symptoms and three different causes over the project’s life, each initially mistaken for the others.

Twenty years ago I wrote a PhD arguing that context is not a set of variables a system can encode, but something that emerges through participation, and that the encoding approach fails in principle, not merely in practice. It is a strange pleasure to watch that argument resurface as a production bug class in a toy artillery game. The identity that mattered here was never encodable server-side. It existed only in the situated history of a particular browser in a particular chain of actions. The fix was not a cleverer encoding. It was giving the client an honest account of its own participation, and giving the record a triage checklist so the next identical symptom gets diagnosed against all three known causes instead of the most recent one.

What I actually learned

The findings, not the game, are the return on six days of work. A uniform writeup applied to every finding, including the one-line fixes, is what makes 89 of them comparable and searchable months later; the overhead was visible and was never relaxed under time pressure, which tells you what it was judged to be worth. Decisions and builds are different kinds of work and should be allowed to move at different speeds. A third of everything that was ever wrong was found by looking before anything broke. And the most useful thing a human can contribute to this kind of collaboration is not architecture diagrams but well-timed judgment, concrete assets, and bug reports offered as symptoms rather than verdicts.

In this story I followed the rabbit. But I turned each page deliberately.

The model used in agents was Claude Sonnet 5