Stuck in review

The review stage is where most people struggle with coding agents. I did too, until I fixed the planning as well as the review.

Last month I was happy that I could close my laptop and Claude would keep going. This month went into what it brings back, because reviewing that work is where most people struggle with agents. Nearly all of this issue circles around that, from my own setup to a Harvard paper.

What I studied

My review process, which until very recently didn’t work great.

The specific problem I struggled with for quite a while was that my agentic reviews never finished. It was virtually impossible for Claude to conclude that no more defects could be found. Every new run turned up something new, seemingly forever.

Mostly it got stuck on documentation and comments, but there was also the occasional remark on symmetry between different parts of the code, function signatures, and scoping. Prose was the worst. A review would always find something to do better, so a change never converged.

The cause is mechanical. Rewriting a sentence makes that sentence part of the change, so the next review reads the new wording and prefers a third one. A bar that admits “could be clearer” never runs out of candidates.

This process of quality assurance can be tackled on two fronts. I did several iterations of tweaking and polishing how Claude reviews the work, which helped quite a bit. To fully solve it, however, I had to attack it from the other front as well — deciding what goes into a change.

It’s curious that it took me so long. I’ve known for decades that controlling scope helps, and I’ve worked that way with Claude from the start: I define why we need the change, clarify the acceptance criteria, set a definition of done and review Claude’s plan before it writes any code. Because the planning is that rigorous, I can trust its implementation.

What the planning lacked was a criterion of size and cohesion. Cutting work into small units is an old principle, and it serves just as well with agents, if not better. A small change is easier to get right and easier to review afterwards.

I’ve held for a long time that size is only a symptom, though. What you’re really after is cohesion, which is the better way to slice work. So the rule I use is to scope work as one change that a reviewer can hold in their head and judge against one stated intent. A big change that’s all one concern stays whole, while a small one that mixes two gets split.

Once I planned that way, a review run came back and reported that nothing worthwhile was left.

I was skeptical, which is a good default. But the changes were now small enough to review by hand, and I couldn’t find anything either. A dozen or so rounds of that and I felt ready to trust it.

My whole cycle works now, from defining the work and planning the implementation to reviewing the change and deploying the new version. It took time to get here, and I’m certain I’ll have to keep tweaking it.

A tool that earned its place

Small, separate changes need somewhere to live apart from each other, and git worktrees are how I do that. One repository, several checkouts, each on its own branch.

Every task now gets its own tree, and a tree holds exactly one task. Two agents can work side by side without touching each other’s files, and when a change reaches review it holds that one task and nothing else.

It isn’t free, though. A new tree only gets the tracked files, so everything else has to be provisioned. In one of my projects that means linking the shared secrets, giving each tree its own install directory so two sessions don’t overwrite each other’s binaries, and copying Claude’s local settings so its tools work there too. Each of them fails silently when it’s missing, so a hook does it on every session start.

Network ports were harder. I gave each tree its own at first, but external OAuth2 sign-in only accepts a redirect to the one I’d registered, so I backed that out and resigned myself to having only one tree run the app at a time.

The cost is in these mechanics, and keeping them updated. The hook to provision new trees is close to 900 lines and it went through six revisions in its first five days.

Its value outweighs its cost, but it’s worth knowing about the tradeoff.

A book worth your time

Do Androids Dream of Electric Sheep? by Philip K. Dick — five stars.

It’s been on my list since Blade Runner, and I ended up loving it. I generally love the post-apocalypse theme, and the book raises a lot of very interesting philosophical questions.

The one that stayed with me is how we value life. The book sorts sentient beings into neat hierarchies: real animals above electric ones, humans above androids, people of average intelligence above the “specials” like J.R. Isidore. Everyone treats those lines as obvious, but they quickly crumble as soon as you look at them closely.

The test in the book that tells androids from humans also has to be updated for every new model, which was an odd thing to read in a month spent working out when to trust a reviewer’s “nothing here”.

My full review has the rest.

From the industry

A short follow-up first. In September I wrote that AI context gets curated for the individual or for the whole organization, that nothing serves the levels in between, and that products would show up in 2027.

Then Hex raised $100 million on a story that sounded a lot like mine. I can’t really take credit for it, though, since Hex published its version of the problem three weeks before I did. And its context is workspace-wide. That’s the organization end again, for one domain.

Hex does have versioned guides, diffs, and previews on pull requests, which is the code-style handling I said context needed. Who gets what is still the open part, so I’m keeping the prediction.

What stuck more was a line further down. Barry McCardel, who wrote the post, on what agents did to BI: “Everyone can build! But now it’s chaotic, and all the trust is gone.” Part of Hex’s answer, in its 4.0 post, is “a first class tool for reviewing agent-generated analytics code”.

So a company that just raised $100 million to rebuild for agents is talking about review. That lines up well with the numbers in the next paper.

Artificial Intelligence in the Firm is a Harvard working paper by Fiona Chen and James Stratton, updated on 1 November, covering 720,542 workers at 655 firms through Jellyfish’s engineering data.

After firms adopted coding agents, lines of code rose 44% and pull requests 35.7%. Resolved issues rose 13.7%, but the statistical support for that number is weak.

The time from submitting code to getting it approved rose 56.1%. It waits longer for a reviewer, and it takes longer once one gets to it. The authors put it in five words: “Code review becomes the bottleneck.”

The detail I keep coming back to is on page 39. Pull requests started at about 1,200 lines added, and agents didn’t change that in any significant way. So teams are opening a third more pull requests, and they’re already big on average.

We’ve known for decades what both of those do. A 1,200-line change is slow to review well, and when changes arrive faster than reviewers can take them, they queue. Agents sped up the arrivals and left the reviewers where they were. One engineer interviewed for the paper describes a project where a single senior expert could approve changes and only had the bandwidth to do it once a week.

This is why I don’t buy the loudest AI proponents when they say the old rules are finished. Agentic development differs from what came before by degree, and the principles we paid for over decades still hold. Agents push on them harder, so you feel it sooner when you skip one.

79.1% of the firms have adopted AI code review, and the bottleneck is still there, because they largely keep humans in the loop. I’m not sure they should, since I’ve just spent a month taking myself out of my own loop. The paper doesn’t test what happens when work gets cut smaller, so what follows is my experience and not its finding.

If I were running a team, I’d start with what we already know. Plan the work in pieces small enough that one person can hold each in mind: the design decision first, then the change to the domain model, then the database and infrastructure that serve it, and the user interface last, each in a pull request of its own. Then let the agents review each change before the pull request is opened, with a bar that lets the review finish. By the time a human looks, the findings have been fixed or refuted, and the human may not need to look at all.

I’ve handed a few pull requests to a team this way, with me developing the change alongside my agent and them reviewing it. So far, no human has found anything the agent hadn’t already anticipated and fixed.

A few pull requests is not much evidence, and I’m mostly one person working with agents, which isn’t the setup the paper describes. Whether this holds in a team with a real queue and overloaded seniors, I don’t know yet.

It’s worth finding out, though. The paper’s model has output up 57.8% by 2031 if review productivity stays fixed, and 341.4% if it keeps pace with coding. It’s a simulation, but review is the variable it turns on.

That’s what stuck this month. If you’ve gotten review to keep up with your agents, or you think I’ve got this wrong, I’d love to hear it.

The AI at Work course also got a Swedish version and a page for organizations this month, if that’s closer to where you are.