An Agent Reviewed This Pull Request. It Read the Ticket First.
Eleven pull requests shipped to our main branch last month after being reviewed by an agent. That sounds riskier than it is, and the agentic part isn’t even what I find interesting about it.
The thing I keep coming back to is this. Everyone is wiring AI into code review right now, and they’re all doing it with the same handful of models. Those models are turning into a commodity. They get cheaper and better every few months, and the distance between the best one and the next one down keeps shrinking. If we’re all using more or less the same engine, the engine can’t be what sets any of us apart. What sets you apart is what you feed it: the context, the why behind a change. That part is yours, and it’s the part hardly anyone is bothering to build.
Most AI review reads the diff and stops. A diff can tell you what changed. It can’t tell you why it changed, or what it was meant to do, or whether the clean, passing, well-written code in front of you is even the right code to be writing. We don’t just want to build the thing right, we want to build the right thing. I built the reviewer around that gap rather than around the model, and I think that’s the decision that matters.
“What sets you apart is what you feed it.”
Before I get to the reviewer, though, there’s a simpler and slightly embarrassing problem worth naming.
Most review is already fake
Code review on most teams is two-tier, we just don’t say it out loud.
There’s the pull request someone actually reads. They pull it down, sit with it, leave real comments. Then there’s the other kind, the “LGTM” nine seconds after it lands, because it’s a version bump or a config tweak and everyone knows it’s fine. That second kind is the majority, and it isn’t really review. It’s a person spending real, expensive minutes to produce a rubber stamp, over and over, all day, on changes they were never going to block.
We do a lot of that and pretend it counts. It leaves a green checkmark, so it looks like review happened, but nobody actually looked. And it isn’t free, because it quietly eats the attention the hard reviews need. The engineer who approved fifteen trivial PRs before lunch is the same one who then has to think clearly about the migration that could take production down, and by the time that one lands, a good chunk of the clear thinking is already gone.
So I stopped treating the two as the same thing and split them.
The safe stuff gets stamped
The classifier does one job. It works out how much a change could hurt you, and routes it accordingly.
I built two lanes, not three. There’s no “medium risk” tier, on purpose, because medium risk is where accountability quietly disappears. Everyone assumes someone else has the middle, so nobody does. A change is either safe enough to go on its own or it isn’t, and when I can’t say for sure, it isn’t.
Safe is opt-in rather than assumed. A PR only reaches the automatic lane if every file it touches is on a list I’ve explicitly marked as low blast radius, and the default for anything not on that list is human required. On top of that there are a few hard gates. Nothing touching auth, billing or migrations goes through automatically. The diff has to be under a size limit, since a small wrong change is a quick revert while a large one can eat your afternoon. Every check has to be green. And we lean on feature flags.
That last part matters more than it sounds, because it splits merging and releasing into two separate decisions. The code can land on main by itself while the call to actually turn it on for customers stays with a person, which is where I want it.
The whole thing is deliberately boring. The classifier can only take a human off a change when it can strongly justify doing so, and it never adds risk of its own. The most it can do is decline to spend attention on something that never needed it. The upshot is that nobody starts their morning nodding at a change they barely read. That was the point.
The reviewer reads the ticket
“You can’t read intent off a diff.”
This is where the bet actually lives.
Put a better model under the reviewer and it gets a bit sharper at spotting bugs in the lines that changed. That’s genuinely useful, but it’s also the easy part, the part every tool already does, because it’s what the model gives you for nothing. What no model can do on its own is tell you whether the code does the right thing, because the right thing isn’t in the diff. It’s in the ticket someone wrote, the epic it hangs off, the scoping call made three weeks ago about what the feature is even for. None of that ships in the box. You have to go and get it and wire it in.
So that’s what ours does. Before it reads a single line of code it pulls the Jira ticket off the branch, the parent epic, the product and technical scoping attached to them, the repo’s own conventions, and whatever comments are already on the PR. Then it reads the diff with all of that in hand.
What I want it to catch is the thing a diff reader structurally can’t: code that works perfectly and solves the wrong problem. The kind that passes every test, gets approved, and turns into a rewrite two weeks later when someone notices it built something the ticket never asked for. No linter finds that, because there’s nothing wrong with the code. The wrongness is in the intent behind it, and you can’t read intent off a diff.
There’s a second thing the context buys that I didn’t expect. Because the reviewer already has all that background, it also knows what not to say. That context is for the reviewer’s benefit, not the author’s, since the author already knows the project. The reviews that actually land are the ones that read everything and then say only the two or three things that matter, and it’s the context that lets them be that short.
What actually happened
I’d rather tell you what the eleven PRs actually did than what I was hoping they’d do.
On correctness it’s been good. It caught real bugs across those eleven, the kind that slip past CI and a tired skim and come back three weeks later as an incident, and it caught all of them before the code merged. One was on a change to a tool that is itself meant to catch dangerous edits, where it found two holes that would have let bad changes through. That more or less paid for the whole thing on its own.
It also plays well with the other bots, which I didn’t expect to care about as much as I do. We run a couple of other review agents, and on one PR they’d already flagged the same issue. Rather than pile on and repeat it, ours linked to their threads and went looking for what they’d missed. It read the room. That’s a small thing, but it’s roughly the difference between a reviewer people listen to and one they mute.
And then the part I have to be straight about. The catch I actually built it for, the clean code that quietly does the wrong thing, hasn’t really shown up yet. Eleven PRs and no clear intent mismatch. I’m oddly fine with that, because the likeliest explanation is that people aren’t shipping code that misses the point, which is the outcome I wanted anyway. The day it does catch one is the day the context earns its keep rather than the model. Until then we keep feeding our corrections back into it, so it gets a little better on the reviews it has already done.
It’s noisy sometimes, too. A couple of junk comments a run, opinions I overrule, nits nobody needed. I don’t mind. A reviewer that never annoys anyone isn’t looking hard enough, and one that’s too loud is a much easier thing to tune down than a quiet one is to wake up. We’re still tuning it, and we will be for a while. That’s the job.
Where it stands
Auto-merge is live for the low-risk lane. The risky work still goes through a person, and it will for a while yet.
The reason I spent the time on this comes down to one belief. The models will keep getting better, and every team gets that upgrade for free, on the same day, from the same vendors. So the model won’t be what separates the teams that pull ahead. What separates them is the slow, unglamorous work of feeding the machine enough to understand not just the code but the reason the code exists. That’s the part almost nobody is doing, and it’s the part I’m betting on.