What Humans Still Check After an AI Review Bot

The pull request had been open for nine minutes when the review bot posted its comment: a tidy wall of green checkmarks, two amber warnings about cyclomatic complexity, and a cheerful note that the diff followed the team's formatting conventions. The author, a backend engineer named Priya, read it, nodded, and then did something the bot could not see. She opened the changed files in her editor and read all fourteen of them line by line, out loud, muttering under her breath.

That habit is the entire subject of this piece. Automated review has gotten genuinely good at a narrow band of the job, and the band is narrower than the enthusiasm suggests. What remains, after the linter has spoken and the static analyzer has signed off, is the work that humans still do by hand, and it is the work that decides whether software survives contact with production.

The bot reads the diff; the human reads the blast radius

A review bot lives inside the patch. It sees the lines that changed, the functions those lines touch, the tests in the same directory. It does not see that the function being modified is called from a cron job in a separate repository by a colleague who left the company two years ago and whose documentation is a single line of comment.

Human reviewers carry that map in their heads, imperfectly but usefully. When a bot flags a null check, it is flagging a local fact. When a senior engineer flags the same line, she is often flagging a memory: the incident last spring, the customer escalation, the three-hour call where this exact code path caused a cascade. The bot has no incident history. It has a ruleset.

This is why the most valuable review comments tend to be short and slightly cryptic to outsiders. "This breaks the nightly sync." "We tried this in 2021." "Ask Marcus before shipping." None of these could be generated by a tool, because none of them are contained in the code.

Naming, and the small humiliations of bad names

A bot will happily approve a variable called data, a function called handleStuff, and a class called Manager. It has no opinion on whether these names will make a new hire cry in six months. Style guides cover casing and length; they do not cover the quieter question of whether a name tells the truth about what the thing does.

Human reviewers still argue about naming, and they are right to. A name that lies is a bug that ships silently and compounds. The function called validateEmail that also normalizes the string and writes an audit log. The config key called timeout that is measured in minutes, not seconds, because someone made that decision in 2018 and nobody has changed it since. The bot sees a variable. The human sees a future support ticket.

Priya, in her pass, once left a single comment on a pull request: "This returns null on success and throws on failure. Please rename it before the next person assumes the opposite." The bot had approved the function twelve minutes earlier. The name was tryCreateUser.

The comment that explains why, not what

Automated tools care about presence, not quality. A docstring checker will confirm that a function has three lines of documentation. It will not notice that those three lines restate the function signature in worse English and explain nothing.

What humans still check, painfully and slowly, is whether a comment explains a decision. Why is this retry limited to three attempts? Why does this query skip the index on purpose? Why does the migration delete the old column instead of leaving it? These are the comments that save a future engineer from undoing a fix because nobody wrote down the reason.

The bot cannot distinguish "why" from "what" because both are just text. A reviewer who has been in the codebase long enough to remember the original bug reads the comment and knows immediately: this is a scar, not a decoration. Leave it alone.

Error messages read as if a tired person will see them

Every error path in a codebase is read, eventually, by someone who is already having a bad day. A bot will check that error handling exists. It will not check whether the message inside it is useful, honest, or kind.

"Something went wrong" is technically an error message. So is "Failed to process request." Neither tells the operator which of eleven services failed, what state the data is in, or what to try next. Humans reviewing error paths tend to rewrite these messages into something an on-call engineer can act on at 3 a.m. without opening the source.

There is also the question of what an error message leaks. A bot has no sense of whether a string exposes an internal table name, a stack trace with credentials, or a customer's email address. A human reviewer, if she is paying attention, catches these before the message reaches a log aggregator where it will live for ninety days.

The test that proves the bug is actually fixed

Coverage tools measure lines executed, not lines that matter. A function can have 100 percent coverage and still be entirely wrong, because every test exercises the happy path with the same fixture. Bots reward this. They tick the coverage bar green and move on.

A human reviewer, staring at a bug fix, asks a specific question: does this test fail if I remove the fix? If not, the test is decoration. This check takes thirty seconds and catches an astonishing number of regressions before they ship, and no static analyzer will ever perform it, because it requires imagining a slightly different version of the code and running it in the mind.

Priya's team has a rule, unwritten, that every bug fix comes with a test that was written first and seen to fail. The bot does not enforce this. It cannot. The reviewer does, by asking, every time, and refusing to approve until the answer is yes.

The reviewer's gut, which is really pattern recognition

There is a moment in review when nothing is technically wrong but something feels off. The diff is too large for its stated purpose. The abstraction was introduced three files before its first real use. The refactor touches a module that has been stable for years and has no tests around it. None of these are lint errors. All of them are worth a question.

That feeling is not mysticism. It is a decade of seeing what precedes incidents, compressed into a reflex. The bot has no reflex. It has rules written by people who had the reflex, months earlier, and encoded a small piece of it. What remains in the human is the part that has not been encoded yet, and probably cannot be.

This is also why automated review has not eliminated the senior reviewer, despite a decade of predictions. The tools have absorbed the easy checks: formatting, syntax, obvious security patterns, dependency vulnerabilities. What is left is the judgment layer, and the judgment layer is where the actual risk lives.

What the bot does well, and why that matters here

The review bot is not the enemy of careful review. It is the reason careful review is possible. Ten years ago, a human reviewer spent the first twenty minutes of every pull request on formatting, indentation, and whether the imports were alphabetical. Those comments were cheap, boring, and repetitive, and they crowded out the comments that mattered.

Now the bot handles the cheap layer, and the human gets to spend her attention on the expensive layer: blast radius, naming, why, error messages, test validity, and the quiet suspicion that something is wrong. That is a genuine improvement in the economics of code review. It is also why the human layer has become more concentrated and more valuable, not less.

The pull request that passed every check and still shipped a bug

Three months after Priya read those fourteen files, a different team shipped a change that the bot approved in four minutes. Every test passed. Coverage went up. The linter was silent. The change went to production on a Thursday afternoon.

The bug was in the error path. The new code caught an exception and logged it, then returned a success response, because the developer had copied the pattern from a handler where that was correct and had not noticed that this handler's caller relied on the exception to trigger a retry. The bot saw valid syntax and a test that passed, because the test only exercised the success path.

A human reviewer who had asked the question, the one about whether this test would fail if the fix were removed, would have caught it in under a minute. Nobody asked. The review had two approvals, both from people who had read the bot's summary and skimmed the diff.

That is the shape of the failure mode that automated review has quietly enabled. The bot is so competent at the cheap layer that it produces a feeling of completeness, and the feeling of completeness is what lets the expensive layer go unexamined. The checkmarks are not lying. They are just answering a narrower question than the one that matters, and the humans who were supposed to ask the wider question have learned, slowly, to trust the green.

Priya still reads every line out loud. Her team thinks it is eccentric. It has also caught, over the past two years, at least four changes that the bot had already approved and that would have caused incidents within the week. The habit costs her about twenty minutes per pull request. It is the cheapest insurance her team buys.