How to Judge a Model on Bugs from Your Own Repo

A benchmark score is a rumor about a model. The repository sitting on a laptop is evidence. That gap explains why so many teams pick a coding model on public leaderboards and then discover, three weeks in, that it cannot fix the one bug class their codebase produces every single sprint.

The useful question is narrower than "which model is best." It is: given this repo, with its particular mess, which model finds and fixes the bugs that actually live here? Answering that requires a small, deliberate experiment, not a leaderboard.

Public benchmarks measure a repo that is not yours

SWE-bench and its variants pull from large open source projects with mature test suites, consistent style, and issues written by people who know how to describe a defect. That is a real skill. It is also a very particular environment, and most private repositories share almost none of its properties.

A typical internal codebase has partial test coverage, a few files nobody wants to touch, and bug reports that read like "the export breaks when the date column is empty." A model trained and tuned against clean issue text can look excellent on the benchmark and then flounder on that sentence.

The minimum viable test set is thirty closed bugs

Pull thirty bugs from the last six months that were fixed and merged. Thirty is enough to see a pattern and small enough to assemble in an afternoon. Fewer than twenty and a single hard case skews the whole comparison.

Each entry needs four things: the commit before the fix, the issue or ticket text, the diff that resolved it, and the test that now passes. The diff matters because it is the ground truth of what "fixed" looked like. Without it, scoring becomes a vibe check.

Skip the bugs that took a senior engineer three days and a whiteboard. Those are not representative, and no current model will solve them. The test set should be boring on purpose.

Reproduce the bug before letting the model touch it

A bug that cannot be reproduced cannot be judged. Before any model enters the picture, confirm that checking out the pre-fix commit and running the relevant test produces a failure. If the test suite does not cover the bug, write the failing test first.

This step eliminates half the candidate bugs immediately. It also quietly builds the evaluation harness, because that same failing test becomes the pass condition for every model run afterward. No failing test, no entry.

Give each model the same narrow context

Context is the variable that ruins most informal comparisons. One model gets the whole repository, another gets the relevant file, a third gets a carefully curated summary. The results then measure the harness, not the model.

Pick one context strategy and hold it fixed. A reasonable default is the failing test, the file it exercises, and the issue text. That is roughly what a mid-level engineer would be handed on day one of a ticket.

Run each model on the same thirty bugs, in the same order, with the same prompt. Temperature at zero if the tool allows it. Log the raw output, not a summary of it.

Score the fix, the test, and the collateral damage

A single pass or fail number hides the thing that matters most in practice. Three separate scores tell a much more useful story.

First, did the failing test pass after the patch? Binary.

Second, did the rest of the suite still pass? A model that fixes the bug by breaking two unrelated tests has not solved anything; it has moved the problem.

Third, does the diff resemble the human fix in scope? A two-line change and a forty-line refactor can both make the test green, and only one of them will survive code review without a fight. This is where the model's taste shows up.

Track all three per bug. The pattern across thirty rows is what the decision rests on.

A model that fixes nothing but explains the bug clearly has value

Some models produce plausible patches that miss the actual defect, yet their written diagnosis points a human directly at the right file and the right line. That is a different product than an autonomous fixer, and it deserves its own column in the results.

If the team's workflow is a developer reading the model's reasoning and then writing the fix themselves, the diagnosis score matters more than the patch score. Judging both with the same rubric produces a misleading ranking.

Watch what happens on the second attempt

Single-shot performance is the easy number. Real usage involves feedback: the tests fail, the developer pastes the error, the model tries again. A model that fails once and succeeds on the second pass is often more useful than one that solves a different set of bugs in one shot.

Give each model up to three attempts per bug, feeding back only the test output. Count how many bugs get solved by attempt three. The gap between attempt one and attempt three is a crude measure of how well the model uses feedback, and it tends to predict real-world satisfaction better than the first-attempt score.

Group the bugs by the shape of the defect

Thirty bugs sorted into categories produce a sharper picture than thirty bugs in a row. Off-by-one errors in loops. Null handling at boundaries. Async race conditions. Type mismatches across a serialization layer. Stale cache reads. Each category stresses a different part of a model's ability.

The categorization usually reveals that one model wins on null handling while another dominates on concurrency, and the aggregate score hides both facts. Since a real repository has a lopsided mix of bug types, the relevant question becomes: which model is strong in the categories this repo produces most?

A payments service that lives and dies on boundary conditions should weight null handling heavily. A frontend with a chatty data layer should weight async and caching bugs. The leaderboard does not know the difference. The thirty bug sample does.

Cost per solved bug beats cost per token

Pricing pages quote per million tokens, which is not a unit anyone makes decisions in. The number that matters is total spend divided by bugs solved, including the tokens burned on failed attempts.

A cheaper model that needs three attempts on every bug and a more expensive one that solves most of them first try can land in the same place on the invoice. Multiply by the team's bug volume and the comparison becomes concrete. A developer-hour saved on a ticket is worth more than the token bill for a week.

Rerun the sample when the repo changes under the model

The result of this experiment has a shelf life. A new framework version, a large refactor, or a shift in coding conventions can change which model performs best. The public leaderboard moves monthly; the private sample should move quarterly.

Keep the thirty bugs as a versioned artifact in the repository, the same way the test suite is versioned. Add new bugs as they are fixed, retire the ones that no longer reproduce. Over a year, the sample becomes a small, honest record of what kinds of defects this codebase actually generates, which is useful well beyond model selection.

The forty-line refactor is the tell

Of all the signals in the sample, the one that best predicts whether a model will earn trust on a team is how often it rewrites code it was not asked to touch. A model that fixes a null check by also reorganizing the surrounding function will pass the test and still get rejected in review, and the rejection costs a developer twenty minutes of reading a diff they did not want.

Count the average diff size per solved bug and compare it against the human fix. The model whose diffs cluster closest to the human line count, while still passing the suite, is usually the one that survives contact with a real pull request queue. That is the number to put on the slide, next to the pass rate, when the decision gets made.