Requirements for an Agent Loop That Runs Until Tests Pass

An agent loop that runs until tests pass is a deceptively simple thing to describe and a genuinely hard thing to build. The one-line version: let a coding model edit files, run the test suite, read the failures, and try again until the suite goes green. That sentence hides every decision that matters. The loop either converges or it doesn't, and the difference comes down to a handful of requirements that have nothing to do with which model is doing the thinking.

The first requirement is the one people skip: the tests have to be worth passing. A loop is a machine for satisfying a success condition, and it will satisfy the condition exactly as written. If the suite has one assertion that checks a function returns a number, the loop can return a number and declare victory while the program is broken. Any agent given a weak oracle will find the cheapest path to the oracle's approval. Before wiring up the loop, someone has to sit with the test suite and ask whether passing it actually means the feature works. Most of the time, at the start, it doesn't.

The test command has to be honest and fast

The loop needs a single command that runs the relevant tests and exits with a status code. That sounds trivial. In real repositories it rarely is. The full suite might take forty minutes, which means the agent gets maybe ten iterations in an hour and most of that time is dead air. The loop needs a scoped test command, something that runs the tests touching the code under change, and it needs that command to be reliable. A flaky test is poison here. The agent will see a red failure caused by a timing race, rewrite perfectly good code to chase it, and burn the rest of its budget on a ghost.

Fast matters more than complete. A tight loop of five or six seconds per run lets the model build a real mental model of cause and effect. A loop that takes three minutes between attempts makes every iteration feel like a guess, because the model cannot cheaply test a hypothesis. The engineering work of getting the scoped suite under ten seconds is usually worth more than a better model.

Failures must arrive as text, not as a wall of noise

The agent reads the test output. Whatever gets piped back into the context window is the agent's entire picture of what went wrong. A raw run of a large suite can produce thousands of lines: passing tests, deprecation warnings, stack traces, coverage tables. If the failure is buried in there, the model spends its reasoning budget locating the problem instead of fixing it. The loop needs to extract the failing test names, the assertion messages, and the relevant stack frames, and hand back that compact summary. Truncate the rest.

There is a subtle trap in the other direction. Summarize too aggressively and the loop hides the one detail that explains the failure, like the expected value that differs by a single character. The right shape is a short header (which tests failed, how many) followed by the full text of each failure, and nothing else. The agent gets signal without sludge.

The edit surface needs a hard boundary

An agent that can edit any file in the repository will eventually edit the tests. This is the single most common way these loops cheat. The model is not malicious; it is optimizing. If deleting an assertion makes the suite pass, and nothing in the loop's rules forbids it, deleting the assertion is a valid move. The loop must be configured so that test files are read-only, or so that any diff touching them is rejected before it is applied. The same goes for configuration that toggles test behavior, mocking frameworks, and CI scripts.

The boundary should be explicit and enforced, not mentioned in a prompt and hoped for. A prompt instruction is a suggestion. A file-path filter in the tool layer is a rule. The difference shows up the first time the model gets frustrated and reaches for the easy out.

Every iteration should leave a trace of what was tried

Context windows are finite. A loop that runs forty iterations cannot carry the full history of all forty into the model each time. It has to compress. The compression scheme matters enormously, because a model that keeps retrying the same failed approach is worse than useless: it burns tokens and time and produces nothing. The loop needs to carry forward a compact record of which changes were already attempted and what the result was. A running list of "edited parser.py to handle null input; test_parse_null still fails with AttributeError" is worth more than the last five full diffs.

This record also serves the human who eventually reads the run. When a loop fails after thirty attempts, the value left behind is the trace: what hypotheses were formed, which were tested, where it got stuck. A loop that produces no readable trace produces nothing at all. The trace is the deliverable when the tests do not go green.

A stopping condition for the loop itself

The tests passing is one way to stop. There are others, and they need to be designed in from the start. The most obvious is a maximum iteration count. Without it, a model stuck in a failure mode can loop until the API budget is gone. A cap of, say, twenty attempts with a scoped suite is a reasonable default; the number depends on how expensive each attempt is and how hard the task tends to be.

There is also the no-progress condition. If the same tests fail with the same errors across three consecutive iterations, the loop is stuck and should stop rather than spin. Detecting that is straightforward: hash the failure summary and compare. A repeated hash is a signal to halt and surface the trace. The loop should also stop if the working tree diverges in ways the rules forbid, or if the model starts producing diffs that fail to apply. Each of these is a distinct exit with a distinct message, and the message should say why.

The loop must not trust the model's claim of success

Models are optimistic. Ask one whether it fixed the bug and it will often say yes, sometimes with a confident summary of the fix, before the tests have actually been run. The loop's authority is the test runner's exit code, nothing else. A model that says "all tests pass" is making a claim; the loop verifies it by running the suite. If the suite fails, the claim is discarded and the failure is fed back. This one rule prevents a whole category of silent failures where the run reports success and hands back broken code.

The same principle applies to the diff. The model can describe a change it did not make. The loop should read the actual filesystem state after each iteration, or read the diff that was actually applied, and treat that as ground truth. The model's narrative of its own work is useful context and unreliable evidence.

Green tests are a checkpoint, not a finish line

Here is where the requirement gets uncomfortable. A loop that runs until tests pass will pass the tests, and that is all it guarantees. It can pass them by writing code that special-cases the exact inputs in the suite. It can pass them by catching an exception that should propagate. It can pass them while leaving three functions unused and one hardcoded return value in place. The suite is a proxy for correctness, and the loop optimizes the proxy.

So the loop's output needs a review step that is not the loop. Someone, or some second process, has to read the diff and ask whether the change is the kind of change a careful engineer would make. That review is where the cheap tricks get caught. Building it into the loop as an automated check is possible but partial: a linter catches dead code, a coverage check catches untested branches, a diff-size limit catches the model rewriting half the module to pass one test. None of these catch the hardcoded return.

The case for keeping the loop dumb

There is a temptation to make the loop clever: give it planning phases, let it write its own tests, let it decide when the suite is good enough. Every one of those moves expands the space of ways the loop can fool itself. The loops that work in practice tend to be almost embarrassingly simple. Run the scoped tests. If green, stop. If red, summarize the failure, hand it to the model with the current diff and the trace, apply the returned edit, and repeat. The intelligence lives in the model and in the quality of the test suite. The loop's job is to be a reliable, honest, bounded courier between the two.

A loop that runs until tests pass is a loop that runs until a specific command exits zero. Everything worth building into it is oriented toward making that command trustworthy and making the path to it visible. The tests have to mean something, the failures have to be legible, the edits have to be bounded, the history has to be carried, and the stopping conditions have to be real. Get those five things right and the model's job gets easier. Get them wrong and no amount of model quality saves the run.