When Generated Tests Are Worth Keeping
A generated test suite can look impressive in a pull request. Fifty files, a thousand assertions, a coverage number that climbs from 61 to 88 percent. Six months later the same suite is a millstone: slow, brittle, full of sleeps and snapshots that fail whenever a designer nudges a margin. The question of which generated tests deserve to survive that six months has a real answer, and it is narrower than the coverage dashboard suggests.
The case for keeping a generated test rests on one property: it must catch a failure that a human would otherwise have shipped. Everything else, the elegance, the coverage delta, the satisfying green checkmark, is secondary. A test that has never failed for a real reason and cannot be traced to a real risk is not an asset. It is future maintenance debt with a nice commit message.
That sounds harsh. It is meant to. The economics of test code are different from the economics of product code. Product code earns money when it works. Test code costs money every time it runs, every time it breaks, and every time someone has to decide whether the failure is real. A generated test that fails for the wrong reason twice will be deleted or skipped, and the habit of skipping spreads.
The failure that would have reached production
The single best filter is a concrete story. When a generated test fails, ask what the shipped bug would have looked like to a customer. If the answer is specific, such as a checkout total off by a cent on a discounted order, a date field parsed as local time on a server in another region, or a pagination cursor skipping the last item on a page, the test has earned its place. If the answer is vague, the test is probably noise.
This filter is brutal in a useful way. It forces the person reviewing the generated suite to imagine real harm. A test for an internal helper that formats a log line rarely survives that question. A test for the boundary of a retry policy that could hammer a payment provider during an outage usually does.
The story also tells the team what to do when the test eventually fails. A failure with a known harm attached gets investigated within the hour. A failure with no story attached gets muted within the week.
Property tests that fail with a minimal counterexample
Generated tests split into two families, and they behave nothing alike. The first is the property test, where a tool generates many inputs and checks that a rule holds: sorting is idempotent, encoding then decoding returns the original value, the total of a split ledger equals the total before the split.
Property tests are worth keeping when the failure report is genuinely diagnostic. A good property test shrinks a failing input to the smallest case that still breaks the rule. That shrunk case is a gift. It is a sentence a person can read and understand at 2 a.m. The test found the bug, and it explained the bug.
Property tests are worth discarding when the shrinking produces something absurd. A rule about money that fails on the number 1.7976931348623157e308, or a validation rule that fails on a string of 400 null bytes, is technically correct and practically useless. The generator found an edge case, but the edge case cannot happen in the system. The test is now a permanent tax on every refactor.
A middle path exists. Constrain the generator. Tell it the domain: amounts between zero and one million, strings from a realistic alphabet, dates in the next two years. The property test then explores the space that matters and stops wandering into the space that cannot occur. That is a generated test worth keeping, and it took a human ten minutes to aim it.
Snapshot tests and the diff nobody reads
Snapshot tests are the most common generated test and the most frequently abandoned. A tool renders a component or serializes a response, stores the output, and fails whenever the output changes. The first run is free. Every subsequent run charges interest.
A snapshot earns its keep in exactly one situation: the output is a contract with something outside the codebase. An API response consumed by a mobile app, a configuration file consumed by an external system, a generated schema that a partner depends on. In those cases a diff is a warning that a real integration might break, and someone should look.
A snapshot is dead weight when the output is internal. A snapshot of a component's rendered markup will fail when a class name changes, when a wrapper element is added, when a test ID is renamed. None of those failures represent a bug. They represent work. The team learns to run the update command without reading the diff, and at that moment the test has become a ritual.
The ritual is worse than no test. It trains people to accept changes they did not review. The one time the snapshot catches a genuine regression, it will be buried in a diff of forty other updates and approved along with them.
Mutation scores as a decision tool, not a grade
Mutation testing generates small changes to the production code, flips a comparison, removes a condition, and checks whether the test suite notices. A surviving mutant is a place where the code could break and the tests would not care. The score is often reported as a single percentage, and that percentage is close to meaningless on its own.
The useful signal is not the score. It is the list of mutants that survived in code that matters. If a mutant survives in a module that computes tax, that is a finding. If a mutant survives in an equality check on a value the system never distinguishes, that is noise, and the team can mark it as equivalent and move on.
Generated tests are good at killing mutants cheaply. That is their real strength. A machine can produce a hundred small assertions across a hundred branches in minutes. A person writing the same tests by hand would spend a day and get bored halfway through. Used this way, generation is a searchlight that points at the dark corners of the suite.
Used as a grade, generation produces a number that teams optimize instead of a behavior they trust. The score goes up, the suite gets slower, and nobody can say which test would catch the next incident.
The maintenance bill arrives on a schedule
Generated tests fail differently from hand-written ones. A hand-written test usually fails because the code changed and the test was correct. A generated test often fails because the generator's assumptions drifted from the code's reality. The test was correct about a world that no longer exists.
This drift is predictable. It shows up when a dependency updates, when a serializer changes its output format, when a language runtime changes how it sorts strings. The generated test was never wrong. It was just pinned to a detail nobody agreed to maintain.
A team that keeps generated tests needs a policy for the day they fail. The policy should be short. If the failure maps to a real risk, fix the code or the test. If the failure maps to a changed dependency output, update the expectation and note why. If the failure maps to nothing, delete the test. The deletion is the important part. A suite that only grows is a suite that eventually stops running.
What a generated test must carry to survive review
A generated test that makes it through review tends to carry three things. The first is a name that states the rule being checked, not the function being called. A reader should know what broke before opening the file. "Rejects a coupon applied after the order is closed" is a name that carries its weight. "Test coupon service 4" is a name that will be skipped.
The second is a clear owner. Someone, by name or by team, is responsible for the failure when it fires. A generated test with no owner is an orphan, and orphans get muted. The third is a reason to exist that is written down, even if only in the commit message: the incident it would have caught, the contract it protects, the mutant it kills.
Those three things take about a minute each to add. That minute is the difference between a test that gets maintained and a test that gets deleted by whoever is on call next quarter.
The tests a team would write again from scratch
A useful exercise, run once a year, is to imagine the suite was deleted overnight and ask which tests would be rewritten first. The list is usually short and specific: the tests around money, around authentication, around the one integration that has broken twice. That list is the set of generated tests worth keeping.
Everything else can be regenerated, or dropped, or replaced by a single assertion in a place that matters. The coverage number will fall. The suite will run faster. The failures that remain will mean something, and the team will read them.
Keeping a generated test is a commitment to maintain it, not a gift from the generator. The generator produced the candidate. The team decides which ones get a lease. Most should not get one, and the ones that do should be able to say, in one sentence, what they are protecting.
