A test written by the same agent that wrote the code confirms what the code does, not what it should do. Why a green, agent-authored suite says little about intent, and how to audit it.
The pull request touches fourteen files. Nine of them are new tests. CI is green, the coverage report went up, and the description says what every reviewer wants to read: “Added tests for all new behaviour.” The reviewer skims the diff, sees assertions next to every function, and approves. Tests exist, tests pass, nobody has a reason to block it.
Three weeks later a bug reaches production. The faulty path was covered. The tests executed it, line by line, and never checked what came out. One of them asserted that a helper had been called. Another asserted that the result was defined. The suite was green before the bug shipped and green after it, because it was never capable of being anything else.
This is the failure mode of agent-authored test suites, and it is easy to miss because everything on the surface looks like diligence. This article states the position plainly: A test written by the same agent that wrote the code is a change detector, not a verification.
Why generated tests look like progress
The case for them is real and worth stating fairly. Generated tests are cheap. An agent produces in a minute what a person would put off for a sprint. They raise coverage numbers on code that previously had none. They catch regressions during refactors, which is genuine value: if behaviour shifts unexpectedly, a pinned test turns red. And an imperfect test is better than no test, because it at least forces the code to be runnable and importable under a harness.
None of that is wrong. The problem is what happens to the signal. Before agents, a test file in a pull request was weak evidence that somebody had thought about expected behaviour, because writing a test took effort that was spent on thinking. That inference no longer holds. A test file now proves only that an agent was asked for one. Reviewers still apply the old heuristic, and the old heuristic now over-reports.
What the data says
The first large measurement of this was published in June 2026. Banik, Chowdhury and Shamim analysed 86,156 test-file patches from 33,596 agent-authored pull requests across 2,807 GitHub repositories, produced by Codex, Copilot, Devin, Cursor and Claude Code. In All Smoke, No Alarm: Oracle Signals in Agent-Authored Test Code, they report that 80.2% of those patches contain weak or no explicit oracle signals. Strong oracles did correlate with a higher chance of merge (OR 1.28), so reviewers do notice the difference when it is visible. The authors’ broader conclusion is the uncomfortable one: test file counts overestimate verification strength.
The industry survey data points the same way. The SmartBear State of Software Quality and Testing 2026 report, as covered by SD Times on 2026-09-30, found that 92% of respondents say AI is the primary tester of its own code. 46% of teams had shipped AI-generated code that later failed. And 81% of leaders believe AI reliably catches its own errors, against 64% of practitioners, so the people closest to the work are already less convinced. The report’s own description of the setup is a system that generates code and validates it, which it calls a “testing black box”.
Neither source says agent-written tests are useless. Both say the same thing from different angles: the count of tests is not the quantity that matters, and confidence is running ahead of evidence.
The oracle problem, now at scale
Every test needs an oracle: a source of truth for what the correct result is. In a human-written test, the oracle is usually external to the code. It comes from a requirement, a worked example, a conversation with the product owner, a bug report. The tester asks “what should happen?” and writes down the answer before checking whether the code agrees.
An agent that has just written the implementation has a much more convenient oracle available: the implementation. It reads the function, sees that a fee is computed with a rounding step, and writes a test expecting the rounded value. If the rounding was the bug, the test now enforces the bug. The expected value was not derived from intent. It was copied from behaviour.
This is why the blind spots are shared. If the agent misunderstood the requirement while writing the code, it misunderstood it the same way while writing the test. Both artifacts descend from one interpretation. A misreading that produces a wrong function also produces a wrong expectation that is consistent with it, and consistency is exactly what a passing test reports.
Figure 01Two verification loops. In the first, the expectations are derived from the code, so the test can only agree with it. In the second, expectations come from the requirement and the code can contradict them.
Independent verification is not more expensive in raw effort. It is more expensive in a different currency: somebody has to hold an opinion about correct behaviour that does not come from the implementation. That is the work agents cannot shortcut for themselves, and it is the work a passing suite quietly stops asking for.
An assertion audit for agent-authored tests
The fix is not to distrust every generated test. It is to audit them against a short list, on every agent pull request, before the test count is allowed to mean anything.
Does every test assert an outcome a user or caller would notice? Tests that only check that
nothing threw, that a value is defined, or that a collaborator was called are smoke. They are
fine as smoke. They must not be counted as verification.
Where did the expected value come from? A requirement, a worked example or a ticket is an
oracle. A number that matches the implementation’s current output is a recording. If nobody can
point to the source of the expectation, treat it as unverified.
Would the test fail if the code were wrong? Break the code on purpose: flip a comparison,
move a boundary by one, delete a branch. Run the suite. Every mutation that survives marks a
line the tests execute but do not check.
Are snapshots and golden files reviewed as specifications? An agent can regenerate a
snapshot in one command. If the diff was accepted without reading it, the snapshot records the
bug rather than preventing it.
Are mocks hiding the behaviour under test? When the dependency that carries the logic is
stubbed to return the value the test then asserts, the test verifies the stub. Mock the
boundary, not the thing being proved.
Who wrote the expectations? The strongest single factor. A human, a separate specification,
or at least a different prompt with a different context, so that a misreading in the code is not
automatically repeated in the test.
The difference shows up quickly in code. Here is the same discount rule, tested twice.
discount.test.tstypescript
// Generated from the implementation: executes the path, verifies nothing.test('applies discount', () => {const result = applyDiscount(cart, 'SAVE10');expect(result).toBeDefined();expect(discountService.validate).toHaveBeenCalled();});// Derived from the requirement: "SAVE10 takes 10% off the subtotal, not shipping."test('SAVE10 discounts the subtotal only', () => {const cart = { subtotal: 200_00, shipping: 9_90 }; // centsexpect(applyDiscount(cart, 'SAVE10').total).toBe(189_90);});
The first test passes if the discount is 10%, 100%, or zero. The second fails on the day someone applies the percentage to shipping too, because the number 189.90 came from arithmetic on the requirement, not from running the function and pasting its output.
Where generated tests are the right tool
None of this argues against using agents for tests. It argues for being honest about what each kind of test is.
Characterization tests are the clearest legitimate case. Before refactoring legacy code that nobody fully understands, pinning its current behaviour is exactly right, bugs included. The job is to detect change, and an agent can generate that net quickly. The only requirement is that the suite is labelled as a change detector, so nobody later reads it as a statement of correctness. Scaffolding is similar: fixtures, page objects, harness wiring and test data builders are mechanical work where there is no oracle to get wrong. Broad smoke coverage, such as “every route renders and nothing throws”, is also cheap evidence of a specific, limited claim.
Letting agents write the tests
Advantages
Near-zero cost per test, so legacy code finally gets a safety net
Fast scaffolding of harnesses, fixtures and smoke coverage
Regression detection during refactors works regardless of who wrote the test
Frees people to spend their time on the expectations that matter
Costs
Expected values drift toward whatever the implementation currently does
Coverage and test counts rise while verification strength does not
Blind spots in the code are repeated in the tests, so errors agree with each other
Independent expectations still cost human time that no tool removes
Those costs are not an argument to stop. They are the list of things the audit exists to catch.
Verification is the scarce skill
Writing the script used to be the expensive part of test automation, so writing scripts was the job. That cost has collapsed. What did not get cheaper is deciding what correct means, and that decision was always the real contribution. The work moves from authoring tests to owning expectations: capturing intent in a form that survives being handed to a machine, then checking that the machine’s output can actually be contradicted by it.
This is also where measurement has to change. The articles on measuring whether automation helps and identifying high-value automation cover how to judge automation by its value rather than its volume. With agents producing tests at no marginal cost, a raw count says even less than it used to. The question to ask of any suite is whether it fails when the code is wrong.
Decision
The call
Let agents write tests, but never let the same context write the code and the expectations. Expectations come from the requirement, from a person, or from a separately prompted pass that has not seen the implementation. Measure suites by whether they fail when the code is deliberately broken, not by how many tests they contain.
Key takeaways
Key takeaways
A test written by the same agent that wrote the code is a change detector: it confirms what the code does, not what it should do.
In a study of 86,156 agent-authored test-file patches, 80.2% contained weak or no explicit oracle signals, so test file counts overestimate verification strength.
Reviewing a green suite tells you the code and tests agree. When both share one context, agreement is the default, not evidence.
Audit agent tests by asserting outcomes, tracing the source of expected values, breaking the code on purpose, and separating who writes expectations from who writes code.
Generated tests are the right tool for characterization, scaffolding and broad smoke, as long as they are labelled as change detectors.
Measure a suite by whether it fails when the code is wrong, not by how many tests it has.
Automation is not a cheaper way to do manual testing. It is a different instrument entirely. Teams that confuse the two build suites that cost more than the confidence they buy.
"Is this suite actually helping?" is a fair question that most teams cannot answer. A field note on instrumenting one suite for four weeks, and the metrics that changed the conversation.