Test Intelligence

Running the entire test suite after every edit is slow enough that agents stop doing it, and running nothing is how regressions ship. Test intelligence resolves this by knowing which tests actually exercise the code that changed.

Attribution#

Each test is linked to the set of symbols it exercises using two complementary sources. Static analysis follows imports and call chains from the test body, which is fast and complete but over-approximates. Runtime coverage from a prior execution is precise but only covers paths that were actually taken. The union is used for selection; the intersection is used for confidence.

Selection#

Given a change plan, the selector produces an ordered verification set. Tests that cover the changed symbols directly run first, followed by tests covering the blast radius, followed by contract tests at any crossed interface boundary. The agent gets a signal from the most informative tests within seconds rather than after a full suite.

change      4 symbols across 2 modules
selected    17 of 3,428 tests            est. 24s

tier 1  direct coverage                        9 tests
        session/token.test          · 6 cases
        session/expiry.test         · 3 cases
tier 2  blast radius                           6 tests
        api/middleware.test         · 4 cases
        admin/impersonation.test    · 2 cases
tier 3  contract at crossed boundary           2 tests
        contracts/session-v2.test   · 2 cases

excluded    3,411 tests — no path to changed symbols
gap         2 references with no attributed test
            (see blast radius report)

Failure interpretation#

A failing test is returned to the planner with structure rather than as a wall of output: which assertion failed, which symbol in the change plan is implicated, whether the test was passing before the change, and whether it is known to be flaky. The planner uses this to decide between fixing the code, fixing the test, or escalating — and an agent that cannot distinguish "I broke this" from "this was already broken" will confidently do the wrong one.

• Newly failing — the change is implicated; return to planning with the failure attached.
• Already failing — pre-existing; recorded but not treated as a blocker.
• Flaky — historically non-deterministic; re-run before drawing any conclusion.
• Newly passing — a previously failing test now passes; reported, since an unexpected fix often signals a misunderstanding.