When 90% Coverage Means Nothing: Mutation Testing with Pitest

Line coverage tells you which lines your tests executed. It says nothing about whether a test would fail if those lines were wrong. A test that calls a method and asserts isNotNull() earns the same coverage as one that pins down the result, which is how a module reaches 90% coverage with tests that would pass against broken code.

When a team starts measuring test quality, coverage is usually the first number on the dashboard. It's a fine floor. Mutation testing is what I'd put next to it, because it checks whether the tests notice when behavior changes.

How mutation testing works

Pitest (PIT) makes small changes to your compiled classes, one at a time. Each changed version is a mutant: a < turned into <=, a void method call removed, a returned list swapped for an empty one. For each mutant it runs only the tests covering the mutated line, using per-test coverage collected up front.

• A test fails: the mutant is killed.
• All covering tests pass: it survived. Behavior changed and nobody noticed.
• No test reaches the line: no coverage.

The mutation score is the share of mutants detected, and unlike line coverage it's hard to raise without real assertions.

Setup: Maven and JUnit 5

<plugin>
  <groupId>org.pitest</groupId>
  <artifactId>pitest-maven</artifactId>
  <version>1.16.0</version>
  <dependencies>
    <dependency>
      <groupId>org.pitest</groupId>
      <artifactId>pitest-junit5-plugin</artifactId>
      <version>1.2.1</version>
    </dependency>
  </dependencies>
  <configuration>
    <targetClasses>
      <param>com.example.pricing.*</param>
    </targetClasses>
    <targetTests>
      <param>com.example.pricing.*</param>
    </targetTests>
    <excludedClasses>
      <param>com.example.pricing.config.*</param>
    </excludedClasses>
    <mutators>
      <mutator>DEFAULTS</mutator>
    </mutators>
    <threads>4</threads>
    <mutationThreshold>70</mutationThreshold>
  </configuration>
</plugin>

• JUnit 5 support is a plugin for the plugin: it goes in the plugin's own <dependencies>, not your project's. Version 1.2.1 needs Pitest 1.15.2 or later.
• targetClasses and targetTests are globs. Without targetClasses, every class in the module's build output gets mutated.
• DEFAULTS is the default mutator set, named here to make the choice visible. STRONGER adds operators, and mutants to triage.
• mutationThreshold fails the build when the score drops below that percentage.

Run it once the tests are compiled; the report lands in target/pit-reports/index.html:

mvn test-compile org.pitest:pitest-maven:mutationCoverage

Reading the report

Every package and class gets three numbers: Line Coverage, Mutation Coverage and Test Strength. Test strength is killed mutants divided by mutants that had coverage, so it separates two problems. Low line coverage means untested code. High line coverage with low test strength means tests that run code without checking it.

Open a class and each line lists its mutants with an outcome, such as changed conditional boundary → SURVIVED.

Three survivors and the tests that kill them

A small rule, with tests that reach 100% line coverage:

public long discountFor(Basket basket) {
    if (basket.totalCents() < minBasketCents) {
        return 0;
    }
    long discount = basket.totalCents() * percent / 100;
    events.discountApplied(basket.id(), discount);
    return discount;
}

public List<String> eligibleSkus(Basket basket) {
    return basket.items().stream()
            .filter(item -> !item.excludedFromDiscounts())
            .map(Item::sku)
            .toList();
}

// minBasketCents = 20_000, percent = 10; basket(...) builds basket "b-1"
@Test
void appliesDiscountAboveMinimum() {
    assertThat(policy.discountFor(basket(50_000))).isEqualTo(5_000);
}

@Test
void noDiscountBelowMinimum() {
    assertThat(policy.discountFor(basket(10_000))).isZero();
}

@Test
void listsEligibleItems() {
    assertThat(policy.eligibleSkus(basketWithItems())).isNotNull();
}

Every line runs. Pitest still finds three gaps.

1. The untested boundary

changed conditional boundary: < became <=. Neither 50,000 nor 10,000 sits on the boundary, so both tests pass against the mutant. The fix is a test exactly at the threshold, where rules are most often misread:

@Test
void appliesDiscountAtExactlyTheMinimum() {
    assertThat(policy.discountFor(basket(20_000))).isEqualTo(2_000);
}

2. The side effect nobody checked

removed call to .../DiscountEvents::discountApplied: the tests only look at the return value, so dropping the event goes unnoticed. If the event is part of the contract, assert it:

@Test
void publishesTheAppliedDiscount() {
    policy.discountFor(basket(50_000));
    verify(events).discountApplied("b-1", 5_000);
}

3. The assertion that asserts nothing

replaced return value with Collections.emptyList, plus the filter lambda's result replaced with true and with false. All of them survive, because any list is not null. One assertion on the content kills the whole group:

@Test
void listsOnlyItemsThatAllowDiscounts() {
    var basket = basketWith(item("sku-1", false), item("sku-2", true));
    assertThat(policy.eligibleSkus(basket)).containsExactly("sku-1");
}

Keeping it fast

• Scope it. Aim targetClasses at the business rules; exclude configuration, DTOs and generated code. Pitest works per Maven module, so run it where the rules live: mvn -pl pricing-core test-compile org.pitest:pitest-maven:mutationCoverage.
• Use threads. The default is one.
• Keep history. Incremental analysis reuses results for mutants whose code and tests haven't changed. Locally, -DwithHistory keeps the file in the temp directory; in CI, point historyInputFile and historyOutputFile at one cached path. Pitest calls the feature experimental (it assumes dependency changes rarely flip a result), so do a full run now and then.
• Skip it on everyday builds. Nightly, or on pull requests that touch the domain module.

What not to chase

• 100%. Some mutants are equivalent: they don't change behavior, so no test can kill them. Set the threshold near today's score and ratchet it up.
• Mutants in code nobody should test. Exclude that code. Lines that call common logging frameworks are skipped by default.
• Interaction tests for their own sake. Killing every removed-call mutant with verify(...) yields tests that mirror the implementation; verify only the calls that are part of the contract, such as how many requests a client sends when it retries.

Takeaways

• Read test strength next to line coverage. High coverage with low strength means weak assertions.
• Boundaries, unchecked side effects and isNotNull()-style assertions are the usual survivors.
• Scope the run, use threads and history, and treat the threshold as a ratchet.