Mutation Testing in Java with PIT

When evaluating the quality of your tests, most developers turn to code coverage tools. These tools provide a percentage indicating how much of your code is executed by your test suite. At first glance, this seems like the ultimate metric: higher coverage implies better-tested code, right?

Not quite!!!

Code coverage tells only part of the story. A test suite can hit 100% coverage and still miss critical bugs if it isn't actually validating the correctness of the code that it executes. This is where mutation testing comes in. It gives a deeper and more meaningful look at the effectiveness of your test suite by answering: Do my tests catch actual defects, or do they just run the code?

What is Mutation Testing?

Mutation testing is a powerful technique designed to evaluate the effectiveness of your test suite. It works by deliberately introducing small, intentional changes, or "mutations," into your source code to simulate common programming errors. These mutations might include:

• Changing a comparison operator (e.g., < to <=).

• Replacing a boolean value (e.g., true to false).

• Modifying arithmetic operations (e.g., + to -).

Once mutated, your test suite is executed against this mutated code. The goal is simple: determine whether your tests can detect these defects. If the tests fail, we say the mutant is killed, indicating that the tests are robust enough to detect this defect. On the other hand if the tests pass despite running on the mutant, we say the mutant survives. A surviving mutant is an indication that there is a gap in the test coverage, the tests are not exhaustive enough to detect this subtle change of logic.

Now, consider a real-world scenario: during a routine refactoring, a developer inadvertently changes a < operator to <=. If your test suite cannot detect this defect, the bug might go unnoticed until it reaches production—where the consequences could range from subtle miscalculations to catastrophic failures. This highlights why robust tests are critical when refactoring.

The traditional metric of 100% code coverage, while helpful, doesn't guarantee your tests are effective. Coverage tools only measure whether your code is executed by tests, not whether your tests validate the correctness of the executed code. Mutation testing addresses this limitation by challenging your tests to prove they can identify meaningful changes in behavior. It's not enough to merely cover your code—your tests must challenge it.

Example: Using PIT for Mutation Testing

Set up

Add the PIT Maven plugin in your pom.xml:

<plugin>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>LATEST</version>
</plugin>

Let’s add some code

I created a class InsurancePremiumCalculator.java that calculates the amount of money the premium Insured needs to pay based on his age and type of vehicule.

public class InsurancePremiumCalculator {
    private static final double BASE_PREMIUM = 500.0;
    private static final int MIN_AGE = 16;
    private static final int MAX_AGE = 120;

    public double calculatePremium(int age, String vehicleType) {
        if (age < MIN_AGE || age > MAX_AGE) {
            throw new IllegalArgumentException("Invalid age");
        }

        double premium = BASE_PREMIUM;

        // Age factor
        if (age < 25) {
            premium *= 1.5;
        }

        // Vehicle type factor
        premium *= switch (vehicleType.toLowerCase()) {
            case "sports" -> 1.7;
            case "suv" -> 1.3;
            case "sedan" -> 1.0;
            default -> throw new IllegalArgumentException("Invalid vehicle type");
        };

        return Math.round(premium * 100.0) / 100.0;
    }
}

Now for the test class, InsurancePremiumCalculatorTest.java :

class InsurancePremiumCalculatorTest {

    private InsurancePremiumCalculator calculator;

    @BeforeEach
    void setUp() {
        calculator = new InsurancePremiumCalculator();
    }

    @Test
    void testBasicPremium() {
        assertEquals(500.0, calculator.calculatePremium(30, "sedan"));
    }

    @Test
    void testYoungDriverPremium() {
        assertEquals(750.0, calculator.calculatePremium(20, "sedan"));
    }

    @Test
    void testSportsCarPremium() {
        assertEquals(850.0, calculator.calculatePremium(30, "sports"));
    }

    @Test
    void testCombinedFactors() {
        assertEquals(1275.0, calculator.calculatePremium(22, "sports"));
    }

    @Test
    void testSUV() {
        assertEquals(975.0, calculator.calculatePremium(22, "suv"));
    }

    @Test
    void testInvalidAge() {
        assertThrows(IllegalArgumentException.class, () ->
                calculator.calculatePremium(15, "sedan"));
        assertThrows(IllegalArgumentException.class, () ->
                calculator.calculatePremium(121, "sedan"));
    }

    @Test
    void testInvalidVehicleType() {
        assertThrows(IllegalArgumentException.class, () ->
                calculator.calculatePremium(30, "invalid"));
    }

}

Looks pretty exhaustive right ? If you shouldn’t take my word for it, let’s take a look at the tests coverage.

I have a 100% test coverage, every single line in this method is executed in one of my tests suite. Now let’s generate the mutation tests report and take a look at our PIT report.
We generate PIT report using the following command (assuming you have maven installed locally) :

mvn test-compile org.pitest:pitest-maven:mutationCoverage

You should be able to find the PIT report under
target/pit-reports/yourpackage/index.html
Open it in your favorite browser and you should see something like this:

If you click on the class name, you should find a detailed report on the class code mutations.

It seems like our tests suffers from a boundary mutation gap, which means (check PIT docs for more information on mutations types) changing the < or > with a <= and >= was not detected by our tests. In other words, we have no tests that detects whetherour edge cases are respected or not. We can do that by modifying our test suite, the updated version looks something like this :

 package lo;

import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

class InsurancePremiumCalculatorTest {

    private InsurancePremiumCalculator calculator;

    @BeforeEach
    void setUp() {
        calculator = new InsurancePremiumCalculator();
    }

    @Test
    void testBasicPremium() {
        assertEquals(500.0, calculator.calculatePremium(30, "sedan"));
    }

    @Test
    void testYoungDriverPremium() {
        assertEquals(750.0, calculator.calculatePremium(20, "sedan"));
    }

    @Test
    void testSportsCarPremium() {
        assertEquals(850.0, calculator.calculatePremium(25, "sports")); 
        //changing 30 to 25 to test the edge case
    }

    @Test
    void testCombinedFactors() {
        assertEquals(1275.0, calculator.calculatePremium(22, "sports"));
    }

    @Test
    void testSUV() {
        assertEquals(975.0, calculator.calculatePremium(22, "suv"));
    }

    @Test
    void testInvalidAge() {
        assertThrows(IllegalArgumentException.class, () ->
                calculator.calculatePremium(15, "sedan"));
        assertThrows(IllegalArgumentException.class, () ->
                calculator.calculatePremium(121, "sedan"));
    }

    @Test
    void testMaxAge() {
        assertEquals(650.0, calculator.calculatePremium(120, "suv"));
        //explicitly testing the edge cases
    }

    @Test
    void testMinAge() {
        assertEquals(975.0, calculator.calculatePremium(16, "suv"));
        //explicitly testing the edge cases
    }
    @Test
    void testInvalidVehicleType() {
        assertThrows(IllegalArgumentException.class, () ->
                calculator.calculatePremium(30, "invalid"));
    }

}

We run the command again to generate a new report :

BINGO! Our test strength is now 11/11, making our tests more reliable and better equipped to catch bugs if any changes occur.

Conclusion

Mutation testing is an invaluable tool for enhancing test effectiveness and ensuring code quality. By incorporating PIT into your testing strategy, you can uncover hidden weaknesses in your tests and build more robust applications.