Brushing Over Real Problems

TL;DR: You write tests that touch every line but verify nothing, creating false confidence in a broken system.

Problems 😔

• False confidence
• Hidden production defects
• Misleading metrics
• Wasted test effort
• Untested edge cases

Solutions 😃

1. Use mutation testing
2. Test real behaviors
3. Write assertive tests
4. Delete coverage-only tests

Refactorings ⚙️

Context 💬

Many teams set a coverage threshold: 80%, 90%, or even 100%.

When you chase that number, you write tests that call methods without checking the actual results.

A test that calls calculateTax() but only asserts result is not None executes the line.

It doesn't verify the tax calculation is correct.

The dashboard turns green.

Defects survive in production.

This is vanity coverage: cosmetic metrics that hide real problems. Like brushes that make surfaces look smooth while the rot stays underneath.

Mutation testing reveals the truth.

When you mutate production code and no tests fail, your coverage numbers lied.

Sample Code 💻

Wrong 🚫

describe('BankAccount', () => {
  test('deposit', () => {
    const account = new BankAccount(100);
    account.deposit(50);
    // Only checking it didn't crash
    expect(account).toBeDefined();
  });

  test('withdraw', () => {
    const account = new BankAccount(100);
    const result = account.withdraw(30);
    // No assertion about the result!
  });

  test('transfer', () => {
    const source = new BankAccount(200);
    const target = new BankAccount(0);
    // Just calling the method to "cover" the line
    source.transfer(50, target);
  });
});

Right 👉

describe('BankAccount', () => {
  test('deposit increases balance', () => {
    const account = new BankAccount(100);
    account.deposit(50);
    expect(account.balance()).toBe(150);
  });

  test('withdraw decreases balance', () => {
    const account = new BankAccount(100);
    account.withdraw(30);
    expect(account.balance()).toBe(70);
  });

  test('withdraw raises on insufficient funds', () => {
    const account = new BankAccount(50);
    expect(() => account.withdraw(100))
      .toThrow(InsufficientFundsError);
  });

  test('transfer moves money between accounts', () => {
    const source = new BankAccount(200);
    const target = new BankAccount(0);
    source.transfer(50, target);
    expect(source.balance()).toBe(150);
    expect(target.balance()).toBe(50);
  });
});

Detection 🔍

[X] Semi-Automatic

Run a mutation testing tool (PIT for Java, Stryker for JavaScript, mutmut for Python).

Count the surviving mutants.

If coverage is high but mutants survive, you have vanity coverage.

You can also search for assertion-free tests, single assertNotNull() assertions, or tests that still pass after you delete the entire production method body.

Exceptions 🛑

Smoke tests that call endpoints to verify the system starts are acceptable without detailed assertions.

These work when they complement a real test suite, not replace it.

Tags 🏷️

• Testing

Level 🔋

[x] Intermediate

Why the Bijection Is Important 🗺️

Your test suite must map each test to a real behavior in the MAPPER.

When a test covers a line without verifying observable behavior, you break that bijection.

Coverage tools only measure "lines executed."

Chase the number without bijection and your suite looks complete while missing real requirements entirely.

AI Generation 🤖

AI code generators sometimes produce vanity coverage.

Ask one to "add tests to reach 80% coverage," and it writes tests that call methods and assert trivially true facts.

The metric goes up.

Nothing gets verified.

AI Detection 🧲

AI can detect vanity coverage, but only if you ask the right questions.

Try: "Find tests with no real assertions" or "Find tests that pass when I delete the production method body."

Without those prompts, most AI tools see a green test and call it good.

Try Them! 🛠

Remember: AI Assistants make lots of mistakes

Suggested Prompt: Replace vanity coverage tests with tests that verify real behaviors and fail when production code is wrong

Conclusion 🏁

Coverage is a signal, not a goal.

When you treat it as a goal, you create vanity coverage that hides real defects.

Use mutation testing to discover what your suite actually verifies.

Write tests that describe real behaviors, not tests that execute lines.

Relations 👩‍❤️‍💋‍👨

More Information 📕

Mutation Testing

Test Coverage Is Not Enough

Stryker Mutation Testing

Quote

The most dangerous kind of waste is the waste we don't recognize.

Shigeo Shingo

Disclaimer 📘

Code Smells are my opinion.

Credits 🙏

Photo by Jamie Street on Unsplash

This article is part of the CodeSmell Series.