LUNAROPS · OPERATIONAL UPLINK 100% UPTIME 1,247d POSTS 893 JEFF.MOON@LUNAROPS.DEV UTC --:--:--

Testing Strategies That Actually Work

14 min read Software Development
testingtddunit-testsintegration-testingtest-doublesquality
Contents

A test suite is not a correctness proof and was never going to be one. Dijkstra’s line — testing shows the presence of bugs, never their absence — is true and worth remembering, but it leads people to the wrong conclusion. The point of testing is not to prove the code is right; it is to buy confidence to change the code. A codebase with a good suite is one where an engineer can refactor a module, add a feature, or upgrade a dependency and find out within seconds whether they broke something, instead of finding out three weeks later from a customer. That confidence is the product. Everything else — coverage numbers, the testing pyramid, TDD, the choice between mocks and fakes — is in service of maximizing that confidence per unit of effort, because the effort is finite and the tests themselves are code you have to maintain forever.

That framing matters because it turns testing from a moral obligation (“good engineers write tests”) into an economic one (“where does a test dollar buy the most confidence?”). A test that never fails teaches you nothing and costs maintenance. A test so coupled to implementation that it breaks every time you refactor — even when behavior is unchanged — is worse than no test, because it punishes exactly the activity tests are supposed to enable. The entire discipline is about writing tests that fail when behavior breaks and stay silent when it doesn’t, and surprisingly little popular testing advice is actually optimizing for that.

The pyramid, the trophy, and what they are really arguing about

The testing pyramid is the canonical mental model: many fast unit tests at the base, fewer integration tests in the middle, a handful of slow end-to-end tests at the top. The shape encodes a real economic truth — tests get slower, more brittle, and more expensive to maintain as they climb, so you want most of your tests to be cheap and fast.

            /\
           /  \        E2E / UI  (few)  -- slow, flaky, high-fidelity
          /----\
         /      \      Integration (some) -- real collaborators, real I/O
        /--------\
       /          \    Unit (many) -- fast, isolated, milliseconds
      --------------
        more tests, lower as you go down

But the pyramid is contested, and the disagreement is instructive rather than academic. Kent C. Dodds popularized the “testing trophy,” which fattens the integration layer on the argument that integration tests deliver the most confidence per test because they exercise units the way they are actually used together, and modern tooling has made them far less slow than the pyramid assumes. The slogan — “write tests, not too many, mostly integration” — is a deliberate rebuke to teams that chase 100% unit coverage and ship anyway because nothing tested the seams between units.

Both are right about different codebases, and the synthesis is the actual lesson: the right shape depends on where your bugs actually live. If your bugs are in tricky pure logic — pricing, parsing, state machines — invest in unit tests, because they pin that logic precisely and cheaply. If your bugs are in the wiring — the serializer that disagrees with the database schema, the auth middleware that the handler assumed ran — invest in integration tests, because no amount of unit testing the pieces in isolation catches a contract mismatch between them. Look at your last twenty production incidents and let their distribution, not a diagram, set your ratios.

Layer Speed Fidelity Brittleness Best at catching
Unit Milliseconds Low (isolated) Low if behavior-focused Logic errors, edge cases in pure functions
Integration Tens of ms–seconds High (real collaborators) Medium Contract mismatches, wiring, serialization, queries
End-to-end Seconds–minutes Highest (real system) High (flaky, slow) Critical user journeys break end to end

Test behavior, not implementation

The single most important property of a good test is that it is coupled to what the code does, not how it does it. A test that asserts on observable behavior — given this input, the function returns this output, or this side effect happens — survives any refactor that preserves behavior. A test that reaches inside and asserts “it called helper() twice and set this._cache to a Map” breaks the moment you restructure the internals, even when the behavior is identical. The second kind of test actively fights refactoring, which is sabotage, because the whole reason you wrote tests was to make refactoring safe. This is the exact handshake with safe refactoring: tests that pin behavior are the net that lets you change structure freely; tests that pin structure are a net with your own feet tangled in it.

Concretely, this means testing through the public interface and asserting on outcomes. A weak test confirms something happened; a strong test confirms the right thing happened, with values specific enough that a wrong answer can’t slip through:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

// Weak: passes for almost any non-null return. Teaches you nothing on failure.
it('works', () => {
  expect(processOrder(order)).toBeTruthy();
});

// Strong: pins the actual contract. A wrong number fails loudly and tells you which.
it('applies the 10% discount before 8% tax', () => {
  const order = { items: [{ price: 100 }, { price: 50 }], discount: 0.1, taxRate: 0.08 };
  const result = processOrder(order);
  expect(result.subtotal).toBe(150);
  expect(result.discount).toBe(15);
  expect(result.tax).toBe(13.5);     // tax on the discounted 135, not the full 150
  expect(result.total).toBe(148.5);
});

The strong test doubles as documentation: it states, executably, that tax is computed on the discounted subtotal — a business rule a reader would otherwise have to reverse-engineer. Tests are the most reliable documentation a codebase has, because unlike comments they cannot drift out of sync with the code without turning red. The corollary is to name tests as specifications of behavior — applies_discount_before_tax, not test_process_order_2 — so that a failure report reads like a sentence describing what broke.

Test doubles: the necessary evil you can overdose on

Real code touches the network, the clock, the database, the payment gateway. To test a unit in isolation you replace those collaborators with test doubles, and the choices among them have consequences that trip up even experienced teams. The vocabulary is worth getting precise because the words get used interchangeably and the differences matter:

  • A stub returns canned answers: “when asked for user 123, return this object.” It controls inputs to the code under test.
  • A fake is a real, working, lightweight implementation: an in-memory database, a hash-map cache. It behaves like the real thing without the cost.
  • A mock is a stub that also asserts on interactions: “the code must call charge() exactly once with these arguments.” It verifies behavior by checking calls.

The trap is over-mocking. Mock everything and you get a test that asserts your code made a specific sequence of calls to objects that no longer resemble reality — and the test passes happily while production breaks, because the real collaborator’s behavior diverged from your mock’s. This is the same mock/prod divergence that masks broken migrations: a mocked dependency is a frozen assumption about how that dependency behaves, and assumptions rot. The discipline is to mock at the system’s true edges — the third-party API, the email sender — and to prefer fakes over mocks for things you own, because a fake that actually stores and retrieves data catches contract errors a mock will cheerfully wave through.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Over-mocked: asserts on calls to a fiction. Green even if the real repo would reject this.
def test_create_user_overmocked():
    repo = Mock()
    service = UserService(repo)
    service.create("a@b.com")
    repo.save.assert_called_once()        # tests that you called save, not that it worked

# Better: a fake repo with real-ish behavior catches the duplicate-email bug for real.
def test_create_user_rejects_duplicate():
    repo = InMemoryUserRepo()             # actually stores and enforces uniqueness
    service = UserService(repo)
    service.create("a@b.com")
    with pytest.raises(DuplicateEmail):
        service.create("a@b.com")         # exercises the real constraint

For the seams between services — where you can’t share a fake — the answer is contract testing: the consumer records the requests it makes and the responses it expects, and the provider’s CI verifies it still honors that contract. This catches the classic distributed failure where a provider renames a field and silently breaks three consumers who all had green test suites. It is the testing counterpart to the API discipline of stable contracts and versioning — and it is the only practical way to get integration confidence across microservice boundaries without standing up the entire system for every test run.

Flakiness is a correctness bug, not an annoyance

A flaky test — one that passes and fails on the same code without any change — is the most corrosive thing in a test suite, and teams chronically under-react to it. The damage is not the occasional red build; it is that flakiness destroys the suite’s signal. Once developers learn that red sometimes means nothing, they start re-running until green, and then they reflexively re-run a real failure until it happens to pass, and now the suite catches nothing because no one believes it. One tolerated flaky test poisons trust in the whole suite.

Flakiness has a small number of root causes, and every one of them is a real defect — often in the code, not just the test:

  • Time. Tests that depend on wall-clock time, sleeps, or timeouts. Fix: inject a controllable clock; never sleep() and hope.
  • Order dependence. Tests that share mutable state and pass only in a particular order. Fix: isolate state; randomize test order in CI to flush these out.
  • Concurrency and races. The test exposes a real race condition that sometimes loses. The flaky test is doing its job — it found a genuine bug. Fix the code.
  • External dependencies. A test hitting a real network endpoint that is occasionally slow or down. Fix: fake the boundary.
  • Nondeterministic data. Random seeds, map iteration order, “today’s date.” Fix: pin the seed, sort before asserting, freeze the date.

The right policy is to treat a flaky test as a stop-the-line event: quarantine it immediately so it stops poisoning the signal, then fix the root cause with the same rigor as any other bug. The investigation is identical to systematic debugging — reproduce it (run it a thousand times in a loop until it fails), find the hidden state it depends on, pin that state. A test that “fails sometimes” is a system that “fails sometimes,” and the test is the cheapest place you will ever observe that.

TDD, honestly

Test-driven development — write a failing test, write the minimum code to pass it, refactor, repeat — generates strong opinions disproportionate to its actual scope. Used honestly, it is a design technique as much as a testing one: writing the test first forces you to use your own interface before it exists, which surfaces awkward APIs early, and the “minimum code to pass” discipline keeps you from gold-plating. The red-green-refactor loop also guarantees you never have untested code and never write a test that passes vacuously, because you watched it fail first.

But the honest version admits where it doesn’t fit. TDD shines when you understand the desired behavior precisely — a parser, a pricing rule, a state machine with clear inputs and outputs. It fights you when you are exploring — spiking a UI, feeling out an unfamiliar API, doing research-grade work where you don’t yet know what the function should do. Writing tests first for code whose shape you’re still discovering means rewriting the tests every time the design shifts, which is pure friction. The mature stance: TDD is a tool, not a religion. Reach for it when the spec is clear and the logic is tricky; spike first and test after when you’re exploring, then backfill the tests once the design settles. What is not optional is that the code is tested before it merges — whether the test was written thirty seconds before the code or thirty minutes after is a workflow preference, not a moral axis.

Coverage is a gauge, not a target

Code coverage measures which lines ran during the tests. It is genuinely useful as a gauge: a coverage report that shows your new payment logic is 0% covered is telling you something true and actionable, and coverage that suddenly drops on a pull request is a useful review signal. The failure is making it a target. The instant “90% coverage” becomes a gate, Goodhart’s law takes over — a measure that becomes a target stops being a good measure — and developers write tests that execute lines without asserting anything meaningful, manufacturing green numbers that prove nothing.

1
2
3

# 100% line coverage. Asserts nothing. Pure theater for the dashboard.
def test_process_order_coverage():
    process_order(some_order)     # every line ran; no claim about the result

That test “covers” the function and protects against no bug whatsoever. Coverage tells you what code is definitely untested (the uncovered lines); it tells you nothing about whether the covered code is correctly tested. High coverage with weak assertions is a more dangerous state than low coverage, because it manufactures false confidence. Use coverage to find the gaps in code that matters — business logic, error paths, security-sensitive branches — and ignore it for the code that doesn’t need testing: framework glue, trivial getters, generated code, configuration. The goal is confidence, and confidence comes from good assertions on the code that can actually hurt you, not from a number.

A more honest signal, if you want one, is mutation testing: a tool deliberately introduces bugs (flips a > to >=, deletes a line) and checks whether your tests catch them. A test suite that stays green while the code is mutated is a suite that isn’t really testing anything, and mutation testing measures that directly — at real CPU cost, which is why it’s a periodic audit rather than a per-commit gate.

Making the suite an asset, not a tax

A test suite earns its keep only if developers actually run it, which means it has to be fast, reliable, and run automatically. The unit suite should finish in seconds, because a suite that takes ten minutes is a suite developers skip locally and only meet in CI, by which point the feedback loop is broken. Keep the fast tests fast and ruthlessly separate from the slow ones, so the millisecond unit tests can run on every save while the heavyweight end-to-end tests run on a schedule or pre-merge.

The whole thing comes together in continuous integration: every push runs the suite, and a red suite blocks the merge. This is what makes tests load-bearing rather than decorative — they become the gate that a human reviewer can rely on having passed, which is precisely the division of labor that lets code review focus on design and logic instead of “did you run the tests.” A few habits keep the suite an asset over years rather than a slowly-rotting liability:

  • Run before commit, gate before merge. Local for speed, CI for truth.
  • One logical concept per test. Not dogmatically one assertion, but one behavior — so a failure names exactly what broke.
  • Delete tests that earn their deletion. A test asserting behavior you intentionally changed should be updated or removed deliberately — never silently deleted to make the build green, but not preserved out of superstition either.
  • Treat test code as real code. It gets reviewed, refactored, and kept DRY-where-it-helps. Most teams under-invest here and end up with a brittle suite they resent.

Verdict

A testing strategy is a budget-allocation problem dressed up as a moral one. Tests do not prove correctness; they buy the confidence to change code, and the entire discipline is about maximizing that confidence per unit of effort spent and maintained. That reframing settles most of the famous debates. The pyramid versus the trophy is not a doctrine to pick but a question to answer empirically — put your tests where your bugs actually live, fat in the unit layer for tricky logic, fat in the integration layer for wiring and contracts. Test behavior, never implementation, so the suite survives the refactoring it exists to enable. Reach for fakes over mocks for things you own, contract tests across service boundaries, and treat every flaky test as the real bug it is reporting.

The unifying idea: a good test fails when behavior breaks and stays silent when it doesn’t, and everything else is noise. Coverage is a gauge of where you haven’t looked, not a target to game; TDD is a design tool for when the spec is clear, not a loyalty oath; speed and reliability are what make the suite something developers trust instead of route around. Build the suite that way and it stops being a tax you pay and becomes the thing that lets a team move fast without breaking what already works — which was the only reason to write tests in the first place.

Sources

Comments