Testing Strategies

Why Testing Strategies Matter for Microservices

Three uncomfortable facts shape everything in this tutorial.

1. End-to-end tests are not free safety. They are slow (minutes per run), flaky (network, timing, shared state), and they fail in ways that don’t tell you which service is wrong. Teams that lean on them ship slower and feel less confident, not more.
2. Unit tests alone are not enough. A unit test that mocks the HTTP client cannot catch the day someone changed a JSON field name. It can only catch logic mistakes — which it does brilliantly.
3. The boundary between services is the most dangerous code in your system. Contract tests exist specifically to make that boundary boring.

Everything below is about choosing the cheapest layer that catches each class of bug, then running that layer ruthlessly in CI.

The Test Pyramid (Updated)

Mike Cohn’s original pyramid had three layers. Microservices add a fourth that did not exist in monolith days: contract tests. They earn the slot because the cost of getting cross-service contracts wrong is uniquely high.

Honeycomb and Trophy variants

Spotify popularized the honeycomb shape: a thin layer of unit tests, a large middle of integration tests against real dependencies, and a thin layer of integrated UI tests. It works well when most of your code is glue between collaborators rather than complex algorithms. Kent C. Dodds’ testing trophy for frontend projects emphasizes integration over unit for similar reasons. Both are reactions to over-mocked unit tests that pass while the system is broken.

The shape that matches your codebase is the right one. The shape that does not match your codebase — usually inherited from a tutorial — is what causes the pain.

Unit Tests for Service Logic

The classic mistake is mocking your own modules. If OrderTotal.compute(items) is a pure function, do not mock it — call it. Mocks belong at the edges: HTTP clients, database adapters, message publishers. Inside the hexagon, real code calls real code.

# pytest unit test for a pricing rule
import pytest
from pricing import apply_volume_discount

class TestVolumeDiscount:
    def test_no_discount_below_threshold(self):
        assert apply_volume_discount(subtotal=99) == 99

    def test_ten_percent_at_threshold(self):
        assert apply_volume_discount(subtotal=100) == 90

    def test_caps_at_twenty_percent(self):
        assert apply_volume_discount(subtotal=10000) == 8000

    @pytest.mark.parametrize("subtotal", [-1, 0])
    def test_rejects_non_positive(self, subtotal):
        with pytest.raises(ValueError):
            apply_volume_discount(subtotal=subtotal)

Test doubles, classified

WireMock and Mountebank are the standard tools when you need a stub HTTP server during a unit-ish test — useful, but be honest with yourself: a test that hits a stub server is no longer a unit test. Promote it to integration.

Integration Tests with Testcontainers

Testcontainers exists for Java, Python, Go, .NET, Node.js, and Rust. The pattern is the same everywhere: start a container, wait for it to be ready, point the system under test at it, run assertions against the real data store.

// Java + JUnit 5 + Testcontainers + Postgres
import org.junit.jupiter.api.Test;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;

@Testcontainers
class OrderRepositoryIT {

    @Container
    static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>("postgres:16-alpine")
        .withDatabaseName("orders")
        .withUsername("test")
        .withPassword("test");

    @DynamicPropertySource
    static void props(DynamicPropertyRegistry r) {
        r.add("spring.datasource.url", postgres::getJdbcUrl);
        r.add("spring.datasource.username", postgres::getUsername);
        r.add("spring.datasource.password", postgres::getPassword);
    }

    @Autowired OrderRepository repo;

    @Test
    void savesAndLoadsOrderWithItems() {
        Order o = Order.draft("cust-42")
            .addItem("sku-1", 2, 19.99);
        repo.save(o);

        Order loaded = repo.findById(o.getId()).orElseThrow();
        assertEquals(2, loaded.getItems().size());
        assertEquals("cust-42", loaded.getCustomerId());
    }
}

This test catches things a fake never can: column name typos, foreign key violations, JSONB serialization quirks, time-zone bugs, the difference between NUMERIC(10,2) and DOUBLE PRECISION. It runs in a few seconds because Testcontainers reuses containers across tests when you ask it to.

# Python + pytest + testcontainers + Redis
import pytest
import redis
from testcontainers.redis import RedisContainer
from ratelimiter import SlidingWindowLimiter

@pytest.fixture(scope="module")
def redis_client():
    with RedisContainer("redis:7-alpine") as r:
        client = redis.Redis(host=r.get_container_host_ip(),
                             port=r.get_exposed_port(6379))
        yield client

def test_rate_limiter_blocks_after_threshold(redis_client):
    limiter = SlidingWindowLimiter(redis_client, key="user:1", limit=5, window_s=10)
    for _ in range(5):
        assert limiter.allow() is True
    assert limiter.allow() is False     # 6th call is throttled

Consumer-Driven Contract Tests

This is the “deploy to staging and pray” pattern, retired. Contract tests buy you the confidence that end-to-end tests pretend to provide — without the cost. Pact is the dominant tool. Spring Cloud Contract is common in JVM shops. Both implement the same idea.

Consumer side (publishes the expectation)

// JavaScript consumer test using @pact-foundation/pact
const { PactV3, MatchersV3 } = require('@pact-foundation/pact');
const { like, integer, iso8601DateTime } = MatchersV3;
const { fetchOrder } = require('../src/orderClient');

const provider = new PactV3({
    consumer: 'shipping-service',
    provider: 'order-service',
});

describe('order client contract', () => {
    it('gets a confirmed order with customer and items', () => {
        provider
            .given('order 42 exists and is confirmed')
            .uponReceiving('a request for order 42')
            .withRequest({ method: 'GET', path: '/orders/42' })
            .willRespondWith({
                status: 200,
                headers: { 'Content-Type': 'application/json' },
                body: {
                    id: integer(42),
                    customerId: like('cust-7'),       // shipping needs this exact field
                    confirmedAt: iso8601DateTime(),
                    items: [{
                        sku: like('sku-1'),
                        qty: integer(2),
                    }],
                },
            });

        return provider.executeTest(async (mockServer) => {
            const order = await fetchOrder(mockServer.url, 42);
            expect(order.customerId).toBe('cust-7');
            expect(order.items).toHaveLength(1);
        });
    });
});

The test runs against a Pact-controlled mock that records the interaction as a pact file (JSON). The pact gets pushed to a Pact Broker keyed on consumer + provider + version.

Provider side (verifies the expectation)

# Python provider verification using pact-python
from pact import Verifier

verifier = Verifier(
    provider="order-service",
    provider_base_url="http://localhost:8080",
)

success, _ = verifier.verify_with_broker(
    broker_url="https://pact-broker.example.com",
    publish_version=os.environ["GIT_SHA"],
    publish_verification_results=True,
    provider_states_setup_url="http://localhost:8080/pact/setup",
)
assert success, "contract verification failed"

The provider runs against pacts from every consumer. If any consumer’s expectation breaks, the provider’s CI turns red. The provider knows exactly which consumer they’d break and can either coordinate the change or evolve the API additively.

End-to-End Tests

End-to-end tests still earn their slot. There are wiring bugs — an environment variable wrong in the helm chart, a missing OAuth scope, an ingress rule that drops gRPC headers — that no other layer catches. The mistake is treating them as the safety net rather than the smoke check.

Cypress and Playwright are the standard tools for browser-driven flows. For pure API end-to-end, k6 with --vus 1 works well, or REST Assured in Java, or pytest + httpx in Python. Whatever the tool, run them against a deployed environment that mirrors production — not against docker-compose up.

// Playwright end-to-end: place an order and verify shipping created
import { test, expect } from '@playwright/test';

test('guest checkout creates a shipment', async ({ page, request }) => {
    await page.goto('https://staging.example.com');
    await page.getByText('Add to cart').click();
    await page.getByRole('link', { name: 'Checkout' }).click();
    await page.getByLabel('Email').fill('guest@example.com');
    await page.getByLabel('Card').fill('4242 4242 4242 4242');
    await page.getByRole('button', { name: 'Pay' }).click();

    const orderId = await page.locator('[data-testid=order-id]').innerText();

    // Verify the downstream service received it (eventual consistency)
    await expect.poll(async () => {
        const r = await request.get(`https://staging.example.com/api/shipments?order=${orderId}`);
        return (await r.json()).status;
    }, { timeout: 30000 }).toBe('pending');
});

Hermetic test environments

The most expensive part of end-to-end is environment ownership. A shared staging gets corrupted by everyone’s test data. Two answers work in practice: per-PR ephemeral environments (spin up a stack on demand, tear it down on merge), or hermetic isolation in a shared environment via tenant IDs that scope all data. Pick one early; retrofitting either is painful.

Synthetic Tests in Production

APM tells you that requests are failing. Synthetics tell you that specific business flows are failing. The two complement each other: APM watches everything noisily, synthetics watch the things that matter quietly.

// Datadog Synthetics: a multi-step API check that runs every minute
{
  "name": "checkout - guest happy path",
  "type": "api",
  "subtype": "multi",
  "locations": ["aws:us-east-1", "aws:eu-west-1"],
  "options": { "tick_every": 60, "min_failure_duration": 120 },
  "steps": [
    {
      "name": "create cart",
      "request": { "method": "POST", "url": "https://api.example.com/v1/carts" },
      "assertions": [{ "type": "statusCode", "operator": "is", "target": 201 }]
    },
    {
      "name": "add item",
      "request": { "method": "POST", "url": "https://api.example.com/v1/carts/{{cart_id}}/items" },
      "assertions": [{ "type": "responseTime", "operator": "lessThan", "target": 800 }]
    },
    {
      "name": "checkout with test card",
      "request": { "method": "POST", "url": "https://api.example.com/v1/checkout" },
      "assertions": [
        { "type": "statusCode", "operator": "is", "target": 200 },
        { "type": "body", "operator": "validatesJSONPath", "target": "$.order_id" }
      ]
    }
  ]
}

Same idea, generic version using a cron-scheduled HTTP probe and a tagged synthetic user account:

# Generic cron-scheduled synthetic, written in Python, alerts via PagerDuty
# Schedule: */1 * * * *  /usr/local/bin/synthetic-checkout
import os, sys, time, requests
from pagerduty import EventV2

BASE = "https://api.example.com"
SYNTH_USER = os.environ["SYNTH_USER"]      # real account, tagged synthetic=true
TOKEN = os.environ["SYNTH_TOKEN"]
PD_KEY = os.environ["PD_ROUTING_KEY"]

def page(summary):
    EventV2(routing_key=PD_KEY).trigger(
        summary=summary, severity="error", source="synthetic-checkout",
    )

def main():
    started = time.monotonic()
    s = requests.Session()
    s.headers["Authorization"] = f"Bearer {TOKEN}"

    cart = s.post(f"{BASE}/v1/carts", timeout=5).json()
    s.post(f"{BASE}/v1/carts/{cart['id']}/items",
           json={"sku": "SYNTH-SKU-1", "qty": 1}, timeout=5)
    r = s.post(f"{BASE}/v1/checkout",
               json={"payment_method": "synthetic-test-card"}, timeout=10)

    if r.status_code != 200 or "order_id" not in r.json():
        page(f"checkout failed: {r.status_code} {r.text[:200]}")
        sys.exit(1)

    if time.monotonic() - started > 3.0:
        page("checkout slow: >3s")

if __name__ == "__main__":
    main()

Chaos and Load Testing

Chaos engineering — injecting controlled failures — is covered in detail in the circuit breaker tutorial. The short version: you write a hypothesis (“when payment is 2 s slow, checkout success stays above 95%”), inject the failure with LitmusChaos, Chaos Mesh, Gremlin, or AWS FIS, and roll back automatically if your steady-state metrics go red. Here we focus on load and performance testing as the complement.

Load testing with k6

k6 is the de-facto modern load testing tool: scripts in JavaScript, executed by a Go runtime that can drive tens of thousands of virtual users from a single box. Locust is the Python-native equivalent. Either fits cleanly into CI as a regression gate.

// k6 load test: ramp to 200 RPS, hold, ramp down
import http from 'k6/http';
import { check, sleep } from 'k6';
import { Trend } from 'k6/metrics';

const checkoutLatency = new Trend('checkout_latency_ms');

export const options = {
    stages: [
        { duration: '2m', target: 50  },   // ramp up
        { duration: '5m', target: 200 },   // hold at peak
        { duration: '2m', target: 0   },   // ramp down
    ],
    thresholds: {
        // CI fails if these regress
        http_req_failed:   ['rate<0.01'],            // <1% errors
        http_req_duration: ['p(95)<500', 'p(99)<1500'],
        checkout_latency_ms: ['p(95)<800'],
    },
};

export default function () {
    const r = http.post('https://staging.example.com/v1/checkout',
        JSON.stringify({ sku: 'SKU-1', qty: 1 }),
        { headers: { 'Content-Type': 'application/json' } });
    checkoutLatency.add(r.timings.duration);
    check(r, { '200': (resp) => resp.status === 200 });
    sleep(1);
}

Wire this into CI as a nightly job against staging. The thresholds turn it into a regression gate: if a code change pushes p95 from 400 ms to 700 ms, the build fails before it ships. Without thresholds, load tests become reports nobody reads.

Real-World Examples

Spotify’s pyramid evolution. Spotify originally invested heavily in end-to-end tests across hundreds of services. The suite became so slow and flaky that engineers stopped trusting it. They moved to a honeycomb shape: thin unit layer, large middle of integration tests against real dependencies in containers, and a small set of consumer-driven contract tests at service boundaries. The end-to-end layer shrank to a handful of business-critical journeys. Deploy frequency increased while incident rate dropped — documented in their engineering blog.

Netflix Chaos Engineering. Chaos Monkey came from Netflix recognizing that end-to-end tests in staging never matched production conditions. The Simian Army — Latency Monkey, Conformity Monkey, Janitor Monkey — moved testing into production with explicit hypotheses and rollback. Today the discipline of chaos engineering is more useful than the original tooling, and tools like LitmusChaos, Chaos Mesh, and Gremlin productize it.

Google’s hermetic builds and the Beyonders. Google’s test infrastructure is built around hermeticity: tests bring their own dependencies, the build system enforces it, and there is no “works on my machine”. The testing philosophy is documented in Software Engineering at Google. Every test is small (one process), medium (one machine, real deps), or large (multi-machine) — and the pyramid weights are enforced by the test runner.

Stripe’s strict CI gates. Stripe runs tens of thousands of tests on every PR using a custom build system. Contract tests against critical APIs are mandatory; flakiness is treated as a P1 bug because flaky tests erode trust faster than missing tests. Their public engineering writeups describe deliberately tight feedback loops: a test that takes more than a few minutes goes on a budget and gets refactored.

The pattern across all four: they invested heavily in the lower layers of the pyramid and ruthlessly pruned the top. None of them rely on end-to-end tests as their main safety net.

Best Practices