Performance & Scalability

Why Performance & Scalability Matter

Failures of performance are not always crashes. More often they are brownouts: requests still complete, but slowly enough that timeouts fire upstream, retries pile up, queues fill, and the user sees a half-loaded page. Brownouts are harder to diagnose than outages because everything is technically “up.” A throughput SLO (“500 RPS sustained”) and a latency SLO (“P99 < 250 ms”) together define the contract that prevents brownouts from becoming the steady state.

Two numbers that matter

Little’s Law is the only equation in this tutorial you have to memorize. If a service handles 1,000 RPS at 200 ms average latency, it has 200 requests in flight on average. Push latency to 2 seconds and that becomes 2,000 in flight — ten times the threads, ten times the memory, often ten times the database connections. Latency and throughput are not independent. They are two sides of the same equation.

Latency Budgets

A latency budget starts with the number the user feels. For an interactive web request that’s typically 250–500 ms at P95. For a mobile API call that backs a UI animation it might be 50–100 ms. For a payment authorization with a hardware terminal in the loop it might be 2 s at P99. The number is non-negotiable; it falls out of human perception research and product requirements.

Once you have the end-to-end number, you decompose it. Suppose checkout has a 300 ms P95 budget and the request fans out as: edge → auth → cart → pricing → inventory → payment → database.

Notice three things. First, payment is the dominant slice — an external dependency you don’t control. Second, inventory and pricing can be parallel; a sequential implementation would blow the budget. Third, there is deliberate slack at the bottom. A budget with no slack is a budget you will miss every time someone runs a GC.

P50, P95, P99, and the long tail

The other reason percentiles matter for microservices: tail latencies amplify on fan-out. If one downstream call has a P99 of 1%, and your request makes ten parallel calls, the probability that at least one of those calls hits the P99 tail is roughly 1 - 0.99^10 ≈ 9.6%. Your effective P99 at the request level is closer to your downstreams’ P90. The more services on the critical path, the more aggressively you must trim tails.

Horizontal vs Vertical Scaling

Microservices are designed to favor horizontal scaling because every service is small enough to hold in memory on commodity hardware. The work of horizontal scaling is in the architecture: keep services stateless so that any replica can serve any request, push state into databases, caches, and object stores, and make sure routing doesn’t pin users to a specific replica unless you genuinely need session affinity.

Stateful services scale differently

Databases, caches, and message brokers are stateful. Adding a replica doesn’t double capacity for free — the replica has to either replicate the existing data (read replicas, easy) or take ownership of a partition of the keyspace (sharding, hard). The right model:

• Read-heavy workloads — vertical first (one beefy primary), then horizontal via read replicas.
• Write-heavy workloads — vertical until the primary is the biggest sensible box, then shard.
• Caches — horizontal from day one, with consistent hashing so adding a node doesn’t evict everything.
• Brokers (Kafka, NATS, RabbitMQ) — partition the topic across brokers; consumers scale by claiming partitions.

Caching Strategies

Caching is the cheapest performance work you can do per dollar — and the easiest to get wrong. The patterns to know:

Cache aside in Python with Redis

import json
import random
import redis
from typing import Optional

r = redis.Redis(host="redis-prod.svc", port=6379, decode_responses=True)

def get_product(product_id: str) -> Optional[dict]:
    key = f"product:{product_id}"

    # 1. Try cache
    cached = r.get(key)
    if cached is not None:
        if cached == "__miss__":
            return None
        return json.loads(cached)

    # 2. Miss — load from DB
    product = db.fetch_product(product_id)
    if product is None:
        # Negative caching: remember misses too, briefly,
        # so a hot 404 doesn’t hammer the DB.
        r.setex(key, 30, "__miss__")
        return None

    # 3. Populate cache with TTL + small jitter to avoid stampedes
    ttl = 300 + random.randint(0, 30)
    r.setex(key, ttl, json.dumps(product))
    return product

Two details that separate a working cache from a production cache: a small jitter on TTLs so that 10,000 keys created in the same second don’t all expire in the same second, and negative caching so a flood of requests for a deleted product doesn’t become a flood of database lookups for nothing.

Cache stampede and request coalescing

import threading

_locks: dict[str, threading.Lock] = {}
_locks_guard = threading.Lock()

def get_with_coalescing(key: str):
    cached = r.get(key)
    if cached is not None:
        return json.loads(cached)

    # One lock per key; only one thread per process loads.
    with _locks_guard:
        lock = _locks.setdefault(key, threading.Lock())

    with lock:
        cached = r.get(key)                # double-check after acquiring
        if cached is not None:
            return json.loads(cached)
        value = db.fetch(key)
        r.setex(key, 300, json.dumps(value))
        return value

For coalescing across processes, you need a distributed lock (Redis SET NX EX) or a queue-based approach where the first miss writes a sentinel like __loading__ and others poll briefly. Either way, the rule is: one origin call per key per refresh window, no matter how many concurrent misses you get.

The cache hierarchy

Async Processing and Backpressure

Anything that doesn’t need to complete before you respond to the user should not run on the request path. The basic shape:

1. Request handler does the minimum — validate, persist, enqueue.
2. Worker pool drains the queue at its own pace.
3. Client polls for status, or the system pushes a webhook / WebSocket update when done.

This converts a latency problem into a throughput problem — which is much easier to scale. Workers can batch, debounce, and run at full CPU without anyone watching a spinner.

Backpressure: when the producer outruns the consumer

If producers can submit faster than consumers can drain, the queue grows without bound until you run out of memory or disk. Backpressure is the mechanism that pushes the “slow down” signal back to the producer.

The rule: bounded queues only. An unbounded in-memory queue is a memory leak with a polite name. Pick a bound, pick a strategy for what to do when you hit it, and surface a metric so you know when you’re hitting it.

Reactive frameworks — Project Reactor, RxJava, Akka Streams — build backpressure into their type system: a downstream operator requests N items from upstream, and upstream is forbidden from emitting more. Plain queues require you to enforce this manually. Batching and debouncing on the consumer side compound the savings: a worker that flushes every 100 ms or every 500 messages, whichever comes first, can sustain 10× the throughput of one that processes one message at a time.

Database Performance

Connection pooling

A database connection is expensive to create — TCP handshake, TLS, authentication, session setup — and Postgres in particular allocates a process per connection. Opening a connection per request will destroy a database long before traffic does. Pool them.

# Python: SQLAlchemy connection pool tuned for a microservice
from sqlalchemy import create_engine

engine = create_engine(
    "postgresql+psycopg://user:pass@db.svc:5432/orders",
    pool_size=20,            # steady-state connections held open per process
    max_overflow=10,         # burst capacity above pool_size
    pool_timeout=3,          # seconds to wait for a connection — fail fast
    pool_recycle=1800,       # recycle every 30 min to dodge stale connections
    pool_pre_ping=True,      # validate before lending (cheap SELECT 1)
)

The N+1 problem

The single most common slow-query bug. You list 100 orders, then for each order you look up the customer — that’s 1 query for the list and 100 queries for the customers, hence “N+1.” The fix is always the same: fetch the related data in one query (JOIN, IN (...), an ORM select_related / preload), or batch the lookups (DataLoader pattern).

# BAD — 1 + N queries
orders = session.query(Order).limit(100).all()
for o in orders:
    print(o.customer.name)        # lazy-loaded query per order

# GOOD — 1 query with join
from sqlalchemy.orm import joinedload
orders = (session.query(Order)
                 .options(joinedload(Order.customer))
                 .limit(100)
                 .all())

Indexes for the access pattern

Indexes make reads fast and writes a bit slower. The mistake is to index every column “just in case.” Index for the queries you actually run, in the order they actually filter and sort. A composite index on (tenant_id, created_at DESC) serves WHERE tenant_id = ? ORDER BY created_at DESC LIMIT 50 directly. The same index does almost nothing for WHERE created_at > ? alone, because Postgres can’t skip the leading column.

Read replicas, sharding, and hot keys

• Read replicas — cheap horizontal read scaling. Send analytics, list endpoints, and reports to a replica. Accept replication lag (often 50–500 ms) and don’t read your own writes from a replica.
• Sharding — split the data across multiple primaries by a shard key (user_id hash, tenant_id, geographic region). Adds operational complexity; pay this cost only when one primary genuinely cannot keep up with writes.
• Hot keys / hot partitions — one shard handling 10× the traffic of the others. Common when shard key correlates with traffic (e.g., tenant_id when one tenant is huge). Mitigations: salt the key, split the hot tenant onto its own shard, or cache aggressively in front of it.

Autoscaling

HPA on a custom metric

The default HPA watches CPU. CPU is fine for CPU-bound work. For I/O-bound services — most microservices — CPU is a lagging indicator. By the time CPU climbs, your queue depth is already five seconds deep. Scale on the metric that predicts queueing.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order-worker
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: order-worker
  minReplicas: 3
  maxReplicas: 200
  metrics:
    # Primary signal: how many messages are sitting in the queue
    - type: External
      external:
        metric:
          name: kafka_consumergroup_lag
          selector:
            matchLabels:
              topic: "orders"
              consumergroup: "order-worker"
        target:
          type: AverageValue
          averageValue: "500"      # aim for ~500 messages of lag per pod
    # Safety net so we don’t pin CPUs at 100% under burst
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 75
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0     # scale up fast
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300   # scale down slow
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60

Two parts of this YAML deserve attention. First, the metric is queue lag, not CPU. For a worker, lag is what the user feels. Second, the behavior block makes scale-up fast and scale-down slow — an asymmetry that matters because adding a pod that you didn’t need costs cents, while removing a pod you did need costs an outage.

Cold starts and predictive scaling

A new replica is not instantly useful. It pulls an image, starts the runtime, JIT-compiles, fills caches, opens connection pools. For a JVM service this can be 30–90 seconds; for serverless functions, often a few seconds; for a heavy Python service with model weights, minutes. During that warmup, the new pod is essentially a hot spare with bad latency.

Mitigations: readiness probes that hold the pod out of rotation until ready; over-provisioning to absorb traffic during scale-up; predictive autoscaling (Knative, KEDA Cron-based scalers, AWS predictive scaling) that scales ahead of known traffic patterns based on time of day rather than waiting for the metric to spike.

Profiling and Load Testing

Profiling: where does the time actually go

• CPU profiling — flame graphs show which call paths dominate CPU time. pprof for Go, py-spy / scalene for Python, async-profiler for the JVM, perf for native code. Always do this on production-like hardware, not your laptop.
• Allocation profiling — what’s allocating and triggering GC. The slowest code is often code that allocates objects in a hot loop.
• Distributed tracing — OpenTelemetry, Jaeger, Tempo. Shows where a single request spent its time across services. Indispensable for finding which downstream owns the P99 tail.
• Continuous profiling — Pyroscope, Parca, Polar Signals, Google Cloud Profiler. Always-on profiling in production at low overhead. The flame graph for the user complaint that came in last Tuesday is still there.

Load testing with k6

// k6 load test — ramp to 500 RPS, hold, then ramp down.
// k6 is the modern default; Locust, Gatling, Vegeta cover the rest.
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  scenarios: {
    checkout: {
      executor: 'ramping-arrival-rate',
      startRate: 10,
      timeUnit: '1s',
      preAllocatedVUs: 100,
      maxVUs: 2000,
      stages: [
        { target: 100, duration: '2m'  },   // warm up to 100 RPS
        { target: 500, duration: '5m'  },   // climb to 500 RPS
        { target: 500, duration: '10m' },   // hold at 500 RPS
        { target: 0,   duration: '2m'  },   // ramp down
      ],
    },
  },
  thresholds: {
    'http_req_duration{status:200}': ['p(95)<250', 'p(99)<500'],
    'http_req_failed': ['rate<0.001'],   // fail the run if >0.1% errors
  },
};

export default function () {
  const r = http.post('https://api.example.com/checkout', JSON.stringify({
    cart_id: 'c-12345',
    payment_token: 'tok_test',
  }), { headers: { 'Content-Type': 'application/json' } });

  check(r, {
    'status is 200': (res) => res.status === 200,
    'body has order_id': (res) => res.json('order_id') !== undefined,
  });
  sleep(1);
}

The thresholds block is what makes a load test useful in CI. The test fails if P95 exceeds 250 ms or error rate exceeds 0.1%. Tie that to a pull-request check and performance regressions get caught at the same speed bug regressions do. Locust covers the same ground in Python; Gatling in Scala for higher concurrency on a single load generator; Vegeta for simple constant-rate hammering from the CLI.

Querying P99 in Prometheus

# P99 latency over the last 5 minutes, per service.
# Assumes a histogram named http_request_duration_seconds with le buckets.
histogram_quantile(
  0.99,
  sum by (service, le) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

# Burn-rate alert: P99 over the last 5m vs. the SLO of 250ms,
# multiplied by request volume so noisy low-traffic windows don’t fire.
(
  histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
  > 0.25
) and (
  sum(rate(http_request_duration_seconds_count[5m])) > 10
)

Real-World Examples

Twitter’s timeline fan-out. When a user posts a tweet, Twitter doesn’t compute timelines on read. It pre-computes them on write — the “fan-out on write” model. Each follower’s home timeline is a Redis list; a new tweet is appended to every follower’s list. Reads become an O(1) lookup instead of an expensive query that joins follows and tweets. The wrinkle is celebrities: fanning out a Taylor Swift tweet to 90 million followers synchronously is impossible. Twitter switches to fan-out on read for high-degree accounts — a hybrid that trades write cost for the worst tail.

Discord’s Elixir-based fan-out. Discord runs millions of concurrent WebSocket connections per server using Elixir’s lightweight processes (BEAM). When a message is posted in a channel, the channel process broadcasts to every subscriber process — cheaply, because each “process” is a few KB instead of an OS thread. Their published numbers describe handling five million concurrent users on a relatively small fleet. The lesson: language and runtime choice can be a performance lever an order of magnitude bigger than micro-optimizations.

Netflix pre-warming for the Friday night peak. Netflix knows traffic is going to triple at 8pm Eastern. They don’t wait for the autoscaler to figure that out. They pre-scale the fleet ahead of the curve, pre-warm caches with predicted hot content, and run chaos experiments in the morning when scale-up is cheap. Predictive scaling combined with pre-warming turns a steep latency cliff into a gentle hill.

Cloudflare’s edge caching. Cloudflare serves a meaningful fraction of the public internet from edge nodes within ~50 ms of every user. Static assets, then API responses with sensible Cache-Control, then increasingly dynamic content via Workers and KV at the edge. The architecture is “cache as much as possible as close as possible to the user” taken to the global limit. The result: most requests never traverse the public internet to an origin.

Stripe’s API latency culture. Stripe publishes their P99 latency targets and treats regressions as outage-grade incidents. The discipline shows up everywhere: aggressive timeouts, idempotency keys for safe retries, hot-path code reviewed line by line, custom database routing to keep critical writes off congested shards. The aggregate effect is an API that feels boringly fast even at peak. Performance is a culture, not a project.

Best Practices