Service Discovery

Why Service Discovery Matters

Discovery exists because the assumptions of the network changed. In a static datacenter you knew, when you deployed, exactly which box ran which service and on which port. You wrote that into /etc/hosts, an HAProxy config, or a wiki page, and it stayed true for months.

Containers broke every part of that. Pods are scheduled on whatever node has capacity. They get a new IP each time they restart. A horizontal scale-up triples the instance count in 30 seconds; a rolling deploy replaces every one of them inside two minutes. Auto-healing kills a sick instance and the replacement comes up somewhere else entirely. There is no longer a stable place where order-service lives. There is only a stable name — order-service — and a system that knows, right now, which IPs answer to that name.

What “discovery” actually means

The job decomposes into three responsibilities. Different systems wire them up differently, but the responsibilities are universal:

Client-Side vs Server-Side Discovery

Client-side discovery puts intelligence in the caller. The caller pulls the registry list, caches it, picks an instance, and retries on failure. Netflix Eureka was the canonical example: the Java client embedded the registry list and rebuilt it from the Eureka server every 30 seconds.

Server-side discovery hides the registry behind a stable address. The caller talks to one well-known endpoint — an AWS ELB, an NGINX, a Kubernetes Service IP — and the LB does the lookup. The caller knows nothing about pods, instances, or registries.

The industry has converged on server-side — specifically the service-mesh flavor of it — for one reason: you don’t want to ship a Eureka client for every language your team writes in. The mesh sidecar gives every pod “client-side” smarts without requiring any language-specific library.

Service Registries

The classic service registries:

Self-registration vs third-party registration

Two ways to get an instance into the registry:

• Self-registration: The service registers itself on startup and de-registers on shutdown. Simple, but every service needs the registry client baked in, and a hard crash skips de-registration.
• Third-party registration: An external agent — the K8s controller, an ECS scheduler, a Consul agent — watches the orchestrator and registers/de-registers on the service’s behalf. More moving parts; cleaner failure semantics.

Modern stacks default to third-party. Kubernetes does it for you: when a pod becomes Ready, the kubelet posts to the API server and the Endpoints object updates automatically. You write zero code for that.

Consul self-registration via JSON

// /etc/consul.d/order-service.json
// Picked up by the local Consul agent on startup.
{
  "service": {
    "id":      "order-service-7f3a",
    "name":    "order-service",
    "address": "10.0.4.23",
    "port":    8080,
    "tags":    ["v2", "prod", "region=us-east-1"],

    "meta": {
      "version": "2.14.0",
      "git_sha": "a91b3cf"
    },

    "checks": [
      {
        "name":     "http-health",
        "http":     "http://10.0.4.23:8080/health",
        "interval": "10s",
        "timeout":  "2s",
        "deregister_critical_service_after": "1m"
      },
      {
        "name":     "tcp-port",
        "tcp":      "10.0.4.23:8080",
        "interval": "5s"
      }
    ]
  }
}

Two checks because they fail for different reasons. The TCP check catches “the process is wedged but the port is still bound.” The HTTP check catches “the port answers but the app is broken.” deregister_critical_service_after is the safety net — if the instance has been failing for a minute it gets removed from lookups even if the agent didn’t hear about a clean shutdown.

DNS-Based Discovery

Kubernetes’ built-in discovery is DNS-based. Every Service object becomes a DNS record inside the cluster:

# A normal ClusterIP Service: stable virtual IP, hides individual pods.
# DNS: order-service.payments.svc.cluster.local → 10.96.42.1
apiVersion: v1
kind: Service
metadata:
  name: order-service
  namespace: payments
spec:
  selector:
    app: order-service
  ports:
    - name: http
      port: 80
      targetPort: 8080
  type: ClusterIP
---
# A *headless* Service: no virtual IP, DNS returns one A record per pod.
# DNS: order-service-headless.payments.svc.cluster.local →
#      10.0.4.23, 10.0.4.24, 10.0.4.25
apiVersion: v1
kind: Service
metadata:
  name: order-service-headless
  namespace: payments
spec:
  clusterIP: None              # the magic flag
  selector:
    app: order-service
  ports:
    - name: http
      port: 80
      targetPort: 8080

The headless variant is what you use when the client wants to do its own load balancing — gRPC clients, StatefulSet members talking to each other, sharded systems where the caller needs to know exactly which pod owns which key.

The TTL trap

ECS Service Discovery is the AWS-managed equivalent. You attach a service to an AWS Cloud Map namespace; ECS keeps Route 53 records in sync with the running tasks. Same trade-offs, plus the AWS bill.

SRV records vs A records

Plain A records give you an IP. SRV records give you IP and port plus weight and priority — useful when ports are dynamic, common in Consul setups (_order-service._tcp.service.consul). Most modern clients still prefer A + a known port; SRV support is uneven across language stdlibs.

Service Mesh Discovery

A service mesh splits in two:

• Data plane: the sidecars (Envoy in Istio, linkerd2-proxy in Linkerd). Every byte of service-to-service traffic flows through them. They handle discovery, load balancing, mTLS, retries, circuit breaking.
• Control plane: the brain (istiod, Linkerd’s control plane). It watches the orchestrator, computes routing tables, and pushes them to every sidecar via xDS or its equivalent.

Discovery becomes a non-issue from the application’s point of view. You write http://order-service/orders/123. The sidecar intercepts it, checks its locally cached endpoint table (pushed from the control plane), picks an instance, opens an mTLS connection, and proxies the request. The pod IP changes? The control plane notices, recomputes, pushes — usually within a second. Your code never knew there was a problem.

Linkerd traffic split

# Send 90% of traffic to v1, 10% to v2 — discovery + weighted routing
# done by the mesh, completely transparent to clients.
apiVersion: split.smi-spec.io/v1alpha1
kind: TrafficSplit
metadata:
  name: order-service-rollout
  namespace: payments
spec:
  service: order-service          # the apex service clients call
  backends:
  - service: order-service-v1
    weight: 90
  - service: order-service-v2
    weight: 10

The application keeps calling order-service. The mesh splits the traffic. To migrate to v2 you change the weights; no redeploy of any caller.

Health Checks

# Each probe answers a different question — wire them up correctly.
apiVersion: v1
kind: Pod
metadata:
  name: order-service
spec:
  containers:
  - name: app
    image: orderservice:2.14.0
    ports: [{ containerPort: 8080 }]

    startupProbe:               # app needs <= 60s to load caches
      httpGet: { path: /healthz, port: 8080 }
      periodSeconds: 5
      failureThreshold: 12      # 12 * 5s = 60s grace

    livenessProbe:              # cheap, port-level — just “am I alive”
      tcpSocket: { port: 8080 }
      periodSeconds: 10
      failureThreshold: 3

    readinessProbe:             # real work — am I able to serve?
      httpGet:
        path: /ready            # checks DB pool, downstream deps, queue depth
        port: 8080
      periodSeconds: 5
      failureThreshold: 2
      successThreshold: 1

Application-level vs port-level checks

A tcpSocket check is cheap and tells you the process is up. An HTTP check on a real path tells you the process is up and the framework is routing requests and the handler is returning. Port-level for liveness, application-level for readiness, is a sane default.

Load Balancing Algorithms

Consistent hashing for cache locality

If your service has a per-user cache, sending the same user to the same instance every time gives you a hot cache instead of three cold ones. Consistent hashing on user_id does that, and survives instance churn without invalidating most of the cache. Envoy and HAProxy both support it natively.

Zone affinity for cost

Cross-AZ traffic on AWS is billed at $0.01/GB each way. For a service that does 10 TB/day of internal calls, that’s $200/day in network fees alone. Zone-aware load balancing — prefer a same-AZ instance unless none is healthy — cuts that to a fraction. Istio’s locality load balancing and Linkerd’s topology-aware routing both implement this with a single config flag.

Eureka client config (Java)

// application.yml — Spring Cloud Netflix Eureka client
spring:
  application:
    name: order-service

eureka:
  client:
    serviceUrl:
      defaultZone: http://eureka-1:8761/eureka/,http://eureka-2:8761/eureka/
    registryFetchIntervalSeconds: 30     # pull registry every 30s
    instanceInfoReplicationIntervalSeconds: 30
    fetchRegistry: true
    registerWithEureka: true

  instance:
    preferIpAddress: true
    leaseRenewalIntervalInSeconds: 10      // heartbeat every 10s
    leaseExpirationDurationInSeconds: 30   // expire after 30s of silence
    metadataMap:
      version: "2.14.0"
      zone: us-east-1a

ribbon:
  NFLoadBalancerRuleClassName: com.netflix.loadbalancer.ZoneAvoidanceRule

Failure Modes

Split-brain in the registry

Strongly consistent registries (Consul, etcd, ZooKeeper) use a Raft or Paxos quorum. Lose quorum — usually because of a network partition that splits the cluster — and writes stop. New pods can’t register. Health checks can’t update state. Old reads still work, which is its own problem because you’re routing on stale data.

Eventually consistent registries (Eureka by design, Cloud Map in some modes) prefer to keep working with stale data over going read-only. That means you’ll occasionally route to a dead instance during a partition — but the failure is local, not global. Netflix chose this trade-off explicitly.

The registry as a single point of failure

Stampedes when a popular service flaps

Imagine a popular downstream — auth-service — whose health check briefly flaps from healthy to unhealthy and back. Every caller that was holding a connection drops it. Every caller that needs auth opens a new connection. The replacement instances get hit by the entire fleet’s reconnection load, half of them buckle under the spike, more flapping ensues. This is a stampede.

Defenses: stagger reconnect with jitter, cap the rate of endpoint-table updates pushed to clients, prefer slow-start load balancing (give a newly-Ready instance a couple of seconds to warm up before it gets full load), and don’t mark a downstream as unhealthy on a single failed probe — require N consecutive failures.

Real-World Examples

Netflix Eureka. The reference for client-side discovery at cloud scale. Eureka servers cluster in a peer-to-peer ring, replicate registrations eventually, and explicitly choose availability over consistency. Eureka’s self-preservation mode is the famous quirk: if too many heartbeats stop arriving (network partition vs. mass instance death), it stops expiring registrations rather than risk emptying the registry. Eureka is in maintenance now; the patterns — client-side LB with Ribbon, hystrix circuit breakers in front — live on in Spring Cloud and Resilience4j.

Airbnb’s SmartStack (deprecated). A clever pre-mesh design: Nerve registered services to ZooKeeper, and Synapse on each host watched ZooKeeper and rewrote local HAProxy configs. Apps talked to localhost:port via HAProxy. It was a sidecar pattern before sidecars had a name. Airbnb retired it as Kubernetes and Envoy made it redundant — but it’s the clearest historical example of why the mesh model won.

HashiCorp Consul at scale. Consul scales by federating: each datacenter runs its own Consul cluster, and they gossip. Cross-DC service lookups are first-class. Big shops — Roblox, Cruise, Cloudflare — run Consul as the source of truth for tens of thousands of services across many datacenters and clouds. Consul Connect adds mTLS and an Envoy sidecar, turning Consul into a full mesh.

Kubernetes’ built-in discovery. The default for new systems. The kubelet, kube-proxy, and CoreDNS together implement registration, health, lookup, and load balancing without any third-party software. Service of type ClusterIP gets a stable virtual IP backed by iptables/IPVS rules; CoreDNS resolves the service name to that VIP; kube-proxy round-robins to the current pod IPs. It’s not the most flexible system, but for “I just want service A to call service B”, it’s zero-friction.

AWS Cloud Map + ECS Service Discovery. The managed-AWS equivalent. ECS tasks auto-register into a Cloud Map namespace; Cloud Map keeps Route 53 records (or instance attributes for an API-based lookup) in sync. Pairs naturally with App Mesh or Service Connect when you want sidecar-style discovery without running Istio yourself.

Best Practices