Service Mesh

Why Service Mesh Matters

Without a mesh, every team writes their own version of these features:

Notice the pattern: the mesh doesn’t add capabilities — you could already do all of this. It moves them into a layer where they’re uniform, observable, and changeable without redeploying application code. That’s the whole pitch.

The Sidecar Pattern

The sidecar is the defining shape of a mesh. Every pod that joins the mesh runs two containers: the application, and a proxy (Envoy in Istio, linkerd2-proxy in Linkerd). All inbound and outbound TCP traffic is transparently redirected to the proxy — usually via iptables rules installed by an init container, or via eBPF in newer Cilium-based meshes.

Sidecar injection

You don’t add the proxy by hand. You annotate a namespace and the mesh’s mutating admission webhook injects the sidecar into every new pod:

# Namespace opt-in: every pod created here gets an Envoy sidecar.
apiVersion: v1
kind: Namespace
metadata:
  name: orders
  labels:
    istio-injection: enabled
---
# Per-pod opt-out (sometimes you really don’t want a sidecar — e.g. a job).
apiVersion: v1
kind: Pod
metadata:
  name: legacy-batch
  namespace: orders
  annotations:
    sidecar.istio.io/inject: "false"

The webhook rewrites the pod spec on its way into etcd. After admission, the pod has two containers, an init container that programmed iptables, and the application is none the wiser.

Control Plane vs Data Plane

The split matters operationally: if Istiod is unhealthy for five minutes, existing connections keep flowing because Envoys are running with their last-known-good config. New pods can’t join the mesh and policy changes don’t propagate, but live traffic keeps moving. That graceful-degradation property is a deliberate design choice, and it’s why the data plane is the part that has to be bulletproof.

SPIFFE and workload identity

Identity in a modern mesh is not “the IP address” or “a JWT we hope nobody steals.” It’s a SPIFFE ID — a URI like spiffe://cluster.local/ns/orders/sa/orders-app — baked into the X.509 certificate that the control plane mints for every workload. The cert is short-lived (24h is typical) and rotated automatically. Authorization decisions reference the SPIFFE ID, not network coordinates.

What a Mesh Gives You for Free

“For free” is a stretch — the mesh is not free, see the cost section — but for the application code, all of these stop being your problem:

The reason this list is the pitch: every item on it is something every distributed systems team eventually builds, badly, three times, in three different languages. The mesh charges you operational complexity in exchange for never building any of it again.

Istio Deep Dive

VirtualService: routing rules

A VirtualService defines what happens to traffic addressed to a host. Here’s a 90/10 weighted split — the canonical way to do a canary release without touching application code:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: payments
  namespace: payments
spec:
  hosts:
    - payments.payments.svc.cluster.local
  http:
    - match:
        - headers:
            x-canary:
              exact: "true"
      route:
        - destination:
            host: payments
            subset: v2           # forced to v2 if header present
    - route:
        - destination:
            host: payments
            subset: v1
          weight: 90            # 90% of normal traffic to stable
        - destination:
            host: payments
            subset: v2
          weight: 10            # 10% to canary
      timeout: 3s
      retries:
        attempts: 2
        perTryTimeout: 1s
        retryOn: 5xx,reset,connect-failure

DestinationRule: subsets and circuit breaking

VirtualService says where traffic goes; DestinationRule says how the proxy talks to it. This is also where you configure the sidecar’s circuit breaker (Envoy calls it “outlier detection”):

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: payments
  namespace: payments
spec:
  host: payments.payments.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http2MaxRequests: 1000
        maxRequestsPerConnection: 10
    outlierDetection:               # Envoy’s circuit breaker
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50       # never eject >50% of endpoints
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2

That single CRD gives every caller of payments identical pool sizing and identical eject-the-bad-pod behavior. No SDK, no language-specific config — the sidecar enforces it on every connection.

PeerAuthentication: enforce mTLS

# Require mTLS for every service in the payments namespace.
# Plain-text traffic from non-meshed callers is rejected.
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: payments
spec:
  mtls:
    mode: STRICT

AuthorizationPolicy: who can call what

# Only the checkout service account may POST to /charge on payments.
# Identity is the SPIFFE URI, not an IP or a JWT.
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: payments-charge
  namespace: payments
spec:
  selector:
    matchLabels:
      app: payments
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - "cluster.local/ns/checkout/sa/checkout-app"
      to:
        - operation:
            methods: ["POST"]
            paths: ["/charge"]

Gateway: north-south entry into the mesh

# Terminate TLS for api.example.com at the edge and route into the mesh.
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: public
  namespace: edge
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      tls:
        mode: SIMPLE
        credentialName: api-example-com-cert
      hosts:
        - "api.example.com"

Linkerd as Alternative

Linkerd uses the SMI (Service Mesh Interface) and now Gateway API GAMMA for traffic management. A Linkerd traffic split for the same canary scenario looks like this:

# Linkerd canary with the SMI TrafficSplit CRD — 90/10.
apiVersion: split.smi-spec.io/v1alpha1
kind: TrafficSplit
metadata:
  name: payments-rollout
  namespace: payments
spec:
  service: payments           # the apex service callers hit
  backends:
    - service: payments-v1
      weight: 90
    - service: payments-v2
      weight: 10

That’s the entire weighted-split story in Linkerd. No DestinationRule, no subsets — just two backing Services and a weight. The trade-off is real: less knob surface means less to misconfigure, but also less to express when you actually need header-based routing or sticky sessions.

Mesh vs Gateway

The overlap is real and growing. Modern gateways (Envoy Gateway, Kong, Gloo) and modern meshes both speak Gateway API and both can do retries and TLS. But the gateway lives at the edge and treats your cluster as a black box; the mesh lives inside the cluster and treats every workload as a first-class participant. The right architecture in 2026 is gateway-in-front-of-mesh, where the gateway hands traffic to a mesh ingress and the mesh takes it from there.

The Real Cost

When NOT to adopt a mesh

• You have fewer than ~20 services and one language. An HTTP client library does the job.
• Your latency budget is single-digit milliseconds end-to-end — the proxy hop will eat it.
• You don’t have a platform team that owns the mesh as a product. Without an owner the mesh becomes everyone’s problem and nobody’s.
• You’re running on bare VMs without Kubernetes. The mesh value drops sharply outside an orchestrator.
• You’re still figuring out service boundaries. A mesh on top of an unstable architecture amplifies confusion.

The clearest signal that you do want a mesh: you have multiple language stacks, mTLS is a compliance requirement, and your platform team is already begging engineers to standardize their HTTP clients. At that point the mesh stops being a tax and starts being the cheapest thing on the menu.

Real-World Examples

Lyft built Envoy in 2016 because they had hundreds of services across Python, Go, and Java and could not maintain N versions of resilience logic. Envoy was donated to the CNCF and became the data plane for Istio, AWS App Mesh, Consul Connect, Gloo, and most modern API gateways. Every time you read about a service mesh in 2026, the proxy is almost always Envoy.

Google ran an internal mesh for years before Istio existed — the architecture seeded Istio when Google open-sourced it with IBM and Lyft in 2017. Google’s internal experience is the reason the control-plane / data-plane split is so disciplined: at their scale, you cannot afford a control-plane failure to take down the data plane.

eBay migrated their service-to-service traffic onto a mesh to standardize observability across their multi-language estate — the win they kept emphasizing in talks was not features, it was uniformity. Every team got the same dashboards without negotiating it.

Salesforce uses a mesh to enforce mTLS and zero-trust between thousands of internal services in regulated environments. The compliance story — “every internal call is mutually authenticated and encrypted, with rotating short-lived certs” — is much easier to make to an auditor when one platform says it for everyone.

Airbnb’s service-mesh adoption story is the cautionary one: they moved incrementally, started with observability before policy, and were public about how much engineering effort the rollout took. The lesson they shared in talks: budget more than you think for control-plane operations and treat mesh upgrades as first-class platform work.

Cilium service mesh is the eBPF-native variant: instead of an Envoy sidecar per pod, a per-node eBPF datapath does L4 mTLS in the kernel and a shared Envoy handles L7 only when you need it. The operational pitch is identical to Istio Ambient — lower per-pod overhead, no app restarts to upgrade — and it’s where the mesh world is converging.

Best Practices