Service Communication

Why Communication Patterns Matter

This tutorial walks the four shapes that cover essentially everything you’ll build: REST, gRPC, GraphQL, and asynchronous messaging. For each one we’ll cover when to reach for it, what it costs, and the production gotchas that hurt the most.

Synchronous vs Asynchronous

Before the protocol choice, the bigger choice: do you need an answer now, or do you need the work to eventually happen? That’s the only question that matters at this layer.

The most common mistake is treating these as a strict dichotomy. Most production paths are both: a synchronous request returns “accepted, here’s your tracking ID,” and the heavy work happens asynchronously behind it.

REST: The Workhorse

REST is what you should choose when in doubt. It is also what you should choose when partners or third-party clients will use the API — OpenAPI is the lingua franca for documentation, mock servers, and SDK generation.

Resource design that actually scales

• Nouns, not verbs. POST /orders, not POST /createOrder. The HTTP verb is the verb.
• Plural collection, singular item. /orders for the list, /orders/{id} for one.
• Idempotent verbs do what they say. GET, PUT, DELETE are safe to retry; POST is not unless you ship an idempotency key.
• Versioning lives in the URL or the Accept header. /v1/orders is uglier but unambiguous; partners always get it right.
• Pagination is mandatory. Cursor-based for streams, offset-based for tabular — never “just return everything.”

# Flask example: a small but well-shaped REST resource.
from flask import Flask, jsonify, request, abort

app = Flask(__name__)

products = {
    1: {"id": 1, "name": "Laptop", "price": 999.99, "stock": 50},
    2: {"id": 2, "name": "Mouse",  "price": 29.99,  "stock": 200},
}

@app.route("/v1/products", methods=["GET"])
def list_products():
    # Pagination — never return unbounded lists.
    limit  = min(int(request.args.get("limit", 50)), 200)
    cursor = int(request.args.get("cursor", 0))
    items  = [p for p in products.values() if p["id"] > cursor][:limit]
    next_cursor = items[-1]["id"] if items else None
    return jsonify({"items": items, "next_cursor": next_cursor})

@app.route("/v1/products/<int:product_id>", methods=["GET"])
def get_product(product_id):
    product = products.get(product_id)
    if not product:
        # RFC 7807 problem+json — see the error handling section below.
        return jsonify({
            "type": "https://errors.example.com/not-found",
            "title": "Product not found",
            "status": 404,
            "instance": f"/v1/products/{product_id}",
        }), 404
    return jsonify(product)

@app.route("/v1/products", methods=["POST"])
def create_product():
    # Idempotency key makes POST safe to retry.
    idem = request.headers.get("Idempotency-Key")
    if not idem:
        abort(400, "Idempotency-Key header required")
    if idem in idempotency_store:
        return idempotency_store[idem]   # replay the exact same response

    data = request.get_json()
    new_id = max(products) + 1
    products[new_id] = {"id": new_id, **data}
    response = (jsonify(products[new_id]), 201)
    idempotency_store[idem] = response
    return response

{
    "type": "https://errors.example.com/insufficient-stock",
    "title": "Insufficient stock",
    "status": 409,
    "detail": "Requested 50 units of SKU-1234; only 12 available.",
    "instance": "/v1/orders/abc-123",
    "available_stock": 12
}

The OpenAPI contract is your real API

Hand-written API docs lie within the week. An OpenAPI (formerly Swagger) spec is the only documentation that stays honest, because every other artifact — mock servers, SDKs, contract tests, gateway routing — is generated from it. Treat the spec as code: review it in PRs, version it, and break the build when handlers don’t match it.

gRPC and Protobuf

gRPC pays off when you have many internal services calling each other a lot. The wins are concrete: smaller payloads (binary encoding, no field names), lower CPU (fast codegen-based serialization), real streaming (server, client, and bidirectional), and a typed contract that catches mismatches at compile time instead of in production.

The contract: a .proto file

// product.proto — this file IS the API.
syntax = "proto3";

package ecommerce.v1;

service ProductService {
    rpc GetProduct (ProductRequest) returns (ProductResponse);
    rpc ListProducts (ListProductsRequest) returns (stream ProductResponse);
    rpc CreateProduct (CreateProductRequest) returns (ProductResponse);
}

message ProductRequest {
    int32 product_id = 1;
}

message ListProductsRequest {
    int32 page_size = 1;
    string page_token = 2;
}

message CreateProductRequest {
    string name = 1;
    double price = 2;
    int32 stock = 3;
}

message ProductResponse {
    int32 id = 1;
    string name = 2;
    double price = 3;
    int32 stock = 4;
}

From this one file you generate server stubs and client libraries in Go, Python, Java, Kotlin, Swift, TypeScript, C#, Rust — whatever you need. The fields are wire-tagged by number (= 1, = 2), which is what makes Protobuf forward- and backward-compatible: never reuse a field number, never change its type, and your old clients keep working forever.

Server in Python

import grpc
from concurrent import futures
import product_pb2, product_pb2_grpc

class ProductServicer(product_pb2_grpc.ProductServiceServicer):
    def __init__(self):
        self.products = {1: {"id": 1, "name": "Laptop", "price": 999.99, "stock": 50}}

    def GetProduct(self, request, context):
        product = self.products.get(request.product_id)
        if not product:
            context.set_code(grpc.StatusCode.NOT_FOUND)
            context.set_details("Product not found")
            return product_pb2.ProductResponse()
        return product_pb2.ProductResponse(**product)

    def ListProducts(self, request, context):
        # Server-side streaming — yield each product as it’s available.
        for p in self.products.values():
            yield product_pb2.ProductResponse(**p)

def serve():
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
    product_pb2_grpc.add_ProductServiceServicer_to_server(ProductServicer(), server)
    server.add_insecure_port("[::]:50051")
    server.start()
    server.wait_for_termination()

GraphQL

GraphQL solves the over-fetching and under-fetching problem that REST has when many client form factors share the same backend. It is also a strong fit for API aggregation — one query that pulls together data from N microservices — via federation (Apollo Federation, GraphQL Mesh).

The schema

type Product {
    id: ID!
    name: String!
    price: Float!
    stock: Int!
    reviews: [Review!]!
}

type Review {
    id: ID!
    rating: Int!
    comment: String
    user: User!
}

type User {
    id: ID!
    name: String!
    email: String!
}

type Query {
    product(id: ID!): Product
    products(limit: Int = 20): [Product!]!
}

type Mutation {
    createProduct(name: String!, price: Float!, stock: Int!): Product!
}

The query and the response have the same shape

# Client query — ask for exactly the fields you need.
query {
    product(id: "1") {
        name
        price
        reviews {
            rating
            user { name }
        }
    }
}

# Response — mirror image of the query.
{
    "data": {
        "product": {
            "name": "Laptop",
            "price": 999.99,
            "reviews": [
                {"rating": 5, "user": {"name": "Jane"}}
            ]
        }
    }
}

If you have one or two clients hitting one backend, GraphQL is overkill. If you have a dozen client apps consuming a hundred microservices, GraphQL (federated) is often the only sane way to keep the API surface coherent.

Asynchronous Messaging

Two broad shapes dominate, and they’re not interchangeable:

For deep coverage of broker selection, queue patterns (work queue, pub/sub, routing, topics), and consumer group semantics, see Messaging Patterns. For event-driven architecture as a system shape (event sourcing, CQRS, sagas), see Event-Driven Architecture. The summary below is the part you need to choose between sync and async at the call-site level.

RabbitMQ producer & consumer (the “work queue” shape)

import pika, json

# Producer — publish a durable message and walk away.
conn = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
ch = conn.channel()
ch.queue_declare(queue="order_processing", durable=True)

ch.basic_publish(
    exchange="",
    routing_key="order_processing",
    body=json.dumps({"order_id": "ORD-12345", "total": 1999.98}),
    properties=pika.BasicProperties(delivery_mode=2),  # persistent
)

# Consumer — one job at a time, ack on success, requeue on failure.
def handle(ch, method, props, body):
    order = json.loads(body)
    try:
        process_order(order)
        ch.basic_ack(delivery_tag=method.delivery_tag)
    except Exception:
        ch.basic_nack(delivery_tag=method.delivery_tag, requeue=True)

ch.basic_qos(prefetch_count=1)         # fair dispatch
ch.basic_consume(queue="order_processing", on_message_callback=handle)
ch.start_consuming()

Kafka producer & consumer (the “event log” shape)

from kafka import KafkaProducer, KafkaConsumer
import json

producer = KafkaProducer(
    bootstrap_servers=["localhost:9092"],
    value_serializer=lambda v: json.dumps(v).encode(),
    key_serializer=lambda k: k.encode() if k else None,
)

# Key determines partition — ordering is per-key, not global.
producer.send(
    "order-events",
    key="ORD-12345",
    value={"event_type": "OrderCreated", "order_id": "ORD-12345", "total": 1999.98},
)
producer.flush()

# A consumer group reads the topic; many groups can read the same topic independently.
consumer = KafkaConsumer(
    "order-events",
    bootstrap_servers=["localhost:9092"],
    group_id="billing-service",
    value_deserializer=lambda m: json.loads(m.decode()),
    auto_offset_reset="earliest",
)
for msg in consumer:
    handle_event(msg.value)

Communication Reliability

This section is a fast tour. For the full treatment of circuit breakers, retries, bulkheads, and chaos engineering, see Circuit Breaker & Resilience.

Timeouts: pick numbers and write them down

Every outbound call needs an explicit timeout. The default in most HTTP clients is “forever,” which is exactly the timeout that turns a slow downstream into a cascading outage.

The timeout for any service should be shorter than the timeout of whoever calls it. Otherwise the upstream gives up first and the work it asked for keeps running anyway — pure waste.

Retry with exponential backoff and jitter

import random, time
from typing import Callable, TypeVar

T = TypeVar("T")

def retry_with_backoff(
    fn: Callable[[], T],
    max_attempts: int = 5,
    base_delay: float = 0.2,
    max_delay: float = 5.0,
    retryable: tuple = (TimeoutError, ConnectionError),
) -> T:
    last = None
    for attempt in range(max_attempts):
        try:
            return fn()
        except retryable as e:
            last = e
            if attempt == max_attempts - 1:
                break
            # Full jitter: pick a random delay in [0, exp_backoff)
            backoff = min(max_delay, base_delay * 2 ** attempt)
            time.sleep(random.uniform(0, backoff))
    raise last

Circuit breakers

Wrap every outbound call to an external dependency in a circuit breaker. After a configured failure rate is exceeded the breaker trips OPEN, and subsequent calls fail immediately instead of hanging on the timeout. This is what prevents one slow downstream from eating all of your service’s threads. See Circuit Breaker & Resilience for implementation, tuning, and observability of breakers, retries, and bulkheads.

Error Handling Across Services

Classify errors at the source

RFC 7807 problem details — the structured error body

# Content-Type: application/problem+json
{
    "type": "https://errors.example.com/insufficient-stock",
    "title": "Insufficient stock",
    "status": 409,
    "detail": "Requested 50 units of SKU-1234; only 12 available.",
    "instance": "/v1/orders/abc-123",
    "sku": "SKU-1234",
    "requested": 50,
    "available": 12,
    "correlation_id": "01HXYZ..."
}

Real-World Examples

Stripe: REST done right

Stripe’s public API is REST + JSON, with a few opinionated extensions that the industry has steadily copied:

• Idempotency keys on every write. Send Idempotency-Key: <uuid>; the server stores the response for 24 hours and replays it on retry.
• Versioning by date. The Stripe-Version: 2023-10-16 header pins each merchant to a specific API snapshot. Stripe can ship breaking changes without breaking anyone.
• Errors as types. Every error has a stable type and a code. Client SDKs branch on these, not on HTTP status alone.
• Cursor-based pagination. Every list endpoint has has_more and a starting_after cursor — consistent across the whole API.

Google: gRPC at planet scale

Google built gRPC on top of an internal RPC framework (Stubby) used for over a decade across thousands of services. Every internal Google service-to-service call is gRPC over HTTP/2 over a custom transport, with Protobuf contracts checked into a single monorepo. The .proto files are the API surface; SDKs in every language are generated by the same toolchain. This is why gRPC is opinionated about things like deadlines (context.Deadline) and metadata propagation — those are Google’s production lessons turned into protocol features.

Shopify: GraphQL Admin API

Shopify’s public Admin API is GraphQL. The reason is the surface area: tens of thousands of third-party apps, each needing different slices of merchant data — orders, customers, products, fulfillment, inventory. With REST that’s either hundreds of endpoints (each one a backwards-compat liability) or massive over-fetching. With GraphQL each app asks for exactly what it needs. Shopify enforces a query-cost limit (calculated from the query AST) so a careless app can’t blow up the database; the cost is published as part of the schema so app authors can budget against it.

Uber: a mixed RPC stack

Uber’s service mesh is a deliberate mix:

• Internal service-to-service: gRPC, with Protobuf contracts. Mostly internal, low-latency, polyglot fleets — the gRPC sweet spot.
• Mobile and partner APIs: REST + JSON, with a thin GraphQL layer (DOSA / Schemaless internally) for client aggregation.
• Event backbone: Apache Kafka, for everything from trip-state changes to ML training pipelines — trillions of messages a day.
• Background jobs: Cadence (Uber’s open-source workflow engine) for long-running multi-step orchestrations like driver onboarding and payment reconciliation.

The lesson is that real production stacks are polyglot at the protocol layer. There is no “one true API style” — pick the right shape per call.

Best Practices

Common anti-patterns