Field guide · zero-trust microservices
Smart endpoints,
dumb pipes.
You have a lock. You don’t fully trust it — so you add a lock for the lock, and repeat. That’s Zero Trust. This is how the gateway, the service mesh, and your app split the work with zero duplicated code, and how mTLS turns the network itself into the un-bypassable second lock.
the ideaDefense in depth. No single control is trusted to hold. An attacker who picks the first lock (the gateway) runs into a second (the mesh), then a third (your app). Each layer re-proves the request — unlocking one never unlocks the next.
01The three-layer lock
A request crosses three checkpoints. The gateway and mesh do fast, generic checks (coarse-grained); only your app can answer “is this row yours?” (fine-grained). Tap a stop to see its job — and what the wire carries.
02The clean boundary — what goes in the pipe
To avoid redundancy, push crypto + identity into the pipe so the app never re-checks them. The trap is pushing everything in: the pipe can’t see your database, so it can’t answer ownership — and the app becomes a confused deputy. Flip the toggle.
The pipe gateway + mesh
The app smart endpoint
- Does THIS user own THIS invoice?
- Has the user paid before download?
- Return 403 if the row isn’t yours
the ruleThe pipe verifies identity ("is this a valid employee?"). The app verifies relationship ("may this employee see this customer’s invoice?"). Zero overlap, zero redundancy.
03Header trust vs the double lock
Trust the injected header (fast, tiny app code) or re-parse the token everywhere (paranoid, heavy)? You don’t have to choose — mTLS plus localhost binding gives Zero Trust isolation with zero app crypto.
mTLS + localhost the compromise — get both
- Zero Trust network isolation AND zero app code
- The app rejects any peer without a mesh-signed cert
- Lock moves from the CPU to the network card
04The one trap: “Blind Trust”
Once the app blindly trusts X-User-Id, anyone who reaches its port can forge it. The fix is to make sure nobody can bypass the mesh to talk to the app directly. Turn mTLS off, then run the attack.
curl -H "X-User-Id: admin":8443 (mTLS)defendedThe app demands a mesh-signed client certificate before reading a single header. The attacker has no such cert, so the TLS handshake fails and the connection is torn down before one byte of HTTP is parsed — the forged X-User-Id header is never even read.
05How mTLS works — the handshake
The mesh is a Certificate Authority. Two peers swap certificates, then pass two checks: ownership (a challenge-response) and authority (a CA signature). Both must pass — which is why a man-in-the-middle can’t fake it.
Here is my certificate (public)Both sides send their CA-signed certificate. Certificates are public — a MITM can even copy them off the wire. That alone proves nothing yet.
06The chain of trust — issued, then checked
The whole story in one picture, because the two halves only make sense together. Phase ①: a pod is born, is handed ca.crt, mints its own key pair, sends a CSR, and gets back a certificate — its public key plus a signature made with the mesh private key. Phase ②: two pods swap those certificates and each checks the signature inside the other’s using the ca.crt it already had. Step through both.
1A pod is created
Kubernetes schedules the pod. Before your code boots, the mesh hands it one public file: ca.crt — the CA’s own certificate. That is the only thing the pod needs in order to check anybody else later.
app A → sidecar A ⇄ mTLS ⇄ sidecar B → app B. The mTLS terminates sidecar ↔ sidecar, and the certificate belongs to the pod — which is exactly why the signature check happens inside the pod, against a ca.crt that is already local. No round-trip to the control plane, so verification cannot be a bottleneck.One caveat worth knowing: this is the sidecar model (Istio classic, Linkerd). In Istio’s newer ambient mode there is no sidecar — a per-node agent terminates mTLS instead, so the handshake happens on the node rather than inside your pod.
The same five issuance beats, with the file each one produces:
CA signs it · the mesh (CA)
The control plane checks the pod’s identity, then stamps the CSR with its master private key.
pod.crt (signed)07The file zoo — .key · .csr · .crt · .pub
Four look-alike files, four different jobs. One is a secret you guard with your life; the rest are safe to hand out. Tap a card.
08Anatomy of a certificate
A certificate is a passport: identity fields (CN, O, OU, C), the public key, and the CA’s signature over all of it. Tap a field.
CN — Common Name — the app’s identity. The mesh matches on this to decide which service is allowed to call the payment-service.
09Why pod-level, not service-level
A Kubernetes Service is a routing rule — a ghost with no CPU or memory to hold a key. And a shared key means one breached pod compromises them all. TLS is point-to-point, so the certificate lives in the pod.
Kubernetes Service
A DNS name + an iptables rule in the kernel. No process, no memory — it just points traffic at a real pod. It can’t hold a key or run a handshake.
The pod
A real container with CPU + memory. It holds its own private key, does the crypto, and if breached only that one identity is lost — the CA revokes it without touching the rest.
owns the identitywhy not shareOne key per Service means every pod shares one private key — breach one, spoof all. Per-pod keys contain the blast radius.
10Does it scale? 10 pods or 10,000
Pods never store each other’s certificates — each holds only ca.crt and verifies peers with local math. Per-pod memory stays flat; only the CA’s work grows. Drag the slider.
thundering herdDeploy 100 pods at once and they all flood the CA with signing requests. Fast Go/C++ control planes sign thousands/sec; scale them out to share the load.
memory hogThe real cost isn’t certs — it’s routing config. Sidecar egress scoping tells each proxy only the handful of services it actually calls, keeping memory tiny.
11Kong instead of a mesh
No mesh? The gateway can be the CA. Same cryptographic guarantee, simpler ops — a static shared cert instead of per-pod rotation. Often the right call for medium systems.
Service mesh
- Dynamic — a control plane issues a unique cert per pod
- Auto-rotated every ~24h
- Great past hundreds of services; costs CPU + RAM
Kong as the CA
- Static — one shared client cert for all backends
- You rotate it on a schedule (or via cert-manager)
- Perfect for medium systems — a full mesh is overkill
// Node.js app enforcing mTLS with Kong — no mesh required
const https = require('https')
const fs = require('fs')
https.createServer({
key: fs.readFileSync('/etc/tls/pod.key'), // the pod's own key
cert: fs.readFileSync('/etc/tls/pod.crt'), // + its cert
ca: fs.readFileSync('/etc/tls/ca.crt'), // the CA's cert (holds its public key)
requestCert: true, // demand Kong's client certificate
rejectUnauthorized: true, // drop anyone the ca.crt can't vouch for
}, (req, res) => {
// Reached here ⇒ traffic is cryptographically proven to be Kong.
const userId = req.headers['x-user-id'] // now safe to trust
res.end(`Hello ${userId}`)
}).listen(8443)same guaranteeThe cryptographic math is identical to a mesh: Kong holds the client cert, the app verifies it against ca.crt. A hacker hitting the app directly, without Kong’s cert, is dropped.
12Try it — OpenSSL in five commands
The mesh’s magic is just standard cryptography. Here’s the whole chain of trust — CA key, CA cert, pod key + CSR, sign, verify — with the exact commands. Step through them.
$ openssl genrsa -out ca.key 4096$ openssl req -x509 -new -key ca.key \
-subj "/O=Pangaea/CN=Mesh-Root-CA" -out ca.crt$ openssl genrsa -out pod.key 2048
openssl req -new -key pod.key \
-subj "/O=Pangaea/OU=Platform/CN=order-service.internal" \
-out pod.csr$ openssl x509 -req -in pod.csr \
-CA ca.crt -CAkey ca.key -days 1 -out pod.crtSignature ok subject=/O=Pangaea/OU=Platform/CN=order-service.internal → pod.crt (the signed passport)
The CA signs the CSR with ca.key — turning it into pod.crt. This is the exact step the mesh control plane runs for every pod, automatically, at birth.
13HTTPS is mTLS with one lock
The padlock in your browser is the same handshake — minus one direction. The server proves itself via public root CAs; you prove yourself with a login. Flip the mode.
the mirrorHTTPS is mTLS with one lock removed. The internet can’t issue a cert to 8 billion browsers, so the server proves itself (via public root CAs) and you prove yourself with a login instead.
·Glossary
- Zero Trust
- A security model that trusts nothing by default — every request re-proves who it is at every hop, so unlocking one layer never grants the next. "A lock for a lock, repeat."
- Defense in depth
- Stacking independent controls so no single failure is fatal. If an attacker picks the first lock (the gateway) they hit a second (the mesh) and a third (the app).
- Smart endpoints, dumb pipes
- A microservices principle: keep business logic in the services (the smart endpoints) and let the network layer (the pipes) do only routing + generic security — never app-specific rules.
- API gateway
- The single front door at the edge. It validates the JWT, rate-limits, blocks bad IPs, and strips client-sent identity headers before any traffic enters your network. Kong is a popular one.
- Service mesh
- An infrastructure layer that puts a sidecar proxy next to every service to handle mTLS, retries, and traffic policy — without touching your app code. Istio, Linkerd, Consul.
- Sidecar
- A proxy container (usually Envoy) that runs beside your app in the same pod and intercepts all its network traffic — the mesh’s worker on the ground.
- Coarse-grained vs fine-grained
- Coarse-grained checks are global (is the token valid? does it have the role?) and belong in the pipe. Fine-grained checks are per-object (does THIS user own THIS invoice?) and only the app can do them.
- AuthN vs AuthZ
- Authentication (AuthN) = who are you? Authorization (AuthZ) = what are you allowed to do? The pipe handles AuthN + coarse AuthZ; the app owns fine-grained AuthZ.
- JWT
- A signed token carrying a user’s identity + claims. The gateway/mesh verifies its signature with the issuer’s public key, so the app can trust its contents without re-parsing it.
- Confused deputy
- When a trusted component (your app) is tricked into misusing its authority because it blindly trusted an upstream (the gateway) that was misconfigured — e.g. a wildcard route leaking an admin path.
- Header injection
- An attacker sending a forged identity header (like X-User-Id: admin) directly to an app that blindly trusts it. Fixed by stripping the header at the edge and only accepting it from the sidecar.
- mTLS
- Mutual TLS — both sides of a connection present a certificate and prove they own its private key. It turns the network itself into a lock: an app refuses any peer without a mesh-signed certificate.
- TLS
- Transport Layer Security — the encryption behind HTTPS. Standard (one-way) TLS verifies only the server; mTLS verifies both ends.
- Certificate Authority
- The trusted issuer. It holds the master private key and is the only thing that can sign certificates. In a mesh it’s the control plane (istiod); on the public internet it’s DigiCert / Let’s Encrypt.
- ca.crt
- The CA’s own certificate, shared with every pod — it carries the CA’s public key (not a bare key file; see .pub). It contains no secret, so it’s safe to distribute — pods use it to verify any certificate the CA signed, with pure local math (no network call).
- Private key
- The secret half of a key pair — never leaves its container. It creates signatures and decrypts. The CA’s private key is the single most guarded secret: it’s the only thing that can mint valid certificates.
- Public key
- The shareable half of a key pair. Anyone can hold it; it verifies signatures made by the matching private key and encrypts data only that private key can read.
- Digital certificate
- A public key + an identity (CN) wrapped with the CA’s signature — like a passport. It proves not just "I own this key" but "a trusted authority vouches this key is order-service".
- CSR
- Certificate Signing Request — the application form a pod sends to the CA: its identity + its public key + a self-signature proving it owns the matching private key. The CA turns it into a .crt.
- X.509
- The standard structure for a certificate — the fields (Subject, Public Key, Issuer, Validity, Signature) baked into one Base64 block between BEGIN/END CERTIFICATE markers.
- Distinguished Name
- The identity fields in a certificate. CN (Common Name) = the app identity, e.g. order-service.internal; O = organization; OU = team; C = two-letter country. The mesh matches on CN to authorize peers.
- Challenge-response
- The proof-of-ownership step of a handshake: "sign this random string with your private key." (Sign, not encrypt — a private key makes signatures; the public half verifies them.) Only the true key owner can, so a thief who copied a public certificate off the wire fails instantly.
- Man-in-the-middle
- An attacker intercepting traffic between two parties. mTLS defeats it: certificates are public and can be stolen, but without the matching private key and a CA signature, a forged one is rejected.
- Thundering herd
- When many pods spin up at once and flood the CA with signing requests simultaneously. Solved by fast multi-threaded control planes that can be scaled horizontally.
- Sidecar egress scoping
- Telling the mesh which services a pod actually talks to, so the control plane only pushes that slice of routing config — keeping each sidecar’s memory tiny instead of mapping the whole cluster.
- .pem / .key
- PEM is the Base64 text envelope (BEGIN/END markers) that holds keys and certs. A .key (or private .pem) is the secret; a .crt is often PEM-encoded too.
- .pub
- An SSH public key — a raw one-line key with no CA, no CN, no signature. You paste it into GitHub and it’s blindly trusted. Different family from mTLS certificates (.crt), which are CA-signed.
- ACME
- The protocol Let’s Encrypt uses to automate public certificates: prove you control a domain (serve a token over HTTP/DNS), then get a 90-day cert that a background job auto-renews.
- Root CA
- The top of the public trust chain — pre-installed in every browser and OS. Sites don’t use it directly; a chain of intermediate CAs signs the actual leaf certificate (how Cloudflare issues millions).
- Localhost binding
- Configuring an app to listen only on 127.0.0.1 so external traffic physically cannot reach it — it must pass through the local sidecar first. A network-level double lock with zero app code.