Field guide · zero-trust microservices

Smart endpoints,
dumb pipes.

You have a lock. You don’t fully trust it — so you add a lock for the lock, and repeat. That’s Zero Trust. This is how the gateway, the service mesh, and your app split the work with zero duplicated code, and how mTLS turns the network itself into the un-bypassable second lock.

the ideaDefense in depth. No single control is trusted to hold. An attacker who picks the first lock (the gateway) runs into a second (the mesh), then a third (your app). Each layer re-proves the request — unlocking one never unlocks the next.

01The three-layer lock

A request crosses three checkpoints. The gateway and mesh do fast, generic checks (coarse-grained); only your app can answer “is this row yours?” (fine-grained). Tap a stop to see its job — and what the wire carries.

Network · coarse-grained
  • Enforce mTLS on the wire
  • Parse the JWT, inject a trusted X-User-Id
  • Hand off to the app over localhost only

on the wire X-User-Id: 12345 (verified, injected)

02The clean boundary — what goes in the pipe

To avoid redundancy, push crypto + identity into the pipe so the app never re-checks them. The trap is pushing everything in: the pipe can’t see your database, so it can’t answer ownership — and the app becomes a confused deputy. Flip the toggle.

The pipe gateway + mesh

  • Verify the JWT signature
  • Rate-limit + block bad IPs
  • Terminate TLS / enforce mTLS
  • Check the "editor" role/scope

The app smart endpoint

  • Does THIS user own THIS invoice?
  • Has the user paid before download?
  • Return 403 if the row isn’t yours

the ruleThe pipe verifies identity ("is this a valid employee?"). The app verifies relationship ("may this employee see this customer’s invoice?"). Zero overlap, zero redundancy.

03Header trust vs the double lock

Trust the injected header (fast, tiny app code) or re-parse the token everywhere (paranoid, heavy)? You don’t have to choose — mTLS plus localhost binding gives Zero Trust isolation with zero app crypto.

mTLS + localhost the compromise — get both

gains
  • Zero Trust network isolation AND zero app code
  • The app rejects any peer without a mesh-signed cert
  • Lock moves from the CPU to the network card
costs

04The one trap: “Blind Trust”

Once the app blindly trusts X-User-Id, anyone who reaches its port can forge it. The fix is to make sure nobody can bypass the mesh to talk to the app directly. Turn mTLS off, then run the attack.

attacker podcurl -H "X-User-Id: admin"
your app:8443 (mTLS)

defendedThe app demands a mesh-signed client certificate before reading a single header. The attacker has no such cert, so the TLS handshake fails and the connection is torn down before one byte of HTTP is parsed — the forged X-User-Id header is never even read.

05How mTLS works — the handshake

The mesh is a Certificate Authority. Two peers swap certificates, then pass two checks: ownership (a challenge-response) and authority (a CA signature). Both must pass — which is why a man-in-the-middle can’t fake it.

mesh sidecarEnvoy
1 / 4
Here is my certificate (public)
Exchange certificates
your app:8443

Both sides send their CA-signed certificate. Certificates are public — a MITM can even copy them off the wire. That alone proves nothing yet.

① ownership do you hold the private key?② authority did our CA sign it?

06The chain of trust — issued, then checked

The whole story in one picture, because the two halves only make sense together. Phase ①: a pod is born, is handed ca.crt, mints its own key pair, sends a CSR, and gets back a certificate — its public key plus a signature made with the mesh private key. Phase ②: two pods swap those certificates and each checks the signature inside the other’s using the ca.crt it already had. Step through both.

mesh CA · control planeca.key 🔒ca.crtthe one secret · never leavespod · order-servicepod.key 🔒pod.pubminted in memory · stays inside① ca.crt — on “pod created”ca.crt stored in the pod③ CSR = pod.pub + CNa temporary form — discarded once signed④ sign with ca.key⑤ pod.crtpod.pub + CN=order-serviceSIG ← signed by ca.key

1A pod is created

Kubernetes schedules the pod. Before your code boots, the mesh hands it one public file: ca.crt — the CA’s own certificate. That is the only thing the pod needs in order to check anybody else later.

Where the traffic actually goes. Yes — pod-to-pod traffic is transparently redirected through the sidecars (an init container installs the iptables rules), so the path is app A → sidecar A ⇄ mTLS ⇄ sidecar B → app B. The mTLS terminates sidecar ↔ sidecar, and the certificate belongs to the pod — which is exactly why the signature check happens inside the pod, against a ca.crt that is already local. No round-trip to the control plane, so verification cannot be a bottleneck.

One caveat worth knowing: this is the sidecar model (Istio classic, Linkerd). In Istio’s newer ambient mode there is no sidecar — a per-node agent terminates mTLS instead, so the handshake happens on the node rather than inside your pod.

The same five issuance beats, with the file each one produces:

CA signs it · the mesh (CA)

The control plane checks the pod’s identity, then stamps the CSR with its master private key.

pod.crt (signed)

07The file zoo — .key · .csr · .crt · .pub

Four look-alike files, four different jobs. One is a secret you guard with your life; the rest are safe to hand out. Tap a card.

08Anatomy of a certificate

A certificate is a passport: identity fields (CN, O, OU, C), the public key, and the CA’s signature over all of it. Tap a field.

X.509 · pod.crt

CNCommon Name — the app’s identity. The mesh matches on this to decide which service is allowed to call the payment-service.

09Why pod-level, not service-level

A Kubernetes Service is a routing rule — a ghost with no CPU or memory to hold a key. And a shared key means one breached pod compromises them all. TLS is point-to-point, so the certificate lives in the pod.

Kubernetes Service

A DNS name + an iptables rule in the kernel. No process, no memory — it just points traffic at a real pod. It can’t hold a key or run a handshake.

routing only

The pod

A real container with CPU + memory. It holds its own private key, does the crypto, and if breached only that one identity is lost — the CA revokes it without touching the rest.

owns the identity

why not shareOne key per Service means every pod shares one private key — breach one, spoof all. Per-pod keys contain the blast radius.

10Does it scale? 10 pods or 10,000

Pods never store each other’s certificates — each holds only ca.crt and verifies peers with local math. Per-pod memory stays flat; only the CA’s work grows. Drag the slider.

Per-pod memory30 MB
O(1) — flat. Each pod holds only its own cert + one ca.crt, whatever the cluster size.
Control-plane CPU21%
O(N) — grows with pods. Scale the CA horizontally (3–5 istiod) to absorb it.

thundering herdDeploy 100 pods at once and they all flood the CA with signing requests. Fast Go/C++ control planes sign thousands/sec; scale them out to share the load.

memory hogThe real cost isn’t certs — it’s routing config. Sidecar egress scoping tells each proxy only the handful of services it actually calls, keeping memory tiny.

11Kong instead of a mesh

No mesh? The gateway can be the CA. Same cryptographic guarantee, simpler ops — a static shared cert instead of per-pod rotation. Often the right call for medium systems.

Service mesh

  • Dynamic — a control plane issues a unique cert per pod
  • Auto-rotated every ~24h
  • Great past hundreds of services; costs CPU + RAM

Kong as the CA

  • Static — one shared client cert for all backends
  • You rotate it on a schedule (or via cert-manager)
  • Perfect for medium systems — a full mesh is overkill
// Node.js app enforcing mTLS with Kong — no mesh required
const https = require('https')
const fs = require('fs')

https.createServer({
  key:  fs.readFileSync('/etc/tls/pod.key'),   // the pod's own key
  cert: fs.readFileSync('/etc/tls/pod.crt'),   // + its cert
  ca:   fs.readFileSync('/etc/tls/ca.crt'),    // the CA's cert (holds its public key)

  requestCert: true,        // demand Kong's client certificate
  rejectUnauthorized: true, // drop anyone the ca.crt can't vouch for
}, (req, res) => {
  // Reached here ⇒ traffic is cryptographically proven to be Kong.
  const userId = req.headers['x-user-id']   // now safe to trust
  res.end(`Hello ${userId}`)
}).listen(8443)

same guaranteeThe cryptographic math is identical to a mesh: Kong holds the client cert, the app verifies it against ca.crt. A hacker hitting the app directly, without Kong’s cert, is dropped.

12Try it — OpenSSL in five commands

The mesh’s magic is just standard cryptography. Here’s the whole chain of trust — CA key, CA cert, pod key + CSR, sign, verify — with the exact commands. Step through them.

~/mesh-pki
$ openssl genrsa -out ca.key 4096
$ openssl req -x509 -new -key ca.key \
  -subj "/O=Pangaea/CN=Mesh-Root-CA" -out ca.crt
$ openssl genrsa -out pod.key 2048
openssl req -new -key pod.key \
  -subj "/O=Pangaea/OU=Platform/CN=order-service.internal" \
  -out pod.csr
$ openssl x509 -req -in pod.csr \
  -CA ca.crt -CAkey ca.key -days 1 -out pod.crt
Signature ok
subject=/O=Pangaea/OU=Platform/CN=order-service.internal
→ pod.crt  (the signed passport)

The CA signs the CSR with ca.key — turning it into pod.crt. This is the exact step the mesh control plane runs for every pod, automatically, at birth.

13HTTPS is mTLS with one lock

The padlock in your browser is the same handshake — minus one direction. The server proves itself via public root CAs; you prove yourself with a login. Flip the mode.

mesh sidecarpresents a cert client cert
both verify
your apppresents a cert server cert
Who is verifiedBoth client AND server
Who is the CAYour private mesh / Kong
Trust comes fromA ca.crt shipped to each pod
Used forService-to-service inside the cluster

the mirrorHTTPS is mTLS with one lock removed. The internet can’t issue a cert to 8 billion browsers, so the server proves itself (via public root CAs) and you prove yourself with a login instead.

·Glossary

Zero Trust
A security model that trusts nothing by default — every request re-proves who it is at every hop, so unlocking one layer never grants the next. "A lock for a lock, repeat."
Defense in depth
Stacking independent controls so no single failure is fatal. If an attacker picks the first lock (the gateway) they hit a second (the mesh) and a third (the app).
Smart endpoints, dumb pipes
A microservices principle: keep business logic in the services (the smart endpoints) and let the network layer (the pipes) do only routing + generic security — never app-specific rules.
API gateway
The single front door at the edge. It validates the JWT, rate-limits, blocks bad IPs, and strips client-sent identity headers before any traffic enters your network. Kong is a popular one.
Service mesh
An infrastructure layer that puts a sidecar proxy next to every service to handle mTLS, retries, and traffic policy — without touching your app code. Istio, Linkerd, Consul.
Sidecar
A proxy container (usually Envoy) that runs beside your app in the same pod and intercepts all its network traffic — the mesh’s worker on the ground.
Coarse-grained vs fine-grained
Coarse-grained checks are global (is the token valid? does it have the role?) and belong in the pipe. Fine-grained checks are per-object (does THIS user own THIS invoice?) and only the app can do them.
AuthN vs AuthZ
Authentication (AuthN) = who are you? Authorization (AuthZ) = what are you allowed to do? The pipe handles AuthN + coarse AuthZ; the app owns fine-grained AuthZ.
JWT
A signed token carrying a user’s identity + claims. The gateway/mesh verifies its signature with the issuer’s public key, so the app can trust its contents without re-parsing it.
Confused deputy
When a trusted component (your app) is tricked into misusing its authority because it blindly trusted an upstream (the gateway) that was misconfigured — e.g. a wildcard route leaking an admin path.
Header injection
An attacker sending a forged identity header (like X-User-Id: admin) directly to an app that blindly trusts it. Fixed by stripping the header at the edge and only accepting it from the sidecar.
mTLS
Mutual TLS — both sides of a connection present a certificate and prove they own its private key. It turns the network itself into a lock: an app refuses any peer without a mesh-signed certificate.
TLS
Transport Layer Security — the encryption behind HTTPS. Standard (one-way) TLS verifies only the server; mTLS verifies both ends.
Certificate Authority
The trusted issuer. It holds the master private key and is the only thing that can sign certificates. In a mesh it’s the control plane (istiod); on the public internet it’s DigiCert / Let’s Encrypt.
ca.crt
The CA’s own certificate, shared with every pod — it carries the CA’s public key (not a bare key file; see .pub). It contains no secret, so it’s safe to distribute — pods use it to verify any certificate the CA signed, with pure local math (no network call).
Private key
The secret half of a key pair — never leaves its container. It creates signatures and decrypts. The CA’s private key is the single most guarded secret: it’s the only thing that can mint valid certificates.
Public key
The shareable half of a key pair. Anyone can hold it; it verifies signatures made by the matching private key and encrypts data only that private key can read.
Digital certificate
A public key + an identity (CN) wrapped with the CA’s signature — like a passport. It proves not just "I own this key" but "a trusted authority vouches this key is order-service".
CSR
Certificate Signing Request — the application form a pod sends to the CA: its identity + its public key + a self-signature proving it owns the matching private key. The CA turns it into a .crt.
X.509
The standard structure for a certificate — the fields (Subject, Public Key, Issuer, Validity, Signature) baked into one Base64 block between BEGIN/END CERTIFICATE markers.
Distinguished Name
The identity fields in a certificate. CN (Common Name) = the app identity, e.g. order-service.internal; O = organization; OU = team; C = two-letter country. The mesh matches on CN to authorize peers.
Challenge-response
The proof-of-ownership step of a handshake: "sign this random string with your private key." (Sign, not encrypt — a private key makes signatures; the public half verifies them.) Only the true key owner can, so a thief who copied a public certificate off the wire fails instantly.
Man-in-the-middle
An attacker intercepting traffic between two parties. mTLS defeats it: certificates are public and can be stolen, but without the matching private key and a CA signature, a forged one is rejected.
Thundering herd
When many pods spin up at once and flood the CA with signing requests simultaneously. Solved by fast multi-threaded control planes that can be scaled horizontally.
Sidecar egress scoping
Telling the mesh which services a pod actually talks to, so the control plane only pushes that slice of routing config — keeping each sidecar’s memory tiny instead of mapping the whole cluster.
.pem / .key
PEM is the Base64 text envelope (BEGIN/END markers) that holds keys and certs. A .key (or private .pem) is the secret; a .crt is often PEM-encoded too.
.pub
An SSH public key — a raw one-line key with no CA, no CN, no signature. You paste it into GitHub and it’s blindly trusted. Different family from mTLS certificates (.crt), which are CA-signed.
ACME
The protocol Let’s Encrypt uses to automate public certificates: prove you control a domain (serve a token over HTTP/DNS), then get a 90-day cert that a background job auto-renews.
Root CA
The top of the public trust chain — pre-installed in every browser and OS. Sites don’t use it directly; a chain of intermediate CAs signs the actual leaf certificate (how Cloudflare issues millions).
Localhost binding
Configuring an app to listen only on 127.0.0.1 so external traffic physically cannot reach it — it must pass through the local sidecar first. A network-level double lock with zero app code.
Built by Pangaea Labs

Illustrative walkthrough. Real service meshes (Istio, Linkerd, Consul) and Kong automate every step here; the OpenSSL commands are the same math done by hand. Certificate values shown are examples.