Guide contents

Lesson 87 of 129

Microservices Security and Zero Trust

A monolith has a small number of entry points and one process boundary. A microservices system has many services communicating over the network, so the attack surface includes every service endpoint and every network path between them. Security cannot rely on a trusted internal network.

Zero trust

Zero trust is a security model with these principles:

  • No implicit trust from network location. A request from inside the cluster or VPC is not assumed to be legitimate.
  • Authenticate and authorize every request, for both the calling workload and, where relevant, the end user.
  • Least privilege. Each identity gets only the permissions it needs.
  • Assume breach. Limit lateral movement so that a compromised service cannot freely reach others, and log activity for detection.
  • Encrypt all traffic, including internal traffic.

The older perimeter model puts strong controls at the network edge and trusts everything behind it. Once an attacker is inside (through one vulnerable service), the perimeter model gives access to everything reachable.

Identities in a microservices system

Two identities appear in a request:

Identity Question Mechanism
Workload (service) identity Which service is calling? Client certificates (mTLS), platform-issued tokens
End-user identity On whose behalf is the call made? Access tokens such as JWTs (see authentication, OAuth2, JWT, and sessions)

Both are needed for correct authorization: service A may be allowed to call service B, and the user in the request may or may not be allowed to see the requested resource.

Mutual TLS and service identity

In ordinary TLS, only the server presents a certificate, and the client verifies it. In mutual TLS (mTLS), both sides present certificates and verify each other, so each side knows the cryptographic identity of the peer. The connection is also encrypted and protected against tampering (see encryption, TLS, and attack mitigation).

Operating mTLS across many services requires automation:

  • A certificate authority (CA) issues a certificate to each workload.
  • Certificates are short-lived (hours or less) and renewed automatically, which limits the value of a stolen key and reduces reliance on revocation.
  • Trust bundles (the CA certificates that peers trust) are distributed to workloads.

A service mesh does this transparently: the control plane issues certificates to sidecar proxies, and the proxies establish mTLS between them without changes to application code.

SPIFFE

SPIFFE (Secure Production Identity Framework for Everyone) is a standard for workload identity that does not depend on network addresses or IPs, which change in dynamic environments.

  • A SPIFFE ID is a URI naming a workload, such as spiffe://example.org/ns/shop/sa/orders.
  • An SVID (SPIFFE Verifiable Identity Document) proves the identity, as an X.509 certificate or a JWT.
  • A workload API lets a workload obtain its SVIDs and trust bundles. Before issuing, the system attests the workload: it verifies properties, such as the Kubernetes namespace and service account, or the node identity, and does not rely on a pre-shared secret.
  • SPIRE is the reference implementation of SPIFFE.

Attestation solves the bootstrap problem of how a new workload first proves who it is (see the secret zero problem in configuration and secrets management).

sequenceDiagram
    participant A as Service A (orders)
    participant CA as Identity system (SPIRE / mesh CA)
    participant B as Service B (payments)
    A->>CA: attest workload, request identity
    CA-->>A: short-lived certificate (spiffe://.../orders)
    B->>CA: attest workload, request identity
    CA-->>B: short-lived certificate (spiffe://.../payments)
    A->>B: TLS handshake, both present certificates
    B->>B: verify chain, read caller identity
    B->>B: policy: may orders call payments POST /charge?
    B-->>A: allowed, response over encrypted channel

In a mesh such as Istio, both requirements are declared as configuration. The first policy requires mTLS for all workloads in a namespace. The second allows only the orders service account to call POST /charge on payments, and denies other callers because an ALLOW policy applies to the selected workload:

YAML
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata: { name: default, namespace: shop }
spec:
  mtls: { mode: STRICT }
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: { name: payments-allow-orders, namespace: shop }
spec:
  selector: { matchLabels: { app: payments } }
  action: ALLOW
  rules:
    - from:
        - source: { principals: ["cluster.local/ns/shop/sa/orders"] }
      to:
        - operation: { methods: ["POST"], paths: ["/charge"] }

Propagating end-user identity

The gateway authenticates the end user, commonly by validating an access token (for example a JWT signed by the identity provider). Internal services then need the user's identity to make authorization decisions. Approaches:

Approach How it works Trade-offs
Forward the original token Each service receives the user's access token and validates its signature, expiry, issuer, and audience. Simple. The token's scope is broad, and any service that receives it can replay it to others. Intermediate services see the full token.
Token exchange A service exchanges the incoming token at the identity provider for a new, narrower token for the next hop (OAuth 2.0 Token Exchange, RFC 8693), with a different audience and reduced scopes. Limits replay and blast radius. Adds a call to the identity provider.
Internal token from the gateway The gateway validates the external token and issues a short-lived internal token or signed header with only the needed claims. Keeps external tokens at the edge. Internal token format and signing keys must be managed.
Trust headers without signature (for example X-User-Id) Services accept identity from a plain header. Safe only if the network guarantees that the header cannot be forged, for example with mTLS between authenticated services. A compromised service can then impersonate any user. Avoid.

Practices for tokens: keep lifetimes short, restrict the audience, validate signature, issuer, expiry, and audience in every service that accepts a token, and never log tokens.

Where a service acts on its own behalf (a scheduled job, a batch process), it uses its workload identity or the OAuth 2.0 client credentials flow.

Authorization: gateway versus service

Authorization decides whether an authenticated identity may perform an action on a resource.

Level Checks Strengths Weaknesses
Gateway (edge) Token validity, coarse scopes, route-level rules, rate limits One place, blocks unauthenticated traffic early, reduces load on services Cannot know resource-level facts (does this order belong to this user?). Bypassed if internal services are reachable in other ways.
Service level Resource ownership, business rules, fine-grained permissions Has the data and domain knowledge to decide. Works even if the edge is bypassed. Logic may be duplicated or inconsistent across services.
Service-to-service policy Which callers may invoke which endpoints, from workload identity Limits lateral movement. Can be enforced by a mesh or a policy engine. Policies must be maintained as services evolve.

Defense in depth uses all three: the gateway rejects invalid requests, services verify tokens and check resource-level permissions, and network-level policy restricts which workloads can communicate. Services must not trust that the gateway has already authorized a request.

For consistent policy, some systems use a policy engine such as Open Policy Agent, which evaluates declarative policies written outside the service code. Policy decisions are then made in a sidecar or library near the service, with policy distributed centrally. Authorization models include role-based access control (RBAC), attribute-based access control (ABAC), and relationship-based models.

Network and platform controls

  • Network policies (for example Kubernetes NetworkPolicy) default-deny traffic between pods and allow only required paths.
  • Egress control restricts which external destinations services may call.
  • Minimal container images, non-root users, and read-only filesystems reduce what a compromised service can do (see containers and Docker).
  • Image scanning and signing in the CI/CD pipeline, and dependency vulnerability scanning.
  • Least-privilege cloud permissions per service, so that a stolen credential grants access to one resource set only.
  • Input validation at every service boundary, since internal callers can send malformed data.
  • Audit logging of authentication decisions, authorization denials, and administrative actions, linked by trace ID.

Secrets

  • Store secrets in a secret store, not in images, source control, or plain environment definitions.
  • Give each service its own credentials, scoped to what it needs.
  • Prefer short-lived, dynamically issued credentials and automate rotation.
  • Authenticate to the secret store with the workload identity.

Details are in configuration and secrets management.

Trade-offs

  • mTLS and per-request authorization add CPU and latency (usually small compared to network and application time) and operational work. Automation through a mesh or SPIRE is needed at scale.
  • Certificate lifetimes, clock accuracy, and trust bundle distribution become failure points. An expired certificate causes an outage.
  • Token exchange adds dependency on the identity provider. Cache validated tokens and use local signature validation with cached keys to reduce calls.
  • Fine-grained service-to-service policy improves containment but needs maintenance as call graphs change.

Interview questions

  • What does zero trust mean for internal traffic? Network location grants no trust. Every call is authenticated (typically mTLS between workloads, tokens for users), authorized against policy, and encrypted.
  • How does mTLS differ from TLS? In TLS only the server proves its identity. In mTLS the client also presents a certificate, so both peers are authenticated.
  • Why use short-lived certificates? A stolen key is useful only until expiry, so the exposure window is short, and revocation infrastructure matters less. Automated renewal makes short lifetimes practical.
  • Is authorization at the gateway sufficient? No. The gateway cannot check resource-level rules, and internal services may be reached without going through it. Services verify identity and permissions themselves.
  • How should a downstream service learn who the end user is? Through a validated token, either forwarded or exchanged for a narrower one, or a signed internal token from the gateway. An unsigned identity header is safe only if no untrusted party can set it.