Configuration and Secrets Management

Configuration is every value that varies between deployments of the same code: database addresses, feature limits, timeouts, endpoints of other services, credentials. A secret is configuration whose disclosure causes harm: passwords, API keys, private keys, tokens. Microservices multiply both, since each service has its own configuration and many services need credentials for databases, brokers, and other services.

Separating configuration from code

The twelve-factor app methodology states that configuration belongs in the environment, not in the code. A useful test is whether the codebase could be made public without exposing any credential, and whether the same build artifact can run in every environment.

• Build one artifact (for example one container image) and inject configuration when it starts.
• Do not commit environment-specific files or secrets to the repository.
• Do not group configuration into named environments inside the code (if env == "prod"). Treat each setting as an independent value.

Configuration sources, from simple to more managed:

In Kubernetes, a ConfigMap and a Secret provide values to pods as environment variables or files.

Configuration services

A configuration service stores configuration centrally and serves it to applications. Examples: Spring Cloud Config, Consul KV, etcd, AWS AppConfig, AWS Systems Manager Parameter Store.

Properties to look for:

• Versioning and audit: who changed what, and the ability to revert.
• Hierarchy: defaults, then per-environment, then per-service, then per-instance overrides.
• Validation before rollout: schema checks so that a typo does not reach all services at once.
• Gradual rollout of changes. A configuration change can break production as easily as a code change, so it deserves the same staged rollout (see CI/CD and deployment strategies).

Trade-offs: a central service is another dependency. Services should read configuration at startup and cache it, and continue with the last known good values if the service is unreachable. A service that cannot start without the configuration service has coupled its availability to it.

Dynamic configuration

Some settings should change without restarting a service: log levels, timeouts, rate limits, feature flags. Mechanisms:

• Polling a config service at an interval.
• Push or watch, where the store notifies clients of changes (etcd watches, Consul blocking queries).
• File reload, where a process watches a mounted file, or receives a signal such as SIGHUP.

Requirements: apply a new configuration atomically (parse and validate the whole new value, then swap it in), and keep the old value if validation fails. Settings that are read once, such as listening ports and connection pool sizes at initialization, still need a restart or a pool rebuild. Dynamic configuration changes system behavior without a deploy, so changes must be logged and be reversible.

Secrets management

Secrets need stronger controls than ordinary configuration:

• Encrypted at rest and in transit.
• Access restricted per identity, following least privilege.
• Every access recorded in an audit log.
• Rotatable without a code change.
• Never written to source control, images, logs, or error messages.

Common failure: secrets in the repository or in image layers, where they persist in history and are copied to every clone.

Secret stores

A secret store is a dedicated system that stores secrets encrypted, authenticates callers, authorizes by policy, and audits access.

• HashiCorp Vault stores secrets, authenticates clients through methods such as Kubernetes service account tokens or cloud IAM identities, applies policies, and can generate credentials on demand.
• Cloud secret managers such as AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integrate with the provider's IAM.
• KMS (key management service) holds encryption keys and performs encrypt and decrypt operations without exposing the key. Envelope encryption encrypts data with a data key, and encrypts the data key with a master key held in the KMS. Rotating the master key then requires re-encrypting only the small data keys.

Envelope encryption steps: (1) the application asks the KMS for a new data key and receives it in plaintext together with a copy encrypted under the master key. (2) It encrypts the data locally with the plaintext key and discards that key from memory. (3) It stores the ciphertext together with the encrypted data key. (4) To decrypt, it sends the encrypted data key to the KMS, receives the plaintext key, and decrypts locally. The KMS handles only small requests, and the master key never leaves it.

sequenceDiagram
    participant S as Service
    participant I as Identity provider
    participant V as Secret store
    participant D as Database
    S->>I: prove identity (platform token)
    I-->>S: identity token
    S->>V: authenticate with token, request database credentials
    V->>D: create user with limited role and TTL
    V-->>S: username, password, lease (TTL)
    S->>D: connect with short-lived credentials
    Note over S,V: renew the lease before expiry, or fetch new credentials

The service needs an initial way to identify itself to the store. This is the secret zero problem. It is solved by using the platform's identity, such as a Kubernetes service account token or a cloud instance identity, so no long-lived bootstrap secret is stored in the service.

Delivery to the application

Rotation

Rotation replaces a secret with a new one at regular intervals, and immediately after suspected exposure. It limits how long a leaked credential is useful.

Zero-downtime rotation of a shared credential such as a database password:

1. The system accepts both the old and the new credential (two active credentials, or two users).
2. Clients pick up the new credential (reload or restart).
3. After every client has moved, the old credential is revoked.

Strategies:

• Dual credentials: keep two valid at the same time during the overlap.
• Dynamic secrets: the store creates a unique credential per client with a short lifetime, such as minutes to hours, and revokes it at expiry. A leaked credential expires soon, each credential identifies its holder in audit logs, and no manual rotation is needed.
• Certificates: issue short-lived certificates and automate renewal (see microservices security and zero trust).

Applications must handle rotation: reload credentials without restart where possible, and retry once with fresh credentials on an authentication failure.

Failure modes

• Secret store unavailable at startup: cache secrets already obtained, and retry with backoff.
• A bad configuration pushed to all services at once: validate, roll out gradually, and support fast rollback.
• Secrets in logs: mask known secret values and never log full requests or environment dumps.
• Overly broad access: a single credential shared by many services makes revocation and auditing impractical. Issue per-service credentials.

Interview questions

• Why not store configuration in the image? The image would differ per environment, and secrets would be baked into layers that anyone with image access can read. One image with injected configuration keeps builds identical across environments.
• What is the difference between a config service and a secret store? A config service distributes non-sensitive settings with versioning. A secret store adds encryption, per-identity policy, audit, and leasing or rotation for sensitive values.
• How do you rotate a database password without downtime? Allow two valid credentials, roll clients to the new one, then revoke the old. Dynamic secrets avoid manual rotation by issuing short-lived per-client credentials.
• What is envelope encryption? Data is encrypted with a data key, and the data key is encrypted with a master key held in a KMS. Master key rotation needs only re-wrapping the data keys.
• What is the secret zero problem? A service needs a credential to authenticate to the secret store. It is addressed with a platform-issued identity, such as a service account token, instead of a stored bootstrap secret.