CI/CD and Deployment Strategies

Continuous integration (CI) merges changes into a shared branch frequently and verifies each one with an automated build and tests. Continuous delivery keeps every verified change in a deployable state, with release as a manual decision. Continuous deployment releases every change that passes the pipeline automatically. In a microservices system, each service has its own pipeline, so services can be released independently.

Pipeline stages

A pipeline is an automated sequence of stages triggered by a commit. Each stage gates the next.

graph LR
    C["Commit"] --> B["Build and unit tests"]
    B --> S["Static analysis and dependency scan"]
    S --> I["Package image and push to registry"]
    I --> T["Integration and contract tests"]
    T --> ST["Deploy to staging"]
    ST --> P["Deploy to production"]
    P --> V["Verify: health, metrics"]

Principles:

• Build once, deploy the same artifact everywhere. Environment differences come from configuration, not from rebuilding.
• Keep pipelines fast. Slow pipelines encourage large, infrequent changes, which are riskier.
• Keep the mainline releasable. Use short-lived branches or trunk-based development, with unfinished features hidden behind feature flags.
• Declare deployments in version control. In GitOps, a controller in the cluster applies the state described in a Git repository, so Git history is the deployment history.

Deployment strategies

The strategies differ in how new and old versions overlap, how much traffic the new version sees, and how quickly the change can be undone.

Recreate

Stop all old instances, then start the new version. It is simple, but causes downtime. It is used when two versions cannot run at once.

Rolling update

Replace instances gradually: start a new instance, wait until it is ready, stop an old one, repeat. Capacity stays close to constant (in Kubernetes, controlled by maxSurge and maxUnavailable).

• Needs no duplicate environment.
• Old and new versions serve traffic concurrently, so they must be compatible with each other and with the shared database.
• Rollback is another rolling update in reverse, which takes as long as the rollout.

Blue/green

Maintain two full environments. Blue serves production. Green receives the new version and is tested while receiving no user traffic. Traffic is then switched from blue to green at the load balancer or router. Blue stays available for rollback.

graph LR
    U["Users"] --> R["Router / load balancer"]
    R -->|"before switch"| BL["Blue: v1"]
    R -.->|"after switch"| GR["Green: v2"]
    BL --> DB[("Shared database")]
    GR --> DB

• Rollback is a fast traffic switch.
• Requires double capacity during the release.
• The switch is all-or-nothing, so a defect affects all users at once.
• The database is normally shared, so schema changes must work for both versions.

Canary

Route a small fraction of traffic to the new version (for example 1%), compare its error rate and latency to the stable version, and increase the share in steps (5%, 25%, 50%, 100%) if metrics stay healthy. Canary analysis can be automated by comparing metrics between the canary and baseline, and aborting on regression.

• Limits the blast radius of a defect to a small share of users.
• Requires traffic splitting (load balancer weights, service mesh, or ingress rules) and good metrics.
• Splitting can be by request percentage, or by user attribute so that a user sees a consistent version.
• Low traffic volume makes canary metrics statistically weak. At 1% of 100 requests per second the canary receives 1 request per second, or 600 requests in 10 minutes, so a single error is already a 0.17% error rate and small regressions cannot be told apart from noise.

Shadow (traffic mirroring)

Copy production requests to the new version and discard its responses. Users see only the old version's responses.

• Tests the new version with real traffic and no user impact.
• Side effects must be neutralized. Writes to databases, payments, and emails from the shadow instance must be suppressed or directed to isolated stores.
• Doubles the load on downstream dependencies that are not stubbed.

Comparison

Deployment and release are separable. A version can be deployed with a feature disabled and released later by enabling a flag, which allows rollback without redeploying.

Database migrations with deployments

A schema change cannot be applied atomically with the application rollout. During a rolling or canary deploy, old and new code run against the same schema. The standard approach is expand and contract, in separate releases:

The order matters because a rollout is not atomic: the schema must already support the new code before that code runs, and must keep supporting the old code until no old instance remains.

1. Expand: add the new column, table, or index in a backward-compatible way. Old code ignores it.
2. Deploy new code that writes to both old and new structures, and reads from one of them.
3. Backfill existing rows into the new structure.
4. Switch reads to the new structure.
5. Contract: after no running version uses the old structure, remove it.

Example: renaming a column. Add the new column, write both, backfill, read the new one, then drop the old one. A direct RENAME breaks any running instance of the old version.

Practices: apply migrations as a separate, reviewed pipeline step. Make each migration backward compatible with the previous application version. Avoid long locks on large tables. Destructive changes come last, after a rollback is no longer needed.

Rollback

• Application rollback is redeploying the previous artifact, switching traffic back (blue/green), or setting the canary weight to zero. It works only if the previous version can still operate on the current database and data.
• Data changes are not rolled back by redeploying code. Expand and contract keeps the previous version compatible, which keeps rollback possible.
• Roll forward (ship a fix) is sometimes faster than rollback, especially after an irreversible migration.
• Automate the trigger. Health checks and SLO-based alerts after deploy can halt a rollout or revert it.

Interview questions

• How do blue/green and canary differ? Blue/green switches all traffic between two full environments at once, giving fast rollback but all-at-once exposure. A canary exposes a small, growing fraction of users to the new version, limiting the effect of defects.
• How do you rename a database column without downtime? Use expand and contract. Add the new column, write to both, backfill, move reads, and drop the old column in a later release.
• Why must services in a rolling update be backward compatible? During the rollout, old and new versions run at the same time and call each other and share data. Incompatible changes cause errors during the overlap.
• What is the difference between deployment and release? Deployment puts code on servers. Release exposes behavior to users. Feature flags separate the two.
• How would you make deployment safe for a service with low traffic? Use synthetic or replayed traffic for canary analysis, or use blue/green with smoke tests, since a small real-traffic canary gives too little signal.