Video Streaming and CDN: Framing, Scale Math, and Architecture

Chapter 15: Global Video Streaming & Edge CDN. Part II, Production System Blueprints. This lesson covers sections 1 to 3 and the chapter continues in Video Streaming and CDN: Deep Dives, Data Models, and Failure Modes.

At a glance

1. Executive Summary & The Problem Space

The Scene: The 4K World Cup Premiere Crash

At 3:00 PM on a Sunday afternoon, the final match of the World Cup kicks off. Over 45 million concurrent viewers open their smart TV apps, laptops, and mobile devices to watch the live 4K stream.

Within 45 seconds of the opening whistle, the streaming infrastructure enters a catastrophic death spiral.

The origin storage cluster backing the video streaming service experiences a crushing thundering herd. Because the edge content delivery networks (CDNs) were cold, over 120 global edge Points of Presence (PoPs) simultaneously miss on Chunk 0 (segment_001_4k.m4s) and forward their HTTP GET requests directly to the central Amazon S3 origin bucket.

S3 request rate limits (5,500 GET requests per second per prefix) are exceeded by a factor of twenty. S3 begins returning HTTP 503 Slow Down responses. The edge PoP servers, receiving 503 errors, fail to cache the chunk and mark the origin as degraded.

Meanwhile, on millions of client devices, the local video playback buffer empties within four seconds. Smart TVs across the globe freeze, displaying spinning loading wheels. Desperate viewers repeatedly hit refresh or restart their apps, multiplying ingress traffic by 400%.

In the incident bridge, a Senior engineer frantically suggests requesting an emergency rate limit increase from AWS support and scaling the origin API gateway fleet to 2,000 pods.

A Staff engineer unmutes and identifies the structural flaw: "Our origin architecture has no shield. We have 150 edge PoPs asking the origin for the exact same static byte range at the exact same millisecond. We are paying millions in cross-region egress and melting S3 because our CDN lacks request coalescing and an Origin Shield tier."

The fix was not begging the cloud provider for higher IOPS. The fix was architectural: inserting a dedicated Origin Shield tier with SingleFlight request coalescing (so that 10,000 edge misses collapse into a single origin fetch), standardizing on 2.0-second GOP-aligned chunking, and deploying an Adaptive Bitrate (ABR) engine that downshifts video quality before the playback buffer empties.

Clarifying Intent: Three Incompatible Video Systems

When an interviewer asks you to "Design YouTube" or "Design Netflix," they are testing whether you understand that video systems divide into three structurally different architectures:

The Three Video Streaming Intents

1. Intent 1: Ultra-Low Latency Live (WebRTC / Interactive): Latency must match real-time television (< 2 seconds). Caching is minimal because content is consumed once in real time. Sacrifices compression efficiency for streaming speed.
2. Intent 2: Video-on-Demand (VOD Catalog / Netflix): Content is known in advance. The system invests massive compute upfront (per-title encoding optimization) to shave off bits while maximizing peak signal-to-noise ratio (PSNR). Relies on 99%+ CDN cache hit rates.
3. Intent 3: User-Generated Content at Scale (The Staff YouTube Standard): Millions of raw video uploads per day from arbitrary devices. Requires an asynchronous, chunked distributed transcoding pipeline that parallelizes encoding across GPU fleets, generates adaptive bitrate ladders (HLS/DASH), and serves billions of views through a multi-tier edge CDN.

STAFF MOVE

Senior: "We will store the uploaded MP4 in S3 and configure CloudFront in front of it with a 24-hour cache TTL."

Staff: "A monolithic MP4 cannot stream adaptively. Clients on 3G cellular connections would stall, while clients on 4K fiber would be starved of resolution. I will build an asynchronous transcoding DAG that slices raw video into 2.0-second GOP-aligned segments across an Adaptive Bitrate (ABR) ladder (from 360p up to 4K AV1/HEVC). We will front S3 with an Origin Shield tier employing SingleFlight collapsed forwarding, reducing origin requests during viral premieres by 98.6%."

Why: The Staff engineer designs for heterogeneous client network conditions and protects origin economics through multi-tier caching.

Scale & Back-of-the-Envelope Economics

To ground our architectural design in production reality, we model a hyperscale video streaming platform (comparable to YouTube or Netflix):

• Daily Active Viewers: 200,000,000 daily active users.
• Concurrent Video Streams (Peak): 20,000,000 simultaneous streams.
• Daily Video Uploads (UGC): 500,000 new videos uploaded per day (~6 videos/sec).
• Average Video Duration: 10 minutes (600 seconds).
• Average Video Stream Bitrate: 3.0 Mbps (mix of 1080p, 720p, and 480p).

Peak Egress Bandwidth:

20,000,000 streams * 3,000,000 bits/sec = 60,000,000,000,000 bits/sec = 60 Terabits per second (Tbps)!

Monthly Cloud Egress Bill Without CDN:

At AWS standard egress rates of $0.05 per Gigabyte, transferring 60 Tbps continuously would cost over $9,700,000 per month! A high-performing CDN with a 98.5% cache hit rate slashes origin cloud egress down to under $150,000/month.

Scale & Hardware Capacity Requirements

2. Requirements & High-Level Architecture

Functional Requirements

1. Video Ingestion & Upload: Creators can upload raw video files of arbitrary format (MP4, MOV, MKV) up to 50 GB with resumable multipart uploads.
2. Automated Transcoding: The system automatically converts raw video into an Adaptive Bitrate (ABR) ladder (4K, 1080p, 720p, 480p, 360p) across modern codecs (AV1, HEVC, H.264).
3. Adaptive Streaming Playback: Viewers can stream video smoothly across variable network connections (WiFi, 5G, congested 3G) without manual quality selection.
4. Global Manifest Distribution: Generates compliant Apple HLS (.m3u8) and MPEG-DASH (.mpd) playlists with DRM encryption hooks.

Non-Functional Requirements (Staff Invariants)

1. Sub-350ms Time-to-First-Frame (TTFF): When a user taps "Play", video must render on screen in less than 350 milliseconds globally.
2. Rebuffering Ratio < 0.25%: Playback stalls (spinning wheels) must account for less than 15 seconds per 100 minutes of watched video.
3. Origin Shield Invariant: A sudden 100x traffic spike on a newly released video must never increase origin S3 request rate by more than 2x.
4. Zero Audio/Video Desynchronization: Bitrate transitions during network fluctuations must maintain sample-accurate audio/video synchronization.

3. End-to-End System Architecture

The architecture decouples the asynchronous, compute-intensive transcoding DAG from the ultra-low-latency edge delivery network:

Figure 15.1: End-to-End Global Video Ingestion, Transcoding DAG & Edge CDN Pipeline

Architectural Components in Depth

COMPONENT 1: VIDEO UPLOAD & RESUMABLE INGESTION GATEWAY

The creator ingestion tier managing gigabyte-sized raw video payloads:

• Presigned Multipart Ingestion Protocol: The mobile or desktop creator client initiates an upload session via POST /v1/videos/upload-session. The Ingestion Service authenticates the creator, allocates an upload ticket, and generates Amazon S3 Presigned Multipart URLs segmented into 10 MB parts.
• Direct-to-Object-Storage Upload: The client streams binary chunks directly to the S3 Staging Bucket using HTTP PUT requests over TLS, computing client-side SHA-256 checksums per part.
• Zero Ingress Server Saturation: Because media bytes bypass application servers entirely, API gateways remain completely free of memory pressure, socket exhaustion, and heavy garbage collection cycles.
• Resumable Upload State Machine: If a creator's network drops at part 84 of 100, the client queries GET /v1/videos/{id}/upload-status, receives a list of verified parts, and resumes directly from part 85 without re-uploading previous gigabytes.
• S3 ObjectCreated Event Egress: Once the final part completes, S3 emits an ObjectCreated:CompleteMultipartUpload notification to an Apache Kafka topic: video.uploaded.raw.

COMPONENT 2: GOP CHUNK SPLITTER & DAG ORCHESTRATOR

The metadata inspector and distributed task graph generator:

• Media Validation Engine: Stateless workers consume video.uploaded.raw events and run ffprobe to validate container integrity, frame rate, audio channel configurations, and color spaces, instantly quarantining corrupted files.
• GOP (Group of Pictures) Splitter: Divides raw high-resolution video into fixed 2.0-second GOP chunks. Crucially, every split is executed strictly on an Instantaneous Decoder Refresh (IDR) keyframe, ensuring every chunk is 100% self-contained and independently decodable without depending on previous or future chunks.
• DAG Job Generation: For a 10-minute video (300 chunks) transcoding into a 5-tier ABR ladder (1080p, 720p, 480p, 360p, 240p in H.264, HEVC, and AV1), the orchestrator compiles a Directed Acyclic Graph (DAG) containing (300 × 5 = 1,500) independent parallel chunk tasks.
• Task Scheduling Queue: Pushes chunk encoding jobs to high-throughput Redis/SQS queues partitioned by priority tier (e.g., premium creator jobs jump the queue ahead of long-tail uploads).

COMPONENT 3: GPU TRANSCODING WORKER FLEET

A horizontally scalable fleet of GPU-accelerated worker instances (e.g., AWS EC2 g5.xlarge with NVIDIA A10G Tensor Core GPUs):

• Hardware-Accelerated Encoding: Workers pull 2-second raw snippet jobs from S3 staging and encode chunks in parallel using NVENC hardware encoders or SVT-AV1.
• Two-Pass Variable Bitrate (VBR): Employs two-pass VBR with perceptual video quality tuning (Netflix VMAF > 93), allocating higher bitrates to high-motion scenes (e.g. sports or action) while compressing low-motion scenes (e.g. talking heads) by up to 60%.
• Direct Segment Egress: The transcoded chunk (segment_042_1080p.m4s) is written directly to the persistent public S3 Segment Storage bucket.
• Fault-Tolerant Worker Retries: If a spot GPU instance is preempted mid-encode, the task's visibility timeout expires in SQS and an alternate worker re-encodes only that single 2-second snippet. A full 2-hour movie transcodes across 1,000 workers in under 4 minutes!

COMPONENT 4: MANIFEST ASSEMBLER & MASTER PLAYLISTS

The packaging and DRM authorization engine:

• HLS & DASH Packaging: Once all chunk tasks for a video complete, the service compiles the Apple HLS master playlist (master.m3u8) and individual variant playlists (1080p.m3u8, 720p.m3u8), as well as the unified MPEG-DASH Media Presentation Description (manifest.mpd).
• Dynamic DRM Encryption (CENC): Encrypts media chunks using ISO/IEC 23001-7 Common Encryption (AES-128 CTR). Injects DRM signaling tags into manifests, allowing Apple FairPlay, Google Widevine, and Microsoft PlayReady clients to acquire decryption keys from a single unified encoding.
• Search & Feed Indexing Egress: Publishes a VideoPublished event to Kafka, alerting recommendation models, user feed generators, and search indexing pipelines that the asset is ready for streaming.

COMPONENT 5: ORIGIN SHIELD TIER

A dedicated tier of caching proxies (Nginx / Envoy / Apache Traffic Server) co-located in the same cloud region as the S3 origin:

• SingleFlight Request Coalescing: When 10,000 edge PoPs miss on the same video chunk simultaneously, the designated Origin Shield node locks the cache key in RAM and forwards exactly one HTTP GET request to S3.
• Memory Streaming Multiplex: All other 9,999 waiting client connections stream the resulting byte response directly from the shield's in-memory response buffer as it arrives from S3.
• Consistent Hashing by Segment URL: Edge PoPs hash chunk URLs across shield nodes, ensuring maximum cache locality and eliminating duplicate cached segments in the shield tier.
• Origin Shielding Efficiency: Slashes S3 read operations by over 98.5%, shielding S3 from rate-limit throttling and cutting cloud egress costs by hundreds of thousands of dollars per month.

COMPONENT 6: MULTI-CDN EDGE POP NETWORK & ANYCAST ROUTING

A globally distributed edge network combining proprietary ISP edge appliances (Open Connect) and commercial CDNs (Cloudflare, Fastly, Akamai):

• BGP Anycast Routing: Routes viewers to their geographically closest edge PoP in under 15ms network RTT.
• Tiered Storage Hierarchy: In-memory RAM cache (for viral segments and initial chunk 0) -> Fast NVMe SSD storage (for popular catalog titles) -> Origin Shield.
• Strict Cache-Control Headers: Media chunks (.m4s, .ts) are immutable and tagged with Cache-Control: public, max-age=31536000, immutable. Master manifests are dynamic and cached for only 2 seconds.
• TCP BBR Congestion Control: Implements Google BBR congestion control on edge sockets, delivering 30% higher throughput over congested last-mile Wi-Fi and mobile cellular connections compared to legacy TCP Cubic.

COMPONENT 7: CLIENT PLAYER & ADAPTIVE BITRATE (ABR) ENGINE

The smart video player runtime embedded in iOS (AVPlayer), Android (ExoPlayer), Smart TVs, and Web (Shaka Player):

• Buffer-Based Algorithm (BBA-2): Drives bitrate switching by continuously monitoring playback buffer occupancy (in seconds of decoded video) rather than relying on noisy, volatile network throughput estimates.
• Proactive Resolution Downshifting: If the buffer drops below the safety threshold (4.0 seconds), BBA downshifts resolution proactively before the buffer starves, completely eliminating buffering spinners.
• Chunk 0 Speculative Pre-Fetching: When a user scrolls past a video card in a mobile feed, the client pre-fetches Chunk 0 of the 360p stream into local memory. When the user taps Play, playback begins instantaneously in 0ms!

ByteByteGo-Style Step-by-Step Video Ingest & Edge Playback Flow

To demonstrate Staff mastery, walk the interviewer through both the asynchronous ingestion pipeline and the low-latency streaming playback path:

Step-by-Step Video Ingestion & Playback Streaming Flow

flowchart TD
    subgraph CREATOR["CREATOR INGESTION PATH"]
        C1["Creator"] -->|"(1) POST /v1/upload-session"| GW["API Gateway"]
        GW -->|"(2) Returns S3 Presigned URLs"| C2["Creator"]
        C2 -->|"(3) Direct PUT 10MB Chunks"| S3S["S3 Staging Bucket"]
        S3S -->|"(4) Emits ObjectCreated event"| DAG["DAG Orchestrator"]
        DAG -->|"(5) 2.0s GOP Split on IDR Keyframes"| DAG2["DAG Orchestrator"]
        DAG2 -->|"(6) Enqueues 1,500 jobs to SQS"| GPU["GPU Worker Fleet<br/>(NVENC Encode)"]
        GPU -->|"(7) Writes .m4s chunks to S3 Origin"| MAN["Manifest Assembler<br/>(Builds master.m3u8)"]
    end
    subgraph VIEWER["VIEWER STREAMING PLAYBACK PATH (#lt;100ms)"]
        V1["Viewer"] -->|"(8) GET master.m3u8"| POP["Edge CDN PoP<br/>(#lt;15ms Anycast)"]
        V2["Viewer"] -->|"(9) GET segment_001_1080p.m4s"| EC["Edge Cache"]
        EC -->|"(10) Cache Miss -> Hashed to Shield"| SH["Origin Shield"]
        SH -->|"(11) SingleFlight Coalescing<br/>(1 S3 request for 10k edge misses)"| S3V["S3 Video Segment Storage"]
    end

STEP 1: UPLOAD SESSION NEGOTIATION

The creator client requests an upload session via POST /v1/videos/upload-session. The API Gateway authenticates the creator, provisions a unique upload ticket, and initiates a multipart upload session.

STEP 2: PRESIGNED S3 MULTIPART URL GENERATION

The Ingestion Gateway generates short-lived, pre-authenticated Amazon S3 Presigned Multipart URLs for 10 MB chunks and returns them in the session response.

STEP 3: DIRECT-TO-S3 MULTIPART INGESTION

The creator client streams binary video chunks directly to the Amazon S3 Staging Bucket using parallel HTTP PUT requests with client-side SHA-256 validation, completely bypassing application servers.

STEP 4: S3 OBJECTCREATED EVENT INGESTION

Upon receiving and verifying all parts, S3 emits an ObjectCreated:CompleteMultipartUpload notification into Apache Kafka topic video.uploaded.raw.

STEP 5: IDR KEYFRAME-ALIGNED 2.0S GOP SPLITTING

The DAG Orchestrator consumes the event and validates the media container via ffprobe. It slices the raw video into uniform 2.0-second Group of Pictures (GOP) chunks, ensuring every split aligns strictly with an Instantaneous Decoder Refresh (IDR) keyframe.

STEP 6: PARALLEL TRANSCODING TASK DAG ENQUEUING

For a 10-minute video, the orchestrator generates a task graph of 1,500 independent chunk encoding jobs across the full ABR ladder and enqueues them into distributed priority SQS queues.

STEP 7: HARDWARE GPU TRANSCODING & SEGMENT STORAGE

A horizontally scaled fleet of GPU worker nodes encodes the 2.0-second snippets in parallel (H.264/HEVC/AV1) using two-pass VBR and writes the transcoded .m4s segments directly to the public S3 Video Segment Storage bucket.

STEP 8: CLIENT MANIFEST REQUEST VIA ANYCAST EDGE POP

The viewer taps "Play" on their client device. The player issues an HTTP GET for master.m3u8, routed to the geographically closest edge CDN Point of Presence (PoP) via BGP Anycast in < 15ms.

STEP 9: EDGE POP VIDEO SEGMENT REQUEST

The player inspects the manifest, selects the starting rendition (e.g. 1080p), and requests segment segment_001_1080p.m4s from the edge cache.

STEP 10: ORIGIN SHIELD CONSISTENT HASH FORWARDING

If the edge PoP experiences a cache miss, the segment URL is hashed across the Origin Shield tier, routing all edge misses for that segment to the same designated shield proxy node.

STEP 11: SINGLEFLIGHT REQUEST COALESCING & ORIGIN FETCH

The Origin Shield executes SingleFlight request coalescing: it locks the cache key in RAM, forwards exactly one HTTP GET to Amazon S3 for 10,000 concurrent edge misses, and streams the incoming byte payload simultaneously to all waiting edge connections.