Video Streaming and CDN: Staff Interview Calibration and Runbooks

Chapter 15 of Part II, Production System Blueprints, continued from Video Streaming and CDN: Deep Dives, Data Models, and Failure Modes. This lesson covers sections 7 to 10 of the chapter.

7. Ten Probing Follow-Up Questions (With Staff Answers)

Q1: "Why use 2-second chunks instead of 6-second or 10-second chunks?"

Senior Trap: "10-second chunks reduce HTTP requests and save bandwidth."

Staff Answer: "10-second chunks reduce HTTP header overhead by ~3%, but they severely impair video streaming ergonomics. A player can only adapt bitrates on chunk boundaries; with 10-second chunks, reacting to network congestion takes up to 10 seconds, causing buffer depletion and playback freezes. Furthermore, Time-to-First-Frame (TTFF) suffers because downloading Chunk 0 takes 5x longer. We standardize on 2.0-second chunks because they provide the optimal balance: sub-350ms startup latency, rapid ABR responsiveness, and manageable CDN request rates."

Q2: "How do you handle audio synchronization when switching video bitrates?"

Senior Trap: "We encode audio inside each video chunk and let the player sync them."

Staff Answer: "Multiplexing audio inside video segments leads to audio glitches and pops during bitrate transitions. We use DASH/HLS Demuxed Streaming: audio is transcoded as an independent, continuous AAC/Opus stream (audio_128k.m3u8) completely decoupled from video. The client player downloads the continuous audio track uninterrupted while seamlessly hot-swapping video renditions in a parallel decoder pipeline. Audio never stutters, and video switches smoothly."

Q3: "What cache-control headers do you place on manifests versus media chunks?"

Senior Trap: "Set a 1-hour cache TTL on everything."

Staff Answer: "Media chunks and manifests have opposite immutability contracts. Video chunks (segment_042.m4s) are immutable content-addressed blobs; their bytes never change. We cache them for 1 year: Cache-Control: public, max-age=31536000, immutable. Master manifests (master.m3u8), however, can change if renditions are updated or live streams progress. We cache manifests for only 2 seconds: Cache-Control: public, max-age=2, stale-while-revalidate=5. This allows instant cache updates without origin stampedes."

Q4: "How do you prevent unauthorized hotlinking of your video streams on third-party sites?"

Senior Trap: "Check the HTTP Referer header at the CDN edge."

Staff Answer: "HTTP Referer headers are easily spoofed by modern clients and scraping tools. We enforce HMAC Signed URL Tokens. When a viewer initiates playback, our API generates an encrypted token encoding the user ID, video ID, client IP subnet, and expiration timestamp: chunk.m4s?token=HMAC_SHA256(secret, uri + ip + expiry). The CDN edge validates the cryptographic signature locally in < 0.1ms using Cloudflare Workers or Fastly VCL before serving bytes. Expired or stolen tokens are rejected with HTTP 403."

Q5: "How do you handle multi-CDN routing and failover if one CDN provider goes down?"

Senior Trap: "We configure Round-Robin DNS between Cloudflare and Akamai."

Staff Answer: "Round-Robin DNS has no visibility into real-time network outages and respects DNS TTLs, delaying failover by hours. We use Client-Side RUM (Real User Measurement) + Multi-CDN Steering: The video player SDK periodically runs lightweight background probes measuring latency and throughput across CDN providers. If CDN A experiences a latency spike or error rate > 1%, the player dynamically rewrites segment URLs to CDN B on the next chunk request. Failover occurs within 2 seconds without user disruption."

Q6: "Why use Apple HLS and MPEG-DASH instead of standard MP4 progressive downloads?"

Senior Trap: "HLS is newer and supports streaming better."

Staff Answer: "A progressive MP4 download requires fetching the entire file sequentially; it cannot adapt to fluctuating network bandwidth. If a user on an MP4 download walks into an area with poor signal, playback halts completely. Furthermore, progressive downloads waste massive CDN egress: if a user watches 30 seconds of a 2-hour movie and abandons it, the player may have already buffered 500 MB of data that is immediately thrown away. Chunked HLS/DASH fetches only 2 seconds of video at a time, saving up to 60% of wasted bandwidth."

Q7: "How do you ensure fast video playback startup (Time-to-First-Frame < 350ms)?"

Senior Trap: "We start by requesting the highest quality 1080p chunk."

Staff Answer: "Requesting a 1080p chunk (1.2 MB) over a mobile connection requires multiple TCP round-trips to scale the TCP congestion window, delaying startup by 1,500ms. We enforce Fast Start Dual-Pumping: The player initiates parallel HTTP GET requests for Chunk 0 of the lowest bitrate rung (360p, ~125 KB) and Chunk 0 of 720p. The lightweight 360p chunk downloads in under 80ms, allowing the hardware decoder to render the first frame immediately. Once the pipeline is primed, the player seamlessly upshifts to 1080p on Chunk 1."

Q8: "How does YouTube handle live video streaming compared to VOD?"

Senior Trap: "It uses the exact same pipeline but processes chunks faster."

Staff Answer: "Live streaming uses Low-Latency HLS (LL-HLS) or CMAF Chunked Transfer Encoding. Instead of waiting for an entire 2-second segment to be encoded and written to disk before advertising it in a manifest, the encoder outputs 200-millisecond chunk parts (part_001.m4s). The edge CDN uses HTTP/1.1 chunked transfer to stream byte parts to the player while the rest of the 2-second segment is still being captured. This compresses glass-to-glass latency from 30 seconds down to under 2 seconds."

Q9: "How do you optimize video storage costs for long-tail videos with zero views?"

Senior Trap: "Delete videos that have no views after 90 days."

Staff Answer: "Deleting user content destroys platform trust. We apply Hierarchical S3 Lifecycle Tiering: (1) Hot renditions (1080p, 720p) for active videos remain in S3 Standard; (2) Chunks untouched for 30 days automatically transition to S3 Infrequent Access (IA); (3) For extreme long-tail videos (zero views in 180 days), we delete the pre-encoded ABR chunks and keep only the single high-quality raw source file in S3 Glacier. If a viewer accesses the video, an on-demand transcoding task re-encodes the video within 2 minutes."

Q10: "How do you defend against CDN cache stampedes during live sporting events?"

Senior Trap: "We configure the CDN with a longer cache TTL."

Staff Answer: "In live streaming, manifest TTLs cannot exceed 2 seconds because new segments are continuously added. When 10 million viewers refresh the manifest simultaneously, standard TTLs fail. We combine two techniques: (1) Stale-While-Revalidate: The edge CDN serves the current manifest from memory while an asynchronous background thread fetches the new manifest from the origin; and (2) Manifest Push: The origin pushes updated manifests directly to edge PoPs via HTTP/2 Server Push or WebSocket channels, eliminating polling traffic entirely."

8. Production Hardening & Operational Runbooks

Production SLOs & Alert Thresholds

Metric	Target Threshold	Severity & Action
Rebuffering Ratio	< 0.25% of watch time	P1 Alert: Audit last-mile CDN throughput & BBA state
Time to First Frame	< 350ms (Global p99)	P2 Alert: Check Chunk 0 pre-warming & dual-pump logic
Origin Shield Hit Rate	> 98.5% Globally	P1 Alert: Check consistent hashing ring & coalescing
Transcoding Task Lag	< 60s queue backlog	P2 Alert: Autoscale GPU worker spot instances

Incident Runbook: Mitigating Origin S3 503 Slow Down Throttling

When high concurrency threatens to overwhelm cloud object storage:

Step 1 (Verify SingleFlight Coalescing): Check metrics on the Origin Shield tier (nginx_request_coalescing_active_locks). If locks are not acquiring, verify that consistent hashing across shield proxies is healthy.
Step 2 (Enable Stale-While-Revalidate): Force global CDN edge nodes to serve cached chunks up to 300 seconds stale during origin 503 errors (proxy_cache_use_stale error timeout updating http_503).
Step 3 (Partition Key Salting): If a specific video asset is hot, activate dynamic URL rewriting at the edge gateway: rewrite chunk_042.m4s to salt_<1..16>/chunk_042.m4s, distributing S3 GET requests across 16 storage prefixes.

9. The Five Structural Fault Lines

Every design decision in video streaming and edge CDN delivery represents a fundamental distributed tradeoff:

The 5 Structural Fault Lines

Fault Line	Option A vs Option B	Who Pays?
1. Startup Latency vs Visual Sharp	Dual-Pumping Low Res vs Pure High-Res Start	Low Res: Product pays initial. High Res: Users pay 2s spinner
2. Chunk Duration Optimization	2.0s Chunks vs 6.0s Chunks	2.0s: Infra pays HTTP requests. 6.0s: Users pay slow ABR shift
3. Origin Shield vs Direct Egress	Dedicated Shield Tier vs Direct Multi-Edge	Shield: Infra pays server cost. Direct: Company pays cloud bill
4. Multi-CDN Redundancy	Multi-CDN RUM Steering vs Single Provider	Multi-CDN: Cools edge caches. Single: Outage vulnerability
5. Transcoding Storage Cost	Pre-Encoded All Rungs vs Just-In-Time Encode	Pre-Encoded: High compute cost. JIT: High startup latency tax

10. Chapter Summary & Staff Decision Matrix

Staff Decision Matrix: Video Streaming & CDN

Decision Point	Default Staff Position
Ingestion Architecture	Direct S3 multipart upload via presigned URLs
Transcoding Execution	Asynchronous GOP chunk splitting + GPU worker DAG
Chunk Duration	2.00 seconds GOP-aligned IDR keyframes across rungs
Delivery Protocols	Demuxed Apple HLS (.m3u8) and MPEG-DASH (.mpd)
Origin Protection	Dedicated Origin Shield tier with SingleFlight
Adaptive Bitrate Logic	Buffer-Based Adaptation (BBA) with hysteresis
Multi-CDN Steering	Client-side RUM telemetry with 2-second failover
Cache TTL Contract	1 year immutable for chunks, 2 seconds for playlist

The Staff Invariant Checklist

Before concluding a video streaming and CDN architecture interview, confirm that you have satisfied these four core requirements:

I have decoupled binary uploads from API servers: Videos stream directly from mobile clients to object storage via presigned URLs.
I have enforced GOP keyframe alignment: All rungs of the ABR ladder share identical 2.0-second IDR keyframe timestamps to prevent decoder crashes.
I have protected the origin with SingleFlight coalescing: The Origin Shield collapses concurrent cache misses into a single origin request.
I have defended the playback buffer: The ABR engine is driven by buffer health rather than volatile instantaneous TCP throughput.