URL Shortener: Failure Modes, Interview Questions, and Runbooks

This lesson covers sections 6 to 10 of Chapter 17 and continues URL Shortener: Deep Dives, Data Models, and Schemas.

6. Failure Modes, Edge Cases & Resilience

FAILURE MODES & RECOVERY RUNBOOKS

Failure Mode	Detection Metric	Automated Recovery Action
1. App Server Crash (Token Loss)	Server ungraceful exit without return	Abandon unused token range; new pod requests fresh lease
2. Edge Redis Cache Eviction Spike	Redis memory > 95% or hit rate < 95%	SingleFlight collapsed proxy reads from database replica
3. Malicious Phishing Campaign	Safe Browsing API or abuse report flag	In-memory edge Bloom filter instantly blocks short code
4. ZooKeeper Cluster Temporary Outage	ZK quorum unavail for > 10 seconds	App nodes use fallback local reserve ranges for 24 hours

Failure Mode 1: KGS Token Loss During Server Crashes

An application server boots up, leases a block of 100,000 IDs ([5,000,000 .. 5,099,999] ), mints 200 short links, and then suffers a kernel panic or out-of-memory crash.

The Senior Mistake: Attempt to reclaim the remaining 99,800 IDs by rolling back the ZooKeeper counter or persisting individual consumed tokens to disk on every write. This destroys write latency.

The Staff Architecture: Safely Discard the Range.

With a 7-character Base62 keyspace containing 3,520,000,000,000 unique IDs, losing 99,800 IDs is statistically negligible. Even if 10,000 servers crashed every year, total abandoned tokens would represent less than 0.0001% of the total capacity. Zero database overhead; zero rollback complexity.

Failure Mode 2: Malicious Phishing & Malware Camouflage

Attackers use the URL shortener to hide malicious phishing URLs, attempting to bypass corporate email scanners and security firewalls.

The Staff Architecture:

Asynchronous Anti-Abuse Scanner: Upon link creation, the URL is submitted to an asynchronous Google Safe Browsing / VirusTotal scanner worker pool.
Intermediary Warning Page for Suspicious Links: If a link is newly created (< 5 minutes old) or originates from an unverified free account, the edge proxy renders an intermediate warning screen: "You are leaving sho.rt to visit [domain]. Proceed with caution."
Edge Bloom Filter Blacklisting: Once a domain is confirmed malicious, the domain hash is pushed to the edge Bloom filter. All short links pointing to that domain are blocked at the edge within 50 milliseconds.

7. Ten Probing Follow-Up Questions (With Staff Answers)

Q1: "Why not use a central Snowflake ID generator instead of KGS range leasing?"

Senior Trap: "Snowflake IDs work great for distributed unique IDs."

Staff Answer: "Twitter Snowflake generates 64-bit IDs where the first 41 bits represent a millisecond timestamp. This timestamp prefix creates severe security and UX drawbacks for URL shorteners: (1) Predictability: The sequential timestamp allows attackers to calculate exactly when links were created and guess adjacent short links; (2) Key Length: A 64-bit integer encoded in Base62 requires 11 characters ("1A2b3C4d5Ef" ), defeating the primary goal of creating a concise, punchy short URL. KGS range leasing uses dense sequential numbers permuted via Feistel ciphers, yielding compact 7-character links with zero predictability."

Q2: "What happens if a user submits a custom vanity URL like sho.rt/blackfriday ?"

Senior Trap: "Check if the string is in the database and insert it if not."

Staff Answer: "Custom vanity aliases bypass the KGS pipeline and introduce potential collision and squatting risks. We handle custom aliases with three controls: (1) Reserved Keyword Blocklist: System routes (/api , /login , /dashboard , /metrics , /.well-known ) are reserved in an in-memory Trie; (2) Length & Character Restrictions: Vanity aliases must be 4 to 32 alphanumeric characters; (3) Unique Constraint with Conditional Insert: The write service executes an atomic INSERT INTO url_mappings (short_code, original_url) VALUES ('blackfriday', ...) ON CONFLICT DO NOTHING . If a row already exists, the API returns HTTP 409 Conflict with suggestion alternatives."

Q3: "How do you handle URL expiration and link deletion at scale?"

Senior Trap: "Run a daily SQL cron job: DELETE FROM url_mappings WHERE expires_at < NOW()."

Staff Answer: "Running a bulk DELETE query across a 10-billion row database locks table pages, bloats write-ahead logs, and causes massive replication lag. We use Lazy Expiration with Storage Tiering: (1) When a user accesses an expired link, the edge proxy detects expires_at < now(), deletes the key from Redis, and returns an HTTP 410 Gone; (2) In DynamoDB or Cassandra, we set native item TTLs where the storage engine purges expired records in low-priority background sweeps without write locks."

Q4: "How do you protect your edge redirect proxy against link-scraping bots?"

Senior Trap: "Rate limit by IP address to 10 requests per second."

Staff Answer: "Sophisticated botnets distribute requests across millions of residential IP addresses, easily bypassing simple IP rate limiting. We combine three defenses: (1) User-Agent & TLS Fingerprinting: Inspect JA3/JA4 TLS client fingerprints to detect automated headless scrapers (e.g. Python Requests, Go-http-client); (2) CAPTCHA Challenge for Rate Spikes: If an IP subnet scans hundreds of non-existent short codes (404 probing), route the connection to an edge Cloudflare Turnstile challenge; (3) Feistel Cipher: Cryptographically eliminates the sequential predictability that makes scraping profitable."

Q5: "How does your system handle the same long URL submitted by different users?"

Senior Trap: "Return the existing short URL so we save database space."

Staff Answer: "De-duplicating long URLs across different users causes major product and security issues: (1) Analytics Contamination: If User A shares a link for a marketing campaign and User B shares the same link for personal use, their click metrics merge into one confusing dashboard; (2) Privacy Leaks: User A could see User B's click volume; (3) Unauthorized Deletion: If User A deletes their link, User B's link breaks. Therefore, we create a new unique short code for every creation request, tying ownership and analytics strictly to user_id . Storage is cheap ($0.02/GB); broken analytics and privacy violations are expensive."

Q6: "Why use ClickHouse for analytics instead of PostgreSQL or Elasticsearch?"

Senior Trap: "PostgreSQL already stores our URLs, so we can just add a clicks table."

Staff Answer: "At 100,000 clicks per second, writing individual click rows into PostgreSQL causes catastrophic write-ahead log bloat and table lock contention. Reading aggregated metrics (e.g. SELECT count(*), country FROM clicks GROUP BY country ) requires scanning billions of rows, taking 30+ seconds. ClickHouse is a columnar DBMS designed for vectorized query execution on SIMD CPU architectures. It compresses click logs by 90%, ingests 500,000 rows/sec in micro-batches, and answers multi-dimensional aggregation queries across 1 billion rows in under 20 milliseconds."

Q7: "How do you prevent a viral link from knocking down your database on a cache miss?"

Senior Trap: "Provision a bigger database with more CPU and read replicas."

Staff Answer: "When a viral link drops out of cache or the cache node crashes, thousands of concurrent requests miss simultaneously. We deploy SingleFlight Request Collapsing at the Edge Proxy: When 5,000 edge threads encounter a cache miss for url:sb24 , the proxy acquires an in-memory lock for that key. Exactly one thread queries the database replica; the other 4,999 threads suspend execution and await the result. Once the first query returns, the proxy populates Redis and satisfies all 5,000 callers from memory. The database sees exactly 1 query."

Q8: "How do you ensure global consistency if a user updates their destination URL?"

Senior Trap: "Never allow users to edit destination URLs."

Staff Answer: "Enterprise customers require the ability to update destination URLs if a typo occurs in a multi-million-dollar print or billboard ad. When an update occurs: (1) Update the PostgreSQL primary row within an audit transaction; (2) Emit an invalidation event to a Redis Pub/Sub channel; (3) All global edge Redis nodes immediately invalidate url:{short_code} ; (4) The subsequent click fetches the new destination from the database replica and re-populates the cache. The propagation completes globally in under 200 milliseconds."

Q9: "Why not use Base64 encoding instead of Base62?"

Senior Trap: "Base64 is more compact and gives more combinations."

Staff Answer: "Standard Base64 encoding includes the characters + and /. Both characters have reserved syntactic meanings in URLs: + represents a space in query strings, and / separates path segments. Using Base64 requires URL escaping (%2B , %2F ), which lengthens the URL and creates parsing errors in legacy SMS clients and chat applications. Base62 uses strictly [0-9, a-z, A-Z] , which is 100% URL-safe without escaping."

Q10: "What is the blast radius if the ZooKeeper KGS cluster goes offline completely?"

Senior Trap: "The entire URL shortener stops working."

Staff Answer: "The KGS cluster is strictly on the link creation write path, never on the redirect read path. If ZooKeeper experiences a total outage: (1) 100% of existing shortened links continue to redirect with zero degradation (< 2ms); (2) Active application servers continue to mint new links using their existing local in-memory token blocks (100,000 IDs per node); (3) Only when an application server completely exhausts its local block and needs a new lease does link creation pause. This gives SREs hours of grace period to restore ZooKeeper before any customer impact occurs."

8. Production Hardening & Operational Runbooks

PRODUCTION SLOs & ALERT THRESHOLDS

Metric	Target Threshold	Severity & Action
Edge Redirect Latency	< 10ms (Global p99)	P1 Alert: Audit edge Redis hit rate & anycast routing
Edge Cache Hit Rate	> 98.5% Globally	P2 Alert: Check LRU churn & memory eviction pressure
KGS Token Pool Level	> 20% unused in pod	P2 Alert: Lease new block before RAM tokens exhaust
Clickhouse Lag	< 2.0s streaming lag	P2 Alert: Check Kafka lag & micro-batch flush size

Incident Runbook: Mitigating Edge Redirect Latency Degradation

When global redirect latency spikes past 30ms:

Step 1 (Check Edge Redis Hit Ratio): Query Datadog metric edge_redis_hit_ratio. If hit ratio drops below 95%, investigate if a cold-cache eviction occurred or if a new viral campaign was launched without cache pre-warming.
Step 2 (Enable Stale-While-Revalidate): Configure edge proxies to serve cached redirect targets even if expired by up to 10 minutes (proxy_cache_use_stale error timeout updating).
Step 3 (Validate SingleFlight Locks): Ensure that edge proxies are coalescing concurrent cache misses to prevent database read replica exhaustion.

9. The Five Structural Fault Lines

Every design decision in URL shortening and clickstream analytics involves a foundational system tradeoff:

THE 5 STRUCTURAL FAULT LINES

Fault Line	Option A vs Option B	Who Pays?
1. Key Generation Strategy	Pre-Allocated KGS vs Online Hash+Retry	KGS: Infra pays ZooKeeper state; Hash: DB & Users pay collision
2. Redirect HTTP Semantics	HTTP 302 Found vs HTTP 301 Permanent	302: Infra pays edge traffic; 301: Marketing pays blind data
3. Click Analytics Pipeline	Asynchronous Kafka vs Synchronous Writes	Async: 1-2s dashboard lag; Sync: Users pay 40ms redirect
4. Security & Predictability	Feistel Permutation vs Raw Auto-Increment	Feistel: 15 CPU instructions; Auto-Inc: Company pays scraping
5. Long-URL Deduplication	Shared Deduplication vs Unique per User	Shared: Contaminates analytics; Unique: Storage pays $0.02/GB

10. Chapter Summary & Staff Decision Matrix

STAFF DECISION MATRIX: URL SHORTENING & ANALYTICS

Decision Point	Default Staff Position
ID Generation Engine	Range-leased KGS (ZooKeeper / Raft) in RAM blocks
Code Obfuscation	64-bit Feistel cipher permutation before Base62
Key Encoding Format	7-character Base62 (`[0-9, a-z, A-Z]`)
Redirect HTTP Status	HTTP 302 Found (preserves 100% click attribution)
Edge Caching Layer	Multi-region Redis / Cloudflare KV with 7-day TTL
Analytics Ingestion	Non-blocking Kafka producer on edge redirect path
Analytical Database	ClickHouse columnar store with Materialized Views
Malicious Link Defense	In-memory edge Bloom filter + Safe Browsing scanner

The Staff Invariant Checklist

Before completing a URL shortener and clickstream analytics interview, verify that you have covered these four core principles:

I have decoupled ID generation from the database write path: The service uses range-leased KGS token blocks in RAM with zero write collisions.
I have defended against competitor enumeration attacks: Short codes are passed through a Feistel cipher permutation to eliminate sequential predictability.
I have selected HTTP 302 over 301 for marketing attribution: The redirect preserves complete clickstream fidelity while offloading traffic to edge Redis caches.
I have decoupled redirect latency from analytical writes: Click events are dispatched asynchronously to Kafka without blocking the HTTP redirect.