File Storage and Sync: Framing, Scale Math, and Architecture

This lesson covers sections 1 to 3 of Chapter 12, Distributed File Storage & Sync Engine. It continues in File Storage and Sync: Chunking, State Machine, and Deep Dives.

Production System Blueprint 9: Content-Defined Chunking, Client State Machines, and Zero-Data-Loss Conflict Resolution

At a glance

1. The Scenario & Problem Framing

PROMPT:

"Design an enterprise-scale distributed file storage and delta synchronization platform (like Dropbox, Google Drive, or Box) supporting 100,000,000 registered users and 25,000,000 daily active users (DAU).

The system must store over 1,000,000,000 files across heterogeneous client environments (Windows, macOS, Linux, iOS, Android) and synchronize changes with sub-5-second cross-device propagation latency. The platform must support files up to 50 Gigabytes in size, minimize bandwidth consumption via Content-Defined Chunking (CDC) and global data deduplication, provide reliable offline sync with zero-data-loss conflict resolution, support on-demand virtual filesystem hydration (Selective Sync), and ensure absolute data durability (99.999999999% - eleven 9s) across distributed storage regions."

Uploading a file to an Amazon S3 bucket via a web form is a basic task for any software engineer. You accept a multipart upload, stream bytes to object storage, and write a record to a database.

Building a cross-platform file synchronization engine operating across millions of laptops, smartphones, and desktop computers is one of the most challenging distributed systems problems in modern computing:

1. The Client-Side Edge Chaos: Unlike server-to-server microservices operating in pristine data centers with 10 Gbps fiber interconnects, file sync runs on untrusted client edge hardware. Laptops close abruptly mid-upload; Wi-Fi drops while traversing elevators; users edit files while disconnected on airplanes for three days; third-party software (compilers, Photoshop, anti-virus scanners) rapidly modifies files in bursts of micro-writes. The primary complexity of file sync lives on the client device, not in the cloud.
2. The Insertion Off-By-One Bandwidth Catastrophe: If you split a 1 Gigabyte file into fixed 4 Megabyte blocks, inserting a single byte at the beginning of the file shifts every subsequent byte offset by 1. A naive fixed-chunking architecture recalculates completely different SHA-256 hashes for every single block, forcing the client to re-upload the entire 1 Gigabyte file over the network for a 1-byte edit.
3. The Silent Data Loss Trap (Blind Last-Write-Wins): When two team members edit the same 200 Megabyte financial spreadsheet offline while flying on separate airplanes, their local versions diverge. In naive backend systems, whichever user lands and connects to Wi-Fi last overwrites the other user's file via Last-Write-Wins (LWW). Three hours of critical financial modeling is permanently destroyed without any warning.
4. The Metadata Lockup on 100,000-File Directory Moves: Users frequently drag a parent directory containing 100,000 files and subfolders into a new location. In naive architectures that store full file paths as primary keys (e.g., /user_102/Documents/Projects/2026/...), a single folder rename triggers 100,000 individual SQL update mutations, causing database row locks to cascade, transaction timeouts, and global sync queue stalls.
5. The Storage Exhaustion Wall: Storing 1 Billion files averaging 5 Megabytes each requires 5 Petabytes of raw storage. Without global content-addressable deduplication (CAS), cross-user redundancy (identical operating system installers, shared corporate templates, viral media) consumes petabytes of wasted cloud disk space, inflating monthly infrastructure bills into hundreds of thousands of dollars.

THE CORE PARADOX

Users expect instantaneous file sync with zero bandwidth overhead, yet files are opaque binary blobs that cannot be automatically merged like text code.

Staff Goal: Decouple metadata coordination from raw blob storage, enforce Rabin Content-Defined Chunking (CDC) on the client to isolate byte shifts, upload missing chunks directly to Content-Addressable S3 storage via presigned URLs, and preserve 100% of concurrent offline edits via the Conflicted Copy preservation protocol.

The L5 vs L6 Contrast — What the Evaluator Actually Hears

The Senior Candidate (L5): Immediately focuses on the server upload API: "We will create a REST API where clients upload files to our backend, our backend writes them to S3, and writes the metadata to PostgreSQL." When asked about performance on large files, they suggest breaking files into fixed 4 MB chunks. When asked about concurrent offline edits, they suggest Last-Write-Wins based on timestamps. They view file sync as a glorified file-upload server.

The Staff Candidate (L6): Reframes the problem around client-side state machine mechanics, content-defined delta mathematics, and zero-data-loss conflict semantics within the first 90 seconds: "File sync is a client-side distributed systems problem. The server is comparatively simple: an immutable Content-Addressable Storage (CAS) blob store and an atomic metadata tree. The true architectural challenge is on the client. First, we must implement Content-Defined Chunking (CDC) using Rabin rolling fingerprints: an insertion shifts only the local chunk, allowing the remaining 99.8% of chunks to retain identical SHA-256 hashes and avoid re-upload. Second, uploads must bypass application servers entirely via presigned S3 URLs. Third, Last-Write-Wins for binary files is an unacceptable anti-pattern; we enforce atomic compare-and-swap metadata commits backed by automatic Conflicted Copy preservation so zero user bytes are ever lost. Finally, we manage the local disk footprint via virtual filesystem on-demand hydration."

2. Step 1: Commit to Intent & Scale Math

Before drawing boxes or designing schemas, an L6 Staff engineer establishes the core operational intent. Cloud file storage divides into three incompatible system profiles:

The Three Incompatible Intents

1. Personal Cloud Synchronization (Dropbox / Google Drive):

• The Constraint: Bandwidth conservation, cross-platform mobile battery limits, and fast local directory response times.
• The Architectural Commit: Content-Defined Chunking (Rabin avg 4 MB), client-side SQLite metadata checkpointing, and virtual filesystem hydration.

2. Team Shared Workspaces (Box / Enterprise Drive):

• The Constraint: High concurrent writer density on shared folders, granular permission inheritance (ACLs), and legal compliance audits.
• The Architectural Commit: Strict optimistic concurrency control (CAS) on metadata, hierarchical directory DAG permissions, and immutable revision histories.

3. Continuous Cloud Backup & Archival (AWS Backup / Backblaze):

• The Constraint: Massive, continuous write throughput without user interaction; read operations are rare (disaster recovery only).
• The Architectural Commit: Append-only streaming block storage, zero deduplication on the hot ingest path, and aggressive tiering to cold Glacier storage.

STAFF MOVE

Senior: "We will stream file uploads through our API servers so we can compute the SHA-256 hash and verify file integrity before saving to S3."

Staff: "Streaming gigabyte-sized binary file uploads through application servers is an infrastructure disaster. At 50,000 concurrent uploads, application servers run out of memory, network interface cards saturate, and a single server crash terminates thousands of active uploads. Application servers should strictly coordinate metadata. The client computes chunk SHA-256 hashes locally, requests presigned S3 PUT URLs from the metadata service, and streams binary chunks directly to S3. The metadata server verifies chunk existence via an asynchronous commit call. This offloads 99.9% of network bandwidth and disk I/O to AWS edge infrastructure."

Why: Demonstrating mastery of edge-offloaded I/O and stateless API design separates Staff engineers from junior backend builders.

Capacity Planning & Scale Math

At 100,000,000 registered users, 25,000,000 Daily Active Users (DAU), and 1,000,000,000 stored files:

1. FILE MODIFICATION VELOCITY & CHUNK INGESTION THROUGHPUT

Active Files Modified per Day: Assuming an average active user modifies or uploads 8 files per day:

Daily File Modifications = 25,000,000 users * 8 files = 200,000,000 file events/day

Average Modification Profile:

• 70% of modifications are small text/document edits (size < 1 MB) -> 1 chunk.
• 25% of modifications are medium edits (PDFs, presentations, 1 MB to 20 MB) -> 3 to 5 chunks.
• 5% of modifications are large media (videos, archives, > 100 MB) -> CDC shifts only 1 or 2 chunks!

Average Changed Chunks per Modification: ~1.4 chunks per file event.

Daily Chunk Upload Volume:

Total Chunks Uploaded = 200,000,000 events * 1.4 chunks ≈ 280,000,000 chunks/day
Average Chunk Ingestion Rate = 280,000,000 / 86,400 seconds ≈ 3,240 chunks/second
Peak Chunk Ingestion Rate (3.5x peak burst) ≈ 11,340 chunk uploads/second

2. STORAGE VOLUME & GLOBAL DEDUPLICATION SIZING

Total Stored Files: 1,000,000,000 files.

Average File Size: 5 Megabytes.

Raw Stored Data (Before Deduplication):

Raw Logical Data = 1,000,000,000 * 5 MB = 5,000 Terabytes = 5 Petabytes

Global Deduplication Factor (Cross-User & Versioning):

In enterprise cloud storage, cross-user deduplication (shared attachments, standard OS binaries, copied corporate slide decks) achieves a 35% reduction. Delta versioning deduplication (retaining only changed 4 MB chunks across file versions) achieves an additional 45% reduction in historical storage.

Net Physical Storage in S3 (CAS Blobs):

Net Physical S3 Storage = 5 Petabytes * (1 - 0.35) ≈ 3.25 Petabytes

At $0.021 per GB/month on Amazon S3 Standard, deduplication saves $36,750 per month ($441,000 annually) in raw cloud storage costs!

3. METADATA DATABASE SIZING (POSTGRESQL / COCKROACHDB)

Each file requires:

File record: file_id (16 bytes), parent_folder_id (16 bytes), owner_id (16 bytes), file_name (64 bytes), version (8 bytes), created_at (8 bytes) ≈ ~128 bytes.

Chunk Manifest Mapping: A 5 MB file consists of ~2 chunks (average chunk size: 4 MB). Each mapping row: file_version_id (16 bytes), chunk_sha256 (32 bytes), chunk_order (4 bytes), byte_offset (8 bytes) ≈ ~60 bytes.

Metadata Database Size:

File Records = 1 Billion * 128 bytes = 128 Gigabytes
Chunk Manifest Records = 2 Billion * 60 bytes = 120 Gigabytes
Indexes & Foreign Keys Overhead = ~250 Gigabytes
Total Metadata Footprint ≈ 500 Gigabytes

The Staff Revelation: Sizing proves that the entire global file metadata tree of 1 Billion files fits into 500 GB of storage, which comfortably resides on a 3-node CockroachDB or sharded PostgreSQL cluster with full in-memory index caching!

Key System Sizing Numbers Worth Memorizing

3. End-to-End System Architecture

The architecture cleanly decouples the stateful local client daemon, the metadata coordination engine, the direct-to-S3 blob transport path, and the asynchronous event processing pipeline:

Figure 12.1: End-to-End Distributed File Storage & Sync Engine Architecture

Architectural Components in Depth

COMPONENT 1: CLIENT SYNC DAEMON RUNTIME (LOCAL DESKTOP / MOBILE)

The stateful background process embedded in the user's operating system (Windows, macOS, Linux, iOS, Android):

• Kernel Filesystem Monitoring: Hooks directly into OS kernel notification subsystems (inotify on Linux, FSEvents on macOS, ReadDirectoryChangesW on Windows) to detect file modifications, creations, deletions, and moves with zero polling overhead.
• 500ms Write-Burst Debouncing: Enforces a 500-millisecond quiet window following any local write event. If an IDE or text editor auto-saves 10 times in 2 seconds, the daemon coalesces the writes into a single sync evaluation, preventing network and CPU thrash.
• Rabin Content-Defined Chunking Engine: Slices modified files using rolling polynomial Rabin fingerprints into variable-size chunks averaging 4 MB. Computes cryptographic SHA-256 hashes per chunk in parallel using SIMD instructions.
• Local Embedded Metadata Store (SQLite): Maintains an embedded SQLite database storing local file paths, file modification timestamps (mtime), device-specific inode numbers, and chunk hashes. Enables instant local delta detection by comparing new Rabin hashes against the local database before contacting the cloud.
• Resumable Parallel Upload Manager: Manages an adaptive pool of worker threads uploading missing chunks directly to cloud storage over HTTPS with exponential backoff retries.

COMPONENT 2: API GATEWAY & METADATA SERVICE

The central brain and orchestration tier of the cloud storage platform:

• Stateless Microservice Cluster: Horizontally scalable Go or Rust worker pods behind an Anycast Layer-7 load balancer. Authenticates users via OAuth2/JWT tokens and enforces workspace permissions.

• Chunk Existence Catalog Check: When a client submits a file upload manifest containing 100 chunk hashes, the Metadata Service checks the global CAS catalog. It identifies which hashes already exist in cloud storage and responds with S3 presigned PUT URLs strictly for the missing chunks.

• Atomic File Version Commit (Compare-And-Swap): When the client finishes uploading missing chunks, it submits a commit request: Commit(file_id, expected_version: 11, new_chunk_hashes). The service executes an atomic SQL transaction:

  UPDATE filesystem_nodes
  SET version = 12, updated_at = NOW()
  WHERE node_id = :file_id AND version = 11;

  If another device committed Version 12 first, the update fails with HTTP 409 Conflict, triggering client-side conflicted-copy preservation.

• Change Event Ingestion: Upon successful commit, publishes a FileModified notification to Apache Kafka topic filesystem.changes.raw.

COMPONENT 3: CONTENT-ADDRESSABLE BLOB STORAGE (CAS)

The immutable blob storage foundation (Amazon S3, Google Cloud Storage, or Ceph):

• Cryptographic Hash Addressing: Every chunk is stored as an immutable object keyed strictly by its hexadecimal SHA-256 hash: s3://bucket/chunks/{sha256_hash}.
• Strict Immutability Guarantee: Chunks are written once and never modified. If a user edits a single sentence in a 1 GB file, only one 4 MB chunk changes. The new chunk is written to S3 under its own hash, while the remaining 249 chunks continue pointing to their existing S3 blobs.
• Cross-Account Global Deduplication: If 10,000 corporate employees across 500 different organizations all save the same 40 MB PDF attachment, the physical chunk bytes exist exactly once in S3, slashing cloud storage expenditure by over 35%.

COMPONENT 4: NOTIFICATION & EDGE SYNC FLEET

The real-time synchronization signalling tier:

• High-Density Connection Termination: Manages millions of persistent, long-lived WebSocket or HTTP/2 Server-Sent Events (SSE) connections to active client sync daemons.
• Sub-50ms Signal Dissemination: Subscribes to Redis Pub/Sub channels keyed by folder_channel:{folder_id}. When a file commits, the notification fleet broadcasts a compact sync signal to all other devices authorized on that folder: { "event": "FILE_UPDATED", "file_id": "f-1048", "version": 12 }.
• Zero-Payload Signal Architecture: The push notification contains zero file contents. It acts purely as a wake-up doorbell. The client daemon wakes up, fetches the latest lightweight metadata manifest from the Metadata Service, computes missing chunks, and streams bytes in the background.

COMPONENT 5: DISTRIBUTED METADATA STORE (COCKROACHDB / SHARDED POSTGRESQL)

The authoritative transactional system of record for the directory tree and file manifests:

• Hierarchical Adjacency List DAG: Stores files and folders as graph nodes referencing their parent folder UUID (parent_folder_id). Moving a folder containing 50,000 files requires updating exactly one row (parent_folder_id = :new_parent), completing in 1.2ms with zero recursion.
• Multi-Version Concurrency Control (MVCC): File records maintain historic version chains, enabling users to view, browse, and rewind folders to any historical timestamp (Point-in-Time Recovery).
• Chunk Catalog & Reference Counting: Tracks every physical chunk stored in S3, maintaining an atomic reference_count indicating how many file versions globally depend on that chunk.

COMPONENT 6: ASYNCHRONOUS CHUNK SWEEPER & GARBAGE COLLECTOR (GC)

The background reclamation worker fleet that purges unreferenced storage:

• Two-Phase Mark-and-Sweep: When files are deleted or overwritten, the database decrements chunk reference counters.
• 7-Day Safety Quarantine: Any chunk whose reference count reaches zero is marked with quarantined_at = NOW() and quarantined for 7 days. This grace window prevents race conditions where User B is currently uploading an identical chunk whose hash check matched 1 minute ago.
• Batched S3 Deletion: After 7 days, background sweepers issue batched DeleteObjects calls to S3, reclaiming petabytes of disk space safely.

COMPONENT 7: ASYNC ENRICHMENT & SEARCH WORKERS

The decoupled worker pool consuming the Kafka filesystem.changes.raw topic:

• Multi-Tier Thumbnail Generation: Generates web and mobile previews (64x64, 256x256, 1024x) for images, videos, and PDFs, storing thumbnails in S3 edge caches.
• Full-Text Inverted Indexing: Asynchronously extracts text from PDFs, Office documents, and text files, streaming tokens to an Elasticsearch cluster for sub-second document search.
• Malware & Virus Scanning: Runs asynchronous ClamAV and heuristic virus detection on newly uploaded chunks, flagging infected files before they synchronize to other team members' laptops.

ByteByteGo-Style Step-by-Step Chunked Delta Sync Flow

To master the Staff interview, trace a local file edit from the user's desktop to cloud storage and peer device hydration:

sequenceDiagram
    participant U as "User edits 1 byte in local file"
    participant C as "Client Sync Daemon"
    participant M as "API Gateway & Metadata Service"
    participant S3 as "Amazon S3 Bucket"
    participant N as "Notification Fleet"
    participant P as "Peer Client Daemon"
    Note over U,M: STEP-BY-STEP CHUNKED DELTA SYNCHRONIZATION FLOW
    Note over U,M: LOCAL EDIT & CDC DETECTION
    U->>C: (1) OS Kernel inotify fires (500ms debounce)
    Note over C: (2) Rabin CDC slices file into 4MB chunks
    Note over C: (3) Computes SHA-256 per chunk
    C->>M: (4) POST /v1/files/{id}/check-chunks (Submits list of chunk hashes)
    Note over M: (5) Queries Chunk Catalog: "Hash B is new, Hashes A, C, D already exist"
    M-->>C: (6) Returns S3 Presigned PUT URL strictly for missing Hash B
    Note over C,S3: DIRECT-TO-S3 STORAGE INGESTION
    C->>S3: (7) Direct HTTP PUT Hash B
    C->>M: (8) POST /v1/files/{id}/commit (Atomic CAS v12)
    Note over M,P: REAL-TIME PEER NOTIFICATION & HYDRATION
    M->>N: (9) Emits event to Kafka & Redis Pub/Sub
    N->>P: (10) Pushes WebSocket signal to Peer Laptops
    P->>M: (11) Wakes up#59; fetches manifest v12 from Metadata
    P->>S3: (12) Downloads only Hash B from S3#59; reassembles file

STEP 1: KERNEL EVENT INTERCEPTION & 500MS DEBOUNCE

The user modifies a byte in a local file. The operating system kernel hooks (inotify on Linux, FSEvents on macOS, ReadDirectoryChangesW on Windows) alert the background sync daemon. The daemon starts a 500-millisecond debounce timer, coalescing rapid successive micro-saves from editors and compilers into a single evaluation event.

STEP 2: CONTENT-DEFINED CHUNKING (RABIN FINGERPRINTING)

The client sync daemon runs the Rabin rolling polynomial algorithm across the file stream. It evaluates dynamic boundary criteria (Rabin_Hash & 0x3FFFFF == 0), slicing the file into variable-size chunks with an average target size of 4 Megabytes.

STEP 3: PARALLEL CRYPTOGRAPHIC SHA-256 HASHING

The daemon computes SHA-256 cryptographic hashes for every chunk using hardware-accelerated SIMD instructions. It compares these hashes against its embedded local SQLite database (local_chunks.db) to identify newly modified blocks.

STEP 4: CHUNK EXISTENCE QUERY (/CHECK-CHUNKS)

The client sends an HTTP POST request to the API Gateway / Metadata Service with the complete list of chunk hashes comprising the new file version: POST /v1/files/f_1048/check-chunks { hashes: ["hash_A", "hash_B", "hash_C"] }.

STEP 5: GLOBAL CAS DEDUPLICATION CATALOG CHECK

The Metadata Service checks the incoming hash array against its global Content-Addressable Storage (CAS) catalog in CockroachDB. It discovers that hash_A and hash_C already exist in cloud storage (uploaded by other users or prior versions), and only hash_B is missing.

STEP 6: S3 PRESIGNED PUT URL GENERATION

Rather than proxying large binary data through application memory, the Metadata Service generates a short-lived (15-minute TTL) presigned AWS S3 PUT URL strictly for the missing hash_B and returns it to the client: { missing: ["hash_B"], upload_urls: { "hash_B": "https://s3.amazonaws.com/cas-bucket/chunks/hash_B?AWSAccessKeyId=..." } }.

STEP 7: DIRECT-TO-S3 CLOUD BLOB TRANSPORT

The client daemon uploads the 4 MB binary payload for hash_B directly to Amazon S3 over TLS via HTTP PUT. Cloud network switches and S3 edge frontends absorb the bandwidth and disk I/O, completely bypassing internal backend app servers.

STEP 8: ATOMIC COMPARE-AND-SWAP (CAS) VERSION COMMIT

Once S3 confirms HTTP 200 OK for hash_B, the client issues a commit RPC: POST /v1/files/f_1048/commit { base_version: 11, new_version: 12, chunks: ["hash_A", "hash_B", "hash_C"] }. The metadata engine executes an atomic Compare-And-Swap transaction updating the version to 12. If a concurrent conflict occurs, it triggers the Conflicted Copy workflow.

STEP 9: INVALIDATION EVENT PUBLICATION

Upon committing Version 12, the Metadata Service publishes a lightweight event to Apache Kafka topic filesystem.changes.raw and broadcasts an invalidation notice across internal Redis Pub/Sub channels keyed by folder ID.

STEP 10: REAL-TIME EDGE WEBSOCKET INVALIDATION SIGNAL

The Edge Notification Fleet receives the Redis Pub/Sub broadcast and immediately pushes a compact 128-byte notification frame down open WebSockets to all authorized peer devices: { event: "FILE_UPDATED", file_id: "f_1048", version: 12 }.

STEP 11: PEER MANIFEST HYDRATION REQUEST

The peer device's background daemon receives the WebSocket doorbell signal. It issues a lightweight HTTP GET request to the Metadata Service: GET /v1/files/f_1048/manifest?version=12 to retrieve the new chunk sequence.

STEP 12: TARGETED DELTA CHUNK DOWNLOAD & LOCAL ASSEMBLY

The peer daemon inspects its local SQLite cache and discovers it already holds hash_A and hash_C. It downloads only missing chunk hash_B (4 MB) directly from S3, verifies its checksum, and splices the blocks together on local disk in under 2 seconds, completing end-to-end delta synchronization.