File Storage and Sync: Interview Calibration, Opinions, and Runbooks

This lesson covers sections 7 to 10 and the chapter checklist of Chapter 12 and continues File Storage and Sync: Chunking, State Machine, and Deep Dives.

7. Staff Interview Calibration: Five Probing Questions

Probe 1: "Why does fixed-size chunking fail when a user edits 1 byte at the start of a large file?"

Senior Trap: "Fixed-size chunking works fine because modern cloud networks have plenty of bandwidth to re-upload the changed chunks."

Staff Answer: "Fixed-size chunking fails due to boundary shift. Slicing a 1 GB file into static 4 MB blocks means an insertion of 1 byte at the very beginning shifts every subsequent byte offset across the entire file. Because cryptographic hashes like SHA-256 exhibit the avalanche effect, every single chunk produces a brand-new hash, forcing a re-upload of all 1,000 Megabytes. We solve this using Content-Defined Chunking (CDC) with Rabin rolling polynomial hashes. In CDC, chunk boundaries are declared based on byte patterns in a 48-byte sliding window. A 1-byte insertion shifts only the immediate local chunk; the remaining 99.8% of chunks retain identical boundaries and SHA-256 hashes, requiring zero re-upload bandwidth."

Probe 2: "Can an attacker determine if a sensitive tax document exists on your platform by uploading its SHA-256 hash?"

Senior Trap: "No, because user files are isolated by account IDs in the database."

Staff Answer: "In naive client-side deduplication systems, yes. An attacker calls the chunk existence API with the document's known hash; if the server responds that the chunk already exists to skip upload, the attacker confirms the document's presence on the platform. We neutralize this vulnerability using Cryptographic Proof of Storage. Before the server links an existing CAS chunk to a user's account without an upload, it challenges the client to compute an HMAC over random byte offsets within the file. An attacker who merely guessed the hash cannot answer the challenge; only a user who physically possesses the file on disk can respond correctly."

Probe 3: "Two users edit the same PowerPoint presentation offline for 3 hours. How do you reconcile their changes without data loss?"

Senior Trap: "We merge their changes using Operational Transformation (OT) or Git-style 3-way merge."

Staff Answer: "Binary files like PowerPoint presentations cannot be merged via OT or CRDTs; attempting an automated binary merge causes catastrophic file corruption. We implement an Optimistic Concurrency Control (CAS) commit protocol backed by Conflicted Copy preservation. The first user to reconnect commits their changes cleanly via UPDATE files SET version = 11 WHERE version = 10. When the second user reconnects, their commit fails with HTTP 409 Conflict. The second user's client downloads the first user's Version 11 and saves their own local edits as a sibling file: 'Presentation (User B's conflicted copy 2026-09-03).pptx'. Both versions sync cleanly to S3, guaranteeing zero data loss."

Probe 4: "A user renames a top-level directory containing 100,000 files. How do you synchronize this without generating 100,000 database writes?"

Senior Trap: "We run an asynchronous background job to update the path string on all 100,000 file rows."

Staff Answer: "Storing absolute path strings in the database is an architectural anti-pattern. We model directory hierarchies as an Adjacency List Directed Acyclic Graph (DAG) where every file and folder stores only its parent's UUID (parent_folder_id). Renaming or moving a 100,000-file folder requires executing a single SQL update on the folder's parent pointer: UPDATE filesystem_nodes SET parent_folder_id = :new_parent WHERE node_id = :folder_id. The database transaction executes in 1 millisecond. Downstream clients receive a single notification event and execute an atomic local OS rename syscall, updating 100,000 files in under 5 milliseconds."

Probe 5: "A user has 500 GB of files in their cloud drive, but their laptop has only a 128 GB SSD. How does your sync client function?"

Senior Trap: "The user must manually choose which folders to download in the app settings (Selective Sync)."

Staff Answer: "Manual folder exclusion is outdated. Modern file sync engines implement On-Demand Virtual Filesystem Hydration using OS-level filesystem drivers (Cloud Filter API on Windows, File Provider on macOS). The client populates the local directory with dehydrated 'placeholder files' that display full names, sizes, and thumbnails, but occupy zero physical bytes on disk. When the user opens a file, the OS driver suspends the read syscall, invokes the sync client to download the required 4 MB chunks from S3 in real time, and transparently resumes the application."

8. Staff Insiders: Controversial Opinions

8.1 "The Hardest Problems in File Sync Live on the Client, Not the Cloud"

Backend engineers instinctively focus on cloud microservices, Kubernetes clusters, and distributed databases. In file sync, the server is comparatively simple: an S3 blob repository and a relational metadata catalog. The true engineering nightmare lives on the client: handling OS-level inotify kernel overflows, checkpointing upload state across sudden laptop lid drops, resolving file-lock contention with anti-virus scanners, and building virtual filesystem drivers. Dedicate your best systems engineers to the client daemon.

8.2 "Collaborative Text Merging (OT/CRDT) Does Not Belong in a General File Sync Engine"

Interview candidates frequently try to impress interviewers by suggesting CRDTs or Operational Transformation to merge conflicting files. This is a severe mistake. General file sync engines handle opaque binary files: zip archives, Photoshop PSDs, compiled binaries, and SQLite databases. You cannot merge two versions of a SQLite database or a JPEG. Implement Conflicted Copy preservation; leave collaborative document editing to dedicated web applications like Google Docs.

8.3 "Client-Side Deduplication Without Cryptographic Proof of Storage Is a Critical Security Hole"

Allowing clients to claim ownership of existing cloud chunks based solely on uploading a SHA-256 hash is an invitation to data exfiltration and side-channel snooping. If your platform supports client-side deduplication, cryptographic proof-of-storage challenge protocols are non-negotiable.

8.4 "Last-Write-Wins Without Conflicted Copies Is Inexcusable Negligence"

Any file sync architecture that silently overwrites an offline user's work based on timestamp ordering is broken by design. Users store their life's work in cloud storage. Conflicted copies may clutter folders, but clutter is easily resolved; permanently lost user data destroys brand trust forever.

8.5 "Never Route Raw File Upload Bytes Through Your Application Servers"

Streaming file bytes through API servers consumes memory, saturates network interfaces, and introduces artificial failure points. Application servers should strictly coordinate metadata and issue presigned S3 URLs. Let the cloud provider's globally distributed edge network handle multi-gigabyte raw byte ingestion.

8.6 "Building a Custom Distributed Object Store on Raw Hard Drives Is Organizational Hubris"

Unless you operate at the astronomical physical scale of Dropbox's Magic Pocket (hundreds of petabytes where AWS S3 margins justify building custom multi-datacenter SAN hardware), do not build your own blob storage engine. Rely on Amazon S3, Google Cloud Storage, or Azure Blob Storage. They provide 11 nines of durability, automatic cross-region replication, and automated hardware lifecycle management that no startup engineering team can replicate reliably.

9. The Architectural Fault Lines & Default Staff Positions

Decision	Fault Line	Who Pays	The Real Operational Cost
Content-Defined vs Fixed Chunking	Client CPU Utilization vs Network Bandwidth	Client Battery	Computing Rabin sliding fingerprints consumes ~15% client CPU during initial hashing.
Direct-to-S3 vs App Server Proxy	Architecture Simplicity vs Infrastructure Scale	Security Surface	Managing short-lived presigned S3 tokens requires strict IAM credential management.
Conflicted Copies vs Last-Write-Wins	Disk Space & Clutter vs Data Loss Risk	User Clutter	Users must occasionally manually review and clean up duplicate conflicted files.
Virtual Filesystem vs Full Local Sync	OS Driver Complexity vs Local Disk Exhaustion	Client Engineering	Writing kernel-level Cloud Filter drivers on Windows and macOS is notoriously difficult.
Global Deduplication vs Zero-Knowledge E2E	Storage Cost Savings vs Cryptographic Privacy	Cloud Storage Bill	Zero-knowledge encryption disables cross-user deduplication, increasing S3 storage by 40%.

DEFAULT STAFF POSITIONS

1. Chunking Engine: Default: Rabin Content-Defined Chunking (average 4 MB chunk target). When to deviate: Fixed-size chunking strictly for read-only static ISOs.

2. Binary Transport Path: Default: Direct Client-to-S3 uploads via presigned PUT URLs. When to deviate: Internal enterprise on-premise air-gapped deployments only.

3. Conflict Resolution: Default: Atomic CAS commit; failed commits create Conflicted Copies. When to deviate: Never use blind Last-Write-Wins for binary files.

4. Directory Hierarchy Model: Default: Adjacency List DAG with parent UUIDs (O(1) folder renames). When to deviate: Never store full absolute path strings in database columns.

5. Local Disk Management: Default: Virtual filesystem on-demand hydration (placeholder files). When to deviate: Full sync for small accounts (<10 GB total drive usage).

10. Failure Modes, 3 AM Incident Runbooks & Active Crisis Drills

Incident 1: S3 Presigned URL Mass Expiration & 403 Upload Avalanche

The Failure: A large user base attempts to upload a 20 GB video file over a slow hotel Wi-Fi connection. The 15-minute expiration timer on the S3 presigned PUT URLs lapses mid-upload, causing Amazon S3 to return continuous HTTP 403 Forbidden errors.

The Threat: Thousands of client sync engines enter uncoordinated retry loops, repeatedly requesting new presigned tokens and re-uploading from byte 0, saturating metadata servers.

Runbook Action:

1. Detect the failure: monitor alert s3_presigned_upload_403_rate > 5%.
2. Implement Dynamic Token Refreshing in Client Daemon:
   • Client refreshes presigned URLs 2 minutes prior to expiration.
   • Failed chunk uploads resume strictly from the failed 4 MB block, never from byte 0!
3. Increase presigned URL TTL dynamically via feature flag to 60 minutes during peak hours: storage-admin --set-presign-ttl 3600

Incident 2: Metadata DB Deadlock on Batch Directory Move

The Failure: An enterprise administrator renames a shared top-level directory containing 80,000 files while hundreds of concurrent users are uploading files inside child folders. The database transaction deadlocks on child foreign key checks.

The Threat: Metadata database connection pool exhausts; global file sync halts for all enterprise users.

Runbook Action:

1. Kill deadlocked transactions: execute SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE wait_event_type = 'Lock';.
2. Engage Asynchronous Hierarchical Lock Acquisition:
   • Directory moves must acquire an exclusive distributed lease in Redis on the root folder ID: SET lock:folder:<id> EX 30 NX
   • Child file commits check the lease; if locked, they queue for 500ms instead of executing conflicting SQL mutations.

Incident 3: Corrupted Chunk Bit-Rot in CAS Blob Store

The Failure: A hard drive bit-flip on underlying cloud storage alters 3 bytes inside chunk 0x3FA9....

The Threat: Clients downloading the file calculate a mismatched SHA-256 hash and fail to reconstruct the file.

Runbook Action:

1. Identify the corrupt chunk via client integrity error reports (client_merkle_mismatch).
2. Restore the chunk from Cross-Region Replication (CRR): Amazon S3 replicates all objects to a secondary region (e.g., us-west-2 to us-east-1).
3. Run automated repair script to overwrite the corrupt object from the replica bucket: aws s3 cp s3://backup-bucket/chunks/0x3FA9 s3://primary-bucket/chunks/0x3FA9

Crisis Drill 1: 50,000-File Inotify Flood Drop

The Scenario: A software developer clones a giant monorepo into their local sync directory, triggering 50,000 filesystem events that overflow the Linux inotify kernel queue.

The Mitigation Protocol:

1. The client daemon catches the IN_Q_OVERFLOW kernel signal.
2. It temporarily pauses the event listener and initiates a Merkle Tree Reconciliation Scan.
3. It walks the local filesystem tree, compares local modification timestamps with its embedded SQLite database, identifies changed files, and resumes standard CDC chunking without missing a single file.

Crisis Drill 2: Two Users Offline for 3 Days Editing Same 2 GB Archive

The Scenario: Two engineers edit the same 2 GB compressed archive offline for 3 days. Both reconnect simultaneously upon landing.

The Mitigation Protocol:

1. User A's commit wins via atomic CAS.
2. User B's commit receives HTTP 409 Conflict.
3. User B's sync client downloads User A's version and creates: Archive (User B's conflicted copy 2026-09-03).zip.
4. Both 2 GB files reside safely in S3. Zero data is lost.

Crisis Drill 3: Ransomware Mass Encryption Detection and Rollback

The Scenario: A ransomware worm infects a user's workstation and encrypts 20,000 files in 3 minutes, uploading scrambled ciphertexts to the sync platform.

The Mitigation Protocol:

1. High-entropy upload velocity triggers the Automated Account Kill-Switch, locking the sync connection.
2. The user logs into the web console from a clean device and selects Account Rollback.
3. The platform executes a point-in-time snapshot restore: POST /account/restore?time=1_hour_ago.
4. The database re-points current version pointers to pre-attack snapshots, restoring 100% of user files in seconds!

Chapter Summary Checklist

STAFF CHECKLIST: Distributed File Storage & Delta Sync

When designing or defending a cloud storage and file sync engine in an L6 loop:

• Content-Defined Chunking: Did you implement Rabin rolling fingerprints to prevent the 1-byte insertion boundary-shift disaster?
• Direct-to-S3 Uploads: Do binary uploads stream directly to object storage via presigned PUT URLs to avoid saturating API servers?
• Conflicted Copy Preservation: Does your conflict model preserve both versions of conflicting binary files rather than using blind Last-Write-Wins?
• Adjacency List DAG: Are directory hierarchies modeled with parent UUIDs to ensure 100,000-file folder moves execute as an O(1) single SQL update?
• Content-Addressable Deduplication: Are chunks stored in S3 by SHA-256 hash to eliminate redundant storage across files and users?
• Cryptographic Proof of Storage: Is client-side deduplication protected by challenge-response protocols to prevent hash guessing leaks?
• Virtual Filesystem Hydration: Can the client display dehydrated placeholder files to support multi-terabyte cloud drives on small SSDs?
• Crash-Resilient State Machine: Does the client daemon checkpoint upload progress in local SQLite to survive sudden laptop lid closures?
• Two-Phase Garbage Collection: Are unreferenced chunks quarantined for a 7-day grace window before physical deletion to prevent in-flight upload race conditions?
• Disaster Snapshot Rollback: Can the platform execute point-in-time snapshot rollbacks to neutralize ransomware mass-encryption attacks?