Design Pastebin

Pastebin lets a user upload a block of text (code, logs, configuration) and receive a short URL that anyone with the link can open. It resembles a URL shortener, with one key difference: the stored value is a body of up to several megabytes instead of a short string. That difference drives the main design decision, which is to store metadata and content in different systems.

Requirements

Functional requirements

• A user submits text and receives a unique URL.
• Anyone with the URL can read the paste. Pastes can be public, unlisted (reachable only by URL), or private (owner only).
• Pastes have an optional expiry (for example 10 minutes, 1 day, 1 month, never). Expired pastes are no longer served.
• Optional metadata: title, syntax language.
• Users can delete their own pastes.

Non-functional requirements

• Durability. A paste that was acknowledged must not be lost before it expires.
• High availability for reads. Reads should survive the loss of a node or an availability zone.
• Low read latency. Time to first byte in the tens of milliseconds for cached pastes.
• Size limit. A maximum paste size (for example 10 MB) bounds memory use and abuse.
• Read-heavy, with a long tail. Most pastes are read a few times. A few are read many times.

Out of scope: real-time collaboration, versioning, search across pastes.

Estimation

Assumptions: 1 million new pastes per day, an average size of 10 KB, a read-to-write ratio of 5:1, and retention of up to 5 years for pastes with no expiry.

• Write rate: 1,000,000 / 86,400 ≈ 12 writes per second. Peak at 3x is about 35 per second.
• Read rate: 12 x 5 = 60 reads per second on average, about 180 at peak.
• Storage per day: 1,000,000 x 10 KB = 10 GB per day.
• Storage for 5 years: 10 GB x 365 x 5 ≈ 18 TB of content.
• Metadata: about 200 bytes per paste (key, title, language, owner, timestamps, size, content pointer). 1,000,000 x 365 x 5 = 1.8 billion pastes, times 200 bytes ≈ 365 GB.
• Bandwidth: reads at 60 per second x 10 KB = 600 KB/s on average (peak about 1.8 MB/s). Writes are 12 x 10 KB = 120 KB/s. Both are small.
• Key space: 1.8 billion pastes fit within 7 base62 characters (62^7 ≈ 3.5 x 10^12).

The request rates are low. Content storage volume is the dominant concern: 18 TB is beyond a single database node, but within the range of object storage without effort. What each number implies:

• 12 writes/s and 180 reads/s at peak fit on a handful of stateless servers and one metadata primary, so no sharding is needed for throughput; sharding is optional headroom.
• 18 TB of content dominates cost and goes to object storage. At about $0.02 per GB-month that is roughly $360 per month per copy, so plain storage without dedup is cheap.
• 365 GB of metadata fits on one node with replicas, and its working set fits in memory. Keep the metadata store small by keeping bodies out of it.
• Cache size: if 10% of the 1 million daily pastes are hot (100,000 × 10 KB = 1 GB) a single small cache node holds the working set.
• Peak bandwidth of about 2 MB/s means the CDN is for latency and spikes, not for bandwidth relief. The real risk is a viral paste, not average load.

See back-of-envelope estimation.

API design

POST /api/v1/pastes
  headers: Authorization (optional), Idempotency-Key (optional)
  body: { "content": "...", "title": "...", "language": "rust",
          "visibility": "public|unlisted|private", "expires_in_seconds": 86400 }
  201 -> { "key": "aZ3k9Qx", "url": "https://paste.example/aZ3k9Qx" }
  413 -> content exceeds size limit

GET /api/v1/pastes/{key}
  200 -> { "key", "title", "language", "content", "created_at", "expires_at" }
  403 -> private paste, wrong user
  404 -> unknown
  410 -> expired or deleted

GET /raw/{key}
  200 -> text/plain body only

DELETE /api/v1/pastes/{key}
  204

The raw endpoint serves the body without JSON so it can be cached by a CDN and fetched by command-line tools. Syntax highlighting is done in the browser, not on the server. It keeps the stored data unmodified and moves CPU cost to the client.

Data model

Two stores with different requirements.

Metadata store. One row per paste:

The access pattern is a lookup by key plus a listing of pastes by owner. A relational database with a secondary index on (owner_id, created_at) fits: the metadata is about 365 GB for five years, which a single relational primary with replicas can hold, and a sharded cluster or a key-value store with a secondary index handles with more headroom. See SQL vs NoSQL.

Content store. Paste bodies are immutable blobs of variable size. Object storage (S3, GCS, or an internal system) suits them: it is cheap per gigabyte, replicates or erasure codes data for durability, and scales to petabytes (see object and blob storage). Storing 10 KB to 10 MB values in the metadata database instead would inflate rows, enlarge backups and replication traffic, and evict useful pages from the database buffer cache. Blobs are written once and never updated, which is what object storage handles best.

The object path is derived from the key, for example pastes/aZ/3k/aZ3k9Qx. Object stores hash-partition internally, so the prefix layout matters little in modern systems, but a key-derived path avoids a lookup.

High-level design

flowchart LR
    C[Client] --> CDN[CDN]
    CDN --> LB[Load balancer]
    LB --> API[Paste service]
    API --> KG[Key generation service]
    API --> MD[(Metadata DB)]
    API --> OS[(Object storage)]
    API --> CA[(Cache)]
    CL[Cleanup worker] --> MD
    CL --> OS

Component responsibilities:

• CDN. Caches GET /raw/{key} for public pastes. See content delivery networks.
• Paste service. Stateless. Validates size and visibility, coordinates the metadata and object writes, and enforces access control.
• Key generation service. Supplies unique keys as described in design a URL shortener: leased counter blocks or a pre-generated key pool.
• Metadata DB. Source of truth for existence, visibility, ownership, and expiry.
• Object storage. Holds paste bodies.
• Cache. Holds metadata and small bodies for hot pastes.
• Cleanup worker. Deletes expired metadata and objects in the background.

Write sequence:

sequenceDiagram
    participant C as Client
    participant P as Paste service
    participant K as Key service
    participant O as Object storage
    participant M as Metadata DB
    C->>P: POST paste
    P->>P: validate size and fields
    P->>K: next key
    K-->>P: aZ3k9Qx
    P->>O: put pastes/aZ3k9Qx
    O-->>P: ok
    P->>M: insert metadata if key absent
    M-->>P: ok
    P-->>C: 201 with URL

Read sequence: the service reads the metadata (cache first, then database), checks that the paste exists, has not expired, and that the requester may see it, then streams the body from object storage or the cache.

Deep dive

Consistency between metadata and content

A write touches two systems, so a crash between them leaves partial state. The order determines which failure appears.

• Object first, metadata second (used above). A crash after the object write leaves an orphan blob that no metadata points to. Users never see an inconsistent paste, because reads start from metadata. A periodic garbage-collection job lists objects older than a safety window whose keys have no metadata row and deletes them.
• Metadata first, object second. A crash leaves a row that points to a missing blob, and readers get errors. This needs a pending state and repair logic.

A third option is a two-phase write with a pending row, which enables a repair job to find half-finished writes, at the cost of an extra write. Distributed transactions across the database and object store are not available, so accept the orphan approach.

Numbers and thresholds: object stores such as S3 give strong read-after-write consistency for new objects, so a reader who sees the metadata row can immediately read the body. Set the orphan GC safety window to at least 24 hours (longer than any retry or in-flight upload) and run it daily. Expected orphans are tiny: only crashes between two steps a few milliseconds apart create them. If the metadata database uses asynchronous replicas, a read routed to a lagging replica right after creation may return 404; route reads of very new keys (less than a few seconds old, or for the creating client) to the primary.

Object first is simpler, and the only cost is cheap orphan cleanup. The metadata insert is the commit point: a paste exists if and only if its metadata row exists. Because a key is used once and content is immutable, retrying the object write is safe. Use the content_sha256 field to verify integrity on write and optionally on read.

Expiry

Each paste has an expires_at. Three mechanisms cooperate:

1. Read-time check. The paste service compares expires_at with the current time and returns 410 for expired pastes. This makes expiry exact regardless of cleanup timing.
2. Background cleanup. A worker scans an index on expires_at in small batches, deletes the object, then deletes the metadata row. Deleting the object first means a crash leaves a metadata row for an already-gone blob, which the read-time check hides and the next run removes. (Deleting the row first would orphan the blob.)
3. Cache and CDN TTL. The cache TTL is set to the smaller of a default TTL and the time remaining until expiry. CDN caching of expiring pastes uses Cache-Control: max-age bounded by the remaining lifetime.

Most pastes in a public service carry an expiry, so cleanup volume is significant: about 12 deletions per second on average. That is small for a batch worker.

Alternatives: per-row TTL in a store that supports it (DynamoDB, Cassandra) removes metadata for free but does not delete the blob, so pair it with an object-store lifecycle rule. Lifecycle rules operate at a granularity of days, so they cannot honor a 10-minute expiry alone, and the read-time check is still required for exactness. Choose the index scan plus read-time check: exactness from the check, cost control from the scan. Batch 500 to 1,000 rows per run, and run every minute so the backlog stays under a minute. Cache and CDN entries for a paste must not outlive expires_at; a deleted paste is purged from the CDN with an explicit invalidation, and until it completes (seconds to minutes) it can still be served, which is acceptable for public content but not for a takedown of sensitive content (use short TTLs for those).

Access control and enumeration

• Public and unlisted pastes rely on unguessable keys for unlisted access. Random or scrambled keys in a 3.5 x 10^12 space, with 1.8 billion pastes, mean a random guess hits a paste with probability about 0.05%. Rate limit reads per IP to make scraping expensive (see design a rate limiter). Unlisted is not private.
• Private pastes require authentication and an owner check on every read. They must not be stored in a shared CDN cache without an authorization mechanism. Mark responses with Cache-Control: private and exclude them from CDN caching. Use longer keys (for example 16 or more random characters) if a paste must be resistant to guessing.
• Encryption at rest is provided by the storage layer. For stronger privacy, clients can encrypt content and keep the key in the URL fragment, which browsers do not send to the server. The service then stores ciphertext it cannot read.

Large pastes and abuse

Enforce the size limit at the load balancer or gateway (Content-Length) and again in the service while streaming. Streaming the body directly to object storage avoids buffering 10 MB in service memory. For pastes above a threshold, the service can return a pre-signed upload URL so the client uploads directly to object storage and then calls the service to finalize. Spam and malware content require per-user and per-IP write rate limits, a blocklist, and reporting and takedown flows.

Deduplication

Storing identical content once saves space. Compute SHA-256 of the body and use it as a content-addressed object name, with reference counts in the metadata layer. Deduplication complicates deletion (an object can be deleted only when its reference count reaches zero) and helps mainly for repeated automated submissions. At 18 TB over five years, plain per-paste storage is affordable and simpler, so deduplication is optional.

Bottlenecks and scaling

• Paste service. Stateless and horizontally scalable. At tens to hundreds of requests per second, a few instances suffice. Deploy across availability zones.
• Metadata DB. 365 GB fits on a single node with replicas, but shard by key hash (sharding and partitioning) if growth demands it. Add read replicas for read scaling (database replication). The owner listing query needs a secondary index or a separate table keyed by owner.
• Object storage. Scales without application involvement. Use lifecycle rules to move old, unread pastes to cheaper tiers.
• Cache. A cache-aside layer for metadata and small bodies (for example bodies under 64 KB) covers the long tail's hot keys. Large bodies are served from object storage or the CDN.
• Hot pastes. A paste shared on social media can produce a sudden spike. CDN caching of the raw endpoint absorbs it, and origin requests are collapsed per key.
• Regional deployment. Serve reads from regional replicas and regional CDN edges. Writes go to a home region and replicate asynchronously.

Sharding and cache choices

• Shard key: paste key hash. Keys are random, so metadata spreads evenly. Sharding by owner makes the listing query local but creates a hot shard for a bot account with millions of pastes; sharding by time puts all new writes on one shard. If the owner listing is needed, keep a separate table keyed by (owner_id, created_at), updated asynchronously.
• Hot paste. One viral paste is one key on one shard and one cache node. CDN caching of the raw endpoint (public content, TTL of 1 to 60 minutes bounded by expiry) absorbs it; on a CDN miss, coalesce origin requests per key.
• Consistency. Strong for the create path (the creator reads their own paste), eventual elsewhere; pastes are immutable, so a stale cache entry is wrong only after deletion or expiry, which the TTL bounds.
• Cache sizing. Cache metadata and bodies under 64 KB with LRU and a TTL of an hour capped at remaining life; as computed, about 1 GB holds the hot set. Store bodies larger than that only in the CDN and object store.

Failure modes

The system degrades by losing writes before reads. At 10x (120 writes/s, 1,800 reads/s peak, 180 TB, 3.6 TB metadata): object storage still scales unchanged, but metadata needs sharding (for example 8 shards of about 450 GB), a real cache cluster replaces the single node, the CDN carries most reads, and dedup or lifecycle tiering of unread pastes becomes worth the complexity. At 100x, abuse control, not storage, is the hardest part.

Trade-offs and follow-up questions

• Why not store bodies in the database? It enlarges rows, replication traffic, and backups, and competes with metadata for cache memory. Object storage is cheaper and built for immutable blobs. Inline storage is acceptable for very small pastes if a threshold is used to choose the path.
• Object-first or metadata-first writes? Object-first leaves harmless orphans that garbage collection removes.
• How to support editing or versions? Store each version as a new immutable object and keep a version list in metadata. Never overwrite blobs in place, since caches and CDNs may serve the old body.
• How to support search? Add an asynchronous indexing pipeline into a full-text index (see inverted index and full-text search) fed by write events, for public pastes only.
• What if a single paste is 1 GB? Reject above the limit; if large files are a product requirement, switch to multipart pre-signed uploads directly to object storage and treat it as a file store.
• What if reads become 100x writes? Push more to the CDN and edge caches; the raw endpoint of immutable content is ideal for long TTLs.
• What if a user asks to delete a paste immediately? Mark it deleted in metadata (reads return 410 at once), delete the blob, and invalidate the CDN. The metadata check is the source of truth.
• What if the same content is pasted a million times? Dedup by content hash with reference counts, or accept the cost: 1 million × 10 KB is 10 GB, which is cheap.
• What if a paste contains malware or illegal content? Reporting flow, hash-based blocklist checked at write and on report, and takedown that deletes the blob and blocks the hash from re-upload.
• How to handle a region failure? Replicate metadata and objects to a second region, and fail reads over to it. The replication lag defines how many recent pastes might be missing. See multi-region and disaster recovery.