Object and Blob Storage

Object storage stores data as immutable objects in a flat namespace, each addressed by a key and accompanied by metadata, and accessed through an HTTP API. It is the storage model behind Amazon S3, Google Cloud Storage, and Azure Blob Storage. It is designed for very large volumes of unstructured data: images, videos, backups, logs, and data-lake files.

Block, file, and object storage

Object stores trade in-place mutation and low latency for scale, durability, and low cost per byte. A database should not be placed on an object store directly, but object stores hold database backups and are the base for many analytics engines.

Object model

• A bucket is a top-level container that has a globally unique name and holds configuration: access policy, region, lifecycle rules, encryption.
• An object is bytes plus metadata, identified by (bucket, key). Keys such as photos/2026/09/img1.jpg look like paths, but the namespace is flat. The / is a delimiter that LIST can use to group results as prefixes.
• Operations: PUT, GET (optionally with a byte range), DELETE, HEAD (metadata only), LIST by prefix.
• Objects are immutable. Updating means writing a new object under the same key. With versioning enabled, older versions are retained and a delete adds a marker.
• Access uses per-object or per-bucket policies, and temporary pre-signed URLs that let a client upload or download one object directly without holding credentials.

Architecture

An object store separates the metadata path from the data path.

flowchart TB
    Client --> FE["Front-end / API servers: auth, routing, request parsing"]
    FE --> MS["Metadata service: key to chunk locations, versions, ACLs"]
    FE --> PL["Placement service: choose storage nodes"]
    FE --> DN1["Storage node 1"]
    FE --> DN2["Storage node 2"]
    FE --> DN3["Storage node 3"]
    MS --> MDB[("Partitioned metadata database")]
    DN1 --- RP["Background: repair, scrubbing, garbage collection"]
    DN2 --- RP
    DN3 --- RP

Front end. Stateless servers behind a load balancer authenticate the request, check policy, and coordinate the read or write.

Metadata service. Maps (bucket, key, version) to object attributes and the list of chunks with their storage locations. It is a partitioned, replicated database sharded by bucket and key (see sharding and partitioning). Ordered storage is needed to serve LIST by prefix efficiently. Metadata is small relative to data but receives most operations, so it is often the scaling bottleneck of the system.

Data store. Many storage nodes hold the bytes on local disks. A placement service chooses nodes, spreading replicas or fragments across failure domains: disks, servers, racks, and availability zones.

Write path

1. The client sends PUT with the object.
2. The front end splits the data into chunks if it is large.
3. Each chunk is written to several nodes (replicas or erasure-coded fragments).
4. After the required number of nodes acknowledge, the front end writes the metadata record.
5. The metadata commit makes the object visible. The client receives success.

Ordering matters: data is written before metadata, so a failure in between leaves orphaned chunks that garbage collection removes, but never a metadata record pointing to missing data.

Read path

The front end looks up the metadata, then fetches the chunks from a healthy node (for byte-range reads, only the needed chunks) and streams them to the client.

sequenceDiagram
    participant C as Client
    participant F as Front end
    participant M as Metadata service
    participant D as Storage nodes
    C->>F: GET /bucket/photos/img1.jpg
    F->>M: lookup key
    M-->>F: chunk list and node locations
    F->>D: read chunks
    D-->>F: chunk data
    F-->>C: object bytes

Chunking

Large objects are divided into chunks (also called parts, extents, or blocks) of a fixed or bounded size, typically megabytes. Chunking:

• Lets one object spread across many nodes and be read in parallel.
• Bounds the unit of replication and repair.
• Enables range reads and resumable transfers.
• Enables deduplication when chunks are addressed by content hash, in systems that choose to do so.

Very small objects are inefficient with per-chunk overhead, so some designs pack many small objects into larger container files (Facebook's Haystack does this for photos to reduce metadata operations).

Durability: replication vs erasure coding

Durability is the probability that stored data is not lost over a period. Object stores advertise very high figures (S3 Standard states eleven nines, 99.999999999%, of annual object durability). It is achieved by redundancy across failure domains, plus continuous checks and repair.

Replication

Store n full copies (commonly 3) on different nodes and zones.

• Simple; reads served from any copy; fast repair by copying one replica.
• Storage overhead is n times the data (3x for three copies).

Erasure coding

Erasure coding splits data into k data fragments and computes m parity fragments from them (Reed-Solomon codes are the common family). Any k of the k + m fragments can reconstruct the data, so the system tolerates the loss of any m fragments.

• Smallest example: k = 2, m = 1. Fragments A and B hold the two halves of the data, and the parity fragment is P = A XOR B. If A is lost, A = B XOR P. If B is lost, B = A XOR P. Any 2 of the 3 fragments reconstruct the data, at 1.5x overhead. Reed-Solomon generalizes this to arbitrary m.
• Storage overhead is (k + m) / k. For k = 6, m = 3: 9 fragments for 6 units of data, 1.5x overhead, tolerating 3 lost fragments. Three-way replication tolerates 2 lost copies at 3x.
• Reading a healthy object needs k fragments from k nodes. Rebuilding a lost fragment needs reading k others, so repair moves more data than replica copying.
• Encoding and decoding consume CPU, and small objects are inefficient to encode.

Many systems use replication for recently written or hot data and convert to erasure coding as the data cools.

Integrity

Each chunk stores a checksum. Reads verify it, and background scrubbing rereads data to detect silent corruption (bit rot). Detected corruption triggers repair from other copies or fragments. When a node or disk fails, a repair process recreates the lost redundancy on other nodes.

Consistency

S3 provides strong read-after-write consistency for object PUT and DELETE and for LIST, since December 2020. Earlier it offered eventual consistency for overwrites and deletes. A design without a strongly consistent metadata service, where readers may see stale metadata, must offer weaker guarantees, so the metadata store is the component that decides the consistency model.

Multipart upload

Uploading one very large object in a single request is fragile: a network error near the end forces starting over, and a single connection limits throughput. Multipart upload splits the upload into parts:

1. Initiate. The client requests an upload and receives an upload ID.
2. Upload parts. The client uploads numbered parts, each independently and in parallel, and retries only failed parts. The server returns an ETag (checksum) per part.
3. Complete. The client sends the list of part numbers and ETags. The server assembles them into one object and commits the metadata atomically.
4. Abort. Uploaded parts of an unfinished upload consume storage until aborted, so lifecycle rules should remove stale incomplete uploads.

sequenceDiagram
    participant C as Client
    participant S as Object store
    C->>S: initiate multipart upload
    S-->>C: upload ID
    C->>S: upload part 1
    C->>S: upload part 2
    C->>S: upload part 3
    Note over C,S: parts can be sent in parallel and in any order
    S-->>C: ETag per part
    C->>S: complete (part numbers and ETags)
    S-->>C: object created

In S3, each part except the last must be at least 5 MiB, a multipart upload may have up to 10,000 parts, and a single object may be up to 5 TiB.

Clients uploading from browsers or mobile apps use pre-signed URLs for each part, so file bytes bypass application servers. See design Google Drive and design YouTube for applications of this.

Storage classes and lifecycle

Object stores offer classes that trade retrieval latency and cost for storage price: frequently accessed, infrequent access, and archival tiers. Lifecycle rules transition objects between classes or expire them by age or prefix. Since delivery to users at scale is usually paired with a content delivery network, the object store acts as the origin.

When to use and not use

Use for large immutable blobs, media, backups, logs, static website assets, and data lake files.

Avoid for:

• Frequently updated small records or transactional data.
• Workloads needing POSIX semantics such as atomic rename, file locks, or append.
• Low-latency random access to small ranges.

Interview questions

• Why store objects and metadata separately? Metadata is small, needs ordered, transactional access for lookups and listings, and is updated often. Data is large and immutable and needs cheap, scalable bulk storage. Separating them lets each scale with its own technology.
• Why is erasure coding cheaper than replication and what does it cost? It gives comparable failure tolerance at 1.2x to 1.5x storage instead of 3x, at the cost of CPU for encoding and more network traffic during repair and degraded reads.
• How does multipart upload help with reliability? Parts upload independently and in parallel, only failed parts are retried, and the object becomes visible only when the complete request commits it.
• How does a LIST with prefix work on a flat namespace? The metadata store keeps keys in sorted order, so a prefix maps to a contiguous key range that is scanned. A delimiter groups keys sharing a prefix into a single common-prefix entry.