Part 5 · 1 chapters · ~8 min

Object Storage Internals

The object model (buckets, keys, versions, immutable blobs), multipart uploads, metadata indexes and strong read-after-write consistency (S3 since December 2020), placement and erasure coding across zones, storage classes and lifecycle rules, conditional writes, MinIO and Ceph RGW, and building on object storage (lakehouses, WAL archives).

Blobs, keys and metadata

code

aws s3api put-object --bucket ledger-archive --key wal/2026/10/07/000000010000004A --body seg --checksum-algorithm SHA256
aws s3api put-object --bucket locks --key leader --body me --if-none-match '*'   # conditional write: create only if absent
aws s3api list-objects-v2 --bucket ledger-archive --prefix wal/2026/10/ --max-keys 1000

# lifecycle: move cold objects to cheaper classes, expire old versions
{ "Rules": [ { "ID": "archive-wal", "Filter": { "Prefix": "wal/" }, "Status": "Enabled",
  "Transitions": [ { "Days": 30, "StorageClass": "GLACIER_IR" } ],
  "NoncurrentVersionExpiration": { "NoncurrentDays": 90 } } ] }

Consistency: since December 2020 Amazon S3 provides strong read-after-write consistency for all operations, so a successful PUT is immediately visible to GETs and LISTs. Conditional writes (If-None-Match, added for PUT in 2024) let object storage act as a simple coordination primitive, which systems like table formats and some databases now use.

INSIDE AN S3-STYLE OBJECT STORE

a key-value store for big immutable blobs

swipe the figure sideways, or tap expand for full screen

1/4

the API

Objects are immutable blobs addressed by bucket and key; you replace, never edit in place. Large uploads use multipart: parts uploaded in parallel and committed together.

immutable blobs by keymultipart for large