Part 4 · 1 chapters · ~8 min

Disks

Operations and throughput, latency and queueing, iostat columns explained, SSD behaviour (garbage collection, write amplification), cloud volume limits and burst credits, biolatency histograms, biosnoop per I/O, and fio for capacity testing.

5

Operations, latency, queues

code
iostat -xz 1
# Device   r/s    w/s   rkB/s   wkB/s  r_await w_await aqu-sz %util
# nvme1n1  120   3000    480   24000     0.4    9.8    29.4  100.0    ← w_await 9.8 ms with 29 queued: saturated at 3,000 writes/s
biolatency-bpfcc -D 10 1          # per-disk latency histograms
biosnoop-bpfcc | head             # each I/O: process, disk, sector, bytes, latency
# cloud: compare 3,000 w/s with the volume's provisioned IOPS (gp3 baseline is 3,000 IOPS)
DISKS: WHAT TO CHECK
iostat columns and what they mean
r/s, w/sOperations per second: comparewith the device or volume IOPSlimit.rkB/s, wkB/sThroughput: compare with bandwidthlimits.r_await, w_awaitAverage latency includingqueueing, in ms.aqu-szAverage queue length: saturation.%utilTime busy: misleading on SSDs andarrays.latency histogramsbiolatency: averages hide bimodallatency.
swipe the figure sideways, or tap expand for full screen
1/4
limits
Cloud volumes have provisioned IOPS and throughput; hitting them looks like rising latency at a flat ceiling of operations per second.
flat ceiling = a limitcheck volume IOPS