Part 4 · 1 chapters · ~8 min

Locks and Contention

The signature of contention (latency up, CPU flat), thread dumps on the JVM, Go mutex and block profiles, async-profiler lock mode, database lock views, connection pool waits as contention, and fixes from shorter critical sections to removing shared state.

9

Seeing waiting

code
# JVM: three thread dumps, 5 s apart; threads stuck on the same lock across all three are the problem
for i in 1 2 3; do jcmd <pid> Thread.print > dump.$i.txt; sleep 5; done
grep -A3 'BLOCKED' dump.*.txt | grep 'waiting to lock' | sort | uniq -c | sort -rn | head

# Go: enable and fetch mutex and block profiles
runtime.SetMutexProfileFraction(5); runtime.SetBlockProfileRate(1_000_000)
go tool pprof http://localhost:6060/debug/pprof/mutex
LOCK CONTENTION
many threads, one lock, and latency that grows with load
worker 1holding lock 50 msworker 2waitingworker 3waitingworker 4waitingthe lockglobal cache mutex / hot row / advisory lock
swipe the figure sideways, or tap expand for full screen
1/4
the shape
Latency rises with concurrency while CPU stays low: threads are waiting, not working. The lab /contention route simulates it with a 50 ms critical section.
latency up, CPU flat: waitinglab: /contention