Part 1 · 2 chapters · ~12 min
Processes and System Calls
Seeing what a process does from the outside: ps and top, /proc, strace and ltrace with timing, lsof for open files and sockets, ss for socket states and queues, file descriptor exhaustion, and macOS equivalents (dtruss, fs_usage, lsof).
3
strace and ltrace
code
strace -f -tt -T -p 4123 -e trace=network,read,write -o /tmp/trace.txt # follow threads, timestamps, durations strace -c -p 4123 # summary: which syscalls take the most time (Ctrl-C to print) ltrace -p 4123 -e malloc+free # library calls (slower still; for native code) # macOS: dtruss -p 4123 (needs SIP adjustments), fs_usage -w -f network 4123
WHAT STRACE SHOWS
a slow request, seen as system calls
swipe the figure sideways, or tap expand for full screen
1/4
accept and read
The process accepts a connection (fd 12) and reads the request. Fast.
accept and read: microsecondsstrace -f -tt -T -p <pid>
4
/proc, lsof and ss
code
cat /proc/4123/status | grep -E 'Threads|VmRSS|voluntary' # threads, memory, context switches ls /proc/4123/fd | wc -l ; cat /proc/4123/limits | grep 'open files' # fd usage vs limit lsof -p 4123 -a -i # its network sockets ss -tanp state established '( dport = :5432 )' | wc -l # connections to Postgres ss -ltn # listen sockets: Recv-Q > 0 on a listener = accept queue backing up ss -ti dst 10.0.4.7 # per-connection TCP info: rtt, retransmits, cwnd
| symptom | tool | reading |
|---|---|---|
| EMFILE "too many open files" | /proc/pid/fd, lsof | sockets or files leaked (not closed); raise the limit only after fixing the leak |
| connections refused under load | ss -ltn | listen backlog full: the app is not accepting fast enough |
| many TIME_WAIT sockets | ss -tan state time-wait | no keep-alive on outbound connections |
| many CLOSE_WAIT sockets | ss -tan state close-wait | the app never closes sockets the peer closed: a bug in your code |