Part 4 · 1 chapters · ~8 min

Pipes and Processes

How the shell runs pipelines (pipe, fork, dup2, exec), concurrency inside pipelines and backpressure, SIGPIPE, exit statuses and pipefail, redirections (>, >>, 2>&1, <(), here-strings), subshells and command substitution, jobs and signals (Ctrl-C, Ctrl-Z, bg, fg, nohup), xargs and GNU parallel, and process inspection (ps, pgrep, lsof).

Plumbing

code

cmd > out.log 2>&1                # stdout to file, then stderr to where stdout points (order matters)
cmd 2>&1 | tee run.log             # see it and keep it
diff <(sort a.txt) <(sort b.txt)   # process substitution: compare two command outputs
read -r count <<< "$(wc -l < app.log)"

yes | head -1                      # yes would run forever; head exits, yes gets SIGPIPE and stops
long_job &  jobs  fg %1            # background, list, foreground; Ctrl-Z suspends, bg resumes
nohup ./worker > worker.log 2>&1 &  # survive logout (for anything real, use systemd or a supervisor)

# parallelism from the shell: 8 at a time
cat ids.txt | xargs -P 8 -n 1 ./reprocess.sh
lsof -p "$(pgrep -o node)" | wc -l   # descriptors held by a process
lsof -i :8080                        # who is listening on a port

WHAT THE SHELL DOES FOR `grep ERROR app.log | wc -l`

pipes, forks and file descriptors

swipe the figure sideways, or tap expand for full screen

1/4

a pipe

pipe() creates a kernel buffer with a read end and a write end, as two file descriptors (C course P10 built this in a shell).

a buffer with two endspipe()