03 — Shell Stream Pipelines & Server Diagnostics

At a glance

After completing this lesson you can:

• Diagnose a wedged Linux server in under 60 seconds using stream pipelines, socket tables (ss -tulpn), and memory stats (/proc/meminfo).
• Compute latency percentiles (p50, p95, p99) and Top-K error offenders from gigabytes of Nginx/FastAPI access logs without loading the entire file into RAM.
• Spot and fix memory-exploding log aggregation code generated by AI assistants.

Time: ~45 minutes · Prerequisites: 02 — Filesystem & Permissions

Reading Companion

• Book: How Linux Works (3rd Edition) — Brian Ward
• Chapter 8 & 9: Resource Utilization, Network Diagnostics (ss, netstat, lsof), and Text Pipeline Filtering (grep, awk, sort, uniq -c).

Why this matters in production

At 03:00 AM, before Grafana dashboards even finish loading or when the metrics agent itself is down, an engineer on call must inspect raw access logs, active TCP sockets (ss -tan), and /proc/meminfo directly on the host. Knowing how to stream-parse structured logs and socket tables separates engineers who resolve incidents in 3 minutes from those who reboot servers blindly.

Deep dive

1. Why Average Latency Lies: Percentiles (p50, p95, p99)

If 99 requests take 5 ms and 1 request takes 10,000 ms (database lock timeout), the average latency is ~105 ms—hiding the fact that 1% of your users experienced a 10-second freeze! Backend SLAs are always defined using tail percentiles (p95, p99).

2. Inspecting Listening Sockets (ss -tulpn) and /proc/meminfo

• MemFree vs MemAvailable: Linux aggressively uses free RAM for the page cache (disk buffer). MemFree often looks tiny (200 MB) on a healthy 16 GB server! The real metric before OOM occurs is MemAvailable (free RAM + reclaimable page cache).
• Socket States (ESTABLISHED, TIME_WAIT, CLOSE_WAIT):
• High TIME_WAIT: Normal on high-throughput servers closing connections cleanly.
• High CLOSE_WAIT: Application bug! The remote client closed the connection, but your Python code forgot to call socket.close(), leaking file descriptors until the server hits Too many open files.

Hands on

python3 phase0-foundations/01-linux-and-shell/03-shell-pipelines-and-diagnostics/examples/01_demo.py
python3 phase0-foundations/01-linux-and-shell/03-shell-pipelines-and-diagnostics/examples/02_trap.py

AI Trap & Code Review

# BAD AI-GENERATED CODE — OOM ON LARGE LOG FILES & WRONG LINUX MEMORY METRIC
def check_server_health(meminfo_text: str, log_path: str) -> bool:
    lines = open(log_path).readlines()  # Loads 20 GB into RAM!
    errors = [l for l in lines if " 500 " in l]
    # Checks MemFree instead of MemAvailable!
    for line in meminfo_text.splitlines():
        if line.startswith("MemFree:"):
            free_kb = int(line.split()[1])
            return free_kb > 500_000 and len(errors) < 100
    return False

Exercises

1. Rung 1 (parse_meminfo_available_mb): Parse /proc/meminfo text and return MemAvailable in integer megabytes (kB // 1024).
2. Rung 2 (top_k_5xx_ips — Fix the AI Bug): Stream an iterable of access log lines ("<ip> <method> <path> <status> <latency_ms>") in $O(U)$ memory (where $U$ is unique offending IPs) and return the top k (ip, count) pairs for 5xx statuses (500–599), sorted by count descending then IP ascending.
3. Rung 3 (compute_latency_percentile): Given a list of latencies in milliseconds and a percentile p (0 < p <= 100), compute the nearest-rank percentile value.
4. Rung 4 (detect_close_wait_leaks): Parse ss -tan output lines ("<State> <Recv-Q> <Send-Q> <Local_Address:Port> <Peer_Address:Port>") and return a dict mapping local port (int) to the count of CLOSE-WAIT sockets, excluding ports with zero CLOSE-WAIT sockets.
5. Rung 5 (summarize_endpoint_slo): Aggregate access log lines by "<method> <path>", returning a dict with total, error_rate (fraction of 5xx rounded to 4 decimals), and p95_ms for each endpoint.

Wrapping up

Can you explain?

1. What is the difference between MemFree and MemAvailable in /proc/meminfo?
2. What does a spike of sockets in the CLOSE-WAIT state prove about your backend service?
3. Why is p99 latency far more useful than mean latency when debugging microservice cascades?

Further Reading & References

• Linux man-pages: proc(5), fcntl(2), signal(7)
• Julia Evans: Bounded pipelines and Unix streams
• Brendan Gregg: Linux Performance Analysis
• The Twelve-Factor App: XI. Logs as event streams