02 — Filesystem, Inodes, Permissions & /proc Inspection

At a glance

After completing this lesson you can:

• Explain how the Linux Virtual Filesystem (VFS) separates directory filenames (dentries) from file metadata and disk blocks (inodes).
• Write crash-safe files using atomic rename (os.replace) so readers never see half-written or truncated JSON/config files.
• Diagnose the classic production outage where df -h shows 100% disk full even after rm /var/log/huge.log was executed.

Time: ~45 minutes · Prerequisites: 01 — Processes & Streams

Reading Companion

• Book: How Linux Works (3rd Edition) — Brian Ward
• Chapter 3: Devices, Links, and Filesystems (Inodes, hard links vs symbolic links, permissions rwx / octal 0o600).
• Chapter 8: /proc filesystem (/proc/<pid>/fd, /proc/<pid>/status).

Why this matters in production

Imagine a production server runs out of disk space (No space left on device). A junior developer SSHes in, sees a 40 GB app.log, runs rm app.log, and celebrates—except df -h still reports 100% disk usage and PostgreSQL crashes!

Why? Because in Linux, a filename is just a pointer (hard link) to an inode. As long as the running Python/Gunicorn process holds an open file descriptor to that inode, the kernel cannot reclaim the 40 GB of blocks on disk.

Deep dive

1. Filenames, Inodes, and Open File Descriptors

Directory Entry ("app.log") ──┐
                              ▼
                        [ Inode #94821 ]
                        - link_count: 1  (becomes 0 after `rm`)
                        - open_fd_count: 1 (held by PID 4120)
                        - size: 40 GB
                        - permissions: 0640
                              │
                              ▼
                     [ Disk Data Blocks ]

The kernel only frees disk blocks when both link_count == 0 AND open_fd_count == 0. To free space without restarting the process, either send a log-reopen signal (SIGUSR1) or truncate the inode in place (: > app.log or os.truncate).

2. Why open("state.json", "w") Corrupts Data

When you call open("state.json", "w"), the kernel immediately truncates the file to 0 bytes (O_TRUNC) before your code writes a single character. If:

1. Another thread/process reads state.json at that microsecond, it reads an empty/partial file (JSONDecodeError).
2. Your process crashes or loses power mid-write, the file is permanently destroyed.

The Production Fix — Atomic Write via Temp File + os.replace(): Write to a temporary file in the same directory (same filesystem/mount point so it shares the inode table), flush() and os.fsync() the descriptor, then call os.replace(tmp_path, target_path). On POSIX, rename(2) within the same filesystem is an atomic kernel operation: readers either see the old complete file or the new complete file, never a half-written one.

Hands on

python3 phase0-foundations/01-linux-and-shell/02-filesystem-and-permissions/examples/01_inodes_and_atomic_write.py
python3 phase0-foundations/01-linux-and-shell/02-filesystem-and-permissions/examples/02_deleted_open_file_trap.py

AI Trap & Code Review

# BAD AI-GENERATED CODE — NON-ATOMIC WRITE & WORLD-READABLE SECRETS
import json

def save_api_credentials(path: str, creds: dict) -> None:
    with open(path, "w") as f:
        json.dump(creds, f)

What a Mid-Level Engineer catches in Code Review:

1. Non-atomic truncation: Truncates path in-place; concurrent readers or mid-write crashes corrupt the file.
2. Insecure default permissions (umask): Standard open() creates files with mode 0666 & ~umask (typically 0644, readable by every user on the Linux host!). Credentials and private keys must be created with 0o600 (rw-------).

Exercises

Run tests with:

python3 -m unittest phase0-foundations/01-linux-and-shell/02-filesystem-and-permissions/test_exercises.py -v

1. Rung 1 (format_octal_permissions): Convert a Unix permission bitmask (e.g., 0o750, 0o600, 0o644) into its 9-character symbolic string ("rwxr-x---", "rw-------", "rw-r--r--").
2. Rung 2 (atomic_write_json — Fix the AI Bug): Replace the buggy in-place open(path, "w") with a crash-safe atomic write in the same directory using a temporary file and os.replace.
3. Rung 3 (audit_secret_file_permissions): Inspect a file path using os.stat; return True if and only if the file is owned by the current UID and has zero group/other permissions (mode & 0o077 == 0).
4. Rung 4 (find_deleted_open_files): Given a simulated /proc/<pid>/fd symlink map (dict[int, str]), return a sorted list of file descriptors pointing to unlinked files (targets ending with " (deleted)").
5. Rung 5 (safe_resolve_within_base): Prevent Path Traversal (../) and Symlink Escape vulnerabilities: resolve user_rel_path inside base_dir using pathlib.Path.resolve() and raise PermissionError if the resolved path escapes base_dir.

Wrapping up

Can you explain?

1. Why must the temporary file in an atomic write be created in the same directory as the target file rather than /tmp?
2. What is the difference between a hard link and a symbolic (soft) link at the inode level?
3. Why does SSH refuse to use ~/.ssh/id_ed25519 if its permissions are 0644?

Further Reading & References

• Linux man-pages: proc(5), fcntl(2), signal(7)
• Julia Evans: Bounded pipelines and Unix streams
• Brendan Gregg: Linux Performance Analysis
• The Twelve-Factor App: XI. Logs as event streams