02 — The Python Data Model, Iterators & $O(1)$-Memory Generators

At a glance

After completing this lesson you can:

• Implement the Python Iterator protocol (__iter__, __next__, StopIteration) from scratch.
• Build lazy generator streaming pipelines (yield, yield from) that process multi-gigabyte log datasets in $O(1)$ constant memory.
• Prevent resource exhaustion and iterator consumption traps in web data streams.

Time: ~50 minutes · Prerequisites: 01 — References, Mutability & Memory

Reading Companion

• Book: Fluent Python (2nd Edition) — Luciano Ramalho
• Chapter 1: The Python Data Model (Special methods, dunder methods).
• Chapter 17: Iterators, Generators, and Classic Coroutines (Iterable vs Iterator, Generator functions, itertools).

Why this matters in production

If an endpoint or background job queries 1,000,000 events and executes [event.dict() for event in query], Python must allocate all 1,000,000 objects in RAM simultaneously, causing massive GC pauses and Linux OOM kills. Using streaming generators (yield) keeps memory usage flat at a few kilobytes regardless of whether the dataset is 10 rows or 100,000,000 rows.

Deep dive

1. Iterable vs Iterator Protocol

• Iterable: An object implementing __iter__() that returns an Iterator.
• Iterator: An object implementing __next__() that returns the next item or raises StopIteration. Every Iterator is also an Iterable whose __iter__() returns self.

Container (Iterable) ────__iter__()────► State Machine (Iterator)
                                            │
                                            ├── __next__() ──► Item 1
                                            ├── __next__() ──► Item 2
                                            └── __next__() ──► raise StopIteration

2. Generator Pipelines

Generators lazily pull data downstream one element at a time. raw_lines -> parse_json -> filter_errors -> batch_chunks

Hands on

python3 phase0-foundations/03-python-internals/02-data-model-and-iterables/examples/01_demo.py
python3 phase0-foundations/03-python-internals/02-data-model-and-iterables/examples/02_trap.py

AI Trap & Code Review

# BAD AI-GENERATED CODE — ACCIDENTAL EXHAUSTION OF GENERATOR
def log_and_process(stream):
    print("Total items:", len(list(stream)))  # EXHAUSTS the generator completely!
    for item in stream:                       # Loop runs ZERO times!
        save(item)

Exercises

1. Rung 1 (RangeStepIterator): Implement a custom iterator class RangeStepIterator(start, stop, step) implementing __iter__ and __next__.
2. Rung 2 (stream_csv_records — Fix the AI Bug): Fix an AI function that loaded an entire file into memory with .readlines(), converting it to yield cleaned records lazily line-by-line.
3. Rung 3 (chunk_stream): Build a generator function chunk_stream(stream, chunk_size) that yields lists of at most chunk_size items from any iterable without reading ahead.
4. Rung 4 (sliding_window_stream): Generator yielding fixed-size tuples (x[i], x[i+1], ...) over an iterable.
5. Rung 5 (fanout_stream): Tee an iterable into $N$ independent generators without buffering all past elements in memory unnecessarily.

Wrapping up

Can you explain?

1. Why does an Iterator raise StopIteration instead of returning None?
2. What happens to a generator function's local variables when execution reaches yield?
3. Why is calling list(gen) twice in a row dangerous?

Further Reading & References

• CPython Internals & Data Model Reference
• PEP 544 — Protocols: Structural subtyping (static duck typing)
• PEP 557 — Data Classes
• Python Garbage Collector (gc module) Architecture