Phase 4 — Distributed Systems, Caching, Queues, Workflows & Search

When a single database and synchronous HTTP request/response cycle are no longer enough, you enter distributed systems. Phase 4 covers the exact stack demanded by top backend teams: Redis, RabbitMQ, Celery, Temporal, Elasticsearch, and gRPC, grounded in Martin Kleppmann's Designing Data-Intensive Applications (DDIA).

Modules & Lessons

01 — Redis & Caching Strategies

• 01-redis-data-structures-and-patterns — Strings, Hashes, Sorted Sets, TTLs, Cache-Aside, Write-Through, and cache invalidation
• 02-stampede-locks-and-rate-limiting — Cache stampede (single-flight + jitter), distributed locks with Lua scripts, and Redis token bucket

02 — Message Brokers & Async Task Queues (RabbitMQ & Celery)

• 01-postgres-skip-locked-queues — Building a durable transactional job queue using PostgreSQL FOR UPDATE SKIP LOCKED
• 02-rabbitmq-exchanges-and-routing — AMQP, exchanges (direct, topic, fanout), consumer acks, prefetch limits, and Dead-Letter Queues (DLQ)
• 03-celery-in-production — Celery task design, canvas workflows (chain, group, chord), retries with exponential backoff, and visibility timeouts

03 — Durable Execution & Reliability Patterns (Temporal)

• 01-idempotency-outbox-and-sagas — At-least-once delivery, idempotency keys, Transactional Outbox pattern, and Saga compensating transactions
• 02-temporal-workflows-and-activities — Deterministic workflows vs non-deterministic activities, event history replay, timeouts, and heartbeats

04 — Search Engines (Elasticsearch)

• 01-inverted-index-and-mappings — How Elasticsearch/Lucene works: analyzers, tokenizers, dynamic vs explicit mappings, and index lifecycle
• 02-full-text-queries-and-aggregations — bool queries, BM25 relevance scoring, fuzzy matching, bucket/metric aggregations, and keeping ES synced with Postgres

05 — Microservices & Inter-Service Communication (gRPC)

• 01-modular-monolith-vs-microservices — Bounded contexts, splitting criteria, distributed monolith anti-patterns, and circuit breakers
• 02-grpc-and-protocol-buffers — Protobuf schema evolution, unary and streaming RPCs in Python, deadlines, and REST vs gRPC vs Events

Capstone Milestone: pulseforge/ v0.5

Scale PulseForge with Redis distributed rate-limiting and deduplication, Transactional Outbox publishing to RabbitMQ/Celery workers, Temporal durable incident-escalation workflows, Elasticsearch full-text log indexing, and a high-speed gRPC ingestion endpoint.