Asynchronism — Message Queues and Task Queues

Synchronous processing means the user waits while the server does all the work before responding. This is fine for fast operations but fails for slow ones: image processing, sending emails, generating reports, calling slow third-party APIs, or any operation that could take more than a second.

A synchronous endpoint that takes 5 seconds per request can only serve 200 requests per minute per server thread. Block the thread for 5 seconds and your server's capacity collapses. Worse, if the slow operation fails, the request fails — there is no retry mechanism built in.

Asynchronous processing solves this: the server receives the request, puts the work onto a queue, immediately responds "accepted" (HTTP 202), and returns the thread to the pool. A separate worker process picks up the job from the queue, does the slow work, and notifies the user when done (via polling, WebSocket, email, or push notification). The request thread is never blocked.

A message queue is a durable buffer that decouples producers (services that generate messages) from consumers (services that process them). Producers push messages into the queue without knowing who will process them or when. Consumers pull messages from the queue at their own pace.

Core properties: • Durability: messages are stored on disk so they survive process restarts. If the worker crashes mid-job, the message is requeued and retried. • Decoupling: producers and consumers can be scaled, deployed, and fail independently. • Buffering: if consumers are slower than producers, the queue absorbs the burst. Producers are never blocked by slow consumers. • Fan-out: one message can be delivered to multiple consumers (publish-subscribe pattern), useful for broadcasting events (e.g., "order placed" → billing service + notification service + analytics service all receive it).

Popular message queue systems: • RabbitMQ: implements AMQP protocol. Supports complex routing (exchanges, bindings, routing keys), message acknowledgements, dead-letter queues, and priorities. Good for task dispatch where routing flexibility matters. • Apache Kafka: distributed log. Consumers read at their own offset. Messages are retained for a configurable period (days or indefinitely). Extremely high throughput. Ideal for event streaming, audit logs, and multiple independent consumer groups reading the same events. • Amazon SQS: managed queue service. At-least-once delivery, up to 14 days retention, automatic scaling. Very low operational overhead. Standard queues are unordered; FIFO queues guarantee order within a message group. • Redis Streams: lightweight in-memory streams with consumer groups. Good for lower-throughput use cases or when Redis is already in the stack.

A task queue is a higher-level abstraction on top of a message queue. Instead of raw messages, you enqueue tasks — named functions with their arguments. Workers pick up tasks and execute them, with built-in retry logic, scheduling, result storage, and monitoring.

Celery (Python): the most popular task queue library. Supports Redis, RabbitMQ, and SQS as brokers. Built-in retry with exponential backoff, task scheduling (cron-like), result backend, chord/chain/group primitives for task orchestration, and Flower for monitoring.

Sidekiq (Ruby): high-performance task queue using Redis. Simple, reliable, and battle-tested at many large Ruby applications.

BullMQ (Node.js): Redis-backed task queue for Node.js with support for job prioritization, rate limiting, and delayed jobs.

Use a task queue (not a raw message queue) when you need: retry with backoff, scheduled/delayed execution, job deduplication, result storage, or a UI to monitor job progress and failures.

If producers generate work faster than consumers can process it, the queue grows unbounded. Eventually the queue exhausts memory or disk, and messages are dropped or the system crashes.

Back pressure is the mechanism by which a system signals upstream producers to slow down when downstream capacity is saturated.

Strategies: • Queue depth monitoring + auto-scaling: watch the queue depth metric and automatically add worker instances when it grows above a threshold. Kubernetes HPA with custom metrics, AWS SQS + Lambda auto-scaling, and ECS service auto-scaling all support this pattern. • Drop or reject: when the queue is full, reject new requests with HTTP 503 (service unavailable) and a Retry-After header. Clients with exponential backoff and jitter will retry at staggered times, naturally smoothing the load. • Rate limiting at the API gateway: prevent producers from submitting more work than the system can handle by enforcing per-client rate limits upstream. • Priority queues: ensure high-priority work (user-facing operations) is processed before low-priority work (analytics, batch jobs) even when the system is saturated.

Ignoring back pressure leads to unbounded queue growth, increasing memory use, increasing consumer lag, degrading latency, and eventually a full system failure. Design for it from the start.

Outbox pattern: when a service needs to both update its database and publish an event atomically, write the event to an "outbox" table in the same database transaction. A separate "outbox publisher" process polls the table and publishes events to the message queue. This guarantees exactly-once publishing relative to the database change without distributed transactions.

Idempotency: because message queues guarantee at-least-once delivery (messages may be delivered more than once in failure scenarios), consumer logic must be idempotent — processing the same message twice must produce the same result as processing it once. Common approach: store processed message IDs in a deduplication table and skip messages already seen.

Dead-letter queue (DLQ): if a message fails processing after N retries (e.g., due to a bug in consumer code or invalid data), move it to a DLQ instead of retrying infinitely. Alert on DLQ depth. Inspect and replay DLQ messages after fixing the underlying bug.

Event sourcing: store the entire history of state changes as an immutable sequence of events (the event log). The current state is derived by replaying events. Naturally integrates with Kafka. Enables audit logs, time-travel debugging, and multiple projections of the same events for different consumers.