Field guide · message brokers
Topics, consumer groups, workers, ordering, retries — the same idea wears a different name in every broker. Here are Kafka, NSQ, NATS (JetStream), RabbitMQ, and AWS lined up so the mental model carries across.
verifiedNSQ delivery is push, not pull — a consumer advertises a RDY count and nsqd pushes up to that many messages (flow-controlled push). Comparisons that call it “pull” are wrong.
A durable, replayable event log.
Producers append events to a log that sticks around; many independent teams read it at their own pace and can rewind to re-read history — think a DVR for event streams.
Reach for it when high throughput, ordered replay, and many consumers of the same data.
A dead-simple work queue.
Push jobs in, add workers to go faster. No partitions, no ceremony — the broker spreads messages across whatever workers are connected.
Reach for it when straightforward task-spreading where you don’t need ordering or replay.
A fast bus that can grow durable.
Core NATS is lightweight fire-and-forget pub/sub; turn on JetStream and you add persistent streams, acknowledgements, and replay — same system, more guarantees.
Reach for it when low-latency messaging that may later need durability and replay.
The flexible router.
An exchange decides which queues each message lands in (direct / topic / fanout), with per-message retries and dead-lettering. The Swiss-army knife of routing.
Reach for it when complex routing rules and classic work-queue patterns.
AWS’s event bus and router.
Your services and SaaS apps emit events; rules match them and forward to targets — a smart switchboard for events across the whole account.
Reach for it when you need to route events between many services by rules.
AWS’s pub/sub fan-out.
Publish once to a topic and it pushes copies to many subscribers at once — usually SQS queues, Lambdas, or webhooks.
Reach for it when one event must fan out to many subscribers immediately.
AWS’s managed queue.
It buffers messages durably until a worker pulls and deletes them; the visibility timeout and DLQ handle retries. One queue = one consumer group.
Reach for it when you need a durable buffer between producers and workers.
AWS’s serverless worker.
It polls SQS (or is invoked by SNS / EventBridge) and auto-scales by concurrency — no servers to run, pay per execution.
Reach for it when you want workers that scale themselves with no servers to operate.
| Concept | Kafkalog + partitions | NSQsimple queue | NATSJetStream | RabbitMQAMQP broker | AWSEB · SNS · SQS · λ |
|---|---|---|---|---|---|
| Topic — where producers publish | Topic | Topic | Subject (captured by a Stream) | Exchange (routes to queues) | EventBridge bus or SNS topic |
| Consumer group — independent processor | Consumer Group | Channel | Durable Consumer (each = 1 CG) | Queue (each = 1 CG) | SQS Queue (each = 1 CG) |
| Workers inside a group | Consumer instance in the CG | Worker on a Channel | Subscribers on one pull consumer / queue group | Competing consumers on a queue | Lambda execution envs (concurrency) or ECS tasks |
| Parallelism model | Capped by # partitions — extra workers idle | Unlimited workers (no partitions) | Unlimited workers (no partitions) | Unlimited workers (no partitions) | Unlimited; parallelism = Lambda concurrency |
| Partitioning / ordering unit | Partition — strict order within it | None | Per-subject order in a stream (strict only MaxAckPending=1) | FIFO per queue (breaks on requeue / many consumers) | SQS FIFO MessageGroupId |
| Backlog storage | Partition log (retention-based) | In-memory / durable nsqd queue | Stream (file/memory, retention-based) | Queue (memory + disk when durable) | SQS queue backlog |
| Fan-out — 1 message → many groups | Many CGs on the same topic | Many channels get the same message | Many consumers/streams on the subject | Fanout/topic exchange → many queues | EventBridge/SNS → many SQS queues |
| Delivery model | Pull | Push — RDY-based flow control | Pull or push (core NATS = push) | Push (prefetch); pull via basic.get | EB/SNS = push; SQS = pull (Lambda polls) |
| Retry behavior | Offset controls replay | Requeue / timeout | Nak / AckWait redeliver; MaxDeliver → DLQ | nack/reject requeue + Dead Letter Exchange | Visibility timeout + DLQ per queue |
| Isolation between groups | Each CG independent | Each channel independent | Each consumer independent (own cursor + acks) | Each queue independent (own backlog + DLX) | Each queue isolates throughput, retries, DLQ |
| Dimension | Kafkalog + partitions | NSQsimple queue | NATSJetStream | RabbitMQAMQP broker | AWSEB · SNS · SQS · λ |
|---|---|---|---|---|---|
| Core concept | Distributed log + partitions | Simple distributed messaging | Subject pub/sub + persistent streams | Exchange routing + queues (AMQP) | Pub/sub + routing + queue + serverless |
| Ordering | Per-partition | Best effort | Per-subject (strict only MaxAckPending=1) | FIFO per queue (breaks on requeue / >1 consumer) | Standard: best effort · FIFO: per MessageGroupId |
| Idle workers if > partitions | Yes — stalled | No | No | No | No |
| Pull or push | Pull | Push (RDY flow control) | Both (core = push) | Push (prefetch); pull via basic.get | EB/SNS = push · SQS = pull |
| Backpressure | Yes (log retention) | Yes (queue backlog) | Yes (retention + MaxAckPending) | Yes (queue depth + prefetch) | Yes (SQS backlog) |
| Partition-like concept | Partitions | None | None (subject-sharding is manual) | None (consistent-hash exchange optional) | FIFO MessageGroupId ≈ partition |
| Scaling method | Add partitions or consumers | Add workers | Add subscribers / shard subjects | Add consumers / hash exchange / quorum queues | Raise Lambda concurrency / add queues |
| Message replay | Strong (offsets) | Limited | Strong (replay by seq/time) | Limited (classic once; Streams replay) | DLQ reprocess / manual replay |
| Storage & cleanup per group | Offsets only — one shared log; renaming a group is free | Own copy (mem + disk) — leaks; drain/delete or use #ephemeral | Cursor over the shared stream (like Kafka); interest/workqueue keeps msgs while wanted | Own copy per queue — delete it, or auto-delete / TTL / max-length | Own copy per queue — auto-expires by retention (default 4d, max 14d); DeleteQueue to remove |
Two layers, always: the broker fans a copy out to each consumer group, then each group splits its copy across its own workers. Pick a broker — watch which groups get a copy, and notice the split inside a group never depends on it. (RabbitMQ adds an exchange-type dial that changes which queues match.)
exchange → queues → consumers split the copy
Topic matches patterns — orders.eu.new matches eu-orders and all-orders (orders.#), but never payments.
orders.eu.neworders.#payments.#100 published → 200 delivered across 2 queues (each gets its own copy). Inside eu-orders, the copy splits 25/25/25/25 across 4 consumers — that split is independent of how many queues got a copy.
The model covers who gets a copy and who processes it. Two details bite in production: whether message order survives, and what “delivered” actually means.
orderingNSQ doesn’t stay FIFO with several consumers on one channel. nsqd pushes each message to a randomly-chosen ready consumer; each consumer buffers up to its RDY count and often handles them concurrently; and any failed message is re-queued and pushed again later — all three break strict order. Need order? Run one single-threaded consumer at RDY=1, or (better) make handlers idempotent and scale freely.
deliveryAt-least-once is tracked per message, not per batch. With RDY=5, nsqd pushes 5 distinct messages, each tracked on its own. A success sends FIN and the buffer refills to 5; a failure or msg_timeout sends REQ and only that one message re-queues. Every message earns at least one FIN before it’s deleted — so a crash after processing but before FIN triggers a redelivery. Build handlers to tolerate duplicates.
cleanupA group either holds a copy, or just a pointer — and only one of those leaks. Log-based brokers (Kafka, RabbitMQ Streams, NATS JetStream under limits retention) make a group a cursor over one shared copy, so a renamed or abandoned group costs nothing — its offsets simply expire. Queue-based ones (NSQ, classic RabbitMQ, SQS) give each group its own copy in memory + disk, so an orphan grows until you reclaim it. For throwaway consumers reach for the ephemeral escape hatch: NSQ #ephemeral channels (memory-only, auto-deleted on last disconnect), NATS ephemeral consumers (drop after inactivity), RabbitMQ auto-delete/exclusive queues — or a message TTL / SQS retention as the safety net.
An assignment must land on exactly one owner; a notification can fan out to everyone else. So you use both patterns at once — route the work to one agent, broadcast the status to the rest. The icon says which nodes are systems and which are people.
Only the assigned agent owns the ticket record. The customer and dashboard get notification copies, not ownership. Delivery is at-least-once, so make “assign” and “notify” idempotent (dedupe on ticket id) — a redelivery mustn’t double-assign or double-notify.