Messaging Systems Explained: Kafka, RabbitMQ, Redis, SQS, NATS, and More

The most useful question I ask before picking a messaging system is not “which one is fastest?” It is: is this message a job to be completed, or an event to be remembered?

Get that right and the shortlist collapses from a dozen products to two or three. Get it wrong and you’ll spend a quarter rebuilding your event backbone on top of a system that was designed to distribute jobs, or you’ll drag Kafka into a codebase that just needed to send some emails.

This post is the guide I wish I’d had the first time I stood in front of a whiteboard trying to justify RabbitMQ over SQS over Kafka. I’ll cover the three underlying models, the vocabulary that actually matters in production, and honest trade-offs for each of the popular systems.

Why messaging exists in the first place

Consider a checkout API. When a customer places an order, the naive implementation does everything inside the HTTP request: save the order, charge the card, reserve inventory, send a confirmation email, notify the warehouse, update analytics.

That works — until one of those steps gets slow, or flaky, or is owned by a team that ships breaking changes on Fridays. Now the entire checkout is at the mercy of the least reliable dependency in the chain.

A messaging system breaks that chain. The API saves the order, publishes an OrderCreated message, and returns. Independent consumers pick up the message and do their part:

Order API → Messaging system → Payment worker
                             → Inventory service
                             → Email worker
                             → Analytics service

You get decoupling (producers don’t know their consumers), asynchronous processing (slow work doesn’t block users), load buffering (a traffic spike queues up instead of falling over), independent scaling, retriable failures, and — if you pick the right kind of system — an event history the whole company can build on.

Three models, not one

“Message queue” is a lazy umbrella. Before you can pick a product, you have to know which of these three shapes you actually need.

Work queue. One message equals one unit of work, and normally exactly one worker should handle it. Five workers on the same queue share the load; each job goes to one of them. This is the model for sending an email, resizing an image, generating a PDF, processing a webhook, running a scheduled task. RabbitMQ, Amazon SQS, Azure Service Bus queues, and BullMQ are all strong here.

Publish/subscribe. One publisher, many independent subscribers. An OrderCreated event might be consumed by inventory, billing, notifications, and analytics — and each one gets its own logical copy. RabbitMQ exchanges, Service Bus topics, Google Cloud Pub/Sub, NATS, and Kafka can all do this.

Event stream. An append-only history. Reading an event doesn’t delete it. Consumers track their own position and can rewind. This is what you want for audit trails, event-driven microservices, analytics pipelines, change data capture, event sourcing, or rebuilding a projection from scratch. Kafka is the canonical example; Redis Streams and NATS JetStream are lighter-weight alternatives.

The real question isn’t “which queue is fastest.” It’s “am I distributing jobs, broadcasting events, or retaining an event history?”

The vocabulary that actually matters

Before comparing products, four concepts show up in every serious design conversation.

Acknowledgement. After processing a message, a consumer acks it. If the consumer crashes before acking, the broker redelivers. RabbitMQ, importantly, distinguishes consumer acknowledgements from publisher confirms: one confirms that a consumer finished the work, the other confirms that the broker accepted responsibility for the message (RabbitMQ documentation). Conflating them is a classic source of “we thought this was durable” incidents.

Delivery guarantee. In practice you’re picking between:

• At-most-once: may be lost, never redelivered.
• At-least-once: never lost, may be delivered more than once.
• Exactly-once: applied once within a carefully defined boundary.

Most real systems are at-least-once, which means your consumers must be idempotent. Store a unique operation ID and reject duplicates. Never assume a payment message can arrive only once; assume it will arrive twice and design accordingly.

Ordering. “Ordered” almost never means globally ordered under unlimited parallelism. It usually means ordered within a queue, a partition, a session, a subject, or a message group. More parallelism generally means weaker global order. If your business logic needs ordering, name the key it’s ordered by, and check that the system you’re picking actually preserves that key.

Retry and dead-letter queues. Retry transient failures with backoff. Send permanently-failing messages to a DLQ so they don’t block the healthy ones. A queue with a single poison message chewing through your workers is the kind of outage that ruins weekends.

The comparison at a glance

That table captures the dominant use case, not every feature each product supports. The nuance lives in the sections below.

RabbitMQ: reliable jobs and grown-up routing

RabbitMQ is the general-purpose broker I reach for when messages represent commands or business tasks — process this payment, send this invoice, retry this webhook, route premium orders differently. Producers publish to an exchange, and bindings route messages from the exchange to one or more queues. Exchange types cover direct routing, patterns, fan-out, and header-based decisions (RabbitMQ exchanges).

Its real strengths are the boring, load-bearing features: per-message acknowledgements, publisher confirms, dead-letter exchanges, priorities, TTLs, and routing rules that are more expressive than “publish to this topic.” For critical data, use quorum queues — they replicate across nodes and only issue publisher confirms once the message is safely replicated to a quorum (RabbitMQ quorum queues).

Reach for RabbitMQ when you need reliable work queues, routing rules more complex than a single topic, per-message ack and redelivery semantics, and your team can either operate a broker or pay a managed provider.

Don’t reach for it as your first choice for a massive, long-lived analytics event history. RabbitMQ Streams exist, but Kafka’s architecture and ecosystem are more naturally shaped for that job.

Apache Kafka: durable event streaming at scale

Kafka stores events in ordered, partitioned logs. Consumers read by offset, different consumer groups keep independent positions, and events stick around according to retention settings — they aren’t deleted just because one consumer read them.

That single design choice is why Kafka wins when the history itself has value: order and payment events that many services consume, database change data capture, clickstream and telemetry ingestion, real-time analytics, event sourcing, rebuilding a search index or materialized view from scratch.

Kafka scales by dividing a topic into partitions. Ordering is preserved within a partition, so if related events must stay in order, they need to share a key. Consumers in the same group divide partitions among themselves; separate groups read independently.

Reach for Kafka when events must be retained and replayed, several independent systems need the same event history, throughput is high, and partition-based ordering fits your domain. Kafka Connect and the surrounding stream-processing ecosystem are a real reason to pick it too.

Don’t reach for it to send a few emails. For a small app, Kafka’s conceptual and operational cost outweighs its benefit — even a managed service doesn’t save you from having to think about partitions, consumer groups, and retention.

Redis Streams: a capable stream inside Redis you already run

Redis Streams is an append-only data structure with message IDs, blocking reads, retention controls, and consumer groups. Consumer groups distribute messages across consumers and track a pending entries list for delivered-but-unacked messages (Redis Streams documentation).

It’s the pragmatic choice when Redis is already in your infrastructure and you need something more durable and replayable than basic Redis Pub/Sub, but you don’t want to introduce a whole new broker.

Reach for it when Redis is already present, traffic is moderate, and you want consumer groups plus short- or medium-term replay without adding Kafka or RabbitMQ.

Just don’t confuse “Redis is fast” with “any Redis deployment is a durable messaging platform.” Memory, persistence mode, replication, trimming, and failover all need attention if this is going to hold data you care about.

BullMQ: honest background jobs for Node.js

BullMQ is the Node.js job queue I reach for by default (BullMQ documentation). Built on Redis, it gives you the things you actually want from an application-level queue: delayed jobs, retries with backoff, priorities, concurrency, repeatable jobs, rate limiting, job dependencies, progress tracking.

It’s a great fit for the workloads a Node backend actually has: sending bulk WhatsApp, SMS, or email, processing uploads, scheduling reminders, retrying third-party API calls, generating reports, spreading CPU- or I/O-heavy work across workers.

A minimal example that shows the pattern I use in real projects:

import { Queue, Worker } from "bullmq";

const connection = {
  host: "127.0.0.1",
  port: 6379,
};

const emailQueue = new Queue("email", { connection });

await emailQueue.add(
  "send-welcome-email",
  { userId: "user_123", email: "user@example.com" },
  {
    attempts: 5,
    backoff: { type: "exponential", delay: 5_000 },
    removeOnComplete: 1_000,
    removeOnFail: 5_000,
  },
);

const worker = new Worker(
  "email",
  async (job) => {
    await sendWelcomeEmail(job.data);
  },
  { connection, concurrency: 10 },
);

worker.on("failed", (job, error) => {
  console.error(job?.id, error);
});

Retries, exponential backoff, and delayed jobs are all first-class (delayed jobs, retrying failing jobs).

Reach for BullMQ when your backend is primarily Node.js or TypeScript, you want jobs plus schedules plus retries plus concurrency plus rate limits, Redis is available, and the queue is an internal application mechanism — not a company-wide event backbone.

BullMQ answers “which worker should execute this job?” Kafka answers “how do many consumers read and replay this event history?” Don’t confuse the two.

NATS and JetStream: low-latency service messaging

Core NATS is a lightweight, subject-based pub/sub system optimized for fast service-to-service communication, request/reply, and live signals.

Core NATS is intentionally ephemeral — a subscriber that’s offline can miss a message. JetStream is the layer that adds persistence, acknowledgements, retention, consumer state, and replay, with an ack-and-redeliver loop that gives you at-least-once delivery (NATS JetStream documentation).

Reach for NATS when latency matters, subject-based routing fits your model, you want request/reply as well as pub/sub, and you want a small operational footprint. Add JetStream the moment a message needs to survive a restart or wait for an offline consumer.

Don’t use Core NATS alone for a payment or order message that must never disappear. That’s what JetStream is for.

Amazon SQS: simple, durable queues on AWS

SQS is a fully managed queue service. No brokers, no partitions, no replication to think about — it integrates with Lambda, ECS, EC2 the way you’d expect from a native AWS service.

Two flavors:

• Standard queues: very high scale, at-least-once delivery, best-effort ordering.
• FIFO queues: ordering within message groups plus deduplication-based exactly-once processing semantics, at lower throughput.

AWS documents standard queues as at-least-once — apps must tolerate the occasional duplicate (Amazon SQS overview). FIFO queues use deduplication IDs and a deduplication window to prevent duplicate sends (SQS FIFO documentation). Even with FIFO, keep your consumers idempotent — “exactly once” as a business outcome usually spans databases and third-party APIs that SQS can’t reason about.

Reach for SQS when you’re on AWS, you want minimal operational overhead, durable work distribution matters more than complex routing, and Lambda-based or autoscaled workers fit the architecture.

Pick FIFO only when ordering or send-side deduplication is genuinely required. Standard is simpler and usually gives you more throughput headroom.

Azure Service Bus: enterprise messaging on Azure

Service Bus is Azure’s answer for both competing-consumer queues and topic/subscription pub/sub, with enterprise features layered on: sessions for ordered handling, scheduled delivery, dead lettering, transactions, filtering, and duplicate detection.

Duplicate detection tracks an application-supplied MessageId for a configured window and drops repeated sends (Microsoft documentation). Microsoft is upfront that this doesn’t remove the need for idempotent receive-side processing — send-side dedup can’t cover every possible repeated business effect (message loss and duplicates guidance).

Reach for Service Bus when you’re on Azure, you need queues plus filtered topic subscriptions, and enterprise workflow features like sessions, transactions, or dead lettering matter.

For massive telemetry ingestion or analytics streams on Azure, compare it against Event Hubs. Service Bus focuses on business messaging; Event Hubs is built for high-throughput event ingestion.

Google Cloud Pub/Sub: managed event distribution on GCP

Pub/Sub is GCP’s fully managed topic-and-subscription service. Each subscription independently pulls messages from a topic, which makes it a natural fit for fanning events out to many systems without operating a broker.

The default model expects consumers to tolerate redelivery. Ordering can be enabled with ordering keys, and exactly-once delivery is available for supported pull subscriptions — with latency and coordination trade-offs (Google Cloud ordering, exactly-once delivery). Pub/Sub also supports replaying retained messages via seek.

Reach for Pub/Sub when you’re on GCP, you need scalable pub/sub with minimal ops, several independent subscriptions consume the same events, and integrations with Cloud Run, Cloud Functions, Dataflow, or BigQuery matter.

A decision table you can actually use

Start from the use case, not the product name.

That last row is worth taking seriously. A messaging platform isn’t mandatory for every application. A Postgres table with status, run_at, an attempt count, and FOR UPDATE SKIP LOCKED is a reasonable first queue when traffic is small — and introducing another service adds more risk than value at that scale.

For everything else, cloud alignment usually matters more than small benchmark differences. Native identity, monitoring, autoscaling, networking, and serverless integrations save real engineering time.

Seven rules that apply no matter which system you pick

1. Make consumers idempotent. Assume a message will arrive twice. Store a unique event or operation ID, use database uniqueness constraints, make external API calls idempotent where you can.

2. Acknowledge only after the work is durable. Never ack before saving the result. A crash between ack and commit is how work permanently disappears.

3. Use the transactional outbox pattern. Saving business data and publishing an event are two separate operations. If the DB commit succeeds but publishing fails, the event is lost. The outbox pattern writes the business change and an outbox record in the same transaction; a relay publishes the outbox event later and marks it sent.

4. Retry only transient failures. A timeout may succeed on retry. An invalid email address will not. Classify errors, use exponential backoff with jitter, cap attempts, and route permanent failures to a DLQ.

5. Monitor queue age, not just queue length. A queue with 10,000 one-second jobs may be healthy. A queue with 20 messages waiting for an hour is not. Watch oldest-message age, processing latency, retry rate, DLQ growth, throughput, and worker saturation.

6. Keep messages small. Put large files in object storage and pass a reference. Small messages are easier to retry, replicate, inspect, and evolve.

7. Version your message schemas. Producers and consumers deploy independently. Add a schema version, make consumers tolerant of additive changes, and consider a schema registry with compatibility rules once the platform grows.

The short version

There is no universally best messaging system. There’s the one that fits the shape of your message.

• BullMQ for application-level jobs in Node.js.
• RabbitMQ for reliable work queues and sophisticated routing.
• Kafka when retained, replayable events form a shared data backbone.
• Redis Streams when Redis is already there and your streaming needs are moderate.
• NATS for fast service communication, JetStream when it must be durable.
• SQS, Service Bus, or Pub/Sub when a managed cloud-native service reduces your operational burden.

The most important architectural decision is still the one you make before you open any docs: is this message a job to be completed, or an event to be remembered? Once that’s clear, the rest is just picking the right tool for the shape you already know.