Message Queues in Practice
RabbitMQ, Redis Streams, or Kafka — understanding the messaging patterns behind decoupled, resilient systems and when each tool fits.
Message queues are the backbone of distributed systems. They decouple producers from consumers, absorb traffic spikes, and enable reliable communication between services that don't need to be online at the same time. But choosing between RabbitMQ, Kafka, Redis Streams, or SQS without understanding the underlying patterns leads to overengineered solutions.
Point-to-Point vs. Pub/Sub
Every messaging pattern falls into one of two categories.
// Point-to-point: one message, one consumer
// Use case: task distribution (send email, process payment)
// Each message is processed by exactly ONE worker
// Pub/Sub: one message, many consumers
// Use case: event broadcasting (order placed → notify analytics, inventory, email)
// Each message is delivered to ALL subscribers| Pattern | Delivery | Consumers | Example |
|---|---|---|---|
| Point-to-point | One consumer per message | Competing workers | Job queues, task distribution |
| Pub/Sub | All subscribers | Independent consumers | Event broadcasting, notifications |
| Fan-out | All consumers, each gets a copy | Independent processing | Multi-system sync |
When You Need a Message Queue
Not every service interaction needs a queue. Direct HTTP calls are fine when:
- The response is needed immediately
- Both services must be available
- Failure should be visible to the caller
Queues add value when:
- The work can be deferred
- Services have different availability requirements
- Traffic is bursty and needs buffering
// ❌ Synchronous chain — one failure breaks everything
app.post("/api/orders", async (req, res) => {
const order = await createOrder(req.body);
await inventoryService.reserve(order.items); // If this fails...
await paymentService.charge(order.total); // ...none of this runs
await emailService.sendConfirmation(order); // ...the user sees an error
await analyticsService.trackPurchase(order);
res.json(order);
});
// ✅ Async event — order is placed, downstream services react independently
app.post("/api/orders", async (req, res) => {
const order = await createOrder(req.body);
await messageQueue.publish("order.placed", {
orderId: order.id,
items: order.items,
total: order.total,
customerId: order.customerId,
});
res.json(order);
});
// Each service consumes the event independently
// inventory-service listens to "order.placed"
// payment-service listens to "order.placed"
// email-service listens to "order.placed"
// analytics-service listens to "order.placed"Delivery Guarantees
Messaging systems offer different levels of delivery reliability.
// At-most-once: fire and forget
// Message may be lost, but never duplicated
// Use for: metrics, logging, non-critical notifications
// At-least-once: guaranteed delivery, possible duplicates
// Consumer MUST be idempotent
// Use for: most business events (payments, emails, orders)
// Exactly-once: no loss, no duplicates (very expensive)
// Requires distributed transactions or deduplication
// Use for: financial transactions (or use at-least-once + idempotency)In practice, at-least-once with idempotent consumers is the right default. Exactly-once delivery is a theoretical guarantee that's extremely expensive to implement correctly.
RabbitMQ: Traditional Message Broker
RabbitMQ excels at routing, acknowledgment, and point-to-point queuing.
import amqp from "amqplib";
// Producer
async function publishOrderEvent(order: Order) {
const connection = await amqp.connect(process.env.RABBITMQ_URL!);
const channel = await connection.createChannel();
await channel.assertExchange("orders", "topic", { durable: true });
channel.publish(
"orders",
"order.placed",
Buffer.from(JSON.stringify(order)),
{ persistent: true }, // Survive broker restarts
);
}
// Consumer
async function startOrderConsumer() {
const connection = await amqp.connect(process.env.RABBITMQ_URL!);
const channel = await connection.createChannel();
await channel.assertExchange("orders", "topic", { durable: true });
const queue = await channel.assertQueue("inventory-service", {
durable: true,
});
await channel.bindQueue(queue.queue, "orders", "order.placed");
channel.prefetch(10); // Process 10 messages at a time
channel.consume(queue.queue, async (msg) => {
if (!msg) return;
try {
const order = JSON.parse(msg.content.toString());
await reserveInventory(order);
channel.ack(msg); // Acknowledge successful processing
} catch (error) {
channel.nack(msg, false, true); // Requeue on failure
}
});
}Kafka: Event Streaming
Kafka isn't a traditional queue — it's a distributed commit log. Messages are persisted and can be replayed.
import { Kafka } from "kafkajs";
const kafka = new Kafka({
clientId: "order-service",
brokers: [process.env.KAFKA_BROKER!],
});
// Producer
const producer = kafka.producer();
await producer.connect();
await producer.send({
topic: "orders",
messages: [
{
key: order.id, // Partition by order ID for ordering guarantees
value: JSON.stringify({ event: "order.placed", data: order }),
},
],
});
// Consumer group — messages distributed among group members
const consumer = kafka.consumer({ groupId: "inventory-service" });
await consumer.connect();
await consumer.subscribe({ topic: "orders", fromBeginning: false });
await consumer.run({
eachMessage: async ({ topic, partition, message }) => {
const event = JSON.parse(message.value!.toString());
if (event.event === "order.placed") {
await reserveInventory(event.data);
}
},
});Choosing the Right Tool
| Feature | RabbitMQ | Kafka | Redis Streams | SQS |
|---|---|---|---|---|
| Routing | Advanced (exchanges, bindings) | Topics + partitions | Simple | Basic |
| Message replay | No (consumed = gone) | Yes (persistent log) | Limited | No |
| Throughput | ~50K msg/s | ~1M msg/s | ~100K msg/s | ~3K msg/s |
| Ordering | Per-queue | Per-partition | Per-stream | FIFO queues only |
| Ops complexity | Medium | High | Low (if using Redis) | None (managed) |
Use RabbitMQ for complex routing, priority queues, and traditional job distribution. Use Kafka for high-throughput event streaming with replay. Use Redis Streams for lightweight queuing when you already have Redis. Use SQS when you want zero operational overhead in AWS.
Key Takeaways
- Not every service call needs a queue — use queues for deferred, independent, or bursty workloads
- At-least-once + idempotent consumers is the practical default for reliable messaging
- RabbitMQ for routing, Kafka for high-throughput event streaming, Redis Streams for lightweight queuing
- Always acknowledge messages explicitly — auto-acknowledgment risks losing messages on consumer crashes
- Partition/key by entity ID for ordering guarantees within a single entity's events
