October DealsAmazon USOctober deal check: compare before you payAmazon US: current deals, useful picks and tech finds.Check DealsClean PCRecommendedOne scan can reveal what keeps slowing WindowsLook for cleanup and repair opportunities.Run ScanOctober DealsAmazon USDeal season is back - check today's better picksAmazon US: current deals, useful picks and tech finds.See Picks×
Skip to content
World desk8 min

Redis Streams: Building Event-Driven Systems Beyond the Cache

Redis Streams can power event-driven workflows with replay and consumer groups. Learn how acknowledgements, pending-message recovery, retention, and failover shape what Redis can safely handle.
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Yes—Redis can do more than cache data. Redis Streams provides an append-oriented event log with replay, independent consumer groups, and acknowledgements. It can suit event-driven workflows that need short-to-moderate retention and can tolerate Redis’s configured persistence and failover behavior. It does not, by itself, guarantee exactly-once processing or lossless failover: consumers must handle retries safely, and operators must choose retention and durability settings deliberately.

What Redis Streams is—and what it is not

Redis Streams is a Redis data structure for recording ordered entries. Producers append entries with XADD; each entry has an ID that Redis generates when asked, or that a producer may specify. Generated IDs are ordered within the stream and include a time component, but they are not a substitute for a business-level ordering rule across independent streams or regions.

Redis documentation describes a stream as an append-only log with additional operations. In practice, entries can be read directly, read through consumer groups, inspected by range, and removed or trimmed. Entries remain available until deleted or trimmed, subject to the deployment’s persistence and replication configuration.

A stream is not an automatic event-processing system. Your application still defines event meaning, validates payloads, performs side effects, deals with poison messages, and decides what to do when dependencies fail. Redis does not make a database update and a stream acknowledgement one atomic transaction.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

How consumer groups distribute and track events

One group shares work; separate groups get independent views

A consumer group lets several named consumers share entries from one stream. Redis tracks delivery progress for the group. When a member reads new entries using XREADGROUP with the special ID >, Redis delivers entries not previously delivered to a consumer in that group. Those entries are assigned among the group’s consumers; the group is for work sharing, not for sending every entry to every member.

Each group has its own progress and pending-entry list (PEL). A separate group can consume the same stream independently, so an order-processing group and an analytics group can each see the event flow without competing for the same group’s entries. This distinction is central: add consumers to one group to divide work; create another group when a separate application needs its own consumption state.

Example: an order event stream

A producer might append a structured event like this:

XADD orders * type order.placed order_id 8472 customer_id 193

The * asks Redis to generate the entry ID. A group can be created to read the stream from its beginning, creating the stream if it does not exist:

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
XGROUP CREATE orders order-processing 0 MKSTREAM

A worker then asks for new group entries:

XREADGROUP GROUP order-processing worker-1 COUNT 10 BLOCK 5000 STREAMS orders >

Redis’s Node.js tutorial uses order lifecycle events such as order.placed, order.paid, order.shipped, and order.cancelled to illustrate this kind of pipeline. These are example event names, not a prescribed schema. In a real system, define stable field names, event versions, identifiers, and validation rules that all producers and consumers understand.

What happens when a consumer fails

Pending is not the same as completed

When a group delivers an entry, Redis records it as pending until the consumer acknowledges it. A worker should issue XACK only after the relevant processing has succeeded:

XACK orders order-processing 1710000000000-0

If a worker crashes before acknowledging, the entry remains in the PEL. The group has not automatically completed the work, but neither does Redis automatically retry it to another consumer just because the first process disappeared.

Inspect and reclaim idle work

Use XPENDING to inspect pending entries and their consumers and idle times. If an entry has been idle longer than a threshold chosen for your workload, a healthy worker can take ownership with XCLAIM or, on Redis 6.2 and later, XAUTOCLAIM. For example:

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
XAUTOCLAIM orders order-processing worker-2 60000 0-0 COUNT 100

This asks Redis to transfer entries idle for at least 60,000 milliseconds to worker-2, scanning from the supplied starting ID. The threshold is an operational decision, not a universal timeout: set it above normal long-running processing time so a slow but healthy worker is less likely to race with a reclaiming worker.

After reclaiming, the new owner still needs to process the entry and acknowledge it. If the stream entry was trimmed or deleted while it was pending, Redis can report its deleted ID in the XAUTOCLAIM response; the payload is then unavailable from that stream entry. Log and handle that case deliberately rather than assuming Redis can replay missing content.

Design for at-least-once delivery

Failure timing can cause an event to be processed more than once. For example, a worker may complete an external side effect and crash before XACK; after reclaim, another worker may perform the side effect again. This is at-least-once behavior, not exactly-once business processing.

Make handlers idempotent where possible: use an event or operation ID to detect repeats, make updates conditional, or keep a deduplication record appropriate to the side effect. Acknowledge too early and a crash before the side effect can lose application work; acknowledge too late and failures can lead to repeat work. Redis cannot make an external database transaction atomic with its acknowledgement.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Replay, retention, and trimming

Read history without advancing a group

XRANGE and XREVRANGE read entries by ID range independently of a consumer group’s delivery cursor. They are useful for inspecting recent activity, rebuilding a projection from retained events, or bootstrapping a reader that needs historical data. A range read does not itself change a consumer group’s progress or acknowledge pending work.

Replay is possible only while the needed entries still exist. Streams do not imply permanent storage: explicit deletion and trimming can remove history, and the data still depends on Redis’s persistence and replication configuration.

Choose a retention window from recovery needs

Trimming can bound memory use, but a trim limit is also a limit on how far back consumers can recover or replay from that stream. Estimate event sizes and arrival rates, then choose a cap that covers the outage, recovery, and replay window the application actually needs. There is no safe universal entry count without those workload inputs.

For example, approximate length-based trimming can be requested when appending:

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
XADD orders MAXLEN ~ 1000000 * type order.placed order_id 8472

Or the stream can be trimmed by minimum ID:

XTRIM orders MINID ~ 1710000000000-0

The approximate marker (~) allows Redis to trim efficiently rather than enforcing an exact boundary on every operation. It is a practical capacity control, not a guarantee that every entry older than a wall-clock duration has been removed; IDs, arrival patterns, and the chosen threshold matter.

Redis 8.2 introduced more fine-grained controls for coordinating trimming and deletion with consumer groups, including KEEPREF, DELREF, and ACKED modes, alongside XDELEX and XACKDEL. These are version-specific features: verify that the server you deploy supports the exact command and behavior before relying on them. They do not remove the need to decide what historical data the application must retain.

When Streams are a practical fit

Redis’s streaming guidance positions Streams for ordered event logs, replay, independent consumer tracking, acknowledgements, and bounded retention. Examples it gives include user-activity event sourcing, sensor monitoring, and per-user notifications. Those examples show possible uses, not a guarantee that Redis is the right system for every application in those categories.

Option Consumption and history Typical consideration
Redis Pub/Sub Redis describes it as fire-and-forget: disconnected subscribers do not get persisted history, replay, or consumer tracking. Fits live notification patterns where subscribers only need messages while connected; not a durable work log.
Redis Streams Entries remain available until trimmed or deleted; range reads and consumer-group tracking support replay and shared work. Can fit workflows with bounded retention that already use Redis, provided durability, memory, and recovery requirements align.
Job queue In Redis’s comparison, completed work is discarded rather than kept as an event history. Prefer when the core abstraction is pending work to complete, not a retained log for independent readers.
Dedicated streaming platform such as Kafka or Pulsar Capabilities and operational model depend on the platform and deployment; long retention or broader streaming needs may be central requirements. Redis notes that a dedicated platform can add disproportionate operational overhead when retention needs are only hours or days. This is workload guidance, not a universal replacement rule.

The choice turns on requirements, not on whether an event can be represented in Redis. Consider how long data must remain replayable, how many independent applications need their own progress, the expected throughput and scale, the team’s operational expertise, and the acceptable data-loss window during failures. Existing Redis infrastructure may reduce the need to operate another cluster for moderate-scale, short-retention work, but does not itself establish that workload’s capacity or durability.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Durability and failover assumptions

Streams and consumer-group state use Redis’s normal persistence and replication mechanisms. The actual guarantees depend on configuration and deployment. Redis documentation warns that default asynchronous replication does not guarantee that the latest XADD or group-state changes have reached a replica before failover. A promoted replica can therefore lack recent data.

When persistence matters, Redis documentation recommends a strong AOF fsync policy. WAIT can request that writes propagate to replicas and can make loss less likely, but it is not a promise of zero loss: Redis describes Sentinel and Cluster failover as best effort, with conditions where a replica missing data may be promoted. Set the persistence and replication policy against the application’s real loss tolerance, and test the failure modes that matter to it. Do not treat a stream as a lossless system-of-record log solely because entries are append-oriented.

Redis’s Active-Active documentation describes additional regional replication semantics. Its behavior should not be conflated with ordinary Redis Open Source replication: verify the product, version, and cross-region behavior you operate, particularly for ordering and replicated consumer-group state.

Operational checks that keep a stream healthy

Redis provides inspection commands including XINFO STREAM, XINFO GROUPS, and XINFO CONSUMERS. Use them alongside application metrics and alerts for:

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
  • Stream length and the oldest retained ID, to see whether retention is keeping pace with the replay window.
  • Pending-entry counts, consumer ownership, and idle time, to find stalled work and assess whether reclaim thresholds are appropriate.
  • Processing latency and acknowledgement rates, to distinguish slow workers from stopped workers.
  • Reclaim activity and entries whose payloads were trimmed, with an application-defined route for poison or unrecoverable events.

A group that is continuously receiving entries but not acknowledging them can accumulate pending work. Conversely, aggressive trimming can remove data before a lagging consumer catches up. Monitor both processing progress and retained history; neither stream length nor pending count alone tells the full recovery story.

Version availability to check before deployment

Redis’s command documentation identifies Streams and basic consumer-group commands as available from Redis Open Source 5.0, XAUTOCLAIM from 6.2, and the enhanced trimming/deletion controls discussed above from 8.2. The documentation describes idempotent message production as beginning in Redis 8.6. Check the server version and managed-service compatibility for the exact commands you plan to use; a command’s existence in current documentation does not mean every Redis deployment supports it.

Consumer groups are conceptually similar to Kafka consumer groups, but Redis documentation cautions that this does not mean they share Kafka’s implementation. Design and validate against the semantics of the Redis product you run rather than assuming a feature transfers between systems.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Leave a Reply

Your email address will not be published. Required fields are marked *

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

More from the Wire

  1. World desk4 min
    How to Spot an AI Voice Scam Before Sending MoneyDon’t rely on how a caller sounds. Pause, call back through a known number, and verify the emergency with another trusted person before sending money.
  2. Mountain View desk4 min
    Google’s SynthID Detector: How to Check AI-Generated Images, Video and AudioGoogle’s SynthID Detector looks for an embedded watermark in supported images, video and audio. Here is what its results do—and do not—show.
  3. Redmond desk20 min
    How to create a link to File or Folder in Windows 11Windows 11 gives you several ways to point to a file or folder without moving or duplicating it. You can create a desktop shortcut,…
Recommended PC Tool
Recommended PC Tool
Windows Errors? Fix Them Before They SpreadFree repair scan
Crashes, No Sound, or Screen Glitches?Free driver scan

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.