Redis Data Structures Beyond Key-Value

Redis data structures beyond strings: sorted sets for leaderboards, streams for event logs, HyperLogLog for cardinality and bitmaps for feature flags.

Published on August 25, 2021Updated on September 9, 20264 min read

redis data-structures backend caching database performance

Most engineers use Redis as a key-value cache. Set a string, get a string, add a TTL. This covers maybe 30% of what Redis can do. The rest of its data structures solve problems that are awkward to model in relational databases and prohibitively expensive to compute at request time.

Sorted sets, streams, HyperLogLog, and bitmaps are purpose-built for patterns that appear constantly in web applications: leaderboards, activity feeds, unique visitor counts, and feature flags. Using the right data structure turns a complex problem into a single Redis command.

Sorted Sets for Leaderboards

Sorted sets store members with scores and keep them ordered automatically. Adding, removing, and ranking operations are all O(log n).

tstypescript

import Redis from 'ioredis';
const redis = new Redis();
 
// ❌ Leaderboard with SQL — expensive query on every page view
// SELECT user_id, score FROM scores ORDER BY score DESC LIMIT 10;
// Gets slower as the table grows. Needs an index. Still scans rows.

tstypescript

// ✅ Sorted set — O(log n) insert, O(log n + m) range query
async function updateScore(userId: string, points: number): Promise<void> {
  await redis.zincrby('leaderboard:weekly', points, userId);
}
 
async function getTopPlayers(count: number): Promise<Array<{ userId: string; score: number }>> {
  // ZREVRANGE returns members in descending score order
  const results = await redis.zrevrange('leaderboard:weekly', 0, count - 1, 'WITHSCORES');
 
  const players: Array<{ userId: string; score: number }> = [];
  for (let i = 0; i < results.length; i += 2) {
    players.push({
      userId: results[i],
      score: parseFloat(results[i + 1]),
    });
  }
  return players;
}
 
async function getPlayerRank(userId: string): Promise<number | null> {
  // ZREVRANK returns 0-based rank (0 = highest score)
  const rank = await redis.zrevrank('leaderboard:weekly', userId);
  return rank !== null ? rank + 1 : null;
}
 
// Usage:
await updateScore('user:alice', 150);
await updateScore('user:bob', 320);
await updateScore('user:carol', 275);
 
const top3 = await getTopPlayers(3);
// [{ userId: 'user:bob', score: 320 }, { userId: 'user:carol', score: 275 }, ...]
 
const aliceRank = await getPlayerRank('user:alice');
// 3

Weekly leaderboards reset by deleting the key or using key names with the week number: leaderboard:2021-W34. No complex SQL, no application-level sorting, no cache invalidation.

Streams for Event Logs

Redis Streams are an append-only log structure with consumer groups — similar to Kafka but built into Redis. Use them for activity feeds, audit logs, and real-time event processing.

tstypescript

// Producer: append events to a stream
async function logActivity(event: {
  userId: string;
  action: string;
  resource: string;
}): Promise<string> {
  const entryId = await redis.xadd(
    'activity:stream',
    '*',   // Auto-generate ID (timestamp-based)
    'userId', event.userId,
    'action', event.action,
    'resource', event.resource,
    'timestamp', new Date().toISOString()
  );
  return entryId;
}
 
// Consumer: read recent events
async function getRecentActivity(count: number): Promise<Array<Record<string, string>>> {
  const entries = await redis.xrevrange('activity:stream', '+', '-', 'COUNT', count);
  return entries.map(([id, fields]) => {
    const obj: Record<string, string> = { id };
    for (let i = 0; i < fields.length; i += 2) {
      obj[fields[i]] = fields[i + 1];
    }
    return obj;
  });
}

tstypescript

// Consumer group: distribute processing across workers
async function setupConsumerGroup(): Promise<void> {
  try {
    await redis.xgroup('CREATE', 'activity:stream', 'processors', '0', 'MKSTREAM');
  } catch {
    // Group already exists — safe to ignore
  }
}
 
async function processEvents(consumerId: string): Promise<void> {
  while (true) {
    const results = await redis.xreadgroup(
      'GROUP', 'processors', consumerId,
      'COUNT', 10,
      'BLOCK', 5000,    // Block for 5s waiting for new events
      'STREAMS', 'activity:stream', '>'
    );
 
    if (!results) continue;
 
    for (const [, entries] of results) {
      for (const [entryId, fields] of entries) {
        // Process the event
        await handleEvent(entryId, fields);
 
        // Acknowledge processing is complete
        await redis.xack('activity:stream', 'processors', entryId);
      }
    }
  }
}

Consumer groups track which events each consumer has processed. If a worker crashes, unacknowledged events are redelivered to another worker. This gives you at-least-once processing without Kafka's operational overhead.

HyperLogLog for Unique Counts

Counting unique visitors, unique search queries, or unique IP addresses exactly requires storing every unique value. HyperLogLog estimates cardinality with ~0.81% error using only 12KB of memory, regardless of the count.

tstypescript

// ❌ Exact unique count — memory grows with unique values
// SET: storing 10M unique visitor IDs = ~400MB
await redis.sadd('visitors:2021-08-25', visitorId);
const count = await redis.scard('visitors:2021-08-25');

tstypescript

// ✅ HyperLogLog — 12KB regardless of cardinality
await redis.pfadd('visitors:hll:2021-08-25', visitorId);
const estimated = await redis.pfcount('visitors:hll:2021-08-25');
// 10,000,000 unique visitors ≈ 9,919,000 estimated (0.81% error)
// Memory: 12KB vs ~400MB
 
// Merge multiple days for weekly unique count
await redis.pfmerge(
  'visitors:hll:week-34',
  'visitors:hll:2021-08-23',
  'visitors:hll:2021-08-24',
  'visitors:hll:2021-08-25'
);
const weeklyUniques = await redis.pfcount('visitors:hll:week-34');
// Union of unique visitors across 3 days — still 12KB

The tradeoff is explicit: you get an estimate, not an exact count. For analytics dashboards where "approximately 1.2 million unique visitors" is as useful as "exactly 1,203,847", HyperLogLog saves orders of magnitude in memory.

Hashes for Object Storage

Redis hashes store field-value pairs within a single key. Use them instead of serializing entire objects into a string — you can read and update individual fields without fetching the whole object.

tstypescript

// ❌ Storing objects as JSON strings
await redis.set('user:123', JSON.stringify({
  name: 'Alice',
  email: 'alice@example.com',
  loginCount: 42,
  lastLogin: '2021-08-25',
}));
 
// To increment loginCount: read, parse, modify, serialize, write
const user = JSON.parse(await redis.get('user:123') ?? '{}');
user.loginCount += 1;
await redis.set('user:123', JSON.stringify(user));
// Race condition if two requests do this simultaneously

tstypescript

// ✅ Hash fields — atomic field-level operations
await redis.hset('user:123', {
  name: 'Alice',
  email: 'alice@example.com',
  loginCount: '42',
  lastLogin: '2021-08-25',
});
 
// Atomic increment — no read-modify-write race condition
await redis.hincrby('user:123', 'loginCount', 1);
 
// Read only the fields you need
const [name, loginCount] = await redis.hmget('user:123', 'name', 'loginCount');
 
// Update a single field without touching others
await redis.hset('user:123', 'lastLogin', new Date().toISOString());

Hashes are memory-efficient for small objects (Redis uses a compact encoding for hashes under 128 fields) and eliminate read-modify-write race conditions through atomic field operations.

Bitmaps for Feature Flags

Bitmaps use individual bits to represent boolean states. One bitmap key can track a boolean flag for millions of users in a few megabytes.

tstypescript

// Track which users have opted into a beta feature
// User IDs map directly to bit offsets
 
async function enableBetaFeature(userId: number): Promise<void> {
  await redis.setbit('feature:dark-mode-beta', userId, 1);
}
 
async function disableBetaFeature(userId: number): Promise<void> {
  await redis.setbit('feature:dark-mode-beta', userId, 0);
}
 
async function hasBetaFeature(userId: number): Promise<boolean> {
  const bit = await redis.getbit('feature:dark-mode-beta', userId);
  return bit === 1;
}
 
// Count how many users have the feature enabled
async function betaUserCount(): Promise<number> {
  return redis.bitcount('feature:dark-mode-beta');
}
 
// Bitwise operations across features
// Users who have BOTH dark-mode AND new-checkout enabled:
await redis.bitop('AND', 'feature:both',
  'feature:dark-mode-beta', 'feature:new-checkout-beta');
const bothCount = await redis.bitcount('feature:both');

10 million users = ~1.25MB per feature flag. Checking a flag is O(1). Counting enabled users across millions is a single BITCOUNT operation. Bitwise AND/OR across flags answers "which users have feature A AND feature B?" in one command.

Key Takeaways

Sorted sets replace complex ranking queries — leaderboards, top-N lists, and range queries in O(log n)
Streams provide Kafka-like event processing with consumer groups, acknowledgment, and redelivery
HyperLogLog estimates unique counts in 12KB regardless of cardinality — ~0.81% error
Hashes store objects with atomic field operations — no read-modify-write race conditions
Bitmaps track boolean flags for millions of entities in megabytes — bitwise operations across flags
Match the data structure to the access pattern — the right Redis structure eliminates application complexity

Redis Data Structures Beyond Key-Value

Sorted Sets for Leaderboards

Streams for Event Logs

HyperLogLog for Unique Counts

Hashes for Object Storage

Bitmaps for Feature Flags

Key Takeaways

Related Posts

Caching Strategies for Web Applications

HTTP Caching: Cache-Control, ETags and CDN Strategies

Connection Pooling: Tuning for Performance Under Load