Understanding Redis Cluster: How Clients Access the Right Shard

Background: Why Do We Need Redis Cluster

The Sentinel mode is based on the master-slave model, enabling read-write separation. It also supports automatic failover, improving system availability. However, since each node stores the same data, it wastes memory and makes online scaling difficult.

As a result, Redis Cluster (a solution for implementing a sharded cluster) came into being. Introduced in Redis 3.0, it enables distributed storage in Redis. It shards data, meaning each Redis node stores different content, thus solving the problem of online expansion. Additionally, it can hold large volumes of data by distributing them across various Redis instances, while also offering replication and failover capabilities.

For instance, if a single Redis instance stores 15GB or more of data, the response time becomes slow. This is due to Redis's RDB persistence mechanism. Redis forks a child process to perform RDB persistence, and the time taken by the fork operation is directly proportional to the size of the Redis data.

At this point, you might naturally think: why not just distribute the 15GB of data across multiple nodes? That’s precisely the motivation behind Redis sharded clusters. So, what exactly is a sharded cluster? Let’s look at an example. If you want to store 15GB of data using Redis, you could either use a single Redis instance or create a sharded cluster with 3 Redis instances. Here's a comparison:

Difference between a sharded cluster and Redis Cluster: Redis Cluster is the official sharded cluster implementation introduced in Redis 3.0.

Now, since the data is distributed across different Redis instances, how does the client know which instance holds the data it wants to access? Let’s take a look at how Redis Cluster handles this.

How Does the Client Know Which Shard to Access?

Redis Cluster uses hash slots to manage the mapping between data and instances.

A sharded cluster is divided into 16,384 slots. Each key-value pair entering Redis is hashed based on its key and assigned to one of these 16,384 slots. The hashing method is simple: the CRC16 algorithm calculates a 16-bit value, which is then modded by 16,384. Every key in the database belongs to one of these 16,384 slots, and every node in the cluster handles a portion of them.

Each node in the cluster is responsible for a portion of the hash slots. Suppose the cluster has three nodes: A, B, and C. Then each node would be responsible for approximately 16,384 / 3 slots. A possible distribution might look like this:

Node A handles hash slots 0–5460
Node B handles hash slots 5461–10922
Node C handles hash slots 10923–16383

So what happens when the client sends a read/write operation to a Redis instance that doesn't hold the corresponding data? Enter the MOVED and ASK redirections.

3. What Happens When the Data Isn’t on the Targeted Instance?

In Redis Cluster mode, a node processes requests as follows:

Using hash slot mapping, the node checks whether the requested Redis key belongs to the current node.
If the hash slot isn’t managed by this node, it returns a MOVED redirection.
If the hash slot is indeed managed by this node and the key exists in the slot, it returns the corresponding result.
If the Redis key does not exist in the slot, it checks whether the hash slot is currently being migrated (MIGRATING).
If the Redis key is being migrated, it returns an ASK error, redirecting the client to the destination server where the key is moving.
If the hash slot is not being migrated, it checks whether the slot is being imported.
If the slot is being imported and has the ASKING flag, the operation proceeds directly; otherwise, a MOVED redirection is returned.

3.1 MOVED Redirection

When a client sends a read/write operation to a Redis instance, and the computed slot does not belong to that node, Redis returns a MOVED redirection error. This response includes the IP and port of the correct Redis node where the slot resides. This is the MOVED redirection mechanism used by Redis Cluster.

3.2 ASK Redirection

ASK redirection typically occurs during cluster resharding. Resharding causes slots to migrate. When a client tries to access the source node, the data may have already been moved to the target node. The ASK redirection helps handle this scenario by directing the client to the new node during the migration process.

4. How Do Nodes Communicate with Each Other?

A Redis cluster is made up of multiple nodes. How do they communicate? Through the Gossip protocol. Gossip is a message propagation protocol where each node periodically selects k nodes from the node list and spreads the information it holds until all nodes reach consensus—this is called algorithm convergence.

Basic idea of the Gossip protocol: A node wants to share some information with other nodes in the network. It periodically and randomly selects a few nodes and sends them the information. Those nodes, in turn, do the same—propagating the information to other randomly selected nodes. Typically, the information is propagated to N nodes, not just one. This N is called the fanout.

Redis Cluster uses the Gossip protocol to exchange information among nodes. This includes data about node failures, new node additions, master-slave changes, slot allocation, and more. The Gossip protocol involves several types of messages, including ping, pong, meet, and fail:

meet message: Notifies a new node to join. The sender informs the recipient to join the current cluster. After the meet message is successfully received, the receiving node joins the cluster and begins periodic ping and pong exchanges.
ping message: Each node sends a ping message to other nodes in the cluster every second. The message includes information about two known nodes—such as their addresses, slots, status, and last communication time.
pong message: Sent as a response to ping or meet messages to confirm successful communication. It also includes information about two known nodes.
fail message: When a node determines that another node in the cluster is offline, it broadcasts a fail message to the entire cluster. Upon receiving this message, other nodes mark the corresponding node as offline.

Notably, each node communicates with others through a cluster bus. This communication uses a special port, which is the regular Redis port plus 10,000. For example, if a node’s service port is 6379, then its cluster communication port is 16379. The nodes use a special binary protocol for communication.

5. What Happens When a Node Fails in the Cluster?

Redis Cluster supports high availability. When a node in the cluster fails, the system uses failover to ensure the cluster continues to provide external services properly.

Node Offline

Redis Cluster uses ping/pong messages for failure detection. This includes subjective and objective offline statuses:

Subjective offline (pfail): One node believes another node is unreachable and marks it as offline. However, this is not a definitive failure judgment—it reflects only one node's perspective and may be inaccurate.
Objective offline (fail): Marks a node as truly offline when multiple nodes in the cluster agree that it is unreachable. This is a consensus-based decision. If the offline node is a master that owns hash slots, a failover process is triggered to replace it.

For example, if Node A marks Node B as subjectively offline, it will send Node B’s status information to other nodes. If Node C receives and parses this message and sees that Node B is marked as pfail, it triggers the objective offline process.

When the offline node is a master, the Redis Cluster initiates a voting process among all masters that own slots. If more than half report it as offline, it is officially marked as objectively offline.

Failover Recovery

Once a failure is detected, if the offline node is a master, a replica (slave) must be elected to replace it in order to maintain high availability. The failover process includes the following steps:

Eligibility Check: Determines whether the replica meets the conditions to replace the failed master.
Failover Delay Preparation: If the replica is eligible, it sets a delay time before initiating failover.
Initiate Election: When the failover delay expires, the replica starts the election process.
Vote Collection: Only masters that hold hash slots are allowed to vote. The candidate replica must collect enough votes (more than half) to proceed with replacing the master.

We are Leapcell, your top choice for hosting backend projects.

Leapcell is the Next-Gen Serverless Platform for Web Hosting, Async Tasks, and Redis:

Multi-Language Support

Develop with Node.js, Python, Go, or Rust.

Deploy unlimited projects for free

pay only for usage — no requests, no charges.

Unbeatable Cost Efficiency

Pay-as-you-go with no idle charges.
Example: $25 supports 6.94M requests at a 60ms average response time.

Streamlined Developer Experience

Intuitive UI for effortless setup.
Fully automated CI/CD pipelines and GitOps integration.
Real-time metrics and logging for actionable insights.

Effortless Scalability and High Performance

Auto-scaling to handle high concurrency with ease.
Zero operational overhead — just focus on building.

Explore more in the Documentation!