Redis - Partitioning
Partitioning is the process of splitting your data into multiple Redis instances, so that every instance will only contain a subset of your keys.
Benefits of Partitioning
It allows for much larger databases, using the sum of the memory of many computers. Without partitioning you are limited to the amount of memory that a single computer can support.
It allows to scale the computational power to multiple cores and multiple computers, and the network bandwidth to multiple computers and network adapters.
Disadvantages of Partitioning
Operations involving multiple keys are usually not supported. For instance, you can't perform the intersection between two sets if they are stored in the keys that are mapped to different Redis instances.
Redis transactions involving multiple keys cannot be used.
The partitioning granuliary is the key, so it is not possible to shard a dataset with a single huge key like a very big sorted set.
When partitioning is used, data handling is more complex. For instance, you have to handle multiple RDB/AOF files, and to get a backup of your data you need to aggregate the persistence files from multiple instances and hosts.
Adding and removing the capacity can be complex. For instance, Redis Cluster supports mostly transparent rebalancing of data with the ability to add and remove nodes at runtime. However, other systems like client-side partitioning and proxies don't support this feature. A technique called Presharding helps in this regard.
Types of Partitioning
There are two types of partitioning available in Redis. Suppose we have four Redis instances, R0, R1, R2, R3 and many keys representing users like user:1, user:2, ... and so forth.
Range Partitioning
Range partitioning is accomplished by mapping ranges of objects into specific Redis instances. Suppose in our example, the users from ID 0 to ID 10000 will go into instance R0, while the users from ID 10001 to ID 20000 will go into instance R1 and so forth.
Hash Partitioning
In this type of partitioning, a hash function (eg. modulus function) is used to convert the key into a number and then the data is stored in different-different Redis instances.
