What we call a shard in this topic has many names depending on which software you’re using. It’s called a partition in Kafka, a range in CockroachDB, a region in HBase and…
- 08.01Sharding and PartitioningWhat we call a shard in this topic has many names depending on which software you’re using. It’s called a partition in Kafka, a range in CockroachDB, a region in HBase and…
- 08.02Pros and Cons of ShardingThe primary reason for sharding a database is scalability. Sharding is a solution if the volume of data or the write throughput has become too great for a single node to…
- 08.03Sharding of Key-Value DataSay you have a large amount of data, and you want to shard it. How do you decide which records to store on which nodes?
- 08.04Sharding by Key RangeOne way of sharding is to assign a contiguous range of partition keys (from a minimum to a maximum) to each shard, like the volumes of a paper encyclopedia, as illustrated in…
- 08.05Sharding by Hash of KeyKey-range sharding is useful if you want records with nearby (but different) partition keys to be grouped into the same shard—for example, this might be the case with…
- 08.06Skewed Workloads and Relieving Hot SpotsConsistent hashing ensures that keys are uniformly distributed across nodes, but that doesn’t mean that the actual load is uniformly distributed. If the workload is highly…
- 08.07Operations: Automatic Versus Manual RebalancingWe have glossed over one important question with regard to rebalancing: does the splitting of shards and rebalancing happen automatically or manually?
- 08.08Request RoutingWe have discussed how to shard a dataset across multiple nodes, and how to rebalance those shards as nodes are added or removed. Now let’s move on to another question: if you…
- 08.09Sharding and Secondary IndexesThe sharding schemes we have discussed so far rely on the client knowing the partition key for any record it wants to access. This is most easily achieved in a key-value data…
- 08.10Recap: ShardingIn this topic we explored different ways of sharding a large dataset into smaller subsets. Sharding is necessary when you have so much data that storing and processing it on a…