Modern Big Data Processing with Hadoop

Hive partition bucketing

Bucketing is a technique of decomposing a large dataset into more manageable groups. Bucketing is based on the hashing function. When a table is bucketed, all the table records with the same column value will go into the same bucket. Physically, each bucket is a file in a table folder just like a partition. In a partitioned table, Hive can group the data in multiple folders. But partitions prove effective when they are of a limited number and when the data is distributed equally among all of them. If there are a large number of partitions, then their use becomes less effective. So in that case, we can use bucketing. We can create a number of buckets explicitly during table creation.