Saving compressed data in HDFS
In this recipe, we are going to take a look at how to store and process compressed data in HDFS.
Getting ready
To perform this recipe, you should already have a running Hadoop.
How to do it...
It's always good to use compression while storing data in HDFS. HDFS supports various types of compression algorithms such as LZO, BIZ2, Snappy, GZIP, and so on. Every algorithm has its own pros and cons when you consider the time taken to compress and decompress and the space efficiency. These days people prefer Snappy compression as it aims to achieve a very high speed and a reasonable amount of compression.
We can easily store and process any number of files in HDFS. To store compressed data, we don't need to specifically make ...
Get Hadoop: Data Processing and Modelling now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
