LZO compression
LZO compression can be split; the LZO block format allows us to decompress the block at a random offset of the file. As the compression has many small blocks of compressed data, it allows the MapReduce jobs to be split along the boundaries thus allowing parallel threads to work on it. Hence, we get incredible speed to write and read.
How to do it…
This is the library that is shipped with most of the Linux distribution. In case you don't have it, you can download it from http://www.oberhumer.com/opensource/lzo/.
Based on the Linux version, you have to download appropriately:
> port fetch lzo2 # If for some reason LZO2 is already installed, please uninstall first before doing this > port edit lzo2 # A vim editor should open // Add the ...
Get HBase High Performance Cookbook now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
