Loading and saving data using the Parquet format
Apache Parquet is a columnar data storage format, specifically designed for big data storage and processing. Parquet is based on record shredding and assembly algorithm in the Google Dremel paper. In Parquet, data in a single column is stored contiguously.
The columnar format gives Parquet some unique benefits. For example, if you have a table with 100 columns and you mostly access 10 columns, in a row-based format you will have to load all 100 columns, as granularity level is at row level. But, in Parquet, you will only load 10 columns. Another benefit is that since all the data in a given column is of the same datatype (by definition), compression is much more efficient.
How to do it...
- Open the ...
Get Spark Cookbook now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
