The Data Science Handbook

Chapter 12Data Encodings and File Formats

Coming from a background of academic physics, my first years in data science were one big exercise in discovering new data formats that I probably should have already known about. It was a bit demoralizing at the time, so let me make something clear upfront: people are always dreaming up new data types and formats, and you will forever be playing catch-up on them. However, there are several formats that are common enough you should know them. It seems that every new format that comes out is easily understood as a variation of a previous format, so you'll be on good footing going forward. There are also some broad principles that underlie all formats, and I hope to give you a flavor of them.

First, I will talk about specific file formats that you are likely to encounter as a data scientist. This will include sample code for parsing them, discussions about when they are useful, and some thoughts about the future of data formats.

For the second half of the chapter, I will switch gears to a discussion of how data is laid out in the physical memory of a computer. This will involve peaking under the hood of the computer to look at performance considerations and give you a deeper understanding of the file formats we just discussed. This section will come in handy when you are dealing with particularly gnarly data pathologies or writing code that aims for speed when you are chugging through a dataset.

12.1 Typical File Format Categories

There ...