We all say we like data, but we donât.
We like getting insight out of data. Thatâs not quite the same as liking the data itself.
In fact, I dare say that I donât quite care for data. It sounds like Iâm not alone.
Itâs tough to nail down a precise definition of âBad Data.â Some people consider it a purely hands-on, technical phenomenon: missing values, malformed records, and cranky file formats. Sure, thatâs part of the picture, but Bad Data is so much more. It includes data that eats up your time, causes you to stay late at the office, drives you to tear out your hair in frustration. Itâs data that you canât access, data that you had and then lost, data thatâs not the same today as it was yesterdayâ¦
In short, Bad Data is data that gets in the way. There are so many ways to get there, from cranky storage, to poor representation, to misguided policy. If you stick with this data science bit long enough, youâll certainly encounter your fair share.
To that end, we decided to compile Bad Data Handbook, a rogues gallery of data troublemakers. We found 19 people from all reaches of the data arena to talk about how data issues have bitten them, and how theyâve healed.
In particular:
- Guidance for Grubby, Hands-on Work
You canât assume that a new dataset is clean and ready for analysis. Kevin Finkâs Is It Just Me, or Does This Data Smell Funny? (Chapter 2) offers several techniques to take the data for a test drive.
Thereâs plenty of data trapped in spreadsheets, a format as prolific as it is inconvenient for analysis efforts. In Data Intended for Human Consumption, Not Machine Consumption (Chapter 3), Paul Murrell shows off moves to help you extract that data into something more usable.
If youâre working with text data, sooner or later a character encoding bug will bite you. Bad Data Lurking in Plain Text (Chapter 4), by Josh Levy, explains what sort of problems await and how to handle them.
To wrap up, Adam Laiacanoâs (Re)Organizing the Webâs Data (Chapter 5) walks you through everything that can go wrong in a web-scraping effort.
- Data That Does the Unexpected
Sure, people lie in online reviews. Jacob Perkins found out that people lie in some very strange ways. Take a look at Detecting Liars and the Confused in Contradictory Online Reviews (Chapter 6) to learn how Jacobâs natural-language programming (NLP) work uncovered this new breed of lie.
Of all the things that can go wrong with data, we can at least rely on unique identifiers, right? In When Data and Reality Donât Match (Chapter 9), Spencer Burns turns to his experience in financial markets to explain why thatâs not always the case.
- Approach
The industry is still trying to assign a precise meaning to the term âdata scientist,â but we all agree that writing software is part of the package. Richard Cottonâs Blood, Sweat, and Urine (Chapter 8) offers sage advice from a software developerâs perspective.
Philipp K. Janert questions whether there is such a thing as truly bad data, in Will the Bad Data Please Stand Up? (Chapter 7).
Your data may have problems, and you wouldnât even know it. As Jonathan A. Schwabish explains in Subtle Sources of Bias and Error (Chapter 10), how you collect that data determines what will hurt you.
In Donât Let the Perfect Be the Enemy of the Good: Is Bad Data Really Bad? (Chapter 11), Brett J. Goldsteinâs career retrospective explains how dirty data will give your classical statistics training a harsh reality check.
- Data Storage and Infrastructure
How you store your data weighs heavily in how you can analyze it. Bobby Norton explains how to spot a graph data structure thatâs trapped in a relational database in Crouching Table, Hidden Network (Chapter 13).
Cloud computingâs scalability and flexibility make it an attractive choice for the demands of large-scale data analysis, but itâs not without its faults. In Myths of Cloud Computing (Chapter 14), Steve Francia dissects some of those assumptions so you donât have to find out the hard way.
We debate using relational databases over NoSQL products, Mongo over Couch, or one Hadoop-based storage over another. Tim McNamaraâs When Databases Attack: A Guide for When to Stick to Files (Chapter 12) offers another, simpler option for storage.
- The Business Side of Data
Sometimes you donât have enough work to hire a full-time data scientist, or maybe you need a particular skill you donât have in-house. In How to Feed and Care for Your Machine-Learning Experts (Chapter 16), Pete Warden explains how to outsource a machine-learning effort.
Corporate bureaucracy policy can build roadblocks that inhibit you from even analyzing the data at all. Marck Vaisman uses The Dark Side of Data Science (Chapter 15) to document several worst practices that you should avoid.
- Data Policy
Sure, you know the methods you used, but do you truly understand how those final figures came to be? Reid Draperâs Data Traceability (Chapter 17) is food for thought for your data processing pipelines.
Data is particularly bad when itâs in the wrong place: itâs supposed to be inside but itâs gotten outside, or it still exists when itâs supposed to have been removed. In Social Media: Erasable Ink? (Chapter 18), Jud Valeski looks to the future of social media, and thinks through a much-needed recall feature.
To close out the book, I pair up with longtime cohort Ken Gleason on Data Quality Analysis Demystified: Knowing When Your Data Is Good Enough (Chapter 19). In this complement to Kevin Finkâs article, we explain how to assess your dataâs quality, and how to build a structure around a data quality effort.
Get Bad Data Handbook now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.