You are previewing Bad Data Handbook.

Bad Data Handbook

Cover of Bad Data Handbook by Q. Ethan McCallum Published by O'Reilly Media, Inc.
  1. Bad Data Handbook
  2. About the Authors
  3. Preface
    1. Conventions Used in This Book
    2. Using Code Examples
    3. Safari® Books Online
    4. How to Contact Us
    5. Acknowledgments
  4. 1. Setting the Pace: What Is Bad Data?
  5. 2. Is It Just Me, or Does This Data Smell Funny?
    1. Understand the Data Structure
    2. Field Validation
    3. Value Validation
    4. Physical Interpretation of Simple Statistics
    5. Visualization
    6. Keyword PPC Example
    7. Search Referral Example
    8. Recommendation Analysis
    9. Time Series Data
    10. Conclusion
  6. 3. Data Intended for Human Consumption, Not Machine Consumption
    1. The Data
    2. The Problem: Data Formatted for Human Consumption
      1. The Arrangement of Data
      2. Data Spread Across Multiple Files
    3. The Solution: Writing Code
      1. Reading Data from an Awkward Format
      2. Reading Data Spread Across Several Files
    4. Postscript
    5. Other Formats
    6. Summary
  7. 4. Bad Data Lurking in Plain Text
    1. Which Plain Text Encoding?
    2. Guessing Text Encoding
    3. Normalizing Text
    4. Problem: Application-Specific Characters Leaking into Plain Text
    5. Text Processing with Python
    6. Exercises
  8. 5. (Re)Organizing the Web’s Data
    1. Can You Get That?
    2. General Workflow Example
      1. robots.txt
      2. Identifying the Data Organization Pattern
      3. Store Offline Version for Parsing
      4. Scrape the Information Off the Page
    3. The Real Difficulties
      1. Download the Raw Content If Possible
      2. Forms, Dialog Boxes, and New Windows
      3. Flash
    4. The Dark Side
    5. Conclusion
  9. 6. Detecting Liars and the Confused in Contradictory Online Reviews
    1. Weotta
    2. Getting Reviews
    3. Sentiment Classification
    4. Polarized Language
    5. Corpus Creation
    6. Training a Classifier
    7. Validating the Classifier
    8. Designing with Data
    9. Lessons Learned
    10. Summary
    11. Resources
  10. 7. Will the Bad Data Please Stand Up?
    1. Example 1: Defect Reduction in Manufacturing
    2. Example 2: Who’s Calling?
    3. Example 3: When “Typical” Does Not Mean “Average”
    4. Lessons Learned
    5. Will This Be on the Test?
  11. 8. Blood, Sweat, and Urine
    1. A Very Nerdy Body Swap Comedy
    2. How Chemists Make Up Numbers
    3. All Your Database Are Belong to Us
    4. Check, Please
    5. Live Fast, Die Young, and Leave a Good-Looking Corpse Code Repository
    6. Rehab for Chemists (and Other Spreadsheet Abusers)
    7. tl;dr
  12. 9. When Data and Reality Don’t Match
    1. Whose Ticker Is It Anyway?
    2. Splits, Dividends, and Rescaling
    3. Bad Reality
    4. Conclusion
  13. 10. Subtle Sources of Bias and Error
    1. Imputation Bias: General Issues
    2. Reporting Errors: General Issues
    3. Other Sources of Bias
      1. Topcoding/Bottomcoding
      2. Seam Bias
      3. Proxy Reporting
      4. Sample Selection
    4. Conclusions
    5. References 
  14. 11. Don’t Let the Perfect Be the Enemy of the Good: Is Bad Data Really Bad?
    1. But First, Let’s Reflect on Graduate School …
    2. Moving On to the Professional World
    3. Moving into Government Work
    4. Government Data Is Very Real
    5. Service Call Data as an Applied Example
    6. Moving Forward
    7. Lessons Learned and Looking Ahead
  15. 12. When Databases Attack: A Guide for When to Stick to Files
    1. History
      1. Building My Toolset
      2. The Roadblock: My Datastore
    2. Consider Files as Your Datastore
      1. Files Are Simple!
      2. Files Work with Everything
      3. Files Can Contain Any Data Type
      4. Data Corruption Is Local
      5. They Have Great Tooling
      6. There’s No Install Tax
    3. File Concepts
      1. Encoding
      2. Text Files
      3. Binary Data
      4. Memory-Mapped Files
      5. File Formats
      6. Delimiters
    4. A Web Framework Backed by Files
      1. Motivation
      2. Implementation
    5. Reflections
  16. 13. Crouching Table, Hidden Network
    1. A Relational Cost Allocations Model
    2. The Delicate Sound of a Combinatorial Explosion…
    3. The Hidden Network Emerges
    4. Storing the Graph
    5. Navigating the Graph with Gremlin
    6. Finding Value in Network Properties
    7. Think in Terms of Multiple Data Models and Use the Right Tool for the Job
    8. Acknowledgments
  17. 14. Myths of Cloud Computing
    1. Introduction to the Cloud
    2. What Is “The Cloud”?
    3. The Cloud and Big Data
    4. Introducing Fred
    5. At First Everything Is Great
    6. They Put 100% of Their Infrastructure in the Cloud
    7. As Things Grow, They Scale Easily at First
    8. Then Things Start Having Trouble
    9. They Need to Improve Performance
    10. Higher IO Becomes Critical
    11. A Major Regional Outage Causes Massive Downtime
    12. Higher IO Comes with a Cost
    13. Data Sizes Increase
    14. Geo Redundancy Becomes a Priority
    15. Horizontal Scale Isn’t as Easy as They Hoped
    16. Costs Increase Dramatically
    17. Fred’s Follies
    18. Myth 1: Cloud Is a Great Solution for All Infrastructure Components
      1. How This Myth Relates to Fred’s Story
    19. Myth 2: Cloud Will Save Us Money
      1. How This Myth Relates to Fred’s Story
    20. Myth 3: Cloud IO Performance Can Be Improved to Acceptable Levels Through Software RAID
      1. How This Myth Relates to Fred’s Story
    21. Myth 4: Cloud Computing Makes Horizontal Scaling Easy
      1. How This Myth Relates to Fred’s Story
    22. Conclusion and Recommendations
  18. 15. The Dark Side of Data Science
    1. Avoid These Pitfalls
    2. Know Nothing About Thy Data
      1. Be Inconsistent in Cleaning and Organizing the Data
      2. Assume Data Is Correct and Complete
      3. Spillover of Time-Bound Data
    3. Thou Shalt Provide Your Data Scientists with a Single Tool for All Tasks
      1. Using a Production Environment for Ad-Hoc Analysis
      2. The Ideal Data Science Environment
    4. Thou Shalt Analyze for Analysis’ Sake Only
    5. Thou Shalt Compartmentalize Learnings
    6. Thou Shalt Expect Omnipotence from Data Scientists
      1. Where Do Data Scientists Live Within the Organization?
    7. Final Thoughts
  19. 16. How to Feed and Care for Your Machine-Learning Experts
    1. Define the Problem
    2. Fake It Before You Make It
    3. Create a Training Set
    4. Pick the Features
    5. Encode the Data
    6. Split Into Training, Test, and Solution Sets
    7. Describe the Problem
    8. Respond to Questions
    9. Integrate the Solutions
    10. Conclusion
  20. 17. Data Traceability
    1. Why?
    2. Personal Experience
      1. Snapshotting
      2. Saving the Source
      3. Weighting Sources
      4. Backing Out Data
      5. Separating Phases (and Keeping them Pure)
      6. Identifying the Root Cause
      7. Finding Areas for Improvement
    3. Immutability: Borrowing an Idea from Functional Programming
    4. An Example
      1. Crawlers
      2. Change
      3. Clustering
      4. Popularity
    5. Conclusion
  21. 18. Social Media: Erasable Ink?
    1. Social Media: Whose Data Is This Anyway?
    2. Control
    3. Commercial Resyndication
    4. Expectations Around Communication and Expression
    5. Technical Implications of New End User Expectations
    6. What Does the Industry Do?
      1. Validation API
      2. Update Notification API
    7. What Should End Users Do?
    8. How Do We Work Together?
  22. 19. Data Quality Analysis Demystified: Knowing When Your Data Is Good Enough
    1. Framework Introduction: The Four Cs of Data Quality Analysis
    2. Complete
    3. Coherent
    4. Correct
    5. aCcountable
    6. Conclusion
  23. Index
  24. About the Author
  25. Colophon
  26. Copyright
O'Reilly logo

Chapter 1. Setting the Pace: What Is Bad Data?

We all say we like data, but we don’t.

We like getting insight out of data. That’s not quite the same as liking the data itself.

In fact, I dare say that I don’t quite care for data. It sounds like I’m not alone.

It’s tough to nail down a precise definition of “Bad Data.” Some people consider it a purely hands-on, technical phenomenon: missing values, malformed records, and cranky file formats. Sure, that’s part of the picture, but Bad Data is so much more. It includes data that eats up your time, causes you to stay late at the office, drives you to tear out your hair in frustration. It’s data that you can’t access, data that you had and then lost, data that’s not the same today as it was yesterday…

In short, Bad Data is data that gets in the way. There are so many ways to get there, from cranky storage, to poor representation, to misguided policy. If you stick with this data science bit long enough, you’ll certainly encounter your fair share.

To that end, we decided to compile Bad Data Handbook, a rogues gallery of data troublemakers. We found 19 people from all reaches of the data arena to talk about how data issues have bitten them, and how they’ve healed.

In particular:

Guidance for Grubby, Hands-on Work

You can’t assume that a new dataset is clean and ready for analysis. Kevin Fink’s Is It Just Me, or Does This Data Smell Funny? (Chapter 2) offers several techniques to take the data for a test drive.

There’s plenty of data trapped in spreadsheets, a format as prolific as it is inconvenient for analysis efforts. In Data Intended for Human Consumption, Not Machine Consumption (Chapter 3), Paul Murrell shows off moves to help you extract that data into something more usable.

If you’re working with text data, sooner or later a character encoding bug will bite you. Bad Data Lurking in Plain Text (Chapter 4), by Josh Levy, explains what sort of problems await and how to handle them.

To wrap up, Adam Laiacano’s (Re)Organizing the Web’s Data (Chapter 5) walks you through everything that can go wrong in a web-scraping effort.

Data That Does the Unexpected

Sure, people lie in online reviews. Jacob Perkins found out that people lie in some very strange ways. Take a look at Detecting Liars and the Confused in Contradictory Online Reviews (Chapter 6) to learn how Jacob’s natural-language programming (NLP) work uncovered this new breed of lie.

Of all the things that can go wrong with data, we can at least rely on unique identifiers, right? In When Data and Reality Don’t Match (Chapter 9), Spencer Burns turns to his experience in financial markets to explain why that’s not always the case.


The industry is still trying to assign a precise meaning to the term “data scientist,” but we all agree that writing software is part of the package. Richard Cotton’s Blood, Sweat, and Urine (Chapter 8) offers sage advice from a software developer’s perspective.

Philipp K. Janert questions whether there is such a thing as truly bad data, in Will the Bad Data Please Stand Up? (Chapter 7).

Your data may have problems, and you wouldn’t even know it. As Jonathan A. Schwabish explains in Subtle Sources of Bias and Error (Chapter 10), how you collect that data determines what will hurt you.

In Don’t Let the Perfect Be the Enemy of the Good: Is Bad Data Really Bad? (Chapter 11), Brett J. Goldstein’s career retrospective explains how dirty data will give your classical statistics training a harsh reality check.

Data Storage and Infrastructure

How you store your data weighs heavily in how you can analyze it. Bobby Norton explains how to spot a graph data structure that’s trapped in a relational database in Crouching Table, Hidden Network (Chapter 13).

Cloud computing’s scalability and flexibility make it an attractive choice for the demands of large-scale data analysis, but it’s not without its faults. In Myths of Cloud Computing (Chapter 14), Steve Francia dissects some of those assumptions so you don’t have to find out the hard way.

We debate using relational databases over NoSQL products, Mongo over Couch, or one Hadoop-based storage over another. Tim McNamara’s When Databases Attack: A Guide for When to Stick to Files (Chapter 12) offers another, simpler option for storage.

The Business Side of Data

Sometimes you don’t have enough work to hire a full-time data scientist, or maybe you need a particular skill you don’t have in-house. In How to Feed and Care for Your Machine-Learning Experts (Chapter 16), Pete Warden explains how to outsource a machine-learning effort.

Corporate bureaucracy policy can build roadblocks that inhibit you from even analyzing the data at all. Marck Vaisman uses The Dark Side of Data Science (Chapter 15) to document several worst practices that you should avoid.

Data Policy

Sure, you know the methods you used, but do you truly understand how those final figures came to be? Reid Draper’s Data Traceability (Chapter 17) is food for thought for your data processing pipelines.

Data is particularly bad when it’s in the wrong place: it’s supposed to be inside but it’s gotten outside, or it still exists when it’s supposed to have been removed. In Social Media: Erasable Ink? (Chapter 18), Jud Valeski looks to the future of social media, and thinks through a much-needed recall feature.

To close out the book, I pair up with longtime cohort Ken Gleason on Data Quality Analysis Demystified: Knowing When Your Data Is Good Enough (Chapter 19). In this complement to Kevin Fink’s article, we explain how to assess your data’s quality, and how to build a structure around a data quality effort.

The best content for your career. Discover unlimited learning on demand for around $1/day.