O'Reilly logo

Data Wrangling with Python by Katharine Jarmul, Jacqueline Kazil

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 8. Data Cleanup: Standardizing and Scripting

You’ve learned how to match, parse, and find duplicates in your data, and you’ve started exploring the wonderful world of data cleanup. As you grow to understand your datasets and the questions you’d like to answer with them, you’ll want to think about standardizing your data as well as automating your cleanup.

In this chapter, we’ll explore how and when to standardize your data and when to test and script your data cleanup. If you are managing regular updates or additions to the dataset, you’ll want to make the cleanup process as efficient and clear as possible so you can spend more time analyzing and reporting. We’ll begin by standardizing and normalizing your dataset and determining what to do if your dataset is not normalized.

Normalizing and Standardizing Your Data

Depending on your data and the type of research you are conducting, standardizing and normalizing your dataset might mean calculating new values using the values you currently have, or it might mean applying standardizations or normalizations across a particular column or value.

Normalization, from a statistical view, often has to do with calculating new values from a dataset to standardize the data on a particular scale. For example, you might need to normalize scores for a test to scale so you can accurately view the distribution. You might also need to normalize data so you can accurately see percentiles, or percentiles across different groups (or cohorts). ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required