O'Reilly logo

Python for Data Analysis, 2nd Edition by William Wesley McKinney

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 10. Data Aggregation and Group Operations

Categorizing a dataset and applying a function to each group, whether an aggregation or transformation, is often a critical component of a data analysis workflow. After loading, merging, and preparing a dataset, you may need to compute group statistics or possibly pivot tables for reporting or visualization purposes. pandas provides a flexible groupby interface, enabling you to slice, dice, and summarize datasets in a natural way.

One reason for the popularity of relational databases and SQL (which stands for “structured query language”) is the ease with which data can be joined, filtered, transformed, and aggregated. However, query languages like SQL are somewhat constrained in the kinds of group operations that can be performed. As you will see, with the expressiveness of Python and pandas, we can perform quite complex group operations by utilizing any function that accepts a pandas object or NumPy array. In this chapter, you will learn how to:

  • Split a pandas object into pieces using one or more keys (in the form of functions, arrays, or DataFrame column names)

  • Calculate group summary statistics, like count, mean, or standard deviation, or a user-defined function

  • Apply within-group transformations or other manipulations, like normalization, linear regression, rank, or subset selection

  • Compute pivot tables and cross-tabulations

  • Perform quantile analysis and other statistical group analyses

Note

Aggregation of time series data, a special ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required