You are previewing Python Data Analysis Cookbook.
O'Reilly logo
Python Data Analysis Cookbook

Book Description

Over 140 practical recipes to help you make sense of your data with ease and build production-ready data apps

About This Book

  • Analyze Big Data sets, create attractive visualizations, and manipulate and process various data types

  • Packed with rich recipes to help you learn and explore amazing algorithms for statistics and machine learning

  • Authored by Ivan Idris, expert in python programming and proud author of eight highly reviewed books

  • Who This Book Is For

    This book teaches Python data analysis at an intermediate level with the goal of transforming you from journeyman to master. Basic Python and data analysis skills and affinity are assumed.

    What You Will Learn

  • Set up reproducible data analysis

  • Clean and transform data

  • Apply advanced statistical analysis

  • Create attractive data visualizations

  • Web scrape and work with databases, Hadoop, and Spark

  • Analyze images and time series data

  • Mine text and analyze social networks

  • Use machine learning and evaluate the results

  • Take advantage of parallelism and concurrency

  • In Detail

    Data analysis is a rapidly evolving field and Python is a multi-paradigm programming language suitable for object-oriented application development and functional design patterns. As Python offers a range of tools and libraries for all purposes, it has slowly evolved as the primary language for data science, including topics on: data analysis, visualization, and machine learning.

    Python Data Analysis Cookbook focuses on reproducibility and creating production-ready systems. You will start with recipes that set the foundation for data analysis with libraries such as matplotlib, NumPy, and pandas. You will learn to create visualizations by choosing color maps and palettes then dive into statistical data analysis using distribution algorithms and correlations. You’ll then help you find your way around different data and numerical problems, get to grips with Spark and HDFS, and then set up migration scripts for web mining.

    In this book, you will dive deeper into recipes on spectral analysis, smoothing, and bootstrapping methods. Moving on, you will learn to rank stocks and check market efficiency, then work with metrics and clusters. You will achieve parallelism to improve system performance by using multiple threads and speeding up your code.

    By the end of the book, you will be capable of handling various data analysis techniques in Python and devising solutions for problem scenarios.

    Style and Approach

    The book is written in “cookbook” style striving for high realism in data analysis. Through the recipe-based format, you can read each recipe separately as required and immediately apply the knowledge gained.

    Downloading the example code for this book. You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub.com/support and register to have the code file.

    Table of Contents

    1. Python Data Analysis Cookbook
      1. Table of Contents
      2. Python Data Analysis Cookbook
      3. Credits
      4. About the Author
      5. About the Reviewers
      6. www.PacktPub.com
        1. eBooks, discount offers, and more
          1. Why subscribe?
      7. Preface
        1. Why do you need this book?
        2. Data analysis, data science, big data – what is the big deal?
        3. A brief of history of data analysis with Python
        4. A conjecture about the future
        5. What this book covers
        6. What you need for this book
        7. Who this book is for
        8. Sections
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
        9. Conventions
        10. Reader feedback
        11. Customer support
          1. Downloading the example code
          2. Errata
          3. Piracy
          4. Questions
      8. 1. Laying the Foundation for Reproducible Data Analysis
        1. Introduction
        2. Setting up Anaconda
          1. Getting ready
          2. How to do it...
          3. There's more...
          4. See also
        3. Installing the Data Science Toolbox
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        4. Creating a virtual environment with virtualenv and virtualenvwrapper
          1. Getting ready
          2. How to do it...
          3. See also
        5. Sandboxing Python applications with Docker images
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        6. Keeping track of package versions and history in IPython Notebook
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        7. Configuring IPython
          1. Getting ready
          2. How to do it...
          3. See also
        8. Learning to log for robust error checking
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        9. Unit testing your code
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        10. Configuring pandas
          1. Getting ready
          2. How to do it...
        11. Configuring matplotlib
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        12. Seeding random number generators and NumPy print options
          1. Getting ready
          2. How to do it...
          3. See also
        13. Standardizing reports, code style, and data access
          1. Getting ready
          2. How to do it...
          3. See also
      9. 2. Creating Attractive Data Visualizations
        1. Introduction
        2. Graphing Anscombe's quartet
          1. How to do it...
          2. See also
        3. Choosing seaborn color palettes
          1. How to do it...
          2. See also
        4. Choosing matplotlib color maps
          1. How to do it...
          2. See also
        5. Interacting with IPython Notebook widgets
          1. How to do it...
          2. See also
        6. Viewing a matrix of scatterplots
          1. How to do it...
        7. Visualizing with d3.js via mpld3
          1. Getting ready
          2. How to do it...
        8. Creating heatmaps
          1. Getting ready
          2. How to do it...
          3. See also
        9. Combining box plots and kernel density plots with violin plots
          1. How to do it...
          2. See also
        10. Visualizing network graphs with hive plots
          1. Getting ready
          2. How to do it...
        11. Displaying geographical maps
          1. Getting ready
          2. How to do it...
        12. Using ggplot2-like plots
          1. Getting ready
          2. How to do it...
        13. Highlighting data points with influence plots
          1. How to do it...
          2. See also
      10. 3. Statistical Data Analysis and Probability
        1. Introduction
        2. Fitting data to the exponential distribution
          1. How to do it...
          2. How it works…
          3. See also
        3. Fitting aggregated data to the gamma distribution
          1. How to do it...
          2. See also
        4. Fitting aggregated counts to the Poisson distribution
          1. How to do it...
          2. See also
        5. Determining bias
          1. How to do it...
          2. See also
        6. Estimating kernel density
          1. How to do it...
          2. See also
        7. Determining confidence intervals for mean, variance, and standard deviation
          1. How to do it...
          2. See also
        8. Sampling with probability weights
          1. How to do it...
          2. See also
        9. Exploring extreme values
          1. How to do it...
          2. See also
        10. Correlating variables with Pearson's correlation
          1. How to do it...
          2. See also
        11. Correlating variables with the Spearman rank correlation
          1. How to do it...
          2. See also
        12. Correlating a binary and a continuous variable with the point biserial correlation
          1. How to do it...
          2. See also
        13. Evaluating relations between variables with ANOVA
          1. How to do it...
          2. See also
      11. 4. Dealing with Data and Numerical Issues
        1. Introduction
        2. Clipping and filtering outliers
          1. How to do it...
          2. See also
        3. Winsorizing data
          1. How to do it...
          2. See also
        4. Measuring central tendency of noisy data
          1. How to do it...
          2. See also
        5. Normalizing with the Box-Cox transformation
          1. How to do it...
          2. How it works
          3. See also
        6. Transforming data with the power ladder
          1. How to do it...
        7. Transforming data with logarithms
          1. How to do it...
        8. Rebinning data
          1. How to do it...
        9. Applying logit() to transform proportions
          1. How to do it...
        10. Fitting a robust linear model
          1. How to do it...
          2. See also
        11. Taking variance into account with weighted least squares
          1. How to do it...
          2. See also
        12. Using arbitrary precision for optimization
          1. Getting ready
          2. How to do it...
          3. See also
        13. Using arbitrary precision for linear algebra
          1. Getting ready
          2. How to do it...
          3. See also
      12. 5. Web Mining, Databases, and Big Data
        1. Introduction
        2. Simulating web browsing
          1. Getting ready
          2. How to do it…
          3. See also
        3. Scraping the Web
          1. Getting ready
          2. How to do it…
        4. Dealing with non-ASCII text and HTML entities
          1. Getting ready
          2. How to do it…
          3. See also
        5. Implementing association tables
          1. Getting ready
          2. How to do it…
        6. Setting up database migration scripts
          1. Getting ready
          2. How to do it…
          3. See also
        7. Adding a table column to an existing table
          1. Getting ready
          2. How to do it…
        8. Adding indices after table creation
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
        9. Setting up a test web server
          1. Getting ready
          2. How to do it…
        10. Implementing a star schema with fact and dimension tables
          1. How to do it…
          2. See also
        11. Using HDFS
          1. Getting ready
          2. How to do it…
          3. See also
        12. Setting up Spark
          1. Getting ready
          2. How to do it…
          3. See also
        13. Clustering data with Spark
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
      13. 6. Signal Processing and Timeseries
        1. Introduction
        2. Spectral analysis with periodograms
          1. How to do it...
          2. See also
        3. Estimating power spectral density with the Welch method
          1. How to do it...
          2. See also
        4. Analyzing peaks
          1. How to do it...
          2. See also
        5. Measuring phase synchronization
          1. How to do it...
          2. See also
        6. Exponential smoothing
          1. How to do it...
          2. See also
        7. Evaluating smoothing
          1. How to do it...
          2. See also
        8. Using the Lomb-Scargle periodogram
          1. How to do it...
          2. See also
        9. Analyzing the frequency spectrum of audio
          1. How to do it...
          2. See also
        10. Analyzing signals with the discrete cosine transform
          1. How to do it...
          2. See also
        11. Block bootstrapping time series data
          1. How to do it...
          2. See also
        12. Moving block bootstrapping time series data
          1. How to do it...
          2. See also
        13. Applying the discrete wavelet transform
          1. Getting started
          2. How to do it...
          3. See also
      14. 7. Selecting Stocks with Financial Data Analysis
        1. Introduction
        2. Computing simple and log returns
          1. How to do it...
          2. See also
        3. Ranking stocks with the Sharpe ratio and liquidity
          1. How to do it...
          2. See also
        4. Ranking stocks with the Calmar and Sortino ratios
          1. How to do it...
          2. See also
        5. Analyzing returns statistics
          1. How to do it...
        6. Correlating individual stocks with the broader market
          1. How to do it...
        7. Exploring risk and return
          1. How to do it...
          2. See also
        8. Examining the market with the non-parametric runs test
          1. How to do it...
          2. See also
        9. Testing for random walks
          1. How to do it...
          2. See also
        10. Determining market efficiency with autoregressive models
          1. How to do it...
          2. See also
        11. Creating tables for a stock prices database
          1. How to do it...
        12. Populating the stock prices database
          1. How to do it...
        13. Optimizing an equal weights two-asset portfolio
          1. How to do it...
          2. See also
      15. 8. Text Mining and Social Network Analysis
        1. Introduction
        2. Creating a categorized corpus
          1. Getting ready
          2. How to do it...
          3. See also
        3. Tokenizing news articles in sentences and words
          1. Getting ready
          2. How to do it...
          3. See also
        4. Stemming, lemmatizing, filtering, and TF-IDF scores
          1. Getting ready
          2. How to do it...
          3. How it works
          4. See also
        5. Recognizing named entities
          1. Getting ready
          2. How to do it...
          3. How it works
          4. See also
        6. Extracting topics with non-negative matrix factorization
          1. How to do it...
          2. How it works
          3. See also
        7. Implementing a basic terms database
          1. How to do it...
          2. How it works
          3. See also
        8. Computing social network density
          1. Getting ready
          2. How to do it...
          3. See also
        9. Calculating social network closeness centrality
          1. Getting ready
          2. How to do it...
          3. See also
        10. Determining the betweenness centrality
          1. Getting ready
          2. How to do it...
          3. See also
        11. Estimating the average clustering coefficient
          1. Getting ready
          2. How to do it...
          3. See also
        12. Calculating the assortativity coefficient of a graph
          1. Getting ready
          2. How to do it...
          3. See also
        13. Getting the clique number of a graph
          1. Getting ready
          2. How to do it...
          3. See also
        14. Creating a document graph with cosine similarity
          1. How to do it...
          2. See also
      16. 9. Ensemble Learning and Dimensionality Reduction
        1. Introduction
        2. Recursively eliminating features
          1. How to do it...
          2. How it works
          3. See also
        3. Applying principal component analysis for dimension reduction
          1. How to do it...
          2. See also
        4. Applying linear discriminant analysis for dimension reduction
          1. How to do it...
          2. See also
        5. Stacking and majority voting for multiple models
          1. How to do it...
          2. See also
        6. Learning with random forests
          1. How to do it...
          2. There's more…
          3. See also
        7. Fitting noisy data with the RANSAC algorithm
          1. How to do it...
          2. See also
        8. Bagging to improve results
          1. How to do it...
          2. See also
        9. Boosting for better learning
          1. How to do it...
          2. See also
        10. Nesting cross-validation
          1. How to do it...
          2. See also
        11. Reusing models with joblib
          1. How to do it...
          2. See also
        12. Hierarchically clustering data
          1. How to do it...
          2. See also
        13. Taking a Theano tour
          1. Getting ready
          2. How to do it...
          3. See also
      17. 10. Evaluating Classifiers, Regressors, and Clusters
        1. Introduction
        2. Getting classification straight with the confusion matrix
          1. How to do it...
          2. How it works
          3. See also
        3. Computing precision, recall, and F1-score
          1. How to do it...
          2. See also
        4. Examining a receiver operating characteristic and the area under a curve
          1. How to do it...
          2. See also
        5. Visualizing the goodness of fit
          1. How to do it...
          2. See also
        6. Computing MSE and median absolute error
          1. How to do it...
          2. See also
        7. Evaluating clusters with the mean silhouette coefficient
          1. How to do it...
          2. See also
        8. Comparing results with a dummy classifier
          1. How to do it...
          2. See also
        9. Determining MAPE and MPE
          1. How to do it...
          2. See also
        10. Comparing with a dummy regressor
          1. How to do it...
          2. See also
        11. Calculating the mean absolute error and the residual sum of squares
          1. How to do it...
          2. See also
        12. Examining the kappa of classification
          1. How to do it...
          2. How it works
          3. See also
        13. Taking a look at the Matthews correlation coefficient
          1. How to do it...
          2. See also
      18. 11. Analyzing Images
        1. Introduction
        2. Setting up OpenCV
          1. Getting ready
          2. How to do it...
          3. How it works
          4. There's more
        3. Applying Scale-Invariant Feature Transform (SIFT)
          1. Getting ready
          2. How to do it...
          3. See also
        4. Detecting features with SURF
          1. Getting ready
          2. How to do it...
          3. See also
        5. Quantizing colors
          1. Getting ready
          2. How to do it...
          3. See also
        6. Denoising images
          1. Getting ready
          2. How to do it...
          3. See also
        7. Extracting patches from an image
          1. Getting ready
          2. How to do it...
          3. See also
        8. Detecting faces with Haar cascades
          1. Getting ready
          2. How to do it...
          3. See also
        9. Searching for bright stars
          1. Getting ready
          2. How to do it...
          3. See also
        10. Extracting metadata from images
          1. Getting ready
          2. How to do it...
          3. See also
        11. Extracting texture features from images
          1. Getting ready
          2. How to do it...
          3. See also
        12. Applying hierarchical clustering on images
          1. How to do it...
          2. See also
        13. Segmenting images with spectral clustering
          1. How to do it...
          2. See also
      19. 12. Parallelism and Performance
        1. Introduction
        2. Just-in-time compiling with Numba
          1. Getting ready
          2. How to do it...
          3. How it works
          4. See also
        3. Speeding up numerical expressions with Numexpr
          1. How to do it...
          2. How it works
          3. See also
        4. Running multiple threads with the threading module
          1. How to do it...
          2. See also
        5. Launching multiple tasks with the concurrent.futures module
          1. How to do it...
          2. See also
        6. Accessing resources asynchronously with the asyncio module
          1. How to do it...
          2. See also
        7. Distributed processing with execnet
          1. Getting ready
          2. How to do it...
          3. See also
        8. Profiling memory usage
          1. Getting ready
          2. How to do it...
          3. See also
        9. Calculating the mean, variance, skewness, and kurtosis on the fly
          1. Getting ready
          2. How to do it...
          3. See also
        10. Caching with a least recently used cache
          1. Getting ready
          2. How to do it...
          3. See also
        11. Caching HTTP requests
          1. Getting ready
          2. How to do it...
          3. See also
        12. Streaming counting with the Count-min sketch
          1. How to do it...
          2. See also
        13. Harnessing the power of the GPU with OpenCL
          1. Getting ready
          2. How to do it...
          3. See also
      20. A. Glossary
      21. B. Function Reference
        1. IPython
        2. Matplotlib
        3. NumPy
        4. pandas
        5. Scikit-learn
        6. SciPy
        7. Seaborn
        8. Statsmodels
      22. C. Online Resources
        1. IPython notebooks and open data
        2. Mathematics and statistics
          1. Presentations
      23. D. Tips and Tricks for Command-Line and Miscellaneous Tools
        1. IPython notebooks
        2. Command-line tools
        3. The alias command
        4. Command-line history
        5. Reproducible sessions
        6. Docker tips
      24. Index