You are previewing Practical Data Analysis Cookbook.
O'Reilly logo
Practical Data Analysis Cookbook

Book Description

Over 60 practical recipes on data exploration and analysis

About This Book

  • Clean dirty data, extract accurate information, and explore the relationships between variables

  • Forecast the output of an electric plant and the water flow of American rivers using pandas, NumPy, Statsmodels, and scikit-learn

  • Find and extract the most important features from your dataset using the most efficient Python libraries

  • Who This Book Is For

    If you are a beginner or intermediate-level professional who is looking to solve your day-to-day, analytical problems with Python, this book is for you. Even with no prior programming and data analytics experience, you will be able to finish each recipe and learn while doing so.

    What You Will Learn

  • Read, clean, transform, and store your data usng Pandas and OpenRefine

  • Understand your data and explore the relationships between variables using Pandas and D3.js

  • Explore a variety of techniques to classify and cluster outbound marketing campaign calls data of a bank using Pandas, mlpy, NumPy, and Statsmodels

  • Reduce the dimensionality of your dataset and extract the most important features with pandas, NumPy, and mlpy

  • Predict the output of a power plant with regression models and forecast water flow of American rivers with time series methods using pandas, NumPy, Statsmodels, and scikit-learn

  • Explore social interactions and identify fraudulent activities with graph theory concepts using NetworkX and Gephi

  • Scrape Internet web pages using urlib and BeautifulSoup and get to know natural language processing techniques to classify movies ratings using NLTK

  • Study simulation techniques in an example of a gas station with agent-based modeling

  • In Detail

    Data analysis is the process of systematically applying statistical and logical techniques to describe and illustrate, condense and recap, and evaluate data. Its importance has been most visible in the sector of information and communication technologies. It is an employee asset in almost all economy sectors.

    This book provides a rich set of independent recipes that dive into the world of data analytics and modeling using a variety of approaches, tools, and algorithms. You will learn the basics of data handling and modeling, and will build your skills gradually toward more advanced topics such as simulations, raw text processing, social interactions analysis, and more.

    First, you will learn some easy-to-follow practical techniques on how to read, write, clean, reformat, explore, and understand your data—arguably the most time-consuming (and the most important) tasks for any data scientist.

    In the second section, different independent recipes delve into intermediate topics such as classification, clustering, predicting, and more. With the help of these easy-to-follow recipes, you will also learn techniques that can easily be expanded to solve other real-life problems such as building recommendation engines or predictive models.

    In the third section, you will explore more advanced topics: from the field of graph theory through natural language processing, discrete choice modeling to simulations. You will also get to expand your knowledge on identifying fraud origin with the help of a graph, scrape Internet websites, and classify movies based on their reviews.

    By the end of this book, you will be able to efficiently use the vast array of tools that the Python environment has to offer.

    Style and approach

    This hands-on recipe guide is divided into three sections that tackle and overcome real-world data modeling problems faced by data analysts/scientist in their everyday work. Each independent recipe is written in an easy-to-follow and step-by-step fashion.

    Downloading the example code for this book. You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub.com/support and register to have the code file.

    Table of Contents

    1. Practical Data Analysis Cookbook
      1. Table of Contents
      2. Practical Data Analysis Cookbook
      3. Credits
      4. About the Author
      5. Acknowledgments
      6. About the Reviewers
      7. www.PacktPub.com
        1. Support files, eBooks, discount offers, and more
          1. Why Subscribe?
          2. Free Access for Packt account holders
      8. Preface
        1. What this book covers
        2. What you need for this book
        3. Who this book is for
        4. Sections
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
        5. Conventions
        6. Reader feedback
        7. Customer support
          1. Downloading the example code
          2. Downloading the color images of this book
          3. Errata
          4. Piracy
          5. Questions
      9. 1. Preparing the Data
        1. Introduction
        2. Reading and writing CSV/TSV files with Python
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
        3. Reading and writing JSON files with Python
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
        4. Reading and writing Excel files with Python
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
        5. Reading and writing XML files with Python
          1. Getting ready
          2. How to do it…
          3. How it works…
        6. Retrieving HTML pages with pandas
          1. Getting ready
          2. How to do it…
          3. How it works…
        7. Storing and retrieving from a relational database
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
        8. Storing and retrieving from MongoDB
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
        9. Opening and transforming data with OpenRefine
          1. Getting ready
          2. How to do it…
          3. See also
        10. Exploring the data with Open Refine
          1. Getting ready
          2. How to do it…
        11. Removing duplicates
          1. Getting ready
          2. How to do it…
        12. Using regular expressions and GREL to clean up data
          1. Getting ready
          2. How to do it…
          3. See also
        13. Imputing missing observations
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
        14. Normalizing and standardizing the features
          1. Getting ready
          2. How to do it…
          3. How it works…
        15. Binning the observations
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
        16. Encoding categorical variables
          1. Getting ready
          2. How to do it…
          3. How it works…
      10. 2. Exploring the Data
        1. Introduction
        2. Producing descriptive statistics
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also…
        3. Exploring correlations between features
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also…
        4. Visualizing the interactions between features
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also…
        5. Producing histograms
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also…
        6. Creating multivariate charts
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also…
        7. Sampling the data
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
        8. Splitting the dataset into training, cross-validation, and testing
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
      11. 3. Classification Techniques
        1. Introduction
        2. Testing and comparing the models
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
        3. Classifying with Naïve Bayes
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
        4. Using logistic regression as a universal classifier
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
        5. Utilizing Support Vector Machines as a classification engine
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
        6. Classifying calls with decision trees
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
        7. Predicting subscribers with random tree forests
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
        8. Employing neural networks to classify calls
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
      12. 4. Clustering Techniques
        1. Introduction
        2. Assessing the performance of a clustering method
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also…
        3. Clustering data with k-means algorithm
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also…
        4. Finding an optimal number of clusters for k-means
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
        5. Discovering clusters with mean shift clustering model
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also…
        6. Building fuzzy clustering model with c-means
          1. Getting ready
          2. How to do it…
          3. How it works…
        7. Using hierarchical model to cluster your data
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also…
        8. Finding groups of potential subscribers with DBSCAN and BIRCH algorithms
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also…
      13. 5. Reducing Dimensions
        1. Introduction
        2. Creating three-dimensional scatter plots to present principal components
          1. Getting ready
          2. How to do it…
          3. How it works…
        3. Reducing the dimensions using the kernel version of PCA
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
        4. Using Principal Component Analysis to find things that matter
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
        5. Finding the principal components in your data using randomized PCA
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
        6. Extracting the useful dimensions using Linear Discriminant Analysis
          1. Getting ready
          2. How to do it…
          3. How it works…
        7. Using various dimension reduction techniques to classify calls using the k-Nearest Neighbors classification model
          1. Getting ready
          2. How to do it…
          3. How it works…
      14. 6. Regression Methods
        1. Introduction
        2. Identifying and tackling multicollinearity
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
        3. Building Linear Regression model
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
        4. Using OLS to forecast how much electricity can be produced
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
        5. Estimating the output of an electric plant using CART
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
        6. Employing the kNN model in a regression problem
          1. Getting ready
          2. How to do it…
          3. How it works…
        7. Applying the Random Forest model to a regression analysis
          1. Getting ready
          2. How to do it…
          3. How it works…
        8. Gauging the amount of electricity a plant can produce using SVMs
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
        9. Training a Neural Network to predict the output of a power plant
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
      15. 7. Time Series Techniques
        1. Introduction
        2. Handling date objects in Python
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
        3. Understanding time series data
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
        4. Smoothing and transforming the observations
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
        5. Filtering the time series data
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
        6. Removing trend and seasonality
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
        7. Forecasting the future with ARMA and ARIMA models
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
      16. 8. Graphs
        1. Introduction
        2. Handling graph objects in Python with NetworkX
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
        3. Using Gephi to visualize graphs
          1. Getting ready
          2. How to do it…
          3. There's more…
          4. See also
        4. Identifying people whose credit card details were stolen
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
        5. Identifying those responsible for stealing the credit cards
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
      17. 9. Natural Language Processing
        1. Introduction
        2. Reading raw text from the Web
          1. Getting ready
          2. How to do it…
          3. How it works…
        3. Tokenizing and normalizing text
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
        4. Identifying parts of speech, handling n-grams, and recognizing named entities
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
        5. Identifying the topic of an article
          1. Getting ready
          2. How to do it…
          3. How it works…
        6. Identifying the sentence structure
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
        7. Classifying movies based on their reviews
          1. Getting ready
          2. How to do it…
          3. How it works…
      18. 10. Discrete Choice Models
        1. Introduction
        2. Preparing a dataset to estimate discrete choice models
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
        3. Estimating the well-known Multinomial Logit model
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
        4. Testing for violations of the Independence from Irrelevant Alternatives
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
        5. Handling IIA violations with the Nested Logit model
          1. Getting ready
          2. How to do it…
          3. How it works…
        6. Managing sophisticated substitution patterns with the Mixed Logit model
          1. Getting ready
          2. How to do it…
          3. How it works…
      19. 11. Simulations
        1. Introduction
        2. Using SimPy to simulate the refueling process of a gas station
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
        3. Simulating out-of-energy occurrences for an electric car
          1. Getting ready
          2. How to do it…
          3. How it works…
        4. Determining if a population of sheep is in danger of extinction due to a wolf pack
          1. Getting ready
          2. How to do it…
          3. How it works…
      20. Index