O'Reilly logo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Elegant SciPy

Book Description

Welcome to Scientific Python and its community. If you’re a scientist who programs with Python, this practical guide not only teaches you the fundamental parts of SciPy and libraries related to it, but also gives you a taste for beautiful, easy-to-read code that you can use in practice. You’ll learn how to write elegant code that’s clear, concise, and efficient at executing the task at hand.

Throughout the book, you’ll work with examples from the wider scientific Python ecosystem, using code that illustrates principles outlined in the book. Using actual scientific data, you’ll work on real-world problems with SciPy, NumPy, Pandas, scikit-image, and other Python libraries.

  • Explore the NumPy array, the data structure that underlies numerical scientific computation
  • Use quantile normalization to ensure that measurements fit a specific distribution
  • Represent separate regions in an image with a Region Adjacency Graph
  • Convert temporal or spatial data into frequency domain data with the Fast Fourier Transform
  • Solve sparse matrix problems, including image segmentations, with SciPy’s sparse module
  • Perform linear algebra by using SciPy packages
  • Explore image alignment (registration) with SciPy’s optimize module
  • Process large datasets with Python data streaming primitives and the Toolz library

Table of Contents

  1. Preface
    1. Who Is This Book For?
    2. Why SciPy?
      1. What Is the SciPy Ecosystem?
    3. The Great Cataclysm: Python 2 Versus Python 3
    4. SciPy Ecosystem and Community
      1. Free and Open Source Software (FOSS)
      2. GitHub: Taking Coding Social
      3. Make Your Mark on the SciPy Ecosystem
      4. A Touch of Whimsy with Your Py
    5. Getting Help
    6. Installing Python
      1. Accessing the Book Materials
    7. Diving In
    8. Conventions Used in This Book
    9. Use of Color
    10. Using Code Examples
    11. O’Reilly Safari
    12. How to Contact Us
    13. Acknowledgments
  2. 1. Elegant NumPy: The Foundation of Scientific Python
    1. Introduction to the Data: What Is Gene Expression?
    2. NumPy N-Dimensional Arrays
      1. Why Use ndarrays Instead of Python Lists?
      2. Vectorization
      3. Broadcasting
    3. Exploring a Gene Expression Dataset
      1. Reading in the Data with pandas
    4. Normalization
      1. Between Samples
      2. Between Genes
      3. Normalizing Over Samples and Genes: RPKM
    5. Taking Stock
  3. 2. Quantile Normalization with NumPy and SciPy
    1. Getting the Data
    2. Gene Expression Distribution Differences Between Individuals
    3. Biclustering the Counts Data
    4. Visualizing Clusters
    5. Predicting Survival
      1. Further Work: Using the TCGA’s Patient Clusters
      2. Further Work: Reproducing the TCGA’s clusters
  4. 3. Networks of Image Regions with ndimage
    1. Images Are Just NumPy Arrays
      1. Exercise: Adding a Grid Overlay
    2. Filters in Signal Processing
    3. Filtering Images (2D Filters)
    4. Generic Filters: Arbitrary Functions of Neighborhood Values
      1. Exercise: Conway’s Game of Life
      2. Exercise: Sobel Gradient Magnitude
    5. Graphs and the NetworkX library
      1. Exercise: Curve Fitting with SciPy
    6. Region Adjacency Graphs
    7. Elegant ndimage: How to Build Graphs from Image Regions
    8. Putting It All Together: Mean Color Segmentation
  5. 4. Frequency and the Fast Fourier Transform
    1. Introducing Frequency
    2. Illustration: A Birdsong Spectrogram
    3. History
    4. Implementation
    5. Choosing the Length of the DFT
    6. More DFT Concepts
      1. Frequencies and Their Ordering
      2. Windowing
    7. Real-World Application: Analyzing Radar Data
      1. Signal Properties in the Frequency Domain
      2. Windowing, Applied
      3. Radar Images
      4. Further Applications of the FFT
      5. Further Reading
      6. Exercise: Image Convolution
  6. 5. Contingency Tables Using Sparse Coordinate Matrices
    1. Contingency Tables
      1. Exercise: Computational Complexity of Confusion Matrices
      2. Exercise: Alternative Algorithm to Compute the Confusion Matrix
      3. Exercise: Multiclass Confusion Matrix
    2. scipy.sparse Data Formats
      1. COO Format
      2. Exercise: COO Representation
      3. Compressed Sparse Row Format
    3. Applications of Sparse Matrices: Image Transformations
      1. Exercise: Image Rotation
    4. Back to Contingency Tables
      1. Exercise: Reducing the Memory Footprint
    5. Contingency Tables in Segmentation
    6. Information Theory in Brief
      1. Exercise: Computing Conditional Entropy
    7. Information Theory in Segmentation: Variation of Information
    8. Converting NumPy Array Code to Use Sparse Matrices
    9. Using Variation of Information
      1. Further Work: Segmentation in Practice
  7. 6. Linear Algebra in SciPy
    1. Linear Algebra Basics
    2. Laplacian Matrix of a Graph
      1. Exercise: Rotation Matrix
    3. Laplacians with Brain Data
      1. Exercise: Showing the Affinity View
      2. Exercise Challenge: Linear Algebra with Sparse Matrices
    4. PageRank: Linear Algebra for Reputation and Importance
      1. Exercise: Dealing with Dangling Nodes
      2. Exercise: Equivalence of Different Eigenvector Methods
    5. Concluding Remarks
  8. 7. Function Optimization in SciPy
    1. Optimization in SciPy: scipy.optimize
      1. An Example: Computing Optimal Image Shift
    2. Image Registration with Optimize
    3. Avoiding Local Minima with Basin Hopping
      1. Exercise: Modify the align Function
    4. “What Is Best?”: Choosing the Right Objective Function
  9. 8. Big Data in Little Laptop with Toolz
    1. Streaming with yield
    2. Introducing the Toolz Streaming Library
    3. k-mer Counting and Error Correction
    4. Currying: The Spice of Streaming
    5. Back to Counting k-mers
      1. Exercise: PCA of Streaming Data
    6. Markov Model from a Full Genome
      1. Exercise: Online Unzip
  10. Epilogue
    1. Where to Next?
      1. Mailing Lists
      2. GitHub
      3. Conferences
    2. Beyond SciPy
    3. Contributing to This Book
    4. Until Next Time...
  11. Appendix. Exercise Solutions
    1. Solution: Adding a Grid Overlay
    2. Solution: Conway’s Game of Life
    3. Solution: Sobel Gradient Magnitude
    4. Solution: Curve Fitting with SciPy
    5. Solution: Image Convolution
    6. Solution: Computational Complexity of Confusion Matrices
    7. Solution: Alternative Confusion Matrix Computing
    8. Solution: Computing the Confusion Matrix
    9. Solution: COO Representation
    10. Solution: Image Rotation
    11. Solution: Reducing the Memory Footprint
    12. Solution: Computing Conditional Entropy
    13. Solution: Rotation Matrix
    14. Solution: Showing the Affinity View
    15. Challenge Accepted: Linear Algebra with Sparse Matrices
    16. Solution: Dealing with Dangling Nodes
    17. Solution: Verify Methods
    18. Solution: Modify the align Function
    19. Solution: scikit-learn Library
    20. Solution: Add a Step to the Start of the Pipe
  12. Index