You are previewing Mastering IPython 4.0.
O'Reilly logo
Mastering IPython 4.0

Book Description

Get to grips with the advanced concepts of interactive computing to make the most out of IPython

About This Book

  • Most updated book on Interactive computing with IPython 4.0;

  • Detailed, example-rich guide that lets you use the most advanced level interactive programming with IPython;

  • Get flexible interactive programming with IPython using this comprehensive guide

  • Who This Book Is For

    This book is for IPython developers who want to make the most of IPython and perform advanced scientific computing with IPython utilizing the ease of interactive computing.

    It is ideal for users who wish to learn about the interactive and parallel computing properties of IPython 4.0, along with its integration with third-party tools and concepts such as testing and documenting results.

    What You Will Learn

  • Develop skills to use IPython for high performance computing (HPC)

  • Understand the IPython interactive shell

  • Use XeroMQ and MPI to pass messages

  • Integrate third-party tools like R, Julia, and JavaScript with IPython

  • Visualize the data

  • Acquire knowledge to test and document the data

  • Get to grips with the recent developments in the Jupyter notebook system

  • In Detail

    IPython is an interactive computational environment in which you can combine code execution, rich text, mathematics, plots, and rich media.

    This book will get IPython developers up to date with the latest advancements in IPython and dive deep into interactive computing with IPython. This an advanced guide on interactive and parallel computing with IPython will explore advanced visualizations and high-performance computing with IPython in detail.

    You will quickly brush up your knowledge of IPython kernels and wrapper kernels, then we'll move to advanced concepts such as testing, Sphinx, JS events, interactive work, and the ZMQ cluster. The book will cover topics such as IPython Console Lexer, advanced configuration, and third-party tools.

    By the end of this book, you will be able to use IPython for interactive and parallel computing in a high-performance computing environment.

    Style and approach

    This is a comprehensive guide to IPython for interactive, exploratory and parallel computing. It will let the IPython get up to date with the latest advancements in IPython and dive deeper into interactive computing with IPython

    Downloading the example code for this book. You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub.com/support and register to have the code file.

    Table of Contents

    1. Mastering IPython 4.0
      1. Table of Contents
      2. Mastering IPython 4.0
      3. Credits
      4. About the Author
      5. About the Reviewer
      6. www.PacktPub.com
        1. eBooks, discount offers, and more
          1. Why subscribe?
      7. Preface
        1. What this book covers
        2. What you need for this book
        3. Who this book is for
        4. Conventions
        5. Reader feedback
        6. Customer support
          1. Downloading the example code
          2. Downloading the color images of this book
          3. Errata
          4. Piracy
          5. Questions
      8. 1. Using IPython for HPC
        1. The need for speed
        2. FORTRAN to the rescue – the problems FORTRAN addressed
          1. Readability
          2. Portability
          3. Efficiency
          4. The computing environment
        3. Choosing between IPython and Fortran
          1. Fortran
          2. IPython
            1. Object-orientation
            2. Ease of adoption
          3. Popularity – Fortran versus IPython
            1. Useful libraries
          4. The cost of building (and maintaining) software
            1. Requirements and specification gathering
            2. Development
            3. Execution
            4. Testing and maintenance
          5. Alternatives
            1. Cross-language development
            2. Prototyping and exploratory development
        4. An example case – Fast Fourier Transform
          1. Fast Fourier Transform
            1. Fortran
            2. Python
          2. Performance concerns
          3. Software engineering concerns
            1. Complexity-based metrics
            2. Size-based metrics
          4. Where we stand now
        5. High Performance Computing
          1. The HPC learning curve
          2. Cloudy with a chance of parallelism (or Amazon's computer is bigger than yours)
          3. HPC and parallelism
          4. Clouds and HPC
        6. Going parallel
          1. Terminology
          2. A parallel programming example
            1. A serial program
            2. A parallel equivalent
            3. Discussion
        7. Summary
      9. 2. Advanced Shell Topics
        1. What is IPython?
        2. Installing IPython
        3. All-in-one distributions
          1. Package management with conda
          2. Canopy Package Manager
        4. What happened to the Notebook?
        5. Starting out with the terminal
        6. IPython beyond Python
          1. Shell integration
          2. History
        7. Magic commands
          1. Creating custom magic commands
        8. Cython
        9. Configuring IPython
        10. Debugging
          1. Post-mortem debugging
          2. Debugging at startup
          3. Debugger commands
        11. Read-Eval-Print Loop (REPL) and IPython architecture
        12. Alternative development environments
          1. Spyder
          2. Canopy
          3. PyDev
          4. Others
        13. Summary
      10. 3. Stepping Up to IPython for Parallel Computing
        1. Serial processes
          1. Program counters and address spaces
          2. Batch systems
          3. Multitasking and preemption
          4. Time slicing
        2. Threading
          1. Threading in Python
          2. Example
          3. Limitations of threading
          4. Global Interpreter Lock
            1. What happens in an interpreter?
            2. CPython
            3. Multi-core machines
            4. Kill GIL
        3. Using multiple processors
        4. The IPython parallel architecture
          1. Overview
          2. Components
            1. The IPython Engine
            2. The IPython Controller
            3. The IPython Hub
            4. The IPython Scheduler
        5. Getting started with ipyparallel
          1. ipcluster
          2. Hello world
          3. Using map_sync
          4. Asynchronous calls
          5. Synchronizing imports
        6. Parallel magic commands
          1. %px
          2. %%px
          3. %pxresult
          4. %pxconfig
          5. %autopx
        7. Types of parallelism
          1. SIMD
          2. SPMD
            1. ipcluster and mpiexec/mpirun
            2. ipcluster and PBS
              1. Starting the engines
              2. Starting the controller
              3. Using the scripts
          3. MapReduce
            1. Scatter and gather
            2. A more sophisticated method
          4. MIMD
          5. MPMD
          6. Task farming and load balancing
            1. The @parallel function decorator
        8. Data parallelism
          1. No data dependence
          2. External data dependence
        9. Application steering
          1. Debugging
          2. First to the post
          3. Graceful shutdown
        10. Summary
      11. 4. Messaging with ZeroMQ and MPI
        1. The storage hierarchy
          1. Address spaces
          2. Data locality
        2. ZeroMQ
          1. A sample ZeroMQ program
            1. The server
            2. The client
          2. Messaging patterns in ZeroMQ
            1. Pairwise
              1. Server
              2. Client
              3. Discussion
            2. Client/server
              1. Server 1
              2. Server 2
              3. Client
              4. Discussion
            3. Publish/subscribe
              1. Publisher
              2. Subscriber
              3. Discussion
            4. Push/Pull
              1. Ventilator
              2. Worker
              3. Sink
              4. Discussion
          3. Important ZeroMQ features
          4. Issues using ZeroMQ
            1. Startup and shutdown
            2. Discovery
        3. MPI
          1. Hello World
          2. Rank and role
          3. Point-to-point communication
          4. Broadcasting
          5. Reduce
          6. Discussion
            1. Change the configuration
            2. Divide the work
            3. Parcel out the work
          7. Process control
            1. Master
            2. Worker
        4. ZeroMQ and IPython
          1. ZeroMQ socket types
          2. IPython components
            1. Client
            2. Engine(s)
            3. Controller
            4. Hub
            5. Scheduler
            6. Connection diagram
          3. Messaging use cases
            1. Registration
            2. Heartbeat
            3. IOPub
        5. Summary
      12. 5. Opening the Toolkit – The IPython API
        1. Performance profiling
          1. Using utils.timing
          2. Using %%timeit
          3. Using %%prun
        2. The AsyncResult class
          1. multiprocessing.pool.Pool
            1. Blocking methods
            2. Nonblocking methods
            3. Obtaining results
            4. An example program using various methods
          2. mp.pool.AsyncResult
            1. Getting results
            2. An example program using various methods
          3. AsyncResultSet metadata
            1. Metadata keys
            2. Other metadata
        3. The Client class
          1. Attributes
          2. Methods
        4. The View class
          1. View attributes
          2. Calling Python functions
            1. Synchronous calls
            2. Asynchronous calls
            3. Configurable calls
            4. Job control
          3. DirectView
            1. Data movement
              1. Dictionary-style data access
              2. Scatter and gather
              3. Push and pull
            2. Imports
              1. Discussion
          4. LoadBalancedView
            1. Data movement
            2. Imports
        5. Summary
      13. 6. Works Well with Others – IPython and Third-Party Tools
        1. R
          1. The rpy2 module/extension
            1. Installing rpy2
          2. Using Rmagic
            1. The %R magic
            2. The %%R magic
            3. Pulling and pushing
          3. Graphics
          4. Using rpy2.robjects
            1. The basics
            2. Interpreting a string as R
        2. Octave
          1. The oct2py module/extension
            1. Installing oct2py
          2. Using Octave magic
            1. The %octave magic
              1. Tricky issues
          3. The %%octave magic
          4. Pushing and pulling
          5. Graphics
          6. Using the Octave module
            1. Pushing and pulling
            2. Running Octave code
        3. Hy
          1. The hymagic module/extension
            1. Installing hymagic
          2. Using hymagic
            1. The %hylang magic
            2. The %%hylang magic
          3. A quick introduction to Hy
            1. Hello world!
            2. Get used to parentheses
            3. Arithmetic operations are in the wrong place
            4. Function composition is everywhere
            5. Control structures in Hy
              1. Setting variable values
              2. Defining functions
              3. if statements
              4. Conditionals
              5. Loops
            6. Calling Python
        4. Summary
      14. 7. Seeing Is Believing– Visualization
        1. Matplotlib
          1. Starting matplotlib
          2. An initial graph
          3. Modifying the graph
          4. Controlling interactivity
        2. Bokeh
          1. Starting Bokeh
          2. An initial graph
          3. Modifying the graph
            1. Customizing graphs
          4. Interactive plots
            1. An example interactive plot
        3. R
          1. Installing ggplot2 and pandas
            1. Using DataFrames
          2. An initial graph
            1. Modifying the graph
            2. A different view
        4. Python-nvd3
          1. Starting Python-nvd3
          2. An initial graph
          3. Putting some tools together
          4. A different type of plot
        5. Summary
      15. 8. But It Worked in the Demo! – Testing
        1. Unit testing
          1. A quick introduction
            1. Assertions
            2. Environmental issues
              1. Before it starts – setup
              2. While it is running – mocks
              3. After it finishes – teardown
            3. Writing to be tested
        2. unittest
          1. Important concepts
          2. A test using setUp and tearDown
            1. One-time setUp and tearDown
          3. Decorators
        3. pytest
          1. Installation
          2. Back compatibility
          3. Test discovery
          4. Organizing test files
          5. Assertions
          6. A test using setUp and tearDown
            1. Classic xUnit-style
              1. Being verbose
          7. Using fixtures
            1. Skipping and failing
          8. Monkeypatching
        4. nose2
          1. Installation
          2. Back compatibility
          3. Test discovery
            1. Running individual tests
          4. Assertions, setup, and teardown
            1. Modified xUnit-style
          5. Using decorators
          6. Plugins
            1. Generating XML with the junitxml plugin
        5. Summary
      16. 9. Documentation
        1. Inline comments
          1. Using inline comments
          2. Function annotations
            1. Syntax
              1. Parameters
              2. Return values
            2. Semantics
          3. Type hints
            1. Syntax
            2. Semantics
        2. Docstrings
          1. Example
            1. Inheriting docstrings
          2. Recommended elements
            1. One-line docstrings
            2. Syntax
              1. Semantics
            3. Multiline docstrings
              1. Syntax
              2. Semantics
                1. The API
                2. Inputs
                3. Functionality
                4. Outputs
                5. Error conditions
                6. Relationship with other parts of the system
                7. Example uses
          3. Example
        3. reStructuredText
          1. History and goals
          2. Customers
          3. The solution
          4. Overview
            1. Paragraphs
            2. Text styles
              1. Literal blocks
            3. Lists
              1. Enumerated lists
              2. Bulleted lists
              3. Definition lists
            4. Hyperlinks
            5. Sections
        4. Docutils
          1. Installation
          2. Usage
          3. Documenting source files
        5. Sphinx
          1. Installation and startup
          2. Specifying the source files
        6. Summary
      17. 10. Visiting Jupyter
        1. Installation and startup
        2. The Dashboard
        3. Creating a notebook
        4. Interacting with Python scripts
        5. Working with cells
          1. Cell tricks
            1. Cell scoping
            2. Cell execution
            3. Restart & Run All
            4. Magics
          2. Cell structure
            1. Code cells
            2. Markdown cells
            3. Raw cells
            4. Heading cells
        6. Being graphic
          1. Using matplotlib
          2. Using Bokeh
          3. Using R
          4. Using Python-nvd3
        7. Format conversion
          1. Other formats
          2. nbviewer
        8. Summary
      18. 11. Into the Future
        1. Some history
        2. The Jupyter project
          1. The Notebook
          2. The console
          3. Jupyter Client
          4. The future of Jupyter
            1. Official roadmap
            2. Official subprojects
              1. Direct creation
              2. Incorporation
            3. Incubation
              1. External incubation
        3. IPython
          1. Current activity
        4. The rise of parallelism
          1. The megahertz wars end
          2. The problem
          3. A parallel with the past
          4. The present
            1. Problems are getting bigger and harder
            2. Computers are becoming more parallel
            3. Clouds are rolling in
            4. There is no Big Idea
              1. Pragmatic evolution of techniques
              2. Better tools
              3. The Next Big Idea
        5. Growing professionalism
          1. The NSF
            1. Software Infrastructure for Sustained Innovation
        6. Summary
      19. Index