Learning Concurrency in Python

Learning Concurrency in Python

by Elliot Forbes
Released August 2017
Publisher(s): Packt Publishing
ISBN: 9781787285378

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Practically and deeply understand concurrency in Python to write efficient programs

About This Book

  • Build highly efficient, robust, and concurrent applications
  • Work through practical examples that will help you address the challenges of writing concurrent code
  • Improve the overall speed of execution in multiprocessor and multicore systems and keep them highly available

Who This Book Is For

This book is for Python developers who would like to get started with concurrent programming. Readers are expected to have a working knowledge of the Python language, as this book will build on these fundamentals concepts.

What You Will Learn

  • Explore the concept of threading and multiprocessing in Python
  • Understand concurrency with threads
  • Manage exceptions in child threads
  • Handle the hardest part in a concurrent system — shared resources
  • Build concurrent systems with Communicating Sequential Processes (CSP)
  • Maintain all concurrent systems and master them
  • Apply reactive programming to build concurrent systems
  • Use GPU to solve specific problems

In Detail

Python is a very high level, general purpose language that is utilized heavily in fields such as data science and research, as well as being one of the top choices for general purpose programming for programmers around the world. It features a wide number of powerful, high and low-level libraries and frameworks that complement its delightful syntax and enable Python programmers to create.

This book introduces some of the most popular libraries and frameworks and goes in-depth into how you can leverage these libraries for your own high-concurrent, highly-performant Python programs. We'll cover the fundamental concepts of concurrency needed to be able to write your own concurrent and parallel software systems in Python.

The book will guide you down the path to mastering Python concurrency, giving you all the necessary hardware and theoretical knowledge. We'll cover concepts such as debugging and exception handling as well as some of the most popular libraries and frameworks that allow you to create event-driven and reactive systems.

By the end of the book, you'll have learned the techniques to write incredibly efficient concurrent systems that follow best practices.

Style and approach

This easy-to-follow guide teaches you new practices and techniques to optimize your code, and then moves toward more advanced ways to effectively write efficient Python code. Small and simple practical examples will help you test the concepts yourself, and you will be able to easily adapt them for any application.

Table of contents

  1. Title Page
  2. Copyright
    1. Learning Concurrency in Python
  3. Credits
  4. About the Author
  5. About the Reviewer
  6. www.PacktPub.com
    1. Why subscribe?
  7. Customer Feedback
  8. Preface
    1. What this book covers
    2. What you need for this book
    3. Who this book is for
    4. Conventions
    5. Reader feedback
      1. Downloading the example code
      2. Errata
      3. Piracy
      4. Questions
  9. Speed It Up!
    1. History of concurrency
    2. Threads and multithreading
      1. What is a thread?
        1. Types of threads
      2. What is multithreading?
    3. Processes
      1. Properties of processes
    4. Multiprocessing
      1. Event-driven programming
      2. Turtle
        1. Breaking it down
    5. Reactive programming
      1. ReactiveX - RxPy
        1. Breaking it down
    6. GPU programming
      1. PyCUDA
      2. OpenCL
      3. Theano
    7. The limitations of Python
      1. Jython
      2. IronPython
      3. Why should we use Python?
    8. Concurrent image download
      1. Sequential download
        1. Breaking it down
      2. Concurrent download
        1. Breaking it down
    9. Improving number crunching with multiprocessing
      1. Sequential prime factorization
        1. Breaking it down
      2. Concurrent prime factorization
        1. Breaking it down
    10. Summary
  10. Parallelize It
    1. Understanding concurrency
      1. Properties of concurrent systems
    2. I/O bottlenecks
    3. Understanding parallelism
      1. CPU-bound bottlenecks
    4. How do they work on a CPU?
      1. Single-core CPUs
      2. Clock rate
      3. Martelli model of scalability
      4. Time-sharing - the task scheduler
      5. Multi-core processors
    5. System architecture styles
      1. SISD
      2. SIMD
      3. MISD
      4. MIMD
    6. Computer memory architecture styles
      1. UMA
      2. NUMA
    7. Summary
  11. Life of a Thread
    1. Threads in Python
      1. Thread state
      2. State flow chart
        1. Python example of thread state
        2. Breaking it down
      3. Different types of threads
        1. POSIX threads
        2. Windows threads
      4. The ways to start a thread
        1. Starting a thread
        2. Inheriting from the thread class
        3. Breaking it down
      5. Forking
        1. Example
        2. Breaking it down
      6. Daemonizing a thread
        1. Example
        2. Breaking it down
    2. Handling threads in Python
      1. Starting loads of threads
        1. Example
        2. Breaking it down
      2. Slowing down programs using threads
        1. Example
        2. Breaking it down
      3. Getting the total number of active threads
        1. Example
        2. Breaking it down
      4. Getting the current thread
        1. Example
        2. Breaking it down
      5. Main thread
        1. Example
        2. Breaking it down
      6. Enumerating all threads
        1. Example
        2. Breaking it down
      7. Identifying threads
        1. Example
        2. Breakdown
      8. Ending a thread
        1. Best practice in stopping threads
        2. Example
        3. Output
      9. Orphan processes
    3. How does the operating system handle threads
      1. Creating processes versus threads
        1. Example
        2. Breaking it down
    4. Multithreading models
      1. One-to-one thread mapping
      2. Many-to-one
      3. Many-to-many
    5. Summary
  12. Synchronization between Threads
    1. Synchronization between threads
      1. The Dining Philosophers
        1. Example
        2. Output
      2. Race conditions
        1. Process execution sequence
          1. The solution
      3. Critical sections
        1. Filesystem
        2. Life-critical systems
    2. Shared resources and data races
      1. The join method
        1. Breaking it down
        2. Putting it together
      2. Locks
        1. Example
        2. Breaking it down
      3. RLocks
        1. Example
        2. Breaking it down
          1. Output
      4. RLocks versus regular locks
      5. Condition
        1. Definition
        2. Example
          1. Our publisher
          2. Our subscriber
          3. Kicking it off
        3. The results
      6. Semaphores
        1. Class definition
        2. Example
        3. The TicketSeller class
          1. Output
          2. Thread race
      7. Bounded semaphores
      8. Events
        1. Example
        2. Breaking it down
      9. Barriers
        1. Example
        2. Breaking it down
          1. Output
    3. Summary
  13. Communication between Threads
    1. Standard data structures
      1. Sets
        1. Extending the class
        2. Exercise - extending other primitives
      2. Decorator
      3. Class decorator
      4. Lists
      5. Queues
        1. FIFO queues
          1. Example
          2. Breaking it down
          3. Output
        2. LIFO queues
          1. Example
          2. Breaking it down
          3. Output
        3. PriorityQueue
          1. Example
          2. Breakdown
          3. Output
      6. Queue objects
        1. Full/empty queues
          1. Example
          2. Output
        2. The join() function
          1. Example
          2. Breakdown
          3. Output
      7. Deque objects
        1. Example
        2. Breakdown
        3. Output
      8. Appending elements
        1. Example
        2. Breaking it down
        3. Output
      9. Popping elements
        1. Example
        2. Breaking it down
        3. Output
      10. Inserting elements
        1. Example
        2. Breaking it down
        3. Output
      11. Rotation
        1. Example
        2. Breaking it down
        3. Output
    2. Defining your own thread-safe communication structures
      1. A web Crawler example
        1. Requirements
        2. Design
        3. Our Crawler class
        4. Our starting point
        5. Extending the queue object
          1. Breaking it down
          2. Output
        6. Future enhancements
        7. Conclusion
        8. Exercise - testing your skills
    3. Summary
  14. Debug and Benchmark
    1. Testing strategies
      1. Why do we test?
      2. Testing concurrent software systems
      3. What should we test?
      4. Unit tests
        1. PyUnit
          1. Example
          2. Output
        2. Expanding our test suite
      5. Unit testing concurrent code
      6. Integration tests
    2. Debugging
      1. Make it work as a single thread
      2. Pdb
        1. An interactive example
      3. Catching exceptions in child threads
    3. Benchmarking
      1. The timeit module
        1. Timeit versus time
        2. Command-line example
        3. Importing timeit into your code
      2. Utilizing decorators
      3. Timing context manager
        1. Output
    4. Profiling
      1. cProfile
        1. Simple profile example
      2. The line_profiler tool
        1. Kernprof
      3. Memory profiling
        1. Memory profile graphs
    5. Summary
  15. Executors and Pools
    1. Concurrent futures
      1. Executor objects
        1. Creating a ThreadPoolExecutor
          1. Example
          2. Output
        2. Context manager
          1. Example
          2. Output
        3. Maps
          1. Example
          2. Output
        4. Shutdown of executor objects
          1. Example
          2. Output
    2. Future objects
      1. Methods in future objects
        1. The result() method
        2. The add_done_callback() method
        3. The .running() method
        4. The cancel() method
        5. The .exception() method
        6. The .done() method
      2. Unit testing future objects
        1. The set_running_or_notify_cancel() method
        2. The set_result() method
        3. The set_exception() method
      3. Cancelling callable
        1. Example
        2. Output
      4. Getting the result
        1. Example
        2. Output
      5. Using as_completed
        1. Example
        2. Output
      6. Setting callbacks
        1. Example
        2. Output
        3. Chaining callbacks
      7. Exception classes
        1. Example
        2. Output
    3. ProcessPoolExecutor
      1. Creating a ProcessPoolExecutor
        1. Example
        2. Output
      2. Context Manager
        1. Example
        2. Output
      3. Exercise
        1. Getting started
      4. Improving the speed of computationally bound problems
        1. Full code sample
        2. Output
    4. Improving our crawler
      1. The plan
        1. New improvements
        2. Refactoring our code
        3. Storing the results in a CSV file
      2. Exercise - capture more info from each page crawl
    5. concurrent.futures in Python 2.7
    6. Summary
  16. Multiprocessing
    1. Working around the GIL
      1. Utilizing sub-processes
        1. Example
        2. Output
    2. The life of a process
      1. Starting a process using fork
      2. Spawning a process
      3. Forkserver
      4. Daemon processes
        1. Example
        2. Breaking it down
        3. Output
      5. Identifying processes using PIDs
        1. Example
        2. Output
      6. Terminating a process
        1. Example
      7. Getting the current process
      8. Subclassing processes
        1. Example
        2. Output
    3. Multiprocessing pools
      1. The difference between concurrent.futures.ProcessPoolExecutor and Pool
      2. Context manager
        1. Example
        2. Output
      3. Submitting tasks to a process pool
        1. Apply
        2. Apply_async
        3. Map
        4. Map_async
        5. Imap
        6. Imap_unordered
        7. Starmap
        8. Starmap_async
        9. Maxtasksperchild
    4. Communication between processes
      1. Pipes
        1. Anonymous pipes
        2. Named pipes
      2. Working with pipes
        1. Example
      3. Handling Exceptions
        1. Using pipes
    5. Multiprocessing managers
      1. Namespaces
        1. Example
      2. Queues
        1. Example
        2. Output
      3. Listeners and clients
        1. Example
        2. The Listener class
        3. The Client class
        4. Output
      4. Logging
        1. Example
    6. Communicating sequential processes
      1. PyCSP
        1. Processes in PyCSP
        2. Output
    7. Summary
  17. Event-Driven Programming
    1. Event-driven programming
      1. The event loop
    2. Asyncio
      1. Getting started
      2. Event loops
        1. The run_forever() method
        2. The run_until_complete() method
        3. The stop() method
        4. The is_closed() method
        5. The close() function
      3. Tasks
        1. Example
        2. The all_tasks(loop=None) method
        3. The current_tasks() function
        4. The cancel() function
      4. Task functions
        1. The as_completed(fs, *, loop=None, timeout=None) function
        2. The ensure_future(coro_or_future, *, loop=None) function
        3. The wrap_future(future, *, loop=None) function
        4. The gather(*coroes_or_futures, loop=None, return_exceptions=False) function
        5. The wait() function
      5. Futures
        1. Example
        2. Output
      6. Coroutines
        1. Chaining coroutines
        2. Output
      7. Transports
      8. Protocols
      9. Synchronization between coroutines
        1. Locks
        2. Queues
        3. Events and conditions
      10. Semaphores and BoundedSemaphores
      11. Sub-processes
    3. Debugging asyncio programs
      1. Debug mode
    4. Twisted
      1. A simple web server example
    5. Gevent
      1. Event loops
      2. Greenlets
      3. Simple example-hostnames
        1. Output
      4. Monkey patching
    6. Summary
  18. Reactive Programming
    1. Basic reactive programming
      1. Maintaining purity
    2. ReactiveX, or RX
      1. Installing RxPY
      2. Observables
        1. Creating observers
        2. Example
        3. Example 2
        4. Breaking it down
        5. Output
      3. Lambda functions
        1. Example
        2. Breaking it down
        3. On_next, on_completed, and on_error in lambda form
        4. Output
      4. Operators and chaining
        1. Filter example
        2. Breaking it down
        3. Chained operators
      5. The different operators
        1. Creating observables
        2. Transforming observables
        3. Filtering observables
        4. Error-handling observables
      6. Hot and cold observables
      7. Emitting events
        1. Example
        2. Breaking it down
        3. Output
      8. Multicasting
        1. Example
        2. Output
      9. Combining observables
        1. Zip() example
        2. Output
        3. The merge_all() operator
        4. Output
      10. Concurrency
        1. Example
        2. Output
    3. PyFunctional
      1. Installation and official docs
      2. Simple example
        1. Output
      3. Streams, transformations, and actions
      4. Filtering lists
        1. Output
      5. Reading/writing SQLite3
      6. Compressed files
      7. Parallel execution
    4. Summary
  19. Using the GPU
    1. Introduction to GPUs
    2. Why use the GPU?
      1. Data science
        1. Branches of data science
          1. Machine learning
          2. Classification
          3. Cluster analysis
          4. Data mining
    3. CUDA
      1. Working with CUDA without a NVIDIA graphics card
    4. PyCUDA
      1. Features
      2. Simple example
      3. Kernels
      4. GPU arrays
    5. Numba
      1. Overview
      2. Features of Numba
        1. LLVM
      3. Cross-hardware compatibility
        1. Python compilation space
        2. Just-in-Time (JiT) versus Ahead-of-Time (Aot) compilation
        3. The Numba process
        4. Anaconda
        5. Writing basic Numba Python programs
        6. Compilation options
          1. nopython
          2. nogil
          3. The cache option
          4. The parallel option 
        7. Issues with Numba
      4. Numba on the CUDA-based GPUs
      5. Numba on AMD APUs
    6. Accelerate
    7. Theano
      1. Requirements
      2. Getting started
        1. Very simple example
        2. Adding two matrices
        3. Fully-typed constructors
      3. Using Theano on the GPU
        1. Example
      4. Leveraging multiple GPUs
        1. Defining the context map
        2. Simple graph example
    8. PyOpenCL
      1. Example
        1. Output
    9. Summary
  20. Choosing a Solution
    1. Libraries not covered in this book
      1. GPU
        1. PyGPU
      2. Event-driven and reactive libraries
        1. Tornado
        2. Flask
        3. Celery
      3. Data science
        1. Pandas
        2. Matplotlib
        3. TensorFlow
    2. Designing your systems
      1. Requirements
        1. Functional requirements
        2. Non-functional requirements
      2. Design
        1. Computationally expensive
        2. Event-heavy applications
        3. I/O-heavy applications
      3. Recommended design books
        1. Software Architecture with Python
        2. Python: Master the Art of Design Patterns
      4. Research
    3. Summary

Product information

  • Title: Learning Concurrency in Python
  • Author(s): Elliot Forbes
  • Release date: August 2017
  • Publisher(s): Packt Publishing
  • ISBN: 9781787285378