Using concurrent.futures thread pools

The concurrent.futures module offers a second kind of executor that we can use in our applications. Instead of creating a concurrent.futures.ProcessPoolExecutor object, we can use the ThreadPoolExecutor object. This will create a pool of threads within a single process.

The syntax for thread pools is almost identical to using a ProcessPoolExecutor object. The performance, however, is remarkably different. In this example of log file analysis, the work is dominated by I/O. Because all of the threads in a process share the same OS scheduling constraints, the overall performance of multithreaded log file analysis is about the same as processing the log files serially.

Using sample log files and a small four-core ...

Get Functional Python Programming - Second Edition now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.