Functional Python Programming - Second Edition

Using the concurrent.futures module

In addition to the multiprocessing package, we can also make use of the concurrent.futures module. This also provides a way to map data to a concurrent pool of threads or processes. The module API is relatively simple and similar in many ways to the multiprocessing.Pool() function's interface.

Here is an example to show just how similar they are:

from concurrent.futures import ProcessPoolExecutor
pool_size = 4
pattern = "*.gz"
combined = Counter()
with ProcessPoolExecutor(max_workers=pool_size) as workers:
    for result in workers.map(analysis, glob.glob(pattern)):
        combined.update(result) 

The most significant change between the preceding example and previous examples is that we're using an instance of the ...