From a functional programming perspective, we've seen three ways to use the map() function concept applied to data items concurrently. We can use any one of the following:
- multiprocessing.Pool
- concurrent.futures.ProcessPoolExecutor
- concurrent.futures.ThreadPoolExecutor
These are almost identical in the way we interact with them; all three have a map() method that applies a function to items of an iterable collection. This fits in elegantly with other functional programming techniques. The performance is different because of the nature of concurrent threads versus concurrent processes.
As we stepped through the design, our log analysis application decomposed into two overall areas:
- The lower-level parsing: ...
