Functional Python Programming - Second Edition

Parsing log files – gathering the rows

Here is the first stage in parsing a large number of files: reading each file and producing a simple sequence of lines. As the log files are saved in the .gzip format, we need to open each file with the gzip.open() function instead of the io.open() function or the __builtins__.open() function.

The local_gzip() function reads lines from locally cached files, as shown in the following command snippet:

from typing import Iteratordef local_gzip(pattern: str) -> Iterator[Iterator[str]]:    zip_logs= glob.glob(pattern)
    for zip_file in zip_logs:
        with gzip.open(zip_file, "rb") as log:
            yield (                line.decode('us-ascii').rstrip()                 for line in log)

The preceding function iterates through all files that match the given ...