PCollection (data)

We have two PCollection classes: specialized container classes and representing datasets of virtually unlimited size. A PCollection represents a pipeline's input, intermediate, and output data while supporting parallelized processing.

PCollections need to be created for any data to be worked upon. At the same time, we also have to understand that PCollections are immutable; that is, the elements of an existing PCollection cannot be changed. They also don’t support random access to individual elements. Every individual element belongs to a pipeline in which it is created. Hence, it cannot be shared between the pipeline objects. One PCollection can be generated from another PCollection after computation. We have a size factor ...

Get Cloud Analytics with Google Cloud Platform now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.