Pig Design Patterns

Rationale for using Pig in data profiling

Implementing the profiling code within the Hadoop environment reduces the dependency on external systems for quality checks. The high-level overview of implementation is depicted in the following diagram:

Implementing profiling in Pig

The following are the advantages of performing data profiling within the Hadoop environment using Pig:

• Implementing the design patterns in Pig reduces data movement by moving the profiling code directly to the data, resulting in performance gains and speeding up the analytics development process.
• By implementing the pattern in Pig, the data quality effort is performed alongside ...