You are previewing Programming Pig.
O'Reilly logo
Programming Pig

Book Description

This guide is an ideal learning tool and reference for Apache Pig, the programming language that helps you describe and run large data projects on Hadoop. With Pig, you can analyze data without having to create a full-fledged application—making it easy for you to experiment with new data sets. This book shows you how.

Table of Contents

  1. Programming Pig
    1. SPECIAL OFFER: Upgrade this ebook with O’Reilly
    2. Preface
      1. Data Addiction
      2. Who Should Read This Book
      3. Conventions Used in This Book
      4. Code Examples in This Book
      5. Using Code Examples
      6. Safari® Books Online
      7. How to Contact Us
      8. Acknowledgments
    3. 1. Introduction
      1. What Is Pig?
        1. Pig on Hadoop
          1. MapReduce’s hello world
        2. Pig Latin, a Parallel Dataflow Language
          1. Comparing query and dataflow languages
          2. How Pig differs from MapReduce
        3. What Is Pig Useful For?
        4. Pig Philosophy
      2. Pig’s History
    4. 2. Installing and Running Pig
      1. Downloading and Installing Pig
        1. Downloading the Pig Package from Apache
        2. Downloading Pig from Cloudera
        3. Downloading Pig Artifacts from Maven
        4. Downloading the Source
      2. Running Pig
        1. Running Pig Locally on Your Machine
        2. Running Pig on Your Hadoop Cluster
        3. Running Pig in the Cloud
        4. Command-Line and Configuration Options
        5. Return Codes
    5. 3. Grunt
      1. Entering Pig Latin Scripts in Grunt
      2. HDFS Commands in Grunt
      3. Controlling Pig from Grunt
    6. 4. Pig’s Data Model
      1. Types
        1. Scalar Types
        2. Complex Types
          1. Map
          2. Tuple
          3. Bag
        3. Nulls
      2. Schemas
        1. Casts
    7. 5. Introduction to Pig Latin
      1. Preliminary Matters
        1. Case Sensitivity
        2. Comments
      2. Input and Output
        1. Load
        2. Store
        3. Dump
      3. Relational Operations
        1. foreach
          1. Expressions in foreach
          2. UDFs in foreach
          3. Naming fields in foreach
        2. Filter
        3. Group
        4. Order by
        5. Distinct
        6. Join
        7. Limit
        8. Sample
        9. Parallel
      4. User Defined Functions
        1. Registering UDFs
          1. Registering Python UDFs
        2. define and UDFs
        3. Calling Static Java Functions
    8. 6. Advanced Pig Latin
      1. Advanced Relational Operations
        1. Advanced Features of foreach
          1. flatten
          2. Nested foreach
        2. Using Different Join Implementations
          1. Joining small to large data
          2. Joining skewed data
          3. Joining sorted data
        3. cogroup
        4. union
        5. cross
      2. Integrating Pig with Legacy Code and MapReduce
        1. stream
        2. mapreduce
      3. Nonlinear Data Flows
      4. Controlling Execution
        1. set
        2. Setting the Partitioner
      5. Pig Latin Preprocessor
        1. Parameter Substitution
        2. Macros
        3. Including Other Pig Latin Scripts
    9. 7. Developing and Testing Pig Latin Scripts
      1. Development Tools
        1. Syntax Highlighting and Checking
        2. describe
        3. explain
        4. illustrate
        5. Pig Statistics
        6. MapReduce Job Status
        7. Debugging Tips
      2. Testing Your Scripts with PigUnit
    10. 8. Making Pig Fly
      1. Writing Your Scripts to Perform Well
        1. Filter Early and Often
        2. Project Early and Often
        3. Set Up Your Joins Properly
        4. Use Multiquery When Possible
        5. Choose the Right Data Type
        6. Select the Right Level of Parallelism
      2. Writing Your UDF to Perform
      3. Tune Pig and Hadoop for Your Job
      4. Using Compression in Intermediate Results
      5. Data Layout Optimization
      6. Bad Record Handling
    11. 9. Embedding Pig Latin in Python
      1. Compile
      2. Bind
        1. Binding Multiple Sets of Variables
      3. Run
        1. Running Multiple Bindings
      4. Utility Methods
    12. 10. Writing Evaluation and Filter Functions
      1. Writing an Evaluation Function in Java
        1. Where Your UDF Will Run
        2. Evaluation Function Basics
          1. Interacting with Pig values
        3. Input and Output Schemas
        4. Error Handling and Progress Reporting
        5. Constructors and Passing Data from Frontend to Backend
          1. Loading the distributed cache
          2. UDFContext
        6. Overloading UDFs
        7. Memory Issues in Eval Funcs
      2. Algebraic Interface
      3. Accumulator Interface
      4. Python UDFs
      5. Writing Filter Functions
    13. 11. Writing Load and Store Functions
      1. Load Functions
        1. Frontend Planning Functions
          1. Determining InputFormat
          2. Determining the location
          3. Getting the casting functions
        2. Passing Information from the Frontend to the Backend
        3. Backend Data Reading
          1. Getting ready to read
          2. Reading records
        4. Additional Load Function Interfaces
          1. Loading metadata
          2. Using partitions
          3. Casting bytearrays
          4. Pushing down projections
      2. Store Functions
        1. Store Function Frontend Planning
          1. Determining OutputFormat
          2. Setting the output location
          3. Checking the schema
        2. Store Functions and UDFContext
        3. Writing Data
          1. Preparing to write
          2. Writing records
        4. Failure Cleanup
        5. Storing Metadata
    14. 12. Pig and Other Members of the Hadoop Community
      1. Pig and Hive
      2. Cascading
      3. NoSQL Databases
        1. HBase
        2. Cassandra
      4. Metadata in Hadoop
    15. A. Built-in User Defined Functions and Piggybank
      1. Built-in UDFs
        1. Built-in Load and Store Functions
        2. Built-in Evaluation and Filter Functions
          1. Built-in math UDFs
          2. Built-in aggregate UDFs
          3. Built-in chararray and bytearray UDFs
          4. Built-in complex type UDFs
          5. Built-in filter functions
          6. Miscellaneous built-in UDF
      2. Piggybank
    16. B. Overview of Hadoop
      1. MapReduce
        1. Map Phase
        2. Combiner Phase
        3. Shuffle Phase
        4. Reduce Phase
        5. Output Phase
        6. Distributed Cache
        7. Handling Failure
      2. Hadoop Distributed File System
    17. Index
    18. About the Author
    19. Colophon
    20. SPECIAL OFFER: Upgrade this ebook with O’Reilly