In this recipe, we cover the Dataset API, which is the way forward for data wrangling in Spark 2.0 and beyond. In Chapter 3, Spark's Three Data Musketeers for Machine Learning - Perfect Together we covered three detailed recipes for dataset, and in this chapter we cover some of the common, repetitive operations that are required to work with these new API sets. Additionally, we demonstrate the query plan generated by the Spark SQL Catalyst optimizer.
Common operations with the new Dataset API
Get Apache Spark 2.x Machine Learning Cookbook now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
