Apache Spark 2.x Machine Learning Cookbook

How it works...

In this recipe, we introduced Spark's Dataset feature which first appeared in Spark 1.6 and which was further refined in subsequent releases. First, we created an instance of a Dataset from a Scala sequence with the help of the createDataset() method belonging to the Spark session. The next step was to print out meta information about the generated Datatset to establish that the creation transpired as expected. Finally, snippets of Spark SQL were used to filter the Dataset by the price column for any price greater than $50, 000.00 and show the final results of execution.