Scala: Guide for Data Science Professionals

Creating a DataFrame from Scala case classes

In this recipe, we'll see how to create a new DataFrame from Scala case classes.

Note

The code for this recipe can be found at https://github.com/arunma/ScalaDataAnalysisCookbook/blob/master/chapter1-spark-csv/src/main/scala/com/packt/scaladata/spark/csv/DataFrameFromCaseClasses.scala.

How to do it...

1. We create a new entity called Employee with the id and name fields, like this:

   case class Employee(id:Int, name:String)
   

   Similar to the previous recipe, we create SparkContext and SQLContext.

   val conf = new SparkConf().setAppName("colRowDataFrame").setMaster("local[2]")
   
   //Initialize Spark context with Spark configuration.  This is the core entry point to do anything with Spark
   val sc = new SparkContext(conf) ...