Creating a DataFrame from Scala case classes
In this recipe, we'll see how to create a new DataFrame from Scala case classes.
Note
The code for this recipe can be found at https://github.com/arunma/ScalaDataAnalysisCookbook/blob/master/chapter1-spark-csv/src/main/scala/com/packt/scaladata/spark/csv/DataFrameFromCaseClasses.scala.
How to do it...
- We create a new entity called
Employee
with theid
andname
fields, like this:case class Employee(id:Int, name:String)
Similar to the previous recipe, we create
SparkContext
andSQLContext
.val conf = new SparkConf().setAppName("colRowDataFrame").setMaster("local[2]") //Initialize Spark context with Spark configuration. This is the core entry point to do anything with Spark val sc = new SparkContext(conf) ...
Get Scala Data Analysis Cookbook now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.