Spark Cookbook

Word count using Streaming

Let's start with a simple example of Streaming in which in one terminal, we will type some text and the Streaming application will capture it in another window.

How to do it...

1. Start the Spark shell and give it some extra memory:

   $ spark-shell --driver-memory 1G
   

2. Stream specific imports:

   scala> import org.apache.spark.SparkConf
   scala> import org.apache.spark.streaming.{Seconds, StreamingContext}
   scala> import org.apache.spark.storage.StorageLevel
   scala> import StorageLevel._
   

3. Import for an implicit conversion:

   scala> import org.apache.spark._
   scala> import org.apache.spark.streaming._
   scala> import org.apache.spark.streaming.StreamingContext._
   

4. Create StreamingContext with a 2 second batch interval:

   scala> val ssc = new StreamingContext(sc, ...