O'Reilly logo

Real-Time Big Data Analytics by Shilpi Saxena, Sumit Gupta

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Programming Spark transformations and actions

In this section, we will leverage the various functions exposed by RDD APIs and analyze our Chicago crime dataset. We will start with simple operations and move on to the complex transformations. First, let's create/define some base classes and then we will develop our transformation logic.

Perform the following steps to write the basic building blocks:

  1. We will extend our Spark-Examples projects and create a new Scala class by the name of chapter.seven.ScalaCrimeUtil.scala. This class will contain some utility functions that will be utilized by our main transformation job.
  2. Open and edit ScalaCrimeUtil.scala and add the following piece of code:
    package chapter.seven class ScalaCrimeUtil extends Serializable{ ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required