Machine Learning: Hands-On for Developers and Technical Professionals

Chapter 11Apache Spark

The Apache Spark project was created by the AMPLab at UC Berkeley as a data analytics cluster computing framework. This chapter is a quick overview of the Scala language and its use within the Spark framework. The chapter also looks at the external libraries for machine learning, SQL-like queries, and streaming data with Spark.

Spark: A Hadoop Replacement?

The debate about whether Spark is a Hadoop replacement might rage on longer than some would like. One of the problems with Hadoop is the same thing that made it famous: MapReduce. The programming model can take time to master for certain tasks. If it's a case of straight totaling up frequencies of data, then MapReduce is fine, but after you get past that point, you're left with some hard decisions to make.

Hadoop2 gets beyond the issue of using Hadoop only for MapReduce. With the introduction of YARN (Yet Another Resource Negotiator) Hadoop acts as an operating system for data with YARN controlling resources against the cluster. These resources weren't limited to MapReduce jobs; they could be any job that could be executed. An excellent example of this was the deployment of JBoss application server containers in the book Apache Hadoop YARN by Arun C. Murthy and Vinod Kumar Vavilapalli (Addison-Wesley Professional, 2014; see the “Further Reading” section at the end of this book for more details).

The Spark project doesn't rely on MapReduce, which gives it a speed advantage. The claim is that it's 100 ...