O'Reilly logo

Apache Spark for Data Science Cookbook by Padma Priya Chitturi

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Machine learning with SparkR

SparkR is integrated with Spark's MLlib machine learning library so that algorithms can be parallelized seamlessly without specifying manually which part of the algorithm can be run in parallel. MLlib is one of the fastest-growing machine learning libraries; hence, the ability to use R with MLlib will create a huge number of contributions to MLlib from R users. As of Spark 1.6, there is support for generalized linear models (Gaussian and binomial) over DataFrames and as per Spark 2.0.2, the algorithms such as Naive Bayes and KMeans are available.

Getting ready

To step through this recipe, you will need a running Spark Cluster either in pseudo distributed mode or in one of the distributed modes, that is, standalone, YARN, ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required