Setting up Spark
Apache Spark is a project in the Hadoop ecosystem (refer to the Using HDFS recipe), which purportedly performs better than Hadoop's MapReduce. Spark loads data into memory as much as possible, and it has good support for machine learning. In the Clustering data with Spark recipe, we will apply a machine learning algorithm via Spark.
Spark can work standalone, but it is designed to work with Hadoop using HDFS. Resilient Distributed Datasets (RDDs) are the central structure in Spark, and they represent distributed data. Spark has good support for Scala, which is a JVM language, and a somewhat lagging support for Python. For instance, the support to stream in the pyspark API lags a bit. Spark also has the concept of DataFrames, but ...
Get Python Data Analysis Cookbook now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
