Apache Spark
Spark was developed as a general-purpose engine for large-scale data processing. It recently released its 1.0 version. Spark has two important features.
The first feature that Spark has is a resilient distributed dataset (RDD). This is a collection of elements partitioned across the nodes of a cluster, which can be operated on in parallel. A file on HDFS or any existing Scala collection can be converted to an RDD collection, and any operation on it can be executed in parallel. RDDs can also be requested to persist in memory, which leads to efficient parallel operations. RDDs have automatic fail-over support and can recover from node failures.
The second important feature of Spark is the concept of shared variables and is used primarily ...
Get Learning Apache Mahout now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
