O'Reilly logo

Python: Advanced Predictive Analytics by Joseph Babcock, Ashish Kumar

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Streaming clustering in Spark

Up to this point, we have mainly demonstrated examples for ad hoc exploratory analysis. In building up analytical applications, we need to begin putting these into a more robust framework. As an example, we will demonstrate the use of a streaming clustering pipeline using PySpark. This application will potentially scale to very large datasets, and we will compose the pieces of the analysis in such a way that it is robust to failure in the case of malformed data.

As we will be using similar examples with PySpark in the following chapters, let's review the key ingredients we need in such application, some of which we already saw in Chapter 2, Exploratory Data Analysis and Visualization in Python. Most PySpark jobs we ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required