Machine Learning with Apache Spark Quick Start Guide

Preprocessing and feature vectorization pipelines

As described earlier, in order to be able to apply our trained decision tree classifier to these real-time tweets, we first need to preprocess and vectorize them exactly as we did with our training and test datasets in Chapter 6, Natural Language Processing Using Apache Spark. However, rather than duplicating the preprocessing and vectorization pipeline logic within our Kafka consumer application itself, we will define our pipeline logic in a separate Python module and within Python functions. This way, any time we need to preprocess text as we did in Chapter 6, Natural Language Processing Using Apache Spark, we simply call the relevant Python function, thereby avoiding the need to duplicate ...