Apache Spark Machine Learning Blueprints

Data and feature preparation

In the Feature extraction section of Chapter 2, Data Preparation for Spark ML, we reviewed a few methods of feature extraction and discussed their implementation in Apache Spark. All the techniques discussed there can be applied to our data here.

Besides feature development, for this project, we will also need to spend a lot of effort in merging various datasets together to obtain more features.

Therefore, for this project, we actually need to conduct feature development, then data merging, and then feature selection, which is to utilize all the techniques discussed in Chapter 2, Data Preparation for Spark ML and Chapter 3, A Holistic View on Spark.

Data merging

To obtain features for predicting, we need to add some external ...