Apache Spark Machine Learning Blueprints

Data and feature development

In the Feature extraction section of Chapter 2, Data Preparation for Spark ML, we have reviewed a few methods for feature extraction and discussed their implementation on Apache Spark. All the techniques discussed there will be applied to our datasets here.

Besides feature development, for this project, we will also need to spend a lot of effort to merge various datasets together to obtain more features.

Therefore, for this project, we actually need to conduct feature development, then data merging and reorganizing, and then feature selection, which is to utilize all the techniques discussed in Chapter 2, Data Preparation for Spark ML, and also in Chapter 3, A Holistic View on Spark. A lot of work has been completed ...