Singular Value Decomposition (SVD) to reduce high-dimensionality in Spark

In this recipe, we will explore a dimensionality reduction method straight out of the linear algebra, which is called SVD (Singular Value Decomposition). The key focus here is to come up with a set of low-rank matrices (typically three) that approximates the original matrix but with much less data, rather than choosing to work with a large M by N matrix.

SVD is a simple linear algebra technique that transforms the original data to eigenvector/eigenvalue low rank matrices that can capture most of the attributes (the original dimensions) in a much more efficient low rank matrix system.

The following figure depicts how SVD can be used to reduce dimensions and then use ...

Get Apache Spark 2.x Machine Learning Cookbook now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.