Spark for Data Science

Singular Value Decomposition

The Singular Value Decomposition (SVD) is one of the centerpieces of linear algebra and is widely used for many real-world modeling requirements. It provides a convenient way of breaking a matrix into simpler, smaller matrices. This leads to a low-dimensional representation of a high-dimensional matrix. It helps us eliminate less important parts of the matrix to produce an approximate representation. This technique is useful in dimensionality reduction and data compression.

Let M be a matrix of size m-rows and n-columns. The rank of a matrix is the number of rows that are linearly independent. A row is considered independent if it has at least one non-zero element and it is not a linear combination of one or more rows. ...