The most difficult part of working with matrices in Spark is to getting used to column order versus row order. It is key to remember that Spark ML uses underlying libraries that work better with column stored mechanisms. Here is an example to demonstrate:
- Given a matrix definition which defines a 2x2 matrix:
val denseMat3 = Matrices.dense(2,2, Array(10.0, 11.0, 20.0, 30.3))
- The matrix is actually stored as :
10.0 20.0 11.0 30.3
You move from left to right in the value set and then from column to column for the placement in the Matrix.
- As you can see, the assumption that the matrix is stored row wise is not in alignment with the Spark approach. The following order is not correct from Spark's perspective:
10.0 11.0
