There's more...

One must be careful not to mix vector facilities provided by the Breeze library with Spark ML vectors. To work with ML library algorithms, you are required to use its native data structures, but you can always convert from ML vectors to Breeze, do all your math operations, and then convert to Spark's desired data structure when using the ML library algorithms (for example, ALS or SVD).

We need the vector and matrix import statements so we can work with the ML library itself, otherwise the Scala vector and matrix will be used by default. This is the source of much confusion when the programs fail to scale on cluster.

The following figure depicts a pictorial view which should help clarify the subject:

Get Apache Spark 2.x Machine Learning Cookbook now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.