There's more...

The public interfaces offered by the Spark ML and MLlib library, whether used for sparse or dense vectors, currently lacks the necessary operators to do full vector arithmetic. We must convert our local vectors to the Breeze library vector to have the operators available for linear algebra.

Prior to Spark 2.0, the method for conversion to Breeze (toBreeze) was available to use, but now the method has changed to asBreeze() and made private! A quick read of the source code is necessary to understand the new paradigm. Perhaps the change reflects Spark's core developers' desire to have less dependency on an underlying Breeze library.

If you are using any version of Spark prior to Spark 2.0 (Spark 1.5.1 or 1.6.1), use the following ...

Get Apache Spark 2.x Machine Learning Cookbook now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.