Apache Spark 2.x Machine Learning Cookbook

How to do it...

1. Start a new project in IntelliJ or in an IDE of your choice. Make sure that the necessary JAR files are included.
2. Import the necessary packages for vector and matrix manipulation:

 import org.apache.spark.sql.{SparkSession} import org.apache.spark.mllib.linalg._ import breeze.linalg.{DenseVector => BreezeVector} import Array._ import org.apache.spark.mllib.linalg.SparseVector

3. Set up the Spark session and application parameters so Spark can run:

val spark = SparkSession .builder .master("local[*]") .appName("myVectorMatrix") .config("spark.sql.warehouse.dir", ".") .getOrCreate()

4. Here we look at creating an ML vector feature from Scala arrays. Let us define a 2x2 dense matrix and instantiate it with an array:

val MyArray1= ...