We declared two Scala arrays, parallelized them into two RDDs that are separate vectors of x() and y(). We then used the zip() method from the RDD API to produce a paired (that is, zipped) RDD. It results in an RDD in which each member is an (x , y) pair. We then proceed to calculate the mean, sum, and so on, and apply the closed form formula as described to find the intercept and slope for the regression line.
In Spark 2.0, the alternative would have been to use the GLM API out of the box. It is worth mentioning that the maximum number of parameters for a closed normal form scheme supported by GLM is limited to 4,096.
We used a closed form formula to demonstrate that a regression line associated with a set of numbers (Y1, ...
