Apache Spark 2.x Machine Learning Cookbook

How it works...

In this recipe, we use the housing data again to demonstrate the Spark 2.0 LinearRegression() API using the L-BFGS optimization option. We read the file in, parse the data, and select specific columns for the regression. We keep the recipe short by accepting default parameters, but set number of iterations (for convergence to a solution) and optimization method to lbfgs before running the .fit() method. We then proceed to output a couple of quick metrics (that is, MSE and RMSE) for demonstration only. We show how to implement/compute these metrics yourself with RDD. Using Spark 2.0 native facilities/metrics and RDDs-based regression recipes, we show how Spark can do these metrics out of the box now, which is testimony to how ...