Apache Spark 2.x Machine Learning Cookbook

Logistic regression with L-BFGS optimization in Spark 2.0

In this recipe, we will use the UCI admission dataset again so we can demonstrate Spark's RDD-based logistic regression solution, LogisticRegressionWithLBFGS(), for an extremely large number of parameters that are present in certain types of ML problem.

We recommend L-BFGS for very large variable space since the Hessian matrix of second derivatives can be approximated using updates. If you have an ML problem with millions or billions of parameters, we recommend deep learning techniques.