Machine Learning with Apache Spark Quick Start Guide

K-means cost function

One of the challenges when using the k-means clustering algorithm is how to choose a suitable value for k upfront, especially if it is not obvious from the wider context of the use case in question. One method to help us is to plot a range of possible values of k on the x axis against the output of the k-means cost function on the y axis. The k-means cost function computes the total sum of the squared distance of every point to its corresponding cluster centroid for that value of k. The goal is to choose a suitable value of k that minimizes the cost function, but that is not so large that it increases the computational complexity of generating the clusters with only a small return in the reduction in cost. We will demonstrate ...