Machine Learning with Apache Spark Quick Start Guide

Stop words

Stop words are common words in a given language that are used to structure a sentence grammatically, but that are not necessarily helpful in determining its underlying meaning or sentiment. For example, in the English language, common stop words include and, I, there, this, and with. A common preprocessing technique is to therefore remove these words from the collection of tokens by filtering based on a language-specific lookup of stop words. Using our previous example, our filtered list of tokens would be ["Machine", "Learning", "Apache", "Spark].