Machine Learning with Apache Spark Quick Start Guide

Feature transformers

The fundamental concept behind natural language processing is treating human text and speech as data—just like the structured and unstructured numerical and categorical data sources we have encountered in this book thus far—while preserving its context. However, natural language is notoriously difficult to understand, even for humans, let alone machines! Not only does natural language consist of hundreds of different spoken languages, with different writing systems, but it also poses other challenges, such as different tones, inflections, slang, abbreviations, metaphors, and sarcasm. Writing systems and communication platforms in particular provide us with text that may contain spelling mistakes, unconventional grammar, ...