Java Data Science Cookbook

Detecting tokens (words) and sentences using OpenNLP

The previous two recipes in this chapter detected tokens (words) and sentences using legacy Java classes and methods in them. In this recipe, we will combine the two tasks of detecting tokens and sentences with an open-source library of Apache named OpenNLP. The reason for introducing OpenNLP with these two tasks that can be accomplished well with the legacy methods is to introduce data scientists to a tool that is really handy and has very high accuracy in several information retrieval tasks on standard and classic corpora. The homepage for OpenNLP can be found at https://opennlp.apache.org/. One strong argument of using this library for tokenization, sentence segmentation, part-of-speech tagging, ...