Elasticsearch: The Definitive Guide

Part III. Dealing with Human Language

I know all those words, but that sentence makes no sense to me.

Matt Groening

Full-text search is a battle between precision—returning as few irrelevant documents as possible—and recall—returning as many relevant documents as possible. While matching only the exact words that the user has queried would be precise, it is not enough. We would miss out on many documents that the user would consider to be relevant. Instead, we need to spread the net wider, to also search for words that are not exactly the same as the original but are related.

Wouldn’t you expect a search for “quick brown fox” to match a document containing “fast brown foxes,” “Johnny Walker” to match “Johnnie Walker,” or “Arnolt Schwarzenneger” to match “Arnold Schwarzenegger”?

If documents exist that do contain exactly what the user has queried, those documents should appear at the top of the result set, but weaker matches can be included further down the list. If no documents match exactly, at least we can show the user potential matches; they may even be what the user originally intended!

There are several lines of attack:

• Remove diacritics like ´, ^, and ¨ so that a search for rôle will also match role, and vice versa. See Chapter 20.

• Remove the distinction between singular and plural—fox versus foxes—or between tenses—jumping versus jumped versus jumps—by stemming each word to its root form. See Chapter 21.

• Remove commonly used words or stopwords like the, and, and ...