Lucene 4 Cookbook

Defining custom TokenFilters

Sometimes, search behaviors may be so specific that we need to create a custom TokenFilter to achieve those behaviors. To create a custom filter, we will extend from the TokenFilter class and override the incrementToken() method.

We will create a simple word-expanding TokenFilter that expands courtesy titles from the short form to the full word. For example, Dr expands to doctor.

How to do it…

Here is the sample code:

public class CourtesyTitleFilter extends TokenFilter { Map<String,String> courtesyTitleMap = new HashMap<String,String>(); private CharTermAttribute termAttr; public CourtesyTitleFilter(TokenStream input) { super(input); termAttr = addAttribute(CharTermAttribute.class); courtesyTitleMap.put("Dr", "doctor"); ...