O'Reilly logo
  • First Last thinks this is interesting:

A tokenizer in Lucene is used to divide the text into tokens,

From

Cover of Mastering ElasticSearch

Note

Such as the phrase: The dog jumps. A tokenizer would separate the phrase into tokens/terms --> "The" , "dog", "jumps"