O'Reilly logo
  • Dennis Hyun thinks this is interesting:

No search engine indexes text directly: rather, the text must be broken into a series of individual atomic elements called tokens. This is what happens during the Analyze Document step

From

Cover of Lucene in Action, Second Edition

Note

Lexical Analyzer!!