Building Parsers with Java™

11.3. New Token Types

A tokenizer is responsible for breaking text into logical elements. Your idea of what a “logical” element is may differ from the defaults provided by class Tokenizer in sjm.parse.tokens. For example, you may want to distinguish real numbers from imaginary numbers, or HTML tags from other kinds of text, or reserved and nonreserved words. Often, when you create a new type of token, you will create one or more new Terminal types to recognize it.

One important reason for changing a tokenizer is to avoid ambiguity. Section 4.2.2, “Terminal Ambiguity,” shows the ambiguity of the following grammar:

query  = (Word | volume)* '?'; 
volume = "cups" | "gallon" | "liter";

This grammar is ambiguous because a word such as "cups" is both ...