O'Reilly logo

Domain Specific Languages by Martin Fowler

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 28. Alternative Tokenization

Alter the lexing behavior from within the parser.

image

28.1 How It Works

In my simple overview of how Parser Generators work, I said that the lexer feeds a stream of tokens to the parser that assembles the stream into a parse tree. The implication is that it’s a one-way interaction: The lexer is a source that the parser simply consumes. As it turns out that isn’t always the case. There are times when the way the lexer does the tokenizing should change depending on where we are in the parse tree—meaning that the parser has to manipulate the way the lexer does the tokenizing.

For a simple example of this problem, ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required