O'Reilly logo

Perl & LWP by Sean M. Burke

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Token Sequences

Some problems cannot be solved with a single-token approach. Often you need to scan for a sequence of tokens. For example in Chapter 4, we extracted the Amazon sales rank from HTML like this:

<b>Amazon.com Sales Rank: </b> 4,070 </font><br>

Here we're looking for the text Amazon.com Sales Rank: , an end-tag for b, and the next token as a text token with the sales rank. To solve this, we need to check the next few tokens while being able to put them back if they're not what we expect.

To put tokens back into the stream, use the unget_token( ) method:

$stream->unget_token(@next);

The tokens stored in @next will be returned to the stream. For example, to solve our Amazon problem:

while (my $token = $stream->get_token(  )) {
  if ($token->[0] eq 'T' and
    $token->[1] eq 'Amazon.com Sales Rank: ') {
    my @next;
    push @next, $stream->get_token(  );
    my $found = 0;
    if ($next[0][0] eq 'E' and $next[0][1] eq 'b') {
      push @next, $stream->get_token(  );
      if ($next[1][0] eq 'T') {
        $sales_rank = $next[1][1];
        $found = 1;
      }
    }
    $stream->unget_token(@next) unless $found;
  }
}

If it's the text we're looking for, we cautiously explore the next tokens. If the next one is a </b> end-tag, check the next token to ensure that it's text. If it is, then that's the sales rank. If any of the tests fail, put the tokens back on the stream and go back to processing.

Example: BBC Headlines

Suppose, for example, that your morning ritual is to have the help come and wake you at about 11 a.m. as they bring two serving ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required