O'Reilly logo

Perl & LWP by Sean M. Burke

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Narrowing In

Now, we could try excluding every kind of thing we know we don't want. We could exclude the mailto: link by excluding all URLs that start with mailto:; we could exclude the guest bio URLs by excluding URLs that contain guestinfo; we could exclude the "Previous" and "Next" links by ignoring any URLs with dayFA in them; and we could think of a way to exclude the image URLs. However, tomorrow the people at Fresh Air might add this to their general template:

<a href="buynow.html"><img alt="Buy the Terry Gross mug"
  src="/mug.jpg" width=450 weight=90></a>

Because that isn't explicitly excluded, it would make its way through and appear as a segment link in every program listed.

It is a valid approach to come up with criteria for the kinds of things we don't want to see, but it's usually easier to come up with criteria to capture what we do want to see. So this is what we'll do.

We could characterize the links we're after in several ways:

  1. These links all contain a <font...> ... </font> sequence and a <b> ... </b> sequence.

  2. They all have an <a ...> tag with an href attribute pointing to a URL.

  3. The URL they point to looks like http://www.npr.org/ramfiles/fa/20010702.fa.ram.

  4. Notably, the URL's scheme is http, it's on the server www.npr.org, its path includes ramfiles, and it ends in .ram.

  5. The (trimmed) link text up to /a always begins with Listen to .

Now, of these, the first criterion is most reminiscent of the sort of things we did earlier with the BBC news extractor. But in this case, ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required