Yahoo! Hacks

Program Yahoo! with Perl

Grabbing data from Yahoo! Search Web Services can be quite easy with just a little bit of Perl.

Perl is great for getting things done quickly, and fetching search results from Yahoo! is no exception. This hack shows a simple way to access Yahoo! with Perl, using minimal code. Think of it as a doorway to Yahoo! that you can drop into your own Perl scripts, or that you can use as a starting point for more complex applications. This script will accept a keyword or phrase, contact Yahoo! Search, and print out the first 10 results.

What You Need

In the spirit of keeping things easy, this hack uses two simple modules that may already be installed on your system: LWP::Simple (http://search.cpan.org/~gaas/libwww-perl-5.803/lib/LWP/Simple.pm) makes the HTTP request; XML::Simple (http://search.cpan.org/~grantm/XML-Simple-2.14/lib/XML/Simple.pm) parses the XML response. If you need to install these modules, you can use CPAN for each module:

	                 perl –MCPAN –e shell
	cpan> install XML::Simple

On a Windows system with ActivePerl, you can install these modules from the command line with the Perl package manager, like this:

	                 ppm install LWP-Simple

The only other piece you’ll need is a unique application ID from Yahoo!, which you can pick up at http://api.search.yahoo.com/webservices/register_application.

The Code

This code builds a Yahoo! Search Web Services request URL using the keyword passed to it when the script is run. Then it parses the response and prints it out in a readable format. Save the following code to a file named yahoo_search.pl:

	#!/usr/bin/perl
	# yahoo_search.pl
	# Accepts a search term and shows the top results.
	# Usage: yahoo_search.pl <Query>
	#
	# You can create an AppID, and read the full documentation
	# for Yahoo! Web Services at http://developer.yahoo.net/

	use strict;
	use LWP::Simple;
	use XML::Simple;

	# Set your unique Yahoo! Application ID
	my $appID = "insert your app ID";
	
	# Grab the incoming search query 
	my $query = join(' ', @ARGV) or die "Usage: yahoo_search.pl <query>\n";
	# Construct a Yahoo! Search Query with only required options
	my $language = "en";
	my $req_url = "http://api.search.yahoo.com/";

		$req_url .= "WebSearchService/V1/webSearch?";
		$req_url .= "appid=$appID";
		$req_url .= "&query=$query";
		$req_url .= "&language=$language";

	# Make the request
	my $yahoo_response = get($req_url);

	# Parse the XML
	my $xmlsimple = XML::Simple->new();
	my $yahoo_xml = $xmlsimple->XMLin($yahoo_response);

	# Initialize results counter
	my $i;

	# Loop through the items returned, printing them out
	foreach my $result (@{$yahoo_xml->{Result}}) {
		$i++;
		my $title = $result->{Title};
		my $summary = $result->{Summary};
		my $url = $result->{Url};
		print "$i. $title\n$summary\n$url\n\n";
		
	}

The final print command sends the information from Yahoo! to STDOUT. You can change what this script shows by rearranging the variables and making this last line more or less complex.

Running the Hack

Simply call the script from the command line:

	                 
                     perl yahoo_search.pl insert word
                  

And be sure to enclose phrases or multiple keywords in quotes:

	                 
                     perl yahoo_search.pl "insert multiword phrase"

Figure 4-3 shows the Yahoo! Search results for the phrase "minimalist Perl“.

Figure 4-3. Yahoo! Search results for “minimalist Perl”

This hack uses minimalist Perl to demonstrate how quickly Yahoo! data can be included in Perl scripts, and this technique can be used as a building block for more advanced scripts. In fact, most of the Perl scripts in this book use this basic method of accessing Yahoo! Search Web Services.