O'Reilly logo

Web Performance Tuning, 2nd Edition by Patrick Killelea

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Automatically Generating Monitoring Scripts Using Sprocket

Now that you know the basics of how to manually write a performance-monitoring script in Perl, I’m going to tell you that you don’t really need to do that. I’ve modified and added to Randal Schwarz’s Perl web proxy server so that it automatically generates monitoring scripts, albeit with some limitations. I call the modified proxy server sprocket. The code is Perl that generates Perl, so it may be hard to follow, but you can download it from http://patrick.net/software/sprocket/sprocket and use it even if you don’t understand exactly how it works.

Here’s how to use it:

  1. First, you’ll need to have the same pieces listed above, all downloadable from http://patrick.net/software/.

  2. Once you have those things installed, get sprocket from http://patrick.net/software/sprocket/sprocket. It is very small and should only take a second or two to download. Put sprocket in a directory from which you can view the resulting PNG images. Your web server’s public_html directory is a good choice.

  3. Now set your web browser’s proxy to the machine sprocket uses by default, port 8008. In Netscape 4, choose Edit Preferences Advanced Proxies Manual Proxy Configuration View HTTP Proxy.

Once your proxy is set, start up sprocket with the -s option for scripting, redirecting the output to the script your want to create. For example:

% sprocket -s > myscript.pl

You’ll see feedback on standard error as your script is being created. For example:

# scripting has started (^C when done)
# set your proxy to <URL:http://localhost:8008/>
# then surf to write a script
# scripted a request
# scripted a request
# scripted a request

In this case, we surfed two pages, resulting in three HTTP requests because one of the pages also contained an image. When you have surfed through the pages you want to monitor, enter Ctrl-C to exit sprocket. (If you try to start it up again right away, you may see a “port in use” error, but that should go away in a minute or so.)

Now myscript.pl contains a script that will duplicate nearly verbatim what you did when you were surfing. The exception is that sprocket will not record Set-Cookie responses, because these are unique to the session you had when you were recording the script. The myscript.pl you just created will listen for new Set-Cookie responses when you run it, so you will have a new session each time you run the script you made. Here is the example myscript.pl file we just created:

#!/usr/bin/perl
use Socket;
use Time::HiRes 'time','sleep';
$proto = getprotobyname('tcp');



#vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
$host   = "vahe";
$port   = 80;
$request = 'GET / HTTP/1.0
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Accept-Charset: iso-8859-1,*,utf-8
Accept-Encoding: gzip
Accept-Language: en
Host: vahe
User-Agent: sprocket/0.10
Proxy-Connection: Keep-Alive



';
$proof  = 'HTTP/1.1 200 OK';

$request            =~ s|[\r\n]*$|\r\n|; # strip extra \r\n
foreach(@cookies) {
   $request        .= "Cookie: $_\r\n"; # tack on cookies
}
$request           .= "\r\n";           # terminate request with blank line

socket(SOCK, PF_INET, SOCK_STREAM, $proto);
$start              = time(  );
$iaddr              = gethostbyname($host);
$paddr              = sockaddr_in($port, $iaddr);
connect(SOCK, $paddr);
$old_fh             = select(SOCK);     # save the old fh
$|                 = 1;                # turn off buffering
select($old_fh);                         # restore

print SOCK $request;

@response           = <SOCK>;
$end                = time(  );

$path               = $request;
$path               =~ m|^[A-Z]* (.*) HTTP|;
$path               = $1;
$file               = $path;
$file               =~ s|/|_|g;

open(FILE, ">>$file") || die "cannot create file";
$date = `date +'%m %d %H %M %S %Y'`;
chop $date;

# validate response here
if (grep(/$proof/, @response)) {
   print FILE $date, " ", $end - $start, "\n";

}
else {
   print FILE $date, " -1\n";
}

close FILE;

$gnuplot_cmd = qq|
   set term png color
   set output "$file.png"
   set xdata time
   set ylabel "latency in seconds"
   set bmargin 3
   set logscale y
   set timefmt "%m %d %H %M %S %Y"
   plot "$file" using 1:7 title "$path" with lines
|;

open(GP, "|/usr/local/bin/gnuplot");
print GP $gnuplot_cmd;
close(GP);

foreach (@response) {
   /Set-Cookie: (.*)/ && push(@cookies, $1);
}

print;
close(SOCK);
#^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^



#vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
$host   = "vahe";
$port   = 80;
$request = 'GET /webpt_sm.gif HTTP/1.0
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png
Accept-Charset: iso-8859-1,*,utf-8
Accept-Encoding: gzip
Accept-Language: en
Host: vahe
Referer: http://vahe/
User-Agent: sprocket/0.10
Proxy-Connection: Keep-Alive



';
$proof  = 'HTTP/1.1 200 OK';

$request            =~ s|[\r\n]*$|\r\n|; # strip extra \r\n
foreach(@cookies) {
   $request        .= "Cookie: $_\r\n"; # tack on cookies
}
$request           .= "\r\n";           # terminate request with blank line

socket(SOCK, PF_INET, SOCK_STREAM, $proto);
$start              = time(  );
$iaddr              = gethostbyname($host);
$paddr              = sockaddr_in($port, $iaddr);
connect(SOCK, $paddr);
$old_fh             = select(SOCK);     # save the old fh
$|                  = 1;                # turn off buffering
select($old_fh);                         # restore

print SOCK $request;

@response           = <SOCK>;
$end                = time(  );

$path               = $request;
$path               =~ m|^[A-Z]* (.*) HTTP|;
$path               = $1;
$file               = $path;
$file               =~ s|/|_|g;

open(FILE, ">>$file") || die "cannot create file";
$date = `date +'%m %d %H %M %S %Y'`;
chop $date;

# validate response here
if (grep(/$proof/, @response)) {
   print FILE $date, " ", $end - $start, "\n";
}
else {
   print FILE $date, " -1\n";
}

close FILE;

$gnuplot_cmd = qq|
   set term png color
   set output "$file.png"
   set xdata time
   set ylabel "latency in seconds"
   set bmargin 3
   set logscale y
   set timefmt "%m %d %H %M %S %Y"
   plot "$file" using 1:7 title "$path" with lines
|;

open(GP, "|/usr/local/bin/gnuplot");
print GP $gnuplot_cmd;
close(GP);

foreach (@response) {
   /Set-Cookie: (.*)/ && push(@cookies, $1);
}

print;
close(SOCK);
#^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^



#vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
$host   = "vahe";
$port   = 80;
$request = 'GET /specs/index.html HTTP/1.0
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Accept-Charset: iso-8859-1,*,utf-8
Accept-Encoding: gzip
Accept-Language: en
Host: vahe
Referer: http://vahe/
User-Agent: sprocket/0.10
Proxy-Connection: Keep-Alive



';
$proof  = 'HTTP/1.1 200 OK';

$request            =~ s|[\r\n]*$|\r\n|; # strip extra \r\n
foreach(@cookies) {
   $request        .= "Cookie: $_\r\n"; # tack on cookies
}
$request           .= "\r\n";           # terminate request with blank line

socket(SOCK, PF_INET, SOCK_STREAM, $proto);
$start             = time(  );
$iaddr              = gethostbyname($host);
$paddr              = sockaddr_in($port, $iaddr);
connect(SOCK, $paddr);
$old_fh             = select(SOCK);     # save the old fh
$|                  = 1;                # turn off buffering
select($old_fh);                         # restore

print SOCK $request;

@response           = <SOCK>;
$end                = time(  );

$path               = $request;
$path               =~ m|^[A-Z]* (.*) HTTP|;
$path               = $1;
$file               = $path;
$file               =~ s|/|_|g;

open(FILE, ">>$file") || die "cannot create file";
$date = `date +'%m %d %H %M %S %Y'`;
chop $date;

# validate response here
if (grep(/$proof/, @response)) {
   print FILE $date, " ", $end - $start, "\n";
}
else {
   print FILE $date, " -1\n";
}

close FILE;

$gnuplot_cmd = qq|
   set term png color
   set output "$file.png"
   set xdata time
   set ylabel "latency in seconds"
   set bmargin 3
   set logscale y
   set timefmt "%m %d %H %M %S %Y"
   plot "$file" using 1:7 title "$path" with lines
|;

open(GP, "|/usr/local/bin/gnuplot");
print GP $gnuplot_cmd;
close(GP);

foreach (@response) {
   /Set-Cookie: (.*)/ && push(@cookies, $1);
}

print;
close(SOCK);
#^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

You can see that the generated script is low-level and contains no subroutines. It is also very repetitive. But these attributes make it very easy to customize. Note that you cannot record an SSL session (unless the user gives permission), because snooping on an SSL session from a proxy would be proof that SSL doesn’t work! But with some effort, you could turn off SSL on your web server, record a session, then modify your generated myscript.pl to use SSL.

The monitoring script will log a timestamp and a latency reading for each item it requests, and automatically generate a graph of the results. The log and graph will appear in the directory sprocket is in. The name of the log file will be the path of the URL with each slash (/) replaced by an underscore (_). The name of the graph will be the name of the log file, but with .png appended.

The first time you run the monitoring script, gnuplot will give a warning that you have only one point on each graph so that it does not know how to adjust the axes. You can ignore this warning.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required