O'Reilly logo

Web Performance Tuning, 2nd Edition by Patrick Killelea

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Monitoring Other Things

You can also monitor the contents of the web pages rather than the latency. For example, the Weblogic application server gives a password-protected web page showing the number of database connections used at any given time. The page is intended only for a human reader, but knowing how to automatically grab web pages, we can “screen scrape” the Weblogic monitoring page and keep a log of the number of database connections used over each day and plot this log. We can snoop the header containing the base 64 encoding of the user ID and password and then use that in our script. Here’s a simple script that will grab the Weblogic T3AdminJDBC page, parse out the number of connections in use, and log that data to a file:

#!/usr/local/bin/perl

use LWP::UserAgent;
use HTTP::Headers;
use HTTP::Request;
use HTTP::Response;

$gnuplot    = "/usr/local/bin/gnuplot";


MAIN: {
   $ua = LWP::UserAgent->new;
   $ua->timeout(60);          # Set timeout to 60 seconds

   &get("http://$ARGV[0]/T3AdminJDBC");

   s/<.*?>/ /g;               # remove all the HTML tags
   /cxn_pool +(\d+) +(\d+)/;  # grab the relevant line
   &log("cxn_pool", $1, $2);

   `$gnuplot *.gp`;
}

sub get {
   local ($url) = @_;

   $request = new HTTP::Request('GET', "$url");

   # This is the base 64 encoding of the weblogic admin login and password.
   $request->push_header("Authorization" => "Basic slkjSLDkf98aljk98797");

   $response = $ua->request($request);

   if (!$response->is_success) {
       #die $request->as_string(  ), " failed: ", $response->error_as_HTML;
       # HTTP redirections are considered failures by LWP.
   }

   # Put response in default string for easy verification.
   $_ = $response->content;
}

# Write log entry in format that gnuplot can use to create a gif.
sub log {

   local ($file, $connections, $pool) = @_;
   $date = `date +'%Y %m %d %H %M %S'`;
   chop $date;

   # Corresponding to gnuplot command: set timefmt "%Y %m %d %H %M %S";

   open(FH, ">>$file") || die "Could not open $file\n";
   printf FH "$date $connections $pool\n";
   close(FH);
}

Once we have our data file, we can plot it using the following gnuplot configuration file:

set term png color
set output "connections.gif"

set xdata time
set timefmt "%H %M %S"
set xrange ["00 00 00":"24 00 00"]
set xlabel "mountain time"
set format x "%H:%M"
set bmargin 3
set ylabel "connections"
set yrange [0:100]

plot "connections.log" using 4:7 title "connections" w l lt 3

And the result is a graph like the one in Figure 4-4.

Graph of database connections over one day, no leak

Figure 4-4. Graph of database connections over one day, no leak

We can also easily store this connection data in a database table itself, as we did with the ps data. Having done that, we can adapt the dynamic graph generation CGI script above to plot connection data. This is very useful for finding “leaks” in database connections — that is, connections that do not get cleaned up after use. This typically happens because of poor or nonexistent error handling, which leaves an unused database connection in limbo. Figure 4-5 is an example CGI-generated graph over 30 days, showing how the database connections build up between restarts of the Weblogic application server. The server was restarted on 10/13, 10/25, 10/30, and 11/3.

Database connections over several weeks, showing leak

Figure 4-5. Database connections over several weeks, showing leak

Using Java for Monitoring

Just because I prefer Perl doesn’t mean you have to use Perl. Here is an example Java program that monitors the first edition of this book’s Amazon sales rank. I wrote a Perl program that does the same thing, and Ian Darwin, the author of O’Reilly’s Java Cookbook, showed me that it’s not much harder in Java. Thanks to Ian for the following example:

import java.io.*;
import com.darwinsys.util.FileIO;
import java.net.*;
import java.text.*;
import java.util.*;
import org.apache.regexp.*;

/** Graph of a book's sales rank on a given bookshop site.
* @author Ian F. Darwin, ian@darwinsys.com, Java Cookbook author,
*      translated fairly literally from Perl into Java.
* @author Patrick Killelea <p@patrick.net>: original Perl version,
*      from the 2nd edition of his book "Web Performance Tuning".
* @version $Id: BookRank.java,v 1.10 2001/04/10 00:28:02 ian Exp $
*/

public class BookRank {
       public final static String ISBN = "0937175307";
       public final static String DATA_FILE = "lint.sales";
       public final static String GRAPH_FILE = "lint.png";
       public final static String TITLE = "Checking C Prog w/ Lint";
       public final static String QUERY = "
                       "http://www.quickbookshops.web/cgi-bin/search?isbn=";

       /** Grab the sales rank off the web page and log it. */
       public static void main(String[] args) throws Exception {

               // Looking for something like this in the input:
               //      <b>QuickBookShop.web Sales Rank: </b>
               //      26,252
               //      </font><br>
               // Patrick Killelea's original RE formulation : match number with
               // comma included, just print minus ",". Loses if fall below 100,000.
               RE r = new RE(" Sales Rank: </b>\\s*(\\d*),*(\\d+)\\s");
               // Java: should use "[\d,]+" to extract the number and
               // NumberFormat.getInstance().parse(  ) to convert to int.

               // Open the URL and get a Reader from it.
               BufferedReader is = new BufferedReader(new InputStreamReader(
                       new URL(QUERY + ISBN).openStream(  )));
               // Read the URL looking for the rank information, as
               // a single long string, so can match RE across multi-lines.
               String input = FileIO.readerToString(is);

               // If found, append to sales data file.
               if (r.match(input)) {
                       PrintWriter FH = new PrintWriter(
                               new FileWriter(DATA_FILE, true));
                       String date = // `date +'%m %d %H %M %S %Y'`;
                               new SimpleDateFormat("MM dd hh mm ss yyyy ").
                               format(new Date(  ));
                       // Paren 1 is the optional thousands; paren 2 is low 3 digits.
                     
                       FH.println(date + r.getParen(1) + r.getParen(2));
                       FH.close(  );
               }

               // Whether current data found or not, draw the graph, using
               // external plotting program against all historical data.
               // Could use gnuplot, R, any other math/graph program.
               // Better yet: use one of the Java plotting APIs.

               String gnuplot_cmd =
                       "set term png color\n" +
                       "set output \"" + GRAPH_FILE + "\"\n" +
                       "set xdata time\n" +
                       "set ylabel \"Book sales rank\"\n" +
                       "set bmargin 3\n" +
                       "set logscale y\n" +
                       "set yrange [1:60000] reverse\n" +
                       "set timefmt \"%Y %m %d %H %M %S\"\n" +
                       "plot \"" + DATA_FILE +
                               "\" using 1:7 title \"" + TITLE + "\" with lines\n"
               ;

               Process p = Runtime.getRuntime(  ).exec("/usr/local/bin/gnuplot");
               PrintWriter gp = new PrintWriter(p.getOutputStream(  ));
               gp.print(gnuplot_cmd);
               gp.close(  );
       }
}

And here’s the original in Perl:

#!/usr/local/bin/perl -w

################################
# Set up user agent and proxy. #
################################

use  LWP::UserAgent;
$ua = LWP::UserAgent->new;
$ua->proxy('http', 'http://httpprox:8080');

#######################################################
# Grab the sales rank off the Amazon page and log it.
#######################################################

$url     = "http://www.amazon.com/exec/obidos/ASIN/1565923790/";
$request = new HTTP::Request('GET', "$url");
$response = $ua->request($request);
$_       = $response->content;
m|Amazon.com Sales Rank: </b>\s*(\d*),*(\d+)\s|s && do {
   open(FH, ">>wpt.sales") || die "Could not open wpt.sales\n";
   $date = `date +'%m %d %H %M %S %Y'`;
   chop $date;
   printf FH "%s %s\n", $date, $1.$2;
   close(FH);
};

#########################
# Regenerate the graph. #
#########################

$gnuplot_cmd = qq|
   set term png color
   set output "sales.png"
   set xdata time
   set ylabel "Amazon sales rank"
   set bmargin 3
   set logscale y
   set yrange [1:30000] reverse
   set timefmt "%m %d %H %M %S %Y"
   plot "wpt.sales" using 1:7 title "web performance tuning" with lines
|;

open(GP, "|/usr/local/bin/gnuplot");
print GP $gnuplot_cmd;
close(GP);

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required