O'Reilly logo

Perl & LWP by Sean M. Burke

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Adding Extra Request Header Lines

Here's some simplistic debugging advice: if your browser sees one thing at a given URL, but your LWP program sees another, first try just turning on cookie support, with an empty cookie jar. If that fails, have it read in your browser's cookie file.[1] And if that fails, it's time to start wondering what means the remote site is using for distinguishing your LWP program's requests from your browser's requests.

Every kind of browser sends different HTTP headers besides the very minimal headers that LWP::UserAgent typically sends. For example, whereas an LWP::UserAgent browser by default sends this header line:

User-Agent: libwww-perl/5.5394

Netscape 4.76 sends a header line like this:

User-Agent: Mozilla/4.76 [en] (Win98; U)

And also sends these header fields that an LWP::UserAgent browser doesn't send normally at all:

Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Accept-Charset: iso-8859-1,*,utf-8
Accept-Encoding: gzip
Accept-Language: en-US

(That's assuming you've set your language preferences to U.S. English). That's on top of any Connection: keep-alive headers that may be sent, if the browser or any intervening firewall supports that feature (keep-alive) of HTTP.

Opera 5.12 is not much different:

User-Agent: Opera/5.12 (Windows 98; U)  [en]
Accept: text/html, image/png, image/jpeg, image/gif, image/x-xbitmap, */*
Accept-Language: en
Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0

But a recent version of Netscape ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required