Webbots, Spiders, and Screen Scrapers, 2nd Edition

Final Thoughts

Here are a few additional things to remember when writing webbots that access password-protected websites.

• For clarity, the examples in this chapter use a minimal amount of code to perform a task. In actual use, you’ll want to follow the comprehensive practices mentioned elsewhere in this book for downloading pages, parsing results, emulating forms, using PHP/CURL, and writing fault-tolerant webbots.

• It’s important to note that no form of online authentication is effective unless it is accompanied by encryption. After all, it does little good to authenticate users if sensitive information is sent across the network in cleartext, which can be read by anyone with a packet sniffer.^[60] In most cases, authentication will be combined with ...