Webbots, Spiders, and Screen Scrapers, 2nd Edition

Parsing Poorly Written HTML

Another problem you’ll encounter when parsing web pages is poorly written HTML. A large amount of HTML is machine generated and shows little regard for human readability. Handwritten HTML is often no better, as it often violates accepted standards by ignoring closing tags or by misusing quotes. Browsers may correctly render web pages that have substandard HTML, but your webbot may have trouble parsing them.

Fortunately, a software library known as HTML Tidy^[14] will clean up poorly written web pages. PHP includes HTML Tidy in its standard distributions, so you should have no problem getting it running on your computer. Installing HTML Tidy (also known as just Tidy) should be similar to installing PHP/CURL. Complete installation ...