Webbots, Spiders, and Screen Scrapers, 2nd Edition

Final Thoughts

I’ve attempted to make it known that regular expressions are an important, but limited, tool for webbot developers. I know, however, many people use regular expressions on a daily basis and will not agree with my assessment of their role in this specific environment. My opinion is informed by the types of data I’ve had to parse in my career as a developer; your experience may lead to a different opinion. If you’re harvesting data like email addresses, phone numbers, or credit card numbers without needing to know the specific context of that data, then regular expressions are an excellent tool. But if your task is more typical of what I’ve encountered (such as identifying the price of a specific item, the date of a specific auction, ...