Advantages and disadvantages of using Selenium for web scraping

Because WebDriver uses a real web browser to access the web site, there is no difference than browsing the web by a human. When you navigate to a web page using WebDriver, the browser loads all the website resources (JavaScript files, images, css files, and so on) and executes all the JavaScripts on the page. It also keeps all cookies created by your websites. This makes it very difficult to determine whether a real person or a robot has accessed the website. With WebDriver, this can be done in a few simple steps, although it's really hard to simulate all these actions in a program that sends handmade HTTP requests to the server.

Sometimes, the data to be extracted may not be ...

Get R Web Scraping Quick Start Guide now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.