After we define the portrule, the only step left is to define the action that executes when the portrule returns
true. In our case, we want to check whether
robots.txt exists on the web server we're scanning.
In order to determine whether the server exists, there's a little bit about the Hypertext Transfer Protocol (HTTP) that we need to learn. First of all, the way to request a page is through an HTTP GET request. For example, if we wanted to go to http://google.com/images, our browser would send a request containing
GET /images to the server at
If the status of the GET request is
OK, the web server returns the status code
200. If there is a server-side error, a
500 error will return. If the page is moved, ...