Serving unnecessary requests costs money. For example, blocking bad bots dynamically is a typical use case for using AWS WAF. You will need to create an IPSet containing a list of blocked IP addresses and a rule that blocks requests from these IPs. You will define a web ACL which allows requests by default and contains our rule to exclude blocked IPs. Additionally, you will need define a mechanism to detect bad bots and add their IP addresses to IPSet.
You can use robots.txt to specify which areas of your site or web app should not be scraped and to ensure there are some links pointing to non-scrapable content. Bad bots (ignoring your robots.txt) will request the hidden link and the trigger script will detect the ...
