Truth 51. Sometimes you don’t want to be found
Search engine optimization is the art and the science of making websites—and the content on those sites—visible and accessible to search engines and to searchers. That’s not always a good thing. Sometimes, you don’t want to be found. What then?
Meet robots.txt.
Think of robots.txt as a spider barrier. The robot exclusion standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent most species of web spiders and other robots from accessing all or part of a website.
Adding a robots.txt file to a website means requesting that cooperative robots ignore specific files or directories. There are lots of good ...
Get The Truth About Search Engine Optimization now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
