O'Reilly logo

Search Engine Optimization Bible, Second Edition by Jerri L. Ledford

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 18. Robots, Spiders, and Crawlers

IN THIS CHAPTER

  • What are robots, spiders, and crawlers?

  • What's the Robot Exclusion Standard?

  • Robots meta tag

  • Inclusion with XML site mapping

You're getting well along in the book, so you should have a pretty good handle on what exactly robots, spiders, and crawlers are, right? No doubt you do, but did you know that there is much more to these Internet creatures than just the fact that they crawl from one web site to another?

Spiders, robots, crawlers, or whatever else you choose to call them can determine how well you rank in search engines, so it's best to make friends with them as quickly as possible. Certain strategies will help you find favor with these crawlers (which is the name we'll use to lump them all together), whereas other strategies, unfortunately, will help you find your way right out of search engine rankings.

A sad fact about crawlers is that sometimes you'll be treated unfairly when you've done absolutely nothing wrong. Some of the strategies that you may want to use to improve the optimization of your site can look suspicious to crawlers. When they see those strategies they will automatically penalize you for trying to spam the search engine, even when you're not.

That's what makes it so important for you to understand not only the intimate details about how crawlers work, but also about what makes them happy and what makes them move on without looking back.

What Are Robots, Spiders, and Crawlers?

You should already have a general ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required