Use a robots.txt file

Navigation

SEO Article Categories

Use a robots.txt file

By: Lee Johnson - SEO Manager - CustomerStreet

Comments ( 0)

Dec
14

Do you have a robots.txt file set up on your server?

Here’s what Google says in their Webmaster Guidelines:

Make use of the robots.txt file on your web server. This file tells crawlers which directories can or cannot be crawled. Make sure it’s current for your site so that you don’t accidentally block the Googlebot crawler.

Visit robotstxt.org to learn how to instruct robots when they visit your site.

If you liked that, then try ...

XML Sitemap Guide
The First Sitemaps One of the newer additions to the Webmasters artillery, and a staple of Customer Street domains, is the XML sitemap.

.htaccess – Blocking IP Addresses, Robots and Offline Browsers
In my last post I covered some of the most common applications of .