Home  |  About Alpine SEO  |  Contact Us  |  Site Map    

Search Engine Optimization Strategies: Robots.txt

A robot is a software program that navigates the Internet and automatically reads the information written in the HTML code of your website. Search engine optimization experts know that including a robots.txt file in your root directory can be very helpful. It is often more beneficial to tell the robots what not to spider than to just let them have full access. Search engine optimization professionals use robot.txt files to serve two purposes, to tell the robots where to find the files, or to keep the robots out of certain files. First and foremost, make sure you have only one robots.txt file on your site, and not in the subdirectories, since a robot will never see it. You need to place all the robot exclusion instruction code into a single file.

Another search engine optimization method is to use a robots Meta tag on individual pages instead of the robots.txt file in the root directory.

Note: The following methods rely on the Robot’s cooperation, and are not guaranteed to work for every Robot. If you are looking for better protection from robots and other agents, try using alternative methods such as password protection.

Robots Exclusion Protocol

The Robots.txt Exclusion Protocol allows website administrators to tell visiting robots which parts of their site should not be visited.

In a nutshell, when a Robot visits a website, say http://www.alpineseo.com/, it firsts checks for http://www.alpineseo.com/robots.txt. If it is able to find this document, it will analyze its contents for records like:

User-agent: * Disallow: /

This command will tell it whether or not it is allowed to retrieve the document.

Robots META tag

The Robots META tag allows web administrators to tell visiting robots whether the document may be indexed or used to harvest more links. No server administrator action is required.

Note: Currently only a few robots will implement this.

In the following example, a robot should not index this particular document or analyze it for further links:

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

For more information on Robots we highly recommend you visiting http://www.robotstxt.org/wc/robots.html for more information.

Remember to use all lower case when saving your robots.txt file, and save it into the ROOT directory because that is the only place the robots will look for it. To learn more about robots and other industry trends in search engine marketing sign up for our eNewsletter and stay informed.

Search Engine Optimization Strategies: Anchor Text

Click Here for a FREE Report or to Request a Proposal. Discover for yourself why Alpine Search Engine Optimization is the Intelligent Marketing Solution. Rise Above the Competition... Reach New Heights of Success!

Search Engine Optimization | SEO Process | Products & Services | ROI Reporting
Why Alpine SEO? | Outsource Partnership | Additional Services | About Alpine SEO
Privacy Policy | Contact Us | Site Map

413 Dunne Drive, Fort Collins, CO 80525
Copyright © 2005 Alpine Search Engine Optimization, LLC.  |  All rights reserved.
Phone: (970) 225-3977  |  Fax: (760) 465-8160