Squash Invading Spiders, Robots.txt Options

[RayBurn]

Squash Invading Spiders

Creating a robots.txt file isn't very difficult. Here are some resources that will help you create such a file if you don't have one:

- How to Set Up a robots.txt to Control Search Engine Spiders
- The how and the why of setting up a robots.txt file.
http://www.thesitewizard.com/archive/robotstxt.shtml

- Robots.txt Validator
- Make sure your robots.txt file is correct with this nifty tool.
http://tool.motoricerca.info/robots-checker.phtml

- Robots.txt file Creator
- An online tool that will create a robots.txt file for you. You still have to understand the settings, but the creator will handle the syntax.
http://www.webtoolcentral.com/webmaster/to...file_generator/

Remember, the robots.txt file goes as the top of your web-site directory structure - the same directory as your home page.