How do i setup a robot.txt file

I have found information on creating a robot.txt file but nothing as to where to put it when I am done.

I believe that it goes into the root domain directory.

Do I then specify it with the robots meta tag on every site page or just leave it there and let the spider find it on its own?

Any help is apprecitated.

In reply to:

---

Do I then specify it with the robots meta tag on every site page or just leave it there and let the spider find it on its own?

---

Just place it in the root directory for your domain, the spiders that respect robots.txt will find it OK.

Mark

Thanks Mark, I appreciate the intel.

In reply to:

---

Thanks Mark, I appreciate the intel.

---

No problem, I am always glad to help where I can.

It is probably a good idea to point out that not all spiders will respect the contents of your robots.txt file, but in general most legitimate search engine spiders will.

Also, if you study your logs, you will no doubt see many spam-bots and harvesters 'impersonating' legitimate search engines, so don't be alarmed if you see the 'google-bot' ignoring your robots.txt file, it may not be the real google-bot.

Mark