Robots.txt

I looked back and searched, but I still haven't really found the answer to my question, so I'll ask now!

It's about robots.txt files.

Okay.

Where do I put it? Say I am using the WebFTP, I would put it in the
root/domain.tld folder?

And do I have to create one for each subdomain, or would the one main one in the main domain be enough?

http://www.google.com/search?hl=en&q=where+to+put+robots.txt

Yes, yes, no.

In reply to:

---

Where do I put it? Say I am using the WebFTP, I would put it in the
root/domain.tld folder?

---

Yes, the robots.txt file goes into the root of the web-directory for the domain (or sub-domain).

In reply to:

---

do I have to create one for each subdomain, or would the one main one in the main domain be enough?

---

You will need to place the file in the web-directories of the parent domain and each sub-domain.

Mark

Wow - two answers in as many minutes... well done guys.

I had another question regarding the robots.txt files. Some SEO "experts" seem to be espousing the merits of using robots.txt even if there aren't areas that you need to restrict because they increase SE traffic. Is there any weight behind this statement or is it just mumbo-jumbo?

Thanks,
-James

http://www.jconnors.net -

In reply to:

---

Some SEO "experts" seem to be espousing the merits of using robots.txt even if there aren't areas that you need to restrict because they increase SE traffic.

---

I can't say I've noticed a difference in bot traffic between sites that have a robots.txt and those that don't. You can use the Crawl-delay parameter to limit bot traffic, but I know of no robots.txt parameters that will actually increase bot traffic.

I've found that the best way to increase the frequency of crawling on my sites, at least from Google, is to use Google Sitemaps and (of-course) to link to the site from as many places as possible.

Mark

I don't want to limit or delay them. It was just interesting and a little too good to be true when some of these people were saying "oh my traffic went up 16% as soon as I put up the robots.txt"

At this point, I'd just be happy if googlebot came back to my site. It hasn't visited in a couple weeks :-(

http://www.jconnors.net -

In reply to:

---

I'd just be happy if googlebot came back to my site. It hasn't visited in a couple weeks :-(

---

Is this the same site as linked in your signature? If so, then I don't think you'll have too long to wait. In my experience, links in this forum tend to get the attention of the Googlebot.

Mark

It's one in the same - webmaster tools shows the 19th of april the last time it crawled my baby.

Edit: I just looked at the stats on the webserver and it shows that googlebot had been there in the last seven days... yet webmaster tools says a couple weeks - SO CONFUSED!!

http://www.jconnors.net - City StreetsEdited by nalgene1080 on 05/09/07 06:48 AM (server time).

In reply to:

---

webmaster tools shows the 19th of april the last time it crawled my baby.

---

The date shown in webmaster tools seems to be severely lagged, webmaster tools currently shows the following for one of my sites;

"Googlebot last successfully accessed your home page on Mar 30, 2007."

But the site in question is visited many times each day by the Googlebot, as verified by the TraceWatch realtime stats package that I have installed on the site.

Mark

Do all of your pages come up in a site:yourdomain.tld search? Only the pages that were in existence on the 19th come up for me :-(

EDIT: Well here's one for you - I did the site:jconnors.net search and it seems that Gbot has cached some new pages as late as the 4th of May. Maybe it only changes the name if your homepage vs. blog pages changes/updates?

http://www.jconnors.net - City StreetsEdited by nalgene1080 on 05/09/07 06:59 AM (server time).

In reply to:

---

Do all of your pages come up in a site:yourdomain.tld search?

---

Yes. Well, all except the pages added very recently. I've found that it generally takes around a week for newly added pages to show up in the Google index.

Mark