New Robot.txt feature

Dumb notes here...

Wouldn't it be helpful if DH put a folder explorer feature in the new panel page for robots so that we can easily find/select directories that we don't want indexed?

Since contra-bot-botting is such a hot topic these days... Seems to me that robots.txt should really only include those directories that we know are already indexed that we do not want indexed. It's counter-productive to list directories in robot.txt that you don't want indexed because legitimate bots and ne'er-do-wells are going to read the file and from that point on know exactly what you don't want anyone else to know about. In other words, do not list /secret in your robots.txt file! Errr, right?

-Buck

Did you miss the DH newsletter?
http://patrtan.dreamhosters.com/10_2007.php

Thanks for the response, but I didn't miss the newsletter. Did you miss this part of my post?

In reply to:

---

Wouldn't it be helpful if DH put a folder explorer feature in the new panel page for robots so that we can easily find/select directories that we don't want indexed?

---

That page allows us to manually enter folder names and file extentions, but we need to have another window open with our chosen explorer (SSH, FTP, HTTP) in order to figure out what we want to put into those boxes.

And while that allows us to put whatever we want in the boxes, my second inquiry just challenges people to question what they really want to put in there.

I see your point now. If there are a lot of folders in the home folder, it will be easier to have a folder explorer for us to select which one to be indexed or not.

Maybe we should suggest this to DH.