|
Posted by Paul Furman on 01/05/07 00:17
Bergamot wrote:
> Paul Furman wrote:
>
>>On my other baynatives site I added a line in the longest irrelevant
>>page to prevent indexing that like such:
>><meta name='googlebot' content='noarchive, noindex'>
>
>
> Your subject line indicates you are asking about robots.txt, but you
> haven't mentioned it in either of your messages.
> http://www.robotstxt.org/wc/robots.html
>
I used a googlebot metadata in the one instance.
The best I can figure robots.txt requires you to list each directory
that's forbidden and I've got a huge list or directories that grows
weekly. I was hoping for a robot command that forbids indexing indexless
directories.
Do you know if robots.txt is effected by php calls? All the pages appear
to be located in edgehill.net/1/ but they are really in many deeply
nested subdirectories.
Navigation:
[Reply to this message]
|