Posted by Adrienne Boswell on 08/08/06 14:29

Gazing into my crystal ball I observed softwarelabus@yahoo.com writing in
news:1155046561.133316.32210@n13g2000cwa.googlegroups.com:

> Hi,
>
> I wanted to warn all website owners that some evil web hosts like
> vistapages will periodically place a robots.txt file on your site that
> disallows all search engines. It happened to me.
>
> Over the last several months I've noticed my web traffic dropped to
> nearly zero. A few days ago I noticed a new file, robots.txt. As most
> of you know, if your site has a robots.txt in your websites home
> directory then all search engines will look at it for possible
> instructions. The robots.txt file tells search engines what to do or
> what not to do. In my case, it had simple instructions to disallow all
> user-agents; i.e., telling all sites they cannot come here.
>
> How to check:
> If you web site is called www.mywebsite.com then you want to check the
> following web page:
> www.mywebsite.com/robots.txt
>
> You should also look for this file when you ftp to your site in case
> your web host places a sneaky server script to make robots.txt
> invisible only to you.
>
> Paul
>
>

I really doubt that this was done with evil intent, probably a misguided
system administrator who got tired of seeing 404 errors, but was too lazy
to look up the robots.txt protocol and get it right.

It's perfectly okay to have blank file, that way the bots are happy, and
the system admins are happy, too.


--
Adrienne Boswell at Home
Arbpen Web Site Design Services
http://www.cavalcade-of-coding.info
Please respond to the group so others can share

[Back to original message]