Posted by William Tasso on 08/08/06 19:56

Fleeing from the madness of the arbpen.com jungle
Adrienne Boswell <arbpen@yahoo.com> stumbled into
news:alt.html,alt.internet.search-engines,alt.www.webmaster,uk.net.web.authoring
and said:

> Gazing into my crystal ball I observed softwarelabus@yahoo.com writing in
> news:1155046561.133316.32210@n13g2000cwa.googlegroups.com:
>
>> Hi,
>>
>> I wanted to warn all website owners that some evil web hosts like
>> vistapages will periodically place a robots.txt file on your site that
>> disallows all search engines. It happened to me.
>>
>> Over the last several months I've noticed my web traffic dropped to
>> nearly zero. A few days ago I noticed a new file, robots.txt. As most
>> of you know, if your site has a robots.txt in your websites home
>> directory then all search engines will look at it for possible
>> instructions. The robots.txt file tells search engines what to do or
>> what not to do. In my case, it had simple instructions to disallow all
>> user-agents; i.e., telling all sites they cannot come here.

ouch!

>> ...
>> You should also look for this file when you ftp to your site in case
>> your web host places a sneaky server script to make robots.txt
>> invisible only to you.

How would your host (reliably) know your IP?

> I really doubt that this was done with evil intent,

I doubt it was done by a sys admin at all.

> probably a misguided
> system administrator who got tired of seeing 404 errors, but was too lazy
> to look up the robots.txt protocol and get it right.

far fetched? ok, plausible.

> It's perfectly okay to have blank file, that way the bots are happy, and
> the system admins are happy, too.

Certainly it is

Speaking generally (and not to the o/p in particular) I'd suspect the
developer had been experimenting and forgotton to delete the file before
the latest sync/upload.
--
William Tasso

http://williamtasso.com/words/what-is-usenet.asp

[Back to original message]