Posted by softwarelabus on 08/08/06 14:16

Hi,

I wanted to warn all website owners that some evil web hosts like
vistapages will periodically place a robots.txt file on your site that
disallows all search engines. It happened to me.

Over the last several months I've noticed my web traffic dropped to
nearly zero. A few days ago I noticed a new file, robots.txt. As most
of you know, if your site has a robots.txt in your websites home
directory then all search engines will look at it for possible
instructions. The robots.txt file tells search engines what to do or
what not to do. In my case, it had simple instructions to disallow all
user-agents; i.e., telling all sites they cannot come here.

How to check:
If you web site is called www.mywebsite.com then you want to check the
following web page:
www.mywebsite.com/robots.txt

You should also look for this file when you ftp to your site in case
your web host places a sneaky server script to make robots.txt
invisible only to you.

Paul

[Back to original message]