|
|
Posted by softwarelabus on 08/08/06 15:09
Adrienne Boswell wrote:
> Gazing into my crystal ball I observed softwarelabus@yahoo.com writing in
> news:1155046561.133316.32210@n13g2000cwa.googlegroups.com:
>
> > Hi,
> >
> > I wanted to warn all website owners that some evil web hosts like
> > vistapages will periodically place a robots.txt file on your site that
> > disallows all search engines. It happened to me.
> >
> > Over the last several months I've noticed my web traffic dropped to
> > nearly zero. A few days ago I noticed a new file, robots.txt. As most
> > of you know, if your site has a robots.txt in your websites home
> > directory then all search engines will look at it for possible
> > instructions. The robots.txt file tells search engines what to do or
> > what not to do. In my case, it had simple instructions to disallow all
> > user-agents; i.e., telling all sites they cannot come here.
> >
> > How to check:
> > If you web site is called www.mywebsite.com then you want to check the
> > following web page:
> > www.mywebsite.com/robots.txt
> >
> > You should also look for this file when you ftp to your site in case
> > your web host places a sneaky server script to make robots.txt
> > invisible only to you.
> >
> > Paul
> >
> >
>
> I really doubt that this was done with evil intent, probably a misguided
> system administrator who got tired of seeing 404 errors, but was too lazy
> to look up the robots.txt protocol and get it right.
>
> It's perfectly okay to have blank file, that way the bots are happy, and
> the system admins are happy, too.
I'd say that was a misguided SA alright, lol. IMHO that's when it's
time to call it quits, start looking for another web host because
that's like dropping a nuke on a site.
Paul
Navigation:
[Reply to this message]
|