Posted by Dotan Cohen on 06/24/05 13:34

On 6/24/05, Richard Lynch <ceo@l-i-e.com> wrote:
> On Thu, June 23, 2005 3:37 am, JamesBenson said:
> > http://www.funender.com/phpBB2/about18577.html
>
> Call me crazy, but...
>
> A)
> Doesn't robots.txt have to be in public_html? How the hell can the robots
> read it if it's in the root folder, as they suggest in that forum?
>
> B)
> Are the Bad Bots really going to honor robots.txt in the first place? I
> mean, I know the spammers don't even need to un-obfuscate emails as simple
> as %40 and #&64; but are they really so dumb as to honor robots.txt while
> they write their crawlers?... That's crazy.
>
> Though I guess my theory on that holds true here as well: If you catch a
> million fish every time you cast your line, are you gonna try better bait?
> No. So they don't un-obfuscate even the simplest email mask, and I guess
> some of them play nice as far as robots.txt goes, even though they are out
> to spam you. Seems silly, but that could be Reality.
>
> C)
> Aren't some of those "bad" bots also going to rule out legitimate
> scripting? Am I mistaken that he pre-emptorally denies access to any
> (legit) Python script? Hello? I'm not a fan of Python, but that seems a
> bit much. :-)
> [Just kidding. I'm ambivalent towards Python, really.]
>
> --
> Like Music?
> http://l-i-e.com/artists.htm
>

robots.txt SHOULD go in the webroot, like this:
http://l-i-e.com/robots.txt

I don't know why FuNEnD3R said that it should be above the webroot.
That is a mistake.

Dotan
http://lyricslist.com/lyrics/artist_albums/64/the_beatles.php
Beatles Song Lyrics

[Back to original message]