|
|
Posted by "Richard Lynch" on 10/20/69 11:19
On Thu, June 23, 2005 3:37 am, JamesBenson said:
> http://www.funender.com/phpBB2/about18577.html
Call me crazy, but...
A)
Doesn't robots.txt have to be in public_html? How the hell can the robots
read it if it's in the root folder, as they suggest in that forum?
B)
Are the Bad Bots really going to honor robots.txt in the first place? I
mean, I know the spammers don't even need to un-obfuscate emails as simple
as %40 and #&64; but are they really so dumb as to honor robots.txt while
they write their crawlers?... That's crazy.
Though I guess my theory on that holds true here as well: If you catch a
million fish every time you cast your line, are you gonna try better bait?
No. So they don't un-obfuscate even the simplest email mask, and I guess
some of them play nice as far as robots.txt goes, even though they are out
to spam you. Seems silly, but that could be Reality.
C)
Aren't some of those "bad" bots also going to rule out legitimate
scripting? Am I mistaken that he pre-emptorally denies access to any
(legit) Python script? Hello? I'm not a fan of Python, but that seems a
bit much. :-)
[Just kidding. I'm ambivalent towards Python, really.]
--
Like Music?
http://l-i-e.com/artists.htm
Navigation:
[Reply to this message]
|