|
|
Posted by Runnin' on Empty on 01/06/24 11:56
"Nikita the Spider" <NikitaTheSpider@gmail.com> wrote in message
news:NikitaTheSpider-8FF326.21244123082006@news-rdr-01-ce0-1.southeast.rr.com...
> In article <1156325769.264557.183990@m73g2000cwd.googlegroups.com>,
> "Paul" <desotuatail@aol.com> wrote:
>
>
> Desmond,
> Ken has already given you good practical advice to which I have nothing
> to add. But I'm wondering what you mean by saying your Web site is
> "under attack". Yahoo! Slurp and Googlebot try to be reasonably polite
> when spidering a site.
>
>
They are unless you have a shopping cart on your site... if you're building
carts, you have to be aware that bots will follow any link.
This includes links that may add products to a temp cart, or delete them,
this can play havoc if you are using any kind of real time SKU tracking
code.
Google and Yahoo are pretty good at obeying robots.txt exclusions, certain
image indexer bots are not.
Runnin'
Navigation:
[Reply to this message]
|