Posted by Paul on 09/01/06 09:15

Can I get a web robot to only see one or 2 files as they link to areas
of my site that I do want indexed. LIKE

User-agent: *
Allow: /history.php
Disallow: /

This would help me enormously.



nice.guy.nige wrote:
> While the city slept, Nikita the Spider (NikitaTheSpider@gmail.com)
> feverishly typed...
>
> [...]
> > But I'm wondering what you mean by saying your Web
> > site is "under attack". Yahoo! Slurp and Googlebot try to be
> > reasonably polite when spidering a site.
>
> Indeed... Last time around, Googlebot even made me a cup of tea! ;-)
>
> Cheers,
> Nige
>
> --
> Nigel Moss http://www.nigenet.org.uk
> Mail address will bounce. nige@DOG.nigenet.org.uk | Take the DOG. out!
> "Your mother ate my dog!", "NTt all of him!"

[Back to original message]