|
|
Posted by Kim Andrι Akerψ on 07/24/06 21:30
CRON wrote:
> Hi,
> How do i disallow all search engines access to:
>
> http://www.scouttalk.ie/user.php?userID=1
>
>
> where 1 in the above line can be any number?
Closest thing would be:
User-agent: *
Disallow: /user.php
You can disallow single files or entire directories, but not specific
query strings.
http://www.robotstxt.org/wc/exclusion-admin.html
Keep in mind that the robots.txt file is usually followed by "good"
spiders, such as MSN, Google and Overture. It doesn't specifically
disallow access for search engines, it only serves as a suggestion to
the spiders what they should ignore on their journey; more of a "please
don't include these files/directories in your index".
Rogue spiders/bots might ignore your robots.txt file altogether or even
specifically go to the "disallowed" locations, just to grab exploitable
content.
--
Kim AndrΓ© AkerΓΈ
- kimandre@NOSPAMbetadome.com
(remove NOSPAM to contact me directly)
Navigation:
[Reply to this message]
|