|
|
Posted by Kim Andrι Akerψ on 11/22/05 13:24
Steve Pugh wrote:
> Scott wrote:
> > > Luigi Donatello Asero wrote:
> > > > "Scott" <golden@uslink.net> skrev i meddelandet
> > > > news:437FD314.421C1B77@uslink.net...
> > > > >
> > > > > Is there a tag that I can put on a page that will prevent
> > > > > search engines from indexing the page?
> > > >
> > > > As far as I know you could insert the adress of the page into
> > > > a file called robots.txt and indicate which search engine you
> > > > do not want to index it.
> >
> > OK, I figured out what to write in robots.txt. What I'm wondering
> > is exactly where to place that file on the host server.
>
> At the root of your site.
>
> If a spider wants to visit http://www.example.com/foo/bar/page.html
> then it will look for http://www.example.com/foo/bar/robots.txt,
> http://www.example.com/foo/robots.txt and
> http://www.example.com/robots.txt and apply all the rules it finds.
> > From your point of view having a single robots.txt in your root
> > folder
> makes for easy maintenance.
Where did you get that idea?
http://www.robotstxt.org/wc/exclusion-admin.html
<quote>
Note that there can only be a single "/robots.txt" on a site.
Specifically, you should not put "robots.txt" files in user
directories, because a robot will never look at them. If you want your
users to be able to create their own "robots.txt", you will need to
merge them all into a single "/robots.txt".
</quote>
--
Kim AndrΓ© AkerΓΈ
- kimandre@NOSPAMbetadome.com
(remove NOSPAM to contact me directly)
Navigation:
[Reply to this message]
|