Posted by Steve Pugh on 11/22/05 11:50

Scott wrote:
> > Luigi Donatello Asero wrote:
> > > "Scott" <golden@uslink.net> skrev i meddelandet
> > > news:437FD314.421C1B77@uslink.net...
> > > >
> > > > Is there a tag that I can put on a page that will prevent search
> > > > engines from indexing the page?
> > >
> > > As far as I know you could insert the adress of the page into a file
> > > called robots.txt and indicate which search engine you do not want to
> > > index it.
>
> OK, I figured out what to write in robots.txt. What I'm wondering is exactly
> where to place that file on the host server.

At the root of your site.

If a spider wants to visit http://www.example.com/foo/bar/page.html
then it will look for http://www.example.com/foo/bar/robots.txt,
http://www.example.com/foo/robots.txt and
http://www.example.com/robots.txt and apply all the rules it finds.
>From your point of view having a single robots.txt in your root folder
makes for easy maintenance.

Steve

[Back to original message]