The Importance of Spider Analysis in a Well-Rounded Search Engine Marketing Plan
| Category: Marketing Strategy | Date: 2003-04-15 |
The field of search engine optimization is a constantly evolving industry with important changes being made almost daily.
In the beginning, search engine optimizers focused on one thing only: rankings. Tracking of actual sales or conversion rates was almost unheard of.
Slowly, search engine marketers began to realize that all of the 1s in the world wont help if they dont convert to traffic, and all the traffic in the world wont help if it
doesnt convert to sales.
So, search engine experts began analyzing their log files and tracking sales. With todays technology, Web site owners can now tell which of their marketing campaigns are truly effective and which need to be replaced by a different campaign. In other words, through log analysis, you can analyze your human traffic and their visits to your Web site.
But what about your "spider" traffic?
One part of log analysis that has remained surprisingly lacking, both in terms of content and accuracy, is spider analysis.
Though traffic analysis programs may look at spider activity, the information often isnt detailed enough or presented in a format to do you much good. Also, spider and
robot analysis is acknowledged as being a main culprit for inaccurate log analysis measurements.
(http://www.internetnews.com/IAR/article/0,,_908361,00.html)
Therefore, the need for detailed spider analysis has begun entering the minds of search engine marketers.
So, what is "spider analysis"?
You know that when you submit a Web page to an engine for indexing, the engine sends a spider to your site to index the contents of the page. "Spider analysis" is simply analyzing the search engine spider visits to your site.
Through effective spider analysis, you can learn the following information about your site in a concise, easy-to-read format.
* Has your site been spidered?
* If so, by which engines?
* When did the spiders visit?
* Which directories and pages did they visit?
* Are certain pages getting respidered more often,
signaling their importance to the search engines?
* Are certain pages not getting spidered at all?
* Are the spiders indexing inappropriate content?
* Are the spiders getting everything they want and need, or
are they receiving error messages?
* Was your site spidered within the specified time agreed
upon in the pay inclusion programs youre participating
in?
* Is your site getting respidered on a regular basis, as
agreed upon in your participating pay inclusion programs?
Another important issue in spider analysis is robots.txt files.
Though the object of search engine marketing is to help spiders find all the pages on your Web site, there will be times when you want to keep spiders out of certain pages.
You can do this with a robots.txt file.
What is a robots.txt file?
A robots.txt file is a text file that is placed on your server that instructs the search engine spiders not to crawl or index certain sections or pages of your site.
But why would you want to keep the spiders out?
Lets say that youre creating a new page for your site, and youve placed the page online while you tweak and edit it. The page isnt ready for visitors, so you wouldnt want it indexed yet. Or, lets say youve placed some employee guidelines on your site. The guidelines are of interest only to your employees, and theres no reason for the
public to view them.
Using a robots.txt file, you can keep the spiders out of those pages.
Therefore, when working with spiders or robots, you want to be able to:
* Create a robots.txt file quickly and easily;
* Use a robots.txt file to present optimized pages to specific engines. For example, using a robots.txt file, you can focus English language robots onto the relevant
pages and direct robots from international search engines to the localized content areas of your site;
* Send e-mail harvesting programs away from your site to keep your e-mail spam down;
* View highlighted pages requested in error by the spiders;
and,
* Direct search engine spiders to relevant areas of your
Web site.
How do you create a robots.txt file?
Creating a robots.txt file manually is tricky at best. One little mistake will make the entire file invalid, and youre opening your Web site up to spiders operating on
their own free will. Plus, if youve created engine-specific pages with similar content, the spiders may discover those nearly duplicate pages, and you could find
yourself in trouble for spamming.
While there have been software programs on the market to create robots.txt files, thats been their sole function. Spider analysis hasnt been a part of their features, until
now.
Robot-Manager Professional Edition is a software program that concentrates totally on spider visits to your site.
Recently, I had the good fortune of reviewing a new software program that performs two major functions: spider analysis and robots.txt creation.
In a nutshell, Im amazed, and I have to predict that the software, Robot-Manager Professional Edition, will rapidly become a "must have" tool for search engine marketers.
To begin with, the program is easy to use. I dont know about you, but I dont have time to learn a complicated software program. Though the program is intuitive and
doesnt require the reading of a complicated users guide, youll still find detailed help topics that can answer any questions.
The program begins with Step 1, where you choose which spiders you want to work with. In Step 2, you download your file directory tree, which is where you begin to tell the program which files you want to keep spiders out of. The robots.txt file is automatically created in Step 3, and you can instantly upload the file to your directory. The program allows you to edit the file manually as well.
In Step 4, the program gets down to the real "meat" of the analysis: analyzing your log files for spider visits. When downloading and analyzing the log files from the domain I worked with, it automatically added 30 new spiders to the spider list.
On the Spider Visits page, I found it particularly helpful to view the visits by Web page. In this manner, I could quickly see which spiders have visited which pages of the site and when. Just think of the value of this information when working with clients. You can prove to them that spiders have visited their sites, even if the pages arent yet indexed. You can also view spider visits by date or by spider.
Also, international spiders are included in the program, which is ideal for those sites that are aiming for a corner of the international market.
Where can you go to see this program for yourself?
Visit Website Management Tools
(http://www.websitemanagementtools.com/) and look for Robot-ManagerT 3.0 Professional Edition. Though the site also offers a Standard Edition of the software, I strongly advise you to look at the Professional Edition instead. Why? Because the Standard Edition doesnt include the spider analysis portion of the software, which is a "must have," in my opinion.
You can download a trial version of the software at no cost, then test drive it before purchasing. With the trial version, only 5 spiders can be selected and 20 spider
visits picked up from the log file.
In Conclusion . . .
As the search engine marketing industry continues to move forward, three major categories of tools or services need to be considered:
* Web page checking, submission, and ranking tools;
* Web traffic analysis tools for analyzing human visitors
to your site; and,
* Spider analysis tools for analyzing spider activity.
If your current search engine marketing plan doesnt cover each of those crucial areas, you need to look into expanding your tool arsenal. The information you can gain by analyzing your human and spider traffic will prove invaluable to you as you work toward strengthening your online business.
About the author.
This article was written by Robin Nobles, a professional freelance writer and the Director of Training of the Academy of Web Specialists academywebspecialists.com/more_info, where she has trained several thousand people in her online courses in search engine marketing strategies. She also teaches onsite search engine marketing workshops http://www.searchengineworkshops.com)with John Alexander, and she has written three books that can be ordered through Amazon.
Copyright 2002 Robin Nobles. All rights reserved.
RobinN@acws.com
http://www.academywebspecialists.com/more_info
In the beginning, search engine optimizers focused on one thing only: rankings. Tracking of actual sales or conversion rates was almost unheard of.
Slowly, search engine marketers began to realize that all of the 1s in the world wont help if they dont convert to traffic, and all the traffic in the world wont help if it
doesnt convert to sales.
So, search engine experts began analyzing their log files and tracking sales. With todays technology, Web site owners can now tell which of their marketing campaigns are truly effective and which need to be replaced by a different campaign. In other words, through log analysis, you can analyze your human traffic and their visits to your Web site.
But what about your "spider" traffic?
One part of log analysis that has remained surprisingly lacking, both in terms of content and accuracy, is spider analysis.
Though traffic analysis programs may look at spider activity, the information often isnt detailed enough or presented in a format to do you much good. Also, spider and
robot analysis is acknowledged as being a main culprit for inaccurate log analysis measurements.
(http://www.internetnews.com/IAR/article/0,,_908361,00.html)
Therefore, the need for detailed spider analysis has begun entering the minds of search engine marketers.
So, what is "spider analysis"?
You know that when you submit a Web page to an engine for indexing, the engine sends a spider to your site to index the contents of the page. "Spider analysis" is simply analyzing the search engine spider visits to your site.
Through effective spider analysis, you can learn the following information about your site in a concise, easy-to-read format.
* Has your site been spidered?
* If so, by which engines?
* When did the spiders visit?
* Which directories and pages did they visit?
* Are certain pages getting respidered more often,
signaling their importance to the search engines?
* Are certain pages not getting spidered at all?
* Are the spiders indexing inappropriate content?
* Are the spiders getting everything they want and need, or
are they receiving error messages?
* Was your site spidered within the specified time agreed
upon in the pay inclusion programs youre participating
in?
* Is your site getting respidered on a regular basis, as
agreed upon in your participating pay inclusion programs?
Another important issue in spider analysis is robots.txt files.
Though the object of search engine marketing is to help spiders find all the pages on your Web site, there will be times when you want to keep spiders out of certain pages.
You can do this with a robots.txt file.
What is a robots.txt file?
A robots.txt file is a text file that is placed on your server that instructs the search engine spiders not to crawl or index certain sections or pages of your site.
But why would you want to keep the spiders out?
Lets say that youre creating a new page for your site, and youve placed the page online while you tweak and edit it. The page isnt ready for visitors, so you wouldnt want it indexed yet. Or, lets say youve placed some employee guidelines on your site. The guidelines are of interest only to your employees, and theres no reason for the
public to view them.
Using a robots.txt file, you can keep the spiders out of those pages.
Therefore, when working with spiders or robots, you want to be able to:
* Create a robots.txt file quickly and easily;
* Use a robots.txt file to present optimized pages to specific engines. For example, using a robots.txt file, you can focus English language robots onto the relevant
pages and direct robots from international search engines to the localized content areas of your site;
* Send e-mail harvesting programs away from your site to keep your e-mail spam down;
* View highlighted pages requested in error by the spiders;
and,
* Direct search engine spiders to relevant areas of your
Web site.
How do you create a robots.txt file?
Creating a robots.txt file manually is tricky at best. One little mistake will make the entire file invalid, and youre opening your Web site up to spiders operating on
their own free will. Plus, if youve created engine-specific pages with similar content, the spiders may discover those nearly duplicate pages, and you could find
yourself in trouble for spamming.
While there have been software programs on the market to create robots.txt files, thats been their sole function. Spider analysis hasnt been a part of their features, until
now.
Robot-Manager Professional Edition is a software program that concentrates totally on spider visits to your site.
Recently, I had the good fortune of reviewing a new software program that performs two major functions: spider analysis and robots.txt creation.
In a nutshell, Im amazed, and I have to predict that the software, Robot-Manager Professional Edition, will rapidly become a "must have" tool for search engine marketers.
To begin with, the program is easy to use. I dont know about you, but I dont have time to learn a complicated software program. Though the program is intuitive and
doesnt require the reading of a complicated users guide, youll still find detailed help topics that can answer any questions.
The program begins with Step 1, where you choose which spiders you want to work with. In Step 2, you download your file directory tree, which is where you begin to tell the program which files you want to keep spiders out of. The robots.txt file is automatically created in Step 3, and you can instantly upload the file to your directory. The program allows you to edit the file manually as well.
In Step 4, the program gets down to the real "meat" of the analysis: analyzing your log files for spider visits. When downloading and analyzing the log files from the domain I worked with, it automatically added 30 new spiders to the spider list.
On the Spider Visits page, I found it particularly helpful to view the visits by Web page. In this manner, I could quickly see which spiders have visited which pages of the site and when. Just think of the value of this information when working with clients. You can prove to them that spiders have visited their sites, even if the pages arent yet indexed. You can also view spider visits by date or by spider.
Also, international spiders are included in the program, which is ideal for those sites that are aiming for a corner of the international market.
Where can you go to see this program for yourself?
Visit Website Management Tools
(http://www.websitemanagementtools.com/) and look for Robot-ManagerT 3.0 Professional Edition. Though the site also offers a Standard Edition of the software, I strongly advise you to look at the Professional Edition instead. Why? Because the Standard Edition doesnt include the spider analysis portion of the software, which is a "must have," in my opinion.
You can download a trial version of the software at no cost, then test drive it before purchasing. With the trial version, only 5 spiders can be selected and 20 spider
visits picked up from the log file.
In Conclusion . . .
As the search engine marketing industry continues to move forward, three major categories of tools or services need to be considered:
* Web page checking, submission, and ranking tools;
* Web traffic analysis tools for analyzing human visitors
to your site; and,
* Spider analysis tools for analyzing spider activity.
If your current search engine marketing plan doesnt cover each of those crucial areas, you need to look into expanding your tool arsenal. The information you can gain by analyzing your human and spider traffic will prove invaluable to you as you work toward strengthening your online business.
About the author.
This article was written by Robin Nobles, a professional freelance writer and the Director of Training of the Academy of Web Specialists academywebspecialists.com/more_info, where she has trained several thousand people in her online courses in search engine marketing strategies. She also teaches onsite search engine marketing workshops http://www.searchengineworkshops.com)with John Alexander, and she has written three books that can be ordered through Amazon.
Copyright 2002 Robin Nobles. All rights reserved.
RobinN@acws.com
http://www.academywebspecialists.com/more_info
Copyright © 2005-2006 Powered by Custom PHP Programming