Spiders & Robots

[Stu]

Does anyone know what the benefit of having spiders and bots is on the site. I know the major search engines have their own, which they use to check your site out before ranking. Is it better to have lots of them crawl you site on a regular basis and will this help either you overall page rank and/or organic search results?

We now have 15 that visit the site and the latest is one called Exabot.

[achris83]

Exabot is the crawler associated with www.exalead.com its a France based search engine, it might look a little different in your logs because of the way it browses your site. It views the full site, taking screenshots as it goes to display with the search results.

In general the number of bots browsing your site is nothing really to be concerned about so long as they are for the most part on behalf of search engines, and not malicious little buggas that are going to be signing up to any services you offer and/or spamming your site and any emails it finds on it.

If you want more control over where bots go and what they see on your site you can use a file called 'robots.txt' on your site and/or use no follow links, you can get more information on how to use the 'robots.txt' file at http://en.wikipedia.org/wiki/Robots.txt. Although most spiders and bots will respect your guidelines some of the more malicious bots will just ignore it, and there aint allot you can do to prevent them from getting through other than blocking there IP addresses from browsing your site.

Hope this helps...

[CJS]

Hi Stu

Bots are great, in that they help search engines index your pages which ultimately lead to them being published in the search results.

You can block bots using a special file called a robot.txt file but you would be mad to do that.

You can also selectively block bots, or even selelectively block directories that you dont want them to crawl.

Another alternative (well complimentary rather than alternative) method is to use XML sitemaps. The main one being google sitemap you can then
choose which files you want to submit for indexing. The best way to find out more is to sign to up google webmaster tools at https://www.google.com/webmasters/

Let me know if you need any other info.

Mat

[CookBoy]

Good post Mat, it's nice to see another new member with some usefull knowledge.

[JinaAlbert]

Spiders and robots are necessary for search engines.The search engine is too complex and fast which is beyond the reach of humans.
To speed up the crawling process bots are needed.They crawls up the site much quickly and index on google.They are automatic web crawlers.