To participate in the bot project you modify your web server configuration file and alter the way your robots.txt is served. You do not need to make any changes to your robots.txt and if you wish to drop out of the project, it's quite simple - just remove the code!
The web server is configured so that robots.txt is an executable so when it is requested, a bot will see what was there before but also executes code to update the bot project database with details of the request.
Why would I want to do that?
You'll get to access all bot related statistics to your site(s) and you'll be part of a project where the webmaster community can work together to gain useful information about bots.
You'll be able to login and see what bots have visited your site, how regularly, where they're from, who they are and so on.
Unless you specify otherwise, your site statistics will only be viewable by you. The summarised data comprised of all sites will be available to all and sundry.
You can even have an email each time a bot is on your site so you can monitor your SEO activities (EG see how long a Blog ping takes to call a bot).
Ok, sounds interesting, I'll give it a go - what's next?
You have to sign up to be able to participate as we need a unique user id so you can view your site statistics but so you have an idea of what's required here's a little overview:
PHP / Apache
You edit your Apache config using a rewrite command and create a new file called robots.php. Your robots.txt remains untouched.
You can read specific information in the forums.
ASP.NET / IIS
Very similar in principle to the PHP method but with a few more lines of code.
You can read specific information in the forums.
So you want me to put your code on my web server...?
Well, yeah. You can see all the code - it's not encrypted or anything so feel free to take a good look and ensure you're happy with it.
How do I know your code isn't doing other stuff?
If you're not familar with the code then post it on some forums or ask about The Bot Project.
If you're still not sure then don't do it. We'd love to have you involved but you have to be comfortable with the whole thing.
It's free right? So why are you doing it?
Because there are hundreds or thousands of bots out there going through our sites when it suits them doing all sorts of stuff. We think it would be very useful and interesting to know more, on a larger scale, about these bots.
We're doing it because we already had the code for our own sites and thought it'd be cool to do something like this. Our intention is to get others involved to steer the project, contribute code, assist in the forums and so on. We're certainly not interested in charging for it.
You can read more over at the FAQ.
Donations are appreciated to pay for bandwidth and disk space.
