I'll find the links to the relevant articles later but Google, Yahoo! and Live Search have all said that the robots.txt file should be no more than 1MB.
That means they're not expecting an instant reply for the file so as long as your server is not massively over loaded there will be no problem in terms of the request for robots.txt being slightly slower than it would if it was just a text file.
We've tried to make the bot tracking code as quick as possible.
If the bot project server is unavailable for any reason, this should not stop your robots.txt file being served.
If you're having problems with serving the modified version of of robots.txt then you've probably got bigger problems with your site / web server anway.... :)
There is nothing dodgy about what we're doing here. There's nothing that the search engines would object to - indeed, they may well get involved with the project.
Unless they actively search for The Bot Project code, they won't even know it's there.
