I'm not exactly sure what you're looking for here. I see 2 things. First, security best practices and then a question about stopping robots.
Security is a big thing. More than I can write here. There have been many books on the topic. I would simply start with drupals best practices. There is a security team and setup/release cycle/development is based around sound security practices.
For the robots, who are you trying to stop. Google, Yahoo, MS, and the other big players will respect the Disallow. If you want to stop them all together you can use rules in the .htaccess file to deny them access.
I am interested in knowing what best practices you follow to secure your drupal sites.
Specifically:
Which directories are protected; which are not?
The location (within the root directory or not) and what permission to use on an user upload directory?
Do you alter the default robot.txt file? I am aware that User-agent: * Disallow: / should keep crawlers off the site but they are not obligated to follow the robot.txt directives.
Thank you.