I heard that I can edit my .htaccess file to stop malicious bots/spiders from accessing my site. Does this work?
Many advanced webmasters write complex rules into their .htaccess file in an attempt to stop malicious spiders from accessing their web site. For example, one such rule might be to deny access to a client if its name was “EmailHarvester”. Once this rule is in place, then any client that identifies itself as EmailHarvester would then be denied access to the domain. However, authors of malicious software are fully aware of the methods being used to exclude them. They are not concerned about writing standards compliant software, so specifically design their software to avoid such rules. In the above example, it would be easy for the author of ‘EmailHarvester’ to bypass this access rule, simply by providing a bogus name (or no name at all) for his spider. He could also provide the ability for users to specify a name themselves, which would make this impossible to detect based on the above rule. An example of such behavior can probably be found on your own domain. If you examine your access
