Surely listing sensitive files is asking for trouble?
Some people are concerned that listing pages or directories in the /robots.txt file may invite unintended access. There are two ansers to this. The first answer is a workaround: You could put all the files you don’t want robots to visit in a separate sub directory, make that directory un-listable on the web (by configuring your server), then place your files in there, and list only the directory name in the /robots.txt. Now an ill-willed robot can’t traverse that directory unless you or someone else puts a direct link on the web to one of your files, and then it’s not /robots.txt fault. For example, rather than: User-Agent: * Disallow: /foo.html Disallow: /bar.html do: User-Agent: * Disallow: /norobots/ and make a “norobots” directory, put foo.html and bar.html into it, and configure your server to not generate a directory listing for that directory. Now all an attacker would learn is that you have a “norobots” directory, but he won’t be able to list the files in there; he’d need to gu
