robots.txt:
Defines indexing protocol for visiting search engine robots/spiders. Edit file to define directories and/or files not to be indexed.
Code:
Save as "robots.txt" in the root directory:
Highlight
User-agent: * Disallow: / User-agent: Mozilla/3.01 (hotwired-test/0.1) Disallow: /cgi-bin Disallow: /admin Disallow: /cgi-bin/file Disallow: /images Disallow: /.realms Disallow: /logs Disallow: /ftp Disallow: /htsdata Disallow: /mailbots Disallow: /stats User-agent: Slurp Disallow: /cgi-bin Disallow: /admin Disallow: /cgi-bin/file Disallow: /images Disallow: /.realms Disallow: /logs Disallow: /ftp Disallow: /htsdata Disallow: /mailbots Disallow: /stats User-agent: Ultraseek Disallow: /cgi-bin Disallow: /admin Disallow: /cgi-bin/file Disallow: /images Disallow: /.realms Disallow: /logs Disallow: /ftp Disallow: /htsdata Disallow: /mailbots Disallow: /stats User-agent: smallbear Disallow: /cgi-bin Disallow: /admin Disallow: /cgi-bin/file Disallow: /images Disallow: /.realms Disallow: /logs Disallow: /ftp Disallow: /htsdata Disallow: /mailbots Disallow: /stats