discourse-legacysite-perl/site/slowtwitch.com/www/robots.txt
2024-06-17 22:30:36 +10:00

11 lines
197 B
Plaintext

User-Agent: *
Disallow: /cgi-bin
Disallow: /log
Disallow: /secure
Disallow: /statistics
Disallow: /flash/
Disallow: /forum/Slowtwitch_Forums_C1/Lavender_Room_F4
User-Agent: bingbot
Crawl-delay: 1