PostText/public/robots.txt
2024-08-06 14:08:05 -04:00

30 lines
507 B
Plaintext

User-agent: search.marginalia.nu
Disallow: /cursors/
Disallow: /images/
Disallow: /fonts/
Disallow: /moderator/
Disallow: /login
Disallow: /logout
Disallow: /human/
User-agent: DuckDuckBot
Disallow: /cursors/
Disallow: /images/
Disallow: /fonts/
Disallow: /moderator/
Disallow: /login
Disallow: /logout
Disallow: /human/
User-agent: Yahoo! Slurp
Disallow: /cursors/
Disallow: /images/
Disallow: /fonts/
Disallow: /moderator/
Disallow: /login
Disallow: /logout
Disallow: /human/
User-agent: *
Disallow: /