You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
12 lines
231 B
12 lines
231 B
7 months ago
|
# exclude help system from robots
|
||
|
|
User-agent: *
|
||
|
|
Disallow: /manual/
|
||
|
|
Disallow: /doc/
|
||
|
|
Disallow: /gif/
|
||
|
|
# but allow htdig to index our doc-tree
|
||
|
|
User-agent: susedig
|
||
|
|
Disallow:
|
||
|
|
# disallow stress test
|
||
|
|
user-agent: stress-agent
|
||
|
|
Disallow: /
|