View Issue Details
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0000881||Main CAcert Website||website content||public||2010-10-15 08:47||2011-06-22 00:09|
|Priority||normal||Severity||minor||Reproducibility||have not tried|
|Summary||0000881: crawlers have permission to crawl through the cacert website|
|Description||Andreas noted that there is no robots.txt to disallow crawling.|
robots.txt created in /www nearby index.php.
|Tags||No tags attached.|
||added file, exported to cacert1|
||patch: git diff c6ed18141161adf6b17ea07d9c6a8eeb37f6eaa1..00e56b6ea045915eb2869688d54fe7d05661faf2|
The above robots.txt may not be appropriate for production it was a quick hack to preserve the ressources of the test-system which has only a low bandwidth connection at the moment.
If indexing of the main site is on purpose we should supply an explicit robots.txt with allowance so it's documented.
||We should not stop crawling for the production system, we should limit it properly. The above mentionened robots.txt is not adequat.|
|2010-10-15 08:47||edgarwahn||New Issue|
|2010-10-15 08:56||edgarwahn||Note Added: 0001751|
|2010-10-15 08:59||edgarwahn||Note Added: 0001753|
|2010-10-15 13:14||Andreas Baess||Note Added: 0001754|
|2010-10-17 10:42||Sourcerer||Note Added: 0001757|
|2011-06-22 00:09||NEOatNHNG||Source_changeset_attached||=> cacert-devel master 6bc7253c|