How to make a robots.txt that disallows 'label' paths

Cassandra Targett September 24, 2014

I have a robots.txt on my Confluence site and would like to disallow crawlers from crawling the labels. I've got the following DISALLOW lines in my robots.txt but it doesn't appear to be working:

Disallow: /label/
Disallow: /labels/
Disallow: /*label*

I borrowed the last line from https://github.com/childnode/atlassian-confluence-config/blob/master/robots.confluence.txt - I thought DISALLOW didn't support regex, but I threw it in there anyway.

Does anyone know a better path to disallow that would prevent the label pages from being crawled?

(By the way, I know it's not working because I have a crawler that is crawling the site. It is possible there is a bug in the crawler, but is respecting all my other rules so it seems unlikely that it would just arbitrarily ignore a valid rule.)

Thanks in advance for any help.

1 answer

0 votes
Tiago Comasseto
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
September 24, 2014

The robots.txt looks correct to me. I don't have ways to test it at this moment, but just wondering, have you restarted the application so that it can pick up the new settings? Cheers

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events