Are Public Spaces indexed by search engines?

Question

Our intent is to use Confluence as a Knowledge Base for customers. After we train and onboard the new customer, we would like to send them one link to our knowledge base. If they have a question later on they can visit and (hopefully) get an answer.

On the paid tier, we see the space permissions can be set to 'Public' which enables anonymous users to freely access the knowledge base.

Questions:

* Are public spaces indexable by search engines? If I Google-search a question about my product will our knowledge base appear in search results?

* Is the public space truly "anyone with the link can view"? Does the customer have to know the link in order to access our knowledge base?

Thank you!

Shannon S · Answer

Hi @Metin _savignano software solutions_,

Thanks for your reply. Since my response is from 2021, it's outdated, and I understand there have been some changes since then.

Can you please raise a support ticket with the cloud, as we can better assist you there?

https://support.atlassian.com/contact/

Thank you, and take care!

Shannon S | Atlassian Community support

Metin _savignano software solutions_ · Answer

@Shannon S That's actually not true. While our site does allow public anonymous access, for some reason, it returns a robots.txt that forbids search engines to index our pages:

# This robots.txt content is autogenerated
User-agent: *
Allow: /wiki/external/
Disallow: /wiki/
Disallow: /sr/
Disallow: /si/
Disallow: /charts
Disallow: /secure/ConfigureReport.jspa
Disallow: /secure/ConfigureReport!default.jspa
Disallow: /secure/admin/
Disallow: /l/cx/
Disallow: /login
Disallow: /logout

Note the Disallow for /wiki/, which effectively forbids access to all pages.

It's not possible for us to have our pages indexed by Google, which is a big problem, and I do not understand why Atlassian has done this.

Shannon S · Answer

Hello Bert,

Welcome to Atlassian Community. It's nice to have you join us!

When content in Confluence is public, anyone on the internet can see your content,  including search engine crawlers. If you visit your site in an Incognito window in your browser, the content you see is what a search engine can index.

If you want to see how that looks, try right now by searching for site:*atlassian.net in a Google search. You can see that anyone could obtain a link to your site this way.

I found two feature requests related to this issue that are long-standing but still active. You'll want to keep an eye on these, so click vote to be added to the list.

* CLOUD-5915 Allow users to modify their robots.txt (this would enable you to block search engine bots)
* CLOUD-5685 Stop search engines from indexing Atlassian OnDemand instances by default

If you don't intend to provide your customers with Confluence licenses, the alternative option would be to pair Confluence with Jira Service Management (JSM). JSM provides free customer accounts, so anyone listed as a customer can see the Confluence knowledge base, and you don't risk search engines indexing your site. See Set up a knowledge base so customers can serve themselves for more information.

I hope that helps, but please let me know if you have any questions about that.

Take care,

Shannon