Best way for creating customer-facing views that are not accessible by crawlers?

[Stephanie Fischer]

We have licensed Jira Product Discovery for building roadmaps and created a roadmap view curated for key customers.

I wanted to turn on the public link, but our InfoSec declined because then it would be accessible to crawlers.

I found this option, but we don't want to ask external customers to make Atlassian accounts even if it's free. https://www.atlassian.com/blog/jira-product-discovery/publish-views-external

And our sales rep recommended IP Allowlisting, but that is probably more challenging to organize than asking customers to make accounts.

So, my question is whether it's possible to have a public page without allowing it to be crawled by bots?

[Jens Schumacher - Released_so · Accepted solution]

Hi Stephanie, 

Short answer: not really, at least not in a reliable way. 

A “public link” by definition means anyone on the internet can access it. You can add noindex headers or robots.txt rules, but those only work for well-behaved search engines. They don’t stop generic crawlers, link scanners, security bots, or anyone who receives the URL and shares it. From an InfoSec perspective, that’s still considered publicly accessible.

You could consider a third party app like Released (I'm one of the founders) which allows you to restrict access to a portal to specific customers, or customer domains, and users can login using their Atlassian ID, Google or via magic link.

You could even embed the portal into your own app or website and authenticate the users yourself, not requiring a login at all. 

To give you an idea what it looks like, here a link to our public roadmap: https://hub.released.so/released/roadmap/ad0e75c1

Let me know if you have any question. 

[Stephanie Fischer]

Thank you for sharing!