Access issue when adding custom website connector for Rovo

[Philip]

I'm trying to build a company-wide internal support agent that has access to all of our company resources. Does anyone else find Atlassian to be very restrictive when trying to add a Custom Website connector?

For context, we also use Intercom Fin AI for customer-facing support, and with Fin AI, I've had no problem adding a variety of custom websites to our Intercom AI index. With Atlassian though, all of the custom websites that I need to add can't be added due to this error:
"Can't access your site

We're having trouble connecting to your site. Check the details, or see our docs for other suggestions."

Specifically I'm having the issue with all of these public resources:

• support.wunderite.com - Intercom knowledgebase - this one get's successfully added, but it's clear that it's not being indexed. If I try to add the Intercom domain version of our knowledgebase, it's blocked. This is a huge blocker for us because we have no way to update Intercom's robots.txt and our knowledgebase is one of our most useful resources. A direct integration with Intercom would be great and maybe resolve this issue.
• docs.wunderite.com - internal API docs - we should be able to update our robots.txt to explicitly allow atlassian-bot
• releases.wunderite.com - release notes via Released.so. I met with the CEO yesterday and he said he'll look into updating their permissions. Released.so does have an integration with Confluence, so as a workaround, I could at least sync our release notes to Confluence.

As far as I can tell, none of these sites are explicitly blocking AI agents, but they also don't explicitly allow atlassian-bot. It seems like Atlassian is far more restrictive than other agents I've used, so I'm not able to add critical knowledge sources. Has anyone else encountered this or know how to address it?

[Mohanraj Thangamuthu · Accepted solution]

Hello, Good day. I would like to know whether you have already reviewed the following document regarding connecting custom sites. https://support.atlassian.com/organization-administration/docs/connect-custom-website-to-rovo/

[Dr Valeri Colon _Connect Centric_]

Hi @Philip Rovo’s website connector is much stricter than other AI indexers. It requires the site to be fully reachable by Atlassian’s crawler, allowlisting atlassian-bot in robots.txt or headers, and permit cross-domain fetching. Many public sites pass human access but fail crawler checks or block required resources (scripts, redirects, auth). If a site doesn’t explicitly allow the crawler, Rovo usually won’t index it. Updating robots.txt to allow User-agent: atlassian-bot is the most reliable fix; otherwise open a Support ticket so Atlassian can confirm the exact crawl failure.

[Philip]

Thank you for confirming re: Atlassian being strict. I did review the documentation before posting this. As mentioned, most of the links that I shared are problematic because they're third party tools, so I have no control over robots.txt. Intercom is the big one - our knowledgebase. A direct integration with Intercom would be great so that I could resolve the issue that way.

[Mohanraj Thangamuthu]

Hello Philip, please create a ticket with our support team at https://support.atlassian.com/contact/. Let me know if you are facing any issues with creating the ticket. Thanks!

[Philip]

I'm on the free Rovo plan, so unfortunately I'm not allowed to create a support ticket for "Technical issues and bugs" for Rovo. Is there another category that I should be using?

[Mohanraj Thangamuthu]

Thanks. I have created a support request on your behalf with concern team, our team will reach out to you over email. Thanks!

[Philip]

I confirmed that adding atlassian-bot to robots.txt on docs.wunderite.com allowed Atlassian to crawl the site. But again, the bigger issue is that I'm blocked on 3rd-party sites. It sounds like Released.so will make an exception for Atlassian. I also opened a ticket with Intercom, but it sounds like I'll be blocked on that unless there's a direct integration or if Intercom makes any changes on their end (not holding my breath for that one).