Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

Next challenges

Recent achievements

  • Global
  • Personal


  • Give kudos
  • Received
  • Given


  • Global

Trophy case

Kudos (beta program)

Kudos logo

You've been invited into the Kudos (beta program) private group. Chat with others in the program, or give feedback to Atlassian.

View group

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Website won’t crawl in SEMRush & other SEO tools

We keep getting a crawl error when trying to use SEMRush and other audit tools to check our site for issues and errors. I’ve checked our codes and our robots.txt and nothing is being blocked and I’ve allowed the SEMRush bots crawl access. The site will crawl in some online tools, but with others we get errors like “URL is invalid” and in SEMRush we continue to get a crawl error due to a “server error” however we can’t find anything wrong on our end. Any insight as to what the issue may be?

2 answers

0 votes
Alyss Atlassian Team Sep 18, 2018

@Ally Adamson Diagnosing the issue may be dependent on figuring out a more specific server error that SEMRush is getting. 

You could start with Google's URL inspection tool to identify obvious indexing issues.

If SEMRush can't provide a specific server error/status code (401, 500, etc), then you could try to recreate the error. The most straightforward way would be to make a GET HTTP request to the URL and set the user-agent request header for a web crawler.

If you aren't comfortable with cURL requests or terminals, I'd recommend Postman as a more user-friendly interface. 

For me personally is a waste time trying to do this by yourself. I am at the IT faculty, and a friend of mine asked to do some seo for his fresh website. At first, it semmed easy, just posting some comments on forums with links in it. I've tried so many ways, apparently I needed more tools than I thought. The SEMRush wasn't working, the robots.txt codes as well. At that moment I was ready to give up, when I accidentally found the malaysia seo service which provided a highly effective search engine optimization for a low price actually, for a quick start up in online business this is the best option I found on internet

Suggest an answer

Log in or Sign up to answer
Community showcase
Posted in Bitbucket

Contest: Share your custom Bitbucket Pipe and win

Announced in this blog, this holiday season we’re celebrating all things CI/CD and between now and the end of 2019 we’ll be showcasing content, use cases, feature announcements and more. One featur...

2,132 views 11 6
Join discussion

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you