Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

Next challenges

Recent achievements

  • Global
  • Personal

Recognition

  • Give kudos
  • Received
  • Given

Leaderboard

  • Global

Trophy case

Kudos (beta program)

Kudos logo

You've been invited into the Kudos (beta program) private group. Chat with others in the program, or give feedback to Atlassian.

View group

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

This morning, due to no action on our part, our confluence installation on aws shows a 502: Bad Gateway error. Stopping and restarting the EC2 instances where confluence is installed solves the problem....temporarily. Shortly afterwards (in a couple of hours), the site goes down with the same error again.

 

Everything was working fine up until yesterday, we've made no changes to the AWS setup or installation files at all. How can we fix this?

4 answers

1 accepted

0 votes
Answer accepted

Formal answer: we were attacked from the vulnerability detailed here: https://community.atlassian.com/t5/Confluence-discussions/khugepageds-eating-all-of-the-CPU/td-p/1055337

 

Following the steps in that thread fixed it for us (kill hkugepads process, clear cron jobs).

1 vote
Diego Atlassian Team Apr 17, 2019

Hello there!

Could you clarify some points for us?

  1. Which version of Confluence are you running?
  2. Is CPU usage unusually high in your server?

Looking forward to your reply!

Hi, thanks for offering to help. Answers below.

 

1. How do I find my confluence version? I cannot access confluence from the url, so I can't just go to the admin panel and find out (bad gateway error). But I can access it via the command line. How do I find out/what command/which file?

 

2. CPU usage according to AWS is at 100%. That's pretty high, right? Unfortunately I don't have statistics of CPU usage before this morning, because when I stopped and restarted my AWS instances to see if I could solve the problem, they were somehow automagically terminated and re spun/re-built. So the ones I currently have running are somehow "new" ones. Stopping/starting the AWS instances solved the bad gateway problem for an hour or two, then it came back.

Our version is 6.12.2 .

 

I was able to get confluence back up by following instructions in this thread:

https://community.atlassian.com/t5/Confluence-discussions/khugepageds-eating-all-of-the-CPU/td-p/1055337

Diego Atlassian Team Apr 18, 2019

Hey there!

Great to know that you solved the issue! I was a little late to reply here but thankfully everything is ok.

Thanks for sharing your Connie version. With this information I can recommend you to upgrade your instance to at least 6.12.4. This is due to these two CVEs:

Confluence Security Advisory - 2019-03-20 | March 2019 Confluence Server Advisory - WebDAV and Widget Connector vulnerabilities

 

Confluence Security Advisory - 2019-04-17 | Confluence - Path traversal vulnerability - CVE-2019-3398

 

Also, try running another malware check on your instance just to be sure that everything is in order.

Further! Before the upgrade or any other change you may apply to your instance, backup these:

  • <confluence-home> folder
  • <confluence-install> folder
  • Database

More information:

Confluence Home and other important directories

 

Let us know your thoughts!

Hi @arcadiaengineering ,

A 502 error isn't going to tell you much with further investigation.

I'd start by following this guide:

https://blog.hubspot.com/marketing/502-bad-gateway

That will be a good starting point.

Regards,

Gary

I just ran top on the confluence server. It claims command khugepageds is taking 399% of the CPU. Is this helpful info? Not sure what to do next...

Looks like we may be suffering from this same attack: https://community.atlassian.com/t5/Confluence-discussions/khugepageds-eating-all-of-the-CPU/td-p/1055337

 

Following the steps in that thread, will post if still need more assistance. Thx.

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Confluence Cloud

Introducing Confluence Cloud for Microsoft Teams (Available for Preview!)

Hi Atlassian Community, Remote work has shifted how teams collaborate, and we’ve heard from many of you that Microsoft Teams has become mission critical to many of your workflows, from how you chat...

2,407 views 16 19
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you