What causes a '503' Error in Conflunce?

[Ashley Terry]

Running Ubuntu 16.04.6 with Confluence and Crowd installed as a service.

When looking through logs and service status I constantly get this 'PID File found but PID doesn't match any running processes. Stopping Tomcat Aborted' If I go in and kill the service manually and remove the PID from the catalina.pid file, confluence will start but crash about 15 min later with the same issues. I can post whatever I need to get this resolved. 

[Alex Medved _ConfiForms_]

Do you see the 100% CPU utilization as well?

Just an idea, may be you have got affected by this issue https://community.atlassian.com/t5/Confluence-discussions/khugepageds-eating-all-of-the-CPU/td-p/1055337

We have seen the same on one of our test servers, confluence was crashing, but the apache front server stayed alive and this has resulted in 503 (thrown from apache proxy).

But the original cause was the trojan mentioned in the linked question

[Ashley Terry]

Now i'm wondering how I just never checked there...

It doesn't seem to be the same virus though. I'm seeing 100% utilization, but process names are all randomized characters.

I did see some tmp.$gibberish files in the crontabs, I removed them. Anything you can think of about this particular malware/virus? Also, I'm trying to update confluence as I am on 6.8...I feel stupid again, but i'm not seeing where to update it?

[Alex Medved _ConfiForms_]

Very good article with links could be found here https://community.atlassian.com/t5/Trust-Security/Confluence-exploit-recovery-tips/gpm-p/1061618

[Brant Schroeder]

Looks as though running as root is causing your errors above, you will need to run Confluence as a non-root user. There are details when setting up Confluence which allows you to set the Confluence user and password. Here are further details in regards to this:

Running Confluence as a service means that Confluence will automatically start up when Linux is started.

If you choose to run Confluence as a service:

• You must use sudo to run the installer to be able to install Confluence as a service.
• The installer will create a dedicated user account, confluence, that will run the service.

If you choose not to run Confluence as a service:

• You will start and stop Confluence by running the start-confluence.sh file in your Confluence installation directory.
• Confluence will be run as the user account that was used to install Confluence, or you can choose to run as a dedicated user.
• Confluence will need to be restarted manually if your server is restarted.

Reference documentation: Installing Confluence on Linux

If you do not know the password to the Confluence user you will need to reset it. Once reset you may use that password to run Confluence as a service.

[Ashley Terry]

Well, I had the issue go away for a while, but it is now happening daily. I can reboot the server and the service starts fine and runs. After an unknown period of time the confluence instance just spits out the 503 error. If I reboot the server we go through that cycle again. The services start as the confluence user. I really need help here, we use this for our office documentation and it continuously goes down.

Additionally, I have raised the JVM heap memory as that was a suggestion on a confluence KB article I read about it reaching the max amount of mem and crashing.