Confluence web server not responding but no errors in confluence.log

[Rich Webb]

Yesterday at some point we tried to save a change to a document in Confluence and it wouldn't save and gave an error (sorry don't know what it was) and then the site disappeared altogether. 

I tried restarting the server (which also houses Jira and Bitbucket which both came back up. Confluence did not.

I checked the atlassian-confluence.log file and there are no errors for the time period in which I did the restart.  There are a few other errors all having this content:

Servlet.service() for servlet [noop] in context with path [] threw exception

These are from several earlier dates but nothing in the last two days.

I'm at a loss because I have no idea where to begin troubleshooting this without any error to go on in the logs.

EDIT: Additional information - the system is running on CentOS Linux and is self-hosted. Thought that would be relevant.

EDIT:  I've tried a couple of things to get the system running again and it appears that there was a security breach.  I found processes in /tmp with random character names running as the confluence user.  Perhaps this is normal but not sure - doesn't seem right.  So I started with a complete backup of the VM.  I stopped and deleted the processes.  I renamed my data directory and did a reinstall of confluence and the new version appeared to run fine - saw the startup wizard on the site.  I shut down confluence and renamed the new data directory and replaced with my data directory - set the appropriate permissions.  Attempted to start confluence and the Java process started and then died.  Seems something in my data directory is causing the process to crash out.  I don't know where to look for logs.

I've also tried resetting the install so it's fresh install of my version with my data in place and attempted an upgrade.  Upgrade completes successfully and thinks it has started the process but the Java process is not running.  Guessing same thing happened - it started and then stopped.

At this point I'm hoping I didn't lose my data. I'm sure the database must be intact but also not sure how to check that all out either.  Really need someone that knows Confluence well to help with this.

Thanks

Rich

[Daniel Eads]

Hi Rich,

A couple questions that will help us check and see which direction to start looking:

1. What version of Confluence do you have?
2. Is the CPU on the box abnormally high right now?

Thanks,
Daniel | Atlassian Support

[Rich Webb]

I started with 6.12.2 and I was just editing my original comment when you posted so you may want to go back and read what I've attempted.  It is now on 6.15.2 but I have a backup so I can reset back to prior to the upgrade if necessary.

I wouldn't say cpu is abnormally high but I also have Bitbucket and Jira running on this machine too so it's got it's work cut out for it.  It is assigned 16GB of ram.

Thanks Daniel - I hope we can get this back.  Lots of important data in there.  

Rich

[Rich Webb]

If I restore a good backup from February, can I just do a mysqldump on the database on the damaged server to get the database and restore to the restored backup? 

I know there is also other data like attachments stored in /var/atlassian/application-data/confluence

What else may I need to get off the damaged server?  I don't know if I can copy the entire data directory or not.  I will make a copy of it just in case though.

Rich

[Daniel Eads]

I've tried a couple of things to get the system running again and it appears that there was a security breach.  I found processes in /tmp with random character names running as the confluence user.

Yes, this was my suspicion but wanted to confirm you were on a vulnerable version before throwing a bunch of information at you. What you've described is an exploit / malware attack.

From what you've said, I would tend to agree that your database sounds intact. I wouldn't recommend going to a February backup just yet as that's pretty far back, and you'll need to upgrade Confluence again to mitigate the security vulnerability.

I'm not sure if you mean you've installed Confluence on a totally new server/VM or simply that you've done a fresh install on the same host but copied over your data directory. Could you clarify this point?

In another recent answer I went over in-depth what might be done on a currently-infected server (including upgrading Confluence) - having a quick read through that would probably be helpful as it does describe how to use malware cleanup tools.

From my current understanding of your situation (not sure if it's a totally new VM or the same one with a new Confluence install / copy of the data directory, Confluence won't start, database seems ok from your description), I'd recommend this course of action:

1. Check top/crontab for any funny looking processes (as described in the linked article) on the old/existing server
2. Run a malware scan / cleanup tool on the old/existing server
3. Try starting Confluence 6.15.2 and note any errors in the application log here

Reading through your post a third/fourth time to make sure I didn't miss any details, based on what you've described it does sound like there might be a malicious process still running. Once that's cleared out we can work on getting Confluence up!

As for your question about "what do I need to get off the damaged server", the data directory can be copied completely. Your old install directory may or may not have customizations - Confluence would tell you this from the "modified files" section in System Information if it was running. Since it's not running, if you used the installer to upgrade, it also writes a .txt file at the parent directory of your install telling you what files were modified in the install dir.