404 not found after power outage

[Brian Willmert]

After a power outage, our confluence server comes up 404 not found.

During boot when all of the services are starting in Ubuntu, I see:

*Starting PostgreSQL 9.3 database server           [fail]

Error messages from the catalina logs in /opt/atlassian/confluence/logs show:

WARNING [Main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule] {server} setting property 'debug' to '0' did not find a matching property

SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal one or more listeners failed to start.  Full details will be found in the appropriate container log file

SEVERE [standalone-startStop-1] org.apache.catalina.startup.HostConfig.start Unable to create directory for deployment: opt/atlassian/confluence/conf/Standalone/localhost

Now I am a windows admin and I dabble in Linux.  It seems to me that Postgre got corrupt during the power outage, which makes Confluence not able to come up.  Where should I be looking to fix this without loosing any data?

Also, I have another VM running an almost identical confluence setup (just with different data).  If it helps to copy anything from that VM to this one to get it working, that is an option.

Thanks for the help,

Brian

[Andy Heinzer · Accepted solution]

Hi Brian,

Sorry to hear that you seem to having some database problems here.  There are two ways I can see that might allow you to recover this data.   One is if you had a postgres backup of that database, you could just restore this database backup to another 9.3 postgres instance and then simply edit your $CONFHOME/confluence.cfg.xml file in order to point your Confluence instance to this new database location.  Then you can just start up confluence and it will work with this new database location.

If you don't have a database backup made with postgres database tools, then the other option would be to try to restore this data from an XML backup that might have been created automatically on the system.   The complete steps to follow in this case are explained in Migrating to another database.   By default Confluence is storing these (if enabled) in the $CONFHOME/backup/ folder.   This method is not always recommended for large environments, but it might be able recover your data.  You would still need to first create a new database as in Database setup for PostgreSQL.

These steps don't really address the problems though with the existing SQL instance, instead these steps are really focused on just getting Confluence back up and working with your data.

[Brian Willmert]

@Andy Heinzer, Thank you for the suggestions, it really helped.  I'm writing out my entire process in case anyone else searches for a similar problem.

After much troubleshooting with postgres, I simply could not do anything with it.  I found other forums with suggestions of doing a pg_resetxlog.  I had to install another package (postgre-xc I think it was) in order for resetxlog to work. This did give results as I didn't see failures during startup anymore, and I was able to restart postgresql service without errors.  It was suggested that after running this resetxlog, I should do a dump and restore of the database to clear out any potential problems in the database.  However, this wouldn't work because I was then getting errors saying that the role postgres didn't exist.

Also, even though I couldn't dump/restore the database, I tried restarting confluence since at least the service was starting, but I still got page not found.  That got me thinking that since I never got the login screen at all, maybe it was a problem with tomcat instead.

I hadn't been taking manual backups, but the automatic backups DID exist in the backups folder you mentioned.  So I copied them off to another system, as well as the attachments folder (wasn't sure if attachments were included in the automatic backup). 

First I tried uninstalling and reinstalling confluence. The install seemed to go through without errors but still no login screen.  Though admittedly I didn't try also manually uninstalling tomcat.

I then tried an uninstall of Confluence, then a apt-get --purge remove postgresql, but the removal of postgresql errored out and I couldn't get it clean.

At this point, since I'm a windows guy, further troubleshooting would have taken me much longer than time I had to burn.  So I ended up booting up off the Ubuntu CD and just rebuilding the stupid thing from scratch.  I still had the original installer file so I was installing the same version.

Install went great, and during the install I picked "restore" instead of starting with a fresh database.  I restored the latest backup that I copied off of the system earlier and that process worked without any problems.  I then copied the attachments folder back as well (again because I wasn't sure if the backup included it).

I had the users check the contents and they said everything looks exactly the way it did before the outage.

I'm back up and running, and I'm going to adjust my backup policy for that lab going forward.

Thanks for the help,

Brian

[Andy Heinzer]

Hi Brian,

Thanks for posting back with all these details.  I am glad to hear that you got it back up and working.   I would agree though that we don't typically want to rely on those XML backups.  For really large environments, there have been known issues with the integrity of those backups.  As such using native database tools, like pgAdmin, would be much preferred in terms of making sure you have your data backed up.

Regards,
Andy