Confluence suddenly stopped responding - Local Database Issue?

[Mark T]

I am unable to access my confluence page on 8090.

At one occasion, I've got an error that says: 

Database: Database connection failed

I am using a Basic license (non-trial) using local H2 Database, as I am still evaluating the product, and I just realised I dont have any Support.

Any good starting point for this issue? I looked into the logs but I cant find anything obvious other than the WARNING ones. I just rebooted my server and this are the fresh logs:

2019-12-03 14:04:51 Commons Daemon procrun stderr initialized
03-Dec-2019 14:04:54.100 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server] failed to set property [debug] to [0]
03-Dec-2019 14:04:54.288 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'debug' to '0' did not find a matching property.
03-Dec-2019 14:04:54.303 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine] failed to set property [debug] to [0]
03-Dec-2019 14:04:54.319 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [debug] to [0]
03-Dec-2019 14:04:54.366 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context] failed to set property [debug] to [0]
03-Dec-2019 14:04:54.397 WARNING [main] org.apache.catalina.core.StandardContext.setPath A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [${confluence.context.path}/synchrony-proxy] does not meet these criteria and has been changed to [/${confluence.context.path}/synchrony-proxy]
03-Dec-2019 14:04:54.397 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context] failed to set property [debug] to [0]
03-Dec-2019 14:04:54.881 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
03-Dec-2019 14:04:55.163 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [1,393] milliseconds
03-Dec-2019 14:04:55.178 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Tomcat-Standalone]
03-Dec-2019 14:04:55.178 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.19]
03-Dec-2019 14:04:59.053 INFO [Catalina-utility-1] org.apache.catalina.core.ApplicationContext.log Spring WebApplicationInitializers detected on classpath: [com.atlassian.synchrony.proxy.SynchronyDispatcherServletInitializer@29e73a7f]
03-Dec-2019 14:04:59.272 INFO [Catalina-utility-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
03-Dec-2019 14:04:59.631 INFO [Catalina-utility-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'dispatcher'
03-Dec-2019 14:04:59.631 INFO [Catalina-utility-1] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization started
03-Dec-2019 14:04:59.631 INFO [Catalina-utility-1] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.prepareRefresh Refreshing WebApplicationContext for namespace 'dispatcher-servlet': startup date [Tue Dec 03 14:04:59 NZDT 2019]; root of context hierarchy
03-Dec-2019 14:04:59.709 INFO [Catalina-utility-1] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.loadBeanDefinitions Registering annotated classes: [class com.atlassian.synchrony.proxy.websocket.WebSocketConfig,class com.atlassian.synchrony.proxy.web.SynchronyWebMvcConfig]
03-Dec-2019 14:05:00.475 INFO [Catalina-utility-1] org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler.initialize Initializing ExecutorService 'defaultSockJsTaskScheduler'
03-Dec-2019 14:05:00.538 INFO [Catalina-utility-1] org.springframework.web.socket.server.support.WebSocketHandlerMapping.registerHandler Mapped URL path [/v1/bayeux-sync1] onto handler of type [class org.springframework.web.socket.server.support.WebSocketHttpRequestHandler]
03-Dec-2019 14:05:01.459 INFO [Catalina-utility-1] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.initControllerAdviceCache Looking for @ControllerAdvice: WebApplicationContext for namespace 'dispatcher-servlet': startup date [Tue Dec 03 14:04:59 NZDT 2019]; root of context hierarchy
03-Dec-2019 14:05:01.569 INFO [Catalina-utility-1] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/healthcheck]}" onto public com.atlassian.synchrony.proxy.web.HealthCheckResult com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.getSynchronyProxyInfo()
03-Dec-2019 14:05:01.569 INFO [Catalina-utility-1] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/reload],methods=[PUT]}" onto public org.springframework.http.ResponseEntity com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.reloadConfiguration(com.atlassian.synchrony.proxy.web.SynchronyProxyConfigPayload)
03-Dec-2019 14:05:01.584 INFO [Catalina-utility-1] org.springframework.web.servlet.handler.SimpleUrlHandlerMapping.registerHandler Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.DefaultServletHttpRequestHandler]
03-Dec-2019 14:05:01.725 INFO [Catalina-utility-1] org.springframework.context.support.DefaultLifecycleProcessor.start Starting beans in phase 2147483647
03-Dec-2019 14:05:01.756 INFO [Catalina-utility-1] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization completed in 2125 ms

==

UPDATE

Thanks!I am new to Confluence, and using local H2 Database.

This the error message:

==========

Database: Database connection failed

Confluence failed to establish a connection to your database.
This could be because:

• Your database isn't running
• The configuration of your confluence.cfg.xml file is incorrect (user, password, or database URL etc.)
• There is a network issue between Confluence and your database (e.g. firewall, database doesn't allow remote access etc.)

===============

Found SEVERE log. Wondering iof its my version of Java?

==

10-Dec-2019 07:35:11.901 SEVERE [http-nio-8090-exec-10] org.apache.catalina.core.StandardHostValve.custom Exception Processing ErrorPage[errorCode=500, location=/500page.jsp]
org.apache.jasper.JasperException: An exception occurred processing [500page.jsp] at line [120]

117:
118: if (sysInfoService != null)
119: {
120: confluenceInfo = sysInfoService.getConfluenceInfo();
121: memoryInfo = sysInfoService.getMemoryInfo();
122: dbInfo = sysInfoService.getSafeDatabaseInfo();
123: sysinfo = GeneralUtil.convertBeanToMap(sysInfoService.getSystemProperties());

[Daniel Eads]

Hey @Mark T , welcome to the Community and thanks for giving Confluence a try.

The embedded H2 database is easy to start with, but unfortunately known to be difficult to work with and gives less-than-helpful error messages sometimes. It's more or less a file in Confluence's home directory, which over time leads to potential issues with recovering data if anything should happen to corrupt it (not to mention performance issues).

That being said, given that the H2 database is a file, it is possible for filesystem permissions to prevent Confluence from being able to access the H2 database file. Are you running Confluence under the same user that you installed it with? What are the permissions on Confluence's home directory? If you're using Linux, you can check the permissions of Confluence's home by running the ls command.

If you're running Confluence with the confluence user, you can also use these commands (putting in the correct confluence home directory path) to make sure the home directory permissions are correct:

$ chown -R confluence <home-directory>
$ chmod -R u=rwx,go-rwx <home-directory>

 Let me know how this goes, and we'll continue troubleshooting if necessary!

Cheers,
Daniel | Atlassian Support

[Mark T]

Hi Daniel, 

I am running on Windoiws Server 2016, Checked my permissions and I can see its permissions are configured for SYSTEM., and applied the same to my account just in case.

I notice high CPU on the Commons Daemon Server as well. I have provisoned 2 CPUs and 8 GB ram for this server. 

[Mark T]

Any more definite way of identifying the issue?

This sounds to me a serious risk,  if logs can't exactly tell what's wrong. :(

[Daniel Eads]

If the file permissions are correct, I'd suspect corruption of the H2 database at this point.

The best path forward would be to spin up a traditional database and port your data over there, hopefully using an existing backup. Here's how I recommend tackling it:

1. Check for the presence of an automatic backup - out of the box, Confluence runs a backup job that should have the data of the most recent day Confluence was running, provided the backup job wasn't disabled by an administrator. More details about this backup job are here, but in short you should find the XMLs generated by this job in the backups directory of Confluence's home folder. If you used the defaults in the installer, this will be:
   

   C:\confluence\data\backups

2. Create a database in a supported database engine - instructions for each of the supported databases are here:
   
   • PostgreSQL
   • MySQL
   • Microsoft SQL Server
   • Oracle
3. Rename the dbconfig.xml file in Confluence's home directory to something like "dbconfig.xml.old". When Confluence starts up, it will run the configuration wizard if it doesn't find this file. If you installed using the default paths, this file will be at:
   

   C:\confluence\data\dbconfig.xml

4. Restart Confluence. Since you renamed the configuration file in the last step, Confluence will re-run the configuration wizard when it starts up. Configure it to point at your newly-set-up database. Make sure to chose the Production Option when installing Confluence so that it allows you to configure a supported database.
5. After the database configuration step, you'll see a "Load content" screen. The bottom option on this screen is "Restore from Backup". Click this option and choose the XML file you found in Step 1. After the import runs, you should have all the content that existed in your Confluence site at the time of the last automated backup.

If that's not panning out for some reason, it is possible to open the H2 database with a tool such as DB Visualizer, although it is definitely better to use an XML backup to switch to a different DBMS if at all possible.

Cheers,
Daniel

[Jeff Turner]

Jira finds its H2 local database with a filesystem path. Perhaps that path was relative, and you started Jira from another location? Or perhaps the path changed?

There must be more to the error page than just "Database: Database connection failed". Details matter.