System will not start up

[Todd Huntley]

System hosted on VM Ubuntu.  VM went offline early this morning.  Restarted system but, confluence will not come back online.  the Confluence service is running but, various error in logs that I am unsure of issue.

[AnnWorley · Accepted solution]

Hi Todd,

If this is happening in Production and severely impacting your business, please let me know so we can work on it with more urgency.

Please check the <Confluence_home_dir>/logs/atlassian-confluence.log. Search for the latest entry of "Starting Confluence" and see at what point in the subsequent logging errors are starting to appear. Please let me know what you find in the log. Note: The <confluence_home> directory is the path defined in the following file: <confluence_install>/confluence/WEB-INF/classes/confluence-init.properties

Some other questions: Do you see any error in the browser when you try to access Confluence? Are there any other applications on the VM which are accessible? If there is a proxy, is that back up since the VM went down?

Thanks,

Ann

[Todd Huntley]

Hi Ann,

Thanks for getting back so quickly.  Confluence is the only application running on this Ubuntu VM.  There is no proxy.  It is production. I deleted the log file as to get a clearer picture of the errors.  I restarted the service and I see a lot of WARN and this is the first

ERROR: 2018-03-14 13:40:13,981 ERROR [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hostComponentProvider': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire method: public void com.atlassian.plugin.spring.SpringHostComponentProviderFactoryBean.setSpringHostComponentProviderConfig(com.atlassian.plugin.spring.SpringHostComponentProviderConfig); nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bundledPluginLoader' defined in class path resource [services/pluginServiceContext.xml]: Cannot resolve reference to bean 'osgiPluginFactory' while setting constructor argument with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'osgiPluginFactory' defined in class path resource [services/pluginServiceContext.xml]: Cannot resolve reference to bean 'osgiPersistentCache' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'osgiPersistentCache': FactoryBean threw exception on object creation; nested exception is com.atlassian.plugin.osgi.container.OsgiContainerException: Unable to clean the cache directory: /var/atlassian/application-data/confluence/plugins-osgi-cache/felix
Related cause: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cacheStatisticsManager' defined in class path resource [services/cacheServiceContext.xml]: Invocation of init method failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ehCacheStatisticsManager' defined in URL [jar:file:/opt/atlassian/confluence/webapps/../confluence/WEB-INF/lib/confluence-cache-ehcache-6.4.0.jar!/cacheProviderContext.xml]: Cannot resolve reference to bean 'userI18NBeanFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'userI18NBeanFactory' defined in class path resource [i18NContext.xml]: Cannot resolve reference to bean 'localeManager' while setting bean property 'localeManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'localeManager' defined in class path resource [i18NContext.xml]: Cannot resolve reference to bean 'languageManager' while setting bean property 'languageManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'languageManager' defined in class path resource [applicationContext.xml]: Cannot resolve reference to bean 'settingsManager' while setting bean property 'settingsManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'settingsManager' defined in class path resource [applicationContext.xml]: Cannot resolve reference to bean 'upgradeManager' while setting bean property 'upgradeManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'upgradeManager' defined in class path resource [upgradeSubsystemContext.xml]: Cannot resolve reference to bean 'conditionallyEnableMailArchivePluginUpgradeTask' while setting bean property 'pluginDependentUpgradeTasks' with key [0]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'conditionallyEnableMailArchivePluginUpgradeTask' defined in class path resource [upgradeSubsystemContext.xml]: Cannot resolve reference to bean 'pluginController' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'pluginController' defined in class path resource [services/pluginServiceContext.xml]: Cannot resolve reference to bean 'pluginManager' while setting bean property 'pluginController'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'pluginManager' defined in class path resource [services/pluginServiceContext.xml]: Cannot resolve reference to bean 'pluginLoaders' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'pluginLoaders' defined in class path resource [services/pluginServiceContext.xml]: Cannot resolve reference to bean 'bundledPluginLoader' while setting constructor argument with key [0]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bundledPluginLoader' defined in class path resource [services/pluginServiceContext.xml]: Cannot resolve reference to bean 'osgiPluginFactory' while setting constructor argument with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'osgiPluginFactory' defined in class path resource [services/pluginServiceContext.xml]: Cannot resolve reference to bean 'osgiPersistentCache' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'osgiPersistentCache': FactoryBean threw exception on object creation; nested exception is com.atlassian.plugin.osgi.container.OsgiContainerException: Unable to clean the cache directory: /var/atlassian/application-data/confluence/plugins-osgi-cache/felix
Related cause: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'databasePluginLoader' defined in class path resource [services/pluginServiceContext.xml]: Cannot resolve reference to bean 'databasePluginLoaderTarget' while setting bean property 'target'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'databasePluginLoaderTarget' defined in class path resource [services/pluginServiceContext.xml]: Cannot resolve reference to bean 'osgiPluginFactory' while setting constructor argument with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'osgiPluginFactory' defined in class path resource [services/pluginServiceContext.xml]: Cannot resolve reference to bean 'osgiPersistentCache' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'osgiPersistentCache': FactoryBean threw exception on object creation; nested exception is com.atlassian.plugin.osgi.container.OsgiContainerException: Unable to clean the cache directory: /var/atlassian/application-data/confluence/plugins-osgi-cache/felix
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:334)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1214)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:753)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:538)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:444)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:326)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
at com.atlassian.spring.container.ContainerContextLoaderListener.contextInitialized(ContainerContextLoaderListener.java:32)
at com.atlassian.confluence.setup.ConfluenceBootstrappedContextLoaderListener.contextInitialized(ConfluenceBootstrappedContextLoaderListener.java:41)

[Todd Huntley]

Apologies, this is the first ERROR:  

2018-03-14 13:40:13,981 ERROR [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hostComponentProvider': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire method: public void com.atlassian.plugin.spring.SpringHostComponentProviderFactoryBean.setSpringHostComponentProviderConfig(com.atlassian.plugin.spring.SpringHostComponentProviderConfig); nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bundledPluginLoader' defined in class path resource [services/pluginServiceContext.xml]: Cannot resolve reference to bean 'osgiPluginFactory' while setting constructor argument with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'osgiPluginFactory' defined in class path resource [services/pluginServiceContext.xml]: Cannot resolve reference to bean 'osgiPersistentCache' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'osgiPersistentCache': FactoryBean threw exception on object creation; nested exception is com.atlassian.plugin.osgi.container.OsgiContainerException: Unable to clean the cache directory: /var/atlassian/application-data/confluence/plugins-osgi-cache/felix
Related cause: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cacheStatisticsManager' defined in class path resource [services/cacheServiceContext.xml]: Invocation of init method failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ehCacheStatisticsManager' defined in URL [jar:file:/opt/atlassian/confluence/webapps/../confluence/WEB-INF/lib/confluence-cache-ehcache-6.4.0.jar!/cacheProviderContext.xml]: Cannot resolve reference to bean 'userI18NBeanFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'userI18NBeanFactory' defined in class path resource [i18NContext.xml]: Cannot resolve reference to bean 'localeManager' while setting bean property 'localeManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'localeManager' defined in class path resource [i18NContext.xml]: Cannot resolve reference to bean 'languageManager' while setting bean property 'languageManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'languageManager' defined in class path resource [applicationContext.xml]: Cannot resolve reference to bean 'settingsManager' while setting bean property 'settingsManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'settingsManager' defined in class path resource [applicationContext.xml]: Cannot resolve reference to bean 'upgradeManager' while setting bean property 'upgradeManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'upgradeManager' defined in class path resource [upgradeSubsystemContext.xml]: Cannot resolve reference to bean 'conditionallyEnableMailArchivePluginUpgradeTask' while setting bean property 'pluginDependentUpgradeTasks' with key [0]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'conditionallyEnableMailArchivePluginUpgradeTask' defined in class path resource [upgradeSubsystemContext.xml]: Cannot resolve reference to bean 'pluginController' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'pluginController' defined in class path resource [services/pluginServiceContext.xml]: Cannot resolve reference to bean 'pluginManager' while setting bean property 'pluginController'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'pluginManager' defined in class path resource [services/pluginServiceContext.xml]: Cannot resolve reference to bean 'pluginLoaders' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'pluginLoaders' defined in class path resource [services/pluginServiceContext.xml]: Cannot resolve reference to bean 'bundledPluginLoader' while setting constructor argument with key [0]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bundledPluginLoader' defined in class path resource [services/pluginServiceContext.xml]: Cannot resolve reference to bean 'osgiPluginFactory' while setting constructor argument with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'osgiPluginFactory' defined in class path resource [services/pluginServiceContext.xml]: Cannot resolve reference to bean 'osgiPersistentCache' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'osgiPersistentCache': FactoryBean threw exception on object creation; nested exception is com.atlassian.plugin.osgi.container.OsgiContainerException: Unable to clean the cache directory: /var/atlassian/application-data/confluence/plugins-osgi-cache/felix

[AnnWorley]

Unable to clean the cache directory: /var/atlassian/application-data/confluence/plugins-osgi-cache/felix

is usually related to file system permissions. Please see Confluence does not start due to 'Unable to clean the cache directory' error for a resolution suggestion.

Please examine the way the service is set up to make sure the right OS user is starting Confluence.

[Todd Huntley]

This is the service and the user creds that I am staring up: -rwxr-xr-x 1 root root 376 Sep 15 10:58 confluence

I am starting the service up from this user:  root@TRGCONF01:/etc/init.d#

So I cleared out all the plugin cache and the log to get a clean view on startup.  Here is the first error that comes in:

2018-03-14 14:27:41,869 ERROR [ThreadPoolAsyncTaskExecutor::Thread 21] [plugin.osgi.factory.OsgiPlugin] onPluginContainerFailed Unable to start the plugin container for plugin 'com.atlassian.analytics.analytics-client'
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'productEventListener' defined in URL [bundle://227.0:0/META-INF/spring/atlassian-plugins-components.xml]: Unsatisfied dependency expressed through constructor argument with index 5 of type [com.atlassian.analytics.client.logger.AnalyticsLogger]: Error creating bean with name 'analyticsLogger' defined in URL [bundle://227.0:0/META-INF/spring/atlassian-plugins-components.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.analytics.client.logger.Log4jAnalyticsLogger]: Constructor threw exception; nested exception is java.lang.NullPointerException; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'analyticsLogger' defined in URL [bundle://227.0:0/META-INF/spring/atlassian-plugins-components.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.analytics.client.logger.Log4jAnalyticsLogger]: Constructor threw exception; nested exception is java.lang.NullPointerException

[Todd Huntley]

It there a way for me to get the current log file over to you?  Basically seeing this over and over now:  

2018-03-14 14:45:11,291 WARN [synchrony-interop-executor:thread-2] [plugins.synchrony.bootstrap.DefaultSynchronyProxyMonitor] pollHealthcheck Could not ping the synchrony-proxy [http://127.0.0.1:80/synchrony-proxy/healthcheck]: {}
org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused (Connection refused)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at com.atlassian.confluence.plugins.synchrony.bootstrap.DefaultSynchronyProxyMonitor.pollHealthcheck(DefaultSynchronyProxyMonitor.java:76)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)

[AnnWorley]

@Todd Huntley, I am creating a support request for you so you can attach logs, please stand by for an email from our portal.