Thanks, it helped. Only the script was located at: /var/spool/cron/
Based on your version of Confluence (6.0.1) and symptoms, it sounds like your instance was affected by an opportunistic attack against the CVE-2019-3396 Widget Connector vulnerability from March 20th (see Confluence Security Advisory - 2019-03-20). We've seen an infection going around that injects malware and the bitcoin miner it tries to run uses all the CPU available on the box. Initially the kerberods malware was being deployed as the payload, but other attacks might be trying to inject different payloads.
I'd recommend tackling things in this order:
Malicious processes
The top command will help you find processes (probably running under the confluence user account) that are consuming a large amount of CPU. If Confluence is currently stopped, you can probably plan on killing any processes running as the confluence user. note the process ID (pid) from the top output and then kill the process using kill -9 followed by the pid. Example:
sudo kill -9 12395
Clean up your crontab
Since most malware adds a cronjob that relaunches the malware every few minutes, you'll also need to check the crontab file and remove any suspicious-looking entries. For Ubuntu, this is stored in the /var/spool/cron/crontabs/ directory. Normally you should use the crontab command to edit the crontab, but for cleanup purposes we'll be inspecting the file for any pre-existing entries.
Using vim (or whichever text editor you're comfortable with), you'll open the file and remove suspicious-looking jobs.
sudo vim /var/spool/cron/crontabs/confluence
Confluence comes up on system startup through the SysV/systemd daemons, so we would expect the confluence user's crontab to not exist under normal circumstances. It's most likely the case that any entries in this file are malicious, but make sure you check them before deleting them entirely.
Upgrade Confluence
Once your CPU is under control and new malicious process aren't spawning, you need to upgrade Confluence to a version that isn't affected by the vulnerability. I'd recommend looking at one of these versions (latest releases as of this post):
Use a malware scanner
Finally, you need to clean up any remaining traces of malware on your system. The LSD malware cleanup tool will be useful for removing the Kerberods malware. Other malware payloads might need different cleanup tools depending on which attack and payload were used. A good starting place for detecting other types of infections are the scanners linked here. Once a particular infection is identified, googling for "____ removal tool" is a good place to start if the scanner was unable to remove the malware automatically.
Please let me know if you have more questions!
Daniel | Atlassian Support
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
"atlassian-confluence.log"
2019-04-16 07:16:23,987 ERROR [localhost-startStop-1] [confluence.sql.migration.SqlConfigMigrationListener] onPluginEnabled Hibernate session not yet ready : java.lang.IllegalStateException: No Hibernate Session bound to thread, and configuration does not allow creation of non-transactional one here
2019-04-16 07:16:24,158 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] logTime Plugin system lateStartup ended
2019-04-16 07:16:24,998 INFO [lifecycle:thread-10] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] startup Starting Synchrony and enabling Collaborative Editing
2019-04-16 07:16:25,037 WARN [lifecycle:thread-8] [expose.jmx.schedule.JmxInstrumentSchedulerImpl] onStart atlassian-instrumentation-jmx expose scheduler started.
2019-04-16 07:16:25,971 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: false
2019-04-16 07:16:25,972 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled app config synchrony.proxy.enabled: true
2019-04-16 07:16:26,025 WARN [lifecycle:thread-4] [hql.internal.ast.HqlSqlWalker] generatePositionalParameter [DEPRECATION] Encountered positional parameter near line 1, column 93 in HQL: [FROM com.atlassian.confluence.impl.schedule.caesium.SchedulerClusteredJob t WHERE t.jobId = ?]. Positional parameter are considered deprecated; use named parameters or JPA-style positional parameters instead.
2019-04-16 07:16:26,835 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment Synchrony working dir: /home/naftahome/confluence
2019-04-16 07:16:26,836 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment /home/naftahome/atlassian-confluence-6.0.1/jre/bin/java -classpath /root/confbackup/atlassian-confluence-6.0.1/temp/2.1.0-master-e100417c.jar:/root/confbackup/atlassian-confluence-6.0.1/confluence/WEB-INF/lib/postgresql-42.1.1.jar -Xss2048k -Xmx1g synchrony.core sql
2019-04-16 07:16:27,293 INFO [ListenableFutureAdapter-thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: false
2019-04-16 07:16:27,293 INFO [ListenableFutureAdapter-thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled app config synchrony.proxy.enabled: true
2019-04-16 07:16:27,324 INFO [ListenableFutureAdapter-thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Base URL: http://nafta.wiki/synchrony-proxy,http://nafta.wiki/synchrony-proxy
2019-04-16 07:16:27,324 INFO [ListenableFutureAdapter-thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Service URL: http://nafta.wiki/synchrony-proxy/v1
2019-04-16 07:16:27,324 INFO [ListenableFutureAdapter-thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony Internal Service URL: http://127.0.0.1:8091/synchrony/v1
2019-04-16 07:16:27,627 INFO [UpmScheduler:thread-1] [atlassian.confluence.user.DefaultUserAccessor] getUserNamesWithConfluenceAccess Found USE permission with no associated username or group: [USECONFLUENCE,0,null,null,null]
2019-04-16 07:16:37,581 WARN [ListenableFutureAdapter-thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProxyMonitor] pollHealthcheck Could not ping the synchrony-proxy [http://127.0.0.1:8090/synchrony-proxy/healthcheck]: Read timed out
2019-04-16 07:16:37,588 WARN [ListenableFutureAdapter-thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProxyMonitor] startHealthcheck The synchrony-proxy has not been started yet. Another healthcheck will happen in 30 seconds.
2019-04-16 07:16:50,426 WARN [ListenableFutureAdapter-thread-1] [plugins.synchrony.config.DefaultSynchronyConfigurationManager] enableSharedDrafts [Collab editing plugin] Enabling Shared Drafts
2019-04-16 07:16:55,500 INFO [localhost-startStop-1] [confluence.upgrade.impl.DefaultUpgradeGate] waitForBooleanValue Waiting to find if plugin dependent upgrades are required. Maximum wait time will be 90 seconds.
2019-04-16 07:16:55,500 INFO [localhost-startStop-1] [confluence.upgrade.impl.DefaultUpgradeGate] waitForBooleanValue plugin dependent upgrades are required : false.
2019-04-16 07:16:57,111 WARN [Caesium-1-4] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob completed unsuccessfully with response JobRunnerResponse[runOutcome=ABORTED,message='LaaS performance logging is turned off']
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.