After reboot, Not running confluence and Not ERROR Log

[jyj0629]

I use jira and confluence for 7 month.
(untill one month, I reboot OS, jira and confluence is running well)

I upgrade confluence before one month
- So sorry, I can not remember exact term and version number. Before and after.
- If you want to know about confluence version, please talk to me how I can check that.
 I will answer

I don't understand why I can not running confluence after reboot.

I can not find any error log (catalina.out) like below.
I try to find some method on the internet for several time, but I can’t find answer.
please answer to me.

P.S) OS is Ubuntu and I got your license for one year.
I use your product for 7 month. (After reboot, Jira and mysql is running well)
I check firewall. (firewall is not running.)
I check “confluence.cfg.xml” in the “confluence-init.properties.”.
mysql login id/pwd is right. (jira is running well)
I think OS have a enough free memory below.

--------------------------------------------------------------------------------------------------

<Below Catalina Log File>

13-Apr-2019 03:45:37.715 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server} Setting property 'debug' to '0' did not find a matching property.
13-Apr-2019 03:45:37.807 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'debug' to '0' did not find a matching property.
13-Apr-2019 03:45:37.934 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine} Setting property 'debug' to '0' did not find a matching property.
13-Apr-2019 03:45:37.943 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'debug' to '0' did not find a matching property.
13-Apr-2019 03:45:37.992 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.
13-Apr-2019 03:45:38.018 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.
13-Apr-2019 03:45:38.340 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
13-Apr-2019 03:45:38.358 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
13-Apr-2019 03:45:38.370 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 741 ms
13-Apr-2019 03:45:38.384 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Tomcat-Standalone]
13-Apr-2019 03:45:38.384 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/9.0.8

<Process status> 

# ps -ef | grep tomcat
jira 2567 1 22 03:37 ? 00:06:06 /opt/atlassian/jira/jre//bin/java -Djava.util.logging.config.file=/opt/atlassian/jira/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms384m -Xmx768m -Djava.awt.headless=true -Datlassian.standalone=JIRA -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Dmail.mime.decodeparameters=true -Dorg.dom4j.factory=com.atlassian.core.xml.InterningDocumentFactory -XX:-OmitStackTraceInFastThrow -Datlassian.plugins.startup.options= -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xloggc:/opt/atlassian/jira/logs/atlassian-jira-gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=20M -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCCause -Dignore.endorsed.dirs= -classpath /opt/atlassian/jira/bin/bootstrap.jar:/opt/atlassian/jira/bin/tomcat-juli.jar -Dcatalina.base=/opt/atlassian/jira -Dcatalina.home=/opt/atlassian/jira -Djava.io.tmpdir=/opt/atlassian/jira/temp org.apache.catalina.startup.Bootstrap start
root 7204 1838 0 04:04 pts/0 00:00:00 grep tomcat

# ps -ef | grep conflu
avahi 879 1 0 03:31 ? 00:00:00 avahi-daemon: running [wedance-confluence.local]
conflue+ 1892 1 99 03:35 ? 01:30:32 /boot/vmlinuz
conflue+ 1921 1 0 03:35 ? 00:00:00 ./Oo5mIH6 ./I7f979N
conflue+ 1922 1 0 03:35 ? 00:00:00 ./Oo5mIH6 ./PCAaKB6
conflue+ 7277 1921 0 04:05 ? 00:00:00 sleep 10
conflue+ 7282 1922 0 04:05 ? 00:00:00 sleep 5
root 7284 1838 0 04:05 pts/0 00:00:00 grep conflu

<Memroy and CPU Staus : top>

top - 04:34:02 up 1:02, 2 users, load average: 3.17, 3.20, 3.17
Tasks: 258 total, 1 running, 190 sleeping, 0 stopped, 1 zombie
%Cpu(s): 37.7 us, 0.1 sy, 0.0 ni, 59.8 id, 2.3 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 8092724 total, 4004964 free, 2384344 used, 1703416 buff/cache
KiB Swap: 2097148 total, 2097148 free, 0 used. 5253924 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1892 conflue+ 20 0 328908 8236 0 S 299.6 0.1 176:42.63
1279 wedance 20 0 3858600 137492 70500 S 0.9 1.7 0:05.05 gnome-shell
2567 jira 20 0 6791712 1.402g 34152 S 0.4 18.2 7:23.24 java
10846 wedance 20 0 52756 4480 3708 R 0.4 0.1 0:00.02 top
1 root 20 0 225532 9484 6820 S 0.0 0.1 0:01.83 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/0:0H
6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_percpu_wq
7 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/0
8 root 20 0 0 0 0 I 0.0 0.0 0:00.53 rcu_sched
9 root 20 0 0 0 0 I 0.0 0.0 0:00.00 rcu_bh
10 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
11 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/1
14 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
15 root rt 0 0 0 0 S 0.0 0.0 0:00.01 migration/1
16 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/1
18 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/1:0H
19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/2
20 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/2

[jyj0629]

Do you know why that process is using CPU fully?

Confluence is downed and that is appeared.

what is that and how can i do?

I killed that every time but this situation is repeatly over and over...

[Daniel Eads]

Based on this, it seems like you might have been affected by a malware attack from the CVE-2019-3396 Widget Connector vulnerability from March 20th (see Confluence Security Advisory - 2019-03-20). Do you remember exactly when you upgraded Confluence? We started seeing attacks in the wild for this vulnerability on April 10.

Confluence 6.15.2 is a version that is patched against the vulnerability. It was released on April 5 so I am wondering if you maybe upgraded late last week (and possibly got infected with malware before upgrading).

If it was a malware attack, you'll still need to clean up the malware from the server. The LSD malware cleanup tool will be useful for removing the Kerberods malware, which we saw most commonly on April 10 specifically. Other malware payloads might need different cleanup tools depending on which attack and payload were used. A good starting place for detecting other types of infections are the scanners linked here. Once a particular infection is identified, googling for "____ removal tool" is a good place to start if the scanner was unable to remove the malware automatically.

[Daniel Eads]

Hey there,

In addition to Dave's great troubleshooting steps, you can also determine the version of Confluence even if it is off (which will be helpful information for us).

The installation directory for Confluence contains a file called README.txt. At the top of this file, the version of Confluence is listed.

Thanks!
Daniel | Atlassian Support

[jyj0629]

Thanks for your advise.

I found confluence version 

"Confluence 6.15.2" 

How can i do?

[Dave Theodore [Coyote Creek Consulting]]

Unfortunately, none of this is very useful in troubleshooting. It does seem, from your ps output, that Confluence is not running.  I find it better to grep for "java" as you will see all Java processes this way.

Here's what I would do to troubleshoot.

1. Become the user that Jira runs as.
2. Move all log files from the log directories in the installation and home directories. At minimum, rename catalina.out and atlassian-jira.log to some other names. The idea is to have a clean set of log files that capture a single start up of the application.
3. Start Jira and watch the logs
4. Look for the first ERROR instance in the log, as this is your likely culprit. There will be stack trace following that with some hopefully useful information.
5. Repeat this with Confluence

Once you get both applications running, you need to verify that your start scripts are running as expected. Stopping and starting should be enough to exercise them as they would be during a reboot.

If you still can't figure out what is going on, please post the full log on pastebin or some other similar service so we can have a look at the full log files.