nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Every Day Confluence dies. PID file exists with no process

stratus-ss

Every day without fail I have to log into my confluence box to run the start-confluence.sh script.

It is running on top of a CentOS 6.3 (64-bit) install. Confluence is running with a MySQL (5.1.67) database on a dedicated box. The VM has been provisioned with 2 CPUs @ 2.0 GHZ with 4G of ram and 2G of swap.

The Hard drive partitioning is as follows:

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/vg_confluence-root 9.9G 2.6G 6.8G 28% /

tmpfs 1.9G 88K 1.9G 1% /dev/shm

/dev/sda1 194M 54M 131M 30% /boot

/dev/mapper/vg_confluence-confluence 82G 622M 77G 1% /confluence

/dev/mapper/vg_confluence-home 5.0G 139M 4.6G 3% /home

After the restart the application runs as expected, but during long periods of idleness confluence stops responding

Accepted answers

All comments

stuartu

Some obvious questions:

1. What error(s) do you see in your log files?
2. What errors do people in their browser when Confluence stops?
3. What version of Confluence?
4. When did this start occurring?
5. Have you installed, updated Confluence and/or plugin(s) recently?

stratus-ss

2013-02-24 17:09:21,038 WARN [http-8080-1] [jersey.spi.inject.Errors] processErrorMessages The following warnings have been detected with resource and/or provider classes:
2013-02-24 17:09:30,584 ERROR [http-8080-4] [atlassian.event.internal.EventPublisherImpl] invokeListeners There was an exception thrown trying to dispatch event 'com.atlassian.confluence.event.events.content.mail.notification.ContentNotificationAddedEvent[source=com.atlassian.confluence.mail.notification.DefaultNotificationManager@234bb715]' from the invoker 'com.atlassian.event.internal.SingleParameterMethodListenerInvoker@6c911796'.
at com.atlassian.confluence.util.ConfluenceErrorFilter.doFilter(ConfluenceErrorFilter.java:22)
at com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:65)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

Feb 24, 2013 5:09:21 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public javax.ws.rs.core.Response com.atlassian.confluence.tinymceplugin.rest.DraftsResource.getDrafts(int,int), should not consume any entity.
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000003f1629c188, pid=2098, tid=140462108604160
#
# JRE version: 6.0_26-b03
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libc.so.6+0x9c188] __gettimeofday+0x18
#

It does reference a file hs_err_pd2098.log but the file looks to be almost all memory/core dumps

stratus-ss

So let me pick off a few of these (first I should state that I attached the logs when submitting the ticket to atlassian, I assumed they would show up here but they obviously did not)

2) The browsers hang and eventually time out

3) Confluence 4.3.7

4) From the first time I completed the install 2-3 weeks ago

5) I have installed no plugins, and have not updated confluence assuming that the binary I downloaded from the website was the most current.

1) I see a few java exceptions similar to this

2013-02-23 22:08:28,011 ERROR [scheduler_Worker-5] [sf.hibernate.util.JDBCExceptionReporter] logExceptions Communications link failure
2013-02-23 22:08:28,015 ERROR [scheduler_Worker-6] [sf.hibernate.util.JDBCExceptionReporter] logExceptions No operations allowed after connection closed.Connection was implicitly closed by the driver.
2013-02-23 22:08:28,016 ERROR [scheduler_Worker-5] [sf.hibernate.util.JDBCExceptionReporter] logExceptions Communications link failure
2013-02-23 22:08:28,018 ERROR [scheduler_Worker-5] [confluence.schedule.quartz.ConfluenceQuartzThreadPool] run Error while executing the Runnable:
at net.sf.hibernate.exception.ErrorCodeConverter.handledNonSpecificException(ErrorCodeConverter.java:90)
at net.sf.hibernate.exception.ErrorCodeConverter.convert(ErrorCodeConverter.java:79)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1119)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3486)

stuartu

There are issues between Confluence and MySQL, given that you have JDBC errors.

We don't run MySQL, so someone else may be able to help, but I'd be re-reading the documentation about database connectivity. Also have a look at your MySQL logs, and possibly your server logs.

https://confluence.atlassian.com/display/CONF43/Database+Setup+For+MySQL

Davet1

Steve,

I think that java version (which I think is 1.6.0_26?) may be problematic.

I have nearly the same setup as you but have been running Java(TM) SE Runtime Environment (build 1.6.0_32-b05) for quite some time without problems.

Are you using the .tar.gz or the installer?

stratus-ss

I am using the binary installer of confluence and the repos for java.

Are you recommending that I build java from source?

stuartu

I'd be surprised if that was the problem, but always good to go with the latest Oracle Java.. Java 1.6 update 41 JDK.

http://www.oracle.com/technetwork/java/javase/downloads/jdk6downloads-1902814.html

Davet1

Not build from source - just unpack the .tar.gz and shove it in a folder.

Actually the latest JDK is 1.7.0_15 but you do not want to be going there. The one stuartu recommends is good.

You can be certain that at least one component of your install is bad - just keep replacing bits till it works like I do

Check your mysql install form top to bottom, JDBC driver, reinstall everything etc

If still no luck then you need to get your crashdumps which I have found to be not that straightforward depending on the nature of the crash.

http://middlewaremagic.com/weblogic/?p=4482

http://weblogs.java.net/blog/kohsuke/archive/2009/02/crash_course_on.html

stratus-ss

I have upgraded to the version of java linked here, verified the version and started confluence. I will give an update in a few days when stability can be determined

stratus-ss

Mar 11, 2013 6:03:51 AM com.sun.jersey.spi.container.servlet.WebComponent filterFormParameters
WARNING: A servlet POST request, to the URI http://confluence/rest/dashboardmacros/1.0/updates, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000003f1629c188, pid=2087, tid=139882862630656
#
# JRE version: 6.0_26-b03
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libc.so.6+0x9c188] __gettimeofday+0x18
#
# An error report file with more information is saved as:
# /opt/atlassian/confluence/bin/hs_err_pid2087.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp

# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000003f1629c188, pid=2087, tid=139882862630656
#
# JRE version: 6.0_26-b03
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode linux-amd64 compressed
oops)
# Problematic frame:
# C [libc.so.6+0x9c188] __gettimeofday+0x18
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#

--------------- T H R E A D ---------------

Current thread (0x00007f3908cfb000): JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon [_thread_blocked, id=2243, stack(0x00007f3904465000,0x00007f3904566000)]

stratus-ss

[code]

Mar 11, 2013 6:03:51 AM com.sun.jersey.spi.container.servlet.WebComponent filterFormParameters
WARNING: A servlet POST request, to the URI http://confluence/rest/dashboardmacros/1.0/updates, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000003f1629c188, pid=2087, tid=139882862630656
#
# JRE version: 6.0_26-b03
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libc.so.6+0x9c188] __gettimeofday+0x18
#
# An error report file with more information is saved as:
# /opt/atlassian/confluence/bin/hs_err_pid2087.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
[/code]

[code]

# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000003f1629c188, pid=2087, tid=139882862630656
#
# JRE version: 6.0_26-b03
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode linux-amd64 compressed
oops)
# Problematic frame:
# C [libc.so.6+0x9c188] __gettimeofday+0x18
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#

--------------- T H R E A D ---------------

Current thread (0x00007f3908cfb000): JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon [_thread_blocked, id=2243, stack(0x00007f3904465000,0x00007f3904566000)]
[/code]

stuartu

> C [libc.so.6+0x9c188] __gettimeofday+0x18

Is something up with that libc package? Is it an old version? Are all your packages up to date?

stratus-ss

The latest OS update was Feb 26, 2013. So I may be a couple of weeks behind but on the whole I would say I am well patched.

Even if something is wrong with libc, there is not much I can do about it until an update comes down the pipes. There is no way I am going to compile libc from sorce

stuartu

> # JRE version: 6.0_26-b03

If you installed the latest (Oracle) Java 1.6, then you aren't using it. The version should be 6.0_43.

You would also want the JDK, not the JRE.

http://www.oracle.com/technetwork/java/javase/downloads/jdk6downloads-1902814.html

stuartu

On linux, I have the JAVA_HOME environment variable set.

I presume you run confluence as a different user from root. Double check the java -version command gives the correct version there to.

Java versions change quite quickly. I wouldn't be too concerned with the minor difference, but 43 would obviously be preferred over 41.