Getting HTTP Status 500 error on opening Confluence login page

[Jaydeep Sarkar]

Hello,

Please find the background and details of the problem below. I've seen similar problem being discussed in this community but even after trying out all of the recommendations, it did not get fixed. Not sure, if I've missed out any of the suggested steps.

Background:

1) The confluence application version 5.4.3 is running on Linux 2.6.32-279.el6.x86_64 on confluence user.

501 14491 1 0 Apr24 ? 00:02:01 /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java -Djava.util.logging.config.file=/opt/atlassian-confluence-5.4.3/conf/logging.properties -Xms256m -Xmx512m -XX:MaxPermSize=256m -Djava.awt.headless=true -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/opt/atlassian-confluence-5.4.3/endorsed -classpath /opt/atlassian-confluence-5.4.3/bin/bootstrap.jar -Dcatalina.base=/opt/atlassian-confluence-5.4.3 -Dcatalina.home=/opt/atlassian-confluence-5.4.3 -Djava.io.tmpdir=/opt/atlassian-confluence-5.4.3/temp org.apache.catalina.startup.Bootstrap start

Problem:

3) When I'm trying to launch loggin screen http://<hostname>.com:8090/

I'm getting the following error on the home page:

HTTP Status 500 -

type Exception report

message

description The server encountered an internal error () that prevented it from fulfilling this request.

exception

com.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.NullPointerException
com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:149)
com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112)
com.atlassian.confluence.setup.ConfluenceEncodingFilter.getGlobalSettings(ConfluenceEncodingFilter.java:45)
com.atlassian.confluence.setup.ConfluenceEncodingFilter.getEncodingInternal(ConfluenceEncodingFilter.java:35)
com.atlassian.confluence.setup.ConfluenceEncodingFilter.getEncoding(ConfluenceEncodingFilter.java:27)
com.atlassian.core.filters.encoding.AbstractEncodingFilter.doFilter(AbstractEncodingFilter.java:38)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44)
com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:65)

root cause

java.lang.NullPointerException
com.atlassian.spring.container.ContainerManager.getComponent(ContainerManager.java:33)
com.atlassian.confluence.util.LazyComponentReference$Accessor.get(LazyComponentReference.java:46)
com.atlassian.util.concurrent.Lazy$Strong.create(Lazy.java:85)
com.atlassian.util.concurrent.LazyReference$Sync.run(LazyReference.java:321)
com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:143)
com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112)
com.atlassian.confluence.setup.ConfluenceEncodingFilter.getGlobalSettings(ConfluenceEncodingFilter.java:45)
com.atlassian.confluence.setup.ConfluenceEncodingFilter.getEncodingInternal(ConfluenceEncodingFilter.java:35)
com.atlassian.confluence.setup.ConfluenceEncodingFilter.getEncoding(ConfluenceEncodingFilter.java:27)
com.atlassian.core.filters.encoding.AbstractEncodingFilter.doFilter(AbstractEncodingFilter.java:38)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44)
com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:65)

Log files:

One checking confluence logs generated at /opt/atlassian-confluence-5.4.3/logs/

-rw-r--r-- 1 confluence confluence 2794106 Apr 26 12:22 catalina.out
-rw-r--r-- 1 confluence confluence 34062 Apr 26 12:22 catalina.2017-04-26.log

It shows following errors:

Apr 26, 2017 12:22:59 PM org.apache.catalina.core.StandardHostValve custom
SEVERE: Exception Processing ErrorPage[errorCode=500, location=/500page.jsp]
java.lang.NullPointerException
at com.atlassian.spring.container.ContainerManager.getComponent(ContainerManager.java:33)
at com.atlassian.confluence.setup.sitemesh.PluginDecoratorMapper.getPluginAccessor(PluginDecoratorMapper.java:74)
at com.atlassian.confluence.setup.sitemesh.PluginDecoratorMapper.getDecorator(PluginDecoratorMapper.java:23)
at com.opensymphony.module.sitemesh.mapper.AbstractDecoratorMapper.getDecorator(AbstractDecoratorMapper.java:45)
at com.opensymphony.module.sitemesh.mapper.InlineDecoratorMapper.getDecorator(InlineDecoratorMapper.java:41)
at com.atlassian.confluence.util.profiling.ConfluenceDecoratorSelector.selectDecorator(ConfluenceDecoratorSelector.java:58)
at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:83)
at com.atlassian.confluence.util.profiling.ProfilingPageFilter.doFilter(ProfilingPageFilter.java:44)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:71)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:211)
at com.atlassian.confluence.web.filter.ConfluenceSecurityFilter.doFilter(ConfluenceSecurityFilter.java:27)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:150)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:71)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:71)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:50)
at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:438)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:374)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:421)
at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:342)
at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:286)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:745)

What all things I've tried:

1) I verified that I can connect to MySql DB.

A snapshot of mysql process.

root 29010 1 0 Apr21 ? 00:00:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/inmyzd0033.in.dst.ibm.com.pid
mysql 29224 29010 0 Apr21 ? 00:04:33 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/lib/mysql/inmyzd0033.in.dst.ibm.com.pid --socket=/var/lib/mysql/mysql.sock

2) I checked the permission level for all directories, sub directories and files under Confluence home /opt/ConfData/

drwxr-xr-x 3 confluence confluence 4096 Feb 27 2014 attachments
drwxr-xr-x 2 confluence confluence 4096 May 30 2016 backups
drwxr-xr-x 2 confluence confluence 4096 Feb 27 2014 bundled-plugins
drwxr-xr-x 2 confluence confluence 4096 Feb 27 2014 config
-rw-r--r-- 1 confluence confluence 2764 Apr 24 19:40 confluence.cfg.xml
drwxr-xr-x 2 confluence confluence 4096 May 13 2015 imgEffects
drwxr-xr-x 4 confluence confluence 4096 Apr 24 19:40 index
drwxr-xr-x 3 confluence confluence 4096 Apr 21 17:19 logs
drwxr-xr-x 2 confluence confluence 4096 Apr 24 19:40 plugins-cache
drwxr-xr-x 5 confluence confluence 4096 Apr 24 17:34 plugins-osgi-cache
drwxr-xr-x 2 confluence confluence 4096 Apr 22 01:48 plugins-temp
drwxr-xr-x 2 confluence confluence 4096 Feb 27 2014 restore
drwxr-xr-x 2 confluence confluence 4096 Apr 5 18:57 temp
drwxr-xr-x 215 confluence confluence 4096 Mar 8 14:50 thumbnails
drwxr-xr-x 3 confluence confluence 20480 Feb 3 15:46 viewfile
drwxr-xr-x 2 confluence confluence 36864 Apr 24 19:39 webresource-temp

3) Cleared up cache and then restarted confluence app i.e. Cleared following directories -

bundled-plugins, plugins-cache, plugins-osgi-cache, plugins-temp

Appreciate if someone could assist me resolving this issues.

Thanks in advance.

- Jaydeep

[Nic Brough -Adaptavist-]

There's lots of things that can cause that error.  You have ruled a number of them out with your investigations, and you mention changes to storage which I suspect are the root cause.

The usual culprit in my notes are:

1. Trying to run Confluence with the wrong version of Java
2. The user running Confluence does not have the right permissions on the confluence-install and/or confluence-home directories
3. You've moved the Confluence home directory
4. There's a corrupted cache
5. There's a corrupted index
6. The database is inaccessible or has the wrong permissions for the user

I reckon it's 2 or possibly 3 from what you've said.

[Jaydeep Sarkar]

Thanks Nic for your response.

Aparently -

Reg your point # 2:

I checked the permission level of home directory:

drwxr-xr-x 3 confluence confluence 4096 Feb 27 2014 attachments
drwxr-xr-x 2 confluence confluence 4096 May 30 2016 backups
drwxr-xr-x 2 confluence confluence 4096 Feb 27 2014 bundled-plugins
drwxr-xr-x 2 confluence confluence 4096 Feb 27 2014 config
-rw-r--r-- 1 confluence confluence 2764 Apr 24 19:40 confluence.cfg.xml
drwxr-xr-x 2 confluence confluence 4096 May 13 2015 imgEffects
drwxr-xr-x 4 confluence confluence 4096 Apr 24 19:40 index
drwxr-xr-x 3 confluence confluence 4096 Apr 21 17:19 logs
drwxr-xr-x 2 confluence confluence 4096 Apr 24 19:40 plugins-cache
drwxr-xr-x 5 confluence confluence 4096 Apr 24 17:34 plugins-osgi-cache
drwxr-xr-x 2 confluence confluence 4096 Apr 22 01:48 plugins-temp
drwxr-xr-x 2 confluence confluence 4096 Feb 27 2014 restore
drwxr-xr-x 2 confluence confluence 4096 Apr 5 18:57 temp
drwxr-xr-x 215 confluence confluence 4096 Mar 8 14:50 thumbnails
drwxr-xr-x 3 confluence confluence 20480 Feb 3 15:46 viewfile
drwxr-xr-x 2 confluence confluence 36864 Apr 24 19:39 webresource-temp

Do you notice any issue here?

Reg point 3:

Additional storage capacity was added to the partition hostingteh file system /dev/sda2.

And Confluence home directory still sitting in that FS.

[confluence@inmyzd0033 ConfData]$ df -h /opt/ConfData/
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 2.0T 811G 1.2T 41% /

So not sure if 2 & 3 can be responsible.

Do you think I should seomething else as well?

Thanks again.

[Nic Brough -Adaptavist-]

You might want to try "df -i" as well when checking the size of the disk.

I assume confluence is being run as a user also called "confluence", who is the owner you've got for the files in home, so that's ok, but you need to check the ownership and access of all the files beneath there AND, as I said before, the confluence installation.

[Jaydeep Sarkar]

Thanks Nic.

This is what I see.

[confluence@inmyzd0033 ~]$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda2 133685280 306728 133378552 1% /
tmpfs 749316 6 749310 1% /dev/shm
/dev/sda1 50200 38 50162 1% /boot
/dev/sdb1 67108864 11 67108853 1% /data

[Jaydeep Sarkar]

Also, the installation directory is /opt/atlassian-confluence-5.4.3 and it has following permission level:

drwxr-xr-x 11 confluence confluence 4096 Feb 12 2014 atlassian-confluence-5.4.3

And whatever comes inside thsi installation directory is as shown below. Permission wise I don't see any issue.

The ownnership of all files and folders inside home /opt/ConfData and installation was changed by

chown -R confluence:confluence *

But still the problem persists.

In the fist response you said-

Trying to run Confluence with the wrong version of Java
The user running Confluence does not have the right permissions on the confluence-install and/or confluence-home directories
You've moved the Confluence home directory
There's a corrupted cache
There's a corrupted index
The database is inaccessible or has the wrong permissions for the user

How do we address point # 5? Could it be because of this?

Thanks

[Jaydeep Sarkar]

Some additional information:

Please note that it was working fine till 5th/6th April. However after additional storage capacity was added, although the application is running but throwing those errors

Thanks!