Confluence not working at all

Ezequiel September 20, 2017

Hey guys,

 

All of a sudden, Confluence (latest server version) stopped working and we can't bring it back to life.

Any ideas? The error we see in the logs is:

19-Sep-2017 20:50:36.211 SEVERE [http-nio-8090-exec-1] org.apache.catalina.core.StandardHostValve.custom Exception Processing ErrorPage[errorCode=500, location=/500page.jsp]
org.apache.jasper.JasperException: java.lang.IllegalStateException: getOutputStream() has already been called for this response
at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:560)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:471)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
at com.atlassian.confluence.util.profiling.ConfluenceActivityFilter.doFilter(ConfluenceActivityFilter.java:39)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
at com.atlassian.confluence.web.filter.HttpRequestStatsFilter.doFilter(HttpRequestStatsFilter.java:47)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
at com.atlassian.gzipfilter.GzipFilter.doFilter(GzipFilter.java:96)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
at com.atlassian.confluence.web.filter.ConfluenceTimingFilter.doFilter(ConfluenceTimingFilter.java:46)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:716)
at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:584)
at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:523)
at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:436)
at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:301)
at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:396)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:176)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:502)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1132)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1533)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1489)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: getOutputStream() has already been called for this response
at org.apache.catalina.connector.Response.getWriter(Response.java:564)
at org.apache.catalina.connector.ResponseFacade.getWriter(ResponseFacade.java:212)
at javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:109)
at javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:109)
at com.atlassian.confluence.web.filter.DebugFilter$LoggingResponseWrapper.getWriter(DebugFilter.java:111)
at javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:109)
at com.atlassian.confluence.web.filter.DebugFilter$LoggingResponseWrapper.getWriter(DebugFilter.java:111)
at org.apache.jasper.runtime.JspWriterImpl.initOut(JspWriterImpl.java:115)
at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:108)
at org.apache.jasper.runtime.PageContextImpl.release(PageContextImpl.java:176)
at org.apache.jasper.runtime.JspFactoryImpl.internalReleasePageContext(JspFactoryImpl.java:120)
at org.apache.jasper.runtime.JspFactoryImpl.releasePageContext(JspFactoryImpl.java:75)
at org.apache.jsp._500page_jsp._jspService(_500page_jsp.java:944)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:438)
... 67 more

 

Thanks,

Ezequiel

4 answers

1 accepted

0 votes
Answer accepted
Leandro September 29, 2017

Hi Ann, I'm Ezequiel's colleague. I'd like to thank you for the links provided, however we were unable to fix confluence following those.

We managed to get confluence working, though, here's what I did:

- Create a mysql dump of confluence DB's

mysqldump -h <hostname> -u <user> -p <password> --database <dbname> > <whatever-name-you-like>.sql

 - Drop the old confluence DB. Create a new DB and grant privileges to the user, as if it was a new installation, cause that's what we're gonna do.

mysql -h <hostname> -u <user> -p <password>
drop database <dbname>;
create database confluence;
create user 'confluence'@'%' identified by '<password>';
grant all privileges on confluence.* to confluence;
exit

- Launch a new instance and install confluence on it. We installed 6.4.1. Don't terminate the old instance, you will need stuff from it!!

Follow this link to install confluence

- Stop confluence and connect to the DB. Drop the new database Confluence used for the installation and source the dump from step #1.

mysql -h <hostname> -u <user> -p <password>
drop database confluence;
create database confluence;
source confluence-dump.sql

 

Leandro September 29, 2017

- After that, we got the following error:

Confluence will not start up because the build number in the home directory [7401] doesn't match the build number in the database [7400].

This link will help you fix build number.

Solution #1 worked for us. 

- Then we had collation and character set issues. Several of them. 

Here's how to fix collation and character set issues

- Finally, we wanted all the images, thumbnails, attachments and etc we had uploaded. 

Go to:

/var/atlassian/application-data/confluence

And make a backup of attachments/ and thumbnails/ directory from the old instance.

Move all this content to the new confluence instance and just uncompress it on the same path.

- Start confluence. It worked like a charm.

Ezequiel September 29, 2017

Awesome!!

Shashidar October 12, 2017

Hi Ann,

May i know the reason for this error in logs?

Even i'm seeing the same error in logs but my production instance is running. How can we resolve the error?

 

Regards,

AnnWorley
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
October 12, 2017

Hi Shashidar,

Which error are you seeing in which log? We discussed several errors on this thread. One was for displaying the error page, one was not being able to ping the synchrony proxy and one was concerned with database character set.

It may be better if you post a separate question so we can keep your issue distinct from this case.

Thanks,

Ann

0 votes
Ezequiel September 22, 2017

Hi Ann,

I was unable to paste the log, so here's a link to it:

 

https://www.dropbox.com/s/t0k7gyvmdqf9ywv/atlassian-confluence.log?dl=0

 
The weirdest thing I see is:


2017-09-21 14:38:49,623 WARN [ListenableFutureAdapter-thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProxyMonitor] pollHealthcheck Could not ping the synchrony-proxy [http://127.0.0.1:8090/synchrony-proxy/healthcheck]: {}
java.net.SocketTimeoutException: Read timed out

AnnWorley
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
September 22, 2017

Thanks very much for the log, it includes the whole startup, as I hoped.

I was surprised to see this at the end of the log:

2017-09-21 14:39:12,007 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] init Confluence is ready to serve

That usually indicates that Confluence is available in the browser, although what is displayed there can turn out to be an error message.

Synchrony Proxy

Like you, I noticed the long stack trace related to the Synchrony health check. My expectation from that error would be issues with collaborative editing. For a useful diagram of how Synchrony works from a network perspective, please see: Administering Collaborative Editing under Proxies.

Non-recommended character set

I was concerned to see this in the log right after start-up, as I have seen Confluence fail to start as a result of character set issues:

2017-09-21 14:37:42,635 WARN [localhost-startStop-1] [confluence.setup.dbcheck.MySQLChecker] checkCharacterSetAndCollation Your database uses non-recommended character set: latin1. See https://confluence.atlassian.com/x/IrYC

Please follow this guide to make sure the right character set is used in the Confluence database: How to Fix the Collation and Character Set of a MySQL Database

0 votes
Ezequiel September 20, 2017

Thanks Ann! Will check and let you know.

0 votes
AnnWorley
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
September 20, 2017

Hi Ezequiel,

Please let me know more about when the instance stopped working. Did it slow down so that you restarted it and it didn't come back up? Or was the service stopped before it was restarted?

It looks like the log entry you posted is from the logs subdirectory of the Confluence installation directory. (It is often called catalina.out.) That log is for the underlying web server and will let us know if there are Tomcat configuration or other basic issues. The error you posted seems to indicate that Tomcat cannot display an error page, which is probably not the root cause of the outage.

In the Confluence home directory there is another logs subdirectory, which contains the atlassian-confluence.log. In that log, please search for Starting Confluence and then scroll down to any errors that Confluence reports during the startup. 

If the errors aren't informative enough, please post them on this thread. If I am not familiar with them I will be happy to research them and let you know my findings.

Thanks,

Ann

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events