Confluence is not accessible periodically

Moses Thomas
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 10, 2020

Dear Atlassian,

I would  have to ask, as i am  experiencing a weird behavior, on and on we are have this  problem that Confluence 7.1.0 worked for some time and then stops working periodically

 2020-02-10 13:56:34,794 WARN [http-nio-8011-exec-118] [atlassian.confluence.util.ReadOnlyLicenseCalculator] getNumberOfLicensedUsers Licensing calculation took more than PT2S
2020-02-10 13:56:36,833 WARN [http-nio-8011-exec-118] [atlassian.confluence.util.ReadOnlyLicenseCalculator] getNumberOfLicensedUsers Licensing calculation took more than PT2S
2020-02-10 13:57:15,640 WARN [pool-25-thread-1] [atlassian.confluence.util.ReadOnlyLicenseCalculator] getNumberOfLicensedUsers Licensing calculation took more than PT2S
2020-02-10 13:59:50,204 ERROR [Catalina-utility-1] [engine.jdbc.spi.SqlExceptionHelper] logExceptions Cannot get a connection, general error
2020-02-10 13:59:50,402 WARN [Catalina-utility-1] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.sal.api.events.SessionDestroyedEvent@3c1bf309 took 175066 ms (warning threshold is 5000 ms)
2020-02-10 13:59:50,404 WARN [Catalina-utility-1] [internal.core.service.VCacheLock] lockWithTimeout Interrupted whilst waiting for a lock on cache:
2020-02-10 13:59:50,405 WARN [Catalina-utility-1] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$11 Failed to read entry from cache 'com.atlassian.bandana.BandanaPersister': Failed due to UNCLASSIFIED_FAILURE

 I have performed upgrade for an  existing application since December 2019 and it has been  working smoothly until  now. I am wondering, when i  downloaded  the installation file atlassian-confluence-7.1.0.tar.gz and extracted the file,  according to the documentation here  https://confluence.atlassian.com/doc/configuring-a-datasource-connection-937166084.html in the  web.xml  the below element was not present in this file and we have never  added this  element and currently this element is not present  and we have been using Confluence over 8yrs, the Mysql database server is self hosted. 

  • Now  my question is could this be the reason why i am having break in accessing Confluence within various  intervals? or is this a confluence  bug?  could some one please assist, this is critical issue for our production instance, and affects the quality of service.

   web.xml element

4. Configure the Confluence web application

Configure Confluence to use this datasource:

  1. Edit <CONFLUENCE_INSTALLATION>/confluence/WEB-INF/web.xml.

  2. Insert the following element just before </web-app> near the end of the file: 

<resource-ref> <description>Connection Pool</description> <res-ref-name>jdbc/confluence</res-ref-name> <res-type>javax.sql.DataSource</res-type> <res-auth>Container</res-auth>

 

  • Is the warning in logs, causing this issue ?  I doubt so, or maybe some kind  of bug, currently i am  not able to identify the root cause  from the logs

Some time such kind of errors  occurred

2020-02-03 13:28:10,047 WARN [http-nio-8011-exec-111] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1580732802678, threadId=381, threadName='http-nio-8011-exec-111', userId='<unknown>', type='web-request', summary='/label/GPPPO/application+auto+cloud+engineering_team+extended_profile+fog+kpi+maxwell10+parameters+pre+predev+profile+redirection+sla+tool'}

Confluence asynchronous events(lots) are generated faster than Confluence can process them this queue is backed up  

Ticket regarding this issue reported to Atlassian https://jira.atlassian.com/browse/CONFSERVER-55028

2020-02-10 16:32:22,979 WARN [http-nio-8011-exec-184] [atlassian.confluence.event.MonitorableCallerRunsPolicy] rejectedExecution Queue Overflow happened: Runnable: [Invoker: SingleParameterMethodListenerInvoker{method=public void com.atlassian.pocketknife.internal.querydsl.cache.PKQCacheClearerImpl.onClearCache(java.lang.Object), listener=com.atlassian.pocketknife.internal.querydsl.cache.PKQCacheClearerImpl@5bbe2b61} (timed); Event: HttpRequestStatsEvent{requestCorrelationId=bab0b2eb69ad6ecb, url=label.maxwell+penguin+sw.%, key=Optional.empty, reqTime=15613, reqStartTime=1581348722407, dbReqTimesInMicros=, dbReqFinishTimes=, requestUserTime=80, requestCpuTime=84, requestGCDuration=125, requestGCCount=5, timingEventKeys=labelsDecoratorStart,labelsDecoratorFinished,sitemeshDecoratorApplyStarted,mainDecoratorStart,confluenceResourceTagsStart,confluenceResourceTagsEnd,mainDecoratorBodyStart,commonsHeaderStart,commonsHeaderFinished,mainDecoratorContentDivStart,mainDecoratorContentDivFinished,mainDecoratorFinished,sitemeshDecoratorApplyFinished,serverRenderEnd, timingEventMillis=15451,15451,15472,15474,15481,15507,15508,15508,15521,15521,15521,15523,15523,15613}]; Executor: [java.util.concurrent.ThreadPoolExecutor@24607637[Running, pool size = 4, active threads = 4, queued tasks = 384, completed tasks = 732]]

I tried the workaround  stated here  but still no success  still fighting with the issue

Confluence Bug: Ticket https://jira.atlassian.com/browse/CONFSERVER-40860

2 answers

1 accepted

0 votes
Answer accepted
Moses Thomas
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 14, 2020

Interesting, that Confluence is vulnerable  this  way.

 

  • Well after deep diving, it  occurs  that we were attack(since we had  public spaces ) and we  block this user IP address in Fire wall
  •  One of my colleague also was able to bring tomcat down with his weird JavaScript  so we disable this  script.
  • Now Confluence has been  running smooth for 2days now.

 

I have embark on implementing a deeper monitoring   for Confluence server, especially tomcat.

0 votes
JP _AC Bielefeld Leader_
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
February 10, 2020

Hi,

we're using Confluence with a datasource connection since 5 years w/o any problems. You might want to check the max. # of connections on the MySQL side and on the datasource configuration. These numers should match or at least there should be more max. connections on the server side than defined by the data source.

Did you monitor the connection usage on the MySQL server side?

Best

JP

Moses Thomas
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 12, 2020

Hi,

Thank you for your reply,

Well from mysql server side the max_connection is 151 ( for both Jira and Confluence so quite OK)

and in Confluence has max_total 60 in the data source in  configuration.

After monitoring this was the Statistics

Aborted_connects 126

Connections = 370129

Max_used_connections = 109

Threads connected =29

So I don't think this is the root cause, I will keep deep digging to find the root cause

 

Best regards,

Moses

JP _AC Bielefeld Leader_
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
February 12, 2020

Hi Moses,

you mentioned in your first post that you added the Connection Pool fragment to the web.xml of Confluence.

Did you also modify the server.xml of the Tomcat server and the confluence.cfg.xml as described in 

https://confluence.atlassian.com/doc/configuring-a-datasource-connection-937166084.html

You might want to compare your setting with the settings we're using in our prod enviroment:

%CONFLUENCE_HOME%/conf/context.xml

<?xml version='1.0' encoding='utf-8'?>
<!-- The contents of this file will be loaded for each web application -->
<Context>

<!-- Default set of monitored resources. If one of these changes, the -->
<!-- web application will be reloaded. -->
<WatchedResource>WEB-INF/web.xml</WatchedResource>
<WatchedResource>WEB-INF/tomcat-web.xml</WatchedResource>
<WatchedResource>${catalina.base}/conf/web.xml</WatchedResource>

<!-- Uncomment this to disable session persistence across Tomcat restarts -->
<!--
<Manager pathname="" />
-->

<!-- Uncomment this to enable Comet connection tacking (provides events
on session expiration as well as webapp lifecycle) -->
<!--
<Valve className="org.apache.catalina.valves.CometConnectionManagerValve" />
-->

<ResourceLink type="javax.sql.DataSource"
name="jdbc/confluence"
global="jdbc/confluence" />

<ResourceLink type="javax.mail.Session"
name="mail/mailsession"
global="mail/mailsession" />

</Context>

%CONFLUENCE_HOME%/conf/server.xml

<Server port="8000" shutdown="SECRET">
<!-- Allow JMX access thru local firewall using defined ports -->
<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener"
rmiRegistryPortPlatform="SECRET" rmiServerPortPlatform="SECRET" />

<!-- Global JNDI resources
Documentation at /docs/jndi-resources-howto.html
-->
<GlobalNamingResources>

<Resource name="jdbc/confluence" auth="Container" type="javax.sql.DataSource"
username="SECRET"
password="SECRET"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://SECRET/confluence?sessionVariables=storage_engine%3DInnoDB&amp;useUnicode=true&amp;characterEncoding=utf8"
defaultTransactionIsolation="READ_COMMITTED"
validationQuery="SELECT 1"
initialSize="20"
maxTotal="400"
maxIdle="20"
minIdle="0"
maxWaitMillis="-1"
timeBetweenEvictionRunsMillis="30000"
minEvictableIdleTimeMillis="60000"
testOnBorrow="true"
removeAbandonedOnBorrow="false"
removeAbandonedOnMaintenance="false"
removeAbandonedTimeout="120"
logAbandoned="true"
/>

</GlobalNamingResources>

<Service name="Tomcat-Standalone">

<Connector port="8009"
minSpareThreads="10"
maxThreads="150"
address="SECRET"
protocol="AJP/1.3"
redirectPort="8444"
enableLookups="false"
URIEncoding="UTF-8"
maxPostSize="-1"
packetSize="65536"/>

<Engine name="Standalone" defaultHost="SECRET" jvmRoute="node0">

<Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase"/>

<Host name="SECRET" appBase="webapps" unpackWARs="true" autoDeploy="false" startStopThreads="4">

<Context path="/confluence" docBase="../confluence" reloadable="false" useHttpOnly="true">
<Manager pathname="" />

<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60" />
</Context>

<Context path="${confluence.context.path}/synchrony-proxy" docBase="../synchrony-proxy" debug="0" reloadable="false" useHttpOnly="true">
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60" />
</Context>
</Host>

</Engine>

</Service>
</Server>

 

%CONFLUENCE_DATA%/confluence.cfg.xml

<?xml version="1.0" encoding="UTF-8"?>

<confluence-configuration>
<setupStep>complete</setupStep>
<setupType>custom</setupType>
<buildNumber>8100</buildNumber>
<properties>
...
<property name="hibernate.connection.datasource">java:comp/env/jdbc/confluence</property>
<property name="hibernate.database.lower_non_ascii_supported">true</property>
<property name="hibernate.dialect">com.atlassian.confluence.impl.hibernate.dialect.MySQLDialect</property>
<property name="hibernate.setup">true</property>
...
</properties>
</confluence-configuration>

 This configuration makes use of the Tomcat Database Pool handling, which is much easier to monitor by JMX. If you only changed the web.xml, you will run into your mentioned problems.

Could you post the relevant section of your server.xml?

Best

JP

JP _AC Bielefeld Leader_
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
February 12, 2020

Oh, I forgot to add the modification to the %CONFLUENCE_HOME%/confluence/WEB_INF/web.xml right before the end of the file:

...

<resource-ref>
<description>Connection Pool</description>
<res-ref-name>jdbc/confluence</res-ref-name>
<res-type>javax.sql.DataSource</res-type>
<res-auth>Container</res-auth>
<res-sharing-scope>Shareable</res-sharing-scope>
</resource-ref>

</web-app>
Moses Thomas
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
February 12, 2020

@JP _AC Bielefeld Leader_  Thanks, I have gone through your post, now suppose all config files are Okay,   and  I DONT HAVE the below element  what do i stand to face ?  cos according to  my previous explanation Confluence has been  working without the below  element in  %CONFLUENCE_HOME%/confluence/WEB_INF/web.xml   until lately so i wounder if this could be the cause?  I doubt so.

 

<resource-ref>
<description>Connection Pool</description>
<res-ref-name>jdbc/confluence</res-ref-name>
<res-type>javax.sql.DataSource</res-type>
<res-auth>Container</res-auth>
<res-sharing-scope>Shareable</res-sharing-scope>
</resource-ref>

</web-app>

 Kind regards,

Mo

JP _AC Bielefeld Leader_
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
February 14, 2020

Hi,

database connections in a Tomcat application server environment can be handled by the application or the Tomcat server itself as a shareable resource. Configuring a shareable resource has the advantage for easier monitoring (JMX) and having a central configuration file (server.xml). Imaging having several applications on a Tomcat server & the DB server changes. You have to modify each application. With a shared resource you just need to change one file.

The fragment in the web.xml tells Confluence to use a shared SQL Datasource named jdbc/confluence defined in the container configuration (server.xml).

Maybe you haven't been using a datasource connection before, but just a also working application JDBC connection.

Great you solved it.

Best

JP

Like Moses Thomas likes this

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events