nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Confluence is not accessible periodically

Moses Thomas_08kyb0

Dear Atlassian,

I would have to ask, as i am experiencing a weird behavior, on and on we are have this problem that Confluence 7.1.0 worked for some time and then stops working periodically

 2020-02-10 13:56:34,794 WARN [http-nio-8011-exec-118] [atlassian.confluence.util.ReadOnlyLicenseCalculator] getNumberOfLicensedUsers Licensing calculation took more than PT2S
2020-02-10 13:56:36,833 WARN [http-nio-8011-exec-118] [atlassian.confluence.util.ReadOnlyLicenseCalculator] getNumberOfLicensedUsers Licensing calculation took more than PT2S
2020-02-10 13:57:15,640 WARN [pool-25-thread-1] [atlassian.confluence.util.ReadOnlyLicenseCalculator] getNumberOfLicensedUsers Licensing calculation took more than PT2S
2020-02-10 13:59:50,204 ERROR [Catalina-utility-1] [engine.jdbc.spi.SqlExceptionHelper] logExceptions Cannot get a connection, general error
2020-02-10 13:59:50,402 WARN [Catalina-utility-1] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.sal.api.events.SessionDestroyedEvent@3c1bf309 took 175066 ms (warning threshold is 5000 ms)
2020-02-10 13:59:50,404 WARN [Catalina-utility-1] [internal.core.service.VCacheLock] lockWithTimeout Interrupted whilst waiting for a lock on cache:
2020-02-10 13:59:50,405 WARN [Catalina-utility-1] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$11 Failed to read entry from cache 'com.atlassian.bandana.BandanaPersister': Failed due to UNCLASSIFIED_FAILURE

I have performed upgrade for an existing application since December 2019 and it has been working smoothly until now. I am wondering, when i downloaded the installation file atlassian-confluence-7.1.0.tar.gz and extracted the file, according to the documentation here https://confluence.atlassian.com/doc/configuring-a-datasource-connection-937166084.html in the web.xml the below element was not present in this file and we have never added this element and currently this element is not present and we have been using Confluence over 8yrs, the Mysql database server is self hosted.

Now my question is could this be the reason why i am having break in accessing Confluence within various intervals? or is this a confluence bug? could some one please assist, this is critical issue for our production instance, and affects the quality of service.

web.xml element

4. Configure the Confluence web application
Configure Confluence to use this datasource:
Edit <CONFLUENCE_INSTALLATION>/confluence/WEB-INF/web.xml.
Insert the following element just before </web-app> near the end of the file:
<resource-ref> <description>Connection Pool</description> <res-ref-name>jdbc/confluence</res-ref-name> <res-type>javax.sql.DataSource</res-type> <res-auth>Container</res-auth>

Is the warning in logs, causing this issue ? I doubt so, or maybe some kind of bug, currently i am not able to identify the root cause from the logs

Some time such kind of errors occurred

2020-02-03 13:28:10,047 WARN [http-nio-8011-exec-111] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1580732802678, threadId=381, threadName='http-nio-8011-exec-111', userId='<unknown>', type='web-request', summary='/label/GPPPO/application+auto+cloud+engineering_team+extended_profile+fog+kpi+maxwell10+parameters+pre+predev+profile+redirection+sla+tool'}

Confluence asynchronous events(lots) are generated faster than Confluence can process them this queue is backed up

Ticket regarding this issue reported to Atlassian https://jira.atlassian.com/browse/CONFSERVER-55028

2020-02-10 16:32:22,979 WARN [http-nio-8011-exec-184] [atlassian.confluence.event.MonitorableCallerRunsPolicy] rejectedExecution Queue Overflow happened: Runnable: [Invoker: SingleParameterMethodListenerInvoker{method=public void com.atlassian.pocketknife.internal.querydsl.cache.PKQCacheClearerImpl.onClearCache(java.lang.Object), listener=com.atlassian.pocketknife.internal.querydsl.cache.PKQCacheClearerImpl@5bbe2b61} (timed); Event: HttpRequestStatsEvent{requestCorrelationId=bab0b2eb69ad6ecb, url=label.maxwell+penguin+sw.%, key=Optional.empty, reqTime=15613, reqStartTime=1581348722407, dbReqTimesInMicros=, dbReqFinishTimes=, requestUserTime=80, requestCpuTime=84, requestGCDuration=125, requestGCCount=5, timingEventKeys=labelsDecoratorStart,labelsDecoratorFinished,sitemeshDecoratorApplyStarted,mainDecoratorStart,confluenceResourceTagsStart,confluenceResourceTagsEnd,mainDecoratorBodyStart,commonsHeaderStart,commonsHeaderFinished,mainDecoratorContentDivStart,mainDecoratorContentDivFinished,mainDecoratorFinished,sitemeshDecoratorApplyFinished,serverRenderEnd, timingEventMillis=15451,15451,15472,15474,15481,15507,15508,15508,15521,15521,15521,15523,15523,15613}]; Executor: [java.util.concurrent.ThreadPoolExecutor@24607637[Running, pool size = 4, active threads = 4, queued tasks = 384, completed tasks = 732]]

I tried the workaround stated here but still no success still fighting with the issue

Confluence Bug: Ticket https://jira.atlassian.com/browse/CONFSERVER-40860

Accepted answers

Moses Thomas_08kyb0

Interesting, that Confluence is vulnerable this way.

Well after deep diving, it occurs that we were attack(since we had public spaces ) and we block this user IP address in Fire wall
One of my colleague also was able to bring tomcat down with his weird JavaScript so we disable this script.
Now Confluence has been running smooth for 2days now.

I have embark on implementing a deeper monitoring for Confluence server, especially tomcat.

All comments

JP _AC Bielefeld Leader_

Hi,

we're using Confluence with a datasource connection since 5 years w/o any problems. You might want to check the max. # of connections on the MySQL side and on the datasource configuration. These numers should match or at least there should be more max. connections on the server side than defined by the data source.

Did you monitor the connection usage on the MySQL server side?

Best

Moses Thomas_08kyb0

Hi,

Thank you for your reply,

Well from mysql server side the max_connection is 151 ( for both Jira and Confluence so quite OK)

and in Confluence has max_total 60 in the data source in configuration.

After monitoring this was the Statistics

Aborted_connects 126
Connections = 370129
Max_used_connections = 109
Threads connected =29

So I don't think this is the root cause, I will keep deep digging to find the root cause

Best regards,

Moses

JP _AC Bielefeld Leader_

Hi Moses,

you mentioned in your first post that you added the Connection Pool fragment to the web.xml of Confluence.

Did you also modify the server.xml of the Tomcat server and the confluence.cfg.xml as described in

https://confluence.atlassian.com/doc/configuring-a-datasource-connection-937166084.html

You might want to compare your setting with the settings we're using in our prod enviroment:

%CONFLUENCE_HOME%/conf/context.xml

<?xml version='1.0' encoding='utf-8'?>
<!-- The contents of this file will be loaded for each web application -->
<Context>

<!-- Default set of monitored resources. If one of these changes, the -->
<!-- web application will be reloaded. -->
<WatchedResource>WEB-INF/web.xml</WatchedResource>
<WatchedResource>WEB-INF/tomcat-web.xml</WatchedResource>
<WatchedResource>${catalina.base}/conf/web.xml</WatchedResource>

<!-- Uncomment this to disable session persistence across Tomcat restarts -->
<!--
<Manager pathname="" />
-->

<!-- Uncomment this to enable Comet connection tacking (provides events
on session expiration as well as webapp lifecycle) -->
<!--
<Valve className="org.apache.catalina.valves.CometConnectionManagerValve" />
-->

<ResourceLink type="javax.sql.DataSource"
name="jdbc/confluence"
global="jdbc/confluence" />

<ResourceLink type="javax.mail.Session"
name="mail/mailsession"
global="mail/mailsession" />

</Context>

%CONFLUENCE_HOME%/conf/server.xml

<Server port="8000" shutdown="SECRET">
<!-- Allow JMX access thru local firewall using defined ports -->
<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener"
rmiRegistryPortPlatform="SECRET" rmiServerPortPlatform="SECRET" />

<!-- Global JNDI resources
Documentation at /docs/jndi-resources-howto.html
-->
<GlobalNamingResources>

<Resource name="jdbc/confluence" auth="Container" type="javax.sql.DataSource"
username="SECRET"
password="SECRET"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://SECRET/confluence?sessionVariables=storage_engine%3DInnoDB&amp;useUnicode=true&amp;characterEncoding=utf8" 
defaultTransactionIsolation="READ_COMMITTED"
validationQuery="SELECT 1"
initialSize="20"
maxTotal="400"
maxIdle="20"
minIdle="0" 
maxWaitMillis="-1"
timeBetweenEvictionRunsMillis="30000"
minEvictableIdleTimeMillis="60000"
testOnBorrow="true"
removeAbandonedOnBorrow="false"
removeAbandonedOnMaintenance="false"
removeAbandonedTimeout="120"
logAbandoned="true"
/>

</GlobalNamingResources>

<Service name="Tomcat-Standalone">

<Connector port="8009"
minSpareThreads="10"
maxThreads="150"
address="SECRET" 
protocol="AJP/1.3"
redirectPort="8444" 
enableLookups="false"
URIEncoding="UTF-8"
maxPostSize="-1"
packetSize="65536"/>

<Engine name="Standalone" defaultHost="SECRET" jvmRoute="node0">

<Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase"/>

<Host name="SECRET" appBase="webapps" unpackWARs="true" autoDeploy="false" startStopThreads="4">

<Context path="/confluence" docBase="../confluence" reloadable="false" useHttpOnly="true">
<Manager pathname="" />

<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60" />
</Context>

<Context path="${confluence.context.path}/synchrony-proxy" docBase="../synchrony-proxy" debug="0" reloadable="false" useHttpOnly="true">
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60" />
</Context> 
</Host>

</Engine>

</Service>
</Server>

%CONFLUENCE_DATA%/confluence.cfg.xml

<?xml version="1.0" encoding="UTF-8"?>

<confluence-configuration>
<setupStep>complete</setupStep>
<setupType>custom</setupType>
<buildNumber>8100</buildNumber>
<properties>
...
<property name="hibernate.connection.datasource">java:comp/env/jdbc/confluence</property>
<property name="hibernate.database.lower_non_ascii_supported">true</property>
<property name="hibernate.dialect">com.atlassian.confluence.impl.hibernate.dialect.MySQLDialect</property>
<property name="hibernate.setup">true</property>
...
</properties>
</confluence-configuration>

This configuration makes use of the Tomcat Database Pool handling, which is much easier to monitor by JMX. If you only changed the web.xml, you will run into your mentioned problems.

Could you post the relevant section of your server.xml?

Best

JP _AC Bielefeld Leader_

Oh, I forgot to add the modification to the %CONFLUENCE_HOME%/confluence/WEB_INF/web.xml right before the end of the file:

...

 <resource-ref>
<description>Connection Pool</description>
<res-ref-name>jdbc/confluence</res-ref-name>
<res-type>javax.sql.DataSource</res-type>
<res-auth>Container</res-auth>
<res-sharing-scope>Shareable</res-sharing-scope>
</resource-ref>

</web-app>

Moses Thomas_08kyb0

Interesting, that Confluence is vulnerable this way.

Well after deep diving, it occurs that we were attack(since we had public spaces ) and we block this user IP address in Fire wall
One of my colleague also was able to bring tomcat down with his weird JavaScript so we disable this script.
Now Confluence has been running smooth for 2days now.

I have embark on implementing a deeper monitoring for Confluence server, especially tomcat.

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

confluence

confluence-server

atl-gruland

performance

bug