Dear Atlassian,
I would have to ask, as i am experiencing a weird behavior, on and on we are have this problem that Confluence 7.1.0 worked for some time and then stops working periodically
2020-02-10 13:56:34,794 WARN [http-nio-8011-exec-118] [atlassian.confluence.util.ReadOnlyLicenseCalculator] getNumberOfLicensedUsers Licensing calculation took more than PT2S
2020-02-10 13:56:36,833 WARN [http-nio-8011-exec-118] [atlassian.confluence.util.ReadOnlyLicenseCalculator] getNumberOfLicensedUsers Licensing calculation took more than PT2S
2020-02-10 13:57:15,640 WARN [pool-25-thread-1] [atlassian.confluence.util.ReadOnlyLicenseCalculator] getNumberOfLicensedUsers Licensing calculation took more than PT2S
2020-02-10 13:59:50,204 ERROR [Catalina-utility-1] [engine.jdbc.spi.SqlExceptionHelper] logExceptions Cannot get a connection, general error
2020-02-10 13:59:50,402 WARN [Catalina-utility-1] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.sal.api.events.SessionDestroyedEvent@3c1bf309 took 175066 ms (warning threshold is 5000 ms)
2020-02-10 13:59:50,404 WARN [Catalina-utility-1] [internal.core.service.VCacheLock] lockWithTimeout Interrupted whilst waiting for a lock on cache:
2020-02-10 13:59:50,405 WARN [Catalina-utility-1] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$11 Failed to read entry from cache 'com.atlassian.bandana.BandanaPersister': Failed due to UNCLASSIFIED_FAILURE
I have performed upgrade for an existing application since December 2019 and it has been working smoothly until now. I am wondering, when i downloaded the installation file atlassian-confluence-7.1.0.tar.gz and extracted the file, according to the documentation here https://confluence.atlassian.com/doc/configuring-a-datasource-connection-937166084.html in the web.xml the below element was not present in this file and we have never added this element and currently this element is not present and we have been using Confluence over 8yrs, the Mysql database server is self hosted.
web.xml element
4. Configure the Confluence web application
Configure Confluence to use this datasource:
Edit
<CONFLUENCE_INSTALLATION>/confluence/WEB-INF/web.xml
.Insert the following element just before
</web-app>
near the end of the file:
<resource-ref> <description>Connection Pool</description> <res-ref-name>jdbc/confluence</res-ref-name> <res-type>javax.sql.DataSource</res-type> <res-auth>Container</res-auth>
Some time such kind of errors occurred
2020-02-03 13:28:10,047 WARN [http-nio-8011-exec-111] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1580732802678, threadId=381, threadName='http-nio-8011-exec-111', userId='<unknown>', type='web-request', summary='/label/GPPPO/application+auto+cloud+engineering_team+extended_profile+fog+kpi+maxwell10+parameters+pre+predev+profile+redirection+sla+tool'}
Confluence asynchronous events(lots) are generated faster than Confluence can process them this queue is backed up
Ticket regarding this issue reported to Atlassian https://jira.atlassian.com/browse/CONFSERVER-55028
2020-02-10 16:32:22,979 WARN [http-nio-8011-exec-184] [atlassian.confluence.event.MonitorableCallerRunsPolicy] rejectedExecution Queue Overflow happened: Runnable: [Invoker: SingleParameterMethodListenerInvoker{method=public void com.atlassian.pocketknife.internal.querydsl.cache.PKQCacheClearerImpl.onClearCache(java.lang.Object), listener=com.atlassian.pocketknife.internal.querydsl.cache.PKQCacheClearerImpl@5bbe2b61} (timed); Event: HttpRequestStatsEvent{requestCorrelationId=bab0b2eb69ad6ecb, url=label.maxwell+penguin+sw.%, key=Optional.empty, reqTime=15613, reqStartTime=1581348722407, dbReqTimesInMicros=, dbReqFinishTimes=, requestUserTime=80, requestCpuTime=84, requestGCDuration=125, requestGCCount=5, timingEventKeys=labelsDecoratorStart,labelsDecoratorFinished,sitemeshDecoratorApplyStarted,mainDecoratorStart,confluenceResourceTagsStart,confluenceResourceTagsEnd,mainDecoratorBodyStart,commonsHeaderStart,commonsHeaderFinished,mainDecoratorContentDivStart,mainDecoratorContentDivFinished,mainDecoratorFinished,sitemeshDecoratorApplyFinished,serverRenderEnd, timingEventMillis=15451,15451,15472,15474,15481,15507,15508,15508,15521,15521,15521,15523,15523,15613}]; Executor: [java.util.concurrent.ThreadPoolExecutor@24607637[Running, pool size = 4, active threads = 4, queued tasks = 384, completed tasks = 732]]
I tried the workaround stated here but still no success still fighting with the issue
Confluence Bug: Ticket https://jira.atlassian.com/browse/CONFSERVER-40860
Interesting, that Confluence is vulnerable this way.
Well after deep diving, it occurs that we were attack(since we had public spaces ) and we block this user IP address in Fire wall
One of my colleague also was able to bring tomcat down with his weird JavaScript so we disable this script.
Now Confluence has been running smooth for 2days now.
I have embark on implementing a deeper monitoring for Confluence server, especially tomcat.
Hi,
we're using Confluence with a datasource connection since 5 years w/o any problems. You might want to check the max. # of connections on the MySQL side and on the datasource configuration. These numers should match or at least there should be more max. connections on the server side than defined by the data source.
Did you monitor the connection usage on the MySQL server side?
Best
JP
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi,
Thank you for your reply,
Well from mysql server side the max_connection is 151 ( for both Jira and Confluence so quite OK)
and in Confluence has max_total 60 in the data source in configuration.
After monitoring this was the Statistics
Aborted_connects 126
Connections = 370129
Max_used_connections = 109
Threads connected =29
So I don't think this is the root cause, I will keep deep digging to find the root cause
Best regards,
Moses
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Moses,
you mentioned in your first post that you added the Connection Pool fragment to the web.xml of Confluence.
Did you also modify the server.xml of the Tomcat server and the confluence.cfg.xml as described in
https://confluence.atlassian.com/doc/configuring-a-datasource-connection-937166084.html
You might want to compare your setting with the settings we're using in our prod enviroment:
%CONFLUENCE_HOME%/conf/context.xml
<?xml version='1.0' encoding='utf-8'?>
<!-- The contents of this file will be loaded for each web application -->
<Context>
<!-- Default set of monitored resources. If one of these changes, the -->
<!-- web application will be reloaded. -->
<WatchedResource>WEB-INF/web.xml</WatchedResource>
<WatchedResource>WEB-INF/tomcat-web.xml</WatchedResource>
<WatchedResource>${catalina.base}/conf/web.xml</WatchedResource>
<!-- Uncomment this to disable session persistence across Tomcat restarts -->
<!--
<Manager pathname="" />
-->
<!-- Uncomment this to enable Comet connection tacking (provides events
on session expiration as well as webapp lifecycle) -->
<!--
<Valve className="org.apache.catalina.valves.CometConnectionManagerValve" />
-->
<ResourceLink type="javax.sql.DataSource"
name="jdbc/confluence"
global="jdbc/confluence" />
<ResourceLink type="javax.mail.Session"
name="mail/mailsession"
global="mail/mailsession" />
</Context>
%CONFLUENCE_HOME%/conf/server.xml
<Server port="8000" shutdown="SECRET">
<!-- Allow JMX access thru local firewall using defined ports -->
<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener"
rmiRegistryPortPlatform="SECRET" rmiServerPortPlatform="SECRET" />
<!-- Global JNDI resources
Documentation at /docs/jndi-resources-howto.html
-->
<GlobalNamingResources>
<Resource name="jdbc/confluence" auth="Container" type="javax.sql.DataSource"
username="SECRET"
password="SECRET"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://SECRET/confluence?sessionVariables=storage_engine%3DInnoDB&useUnicode=true&characterEncoding=utf8"
defaultTransactionIsolation="READ_COMMITTED"
validationQuery="SELECT 1"
initialSize="20"
maxTotal="400"
maxIdle="20"
minIdle="0"
maxWaitMillis="-1"
timeBetweenEvictionRunsMillis="30000"
minEvictableIdleTimeMillis="60000"
testOnBorrow="true"
removeAbandonedOnBorrow="false"
removeAbandonedOnMaintenance="false"
removeAbandonedTimeout="120"
logAbandoned="true"
/>
</GlobalNamingResources>
<Service name="Tomcat-Standalone">
<Connector port="8009"
minSpareThreads="10"
maxThreads="150"
address="SECRET"
protocol="AJP/1.3"
redirectPort="8444"
enableLookups="false"
URIEncoding="UTF-8"
maxPostSize="-1"
packetSize="65536"/>
<Engine name="Standalone" defaultHost="SECRET" jvmRoute="node0">
<Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase"/>
<Host name="SECRET" appBase="webapps" unpackWARs="true" autoDeploy="false" startStopThreads="4">
<Context path="/confluence" docBase="../confluence" reloadable="false" useHttpOnly="true">
<Manager pathname="" />
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60" />
</Context>
<Context path="${confluence.context.path}/synchrony-proxy" docBase="../synchrony-proxy" debug="0" reloadable="false" useHttpOnly="true">
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60" />
</Context>
</Host>
</Engine>
</Service>
</Server>
%CONFLUENCE_DATA%/confluence.cfg.xml
<?xml version="1.0" encoding="UTF-8"?>
<confluence-configuration>
<setupStep>complete</setupStep>
<setupType>custom</setupType>
<buildNumber>8100</buildNumber>
<properties>
...
<property name="hibernate.connection.datasource">java:comp/env/jdbc/confluence</property>
<property name="hibernate.database.lower_non_ascii_supported">true</property>
<property name="hibernate.dialect">com.atlassian.confluence.impl.hibernate.dialect.MySQLDialect</property>
<property name="hibernate.setup">true</property>
...
</properties>
</confluence-configuration>
This configuration makes use of the Tomcat Database Pool handling, which is much easier to monitor by JMX. If you only changed the web.xml, you will run into your mentioned problems.
Could you post the relevant section of your server.xml?
Best
JP
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Oh, I forgot to add the modification to the %CONFLUENCE_HOME%/confluence/WEB_INF/web.xml right before the end of the file:
...
<resource-ref>
<description>Connection Pool</description>
<res-ref-name>jdbc/confluence</res-ref-name>
<res-type>javax.sql.DataSource</res-type>
<res-auth>Container</res-auth>
<res-sharing-scope>Shareable</res-sharing-scope>
</resource-ref>
</web-app>
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
@JP _AC Bielefeld Leader_ Thanks, I have gone through your post, now suppose all config files are Okay, and I DONT HAVE the below element what do i stand to face ? cos according to my previous explanation Confluence has been working without the below element in %CONFLUENCE_HOME%/confluence/WEB_INF/web.xml until lately so i wounder if this could be the cause? I doubt so.
<resource-ref>
<description>Connection Pool</description>
<res-ref-name>jdbc/confluence</res-ref-name>
<res-type>javax.sql.DataSource</res-type>
<res-auth>Container</res-auth>
<res-sharing-scope>Shareable</res-sharing-scope>
</resource-ref>
</web-app>
Kind regards,
Mo
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi,
database connections in a Tomcat application server environment can be handled by the application or the Tomcat server itself as a shareable resource. Configuring a shareable resource has the advantage for easier monitoring (JMX) and having a central configuration file (server.xml). Imaging having several applications on a Tomcat server & the DB server changes. You have to modify each application. With a shared resource you just need to change one file.
The fragment in the web.xml tells Confluence to use a shared SQL Datasource named jdbc/confluence defined in the container configuration (server.xml).
Maybe you haven't been using a datasource connection before, but just a also working application JDBC connection.
Great you solved it.
Best
JP
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.