Failed to add rrd metrics for key: Error

Question

Hi After upgrading from version 7.6.4 to version 7.9.2, we are getting the below error. We are using jira data center, with 2 nodes (Aincrad is the name of one of the nodes.) We've also changed the jira.index.lock.waittime from 30000 to 90000 from jira-config.properties file, however the error remained. 2019-02-06 15:47:26,444 pool-29-thread-1 ERROR [c.o.s.r.diag.rrd.RrdManagerImpl] Failed to add rrd metrics for key: 20301 java.io.IOException: Could not obtain exclusive lock on file: /mnt/jira-db-shared/jira/scriptrunner/rrd/Aincrad/20301.rrd4j] after 3000 milliseconds at org.rrd4j.core.RrdSafeFileBackend.lockFile(RrdSafeFileBackend.java:65) at org.rrd4j.core.RrdSafeFileBackend.(RrdSafeFileBackend.java:33) at org.rrd4j.core.RrdSafeFileBackendFactory.open(RrdSafeFileBackendFactory.java:28) at org.rrd4j.core.RrdBackendFactory.open(RrdBackendFactory.java:434) at org.rrd4j.core.RrdDb.(RrdDb.java:270) at org.rrd4j.core.RrdDb.(RrdDb.java:258) ... 2 filtered at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83) at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:258) at com.onresolve.scriptrunner.runner.diag.rrd.RrdManagerImpl.createOrOpenDatabaseFile(RrdManagerImpl.groovy:181) at com.onresolve.scriptrunner.runner.diag.rrd.RrdManagerImpl$createOrOpenDatabaseFile.callCurrent(Unknown Source) at com.onresolve.scriptrunner.runner.diag.rrd.RrdManagerImpl.addSample(RrdManagerImpl.groovy:146) ... 2 filtered at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1225) at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnCurrentN(ScriptBytecodeAdapter.java:95) at com.onresolve.scriptrunner.runner.diag.rrd.RrdManagerImpl.this$dist$invoke$2(RrdManagerImpl.groovy) at com.onresolve.scriptrunner.runner.diag.rrd.RrdManagerImpl$RrdDataProcessor.methodMissing(RrdManagerImpl.groovy) ... 2 filtered Can someone advice regarding this error? Thanks

Joshua Yamdogo @ Adaptavist · Answer

Hi @Maria Bonello

Are you still having this issue? If you are, could you please a support ticket through our support portal here: https://marketplace.atlassian.com/apps/6820/scriptrunner-for-jira?hosting=server&tab=support

Regards,

Josh

Maria Bonello · Answer

Hi Andrew,

The ScriptRunner version is 5.4.48, which was updated few days ago from version 5.3.9. This is the only plugin that is seems to be used from the stacktrace.

Also, I went through the clusterlockstatus table and there isn't any node holding the cluster wide lock as all the records in the table has null datavalue in the locked_by_node field.

Andy Heinzer · Answer

It looks like you might be seeing problems such as those described in Jira Data Center Functionalities Loss Due to Cluster Wide Lock.  It explains how one node in a data center cluster can create a lock globally, and then explains some ways to correct that.

However I'm not familiar with the code packages listed in your stacktrace at the beginning:

at org.rrd4j.core.RrdSafeFileBackend.lockFile(RrdSafeFileBackend.java:65)
at org.rrd4j.core.RrdSafeFileBackend.<init>(RrdSafeFileBackend.java:33)
at org.rrd4j.core.RrdSafeFileBackendFactory.open(RrdSafeFileBackendFactory.java:28)
at org.rrd4j.core.RrdBackendFactory.open(RrdBackendFactory.java:434)

I see below these are some scriptrunner packages being used.  However my concern is over these rrd4j packages.  Perhaps the version of these packages has not been updated to correspond to the changes in Jira and Jira data center.

Could you share with us the plugins and the versions of those plugins you are using here with Jira Data Center 7.9.2?