Hello,
We are experiencing ongoing blocking session/locking issues on our JIRA instance. We generally get alerts on the database before we experience issues on the application. The impact to the application is that some users cannot perform particular actions (see below) and if the block is not cleared on the database level or the thread ended, the application will eventually stop responding. The issue has been isolated to a particular request/transition but we have found it difficult to reproduce the issue on our test/ QA server. The issue seems to occur during peak periods of the day.
Our DBA's are monitoring the database closely and do not see any other issues. The query that blocks is UPDATE OS_CURRENTSTEP.. I have attached some additional information in the database-info file.
I have been taking threaddumps and have been able to identify the exact request that is causing the block. The issue seems to be related to a particular transition in one of our workflows (attached workflow - transition is 461). This transition does have a custom post function that creates subtasks based on the information in the ticket. I suspect that this post function is in part a cause of the issue but we have not made any changes to the function for over a year and the frequency of the blocking issues have only recently increased. I have done a review of the code and I cannot find any glaringly major issues but I am continuing my investigation.
I understand that Atlassian cannot fully support our custom function and since our instance is quite large and complex it will be difficult to help but if you have any past experiences, ideas or advice how we can resolve this issue, I would greatly appreciatte any help.
Thanks,
Ram
Most probably, you run into this: https://jira.atlassian.com/browse/JRA-26227
I opened a test case here: https://jira.atlassian.com/browse/JRA-26172
Read about it, sounds very familiar, we had this output also.
Please note that the second link provides a workaround to the problem.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Radu,
Thanks for the reply! That bug actually describes our exact issue :-)
As you probably noted, we are also creating subtasks/issues within the post function.
I am going to go through your cases now but glad to hear that you have a workaround atleast.
I will let you know how I go.
Thankyou again - much appreciatte your time to post an answer!
Ram
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
|
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Example of what looks to be the blocking thread:
"TP-Processor14" daemon prio=10 tid=0x00002aaac0deb000 nid=0x1d17 runnable [0x00000000455b6000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at oracle.net.ns.Packet.receive(Packet.java:282)
at oracle.net.ns.DataPacket.receive(DataPacket.java:103)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:230)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:175)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:100)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:85)
.........removed the standard oracle stace trace part....
oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1062)
at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:102)
at org.ofbiz.core.entity.jdbc.SQLProcessor.executeUpdate(SQLProcessor.java:631)
at org.ofbiz.core.entity.GenericDAO.singleUpdate(GenericDAO.java:240)
at org.ofbiz.core.entity.GenericDAO.customUpdate(GenericDAO.java:190)
at org.ofbiz.core.entity.GenericDAO.update(GenericDAO.java:183)
at org.ofbiz.core.entity.GenericHelperDAO.store(GenericHelperDAO.java:207)
at org.ofbiz.core.entity.GenericDelegator.store(GenericDelegator.java:1406)
at org.ofbiz.core.entity.GenericDelegator.store(GenericDelegator.java:1386)
at com.opensymphony.workflow.spi.ofbiz.OfbizWorkflowStore.markFinished(OfbizWorkflowStore.java:231)
at com.opensymphony.workflow.AbstractWorkflow.createNewCurrentStep(AbstractWorkflow.java:1473)
at com.opensymphony.workflow.AbstractWorkflow.transitionWorkflow(AbstractWorkflow.java:1256)
at com.opensymphony.workflow.AbstractWorkflow.doAction(AbstractWorkflow.java:567)
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Example of a locking thread:
"TP-Processor165" daemon prio=10 tid=0x00002aaac0b98000 nid=0x4cb6 waiting on condition [0x000000004ed4d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006a7b932c8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at com.atlassian.util.concurrent.ManagedLocks$ManagedLockImpl.withLock(ManagedLocks.java:308)
at com.atlassian.util.concurrent.LockManagers$Manager.withLock(LockManagers.java:92)
at com.atlassian.jira.project.DefaultProjectManager$NextIdGenerator.getNextId(DefaultProjectManager.java:563)
at com.atlassian.jira.project.DefaultProjectManager.getNextId(DefaultProjectManager.java:129)
at com.atlassian.jira.project.AbstractProjectManager.getNextId(AbstractProjectManager.java:39)
at com.atlassian.jira.project.CachingProjectManager.getNextId(CachingProjectManager.java:87)
at com.atlassian.jira.workflow.function.issue.IssueCreateFunction.execute(IssueCreateFunction.java:51)
at com.opensymphony.workflow.AbstractWorkflow.executeFunction(AbstractWorkflow.java:869)
at com.opensymphony.workflow.AbstractWorkflow.transitionWorkflow(AbstractWorkflow.java:1265)
at com.opensymphony.workflow.AbstractWorkflow.initialize(AbstractWorkflow.java:618)
at com.atlassian.jira.workflow.SimpleWorkflowManager.createIssue(SimpleWorkflowManager.java:219)
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Database Statistics |
|
Issues | 379008 |
Projects | 470 |
Custom Fields | 741 |
Workflows | 343 |
Users | 5484 |
Groups | 466 |
JIRA Info |
|
Uptime | 4 days, 5 hours, 29 minutes, 38 seconds |
Edition | enterprise |
Version | 4.1.1 |
Build Number | 522 |
Build Date | Mon Apr 19 00:00:00 CEST 2010 |
Atlassian Partner | |
Installation Type | Standalone |
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
DB INFO:
OBJECT_NAME
--------------------------------------------------------------------------------
ROW_WAIT_OBJ# ROW_WAIT_FILE# ROW_WAIT_BLOCK# ROW_WAIT_ROW# DBMS_ROWID.ROWID_C
------------- -------------- --------------- ------------- ------------------
OS_CURRENTSTEP
82858 6 1228144 32 AAAUOqAAGAAEr1wAAg
SQL>
Notification Type: PROBLEM
Service: check_dbhealth
Host: GT Database
Address: 10.0.33.1
State: CRITICAL
Date/Time: Tue Oct 11 12:35:15 CEST 2011
Additional Info:
CRITICAL: blocking sessions found
long running: inst: 1 sid: 165 serial: 40174 user: JIRA elapsed: 620 sql_id: 1yu9j7r6xgvj6 sql: UPDATE OS_CURRENTSTEP SET ENTRY_ID=:1 , STEP_ID=:2 , ACTION_ID=:3 , OWNER=:4 , START_DATE=:5 , DUE_DATE=:6 , FINISH_DATE=:7 , STATUS=:8 , CALLER=:9 WHERE ID=:10 ; inst: 1 sid: 554 serial: 45609 user: JIRA elapsed: 859 sql_id: bs664s4c8x21w sql: UPDATE project SET pcounter=:1 WHERE ID=:2 ; blocking: inst: 1 username JIRA sid: 165 type TX held: None req: X ctime: 62
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.