Blocking Locking Issues

Ram Tandukar November 10, 2011

Hello,

We are experiencing ongoing blocking session/locking issues on our JIRA instance. We generally get alerts on the database before we experience issues on the application. The impact to the application is that some users cannot perform particular actions (see below) and if the block is not cleared on the database level or the thread ended, the application will eventually stop responding. The issue has been isolated to a particular request/transition but we have found it difficult to reproduce the issue on our test/ QA server. The issue seems to occur during peak periods of the day.

Our DBA's are monitoring the database closely and do not see any other issues. The query that blocks is UPDATE OS_CURRENTSTEP.. I have attached some additional information in the database-info file.

I have been taking threaddumps and have been able to identify the exact request that is causing the block. The issue seems to be related to a particular transition in one of our workflows (attached workflow - transition is 461). This transition does have a custom post function that creates subtasks based on the information in the ticket. I suspect that this post function is in part a cause of the issue but we have not made any changes to the function for over a year and the frequency of the blocking issues have only recently increased. I have done a review of the code and I cannot find any glaringly major issues but I am continuing my investigation.

I understand that Atlassian cannot fully support our custom function and since our instance is quite large and complex it will be difficult to help but if you have any past experiences, ideas or advice how we can resolve this issue, I would greatly appreciatte any help.

Thanks,

Ram

6 answers

1 accepted

1 vote
Answer accepted
Radu Dumitriu
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
November 10, 2011

Most probably, you run into this: https://jira.atlassian.com/browse/JRA-26227

I opened a test case here: https://jira.atlassian.com/browse/JRA-26172

Read about it, sounds very familiar, we had this output also.

Radu Dumitriu
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
November 10, 2011

Please note that the second link provides a workaround to the problem.

Ram Tandukar November 10, 2011

Hi Radu,

Thanks for the reply! That bug actually describes our exact issue :-)

As you probably noted, we are also creating subtasks/issues within the post function.

I am going to go through your cases now but glad to hear that you have a workaround atleast.

I will let you know how I go.

Thankyou again - much appreciatte your time to post an answer!

Ram

0 votes
Ram Tandukar November 10, 2011
Base URL Blocked
System Date Friday, 11 Nov 2011
System Time 12:43:54 +0100
Current Working Directory /var/lib/heartbeat/cores/root
Java Version 1.6.0_24
Java Vendor Sun Microsystems Inc.
JVM Version 1.0
JVM Vendor Sun Microsystems Inc.
JVM Implementation Version 19.1-b02
Java Runtime Java(TM) SE Runtime Environment
Java VM Java HotSpot(TM) 64-Bit Server VM
User Name root
User Timezone Blocked
User Locale English (United States)
System Encoding ANSI_X3.4-1968
Operating System Linux 2.6.18-194.el5
OS Architecture amd64
Application Server Container Apache Tomcat/6.0.20
Database type oracle10g
Database JNDI address java:comp/env/jdbc/JiraDS
Database URL jdbc:oracle:thin:@nlgtprodorc001a.markit.partners:1521:GTPROD
Database version Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production With the Partitioning, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options
Database driver Oracle JDBC driver 11.2.0.1.0
External user management OFF
Crowd integration OFF

0 votes
Ram Tandukar November 10, 2011

Example of what looks to be the blocking thread:

"TP-Processor14" daemon prio=10 tid=0x00002aaac0deb000 nid=0x1d17 runnable [0x00000000455b6000]

java.lang.Thread.State: RUNNABLE

at java.net.SocketInputStream.socketRead0(Native Method)

at java.net.SocketInputStream.read(SocketInputStream.java:129)

at oracle.net.ns.Packet.receive(Packet.java:282)

at oracle.net.ns.DataPacket.receive(DataPacket.java:103)

at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:230)

at oracle.net.ns.NetInputStream.read(NetInputStream.java:175)

at oracle.net.ns.NetInputStream.read(NetInputStream.java:100)

at oracle.net.ns.NetInputStream.read(NetInputStream.java:85)

.........removed the standard oracle stace trace part....

oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1062)

at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:102)

at org.ofbiz.core.entity.jdbc.SQLProcessor.executeUpdate(SQLProcessor.java:631)

at org.ofbiz.core.entity.GenericDAO.singleUpdate(GenericDAO.java:240)

at org.ofbiz.core.entity.GenericDAO.customUpdate(GenericDAO.java:190)

at org.ofbiz.core.entity.GenericDAO.update(GenericDAO.java:183)

at org.ofbiz.core.entity.GenericHelperDAO.store(GenericHelperDAO.java:207)

at org.ofbiz.core.entity.GenericDelegator.store(GenericDelegator.java:1406)

at org.ofbiz.core.entity.GenericDelegator.store(GenericDelegator.java:1386)

at com.opensymphony.workflow.spi.ofbiz.OfbizWorkflowStore.markFinished(OfbizWorkflowStore.java:231)

at com.opensymphony.workflow.AbstractWorkflow.createNewCurrentStep(AbstractWorkflow.java:1473)

at com.opensymphony.workflow.AbstractWorkflow.transitionWorkflow(AbstractWorkflow.java:1256)

at com.opensymphony.workflow.AbstractWorkflow.doAction(AbstractWorkflow.java:567)

0 votes
Ram Tandukar November 10, 2011

Example of a locking thread:

"TP-Processor165" daemon prio=10 tid=0x00002aaac0b98000 nid=0x4cb6 waiting on condition [0x000000004ed4d000]

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for <0x00000006a7b932c8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)

at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)

at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)

at com.atlassian.util.concurrent.ManagedLocks$ManagedLockImpl.withLock(ManagedLocks.java:308)

at com.atlassian.util.concurrent.LockManagers$Manager.withLock(LockManagers.java:92)

at com.atlassian.jira.project.DefaultProjectManager$NextIdGenerator.getNextId(DefaultProjectManager.java:563)

at com.atlassian.jira.project.DefaultProjectManager.getNextId(DefaultProjectManager.java:129)

at com.atlassian.jira.project.AbstractProjectManager.getNextId(AbstractProjectManager.java:39)

at com.atlassian.jira.project.CachingProjectManager.getNextId(CachingProjectManager.java:87)

at com.atlassian.jira.workflow.function.issue.IssueCreateFunction.execute(IssueCreateFunction.java:51)

at com.opensymphony.workflow.AbstractWorkflow.executeFunction(AbstractWorkflow.java:869)

at com.opensymphony.workflow.AbstractWorkflow.transitionWorkflow(AbstractWorkflow.java:1265)

at com.opensymphony.workflow.AbstractWorkflow.initialize(AbstractWorkflow.java:618)

at com.atlassian.jira.workflow.SimpleWorkflowManager.createIssue(SimpleWorkflowManager.java:219)

0 votes
Ram Tandukar November 10, 2011
Issues 379008
Projects 470
Custom Fields 741
Workflows 343
Users 5484
Groups 466
Uptime 4 days, 5 hours, 29 minutes, 38 seconds
Edition enterprise
Version 4.1.1
Build Number 522
Build Date Mon Apr 19 00:00:00 CEST 2010
Atlassian Partner
Installation Type Standalone
0 votes
Ram Tandukar November 10, 2011

DB INFO:

OBJECT_NAME

--------------------------------------------------------------------------------

ROW_WAIT_OBJ# ROW_WAIT_FILE# ROW_WAIT_BLOCK# ROW_WAIT_ROW# DBMS_ROWID.ROWID_C

------------- -------------- --------------- ------------- ------------------

OS_CURRENTSTEP

82858 6 1228144 32 AAAUOqAAGAAEr1wAAg

SQL>

Notification Type: PROBLEM

Service: check_dbhealth

Host: GT Database

Address: 10.0.33.1

State: CRITICAL

Date/Time: Tue Oct 11 12:35:15 CEST 2011

Additional Info:

CRITICAL: blocking sessions found

long running: inst: 1 sid: 165 serial: 40174 user: JIRA elapsed: 620 sql_id: 1yu9j7r6xgvj6 sql: UPDATE OS_CURRENTSTEP SET ENTRY_ID=:1 , STEP_ID=:2 , ACTION_ID=:3 , OWNER=:4 , START_DATE=:5 , DUE_DATE=:6 , FINISH_DATE=:7 , STATUS=:8 , CALLER=:9 WHERE ID=:10 ; inst: 1 sid: 554 serial: 45609 user: JIRA elapsed: 859 sql_id: bs664s4c8x21w sql: UPDATE project SET pcounter=:1 WHERE ID=:2 ; blocking: inst: 1 username JIRA sid: 165 type TX held: None req: X ctime: 62

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events