Jira suddenly went slow and unresponsive for about an hour

[Rahul Aich [Nagra]]

Hi

Last night our production instance of jira suddenly went extremely slow and unresponsive for about an hour. After about an hour and a half things went back to normal.

Can anyone please help me. I have attached the atlassian-jira and UI error logs.

On UI we got

com.atlassian.jira.exception.DataAccessException: org.ofbiz.core.entity.GenericEntityException: while updating: [GenericEntity:Issue][summary,Build_3[XXXXXX]: .][status,1][votes,0][fixfor,null][assignee,aaaaaaa][security,null][resolution,null][resolutiondate,null][type,1][timespent,null][id,7735][reporter,eli77][project,1052][environment,Platform: TCH73 FGR Box Build: Build 25 Build Type: s_tf][component,null][updated,2013-01-15 12:58:20.0][created,2013-01-15 12:57:31.0][workflowId,959][description,Steps to reproduce: <DELETED ON PURPOSE> (CL# 338027)][priority,2][timeestimate,null][duedate,null][timeoriginalestimate,null][key,TFE2E-825] (SQL Exception while executing the following:UPDATE jiraissue SET pkey=?, PROJECT=?, REPORTER=?, ASSIGNEE=?, issuetype=?, SUMMARY=?, DESCRIPTION=?, ENVIRONMENT=?, PRIORITY=?, RESOLUTION=?, issuestatus=?, CREATED=?, UPDATED=?, DUEDATE=?, RESOLUTIONDATE=?, VOTES=?, TIMEORIGINALESTIMATE=?, TIMEESTIMATE=?, TIMESPENT=?, WORKFLOW_ID=?, SECURITY=?, FIXFOR=?, COMPONENT=? WHERE ID=? (Lock wait timeout exceeded; try restarting transaction))

On atlassian-jira.log, i have about 15k lines of repeating errors at the same instant (22:13:57 Hrs). Here are the unique lines.

2013-01-15 22:13:57,216 http-443-19 ERROR lxxxxxx 8x32075x3 1a6ljxd 172.22.6.81 /secure/WorkflowUIDispatcher.jspa [com.opensymphony.workflow.AbstractWorkflow] Error checking available actions
com.opensymphony.workflow.StoreException: Could not find workflow instance #95670: root cause: SQL Exception while executing the following:SELECT NAME, INITIALIZED, STATE FROM OS_WFENTRY WHERE ID=? (The last packet successfully received from the server was 79,505,636 milliseconds ago. The last packet sent successfully to the server was 79,505,636 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.)

2013-01-15 22:13:57,232 http-443-20 ERROR cstev 514x50962x7 1vo5lhl 10.4.42.84 /secure/CommentAssignIssue.jspa [atlassian.jira.workflow.SimpleWorkflowManager] Error occurred while rolling back transaction.
org.ofbiz.core.entity.GenericTransactionException: Error occurred while rolling back transaction. (No operations allowed after connection closed.Connection was implicitly closed by the driver.)

2013-01-15 22:13:57,420 http-443-14 ERROR xcwa 100x37714x6 wox3k6 172.22.7.0 /secure/WorkflowUIDispatcher.jspa [webwork.util.ValueStack] query="locale" {[id="locale" type="8" values=""]}
java.lang.reflect.InvocationTargetException

2013-01-15 22:13:57,373 http-443-20 ERROR cstev 514x50962x7 1vo5lhl 10.4.42.84 /secure/CommentAssignIssue.jspa [webwork.util.ValueStack] METHOD: "unescapedText", exception:
java.lang.IllegalStateException: getAttribute: Session already invalidated

2013-01-15 22:13:57,310 http-443-20 ERROR cstev 514x50962x7 1vo5lhl 10.4.42.84 /secure/CommentAssignIssue.jspa [velocity] Method getText threw exception for reference $i18n in template templates/jira/issue/field/attachment-edit.vm at [50,21]

2013-01-15 22:13:57,310 http-443-20 ERROR cstev 514x50962x7 1vo5lhl 10.4.42.84 /secure/CommentAssignIssue.jspa [velocity] VMProxyArg.getObject() : method invocation error getting value : org.apache.velocity.exception.MethodInvocationException: Invocation of method 'getText' in class com.atlassian.jira.web.action.issue.CommentAssignIssue threw exception class java.lang.IllegalStateException : getAttribute: Session already invalidated

2013-01-15 22:13:57,310 http-443-20 ERROR cstev 514x50962x7 1vo5lhl 10.4.42.84 /secure/CommentAssignIssue.jspa [com.atlassian.velocity.DefaultVelocityManager] MethodInvocationException occurred getting message body from Velocity: java.lang.IllegalStateException: getAttribute: Session already invalidated

013-01-15 22:13:57,310 http-443-20 ERROR cstev 514x50962x7 1vo5lhl 10.4.42.84 /secure/CommentAssignIssue.jspa [velocity] Method getText threw exception for reference $i18n in template templates/jira/issue/field/resolution-edit.vm at [1,385]
2013-01-15 22:13:57,295 http-443-18 ERROR xcwang 92x36934x7 wox3k6 172.22.7.0 /secure/WorkflowUIDispatcher.jspa [webwork.util.ValueStack] query="locale" {[id="locale" type="8" values=""]}
java.lang.reflect.InvocationTargetException

2013-01-15 22:13:57,279 http-443-20 ERROR cstev 514x50962x7 1vo5lhl 10.4.42.84 /secure/CommentAssignIssue.jspa [webwork.util.ValueStack] query="searchRequest" {[id="searchRequest" type="8" values=""]}
java.lang.reflect.InvocationTargetException

[Renjith Pillai · Accepted solution]

A bad day for you, huh?

Check out https://confluence.atlassian.com/display/BAMKB/MySQL+-+Lock+wait+timeout+exceeded+-+try+restarting+transaction

There are some stuff in that link (link is for Bamboo, works for JIRA as well). Also need to check if users are selecting huge volume of data? How much is set as your query limit jira.search.views.max.limit?

[Vishnukumar Vasudevan]

Looks like DB error. Did you talk to your DB guys ? is there any issue with DataBase ?

[Dave Hergert]

Sounds like a database issue; have you contacted your local DBA to see if anything happened to the database around that time? Based on your information that it lasted for approximately an hour, I am guessing your database went down for a backup or some sort of maintenance that was preventing connections.