JIRA Group.containsUser Issue

Question

We had a period of slow response followed by an outage yesterday due a number of threads in JIRA hanging on the following:

at java.util.concurrent.CopyOnWriteArrayList.indexOf(CopyOnWriteArrayList.java:150)

at java.util.concurrent.CopyOnWriteArrayList.contains(CopyOnWriteArrayList.java:188)

at com.atlassian.jira.user.osuser.JiraOFBizAccessProvider.handles(JiraOFBizAccessProvider.java:73)

at com.opensymphony.user.UserManager.getProvider(UserManager.java:269)

at com.opensymphony.user.UserManager$Accessor.getAccessProvider(UserManager.java:339)

at com.opensymphony.user.Entity.getAccessProvider(Entity.java:100)

at com.opensymphony.user.Group.containsUser(Group.java:121)

Please find the thread dump attached.

We have restarted the application and I haven't seen a repeat of those threads yet but would like any advice on what may have caused the threads to hang. Here are the top three groups by size. Is there known issues when groups get to a particular size? We can try to reduce jira-users if required. We are using jira-users in a few permission schemes as well. Should we take it out of those permission schemes?

jira-users        4661

Markit Internal        2506

confluence-users2471

I also noticed that the issue seems to be triggered primarily during Filter Table requests:

com.atlassian.jira.gadgets.system.IssueTableResource.getFilterTable(IssueTableResource.java:213)

Has anyone experienced similar issues to this before?

Dieter · Answer

I'd recommend to replace jira-users in permission schemes by anyone. Please read this for more details: http://blogs.onresolve.com/?p=102#more-102

Ram Tandukar1 · Answer

Thanks Jeff. We havent added any new plugins in the recent past but our instance is fairly busy with integrations that can make it difficult to determine the root cause. I agree that the threads that are locking on the above trace are not the cause but a symptom of a larger problem but we have noticed the same trace in some long running threads over the last month so I am still wondering if there is something that we can do to improve the group/user management.

Anyway, we havent had a repeat incident and will be upgrading in the next few months so hopefully the new user management systems in recent upgrades will help.

Jeff Kirby · Answer

Generally when we have run out of threads on our Jira server, the locked threads are not the cause but rather the sympton of some other operation that is unnecessarily locking up the server's resources. Once your server runs out of threads, then you're going to see all kinds of operations hanging.  Have you added a new plugin to the system recently? That is usually the culprit for us.