transactions using scripts result in timeouts

Andreas Sumper September 16, 2015

In JIRA 6.4 we have some transactions, that use scripted postfunctions or scripted values of customfields. From time to time (around once per day) the workflows stop working and the transactions result in a timeout.

For example: If a subtask fetches some information from the parent during creation, it will not finish.

To make it work again, we have to restart the complete JIRA isntallation.

Nothing is shown in the log. It seems, that the request does not reach the system at all. Network is not an issue as performing other steps at the same time from the same computer work.

Can anybody point me to somewhere to start to inverstigate. JIRA is not usable with this issue.

4 answers

1 vote
Nic Brough -Adaptavist-
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
September 16, 2015

I think we need to establish the point of failure.  Are the scripts actually starting or not?  Could you insert a simple log.debug("Script XYZ has started") right at the top of the suspect scripts to prove that it is actually stopping within the scripts?  Also, I'd add one just before the exit point too.

The problem here is that "Nothing is shown in the log" means we don't know if it's failing as the scripts are running, before or after them.

Andreas Sumper September 17, 2015

No, they are not getting triggered. I have debug-output on different places in the scripts.

I get a timeout, when trying to execute a transition, that uses any scriptet function.

If trying to open such a particular transition in workflow designer, it will also run into a timeout and not open.

If adding a scripted function to an existing transition and trying to save it, it will also run into a timeout.

We have to restart the whole JIRA instance to make it work again. It will work for some time, but after some hours or more, it will stop working.

 

When the system is in such a state, no errors are shown in the logs concerning the scripts. It seems, the transition or the adding of the scripted function is never triggered.

 

We also experienced issues with background reindex. It seems, that triggering a reindex in the background makes JIRA unavailable for users. At least some functions. 

 

Nic Brough -Adaptavist-
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
September 17, 2015

Good debugging info there, especially the look at the workflow editor and re-indexing. You've established that it's not the scripts themselves, but the attempt to actually open them - that seems to be stopping dead, and it's consistent across different methods of trying to access them. (The indexing looks at scripted fields, and the workflow designer obviously needs to check them when they are referenced in the workflow) So, it sounds like there's something killing the script-runner on any attempt at access. Problem is I don't know what it could be. Could you do a couple more things for us? * List the versions of JIRA, Java and the Script Runner in use - obviously, we should check the compatibility of all three, but as I suspect we're heading for raising a proper support call with Adaptavist, my colleagues are going to ask no matter what (You already said JIRA 6.4, but they'll probably ask for the point release as well - 6.4.??) * Check the resource usage the next time it freezes. Is the CPU or Memory on the server being thrashed? Ideally, the next time it happens, get a thread trace of what the process is doing.

0 votes
Andreas Sumper January 10, 2016

Firt of all, Sorry for not writing back earlier, as I had many things to finish in the last year. Now I can concentrate on the issue again.

Currently it seems, that it is not the script but other things, as it also occurs in other projects when creating subtasks.

We are fetching fields from parent task to the created subtasks in the subtask-workflow-create-function. It seems, that this fetching causes the problems. If some of the fields are set in the "new"-screen, it works. 

I changed my script to use another logfile, as we are having tons of entries in the logfile. As soon as it occurs again, I will give more information on that.

I also monitor the JIRA installation with jconsole. the memory rises up to 8GB (which is the limit) and when the garbage-collection starts, it goes down to 2,5GB. 

 

0 votes
JamieA
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
September 20, 2015

Sounds like a possible deadlock. But I don't think it's anything intrinsic in the plugin, there are plenty of large sites running many script functions all day.

You need to get a thread dump when the system becomes unresponsive or when the bad behaviour starts... how you do that varies depending on the server OS, but Atlassian have plenty of documentation on how.

> I have a stack trace from one of the last crashes (happens around once per day)

A crash seems different from what you're describing, but please give us the stack trace or create a request here: https://productsupport.adaptavist.com/servicedesk/customer/portal/3

 

0 votes
Andreas Sumper September 19, 2015

Jira is 6.4.9 build 64024
Java is 1.8.0_40
ScriptRunner Plugin is 4.1.1 (same behavior on 3.1.4)
I have a stack trace from one of the last crashes (happens around once per day)
The system is not having a very high load, when it comes to a crash.

Suggest an answer

Log in or Sign up to answer