Problem Re-Indexing with Groovy Runner plugin

Andy Grace March 10, 2013

I had an index fail with NoClassDefFoundError - this then meant that all Jira functionality was lost (couldn't search for or create issues, admin tools all messed up)

Stopping Jira, deleting indexes and removing plugins from installedplugins and then re-starting overcame and I gradually added plugins re-indexing as I went and re-index went fine at each stage

I upgraded Groovy Runner from 2.1 Beta 7 to 2.1 Beta 8 and I didn't get the NoClassDefFoundError but the re-index did fail, this time with Too Many Open Files

I upped ullimit to 4096 and it now re-indexes successfully

I don't have that many Groovy Files (1 scripted field is a file and a few workflow transitions have groovy files) and this is a test system with relatively low volume of issues so I'm reticent to move it to production until I overcome the issue.

Any ideas why it might be keeping too many files open (or indeed why the first time failed with NoClassDefFoundError and not the 2nd when I didn't change any scripted files or script transitions/fields)?

6 answers

1 accepted

0 votes
Answer accepted
Andy Grace July 2, 2013

Was resolved by https://jamieechlin.atlassian.net/browse/GRV-211 apologies for the late answer

0 votes
ITSupport May 6, 2015

I was wondering if there is a way to disable indexing Custom Scripted Field made by Script Runner? In our environment we are creating html content. Apparently, when we remove groovy script and add "return '' " indexing works fine but when complete script is generating html index is corrupt.

JamieA
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
May 6, 2015

to disable indexing just set the custom field to have no indexer.

ITSupport May 8, 2015

Thank you very much for your tip! Accept my apologies for not thinking and rushing with question.

JamieA
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
May 8, 2015

no apology necessary, but thanks.

SABVARX April 7, 2017

Thank you for posting the solution on how to disable the indexer! I was really in troubles because the index couldn't run through with script runner enabled. We have two html customfields. After editing the customfiels to have "no search template" it all works fine again.

0 votes
Pierre EK November 27, 2014

We have the same problem here - we had it before and it still exists with the upgraded versions of JIRA and Script Runner. We are now at JIRA 6.3.7, and Script Runner 3.0.6.

If we keep Script Runner enabled, and perform a Reindex of JIRA (full or background), the indexes become corrupted. The problem is solved by disabling Script Runner and relaunching the indexing sad

I am surprised that we are just a few to report this problem, even though it has been existing with our company for a while. We are keeping the plugin disabled, and we're enabling it only when we need to Copy a project in the Built In scripts section...

0 votes
JamieA
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
December 4, 2013

Are you sure that's the reason - can you test with a script that just returns null?

What indexer are you using?

0 votes
Sumit Kumar
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
December 4, 2013

is this resolved ?? I am also facing the same issue.

Script Runner version :: 2.1.9 .. index goes corrupt after adding the scripted fields into the project.

when a null is returned as an output of the issue on which the scripted field are loaded.

0 votes
JamieA
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
March 10, 2013

> NoClassDefFoundError and not the 2nd when I didn't change any scripted files or script transitions/fields

Which class was not found?

No... sounds improbable that nothing changed but you got different results. Have you got the log files?

> I upped ullimit to 4096 and it now re-indexes successfully

4096 is really very low... the ulimit for open files is probably not going to be related to the volume of issues.

JamieA
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
March 10, 2013

Hi Andy,

> it would need to open a file for each issue that has a scripted field that has groovy file?

Possibly... although it would also close it. Seems related to https://jamieechlin.atlassian.net/browse/GRV-206 - maybe you could watch that issue.

Andy Grace March 10, 2013

Hi Jamie

Thanks for the super quick answer - the only change between the 2 times was upping the version of the groovy plugin.

It didn't specify the class that it couldn't find

I upped ulimit from 1024 to 4096 as outlined as it

https://confluence.atlassian.com/display/JIRAKB/Loss+of+Functionality+due+to+Too+Many+Open+Files+Error

I'm not sure how the indexing works and whether it would need to open a file for each issue that has a scripted field that has groovy file?

I do have log files and screenshots - what would be the best way to get them to you?

Cheers

Andy

Andy Grace March 10, 2013

Would I be better changing to inline groovy for the custom fields and transitions rather than opening the file?

Here is a snippet of the log from the first failure the NoClassDef.. so it appeared then that it failed to find the class for the Index failure.. but the original error was the same too many files open - I suspect it must have failed to read from the class path due to too many files open issue

Exception in thread "ISSUE-indexQueue:thread-1" java.lang.NoClassDefFoundError: com/atlassian/jira/index/DefaultIndex$Failure

at com.atlassian.jira.index.DefaultIndex.perform(DefaultIndex.java:45)

at com.atlassian.jira.index.QueueingIndex$Task.index(QueueingIndex.java:144)

at com.atlassian.jira.index.QueueingIndex$Task.run(QueueingIndex.java:125)

at java.lang.Thread.run(Unknown Source)

...

at java.lang.Thread.run(Unknown Source)

Mar 8, 2013 1:09:22 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run

SEVERE: Socket accept failed

java.net.SocketException: Too many open files

at java.net.PlainSocketImpl.socketAccept(Native Method)

at java.net.PlainSocketImpl.accept(Unknown Source)

at java.net.ServerSocket.implAccept(Unknown Source)

at java.net.ServerSocket.accept(Unknown Source)

at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)

Andy Grace March 11, 2013

Hi - so it seems GRV-206 is not related to my problem but to the groovy version.

I've unearthed something deeper in the log - there was a nullpointer being thrown on one of my equalsIgnoreCase checks which was occuring on a null object

Perhaps the script runner is not closing files when hitting exceptions like this and that is why it is now hitting the Too many open files issue?

Shall I create a GRV issue so I can upload logs etc?

JamieA
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
March 11, 2013

No, it will be not be leaving files open. Why don't you see what files are open with lsof? You can create an issue, sure, but I'm not seeing a real problem yet.

Andy Grace March 13, 2013

OK I now have it narrowed down to one scripted field that uses a file - if I delete that field it indexes fine, with the field there it dies with too many files open

It could just be that the script is taking a while to pull results so that during indexing more and more files get open

I'll raise an issue and send you the lsof output and further details

Suggest an answer

Log in or Sign up to answer