Safe way to query JQL in groovy script

[#1]

I've been challenged recently with some performance issues on my environment.

While doing research, I came across this Kbase article and as I was reading the "Cause" section mention the use of PagerFilter.getUnlimitedFilter() I went: "Hey this looks familiar, I do this all the time!"

Sure enough, I probably got that example from a place like this Adaptavist Library Script.

So what should one do to use JQL in a script while not risk blowing up their heap space with an uncontrolled query?

Here is my solution...

I created a short IssueSearchHelper groovy class that I can leverage in my other scripts:

package path.to

import com.atlassian.jira.bc.issue.search.SearchService
import com.atlassian.jira.component.ComponentAccessor
import com.atlassian.jira.user.ApplicationUser
import com.atlassian.jira.web.bean.PagerFilter
import groovy.util.logging.Log4j
import org.apache.log4j.Level

@Log4j
class IssueSearchHelper {
    final Integer MAX_PAGER_SIZE = 200
    final Integer DEFAULT_PAGER_SIZE = 50

    ApplicationUser user = ComponentAccessor.jiraAuthenticationContext.loggedInUser
    SearchService searchService = ComponentAccessor.getComponent(SearchService)
    Integer pageSize = DEFAULT_PAGER_SIZE
    Integer maxIterations

    static void setLogLevel(def level) {
        //quick way to turn on logging
        if (level instanceof String) log.setLevel(Level.toLevel(level))
        if (level instanceof Level) log.setLevel(level)
    }

    void setPageSize(Integer size) {
        if (!size) size = DEFAULT_PAGER_SIZE
        //don't let anyone attempt to get more than the MAX_PAGER_SIZE per page
        this.pageSize = (size > MAX_PAGER_SIZE) ? MAX_PAGER_SIZE : size
    }

    void setUser(ApplicationUser user) {
        //default the user when user:null is specified
        this.user = user ?: ComponentAccessor.jiraAuthenticationContext.loggedInUser
    }

    Long getSearchCount(String jql) {
        def jqlParseResult = searchService.parseQuery(user, jql)
        assert jqlParseResult.isValid(), jqlParseResult.errors

        searchService.searchCount(user, jqlParseResult.query)
    }

    def withJql(String jql, Closure closure) {
        def jqlParseResult = searchService.parseQuery(user, jql)
        assert jqlParseResult.isValid(), jqlParseResult.errors
        def issueCount = searchService.searchCount(user, jqlParseResult.query) as Integer
        if(!issueCount) return null
        Integer localMax = maxIterations ?: issueCount
        Integer localPageSize = pageSize
        if (localMax < pageSize) {
            localPageSize = localMax
        }
        //we'll chunk the results from the end in case the closure causes issue to drop from the search
        log.info "Creating an initial pager with for the last $localPageSize issues out of $issueCount"
        def start = (issueCount as Integer) - localPageSize
        def pager = new PagerFilter(start, localPageSize)
        Integer iterationCount = 0
        while (true) {
            log.debug "Retrieving 1 page of results from $pager.start to $pager.end"
            def results = searchService.search(user, jqlParseResult.query, pager).results
            results.each {
                def issue = ComponentAccessor.issueManager.getIssueObject(it.id)
                log.trace "Obtained a Mutable issue from search result: $issue.key"
                log.trace "Calling the supplied closure"
                closure.call(issue)
            }
            iterationCount += localPageSize
            if (pager.start == 0 || localMax && iterationCount >= localMax) {
                break
            }
            log.trace "Creating another pager starting at $pager.previousStart with a size of $pager.max"
            def remainingIterations = localMax - iterationCount
            if (remainingIterations < localPageSize) {
                pager = new PagerFilter(pager.start - remainingIterations, remainingIterations)
            } else {
                pager = new PagerFilter(pager.previousStart, localPageSize)
            }
        }
    }
}

This encapsulates anything you need to do to perform repeatable actions on a number of issues.

It takes care of grabbing the necessary components and validating your JQL.

Here are a few examples of how you can instantiate the helper class:

import path.to.IssueSearchHelper

//get a default search helper with currentUser, default page size, unlimited results
def defaultSearchHelper = new IssueSearchHelper()

//get a SearchHelper where the issues are retreived based on  a specific user permission
def someUser = ComponentAccessor.userManager.getUserByName('sample_username')
def userSearchHelper = new IssueSearchHelper(user:someUser)

//specify the maximum number of issue to process regardless of how many are present
def limitedSearchHelper = new IssueSearchHelper(maxIterations:10)

//specify the size of each page to process
def pageSizeSearchHelper = new IssueSearchHelper(pageSize:100)

//all paramters combined
def allArgsSearchHelper = new IssueSearchHelper(pageSize:100, maxIterations:250, user:somUser)

Here is how you would actually use it once instantiated:

import com.atlassian.jira.component.ComponentAccessor
import com.atlassian.jira.issue.Issue
import path.to.IssueSearchHelper

def is = ComponentAccessor.issueService

//get a default search helper with currentUser, default page size, unlimited results
def issueSearchHelper = new IssueSearchHelper()

//you can re-use the user variable defaulted by the helper, I'll use that below in the issueService call
def user = issueSearchHelper.user

def jql = 'Assignee = currentUser() and Updated > -1d'

//just return the total number of issue that match the JQL. maxIterations and pageSize are ignored
def count = issueSearchHelper.getSearchCount(jql)

//perform the same operation on each issues in the search
issueSearchHelper.withJql(jql){Issue issue->
    log.info "Found issue $issue.key"
    
    //make a random update
    def iip = is.newIssueInputParameters()
    iip.summary = issue.summary + " - updated"
    def validationResult = is.validateUpdate(user, issue.id, iip)
    assert validationResult.valid : validationResult.errorCollection.errors
    is.update(user, validationResult)
}

Here is another example where you can use the helper in the middle of other code when you need to iterate over your issues to build something else, like a table:

import com.atlassian.jira.issue.Issue
import path.to..IssueSearchHelper
import groovy.xml.MarkupBuilder

def issueSearchHelper = new IssueSearchHelper()

def writer = new StringWriter()
def builder = new MarkupBuilder(writer)
builder.table(class:'aui'){
    def jql = 'updated > -1d'
    issueSearchHelper.withJql(jql){Issue issue->
        tr{
            td issue.key
            td issue.summary
        }
    }
}
writer.toString()

This last example may still give you memory problems because the writer variable will still grow based on the number of issues your JQL returns, but at least you won't be ALSO filling a SearchResults object with all of the issues.

It's still possible to get yourself in trouble if you try hard enough:

def jql = 'created > 1970-01-01'

def allIssues = []
issueSearchHelper.withJql(jql){Issue issue->
    allIssues << issue
}
//allIssues will be just as big as if you had used PagerFilter.unlimitedFilter

Now, it turns out, my performance problems probably had nothing to do with my scripted jql queries ... but now I know better than to risk it by using getUnlimitedFilter()

Have fun scripting :)

[Peter-Dave Sheehan]

For anyone watching this... I encountered an infinite loop situation because my supplied closure was updating issues such that they disappeared from the JQL. 

I edited the code in the original post with a fix. The new version will process the issues in the search in reverse order. Also, there is an exit condition that will be encountered no matter what.