I've been challenged recently with some performance issues on my environment.
While doing research, I came across this Kbase article and as I was reading the "Cause" section mention the use of PagerFilter.getUnlimitedFilter() I went: "Hey this looks familiar, I do this all the time!"
Sure enough, I probably got that example from a place like this Adaptavist Library Script.
So what should one do to use JQL in a script while not risk blowing up their heap space with an uncontrolled query?
Here is my solution...
I created a short IssueSearchHelper groovy class that I can leverage in my other scripts:
package path.to
import com.atlassian.jira.bc.issue.search.SearchService
import com.atlassian.jira.component.ComponentAccessor
import com.atlassian.jira.user.ApplicationUser
import com.atlassian.jira.web.bean.PagerFilter
import groovy.util.logging.Log4j
import org.apache.log4j.Level
@Log4j
class IssueSearchHelper {
final Integer MAX_PAGER_SIZE = 200
final Integer DEFAULT_PAGER_SIZE = 50
ApplicationUser user = ComponentAccessor.jiraAuthenticationContext.loggedInUser
SearchService searchService = ComponentAccessor.getComponent(SearchService)
Integer pageSize = DEFAULT_PAGER_SIZE
Integer maxIterations
static void setLogLevel(def level) {
//quick way to turn on logging
if (level instanceof String) log.setLevel(Level.toLevel(level))
if (level instanceof Level) log.setLevel(level)
}
void setPageSize(Integer size) {
if (!size) size = DEFAULT_PAGER_SIZE
//don't let anyone attempt to get more than the MAX_PAGER_SIZE per page
this.pageSize = (size > MAX_PAGER_SIZE) ? MAX_PAGER_SIZE : size
}
void setUser(ApplicationUser user) {
//default the user when user:null is specified
this.user = user ?: ComponentAccessor.jiraAuthenticationContext.loggedInUser
}
Long getSearchCount(String jql) {
def jqlParseResult = searchService.parseQuery(user, jql)
assert jqlParseResult.isValid(), jqlParseResult.errors
searchService.searchCount(user, jqlParseResult.query)
}
def withJql(String jql, Closure closure) {
def jqlParseResult = searchService.parseQuery(user, jql)
assert jqlParseResult.isValid(), jqlParseResult.errors
def issueCount = searchService.searchCount(user, jqlParseResult.query) as Integer
if(!issueCount) return null
Integer localMax = maxIterations ?: issueCount
Integer localPageSize = pageSize
if (localMax < pageSize) {
localPageSize = localMax
}
//we'll chunk the results from the end in case the closure causes issue to drop from the search
log.info "Creating an initial pager with for the last $localPageSize issues out of $issueCount"
def start = (issueCount as Integer) - localPageSize
def pager = new PagerFilter(start, localPageSize)
Integer iterationCount = 0
while (true) {
log.debug "Retrieving 1 page of results from $pager.start to $pager.end"
def results = searchService.search(user, jqlParseResult.query, pager).results
results.each {
def issue = ComponentAccessor.issueManager.getIssueObject(it.id)
log.trace "Obtained a Mutable issue from search result: $issue.key"
log.trace "Calling the supplied closure"
closure.call(issue)
}
iterationCount += localPageSize
if (pager.start == 0 || localMax && iterationCount >= localMax) {
break
}
log.trace "Creating another pager starting at $pager.previousStart with a size of $pager.max"
def remainingIterations = localMax - iterationCount
if (remainingIterations < localPageSize) {
pager = new PagerFilter(pager.start - remainingIterations, remainingIterations)
} else {
pager = new PagerFilter(pager.previousStart, localPageSize)
}
}
}
}
This encapsulates anything you need to do to perform repeatable actions on a number of issues.
It takes care of grabbing the necessary components and validating your JQL.
Here are a few examples of how you can instantiate the helper class:
import path.to.IssueSearchHelper
//get a default search helper with currentUser, default page size, unlimited results
def defaultSearchHelper = new IssueSearchHelper()
//get a SearchHelper where the issues are retreived based on a specific user permission
def someUser = ComponentAccessor.userManager.getUserByName('sample_username')
def userSearchHelper = new IssueSearchHelper(user:someUser)
//specify the maximum number of issue to process regardless of how many are present
def limitedSearchHelper = new IssueSearchHelper(maxIterations:10)
//specify the size of each page to process
def pageSizeSearchHelper = new IssueSearchHelper(pageSize:100)
//all paramters combined
def allArgsSearchHelper = new IssueSearchHelper(pageSize:100, maxIterations:250, user:somUser)
Here is how you would actually use it once instantiated:
import com.atlassian.jira.component.ComponentAccessor
import com.atlassian.jira.issue.Issue
import path.to.IssueSearchHelper
def is = ComponentAccessor.issueService
//get a default search helper with currentUser, default page size, unlimited results
def issueSearchHelper = new IssueSearchHelper()
//you can re-use the user variable defaulted by the helper, I'll use that below in the issueService call
def user = issueSearchHelper.user
def jql = 'Assignee = currentUser() and Updated > -1d'
//just return the total number of issue that match the JQL. maxIterations and pageSize are ignored
def count = issueSearchHelper.getSearchCount(jql)
//perform the same operation on each issues in the search
issueSearchHelper.withJql(jql){Issue issue->
log.info "Found issue $issue.key"
//make a random update
def iip = is.newIssueInputParameters()
iip.summary = issue.summary + " - updated"
def validationResult = is.validateUpdate(user, issue.id, iip)
assert validationResult.valid : validationResult.errorCollection.errors
is.update(user, validationResult)
}
Here is another example where you can use the helper in the middle of other code when you need to iterate over your issues to build something else, like a table:
import com.atlassian.jira.issue.Issue
import path.to..IssueSearchHelper
import groovy.xml.MarkupBuilder
def issueSearchHelper = new IssueSearchHelper()
def writer = new StringWriter()
def builder = new MarkupBuilder(writer)
builder.table(class:'aui'){
def jql = 'updated > -1d'
issueSearchHelper.withJql(jql){Issue issue->
tr{
td issue.key
td issue.summary
}
}
}
writer.toString()
This last example may still give you memory problems because the writer variable will still grow based on the number of issues your JQL returns, but at least you won't be ALSO filling a SearchResults object with all of the issues.
It's still possible to get yourself in trouble if you try hard enough:
def jql = 'created > 1970-01-01'
def allIssues = []
issueSearchHelper.withJql(jql){Issue issue->
allIssues << issue
}
//allIssues will be just as big as if you had used PagerFilter.unlimitedFilter
Now, it turns out, my performance problems probably had nothing to do with my scripted jql queries ... but now I know better than to risk it by using getUnlimitedFilter()
Have fun scripting :)
Peter-Dave Sheehan
Atlassian Administrator/Groovy Developer
QAD
Santa Barbara, California
526 accepted answers
1 comment