TIL - Carriage Returns in Summary - HAPI issue?

[#1]

Sharing this with the Jira Data Center admins who use ScriptRunner HAPI. 

Problem

I was using HAPI to update some issue fields and it threw an error when it hit an issue with a carriage return in the Summary field, even though I wasn't updating that field.

My script looked something like this:

Issues.search("Project = ABC").each { issue ->
  issue.update {
    issue.setFieldA("New Value")
  }
}

How did that carriage return get into the Summary field? I had imported 200K issues from another system during a migration and it didn't occur to me to check the Summary field for carriage returns before I did the import. The import worked fine and imported the carriage returns into the Summary field without complaint, even though they don't show up when you look at the issue in Jira.

Solution

Since I didn't know which issue caused the problem and there may have been many of them (in fact there were about 20K issues with this problem), I needed to check them all.

My solution was to use the following script:

import java.util.stream.Collectors

def issues = Issues.search('project = ABC')
    .stream()
    .filter { issue -> issue.getSummary().contains("\n")}    
    .collect(Collectors.toList())
    .each { issue ->
        issue.update {
            setSummary(issue.getSummary().replace("\n", " "))
        }
    }

Because of the size of the data set, I used the .stream() to stream the data into the filter and then collect it into a List. Then it was a simple matter to update the Summary replacing the carriage return with a space.

Open Question

I worried that even with the filter, the resulting data would overwhelm the server memory. In my case, it was able to handle it. If you know how to process the filter results without having to collect them into a list first (i.e., run the issue.update() function on each issue as it comes out of the filter, let me know if the comments.

[Matt Doar _Adaptavist_]

Glad to hear you found a solution. I guess you could keep the impact smaller by changing the JQL search to return fewer issues. You know you can use JQL such as "key < TEST-1000" to only return issues with issue keys in that range

I suspect you used the old CSV importer when importing the original 200K issues? Or perhaps a custom one using the Jira REST API? I feel like I've run into this problem sometime ago and it was that the CSV importer didn't check for this case. It also gets funky when you try to have Summary fields that are longer than 255 characters, or perhaps it was 254. 

Do you think that ScriptRunner HAPI should handle making the field contents correct for Jira DC?

[Jamie Echlin _ScriptRunner - The Adaptavist Group_]

Hi Derek. You are right to consider memory issues. The iterator returned by `Issues.search` will hang on to a constant amount of memory, by fetching pages of issues from the Lucene index, only when required 

However, if you use groovy GDK methods like `findAll` then it will collect as many matching issues as it can, which can cause memory issues. So the solution is to use `stream` and `filter`. To fix your code just remove the `collect` line, which is unnecessary:

def issues = Issues.search('project = SSPA')
    .stream()
    .filter { issue -> issue.getSummary().contains("\n")}
    .forEach { issue ->
        issue.update {
            setSummary(issue.getSummary().replace("\n", " "))
        }
    }

I also used forEach rather than each, they should be interchangeable here but `each` was showing a type-checking error for some reason, the code worked fine though.