JsonParseException: Invalid UTF-8 middle byte 0xe4 on RestAPI create project

[Peter Brodt]

Hi 

we run a groovy script to create projects from a jira issue in a workflow post function via RestAPI "createshared". The script runs fine as long as the project name doesn't contain german special character (Umlauts). Then the following error is thrown:

Uncaught exception thrown by REST service: Invalid UTF-8 middle byte 0xe4

Here is my code snippet:

final String PROJECT_NAME_TO_CREATE = "ÄäÖöÜüß"
final String PROJECT_LEAD_USERNAME = issue.reporter.name //Reporter wird Project Lead

def cField5Object = customFieldManager.getCustomFieldObjectsByName('Produkt + Template')
def cField5 = cField5Object[0]
def cFieldValue5 = issue.getCustomFieldValue(cField5)
def String template = String.valueOf(cFieldValue5).drop(14);
final String PROJECT_TEMPLATE = template.substring(0,template.length() - 1)

// Projekt Beschreibung wird der Template Name eingefügt
final String PROJECT_DESCRIPTION = "Basierend auf Template: " + PROJECT_TEMPLATE

projectTemplateID = 20925
ProjectManager projectManager = ComponentAccessor.getProjectManager()
Project templateProjectId = projectManager.getProjectByCurrentKey(projectTemplateKey);

//StringEntity stringEntity = new StringEntity('{"key":"RYMG","name":"Rest Ye Merry Gentlemen","lead":"XV13499"}')
def propertyString = '{"key":"' + PROJECT_KEY_TO_CREATE + '","name":"' + PROJECT_NAME_TO_CREATE + '","lead":"' + PROJECT_LEAD_USERNAME + '","description":"' + PROJECT_DESCRIPTION + '"}'
StringEntity stringEntity = new StringEntity(propertyString)
log.warn("stringEntity: " + propertyString)
HttpPost httpPost = new HttpPost(JIRA_BASEURL + "/rest/project-templates/1.0/createshared/" + projectTemplateID)
log.warn("HttpPost = " + JIRA_BASEURL + "/rest/project-templates/1.0/createshared/" + projectTemplateID)
httpPost.setHeader(HttpHeaders.CONTENT_TYPE, "application/json")
httpPost.setEntity(stringEntity)

//httpPost.setHeader(HttpHeaders.AUTHORIZATION, "Basic dXNlcjpwd2Q=")
httpPost.setHeader(HttpHeaders.AUTHORIZATION, "Basic ${authString}")
HttpClient httpClient = HttpClientBuilder.create().build()
HttpResponse response = httpClient.execute(httpPost)

The RestAPI with curl directly from the commandline works perfect.

Any ideas 

Peter

[PD Sheehan]

I have several thoughts ... 

1) If you are manipulating JSON in groovy, don't try to do it yourself. Use the built-in libraries for that

import groovy.json.JsonBuilder
def projectPayload = [key: PROJECT_KEY_TO_CREATE, name:PROJECT_NAME_TO_CREATE, lead:PROJECT_LEAD_USERNAME, description:PROJECT_DESCRIPTION]
def projectPayloadJson = new JsonBuildere(projectPayload).toPrettyString()

This should fix your immediate issue. Here is a sample console script that shows the difference:

import groovy.json.JsonBuilder

final String PROJECT_NAME_TO_CREATE = "ÄäÖöÜüß"
String PROJECT_KEY_TO_CREATE = 'key'
String PROJECT_LEAD_USERNAME = 'lead'
String PROJECT_DESCRIPTION = 'desc'

def projectPayload = [key: PROJECT_KEY_TO_CREATE, name: PROJECT_NAME_TO_CREATE, lead: PROJECT_LEAD_USERNAME, description: PROJECT_DESCRIPTION]

def propertyString = '{"key":"' + PROJECT_KEY_TO_CREATE + '","name":"' + PROJECT_NAME_TO_CREATE + '","lead":"' + PROJECT_LEAD_USERNAME + '","description":"' + PROJECT_DESCRIPTION + '"}'
log.info "raw object output: $projectPayload"
log.info "proper json: ${new JsonBuilder(projectPayload).toString()}"
log.info "manual json: $propertyString"

 2) If you need to use the rest API, it's much better to use the "TrustedRequestFactory"

import com.atlassian.jira.component.ComponentAccessor
import com.atlassian.jira.config.properties.APKeys
import com.atlassian.sal.api.net.Request
import com.atlassian.sal.api.net.TrustedRequest
import com.atlassian.sal.api.net.TrustedRequestFactory
import com.onresolve.scriptrunner.runner.customisers.PluginModule
import groovyx.net.http.URIBuilder

@PluginModule TrustedRequestFactory trustedRequestFactory

def runAsUser= ComponentAccessor.userManager.getUserByName('admin')
def baseUrl = ComponentAccessor.applicationProperties.getString(APKeys.JIRA_BASEURL)
def request = trustedRequestFactory.createTrustedRequest(Request.MethodType.POST, '/rest/project-templates/1.0/createshared/') as TrustedRequest
request.addTrustedTokenAuthentication(new URIBuilder(baseUrl).host, runAsUser.name)

def projectPayload = [key: PROJECT_KEY_TO_CREATE, name: PROJECT_NAME_TO_CREATE, lead: PROJECT_LEAD_USERNAME, description: PROJECT_DESCRIPTION]
def projectPayloadJson = new JsonBuildere(projectPayload).toPrettyString()
request.setRequestBody(projectPayloadJson, 'application/json')

def response = request.execute()

This way, you won't need to deal with protecting credentials

3) I'm not sure what type of custom field your "Produkt + Template" field is, but I'm sure there is a better way to extract a project key from the selected item. If your field is a "Project Picker" field, then the value of the getCustomFieldValue will be a project object. You could just get the key or id directly from it (or, if you want auto-completion and prevent static type checking error, you can type cast the result):

import com.atlassian.jira.project.Project
def selectedProject = issue.getCustomFieldValue(cf) as Project
def selectedProejctKey = selectedProject.key
def selectedProjectId = selectedProject.id

3) But really, you don't need to use REST API at all for this. you could either use the native ProjectService, or the built-in script "Copy Project.

Here is a native example (for the console) using some of your variables and values:

import com.atlassian.jira.component.ComponentAccessor
import com.atlassian.jira.issue.Issue
import com.atlassian.jira.project.Project
import com.atlassian.jira.bc.project.ProjectCreationData
import com.atlassian.jira.bc.project.ProjectService

def projectService = ComponentAccessor.getComponent(ProjectService)

def issue = ComponentAccessor.issueManager.getIssueObject('KEY-123')
def runAsUser = ComponentAccessor.userManager.getUserByName('admin')
def projectCf = ComponentAccessor.customFieldManager.getCustomFieldObjectsByName('Project Name')[0]

def PROJECT_TEMPLATE = (issue.getCustomFieldValue(projectCf) as Project)
String PROJECT_NAME_TO_CREATE = "ÄäÖöÜüß"
String PROJECT_DESCRIPTION = "Basierend auf Template: " + PROJECT_TEMPLATE.key
String PROJECT_KEY_TO_CREATE = 'KEY'

def projBuilder = new ProjectCreationData.Builder()
projBuilder.
withProjectTemplateKey(PROJECT_TEMPLATE.key).
withKey(PROJECT_KEY_TO_CREATE).
withName(PROJECT_NAME_TO_CREATE).
withDescription(PROJECT_DESCRIPTION).
withLead(issue.reporter)

def validationResult = projectService.validateCreateProjectBasedOnExistingProject(runAsUser, PROJECT_TEMPLATE.id,projBuilder.build())
assert validationResult.isValid(), validationResult.errorCollection
projectService.createProject(validationResult)

 Here is an example with the CopyProject:

import com.atlassian.jira.bc.project.ProjectCreationData
import com.atlassian.jira.component.ComponentAccessor
import com.onresolve.scriptrunner.canned.jira.admin.CopyProject
import com.atlassian.jira.project.Project

def projectManager = ComponentAccessor.projectManager

def runasUser = ComponentAccessor.userManager.getUserByName('admin') //user with create project permission
def issue = ComponentAccessor.issueManager.getIssueObject('KEY-123')
def projectCf = ComponentAccessor.customFieldManager.getCustomFieldObjectsByName('Project Name')[0]

def PROJECT_TEMPLATE = issue.getCustomFieldValue(projectCf) as Project
String PROJECT_NAME_TO_CREATE = "ÄäÖöÜüß"
String PROJECT_DESCRIPTION = "Basierend auf Template: " + PROJECT_TEMPLATE.key
String PROJECT_KEY_TO_CREATE = 'KEY'

def copyProject = new CopyProject()
def inputs = [
        (CopyProject.FIELD_SOURCE_PROJECT)       : PROJECT_TEMPLATE.key,
        (CopyProject.FIELD_TARGET_PROJECT)       : PROJECT_KEY_TO_CREATE,
        (CopyProject.FIELD_TARGET_PROJECT_NAME)  : PROJECT_NAME_TO_CREATE,
        (CopyProject.FIELD_COPY_VERSIONS)        : false,
        (CopyProject.FIELD_COPY_COMPONENTS)      : false,
        (CopyProject.FIELD_COPY_ISSUES)          : false,
        (CopyProject.FIELD_COPY_DASH_AND_FILTERS): false,
]

def errorCollection = copyProject.doValidate(inputs as Map<String, String>, false)
assert !errorCollection.hasAnyErrors(), errorCollection

ComponentAccessor.jiraAuthenticationContext.setLoggedInUser(runasUser)
def output = copyProject.doScript(inputs as Map<String, Object>)
log.info output
//the CopyProject script can't set the Project Lead and Description during the copy, so we set them after
def newProject = projectManager.getProjectObjByKeyIgnoreCase(PROJECT_KEY_TO_CREATE)
projectManager.updateProject(newProject, newProject.name, PROJECT_DESCRIPTION, issue.reporter.name, "", newProject.assigneeType)

I personally prefer the way the built-In CopyProject script handles custom fields contexts (the new project will share any contexts that the template project used, whereas the ProjectService method, the new project is not added to any context).

Sorry... looks like I went a bit overboard with this answer.

Hopefully, it can help some others too.

[PD Sheehan]

I was forgetting the newer syntax for getting a built-in script object:

import com.onresolve.scriptrunner.runner.ScriptRunnerImpl
import com.onresolve.scriptrunner.canned.jira.admin.CopyProject

def copyProject = ScriptRunnerImpl.scriptRunner.createBean(CopyProject)

[Peter Brodt]

Hi Peter,

thanks so much for your help.  After implementing the json builder everything works fine.

I definitely will change the script in respect to your "TrustedRequestFactory" advice.

Yes, there are sureley smarter ways for extracting the project id. But it is a workflow for our end-users to create Jira project or Conflunec space generation tickets. The field in question is not a project picker but a hard coded cascading select list (product[Jira|Confluence] + Templates). Since there are only a couple of templates we allow, we did it the simple way.

The native project service was our first attempt. Unfortunatly we came across an error, that newly created projects sometimes were not visible. After editing the project details of a random project, they became visible. The error did not occur every time but much too often to be acceptable. Neither Adaptavist nor Atlassian were able to help. That's why we switched to REST.

Thanks again for your help. I really appreciate it (:-.

Peter