It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

How to avoid timeouts / 502 errors with long running REST tasks

Anonymouslemming Sep 14, 2015

Hi all,

We need to delete a large number of projects from a number of JIRA instances. I've written a REST plugin to expose the ProjectService.DeleteProjectResult method from the JIRA API. 

This works fine for small projects with few issues, but for larger projects with thousands of issues, this fails. It runs for a while, then we eventually receive a 502 (Bad Gateway) error.

Attempting to delete similar sized projects in the UI has similar behaviour - we see the delete progress for a while, then we get a network timeout error. 

In both cases, the logs imply that the delete activity is ongoing after the failure. We see similar results when doing exports of large instances - a network timeout with the export continuing to run in the background.

What's the right way of keeping these kinds of long-running requests alive? 

I did consider iterating over all issues and deleting them first, but we have dozens of permission schemes in use, and most of these have no delete permission for any role. This was necessary to ensure that users can't delete things and break audit requirements. 

 

The code that the plugin is implementing is as follows:

ProjectService.GetProjectResult projectResult = projectService.getProjectByKey(loggedInUser, projectkey);
Project project = projectResult.getProject();

if (project == null) {
    logger.warn("Unable to lookup " + projectkey + " - Will not delete");
    return Response.status(Response.Status.NOT_FOUND).entity("Failed to lookup project " + projectkey).cacheControl(never()).build();
}
SimpleErrorCollection errorCollection = new SimpleErrorCollection();
ProjectService.DeleteProjectResult deleteProjectResult = projectService.deleteProject(loggedInUser, new ProjectService.DeleteProjectValidationResult(errorCollection, project));
if (deleteProjectResult.isValid()) {
    logger.info("Successfully deleted " + projectkey);
    return Response.status(Response.Status.OK).entity("Deleted Project").cacheControl(never()).build();
} else {
    logger.warn("Failed to delete " + projectkey);
    return Response.status(Response.Status.NOT_FOUND).entity("Failed to delete project").cacheControl(never()).build();
}

3 answers

1 vote
Volodymyr Krupach Sep 15, 2015

The only thing that comes to my mind is to start separate thread and return immediately from your REST. Then in your page you need to have JavaScript that pulls other rest method that checks if the thread is alive. While it's alive you show "progress icon" and show "Done status" when the rest returns "thread completed".

Anonymouslemming Sep 17, 2015

Thanks for the feedback. I'm not quite sure I follow though. I can see how we'd create the thread to carry out the delete action, but how would we use a different REST request to get information about that thread ? What information would we have to return from our original request to allow us to monitor the thread status (and exit status especially!) from another REST request ? Thanks,

Volodymyr Krupach Sep 17, 2015

Rest service classes (classes with methods annotated by annotations from javax.ws.rs) are singletons. It means that all you requests will be served by the same instances so you can keep thread reference in your class and have there 2 REST methods: delete and checkStatus. Also I suggest to make sure that "delete" does not spawn new threads while deletion is in progress.

Anonymouslemming Sep 19, 2015

Could you point me to some documentation that explains that a little better please? I'm not really a developer, I just impersonate one when we have a DevOps task like this :) I get that we could store the thread reference in our class, but I don't understand how we'd manage multiple requests. Assuming that User A kicks off a request to delete project FOO - we store the thread reference and checkstatus monitors that thread. User A tries to delete project FOO again, but we check that this is already in progress and do not allow the action. User B then kicks off a request to delete project BAR. At this point I'm lost. Are you saying that only 1 delete should be alive at a time and that user B shouldn't be able to trigger the delete of BAR until FOO is finished deleting ? Thanks!

Volodymyr Krupach Sep 19, 2015

> Are you saying that only 1 delete should be alive at a time and that user B shouldn't be able to trigger the delete of BAR until FOO is finished deleting ? Yes, exactly. Project deletion could be very heavy operation and I recommend you to allow only one at a time.

1 vote
Jonas Andersson Sep 16, 2015

Are you using a mod proxy infront of your Jira? By the sound of the issue, you do. If this is the case the timeouts can be configured in apache. If i am on the right track, let me know and i can tell you more. And upvote my answer smile I spend a lot of time helping but few take the time to give me a click! smile

 

Anonymouslemming Sep 17, 2015

We're not using mod_proxy, no, sorry.

1 vote

I've had problems with this before and ended up using a jira-python script to delete the issues slowly. Many permission schemes allow members of jira-administrators to do the same thing in multiple projects. Or can you extend your REST add-on to allow you to delete issues with the same authentication?

Anonymouslemming Sep 19, 2015

We have permission schemes that block us from being able to delete issues. I do like the idea though and will look into whether or not we can delete issues in our own plugin without meeting the requirements of the permission schemes - that might solve the issue for us.

Suggest an answer

Log in or Sign up to answer
This widget could not be displayed.
This widget could not be displayed.
Community showcase
Published in Next-gen

Introducing subtasks for breaking down work in next-gen projects

Teams break work down in order to help simplify complex tasks. This is often done iteratively, with tasks being broken down into smaller tasks and so on until the work is accurately captured in well-...

8,153 views 25 36
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you