Anyone know how to keep Jira Cloud Migration Assistant from killing our server?

[Nick Smith]

I'm in the process of planning/practicing our migration to Cloud.

When I run the planning/test parts of the migration tool on our server, it runs as expected. But I found, as I began addressing the issues the list (after the tool had seemingly reached a point where it was done processing anything further until I gave it more input), that our JIRA server had become unresponsive.

I found that the instance running JIRA was in pegging 100% of cpu resources across all processors/cores.

I didn't expect the tool to interrupt production during business hours (egg on my face). I ended up having to restart the server, and it didn't seem to experience this issue afterwards.

..until now, after business hours, I tried it again and it is doing the same thing.

Anyone have any ideas on what to do, where to look, etc.? 

I did notice that in the logs folder queries in /var/atlassian/application-data/jira/log/atlassian-jira-slow-queries.log entries like this:

2021-12-15 18:02:45,491-0500 pool-36-thread-18 WARN <USERNAMEREDACTED> 1075x2261x3 1tw443i /rest/migration/latest/check/67fb93a1a366691aa7972119d6f34f907fe1c904
[c.a.j.i.search.providers.LuceneSearchProvider_SLOW] LuceneQueryExecutionEvent{query={issue in childIssuesOf(<KEYREDACTED>-134)}, queryTermMetrics={org.apache.lucene.s
earch.TermInSetQuery=TermMetric{count=1, isCustomField=false}}, numberOfClausesInQuery=1, executionTime=1940, numberOfResults=0, collectorType='TopDocs'}

So it appear to be running queries in the background....but it seems to me there should be a limiter on this type of thing.

I'm going to let this run all night, maybe it will finish up whatever it is doing with itself.

[Nick Smith · Accepted solution]

So I ended up finally getting a response from support.

I basically had to open a response, deal with a brain dead autoresponder, wait a couple days and then manually click the escalate button in the issue. This...finally put me in touch with someone who was helpful.

He had me:

Set setenv.sh memory to (we had a different min than max):

JVM_MINIMUM_MEMORY="24g"
JVM_MAXIMUM_MEMORY="24g"

Set jira.autoexport=false to in the jira-config.properties file.

And then act incredibly patient for 2 hours. But we did eventually get all of the project/issues migrated.

It was also helpful to increase the core/memory count on the db server to a ridiculous amount.

[Alex Ortiz]

Hello @Nick Smith ,

I ran into a similar issue not too long ago.  

The first thing I would recommend you take a look at, is your SQL database.  Make sure that the version you are running is compatible with the migration assistant.  My specific scenario was that as soon as I kicked off the migration assistant, CPU usage would go to 100%.  Like you, I did this in production in the middle of the day.  Not good.

In my specific scenario, I couldn't update my SQL instance, so I had to stand up a separate Jira instance, export my production instance and import to my temporary instance.  This worked well as I now had a compatible instance and was able to do the migration as planned.

As Pramodh mentioned though, I too would recommend you test out in your dev environment first.  Let me know if I can be of any assistance!

[Nick Smith]

Hi Alex,

Thanks for the info.

The documentation for the migration tool is a mess and seems to assume we all have 10,000 users and a whole team to perform this migration lol.

Anyway, I wasn't able to find any specific database requirements in the docs (we use MySQL 5.7.3 for this system), and the version of JIRA we are using far exceeds the minimum version (8.18.0).

Also, this is probably a super small migration relatively speaking, with our help desk project has less than 12,000 issues in it and the other half dozen or so totaling less than a couple thousand, so I was surprised to have this kind of issue.

We don't have a dev environment though if I can isolate the issue and it's called for, I could stand up a new server/temp db, though this is the kind of thing I was hoping I wouldn't have to do anymore.

[Alex Ortiz]

Based on my research, 5.7.3 should be supported: https://confluence.atlassian.com/adminjiraserver/supported-platforms-938846830.html

Have you tried updating the Jira Cloud Migration Assistant from your "Manage Apps"?  

I would also reach out to Atlassian support as they may be able to take a log file and pinpoint the actual error message.

I know this doesn't answer your question, but hopefully gives you some things to look at.  

I had to stand up a new server and had to get a full export of Jira and reimport before I could successfully use the migration assistant. 

[Nick Smith]

Hi Alex,

Thanks for the input. Jira Cloud Migration Assistant is fully up to date on the source server.

I did reach out to JIRA support Thursday. Saturday night I received a response where it was fairly obvious whoever responded didn't read my email with any carefulness at all. I responded respectfully restating the need I had. Still waiting for a response.

Monday/Tuesday was the time allotted to perform this migration...but I may instead use the time researching alternative solutions from other companies that may take support a bit more seriously if I don't receive a substantive and timely response—unfortunately experience has shown Atlassian support does everything they can to avoid actually being helpful.

I would be interested in how you did the export/reimport when you have to perform this migration and what this actually fixed (db corruption or something)?

[Alex Ortiz]

My specific case was I had a SQL Server that was just too old.  

Are you running your Jira on a VM?  If so, it might be easier for you to take a snapshot and then upgrade your SQL database there.  

Alternatively, I would recommend you get a new VM, install a fresh copy of Jira and configure a fresh DB.

https://confluence.atlassian.com/adminjiraserver/backing-up-data-938847673.html

I've found that doing just the XML backup is enough since I usually perform migrations after hours.  But, it's up to you.

To restore: 

https://confluence.atlassian.com/adminjiraserver/restoring-data-from-an-xml-backup-938847707.html#Restoringdatafromanxmlbackup-restorexmldata

[Nick Smith]

Hi Alex,

Thanks for the ideas.

The database is a MySQL hosted on Amazon RDS and the server JIRA server itself is hosted on an Amazon EC2 instance so I could reroll an updated database and a clean JIRA install for the import if I really need to, but I was really hoping to avoid spending so much time on just getting the data out (I could see that taking me an entire day), but it does give me a contingency plan to discuss so I appreciate that.

[Nick Smith]

I upgraded the Jira Server to 96 vCPU/384GB ram and MySQL database it uses to 16vCPU/64GB RAM. This is a manageable cost for a couple days to get this process completed.

This seems to have helped, as it doesn't seem to completely kill the server availability.

However, when I tell it to run it gives an "idiot light" in the Checking Your Migration section that says Checks are "Running". NO indications of how long this will take or what it is doing. I sometimes wonder if the people who write tools like this have ever dealt with a deadline in their life.

Anyway, I'll keep watching this for a bit and hopefully it will make some progress. If anyone has insight on how long this process takes that would be helpful (or maybe a way to track its progress).

[Pramodh M]

Hi @Nick Smith 

First of all, if you are in the process of planning please don't do this activity on Prod Instance.

Perform all the activities on the test instance and create a playbook as to how the data is migrated to Cloud.

https://www.atlassian.com/migration/plan/cloud-guide

For the error that you are getting, increase the resources of both DB and Jira to maximize the performance speed of the Migration assistant App.

Thanks,
Pramodh

[Nick Smith]

I already quadrupled the resources of the server from what it has been stably running with for years. We are a very, very small organization relatively speaking (< 10 users, less than 15,000 issues total).

32GB RAM / 8 aws vCPUs.