Jira - Long upgrade process. Any way to expedite? Stuck on custom field update.

[Richard Thomas]

We are trying to upgrade 4.3.4 to 5.1.1. When we ran the upgrade on a test box, the "updating custom field" part of the install took 1 1/2 hours. Step 4 of 41 took 7 1/2 minutes.

We attempted an upgrade on our production box this weekend. Step 4 was still running after nearly 40 minutes. Thinking something must be wrong, we aborted the install, rolled everything back and started again. After the third try, we cancelled the update and will be reexamining things to see how to proceed. Our DBA guy says that during the install, he saw two SQL processes blocking each other (I will include a more detailed statement from him later).

Could someone suggest a way to make the install proceed in a timely manner?

[Richard Thomas]

The DBA did the following:

"I have restored the DB and set parallelism to only use 1 instead of all"

Then the update (direct to 5.1.1) proceeded as it had previously on the test box and completed within 1 hour. We will be scheduling another run at the upgrade presently.

[Richard Thomas]

Production upgrade successful.

[Nic Brough -Adaptavist-]

That pre-empts the next question of "what is different between Production and Test" - it's service pack 1 causing the problem, or there's a fix/change in the service pack that the upgrade needed left unfixed.

We've not actually discussed what OS or even database you are on yet, and I think that's relevant in this case. Could you tell us a bit more about them? And what the service pack was applied to?

(I'm assuming Windows is the base, but was it the application server, or the database server, or both? And was it the OS service pack, or something for the database?)

[Richard Thomas]

So we rolled everything back again and tried the upgrade to 4.4. Same exact problem. The log messages were different but I believe it was in the same place. The log stated

2012-09-26 17:05:17,488 main INFO [atlassian.jira.upgrade.UpgradeManagerImpl] Performing Upgrade Task: Converting Custom field values for Select and MultiSelect types to store the id of the option rather than the value.

which I believe means it stalled in the same place (or just after). This section took seven minutes when we first attempted it. When we tried in production, it never came back. The difference was that service pack 1 was installed on production. Service pack 1 has been installed on the test server and since then, we are having this problem. Something in the upgrade process appears to be conflicting.

[Norman Abramovitz]

Sorry could not follow the conversation until now. You need to read through all the upgrade guides to see if anything might affect your upgrade process. The reason for picking on 4.4 is because there was major changes done in this release. You can see for yourself about the number of changes in 4.4 that was documented. Do not forget to check your plugins as well.

https://confluence.atlassian.com/display/JIRA/JIRA+4.4+Upgrade+Notes

I had issues between 4.3 and 4.4 due to the database changes. I also had issues with 5.0.5, 5.0.6 and 5.0.7 related to UPM not functioning correctly. I also had issues with the 5.1.2 and 5.1.3 where some process was blocking the upgrade but could not find the process to kill. I needed to reboot the machine to clear things up.

[Nic Brough -Adaptavist-]

Yes, I'd agree - poking at a critical system is not a good option. Messing around with debugging something that might never be fixed, tinkering with databases when we're not even sure where the porblem is, and so on, I'd skip it and do a two step upgrade. My instinct is to go with Norman's route - 4.4 first, 5.x next.

[Richard Thomas]

We will be having a meeting about this shortly. That may be the decided outcome but it's not just up to me. Unfortunately, this is considered a critical system so just poking at it until it works is probably not going to be an optimal solution.

[Nic Brough -Adaptavist-]

Mmm, that's a message for a DBA more than us, but fair enough.

Can you try an intermediate update?

We could be chasing this for ages and if it is a bug in the process of jumping too many versions, I doubt it's going to be fixed when there's a workaround of "take smaller jumps in version".

[Richard Thomas]

This is from our DB guy:

There were no error messages, but as the upgrades were being performed on the server, the select statement used to supply the parameters for the update statements looked to be blocking the update statements themselves. The wait time type that Confio was giving was CXPacket (Parallelism),

[Richard Thomas]

No, this was a straight upgrade. I proceeded without incident on our test box other than the long time. Steps were simply to prepare backups/rollback, run install, watch logs.

[Nic Brough -Adaptavist-]

The SQL might be nice to see, but I think we need a brief list of the steps you are taking to do the upgrade too.

I've had problems with a custom field blocking the upgrade, fixed by making sure the plugin providing the type was installed in the new system before we even looked at the real data, and I know some version jumps can be a problem in some cases (e.g. I had problems doing 3.0.3 to 3.6, but 3.0.3 to 3.3.0 to 3.6 sailed through fine)

[Nic Brough -Adaptavist-]

Ah, apologies, I didn't see Norman's comment. It definitely sounds like you should try an intermediate version first.

[Richard Thomas]

Please see reply above.

[Norman Abramovitz]

You might want to list your steps. Like did you upgrade to 4.4 first before going to 5.x?