Upgrading taking a long time

[Jonny Polivka]

Upgrading from 7.4.11 to 7.13.2 and the installer gets to the state... 

"Configuring Confluence - Please wait a few moments while we configure Confluence"

...and stays there.  This last attempt I left it running for 90 minutes but the progress bar doesn't update.   Task manager shows the process is using CPU between 0 - 10% so it seems that it's not really hung.  (Or is some thread inside the installer is?)

The service is not running and the database is hosted on a separate server. 

Questions:

• Does the installer generate a log file when ran?  If so location and name?
• What is the installer doing at this state?
• Is there a way to know what exactly it's doing?

Environment

- Windows 2021 R2 x64  (4 cpu / 16GB ram)

- Confluence Server 7.4.11

- DB hosted on SQL Server

Thank you

[Florian Schüller]

Hello @Jonny Polivka ,

is there possibly an XDR (Extended Detection and Response) agent or something similar running on the application server?

I suspect that this could be the reason, as this agent checks all processes and file accesses/modifications to determine if they are legitimate.

[Brant Schroeder]

@Jonny Polivka 

I have seen this happen in the past when the application server did not have enough memory.  I have also seen it when the DB server does not have enough CPU.  What is happing on your DB server when this is going on, is the CPU tapped?  

[Jonny Polivka]

Our app server is 4 CPUs / 16 GB RAM.  I was watching Task Manager and memory was around 3GB the whole time the installer was running.  

Unknown on the sql server.  Will ask our DBA group...but does the installer communicate with the DB?  I didn't see or notice a lot of network activity viewing via Tack Manager.

[Brant Schroeder]

@Jonny Polivka at the end of the install all the db updates take place.  Did you check all you apps to make sure they are compatible?  Sometimes it will get hung on apps that can not upgrade.

[Jonny Polivka]

Hi @Brant Schroeder

I did run the Confluence Update Checker before and disabled two apps that it flagged; Atlassian Rest API Browser and SSO for Atlassian Data Center.  

Did some digging on the test environment and found a few i4j_nlog_*.log files in my user AppData\Local\Temp directory that look to have relevant data.  With those clues I get to ej technologies product install4J, "multi-platform Java installer builder."  

Moving back to the test instance, I ran the 7.13.2 installer again and saw that it created and started logging to file i4j_log_Confluence_<random number>.log file.  So that answers my question if the installer creates a log file.  

Comparing that log file with the existing ones they don't quite match up so they look to be logging two different events triggered by the installer.  (This is just a guess.)

I rolled back production instance so there is no way for me to check for this file.

[Brant Schroeder]

@Jonny Polivka is your test instance the same as production?  Do you sync the data?  Did it upgrade just fine?

[Jonny Polivka]

Hi @Brant Schroeder ,

They are not synced.    That crossed my mind over the holiday so I'll be spending time getting Test synced with Production data then upgrade.  

Test was synced with Production at one time but is overdue to have it happen again. 

[Brant Schroeder]

Did the test instance upgrade successfully?  Are the apps the same on both instances?

[Jonny Polivka]

Test instance did upgrade successfully but it's running an older snapshot of Production data.  

This week I'll revert Test back to 7.4.11, sync the data with Production and run an upgrade to 7.13.2 .   

[Jonny Polivka]

To finish off this thread...  

Finally completed upgrading both Test and Production to the latest LTS release (7.13.3 as off this writing.) 

Did take a copy of production data and use that for test.  The upgrade from 7.4.11 to 7.13.3 took about 3 hours.  The installer duration was shorter, but more than 90 minutes.  After deleting and installing files, it sat there doing something but unknown what for most of the duration.  (What is it doing during this time?) 

After installer completion, I reconfigured the settings it wiped out and started the service and the service sat there again for about 30 - 45 minutes.  Unknown what was happening but my guess is it was doing something with the db.  What's logged in the log files doesn't give any description of the activity and are not updated during this lengthy pause.  Viewing network and process activity was my only window.  

Same install experience with production but it took 80 minutes longer for both the installer and service.  Both have a long pause but are active.  Hard to say what is happening but looking at network and process activity I can see it's not unresponsive or hung.  My wish is that this activity would be logged or a heartbeat be written to the logs so it's known what is going on.  

So the short is.... test, test, test and plan for long outage for production.  

[Dirk Bromberg]

What I figured out is that a lot of mbr stuff is goning on the disk. So I think about rewriting file permissions on data folder. So it's before the change of database.