How do I resolve this problem in a hipchat 2.4.0 upgrade?

[JohnTrammell]

The upgrade process results in this error:

root@hipchat-test:/home/admin# hipchat upgrade --upgrade --force-upgrade
Expected sha512sum and filename: e86b4eff1824b3dac6b0b90c9af1a4ea5cdd265a06980f16ee35247cffd12105752b23d71bd166c013d9706150d34b7ed0c8654f030af590efe88f59f2ab4fa8 hcs-2.4.0-2018.05.10.074812.tar.gz
Downloading the update, this may take a few minutes depending on your network speeds
Calculated sha512sum: e86b4eff1824b3dac6b0b90c9af1a4ea5cdd265a06980f16ee35247cffd12105752b23d71bd166c013d9706150d34b7ed0c8654f030af590efe88f59f2ab4fa8 hcs-2.4.0-2018.05.10.074812.tar.gz
Updating system configuration, this may take a few minutes...
There was an unexpected error: Command '['sudo', '/opt/atlassian/hipchat/sbin/cs']' returned non-zero exit status 1
root@hipchat-test:/home/admin#

Looking at /var/log/chef.log, I see (possibly sensitive things redacted):

================================================================================
Error executing action `run` on resource 'bash[update_mysql_password]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-scriptREDACTED" ----
STDOUT: [mariadb_grants] Validating password specified is indeed set
STDERR: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
---- End output of "bash" "/tmp/chef-scriptREDACTED" ----
Ran "bash" "/tmp/chef-scriptREDACTED" returned 1

Resource Declaration:
---------------------
# In /hipchat-scm/chef-repo/cookbooks/mariadb/recipes/server.rb

47: bash 'update_mysql_password' do
 48: code <<-EOH
 49: /etc/mariadb_grants #{DBPASSWORD} #{DBUSER}
 50: EOH
 51: action :run
 52: end
 53:

Compiled Resource:
------------------
# Declared in /hipchat-scm/chef-repo/cookbooks/mariadb/recipes/server.rb:47:in `from_file'

bash("update_mysql_password") do
 action [:run]
 retries 0
 retry_delay 2
 guard_interpreter :default
 command "\"bash\" \"/tmp/chef-scriptREDACTED\""
 backup 5
 returns 0
 code " /etc/mariadb_grants REDACTED root\n"
 interpreter "bash"
 cookbook_name :mariadb
 recipe_name "server"
end

[2018-06-13T19:57:48+00:00] INFO: Running queued delayed notifications before re-raising exception
[2018-06-13T19:57:48+00:00] ERROR: Running exception handlers
[2018-06-13T19:57:48+00:00] ERROR: Exception handlers complete
[2018-06-13T19:57:48+00:00] FATAL: Stacktrace dumped to /usr/local/src/chef-stacktrace.out
[2018-06-13T19:57:48+00:00] ERROR: bash[update_mysql_password] (mariadb::server line 47) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-scriptREDACTED" ----
STDOUT: [mariadb_grants] Validating password specified is indeed set
STDERR: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
---- End output of "bash" "/tmp/chef-scriptREDACTED" ----
Ran "bash" "/tmp/chef-scriptREDACTED" returned 1
[2018-06-13T19:57:49+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
[2018-06-13T19:57:49+00:00] INFO: Allow custom /etc files
[2018-06-13T19:57:49+00:00] INFO: Re-enabling monitoring
[2018-06-13T19:57:49+00:00] ERROR: Error: Configuration failure

Any suggestions on how to resolve the root password issue? I assume that's at the bottom of the upgrade failure.

Thanks, John

[Arbi Dridi]

Hi John, 

Can you give the resolution steps of the following article a try? 
Hipchat Server database passwords do not match

We're working on making this publicly documented today. 

Thanks.

[JohnTrammell]

Looking at /etc/mariadb_grants (an odd location for a script IMO), I saw these lines:

GRANTED='/hipchat/config/.sql.root.grant'
...

if [ ! -f $GRANTED ] ; then
...

Looking at the filesystem, I saw that file `/hipchat/config/.sql.root.grant` existed, so I removed it and re-ran the script. It appeared to work successfully. I'm not sure where the file came from, perhaps a previous upgrade?