Stash Backup Hangs

[Joost Coelingh]

Hello,

The last few days the backup for stash seems to hang (ran perfectly for more then a month every night). Checking the backup log the last thing i see is this:

.............. etc

This is using the DIY backup scripts, looking into the scripts i think this is the part its getting stuck on:

function stash_backup_wait {

STASH_PROGRESS_DB_STATE="AVAILABLE"

STASH_PROGRESS_SCM_STATE="AVAILABLE"

print -n "[${STASH_URL}] .INFO: Waiting for DRAINED state "

while [ ${STASH_PROGRESS_DB_STATE} != "DRAINED" -a ${STASH_PROGRESS_SCM_STATE} != "DRAINED" ]; do

print -n "."

STASH_PROGRESS_RESULT=`curl -s ${CACERT} -f ${STASH_HTTP_AUTH} -X GET -H "X-Atlassian-Maintenance-Token: ${STASH_LOCK_TOKEN}" -H "Accept: application/json" -H "Content-type: application/json" "${STASH_URL}/mvc/maintenance"`

if [ -z "${STASH_PROGRESS_RESULT}" ]; then

bail "[${STASH_URL}] ERROR: Unable to check for backup progress"

fi

STASH_PROGRESS_DB_STATE=`echo ${STASH_PROGRESS_RESULT} | jq -r '.["db-state"]'`

STASH_PROGRESS_SCM_STATE=`echo ${STASH_PROGRESS_RESULT} | jq -r '.["scm-state"]'`

done

print "done"

info "db state '${STASH_PROGRESS_DB_STATE}'"

info "scm state '${STASH_PROGRESS_SCM_STATE}'"

}

If i run the script by hand with the set +x option its running fine about 60% of the time and failing the other 40% just getting stuck.

I also tried the java backup client also here its giving odd results like this:

2014-08-11 21:00:55,158 INFO [main] c.a.s.i.b.client.spring.SpringMain Initializing

2014-08-11 21:00:56,150 INFO [main] c.a.s.i.b.client.DefaultBackupClient Contacting Stash

2014-08-11 21:00:56,540 INFO [main] c.a.s.i.b.client.RestStashService Using Stash 3.2.0

2014-08-11 21:00:56,662 INFO [main] c.a.s.i.b.client.RestStashService Stash has been locked for maintenance. It may be unlocked with token: b437f1744c372fd57609175ce96af48db801b7aa

2014-08-11 21:00:56,721 INFO [main] c.a.s.i.b.client.RestStashService Starting database backup on Stash. It may be cancelled with token: 941415bc19e7f8f29cfc185ca873a06e10089fda

2014-08-11 21:01:01,414 INFO [main] c.a.s.i.b.c.layout.DefaultStashHome Verifying Stash home

2014-08-11 21:01:03,281 INFO [main] c.a.s.i.b.c.layout.DefaultStashHome Verifying Stash home

2014-08-11 21:01:06,157 INFO [main] c.a.s.i.b.c.layout.DefaultStashHome Verifying Stash home

2014-08-11 21:01:11,270 INFO [main] c.a.s.i.b.c.layout.DefaultStashHome Backing up Stash home

2014-08-11 21:01:16,150 INFO [main] c.a.s.i.b.c.layout.DefaultStashHome Backing up Stash home

2014-08-11 21:01:20,052 INFO [main] c.a.s.i.b.client.DefaultBackupClient Waiting on Stash database backup to complete

2014-08-11 21:02:15,218 INFO [main] c.a.s.i.b.client.RestStashService Cancelling database backup with code: 941415bc19e7f8f29cfc185ca873a06e10089fda

2014-08-11 21:02:15,232 INFO [main] c.a.s.i.b.client.RestStashService Database backup was already manually cancelled on Stash

2014-08-11 21:02:15,233 INFO [main] c.a.s.i.b.client.RestStashService Unlocking Stash using token: b437f1744c372fd57609175ce96af48db801b7aa

2014-08-11 21:02:15,253 ERROR [main] c.a.s.i.backup.client.BackupMain A backup could not be created. Reason: Database backup on Stash does not appear to have been started

com.atlassian.stash.internal.backup.client.BackupException: Database backup on Stash does not appear to have been started

at com.atlassian.stash.internal.backup.client.DefaultBackupClient.throwIfNoLatestTask(DefaultBackupClient.groovy:159) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.CachedMaintenanceReporter.waitForStatus(CachedMaintenanceReporter.groovy:87) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.DefaultBackupClient.waitForBackupTaskCompletion(DefaultBackupClient.groovy:198) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.DefaultBackupClient.downloadDbZipOnBackupCompletion(DefaultBackupClient.groovy:111) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.DefaultBackupClient.archiveHome(DefaultBackupClient.groovy:82) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.FixedRateProgressPublisher.publishWhileExecuting(FixedRateProgressPublisher.groovy:34) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.RestStashService.backup(RestStashService.groovy:66) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.RestStashService.lock(RestStashService.groovy:174) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.DefaultBackupClient.backup(DefaultBackupClient.groovy:50) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.spring.SpringMain.run(SpringMain.groovy:28) [stash-backup-core-1.3.0.jar:na]

at com.atlassian.stash.internal.backup.client.spring.SpringMain.run(SpringMain.groovy) [stash-backup-core-1.3.0.jar:na]

at com.atlassian.stash.internal.backup.client.BackupMain.main(BackupMain.groovy:38) [stash-backup-client.jar:1.3.0]

Any idea why it seems to hang all of a sudden?

Its running during the night, all postgres connections seem idle during that time

Thanks in advance,

Joost

[ThiagoBomfim]

Hi Joost,

Could you please raise an issue on https://support.atlassian.com?

Thank you!

Best regards,

Thiago Bomfim

Atlassian Support