Stash Backup Hangs

Hello,

The last few days the backup for stash seems to hang (ran perfectly for more then a month every night). Checking the backup log the last thing i see is this:

.............. etc

This is using the DIY backup scripts, looking into the scripts i think this is the part its getting stuck on:

function stash_backup_wait {

STASH_PROGRESS_DB_STATE="AVAILABLE"

STASH_PROGRESS_SCM_STATE="AVAILABLE"

print -n "[${STASH_URL}] .INFO: Waiting for DRAINED state "

while [ ${STASH_PROGRESS_DB_STATE} != "DRAINED" -a ${STASH_PROGRESS_SCM_STATE} != "DRAINED" ]; do

print -n "."

STASH_PROGRESS_RESULT=`curl -s ${CACERT} -f ${STASH_HTTP_AUTH} -X GET -H "X-Atlassian-Maintenance-Token: ${STASH_LOCK_TOKEN}" -H "Accept: application/json" -H "Content-type: application/json" "${STASH_URL}/mvc/maintenance"`

if [ -z "${STASH_PROGRESS_RESULT}" ]; then

bail "[${STASH_URL}] ERROR: Unable to check for backup progress"

fi

STASH_PROGRESS_DB_STATE=`echo ${STASH_PROGRESS_RESULT} | jq -r '.["db-state"]'`

STASH_PROGRESS_SCM_STATE=`echo ${STASH_PROGRESS_RESULT} | jq -r '.["scm-state"]'`

done

print "done"

info "db state '${STASH_PROGRESS_DB_STATE}'"

info "scm state '${STASH_PROGRESS_SCM_STATE}'"

}

If i run the script by hand with the set +x option its running fine about 60% of the time and failing the other 40% just getting stuck.

I also tried the java backup client also here its giving odd results like this:

2014-08-11 21:00:55,158 INFO [main] c.a.s.i.b.client.spring.SpringMain Initializing

2014-08-11 21:00:56,150 INFO [main] c.a.s.i.b.client.DefaultBackupClient Contacting Stash

2014-08-11 21:00:56,540 INFO [main] c.a.s.i.b.client.RestStashService Using Stash 3.2.0

2014-08-11 21:00:56,662 INFO [main] c.a.s.i.b.client.RestStashService Stash has been locked for maintenance. It may be unlocked with token: b437f1744c372fd57609175ce96af48db801b7aa

2014-08-11 21:00:56,721 INFO [main] c.a.s.i.b.client.RestStashService Starting database backup on Stash. It may be cancelled with token: 941415bc19e7f8f29cfc185ca873a06e10089fda

2014-08-11 21:01:01,414 INFO [main] c.a.s.i.b.c.layout.DefaultStashHome Verifying Stash home

2014-08-11 21:01:03,281 INFO [main] c.a.s.i.b.c.layout.DefaultStashHome Verifying Stash home

2014-08-11 21:01:06,157 INFO [main] c.a.s.i.b.c.layout.DefaultStashHome Verifying Stash home

2014-08-11 21:01:11,270 INFO [main] c.a.s.i.b.c.layout.DefaultStashHome Backing up Stash home

2014-08-11 21:01:16,150 INFO [main] c.a.s.i.b.c.layout.DefaultStashHome Backing up Stash home

2014-08-11 21:01:20,052 INFO [main] c.a.s.i.b.client.DefaultBackupClient Waiting on Stash database backup to complete

2014-08-11 21:02:15,218 INFO [main] c.a.s.i.b.client.RestStashService Cancelling database backup with code: 941415bc19e7f8f29cfc185ca873a06e10089fda

2014-08-11 21:02:15,232 INFO [main] c.a.s.i.b.client.RestStashService Database backup was already manually cancelled on Stash

2014-08-11 21:02:15,233 INFO [main] c.a.s.i.b.client.RestStashService Unlocking Stash using token: b437f1744c372fd57609175ce96af48db801b7aa

2014-08-11 21:02:15,253 ERROR [main] c.a.s.i.backup.client.BackupMain A backup could not be created. Reason: Database backup on Stash does not appear to have been started

com.atlassian.stash.internal.backup.client.BackupException: Database backup on Stash does not appear to have been started

at com.atlassian.stash.internal.backup.client.DefaultBackupClient.throwIfNoLatestTask(DefaultBackupClient.groovy:159) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.CachedMaintenanceReporter.waitForStatus(CachedMaintenanceReporter.groovy:87) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.DefaultBackupClient.waitForBackupTaskCompletion(DefaultBackupClient.groovy:198) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.DefaultBackupClient.downloadDbZipOnBackupCompletion(DefaultBackupClient.groovy:111) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.DefaultBackupClient.archiveHome(DefaultBackupClient.groovy:82) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.FixedRateProgressPublisher.publishWhileExecuting(FixedRateProgressPublisher.groovy:34) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.RestStashService.backup(RestStashService.groovy:66) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.RestStashService.lock(RestStashService.groovy:174) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.DefaultBackupClient.backup(DefaultBackupClient.groovy:50) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.spring.SpringMain.run(SpringMain.groovy:28) [stash-backup-core-1.3.0.jar:na]

at com.atlassian.stash.internal.backup.client.spring.SpringMain.run(SpringMain.groovy) [stash-backup-core-1.3.0.jar:na]

at com.atlassian.stash.internal.backup.client.BackupMain.main(BackupMain.groovy:38) [stash-backup-client.jar:1.3.0]

Any idea why it seems to hang all of a sudden?
Its running during the night, all postgres connections seem idle during that time
Thanks in advance,
Joost

1 answer

This widget could not be displayed.

Hi Joost,

Could you please raise an issue on https://support.atlassian.com?

Thank you!

Best regards,

Thiago Bomfim

Atlassian Support

Suggest an answer

Log in or Sign up to answer
Community showcase
Published Aug 21, 2018 in Bitbucket

Branch Management with Bitbucket

As a project manager, I have discovered that different developers want to bring their previous branching method with them when they join the team. Some developers are used to performing individual wo...

1,303 views 8 11
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you