Stash Backup Hangs

Hello,

The last few days the backup for stash seems to hang (ran perfectly for more then a month every night). Checking the backup log the last thing i see is this:

.............. etc

This is using the DIY backup scripts, looking into the scripts i think this is the part its getting stuck on:

function stash_backup_wait {

STASH_PROGRESS_DB_STATE="AVAILABLE"

STASH_PROGRESS_SCM_STATE="AVAILABLE"

print -n "[${STASH_URL}] .INFO: Waiting for DRAINED state "

while [ ${STASH_PROGRESS_DB_STATE} != "DRAINED" -a ${STASH_PROGRESS_SCM_STATE} != "DRAINED" ]; do

print -n "."

STASH_PROGRESS_RESULT=`curl -s ${CACERT} -f ${STASH_HTTP_AUTH} -X GET -H "X-Atlassian-Maintenance-Token: ${STASH_LOCK_TOKEN}" -H "Accept: application/json" -H "Content-type: application/json" "${STASH_URL}/mvc/maintenance"`

if [ -z "${STASH_PROGRESS_RESULT}" ]; then

bail "[${STASH_URL}] ERROR: Unable to check for backup progress"

fi

STASH_PROGRESS_DB_STATE=`echo ${STASH_PROGRESS_RESULT} | jq -r '.["db-state"]'`

STASH_PROGRESS_SCM_STATE=`echo ${STASH_PROGRESS_RESULT} | jq -r '.["scm-state"]'`

done

print "done"

info "db state '${STASH_PROGRESS_DB_STATE}'"

info "scm state '${STASH_PROGRESS_SCM_STATE}'"

}

If i run the script by hand with the set +x option its running fine about 60% of the time and failing the other 40% just getting stuck.

I also tried the java backup client also here its giving odd results like this:

2014-08-11 21:00:55,158 INFO [main] c.a.s.i.b.client.spring.SpringMain Initializing

2014-08-11 21:00:56,150 INFO [main] c.a.s.i.b.client.DefaultBackupClient Contacting Stash

2014-08-11 21:00:56,540 INFO [main] c.a.s.i.b.client.RestStashService Using Stash 3.2.0

2014-08-11 21:00:56,662 INFO [main] c.a.s.i.b.client.RestStashService Stash has been locked for maintenance. It may be unlocked with token: b437f1744c372fd57609175ce96af48db801b7aa

2014-08-11 21:00:56,721 INFO [main] c.a.s.i.b.client.RestStashService Starting database backup on Stash. It may be cancelled with token: 941415bc19e7f8f29cfc185ca873a06e10089fda

2014-08-11 21:01:01,414 INFO [main] c.a.s.i.b.c.layout.DefaultStashHome Verifying Stash home

2014-08-11 21:01:03,281 INFO [main] c.a.s.i.b.c.layout.DefaultStashHome Verifying Stash home

2014-08-11 21:01:06,157 INFO [main] c.a.s.i.b.c.layout.DefaultStashHome Verifying Stash home

2014-08-11 21:01:11,270 INFO [main] c.a.s.i.b.c.layout.DefaultStashHome Backing up Stash home

2014-08-11 21:01:16,150 INFO [main] c.a.s.i.b.c.layout.DefaultStashHome Backing up Stash home

2014-08-11 21:01:20,052 INFO [main] c.a.s.i.b.client.DefaultBackupClient Waiting on Stash database backup to complete

2014-08-11 21:02:15,218 INFO [main] c.a.s.i.b.client.RestStashService Cancelling database backup with code: 941415bc19e7f8f29cfc185ca873a06e10089fda

2014-08-11 21:02:15,232 INFO [main] c.a.s.i.b.client.RestStashService Database backup was already manually cancelled on Stash

2014-08-11 21:02:15,233 INFO [main] c.a.s.i.b.client.RestStashService Unlocking Stash using token: b437f1744c372fd57609175ce96af48db801b7aa

2014-08-11 21:02:15,253 ERROR [main] c.a.s.i.backup.client.BackupMain A backup could not be created. Reason: Database backup on Stash does not appear to have been started

com.atlassian.stash.internal.backup.client.BackupException: Database backup on Stash does not appear to have been started

at com.atlassian.stash.internal.backup.client.DefaultBackupClient.throwIfNoLatestTask(DefaultBackupClient.groovy:159) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.CachedMaintenanceReporter.waitForStatus(CachedMaintenanceReporter.groovy:87) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.DefaultBackupClient.waitForBackupTaskCompletion(DefaultBackupClient.groovy:198) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.DefaultBackupClient.downloadDbZipOnBackupCompletion(DefaultBackupClient.groovy:111) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.DefaultBackupClient.archiveHome(DefaultBackupClient.groovy:82) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.FixedRateProgressPublisher.publishWhileExecuting(FixedRateProgressPublisher.groovy:34) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.RestStashService.backup(RestStashService.groovy:66) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.RestStashService.lock(RestStashService.groovy:174) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.DefaultBackupClient.backup(DefaultBackupClient.groovy:50) ~[stash-backup-client.jar:1.3.0]

at com.atlassian.stash.internal.backup.client.spring.SpringMain.run(SpringMain.groovy:28) [stash-backup-core-1.3.0.jar:na]

at com.atlassian.stash.internal.backup.client.spring.SpringMain.run(SpringMain.groovy) [stash-backup-core-1.3.0.jar:na]

at com.atlassian.stash.internal.backup.client.BackupMain.main(BackupMain.groovy:38) [stash-backup-client.jar:1.3.0]

Any idea why it seems to hang all of a sudden?
Its running during the night, all postgres connections seem idle during that time
Thanks in advance,
Joost

1 answer

0 vote

Hi Joost,

Could you please raise an issue on https://support.atlassian.com?

Thank you!

Best regards,

Thiago Bomfim

Atlassian Support

Suggest an answer

Log in or Sign up to answer
How to earn badges on the Atlassian Community

How to earn badges on the Atlassian Community

Badges are a great way to show off community activity, whether you’re a newbie or a Champion.

Learn more
Community showcase
Posted Jun 12, 2018 in Bitbucket

Do you use any Atlassian products for your personal projects?

After spinning my wheels trying to get organized enough to write a book for National Novel Writing Month (NaNoWriMo) I took my affinity for Atlassian products from my work life and decided to tr...

22,746 views 26 12
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you