Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Deleted user
Level
0 / 0 points
Next:
badges earned

Your Points Tracker
Challenges
Leaderboard
  • Global
  • Feed

Badge for your thoughts?

You're enrolled in our new beta rewards program. Join our group to get the inside scoop and share your feedback.

Join group
Recognition
Give the gift of kudos
You have 0 kudos available to give
Who do you want to recognize?
Why do you want to recognize them?
Kudos
Great job appreciating your peers!
Check back soon to give more kudos.

Past Kudos Given
No kudos given
You haven't given any kudos yet. Share the love above and you'll see it here.

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

diy backup bug in scripts?

Hi,

   I am trying to set up diy backup for my bitbucket server instance. I am a newbie, so please be gentle....

I have set up a test bitbucket instance with some users which appears to be working fine. I have created a diy-backup script as per the instructions and it appears to run, but the diagnostic output says Elasticsearch succeeded, but then backup failed; Note that I am not using any of the elasticsearch variables as it is a standard installation, so I am assuming that it shold be calling the no_op version of the Elasticsearch backups. (my example is based upon the postgres-rsync example script)

Running the command ./bitbucket-diy-backup.sh reports;

[common.ls /bitbucket-sh:17 -> source][bb] DEBUG: Using vars file ; './bitbucket.diy-backup.vars.sh'

(rsync stuff - works)

[common.sh:114 -> lock_bitbucket][bb] DEBUG: Running curl -L -s -f -u ******:****** http://localhost:7990/mvc/maintenance.lock
[common.sh:114 -> lock_bitbucket][bb] INFO: Bitbucket has been locked for maintenance. It can be unlocked with:

(etc.)

[bitbcket-diy-backup.sh:118 -> run_in_bg][bb] DEBUG: Started Elasticsearch backup (PID=2823)
[common.sh:120 -> backup_wait][bb] INFO: Waiting for Bitbucket to become ready to be backed up
[bitbcket-diy-backup.sh:0 -> main][bb] INFO: Backing up the database and filesystem in parallel
rm: cannot remove '/backup-root/bitbucket-db/': No such file or directory
[bitbucket-diy-backup.sh:123 -> run_in_bg][bb] DEBUG: Started Database backup (PID=2852)
[bitbucket-diy-backup.sh:124 -> run_in_bg][bb] DEBUG: Started Disk backup (PID=2858)
[disk-rsync.sh:23 -> perform_rsync_home_directory][bb] DEBUG: Running rsync -avh --delete --delete-excluded --exclude=/caches/ --exclude=/data/db.* --exclude=/shared/data/db.* --exclude=/search/data/ --exclude=/shared/search/data/ --exclude=/export/ --exclude=/log/ --exclude=/plugins/.*/ --exclude=/tmp --exclude=/.lock --exclude=/shared/.lock /home/bbuser/bbdata/ /backup-root/bitbucket-home
[bitbucket-diy-backup.sh:127 -> wait_for_bg_jobs][bb] DEBUG: Waiting for Databse backup (PID=2852)
sending incremental file list
[bitbucket-diy-backup.sh:127 -> wait_for_bg_jobs][bb] DEBUG: Waiting for Disk backup (PID=2858)
analytics-logs/16588fa71a97d9104d1549704b342b16.090b4a5c-a7df-430f-afdf-80d2edc0eeac.atlassian-analytics.log

sent 10.76K bytes received 151 bytes 21.81K bytes/sec
total size is 279.91K speedup is 25.67
[bitbucket-diy-backup.sh:127 -> wait_for_bg_jobs][bb] DEBUG: Disk backup finished successfully (PID=2858)
[bitbucket-diy-backup.sh:127 -> wait_for_bg_jobs][bb] DEBUG: Waiting for Elasticsearch backup (PID=2823)
[bitbucket-diy-backup.sh:127 -> wait_for_bg_jobs][bb] DEBUG: Elasticsearch backup finished successfully (PID=2823)
[bitbucket-diy-backup.sh:127 -> wait_for_bg_jobs][bb] ERROR: Database Backup failed with status 1 (PID=2823)
[common.sh:129 -> unlock_bitbucket][bb] DEBUG: running curl -L -s -f - u ******:****** http://loclahost:7990/mvc/maintenance/lock?token=214e9737d46b09adefda6668367458c7fa9abb3a
[bitbucket-diy-backup.sh:130 -> cleanup_incomplete_backup][bb] DEBUG: Cleaning up after failed backup
[bitbucket-diy-backup.sh:0 -> main][bb] ERROR: Backing up bitbucket failed
[common.sh:1 -> run_cleanup][bb] DEBUG: Running cleanup jobs...
(no further output!)


....but it doesnt cleanup properly either. I have to reboot the machine to get out of this one as even entering the correct backup token to unlock it still leaves the backup hung.

My major problem which I can;t get my head around is if the backup succedded why is it reporting an exit code of 1 and what can I do to fix it?

 

2 answers

I see exactly the same behavior 6.8.1 version. Any solution is much appreciated.

Confirmed: What can say - exactly same here with MySQL. I try to use it to backup Bitbucket Docker instance. It's fine - all the files from named volume are processed, but it fails on database part.:

sent 13.74G bytes received 3.55M bytes 68.56M bytes/sec
total size is 13.72G speedup is 1.00
[common.sh:114 -> lock_bitbucket][9da8c79a4313] DEBUG: Running curl -L -s -f -u ******:****** http://atlassian-test:7990/mvc/maintenance/lock
[common.sh:114 -> lock_bitbucket][9da8c79a4313] INFO: Bitbucket has been locked for maintenance. It can be unlocked with:
[common.sh:114 -> lock_bitbucket][9da8c79a4313] INFO: curl -u ... -X DELETE -H 'Content-type:application/json' 'http://atlassian-test:7990/mvc/maintenance/lock?token=271fe3ec99a84a1997eff6e29cdac3664a9f2db4'
[common.sh:115 -> backup_start][9da8c79a4313] DEBUG: Running curl -L -s -f -u ******:****** http://atlassian-test:7990/mvc/admin/backups?external=true
[common.sh:115 -> backup_start][9da8c79a4313] INFO: Bitbucket server is now preparing for backup. If the backup task is cancelled, Bitbucket Server should be notified that backup was terminated by executing the following command:
[common.sh:115 -> backup_start][9da8c79a4313] INFO: curl -u ... -X POST -H 'Content-type:application/json' 'http://atlassian-test:7990/mvc/maintenance?token=c81a803cef12948ebb0ffe9863547c7d151e4f34'
[common.sh:115 -> backup_start][9da8c79a4313] INFO: This will also terminate the backup process in Bitbucket Server. Note that this will not unlock Bitbucket Server from maintenance mode.
[bitbucket.diy-backup.sh:118 -> run_in_bg][9da8c79a4313] DEBUG: Started Elasticsearch backup (PID=85)
[common.sh:120 -> backup_wait][9da8c79a4313] INFO: Waiting for Bitbucket to become ready to be backed up
[bitbucket.diy-backup.sh:0 -> main][9da8c79a4313] INFO: Backing up the database and filesystem in parallel
rm: can't remove '/var/cache/bitbucket-diy/bitbucket-db': No such file or directory
[bitbucket.diy-backup.sh:123 -> run_in_bg][9da8c79a4313] DEBUG: Started Database backup (PID=111)
[bitbucket.diy-backup.sh:124 -> run_in_bg][9da8c79a4313] DEBUG: Started Disk backup (PID=116)
[disk-rsync.sh:23 -> perform_rsync_home_directory][9da8c79a4313] DEBUG: Running rsync -avh --delete --delete-excluded --exclude=/caches/ --exclude=/data/db.* --exclude=/shared/data/db.* --exclude=/search/data/ --exclude=/shared/search/data/ --exclude=/export/ --exclude=/log/ --exclude=/plugins/.*/ --exclude=/tmp --exclude=/.lock --exclude=/shared/.lock /mnt/server-data /var/cache/bitbucket-diy/bitbucket-home/
[bitbucket.diy-backup.sh:127 -> wait_for_bg_jobs][9da8c79a4313] DEBUG: Waiting for Disk backup (PID=116)
sending incremental file list
server-data/caches/tag-index/
server-data/caches/tag-index/cache-entry14465606170308675900
server-data/export/
server-data/export/backup-backup-stash-20191211-085756-098Z.zip
server-data/log/atlassian-bitbucket-access.log
server-data/log/atlassian-bitbucket.log
server-data/log/audit/atlassian-bitbucket-audit.log

sent 7.40M bytes received 35.32K bytes 1.35M bytes/sec
total size is 13.72G speedup is 1,845.82
[bitbucket.diy-backup.sh:127 -> wait_for_bg_jobs][9da8c79a4313] DEBUG: Disk backup finished successfully (PID=116)
[bitbucket.diy-backup.sh:127 -> wait_for_bg_jobs][9da8c79a4313] DEBUG: Waiting for Database backup (PID=111)
[bitbucket.diy-backup.sh:127 -> wait_for_bg_jobs][9da8c79a4313] DEBUG: Waiting for Elasticsearch backup (PID=85)
[bitbucket.diy-backup.sh:127 -> wait_for_bg_jobs][9da8c79a4313] DEBUG: Elasticsearch backup finished successfully (PID=85)
[bitbucket.diy-backup.sh:127 -> wait_for_bg_jobs][9da8c79a4313] ERROR: Database backup failed with status 1 (PID=85)
[common.sh:129 -> unlock_bitbucket][9da8c79a4313] DEBUG: Running curl -L -s -f -u ******:****** http://atlassian-test:7990/mvc/maintenance/lock?token=271fe3ec99a84a1997eff6e29cdac3664a9f2db4
[bitbucket.diy-backup.sh:130 -> cleanup_incomplete_backup][9da8c79a4313] DEBUG: Cleaning up after failed backup
[bitbucket.diy-backup.sh:0 -> main][9da8c79a4313] ERROR: Backing up Bitbucket failed
[common.sh:1 -> run_cleanup][9da8c79a4313] DEBUG: Running cleanup jobs...


Also it fails with cleanup job and I need to recreate Bitbucket container.

How were you able to resolve this issue. I ran into a similar issue while running the DIY script.

Hi Ajay,

    I did start looking into this and started putting debug in to the script to figure out what was going on but I am overloaded with work and so never completed the task. As our team is only 4 member strong what I do is manually shut down the services and the manually backup the bitbucket filesystem and manually backup the Postgres database. 

 

One day I hope to get back on to trying to figure out the issue, but for the moment I can back up using this method. If I ever find the answer I will post it here.

Thanks for your Reply Paul. I will post an answer if i get this script to work too.

 

~Ajay

Quick fix for this bug (but only for mysql). Not tested restoring, but it's an archive so manual restoring isn't an issue:

diff --git a/database-mysql.sh b/database-mysql.sh
index 8e4b29c..4b315cb 100644
--- a/database-mysql.sh
+++ b/database-mysql.sh
@@ -20,9 +20,9 @@ function prepare_backup_db {
}

function backup_db {
- rm -r "${BITBUCKET_BACKUP_DB}"
- run mysqldump ${MYSQL_HOST_CMD} ${MYSQL_BACKUP_OPTIONS} -u "${MYSQL_USERNAME}" -p"${MYSQL_PASSWORD}" \
- --databases "${BITBUCKET_DB}" > "${BITBUCKET_BACKUP_DB}"
+ mkdir -p "${BITBUCKET_BACKUP_DB}"
+ mysqldump ${MYSQL_HOST_CMD} ${MYSQL_BACKUP_OPTIONS} -u "${MYSQL_USERNAME}" -p"${MYSQL_PASSWORD}" \
+ --databases "${BITBUCKET_DB}" > "${BITBUCKET_BACKUP_DB}/${BITBUCKET_DB}.sql"
}

function prepare_restore_db {

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket

Calling any interview participants for Bitbucket Data Center

Hi everyone,  We are looking to learn more about development teams’ workflows and pain points, especially around DevOps, integrations, administration, scale, security, and the related challeng...

504 views 6 4
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you