Bitbucket Server restore hangs at end of processing large dataset

Question

Towards the end of processing a large dataset (1769713 changes, running for about 12 hours), the restore client hangs. The backup was created on a Windows 7 machine used to evaluate Bitbucket Server, and was being restored onto an Ubuntu Server 16.04 EC2 instance (m5.xlarge). The backup was created and restored using bitbucket-backup-client-3.3.4.

Is this related to a known issue, or is there something I'm missing? I started with a clean BB server install on the EC2 instance, deleted the home directory, set up backup-config.properties, then began the restore using '$sudo java -jar bitbucket-restore-client.jar ~/bitbucket-20181023-115550-271.tar'

Here are the last few entries of the log file before the hang:

2018-10-24 00:17:21,794 INFO [main] c.a.b.i.b.c.DefaultRestoreClient Processed 1763000 changes of 1769713
2018-10-24 00:18:29,249 INFO [main] c.a.b.i.b.c.DefaultRestoreClient Processed 1764000 changes of 1769713
2018-10-24 00:19:36,828 INFO [main] c.a.b.i.b.c.DefaultRestoreClient Processed 1765000 changes of 1769713
2018-10-24 00:20:44,560 INFO [main] c.a.b.i.b.c.DefaultRestoreClient Processed 1766000 changes of 1769713
2018-10-24 00:21:52,609 INFO [main] c.a.b.i.b.c.DefaultRestoreClient Processed 1767000 changes of 1769713
2018-10-24 00:23:00,605 INFO [main] c.a.b.i.b.c.DefaultRestoreClient Processed 1768000 changes of 1769713
2018-10-24 00:23:59,200 INFO [main] c.a.b.i.b.c.DefaultRestoreClient Processed 1769000 changes of 1769713

I have some output from attaching strace to the threads created from the main restore process if that might be any help.

chrisssc · Answer

Hi Rodrigo,

I ended up having success re-running the command, but specifying the sudo -su argument as the 'atlbitbucket' user - I can't be 100% sure that this was the problem/the fix, as I don't have the time to re-test it just now, but I thought I'd leave it here just in case anyone runs into the same issue.

Thanks for your answer,

Chris

Rodrigo M · Answer

Hi Chris!

According to Using the Bitbucket Server Backup Client document, you will also need to specify a path/to/bitbucket/home parameter in the jar command when restoring the instance. I also don't recommend running the command as a sudo user. You can try to enable DEBUG mode to check if there are any errors in the logs.

As an alternative, you can try to use another backup method mentioned in the Data recovery and backups document.
following

It's also possible to go through Migrating Bitbucket Server to another server document and follow the steps in order to migrate a Bitucket instance to another server.

Regards
Rodrigo