stash database backup failed using backup client

[Vivian Zhang]

Backup was working for some time. But it started failing after we migrated more repositories to stash. The error we received is:

WARN [threadpool:thread-995154] admin 1320x3126501x1 1utuvpt 172.16.100.94,0:0:0:0:0:0:0:1 "POST /mvc/admin/backups HTTP/1.1" c.a.s.i.m.LatchAndDrainDatabaseStep The DataSource could not be drained; some database connections are still open. Aborting migration...

WARN [threadpool:thread-995154] admin 1320x3126501x1 1utuvpt 172.16.100.94,0:0:0:0:0:0:0:1 "POST /mvc/admin/backups HTTP/1.1" c.a.s.i.m.DefaultMaintenanceTaskMonitor BACKUP maintenance has failed (Cause: BackupException: A backup file could not be created.)

...

Caused by: com.atlassian.stash.internal.migration.MigrationException: One or more database connections were not closed within the allotted timeout. To prevent corruption due to inconsistent state, the migration has been aborted. Please try your migration again when the system is under less load.

The time out happened within 1min when it tried to drain the data source. Is there a way to increase the timeout? I don't see such an option in the backup client config file.

the external database we are using is mysql.

Thanks.

[Vivian Zhang]

I got the NPE error. Looks like it's a known issue for back up client 1.3.0. I'm planning to install 1.3.1 which has the debug enabled and let Atlassian investigate more.

[Johannes Kilian]

It looks like that there are several issues with Stash Backup Client 1.3.0 ;-(

[Vivian Zhang]

I just got notified the final fix is just released. It can be downloaded at Atlassian Marketplace. Look for Stash Backup Client 1.3.1. I installed their beta release before this and it worked already. The release was to fix the NPE error.

[Johannes Kilian]

Thanks for the notification

[Johannes Kilian]

Thanks for the notification

[Johannes Kilian]

@Xavier:

If you are using Stash 3.2.2 you have to update to latest Stash Backup Client 1.3.0

But ... There is a known issue with Stash Backup Client 1.3.0 - which fails due to locked ".lock" file.

2014-08-27 04:40:42,852 ERROR [main] c.a.s.i.b.c.layout.DefaultStashHome E:\GitRepo\shared\.lock could not be read

This can be resolved by excluding those lock files.

For the time being, you can add this line in <tt>backup-config.properties</tt> to exclude <tt>.lock</tt> files as a workaround:

stash.home.excludes=**/.lock

(suggested by Atlassian support desk)

[Xabier Davila]

Hi Vivian, did you end up getting to the bottom of this issue? I'm having the same problem with Stash 3.2.2 using PostgreSQL