Stash backup client: Why do we have a timeout waiting for SCM drain?

I have currently this error message during my backup:

2014-12-31 02:01:28,478 ERROR [main] c.a.s.i.backup.client.BackupMain A backup could not be created. Reason: Operations from one or more SCMs did not finish within the allotted timeout. To prevent corruption due to inconsistent state, the backup has been aborted. Please try backup up again when the system is under less load.

 

I have set in stash-config.properties the following value

backup.drain.database.timeout=120

 

but:

Is this parameter also valid for scm?

What are the risk increasing this wait? Does the client start backing up things even if Database and SCM are not yet both drained?

 

Thanks for anyone who share any hint about this setting.

Regards,

Pierre

 

5 answers

1 vote

Hi Pierre,

Thanks for your observation. Indeed there was a mistake on the KB and I've corrected it so the right parameters (needs tweaking both on the server and the client):

That said, could you please follow the recommendations above and let us know how you go?

Best regards,
Thiago Bomfim
Atlassian Support

Edit: I also changed the KB title so it's more searchable.

Hi Pierre, Did you get a chance to look at the KB above? All the best, Thiago

Thank you Thiago. As setting something in stash-config.properties require to restart stash to be considered, I will do this operation at our next allowed maintenance frame. I keep you informed of the results after next tuesday.

Please do. Cheers!

It should give up after the timeout. At least the DIY backup script examples do that, the internal backup should behave the same.

The timeout exist to allow the backup to wait until internal buffers are flushed and the system is in a consistent state. They don't wait indefinitely to avoid deadlocks (usually a bug, but already happened). 

Thank you Mike for those 2 answers. I do understand that it is usefull to have a timeout to handle any unexpected behavior. Do you have more knowledge on the nature of this backup.drain.database.timeout settings? Is it valid for both db and scm or only for DB? As this drain is used to ensure that the system is in a consistent state, does it mean that the backup wait both drain before starting the sync, and so that there is no risk to increase it? Thanks and regards, Pierre

0 vote

Hi Pierre,

The description of this parameter is on the document below:

  • Stash config properties - Backup

    "Defines the number of seconds Stash will wait for connections to the database to drain and latch in preparation for a backup.

    Value is in SECONDS."

You might have come across the KB below, which gives you more details on how to troubleshoot it:

Best regards,
Thiago Bomfim

 

 

Thank you Thiago, I have read this "Stash config properties - Backup" file already. Does it mean that it concern only database drain and we cannot define any timeout for scm drain activities? My error message is about scm: "Operations from one or more SCMs did not finish within the allotted timeout." After increasing the timeout from 60 to 120 seconds, I do not reproduce the error, but I am not sure that there is a direct link as it seems to be a settings only for databases operations. I have missed this KB, I will consider this possibility to go in UPM safe mode. It is very interessting, thank you! Regards, Pierre Zurmely

Hi Pierre, Yes, it is the connections to the database only. I hope this has helped you. Thiago

Hello,

 

I am still having troubles with the last 1.6 version of the backup tool.

And I am realy confised by this page:

https://confluence.atlassian.com/display/STASHKB/Backup+client+-+Failed+to+drain+SCM

 

Because in this page it seems that there is a confusion between scm and git drain error, and the setting that is acting only on the database aspect of the things.

 

So, how can I handle errors like:

2015-01-21 02:02:08,891 ERROR [main] c.a.s.i.backup.client.BackupMain A backup could not be created. Reason: Operations from one or more SCMs did not finish within the allotted timeout. To prevent corruption due to inconsistent state, the backup has been aborted. Please try backup up again when the system is under less load.

 

Do I have to go in maintenance mode many minutes before starting back up script ?

 

How can I temporise?

 

Suggest an answer

Log in or Sign up to answer
How to earn badges on the Atlassian Community

How to earn badges on the Atlassian Community

Badges are a great way to show off community activity, whether you’re a newbie or a Champion.

Learn more
Community showcase
Posted Jun 06, 2018 in Bitbucket

Do you use Bitbucket Cloud and Jira Cloud? If so, let us know!

Hi Community, I'm Julia and I'm on the Jira Software Cloud marketing team!  We're looking for companies or teams using Bitbucket Cloud and Jira Software Cloud. If your team fits the t...

129 views 6 3
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you