Bitbucket backup fails intermittently with error "SocketTimeoutException: Read timed out"

[Bradley Logan]

I have a backup of Bitbucket scheduled to run nightly using the latest version of the Bitbucket Backup Client, v3.6.0.  Approximately 40% of the time the backup fails with a "java.net.SocketTimeoutException: Read timed out" error.

I turned on debug level logging to see if it would give me a better idea of the problem, but so far I haven't figured it out.  Below is an excerpt of the logs mentioning the error.

Any ideas or suggestions?

2019-10-10 02:06:42,815 DEBUG (50%) Progress publishing has stopped
2019-10-10 02:06:42,842 DEBUG (50%) Unhandled client exception encountered while Bitbucket was performing a database backup
org.apache.wink.client.ClientRuntimeException: java.lang.RuntimeException: java.net.SocketTimeoutException: Read timed out
at org.apache.wink.client.internal.ResourceImpl.invoke(ResourceImpl.java:241)
at org.apache.wink.client.internal.ResourceImpl.invoke(ResourceImpl.java:189)
... 14 more frames available in the log file
2019-10-10 02:06:43,378 INFO (50%) Cancelling database backup with code: 9cb289b36c38dc1c833ff26ec52bafc95cb9c2f9
2019-10-10 02:06:43,379 DEBUG (50%) HTTP POST http://localhost:7990/stash/mvc/maintenance?token=9cb289b36c38dc1c833ff26ec52bafc95cb9c2f9, headers: [Accept:[[application/json]], Content-Length:[[0]], Content-Type:[[application/json]], User-Agent:[[Wink Client v1.1.2]], X-Atlassian-Maintenance-Token:[[2061c7c51e6062aa7c2f2cc76414ca55baf4a452]]]
2019-10-10 02:06:50,757 WARN (50%) Error reporting progress to Bitbucket
org.apache.wink.client.ClientRuntimeException: java.lang.RuntimeException: java.net.SocketTimeoutException: Read timed out
at org.apache.wink.client.internal.ResourceImpl.invoke(ResourceImpl.java:241)
at org.apache.wink.client.internal.ResourceImpl.invoke(ResourceImpl.java:189)
... 15 more frames available in the log file
2019-10-10 02:07:20,831 DEBUG (50%) -> 200 OK (@1J615UPx127x243917x3)

[Bradley Logan]

I haven't been able to figure out the cause of the error, but I modified my script to simply rerun the Bitbucket Backup Client if it fails, and so far it has always worked on the second attempt.

[Stephen Sifers]

Hello Bradley,

Thanks for reaching out to Community for your Bitbucket server backup issues and for including detailed logs into the issue you’re having. From reviewing the logs, there is an issue possibly with the database backup portion of the process. With this said, I would like to see if we’re able to back up the database without error using your database application.

Please attempt a database backup and check the consistency of the backup once completed. Any error while attempting the database backup may result in your Bitbucket server backup to fail.

We look forward to hearing how the database backup goes.

Regards,
Stephen Sifers

[Bradley Logan]

Thank you for the response, Stephen!

We are running mysql as our Bitbucket DB. I ran mysqlcheck, but all tables were reported as "OK".

Then I ran mysqldump successfully without any errors, resulting in a 2.1 MB file (our instance is not large).  I also ran mysqldump in a loop for 15 minutes, dumping every 3 seconds to see if any intermittent error occurred, but found no issues.

Any other suggestions?

[Bradley Logan]

I'm still seeing backups failing about 40% of the time. Any other ideas?