Fisheye - Repository cannot be restarted during/after backup

[Michael Bartmon]

This happens every few days. Several repositories are not restarted after the backup task. The log does not seem to contain anything useful other than a message about not being able to restart.

I am using version 3.2.2.

One of the repositories is called MBCS_Git and I am providing snippets of the log that pertain to that repository:

2014-02-18 01:00:00,916 INFO [quartzScheduler_Worker-8 backup-1392706800017] fisheye MessageLoggingProgressMonitor-update - Shutting down repository MBCS_Git...

2014-02-18 01:00:00,916 INFO [quartzScheduler_Worker-8 backup-1392706800017] fisheye RepositoryHandle-stop - STOP requested on MBCS_Git

2014-02-18 01:00:10,226 DEBUG [IncrementalPinger5 MBCS_Git] fisheye LoggingProcessMonitor-onAfterFinished - Finished process: git ls-remote --heads d:\git\mbcscentral_c took 13394ms

2014-02-18 01:00:10,226 DEBUG [IncrementalPinger5 MBCS_Git] fisheye LoggingProcessMonitor-onBeforeStart - Starting process: git fetch d:\git\mbcscentral_c +refs/heads/*:refs/heads/*

2014-02-18 01:00:12,521 DEBUG [IncrementalPinger1 WinGL] fisheye LoggingProcessMonitor-onAfterFinished - Finished process: git fetch d:\git\WinGL +refs/*:refs/* took 13450ms

2014-02-18 01:00:12,521 DEBUG [IncrementalPinger1 WinGL] fisheye LoggingProcessMonitor-onBeforeStart - Starting process: git branch --verbose --no-abbrev

2014-02-18 01:00:12,549 DEBUG [IncrementalPinger1 WinGL] fisheye LoggingProcessMonitor-onAfterFinished - Finished process: git branch --verbose --no-abbrev took 28ms

2014-02-18 01:00:12,549 DEBUG [IncrementalPinger1 WinGL] fisheye LoggingProcessMonitor-onBeforeStart - Starting process: git ls-remote --heads d:\git\WinGL

2014-02-18 01:00:19,227 DEBUG [IncrementalPinger3 MbcsCentralWeb] fisheye LoggingProcessMonitor-onAfterFinished - Finished process: git ls-remote --heads d:\git\MbcsCentralWeb took 11190ms

2014-02-18 01:00:19,227 DEBUG [IncrementalPinger3 MbcsCentralWeb] fisheye LoggingProcessMonitor-onBeforeStart - Starting process: git fetch d:\git\MbcsCentralWeb +refs/heads/*:refs/heads/*

2014-02-18 01:00:20,614 DEBUG [notificationManager1 ] fisheye DefaultNotificationManager-processNotifications - Processing notifications

2014-02-18 01:00:21,414 WARN [quartzScheduler_Worker-8 backup-1392706800017] fisheye MessageLoggingProgressMonitor-update - Unable to stop FishEye repository MBCS_Git. Caches included in this backup may not be consistent. If the problem persists, try stopping the repositories manually, before backup.

2014-02-18 01:00:23,806 DEBUG [IncrementalPinger5 MBCS_Git] fisheye LoggingProcessMonitor-onAfterFinished - Finished process: git fetch d:\git\mbcscentral_c +refs/heads/*:refs/heads/* took 13580ms

2014-02-18 01:00:23,807 DEBUG [IncrementalPinger5 MBCS_Git] fisheye LoggingProcessMonitor-onBeforeStart - Starting process: git ls-remote --tags d:\git\mbcscentral_c

2014-02-18 01:00:32,187 INFO [quartzScheduler_Worker-8 backup-1392706800017] fisheye MessageLoggingProgressMonitor-update - Restarted repository MbcsMain

2014-02-18 01:00:32,187 DEBUG [FE-CacheWarmer ] fisheye RepositoryHandle-acquire - acquire engine on MbcsMain, count=1, waited 0 ms to acquire.

2014-02-18 01:00:32,187 WARN [quartzScheduler_Worker-8 backup-1392706800017] fisheye MessageLoggingProgressMonitor-update - Unable to restart repository MBCS_Git

2014-02-18 01:00:32,187 WARN [quartzScheduler_Worker-8 backup-1392706800017] fisheye FishEyeCacheBackup$SaverImpl-backup - Unable to restart repository MBCS_Git

2014-02-18 01:00:32,187 WARN [quartzScheduler_Worker-8 backup-1392706800017] fisheye MessageLoggingProgressMonitor-update - The process cannot access the file because another process has locked a portion of the file

2014-02-18 01:00:32,189 INFO [quartzScheduler_Worker-8 backup-1392706800017] fisheye MessageLoggingProgressMonitor-update - Backup failed

2014-02-18 01:00:32,189 ERROR [quartzScheduler_Worker-8 backup-1392706800017] fisheye BackupJob-execute - Error performing backup: The process cannot access the file because another process has locked a portion of the file

2014-02-18 01:00:37,395 DEBUG [IncrementalPinger5 MBCS_Git] fisheye LoggingProcessMonitor-onAfterFinished - Finished process: git ls-remote --tags d:\git\mbcscentral_c took 13588ms

2014-02-18 01:00:37,396 DEBUG [IncrementalPinger5 MBCS_Git] fisheye LoggingProcessMonitor-onBeforeStart - Starting process: git show-ref -d --tags

2014-02-18 01:00:37,458 DEBUG [IncrementalPinger5 MBCS_Git] fisheye LoggingProcessMonitor-onAfterFinished - Finished process: git show-ref -d --tags took 62ms

2014-02-18 01:00:37,458 DEBUG [IncrementalPinger5 MBCS_Git] fisheye RepositoryStatus-setMessage - Status change [MBCS_Git]: Updated from d:\git\mbcscentral_c

2014-02-18 01:00:37,458 INFO [IncrementalPinger5 MBCS_Git] fisheye BaseRepositoryScanner-ping - processing MBCS_Git

2014-02-18 01:00:37,458 DEBUG [IncrementalPinger5 MBCS_Git] fisheye LifecycleRepositoryScanner-doSlurpTransaction - Rolling back cache changes

2014-02-18 01:00:37,458 INFO [IncrementalPinger5 MBCS_Git] fisheye BaseRepositoryScanner-ping - done slurp for /MBCS_Git time = 0ms

2014-02-18 01:00:37,458 DEBUG [IncrementalPinger5 ] fisheye RepositoryHandle-release - release engine on MBCS_Git, count=0 waited 0 ms to release

2014-02-18 01:00:37,458 DEBUG [IncrementalPinger5 ] fisheye RepositoryHandle-tryToBeginStopProcedure - STOPPING MBCS_Git

2014-02-18 01:00:37,458 DEBUG [IncrementalPinger5 ] fisheye InfinityDbHandle-close - closing c:\fisheye_data\var\cache\MBCS_Git\revcache\data.bin

2014-02-18 01:00:37,995 DEBUG [IncrementalPinger5 ] fisheye RepositoryEngine-forceClose - Closed engine MBCS_Git

2014-02-18 01:03:33,754 DEBUG [qtp8303200-15359 ] fisheye ProfilingServletFilter-logRequest - start request GET /login?origUrl=%2Fbrowse%2FMBCS_Git%2FMbcsCentral.iw9%3F%2540annm%3Dblame%26r%3D9d8009b%26r%3De084411ac745017e4b60aebb3b500f8ce03d818d nosession

2014-02-18 01:03:33,769 DEBUG [qtp8303200-15359 ] fisheye ProfilingServletFilter-logRequest - end request GET /login?origUrl=%2Fbrowse%2FMBCS_Git%2FMbcsCentral.iw9%3F%2540annm%3Dblame%26r%3D9d8009b%26r%3De084411ac745017e4b60aebb3b500f8ce03d818d nosession 200

2014-02-18 01:07:21,142 DEBUG [qtp8303200-15363 ] fisheye ProfilingServletFilter-logRequest - start request GET /login?origUrl=%2Fbrowse%2FMBCS_Git%2FEomReorg_2.dtp%3Fr%3D651f64c30c23c9cc58ace0ce22284035cbab08ce%26r%3D9d8009bae0f4a1670c5c6f4b1c18ac21099894d5%26r%3D44e326114e577cb87d0c1791d8af36eee8841773 nosession

2014-02-18 01:07:21,157 DEBUG [qtp8303200-15363 ] fisheye ProfilingServletFilter-logRequest - end request GET /login?origUrl=%2Fbrowse%2FMBCS_Git%2FEomReorg_2.dtp%3Fr%3D651f64c30c23c9cc58ace0ce22284035cbab08ce%26r%3D9d8009bae0f4a1670c5c6f4b1c18ac21099894d5%26r%3D44e326114e577cb87d0c1791d8af36eee8841773 nosession 200

[Piotr Swiecicki · Accepted solution]

Hi Micheal,

Apologies, I didn't notice this was scheduled backup happening overnight. I have raised https://jira.atlassian.com/browse/FE-5089issue, feel free to update it if I have missed anything, and vote on this so it would get quicker into top of our backlog. In the meantime, if you want us to provide you a custom build with extended timeout raise a support request on support.atlassian.complease.

Kind regards,

Piotr

[Piotr Swiecicki]

Hi Michael,

I'm guessing the problem is related to the fact that your repository wasn't stopped within 20 seconds (time backup procedure waits for each repository to stop), so backup issued this warning:

2014-02-18 01:00:21,414 WARN [quartzScheduler_Worker-8 backup-1392706800017] fisheye MessageLoggingProgressMonitor-update - Unable to stop FishEye repository MBCS_Git. Caches included in this backup may not be consistent. If the problem persists, try stopping the repositories manually, before backup.

Request to stop MBCS_Git repository completed after backup finished it's job, so it wasn't restarted in result.

Can you try stopping this repository manually before running backup next time and see if that helps?

I'm guessing FishEye should handle this situation in a better way - trying to cancel running SCM commands more agressively and/or extending timeout allowed. And perhaps ensuring all repositories requested to be stopped and restarted again.

Please confirm you can workaround the problem by stopping repositories manually, and file a bug report so we can take care of this.

Thanks,

Piotr

[Michael Bartmon]

Thanks for the reply.

Thing is that this is an automatic backup at 1:00AM (note that the messages are generated by quartzSecheduler). Therefore there is no possibility for me to stop the repository prior to the backup. I would have to manually stop every repository every evenening and then restart every repository the following morning, every day. That is not a satisfactory solution.