Crucible "on-line" backup?

Hello,

according to official documentation there is no way to use Fisheye/Crucicble until the backup completes.

The question is: is it possible to implement "online backup" for the application, which doesn't impact users/UI for backup time?

 

Thanks.

2 answers

Hi Sergey,

Though you could implement your own backup strategy to backup both the FISHEYE_INST directory (which is Crucible's data directory) and your database, hot backups for Crucible are not recommended, as this can lead to inconsistent backups.

The backup tool offered by Crucible stops users from doing anything to avoid new information to be written while the backup is occurring with the purpose of avoiding any inconsistency.

Regards,

Gustavo Refosco

Hi Gustavo, thank for your quick reply. What I've understood from your comment is that Atlassian can't recommend hot backups. But my question was a bit different - we're looking for "online backup" solution. Is it possible? Thanks.

Hi Sergey, I'm sorry, I got the concept of "online backup" as a hot backup. Could you please define further what you mean by "online backup"? If this is about being able to backup Crucible without stopping it, this is correspondent to the first method presented in https://confluence.atlassian.com/display/CRUCIBLE/Backing+up+and+restoring+Crucible+data through which you can perform the backup via UI - either way you already mentioned the documentation before, so I think you already checked at this. Regards, Gustavo

Hi Gustavo, thanks for link to documentation, this is what I've found on Atlassian site as recommended solution. When we talking about "online backup" solution, it means no impact for customers, no interrupt for service. Let's say we need 24x7 "Code Review" solution. Any idea? Regards, Sergey I can't send my comment yesterday because of this message: Your activity is currently limited because you've commented, answered or asked a question 3 times in the past 24 hours. These limits apply until you earn 25 points, then you can ask, answer and comment as much as you like. Any chance to add 25 points to my humble person?

Do you really need 100% uptime?  Our backup via built-in FE/Cruc tool is configured to include everything but repo/app caches (since those can be recreated) and runs at 1:00 a.m. nightly.  Records show creating that backup zip only takes 4-5 minutes to complete execution. 

What to do with the zip once you've created it is fun.  This Windows batch script has error handling, is configured for FE/Cruc and JIRA and the SQL database, and performs service stop/restarts to ensure MySQL database copy is good, and checks remaining server freespace, and then moves backup zips to a network folder on another server so it's not wasting server space or un-retrievable in the event of a server crash.  It also does a check on the size of the backup so can confirm easily that backup was good.  It auto-deletes backup files for FE/Cruc after 3 days and JIRA after 5 days, since we should have a copy on tape by then via enterprise server backup tools.  It keeps a log of the files that were deleted just in case we ever have to pull anything from tape (need filename). 

Windows Task Scheduler runs it at 2 am and then auto-emails the log to me on completion, and I've never seen it take longer than 10 minutes total.  The thing that takes the longest is probably the free space check since it isn't very efficient.  Not the cleanest script of all time, but it will output any errors to the log file, plus the output log is descriptive and formatted for readability.  It uses a free Sysinternals tool I placed in the root of C (c:\du.exe) to accomplish the file size checks.

So, all that, and time offline is ~15 minutes total per night.  Our I/O speeds on the server are poor (although we have 10gb between servers) so it may be even faster for you. 

 

echo on 
setlocal enableextensions enabledelayedexpansionset
 
destdir=\\domainname.com\project\SVN_BK\Atlassian\Backup%random% 
md %destdir% 
md %destdir%\jira_dataset 
logfile=%destdir%\log-atlassian.txt 
set mailcopy=c:\log-atlassian.txt 
set logdelete=c:\deletionlog-atlassian.txt
 
rem This gets the current date and formats it as YYYY_MM_DD to match Atlassian's format 
set atlasdate=%date:~10,4%_%date:~4,2%_%date:~7,2%
 
set jiraFolder="C:\Program Files\Atlassian\Application Data\JIRA\data" 
set jiraBackupFolder=%destdir%\jira_dataset cruc
 
Service="Atlassian Crucible" 
set jiraService=JIRA161214003955
 
set FECrucbackup=D:\FISHEYE_INST\backup\fisheyeandcrucible_backup_%atlasdate%.zip
 
rem Splitting the folder in two variables allows for handling folders with spaces cleanly with mysqldump 
set mySQLFolder="C:\Program Files\MySQL\MySQL Server 5.5" 
set mySQLBackupFolder="C:\Program Files\MySQL\Backups" 
set result=%mySQLBackupFolder%\jiradbback%atlasdate%.sql
set user=DatabaseAccountWithBackupRightsOnly
 
echo 1) START Atlassian Backup: %date% %time% >> %logfile% 
echo . >> %logfile%
echo 2a) shutdown Crucible >> %logfile% 
echo . >> %logfile%
 
rem As "sc stop" and "net stop" can both return before the service is really stopped, actually 
rem stopping JIRA can take up to 10 minutes, and we need them stopped before backing up SQL, 
rem these subroutines repeat the stop request until the services show as STOPPED. 
rem If they still aren't stopped after 400 attempts, something is wrong with the server, 
rem and the backup should exit.
 
set /a counterA=1 
:sub_StopCruc 
for /F "tokens=3 delims=: " %%H in ('sc query %crucService% ^| findstr "        STATE"') do (if /I "%%H" NEQ "STOPPED" (    
  echo Stopping Atlassian Crucible.    
  net stop %crucService%    
  echo Pass %counterA%    
  set /a counterA+=1    
  if not counterA == 400 GOTO sub_StopCruc    
  if counterA == 400 GOTO sub_Error
   ) 
) 1>>%logfile% 2>&1
 
echo . >> %logfile% 
echo 2b) shutdown JIRA >> %logfile%
set /a counterB=1 
:sub_StopJira 
for /F "tokens=3 delims=: " %%H in ('sc query %jiraService% ^| findstr "        STATE"') do (if /I "%%H" NEQ "STOPPED" (
    echo Stopping Atlassian JIRA.
    net stop %jiraService%
    echo Pass: %counterB%
    set /a counterB+=1
    if not counterB == 400 GOTO :sub_StopJira
    if counterB == 400 GOTO sub_Error
   )
 ) 1>>%logfile% 2>&1 
echo . >> %logfile%
 
echo report Atlassian Crucible service state: >> %logfile% 
sc query %crucService% 1>>%logfile% 2>&1 
echo report Atlassian JIRA service state: >> %logfile% 
sc query %jiraService% 1>>%logfile% 2>&1 
echo Crucible and JIRA services should now be stopped, continuing backup... >> %logfile% 
echo . >> %logfile%
 
echo 3) creating a backup of jiradb SQL database >> %logfile% 
"C:\Program Files\MySQL\MySQL Server 5.5\bin\mysqldump.exe" -u %user% -YourPasswordHere jiradb --result-file=%result% 1>>%logfile% 2>&1 
echo . >> %logfile% 
echo 3a) checking size of SQL backup >> %logfile% 
FOR %%I IN (%result%) DO echo %%~zI bytes 1>>%logfile% 2>&1 echo . >> %logfile% echo 4) copying jiradbback.sql to network >> %logfile% 
xcopy %result% %destdir% 1>>%logfile% 2>&1 
echo . >> %logfile% 
echo 4a) delete local copy of %result% since it has been copied to network >> %logfile% 
del %result% 1>>%logfile% 2>&1 
echo . >> %logfile%
 
echo 5) copying JIRA Data folder to network >> %logfile% 
xcopy %jiraFolder% %jiraBackupFolder% /S 1>>%logfile% 2>&1 
echo . >> %logfile%
 
echo 6) starting JIRA service >> %logfile% 
net start %jiraService% 1>>%logfile% 2>&1 
 
echo 7) starting Crucible service >> %logfile% 
net start %crucService% 1>>%logfile% 2>&1
 
rem FE/Cruc backup move 
echo 8) copying FECruc backup file to network >> %logfile% 
copy %FECrucbackup% %destdir% 1>>%logfile% 2>&1 
echo . >> %logfile% 
 
echo 9) delete local copy of %FECrucbackup% since it has been copied to network >> %logfile% 
del %FECrucbackup% 1>>%logfile% 2>&1 
echo . >> %logfile%
 
echo 10) END Atlassian Backup Complete: %date% %time% >> %logfile% 
echo . >> %logfile%
 
echo 11) Check remaining FreeSpace on Server D Drive: >> %logfile%
rem Checking deviceID in case in future this script ever needs to check space for other drives 
for /f "skip=1 tokens=1-3" %%a in ('wmic logicaldisk get deviceid^,FreeSpace^,Size') do (
    set disk=%%a
    if "!disk!"=="D:" (
       set "FreeBytesDrive=%%b"
       set "TotBytesDrive=%%c"
       echo(
       if defined TotBytesDrive (
          set /a Fmb=!FreeBytesDrive:~0,-6!, Tmb=!TotBytesDrive:~0,-6!, Umb=Tmb-Fmb
          call :ConvertGb Tgb Tmb
          call :ConvertGb Fgb Fmb
          call :ConvertGb Ugb Umb
 
         echo %%a Used:!UGb! - Free:!FGb! / Tot:!TGb! Gb >> %logfile%
       )
    )
 ) echo . >> %logfile%
 
echo 12) Total final size of backup to %destdir% : >> %logfile% 
c:\du.exe -q %destdir% /accepteula 1>>%logfile% 2>&1
 
echo 13) Deleting files and folders more than 3 days old >> %logfile% 
PushD \\YourDomain.com\project\SVN_BK\Atlassian\ && (forfiles /D -3 /C "CMD /C if @ISDIR==TRUE echo Backup Directory @relpath created on @fdate deleted on %date% >> %logfile%") & PopD 1>>%logfile% 2>&1 
PushD \\YourDomain.com\project\SVN_BK\Atlassian\ && (forfiles /D -3 /C "CMD /C if @ISDIR==TRUE echo Backup Directory @relpath created on @fdate deleted on %date% >> %logdelete%") & PopD 1>>%logfile% 2>&1 
PushD \\YourDomain.com\project\SVN_BK\Atlassian\ && (forfiles /D -3 /C "CMD /C if @ISDIR==TRUE echo RD /Q @FILE &RD /Q /S @FILE") & PopD 1>>%logfile% 2>&1 
echo . >> %logfile%
 
rem not worried about keeping a log of these automatic JIRA backups 
echo 14) Deleting daily automatic backups more than 5 days old >> %logfile% 
PushD C:\Program Files\Atlassian\Application Data\JIRA\export && (forfiles /D -5 /C "CMD /C if @ISDIR==TRUE echo RD /Q @FILE &RD /Q /S @FILE") & PopD 1>>%logfile% 2>&1 
echo . >> %logfile%
 
rem Copying log to local server so that it is accessible to server's Scheduled Task for email at completion 
echo 15) Copy logfile to local server for confirmation mail >> %logfile% 
echo F | xcopy %logfile% %mailcopy% /y /c 1>>%logfile% 2>&1
 
GOTO :EOF
 
rem GB with 3 decimal, use only 2 for now, high precision, add padding 
:ConvertGB
   set /a tmp=%2-%2/10+%2*1000/31926
   set "tmp=      !tmp:~0,-3!.!tmp:~-3,2!"
   set tmp=!tmp:~-6!
   set %1=!tmp: .=0.!
   exit /b
 
:sub_Error
   if %counterA% == 400 echo "Crucible would not stop, exiting backup script."
   if %counterB% == 400 echo "JIRA would not stop, exiting backup script."
   exit /b
 
exit

 

 

 

Oh, and it grabs all of your files JIRA attachments and avatars too.

Hi Tini, thanks for sharing your code. Yes, we really need 100% uptime for FeCru. Could you provide additional information about instance, where backup basically runs just 5 minutes: 1) how many projects/repos you have? 2) how long it takes to restore instance from backup? 3) same as 2) but with indexes rebuild? Thanks in advance.

My way definitely takes things offline for a short period. Only reason I can think of to have a 100% uptime requirement is distributed dev team across internal timezones? Unsure if FE/Cruc has support for server replication. 1) 20 repos, 18 projects, 50 users 2) I do this too infrequently to give you an accurate answer. Primary method of migrating data to a test server before a major version update is via the easier FISHEYE_INST copy (with MySQL dump/restore), rather than actual restore from FE/Cruc zip backup file. 3) ~Once a year I probably force an optimized index rebuild after restart due to performance issues, and that can take 20-40 minutes. But, activity is unrelated to any backup and, our bad virtual server I/O throughput probably doesn't help. Virtual Server host, dual Xeons @ 2.2Ghz (8 cores), 8GB RAM, 64-bit OS. FYI our Fisheye indexes a 122GB Subversion instance, which is also co-hosted.

Suggest an answer

Log in or Sign up to answer
How to earn badges on the Atlassian Community

How to earn badges on the Atlassian Community

Badges are a great way to show off community activity, whether you’re a newbie or a Champion.

Learn more
Community showcase
Published Thursday in Confluence

Color tables for a shiny Confluence page

...; ## Developed by: Alana Fernando ## Shared with love ## @param style:title=style type|type=enum|required=true|desc=Choose a style.|enumValues=Style1,Style2,Style3,Style4,Style5 ## @param alignment:title...

156 views 8 11
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you