Clustering Bitbucket Install as DR

[Ken Ng]

I currently have 1 environment clustered with 2 servers setup with shared filer etc...  My question is if I want to do the install in my 2nd clustered location which also uses that shared filer but it's set as read only along with the db for that 2nd location, can I install BB binaries without having to write into the shared filer and db ?  The shared filer will get replicated to the 2nd site along with the db stuff so we want just a basic install of a clustered BB environment in LAB without having the install write to the shared filer and DB.

Lab1

ServerA

ServerB

sharedfiler mounted as read/writable on Lab1  (replication of the data here is replicated to Lab2's sharedfiler (readonly))

db read/writable on Lab1  (replication happens here as well to Lab2)

###############################################################

Lab2 

ServerA 

ServerB

sharefiler mounted as read ONLY on Lab2

db read ONLY on Lab2

I want to know if it's possible to install BB without having to write to a sharedfiler which is Read Only and DB which is read only as well.

So what we ultimately want to do if Lab1 goes down, we can swing the sharedfiler over to Lab2 where it becomes writable since replication has been happening along with the DB being read/write state.

[Evan Slaughter · Accepted solution]

Howdy Ken,

What you'll likely want to do is instead perform a clean install of Bitbucket on the nodes of the DR environment, which will put all of the binaries in the specified $BITBUCKET_INSTALL location, and also create/validate initial $BITBUCKET_HOME and $BITBUCKET_HOME/shared directories on the local filesystem.

After running the installer, though, you can then mount your NFS mount at the "$BITBUCKET_HOME/shared" directory - which should then contain a bitbucket.properties file that points to this read only database replica. It's important to note, though, that Bitbucket won't successfully start while the database or shared file-system are set to read-only (as it creates lock files/records as part of the start-up process), so that read-only block will need to be removed prior to actually attempting to start Bitbucket as part of your DR test/plan.

For more on this, I recommend checking out the following article:

• Disaster Recovery guide for Bitbucket Data Center 

Hope this helps!

Thanks,

Evan Slaughter

[Ken Ng]

Sorry see replay in the same thread, forgot to reply here.

Here's the situation, currently the DR environment already has the NFS mounted already.  So I was thinking maybe I just let the install run its course and it will eventually fail because it's not able to write to the DB and NFS mount but it will still lay out the binaries and configurations locally to that node on the DR side.  When it's time to move to the DR side, we just make the NFS mount and DB writable and start BB.  How does that sound ?  Would that work ?

[Evan Slaughter]

Assuming you're going to be performing the Bitbucket install well before your team attempts testing/moving to the DR environment, is there a reason why you would want to have the NFS mount in place prior to running the installer?

Although it may be possible to run the installer while having the shared home already mounted, doing this is not recommended - as it would be a much cleaner plan to simply unmount the NFS volume, allow the installer to complete as expected (and without any added warnings/errors), and then re-mount the NFS volume on the empty shared home directory after the installer has finished running.

From that point on - it shouldn't matter whether or not the shared NFS mount or the database are writable until the Bitbucket process is started. Then for starting the DR environment, you would stop replication to the database/NFS volume and then remove the read-only locks on both of these assets. Then, after performing the needed adjustments to the bitbucket.properties file along with the other precautions outlined in the provided knowledge article - you would be in the clear to start the Bitbucket services on the nodes of the DR environment.

Thanks,

Evan Slaughter

[Ken Ng]

Thanks for your  assistance with this!

[Ken Ng]

Here's the situation, currently the DR environment already has the NFS mounted already.  So I was thinking maybe I just let the install run its course and it will eventually fail because it's not able to write to the DB and NFS mount but it will still lay out the binaries and configurations locally to that node on the DR side.  When it's time to move to the DR side, we just make the NFS mount and DB writable and start BB.  How does that sound ?