How to add servers into cluster

[Rohit Shukla]

Currently I have 2 servers on my local, I want to add into a clusters.

[Danyal Iqbal · Accepted solution]

Are these dedicated servers or VM's? You could clone the primary node and mount the shared home directory as ${BITBUCKET_HOME}/shared. Synchronize clocks and configure the load balancer.

[Rohit Shukla]

Hello Danyal,

We have 2 VM's in which we have setup of bit bucket data center, and we have done with mounting of shared folder. 

I am not getting how to do configure load balancer, is it necessary? if Yes, can you please help me to understand that how to do setup of this, stuck since long on this.

Do you have chat support as well, as every time discussing on email is time consuming. 

Thanks for your help.

[Danyal Iqbal]

which load balancer do you have? Which load balancing scheme is already implemented. see https://confluence.atlassian.com/enterprise/load-balancer-configuration-options-935383760.html

and the load balancer documentation for final configuration.

[Rohit Shukla]

Thank you Danyal,

Can you please explain on this as well.

When I tried to specify the same Postgres database to the second node, it complained that the database is not empty (of course, it's not empty, it was initialized by the first node).

So I'm absolutely confused.
My bitbucket.properties files is:#>*******************************************************
#> Migrated to database at jdbc:postgresql://X.X.X.X:5432/bit-bucket-1
#> Updated by Rohit on 2018-02-14 T15:55:44.867+05:30
#>*******************************************************
jdbc.driver=org.postgresql.Driver
jdbc.url=jdbc:postgresql://DB_server:5432/bit-bucket-1
jdbc.user=psDba
jdbc.password=dba

# Use multicast to discover cluster nodes (recommended).
hazelcast.network.multicast=false

# If your network does not support multicast, you may uncomment the following lines and substitute
# the IP addresses of some or all of your cluster nodes. (Not all of the cluster nodes have to be
# listed here but at least one of them has to be active when a new node joins.)
hazelcast.network.tcpip=true
hazelcast.network.tcpip.members=X.X.172.183.X:5701, X.X.172.184:5701

# The following should uniquely identify your cluster on the LAN.
hazelcast.group.name=your-bitbucket-cluster
hazelcast.group.password=your-bitbucket-cluster

[Rohit Shukla]

Hello Danyal,

One question is still unanswered for me that,

Is it load balancer really needed, as per document LB is Optional.

Suppose we have 2 node, node 1 and node 2 , and we have setup on both, we are considering node 1 as master in our case, could we treat node1 as a load balancer for us?

[Danyal Iqbal]

LB is not optional for a data centre setup. You need a load balancer to make sure that requests are sent to both nodes either in round robin fashion (simplest scheme ) or according to the load/resources of the node.

of coourse you can configure node 1 as the load balancer. LB is just a piece of software. You need to configure  apache/nginx as the load balancer.

Relax, take a break, have a coffee, read the setup tutorial, assimilate for a few hours and try again. Data centres can not be configured in a hurry :)

[Rohit Shukla]

Okay, thank you. Surely let me work on that.

Just give me answer of this as well..

The current node is unable to safely connect to the cluster and will not service requests
Nodes are connected to the same shared home but different databases

When I tried to specify the same Postgres database to the second node, it complained that the database is not empty (of course, it's not empty, it was initialized by the first node). Rest I have already added above. Please check and let me know.

How we can share one database for 2 node?

Thank you once again.

[Rohit Shukla]

Hello Danyal,

I am waiting on this, please provide me your input on this?

[Danyal Iqbal]

The current node is unable to safely connect to the cluster and will not service requests
Nodes are connected to the same shared home but different databases.

How did you manage to connect them to different databases? 2nd node should be a clone of the 1st node which means that the database connection should be the same as well.

What does the logs on both nodes say?${BITBUCKET_HOME}/log/atlassian-bitbucket.log

Did you install datacentre from scratch on the new node?

Are you sure you have an existing data centre license?check license at http://<bitbucket>:7990/admin/license new nodes should be visible automatically at http://<load-balancer>/admin/clustering, subject to correct configuration of course.


 How we can share one database for 2 node?

This is irrelevant for a datacentre installation. This question is relevant only standalone servers. If it complains that the database is locked, than your existing license/install is not a data centre release.

TIP:

You must ensure your database is configured to allow enough concurrent connections. Bitbucket Server by default uses up to 80 connections per cluster node , which can exceed the default connection limit of some databases.

For example, in PostgreSQL the default limit is usually 100 connections. If you use PostgreSQL, you may need to edit your postgresql.conf file, to increase the value of max_connection s, and restart Postgres.

Only the ${BITBUCKET_HOME}/shared directory should be shared between cluster nodes. All other directories, including ${BITBUCKET_HOME}, should be node-local (that is, private to each node).

Bitbucket Data Center checks during startup that ${BITBUCKET_HOME} is node local and ${BITBUCKET_HOME}/shared is shared, and will fail to form a cluster if this is not true.

Assuming, you get this far, you should be able to start the first node. Than configure your 2nd node in the load balancer.

See: https://confluence.atlassian.com/bitbucketserver/connecting-bitbucket-server-to-postgresql-776640389.html for connecting to a postgresdb.

[Rohit Shukla]

Hello Danyal,

I have installed bit bucket server setup on 2 different nodes, and later I have done mounting for shared folder. 

I am not getting your point that "2nd node should be a clone of the 1st node which means that the database connection should be the same as well." What does it mean and how to do that?

[Danyal Iqbal]

I have installed bit bucket server setup on 2 different nodes, and later I have done mounting for shared folder. 

Bitbucket server is different than data centre (licenses and everything else). You need to install bitbucket data centre from scratch and than migrate your server installation to data centre.

2nd node should be a clone of the 1st node which means that the database connection should be the same as well.

Right click the 1st node and clone the VM (in vsphere/esxi/other virtualisation env etc.)

[Rohit Shukla]

Okay, thanks.

One question:

So cloning of server(node2) should be after installation of bit bucket data center on node1?

[Rohit Shukla]

Hello Danyal,

I have done with the setup on first node, migrate database as well.

Now as you suggested to do cloning of first node, I am facing one problem.

On local VM(esxi) we dont have option of cloning. Do we have any other way to clone or create other node2, so I can add into cluster?

[Danyal Iqbal]

On local VM(esxi) we dont have option of cloning. Do we have any other way to clone or create other node2, so I can add into cluster?

Your ESXI administrator has the rights to clone the machine.  As a workaround, you could install the other node from scratch as well.

[Rohit Shukla]

As a workaround, you could install the other node from scratch as well.

-- For this, should I install bit bucket server on node2, or bit bucket data center.

My first node SSN number is: SEN-L11220666

[Danyal Iqbal]

You need Data centre on all nodes.

[Rohit Shukla]

First node details..

[Rohit Shukla]

Okay, let me do that.

Thanks.

[Rohit Shukla]

Hello Danyal,

Can you please tell me what could be the reason of below errors?

The following problems occurred, which prevents Atlassian Bitbucket from starting:
The current node is unable to safely connect to the cluster and will not service requests
Required property 'buildHash' should be 'aef68c73b38447e13e3a4aa7b4889f565088b473' but is 'e111a8b7b8044728fcd3f5f442428a9614dd41f5'
The current node has been passivated

[Rohit Shukla]

Hello Danyal,

Created clone of first node.

In license I am able to see same server id for both node or servers, is it correct?

Regards,

[Danyal Iqbal]

Yes, you should see the same id on both nodes.

2nd node failed to start up due to different build hash.use the clone for further configuration. See my initial comments.

[Rohit Shukla]

Hello Danyal,

I am getting this error now, after mounting. We already checked permission. It is there. What could be the reason of this?

The following problem occurred, which prevents Atlassian Bitbucket from starting:

• Unable to create and acquire shared lock file '/var/atlassian/application-data/bitbucket/shared/.lock' for Bitbucket shared home directory '/var/atlassian/application-data/bitbucket/shared'.
  
  Please ensure that the user running Bitbucket has permission to write to this directory and that file locking is enabled for your network file system.
  
  If this is already the case, please check the logs for more information.

[Rohit Shukla]

Hello,

Can you please help me on this, I am stuck on this. I checked all possible scenarios.

[Danyal Iqbal]

Make sure the lockd, portmap, and dbus services are all running on your NFS server.

See:https://confluence.atlassian.com/bitbucketserverkb/bitbucket-data-center-fails-to-start-with-a-lock-file-shared-lock-cannot-be-created-in-home-directory-error-779171833.html.

[Rohit Shukla]

Hello,

I have cross checked lockd, portmap and dbus services, looks these are running. I have installed NFS on node 1 itself.

root@in-bucket1:~ # service portmap status
● rpcbind.service - RPC bind portmap service
Loaded: loaded (/lib/systemd/system/rpcbind.service; indirect; vendor preset: enabled)
Drop-In: /run/systemd/generator/rpcbind.service.d
└─50-rpcbind-$portmap.conf
Active: active (running) since Wed 2018-02-21 17:26:19 IST; 16h ago
Main PID: 2929 (rpcbind)
Tasks: 1
Memory: 720.0K
CPU: 60ms
CGroup: /system.slice/rpcbind.service
└─2929 /sbin/rpcbind -f -w

Feb 21 17:26:19 in-bucket1 systemd[1]: Starting RPC bind portmap service...
Feb 21 17:26:19 in-bucket1 systemd[1]: Started RPC bind portmap service.

root@in-bucket1:~ # service dbus status
● dbus.service - D-Bus System Message Bus
Loaded: loaded (/lib/systemd/system/dbus.service; static; vendor preset: enabled)
Active: active (running) since Wed 2018-02-21 15:29:09 IST; 18h ago
Docs: man:dbus-daemon(1)
Main PID: 811 (dbus-daemon)
Tasks: 1
Memory: 1.9M
CPU: 173ms
CGroup: /system.slice/dbus.service
└─811 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation

Feb 21 15:29:09 in-bucket1 systemd[1]: Started D-Bus System Message Bus.
Feb 21 15:29:09 in-bucket1 dbus[811]: [system] AppArmor D-Bus mediation is enabled
Feb 21 15:29:09 in-bucket1 dbus[811]: [system] Activating via systemd: service name='org.freedesktop.PolicyKit1' unit='polkitd.service'
Feb 21 15:29:09 in-bucket1 dbus[811]: [system] Successfully activated service 'org.freedesktop.PolicyKit1'

root@in-bucket1:~ # service lockd status
● lockd.service
Loaded: not-found (Reason: No such file or directory)
Active: inactive (dead)

Is it the reason? If Yes, How I can make sure that lockd is running? Please suggest.

################################################################################################################################

Also can you please cross check entries of /etc/exports & /etc/fstab as well. Am I missing anything?

Node1:(192.168.*.185)

root@in-bucket1:~ # cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
# to NFS clients. See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check)
#
/var/atlassian/application-data/bitbucket/shared 192.168.*.187(rw,nohide,insecure,no_subtree_check,sync,no_root_squash)

################

Node2:(192.168.*.187)

root@in-bucket2:~ # cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/sda1 during installation
UUID=7ae8e345-*-*-9ab8-a5c61d27c6e8 / ext4 errors=remount-ro 0 1
# swap was on /dev/sda5 during installation
UUID=d737e5a7-*3-4de3-9814-61214d1b2ac0 none swap sw 0 0
192.168.*.185:/var/atlassian/application-data/bitbucket/shared /var/atlassian/application-data/bitbucket/shared nfs nfsvers=3,lookupcache=pos,noatime,intr,rsize=32768,wsize=32768 0 0

[Shankar Asam {Appfire}]

clustering can only be possible with bitbucket data center versions/licenses. Pls refer to this guide on data center setup.

[Rohit Shukla]

Thank you Shankar,

Yes, have bitbucket data center version on both servers.