Writing and Testing a Data Center compatible Add-on

Data Center products are all the rage. Highly available, scalable and easy to upgrade.

Make sure your add-ons don't get left behind by declaring (and testing!) their compatibility with the new architecture.

Your first stop should be https://developer.atlassian.com/market/developing-for-the-marketplace/data-center-add-on-development which calls out the basics, key services (like schedulers and listeners) to be aware of, and how to delcare compatbility in your atlassian-plugin.xml.

This article won't regurgittate that great resource, but instead jump into testing!  I assume basic knowledge of plugin development and integration testing.

 

 

Starting a Data Center cluster with atlas-run

Of course development against an actual DC cluster where you would have to manually install and test the plugin won't fly.  But by default, even with DC compatibility declared, atlas-run will only spin up a single instance.  I'll outline the minimum to have atlas-run, atlas-integration-test, etc spin up a DC cluster with your plugin.

Prequisites

Since a cluster needs multiple nodes to talk, they need a shared database. The default in-memory database won't cut it.  Fortunately you can get free MySQL and Postgres Databases from Amazon RDS under their free tier.
https://aws.amazon.com/rds/free/

You'll need the hostname and credentals once that's running.

Additionally you'll need the database drivers, which due to license limitations usually can't just be declared in maven. Download the appropriate .jar file somewhere in your project, and make sure to update the script at the bottom of this article to match.

 

Define a cluster in pom.xml

Using the AMPS plugin's 'Products' definition we define 2 products (of the type that matches your Atlassian product) that will share the Database and homepath.

There are 3 products defined:

  1. Default in memory - single instance
  2. Clustered member instance 1
  3. Clustered member instance 2

The only difference between #2 and #3 is their ports used for ssh/https so they dont clash.

	<build>
		<plugins>
			<plugin>
				<groupId>com.atlassian.maven.plugins</groupId>
				<artifactId>bitbucket-maven-plugin</artifactId>
				<version>${amps.version}</version>
				<extensions>true</extensions>
				<configuration>
                    		<allowGoogleTracking>false</allowGoogleTracking>
				<server>localhost</server>
				<!-- run DC cluster using 'atlas-run \-\-testGroup clusterTestGroup' -->
                    		<!-- you will also need to download mysql driver and place in /home/lib for *all* nodes in cluster. See bitbucket.shared.home values for each node below -->
				<!-- see https://developer.atlassian.com/stash/docs/latest/how-tos/cluster-safe-plugins.html -->
                   		<products>
                        		<!-- standalone non-clustered w/ in-memory db -->
                        		<product>
                            			<id>bitbucket</id>
                            			<instanceId>bitbucket-single</instanceId>
                           			<version>${bitbucket.version}</version>
                            			<dataVersion>${bitbucket.data.version}</dataVersion>
                        		</product>
					<!-- node 1 for clustered -->
					<product>
						<id>bitbucket</id>
						<instanceId>bitbucket-node-1</instanceId>
						<version>${bitbucket.version}</version>
						<dataVersion>${bitbucket.data.version}</dataVersion>
						<systemPropertyVariables>
							<bitbucket.shared.home>${project.basedir}/target/bitbucket-node-1/home/shared</bitbucket.shared.home>
                                			<!-- override the SSH port used for this node -->
                                			<bitbucket.plugin.ssh.port>7997</bitbucket.plugin.ssh.port>
							<!-- override database settings so both nodes use a single database -->
							<jdbc.driver>com.mysql.jdbc.Driver</jdbc.driver>
							<jdbc.url>jdbc:mysql://10.187.116.253:3306/cf_355c208f_d6d1_4f9b_b8c9_538453c89ec7?characterEncoding=utf8&amp;useUnicode=true&amp;sessionVariables=storage_engine%3DInnoDB</jdbc.url>
							<jdbc.user>DontCheckMeIntoSource</jdbc.user>
							<jdbc.password>Seriously</jdbc.password>
							<!-- allow cluster nodes to find each other over TCP/IP thus enabling clustering for this node -->
							<hazelcast.network.tcpip>true</hazelcast.network.tcpip>
							<!-- set to true if your load balancer supports stick sessions -->
							<hazelcast.http.stickysessions>false</hazelcast.http.stickysessions>
							<!-- forces Stash to fully finish starting up before yielding to the func test runner or atlas-run -->							<johnson.spring.lifecycle.synchronousStartup>true</johnson.spring.lifecycle.synchronousStartup>
						</systemPropertyVariables>
		                            	<libArtifacts>
                		                	<!-- ensure MySQL drivers are available to Stash -->
                                			<libArtifact>
                       		             			<groupId>mysql</groupId>
                                	    			<artifactId>mysql-connector-java</artifactId>
                                    				<version>5.1.32</version>
                                			</libArtifact>
                            			</libArtifacts>
						</product>
						<!-- Node 2 -->
						<product>
							<id>bitbucket</id>
							<instanceId>bitbucket-node-2</instanceId>
							<version>${bitbucket.version}</version>
							<dataVersion>${bitbucket.data.version}</dataVersion>
							<!-- override the HTTP port used for this node -->
							<httpPort>7992</httpPort>
							<systemPropertyVariables>
								<bitbucket.shared.home>${project.basedir}/target/bitbucket-node-1/home/shared</bitbucket.shared.home>
								<!-- override the SSH port used for this node -->
								<bitbucket.plugin.ssh.port>7998</bitbucket.plugin.ssh.port>
								<!-- override database settings so both nodes use a single database -->
								<jdbc.driver>com.mysql.jdbc.Driver</jdbc.driver>
								<jdbc.url>jdbc:mysql://10.187.116.253:3306/cf_355c208f_d6d1_4f9b_b8c9_538453c89ec7?characterEncoding=utf8&amp;useUnicode=true&amp;sessionVariables=storage_engine%3DInnoDB</jdbc.url>
								<jdbc.user>DonyCheckIntoSource</jdbc.user>
								<jdbc.password>Seriously</jdbc.password>
								<!-- allow cluster nodes to find each other over TCP/IP thus enabling clustering for this node -->
								<hazelcast.network.tcpip>true</hazelcast.network.tcpip>
								<!-- set to true if your load balancer supports stick sessions -->
								<hazelcast.http.stickysessions>false</hazelcast.http.stickysessions>
								<!-- forces Stash to fully finish starting up before yielding to the func test runner or atlas-run -->
								<johnson.spring.lifecycle.synchronousStartup>true</johnson.spring.lifecycle.synchronousStartup>
							</systemPropertyVariables>
							<libArtifacts>
								<!-- ensure MySQL drivers are available to Stash -->
								<libArtifact>
									<groupId>mysql</groupId>
									<artifactId>mysql-connector-java</artifactId>
									<version>5.1.32</version>
								</libArtifact>
							</libArtifacts>
						</product>
					</products>
					<!-- <log4jProperties>src/main/resources/log4j.properties</log4jProperties> -->
					<testGroups>
						<!-- tell AMPS / Maven which products ie nodes to run for the named testGroup 'clusterTestGroup' -->
						<testGroup>
							<id>clusterTestGroup</id>
							<productIds>
								<productId>bitbucket-node-1</productId>
								<productId>bitbucket-node-2</productId>
							</productIds>
						</testGroup>
						<testGroup>
							<id>default</id>
							<productIds>
								<productId>bitbucket-single</productId>
							</productIds>
						</testGroup>
					</testGroups>
				</configuration>
			</plugin>

 

The additional use of TestGroups is just an easy way for us to fire both products at once.

atlas-run --testGroup clusterTestGroup #slower start, test DC

OR

atlas-run --testGroup default  #faster start, snigle instance

 

Shared Directories and Database issues

The shared home path and limitations on RDS incomming connections added a bit of headache that the trivial script below solves by reducing the total DB connections to each node and pre-configuring their shared directory specified above.

#!/bin/bash

#bye bye whatever is there now!
echo "recreating home directories from scratch"
rm -R target/*


#creat cluster home dirs for mysql lib
mkdir -p target/stash-node-2/home/lib
mkdir -p target/stash-node-1/home/lib
mkdir -p target/stash-node-2/home/shared
mkdir -p target/stash-node-1/home/shared

# replace with path to your mysql/postgres driver jar.
cp src/test/resources/mysql-connector* target/stash-node-2/home/lib
cp src/test/resources/mysql-connector* target/stash-node-1/home/lib


echo "providing stash-config to clustered home for db settings"
# set stash-config.properties with defaults and force db pool below CF max limit of 40 connections
cat >>target/stash-node-1/home/shared/stash-config.properties <<EOF
logging.logger.com.atlassian.stash=INFO
logging.logger.com.atlassian.stash.internal.project=WARN
logging.logger.ROOT=WARN
feature.getting.started.page=false
plugin.branch-permissions.feature.splash=false
db.pool.partition.count=2
db.pool.partition.connection.maximum=9
EOF

cat target/stash-node-1/home/shared/stash-config.properties > target/stash-node-2/home/shared/stash-config.properties

echo "starting test group"
# run with cluster optipn
atlas-run --testGroup clusterTestGroup

Load Balancers

Since each cluster instance runs on a dedicated port, you can test the functionality works in each one by using the approproate port. 

To test a more real world scenario, you'll want to put a load balancer in front. Fortunately Atlassian thought about this need and released a simple plugin to drop a load balancer in front of your cluster as part of the testing lifecycle.

<plugin>
        <groupId>com.atlassian.maven.plugins</groupId>
        <artifactId>load-balancer-maven-plugin</artifactId>
        <version>1.1</version>
        <executions>
            <execution>
                <id>start-load-balancer</id>
                <phase>pre-integration-test</phase>
                <goals>
                    <goal>start</goal>
                </goals>
            </execution>
            <execution>
                <id>stop-load-balancer</id>
                <phase>post-integration-test</phase>
                <goals>
                    <goal>stop</goal>
                </goals>
            </execution>
        </executions>
        <configuration>
          <balancers>
              <balancer>
                <port>7990</port>
                <targets>
                  <target>
                    <port>7991</port>
                  </target>
                  <target>
                    <port>7992</port>
                  </target>
                </targets>
              </balancer>
              <balancer>
                <port>7999</port>
                <targets>
                  <target>
                    <port>7997</port>
                  </target>
                  <target>
                    <port>7998</port>
                  </target>
                </targets>
              </balancer>
          </balancers>
        </configuration>
      </plugin>

 

Credits

This article uses sample code snippets from an Open Source plugin I wrote for Liberty Mutual Insurance - https://github.com/Libertymutual/ssh-key-enforcer-stash/  which forces bitbucket SSH keys to be destroyed and rotated per company's predefined policy.

Much of the code is based on/copied from Atlassian tutorial https://developer.atlassian.com/bitbucket/server/docs/latest/how-tos/cluster-safe-plugins.html

 

2 comments

If I follow these steps for a Jira plugin, will I have have data center compatability for my plugin? 
Note: This plugin will not be pushed into the marketplace

Yes, so long as testing validates everything, the attribute in the xml is what tells JIRA your plugin is compatible.

Comment

Log in or Sign up to comment
Community showcase
Asked Sep 17, 2018 in Data Center

Hi! I'm Cameron Deatsch, enterprise enthusiast and Head of Server at Atlassian. AMA!

Hey team! I'm Cameron Deatsch, Head of Server at Atlassian. At our European Summit just a few weeks ago, I spoke about the latest regarding our Server and Data center products and o...

48,136 views 81 16
View question

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you