Create
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Products
See all
Community resources
Top groups
Explore all groups
Community resources
Learn
Community resources
Events
Community resources

Writing and Testing a Data Center compatible Add-on

July 9, 2017

Data Center products are all the rage. Highly available, scalable and easy to upgrade.

Make sure your add-ons don't get left behind by declaring (and testing!) their compatibility with the new architecture.

Your first stop should be https://developer.atlassian.com/market/developing-for-the-marketplace/data-center-add-on-development which calls out the basics, key services (like schedulers and listeners) to be aware of, and how to delcare compatbility in your atlassian-plugin.xml.

This article won't regurgittate that great resource, but instead jump into testing!  I assume basic knowledge of plugin development and integration testing.

 

 

Starting a Data Center cluster with atlas-run

Of course development against an actual DC cluster where you would have to manually install and test the plugin won't fly.  But by default, even with DC compatibility declared, atlas-run will only spin up a single instance.  I'll outline the minimum to have atlas-run, atlas-integration-test, etc spin up a DC cluster with your plugin.

Prequisites

Since a cluster needs multiple nodes to talk, they need a shared database. The default in-memory database won't cut it.  Fortunately you can get free MySQL and Postgres Databases from Amazon RDS under their free tier.
https://aws.amazon.com/rds/free/

You'll need the hostname and credentals once that's running.

Additionally you'll need the database drivers, which due to license limitations usually can't just be declared in maven. Download the appropriate .jar file somewhere in your project, and make sure to update the script at the bottom of this article to match.

 

Define a cluster in pom.xml

Using the AMPS plugin's 'Products' definition we define 2 products (of the type that matches your Atlassian product) that will share the Database and homepath.

There are 3 products defined:

  1. Default in memory - single instance
  2. Clustered member instance 1
  3. Clustered member instance 2

The only difference between #2 and #3 is their ports used for ssh/https so they dont clash.

	<build>
		<plugins>
			<plugin>
				<groupId>com.atlassian.maven.plugins</groupId>
				<artifactId>bitbucket-maven-plugin</artifactId>
				<version>${amps.version}</version>
				<extensions>true</extensions>
				<configuration>
                    		<allowGoogleTracking>false</allowGoogleTracking>
				<server>localhost</server>
				<!-- run DC cluster using 'atlas-run \-\-testGroup clusterTestGroup' -->
                    		<!-- you will also need to download mysql driver and place in /home/lib for *all* nodes in cluster. See bitbucket.shared.home values for each node below -->
				<!-- see https://developer.atlassian.com/stash/docs/latest/how-tos/cluster-safe-plugins.html -->
                   		<products>
                        		<!-- standalone non-clustered w/ in-memory db -->
                        		<product>
                            			<id>bitbucket</id>
                            			<instanceId>bitbucket-single</instanceId>
                           			<version>${bitbucket.version}</version>
                            			<dataVersion>${bitbucket.data.version}</dataVersion>
                        		</product>
					<!-- node 1 for clustered -->
					<product>
						<id>bitbucket</id>
						<instanceId>bitbucket-node-1</instanceId>
						<version>${bitbucket.version}</version>
						<dataVersion>${bitbucket.data.version}</dataVersion>
						<systemPropertyVariables>
							<bitbucket.shared.home>${project.basedir}/target/bitbucket-node-1/home/shared</bitbucket.shared.home>
                                			<!-- override the SSH port used for this node -->
                                			<bitbucket.plugin.ssh.port>7997</bitbucket.plugin.ssh.port>
							<!-- override database settings so both nodes use a single database -->
							<jdbc.driver>com.mysql.jdbc.Driver</jdbc.driver>
							<jdbc.url>jdbc:mysql://10.187.116.253:3306/cf_355c208f_d6d1_4f9b_b8c9_538453c89ec7?characterEncoding=utf8&amp;useUnicode=true&amp;sessionVariables=storage_engine%3DInnoDB</jdbc.url>
							<jdbc.user>DontCheckMeIntoSource</jdbc.user>
							<jdbc.password>Seriously</jdbc.password>
							<!-- allow cluster nodes to find each other over TCP/IP thus enabling clustering for this node -->
							<hazelcast.network.tcpip>true</hazelcast.network.tcpip>
							<!-- set to true if your load balancer supports stick sessions -->
							<hazelcast.http.stickysessions>false</hazelcast.http.stickysessions>
							<!-- forces Stash to fully finish starting up before yielding to the func test runner or atlas-run -->							<johnson.spring.lifecycle.synchronousStartup>true</johnson.spring.lifecycle.synchronousStartup>
						</systemPropertyVariables>
		                            	<libArtifacts>
                		                	<!-- ensure MySQL drivers are available to Stash -->
                                			<libArtifact>
                       		             			<groupId>mysql</groupId>
                                	    			<artifactId>mysql-connector-java</artifactId>
                                    				<version>5.1.32</version>
                                			</libArtifact>
                            			</libArtifacts>
						</product>
						<!-- Node 2 -->
						<product>
							<id>bitbucket</id>
							<instanceId>bitbucket-node-2</instanceId>
							<version>${bitbucket.version}</version>
							<dataVersion>${bitbucket.data.version}</dataVersion>
							<!-- override the HTTP port used for this node -->
							<httpPort>7992</httpPort>
							<systemPropertyVariables>
								<bitbucket.shared.home>${project.basedir}/target/bitbucket-node-1/home/shared</bitbucket.shared.home>
								<!-- override the SSH port used for this node -->
								<bitbucket.plugin.ssh.port>7998</bitbucket.plugin.ssh.port>
								<!-- override database settings so both nodes use a single database -->
								<jdbc.driver>com.mysql.jdbc.Driver</jdbc.driver>
								<jdbc.url>jdbc:mysql://10.187.116.253:3306/cf_355c208f_d6d1_4f9b_b8c9_538453c89ec7?characterEncoding=utf8&amp;useUnicode=true&amp;sessionVariables=storage_engine%3DInnoDB</jdbc.url>
								<jdbc.user>DonyCheckIntoSource</jdbc.user>
								<jdbc.password>Seriously</jdbc.password>
								<!-- allow cluster nodes to find each other over TCP/IP thus enabling clustering for this node -->
								<hazelcast.network.tcpip>true</hazelcast.network.tcpip>
								<!-- set to true if your load balancer supports stick sessions -->
								<hazelcast.http.stickysessions>false</hazelcast.http.stickysessions>
								<!-- forces Stash to fully finish starting up before yielding to the func test runner or atlas-run -->
								<johnson.spring.lifecycle.synchronousStartup>true</johnson.spring.lifecycle.synchronousStartup>
							</systemPropertyVariables>
							<libArtifacts>
								<!-- ensure MySQL drivers are available to Stash -->
								<libArtifact>
									<groupId>mysql</groupId>
									<artifactId>mysql-connector-java</artifactId>
									<version>5.1.32</version>
								</libArtifact>
							</libArtifacts>
						</product>
					</products>
					<!-- <log4jProperties>src/main/resources/log4j.properties</log4jProperties> -->
					<testGroups>
						<!-- tell AMPS / Maven which products ie nodes to run for the named testGroup 'clusterTestGroup' -->
						<testGroup>
							<id>clusterTestGroup</id>
							<productIds>
								<productId>bitbucket-node-1</productId>
								<productId>bitbucket-node-2</productId>
							</productIds>
						</testGroup>
						<testGroup>
							<id>default</id>
							<productIds>
								<productId>bitbucket-single</productId>
							</productIds>
						</testGroup>
					</testGroups>
				</configuration>
			</plugin>

 

The additional use of TestGroups is just an easy way for us to fire both products at once.

atlas-run --testGroup clusterTestGroup #slower start, test DC

OR

atlas-run --testGroup default  #faster start, snigle instance

 

Shared Directories and Database issues

The shared home path and limitations on RDS incomming connections added a bit of headache that the trivial script below solves by reducing the total DB connections to each node and pre-configuring their shared directory specified above.

#!/bin/bash

#bye bye whatever is there now!
echo "recreating home directories from scratch"
rm -R target/*


#creat cluster home dirs for mysql lib
mkdir -p target/stash-node-2/home/lib
mkdir -p target/stash-node-1/home/lib
mkdir -p target/stash-node-2/home/shared
mkdir -p target/stash-node-1/home/shared

# replace with path to your mysql/postgres driver jar.
cp src/test/resources/mysql-connector* target/stash-node-2/home/lib
cp src/test/resources/mysql-connector* target/stash-node-1/home/lib


echo "providing stash-config to clustered home for db settings"
# set stash-config.properties with defaults and force db pool below CF max limit of 40 connections
cat >>target/stash-node-1/home/shared/stash-config.properties <<EOF
logging.logger.com.atlassian.stash=INFO
logging.logger.com.atlassian.stash.internal.project=WARN
logging.logger.ROOT=WARN
feature.getting.started.page=false
plugin.branch-permissions.feature.splash=false
db.pool.partition.count=2
db.pool.partition.connection.maximum=9
EOF

cat target/stash-node-1/home/shared/stash-config.properties > target/stash-node-2/home/shared/stash-config.properties

echo "starting test group"
# run with cluster optipn
atlas-run --testGroup clusterTestGroup

Load Balancers

Since each cluster instance runs on a dedicated port, you can test the functionality works in each one by using the approproate port. 

To test a more real world scenario, you'll want to put a load balancer in front. Fortunately Atlassian thought about this need and released a simple plugin to drop a load balancer in front of your cluster as part of the testing lifecycle.

<plugin>
        <groupId>com.atlassian.maven.plugins</groupId>
        <artifactId>load-balancer-maven-plugin</artifactId>
        <version>1.1</version>
        <executions>
            <execution>
                <id>start-load-balancer</id>
                <phase>pre-integration-test</phase>
                <goals>
                    <goal>start</goal>
                </goals>
            </execution>
            <execution>
                <id>stop-load-balancer</id>
                <phase>post-integration-test</phase>
                <goals>
                    <goal>stop</goal>
                </goals>
            </execution>
        </executions>
        <configuration>
          <balancers>
              <balancer>
                <port>7990</port>
                <targets>
                  <target>
                    <port>7991</port>
                  </target>
                  <target>
                    <port>7992</port>
                  </target>
                </targets>
              </balancer>
              <balancer>
                <port>7999</port>
                <targets>
                  <target>
                    <port>7997</port>
                  </target>
                  <target>
                    <port>7998</port>
                  </target>
                </targets>
              </balancer>
          </balancers>
        </configuration>
      </plugin>

 

Credits

This article uses sample code snippets from an Open Source plugin I wrote for Liberty Mutual Insurance - https://github.com/Libertymutual/ssh-key-enforcer-stash/  which forces bitbucket SSH keys to be destroyed and rotated per company's predefined policy.

Much of the code is based on/copied from Atlassian tutorial https://developer.atlassian.com/bitbucket/server/docs/latest/how-tos/cluster-safe-plugins.html

 

Comment
Watch
Like # people like this
4130 views

3 comments

Comment

Log in or Sign up to comment
Jeet G
Jeet G July 20, 2017

If I follow these steps for a Jira plugin, will I have have data center compatability for my plugin? 
Note: This plugin will not be pushed into the marketplace

Like
EddieW
EddieW
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
Get recognized
July 20, 2017

Yes, so long as testing validates everything, the attribute in the xml is what tells JIRA your plugin is compatible.

Like
congstar GmbH
congstar GmbH March 15, 2019

Thank you!

Like
EddieW

EddieW

Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
Get recognized

About this author

117 accepted answers

565 total posts

View profile
groups-icon
Data Center
TAGS
AUG Leaders

Atlassian Community Events

    See all events