Introducing Atlassian Labs DC Weaver - formerly Atlassian Cloudformation Forge

[#1]

Atlassian Labs DC Weaver

NB: Atlassian Cloudformation Forge has been renamed to Atlassian Labs DC Weaver

---

Atlassian Labs DC Weaver is a tool for the creation and administration of CloudFormation stacks of Atlassian products, for users without physical access to the underlying AWS services.

The IT Platform Engineering team at Atlassian built this tool internally, to allow product teams to manage their own instances of Confluence, Jira, and Crowd without having access to the AWS console itself.

What can Weaver do?

Weaver can manage Cloudformation deployments of Jira Data Center, Confluence Data Center and Crowd Data Center.

Actions are split into three categories: Stack Actions, Node Management, and Application Management.

Stack Actions

These are actions that are performed against a Cloudformation stack of an Atlassian product.

• Create: Create new instances of Jira, Confluence or Crowd.
• Update Stack Configuration: Update template parameters for a stack.
• Tag Stack: Add tags to the stack, and all its resources.
• Clone Stack: Create a clone of an instance from database and filesystem snapshots. This requires clone templates (see the (now deprecated) public templates for an example). This can be used to automate creating of staging instances, and for DR.
• Destroy: Tear stacks down.

Node Management

These are actions that are performed against an EC2 node, or all the EC2 nodes in the Cloudformation stack.

• Toggle Node Registration: Remove or re-add a node from/to the load balancer.

• Rebuild Nodes: Perform a rolling rebuild of the nodes in the cluster. Wait for each new node to be registered in the load balancer, with a good index (if admin credentials are specified), before destroying the next. This can be used to change the underlying AWS instance type, or to obtain new AMIs.
• Re-initialize Nodes: Run a cfn-reinit on the nodes. This will apply any updates from the Cloudformation stack to the node, while retaining the local data, eg the index. This can provide faster rollouts of changes that do not change the underlying AWS instance type.

Application Management

These are actions that are performed against Jira/Confluence/Crowd.

• Single Node Restart: Restart one node in the cluster. This also gives you a preview of the CPU utilisation on the node, as high CPU is often a reason for needing to restart a node.
• Rolling Restart: Restart all nodes in the cluster one by one. Wait for each node to be re-registered in the load balancer before restarting the next.
• Full Restart: Restart all nodes in the cluster immediately.
• Upgrade: Enter the version you want to upgrade to, select if you want ZDU (for Jira) or Rolling (for Confluence), and select whether to replace the nodes or simply re-init them. If you enter administrator credentials here, each node will be deregistered from the load balancer until its index health reports > 98%, then it will be re-added to the load balancer, and the next node will be upgraded.
• Diagnostics: Run thread and heap dumps again each node in the cluster. Thread dumps are uploaded to S3, and can be downloaded from this tab. For Jira, you can check Index Health across the cluster here too.

• Run SQL: run a predefined SQL script against an instance(NB: script needs to be added to the service by an administrator - this is not free-form SQL).
• Smoke Tests: Run these automated smoke tests against a stack. You can choose whether to run the basic tests, or the modules that create/delete content (eg pages, issues).

The code

Weaver can be downloaded here: https://bitbucket.org/atlassian/atl-labs-dc-weaver/src

Pre-deployment configuration

1. You will need an AWS account, with either the AWS managed "AdministratorAccess" policy or the effective "Allow *" for the following services:
   1. EC2
   2. ECS
   3. ElasticLoadBalancing
   4. IAM
   5. CloudWatch
   6. Route53
   7. S3
   8. SSM 
2. Decide which AWS regions you want to create stacks in
3. Set up Route53 (optional)

Deployment

Weaver is deployed using the Cloudformation template in the Weaver repository.

The template creates:

1. An IAM role with all the access Weaver needs to operate
2. An EC2 node that downloads, configures and runs Weaver
3. A load balancer that can be either internal only or public facing
4. A CNAME for Weaver in your hosted zone (optional)

Post-deployment configuration

Weaver currently stores configuration parameters in:

• config/config.py
• saml_auth/permissions.json (optional - for SAML)

These are downloaded from S3, to persist settings across Weaver deployments. Once the files have been modified locally, you can upload them to S3 using the AWS CLI, eg:

aws s3 cp weaver/saml_auth/permissions.json s3://atl-labs-dc-weaver-<your-aws-account-id>/config/permissions.json

These will then be downloaded and applied to any new Weaver instance deployed in that AWS account.

Analytics

We have enabled simple Google analytics to get an idea of how many people are using Weaver. 

Analytics can be disabled in the Cloudformation template. If analytics are disabled after Weaver is created, spin down to 0 nodes using the template and back up to 1 to apply the change.

SAML

Weaver can use SAML authentication to define who can perform which actions, on which instances, in which regions.

For more information on how to enable SAML in Weaver, see: https://community.atlassian.com/t5/Data-Center-articles/Enabling-SAML-in-Atlassian-Labs-Data-Center-Weaver/ba-p/1781426

Templates

Weaver works with AWS Cloudformation templates. It automatically deploys the (now deprecated) public templates: https://bitbucket.org/atlassian/atlassian-aws-deployment/src/master/

These can be used as a reference, but they are no longer supported.

Weaver can be used with the Atlassian Quickstart templates, or these can be used as a reference to build your own templates.

You can deploy your own templates to Weaver by specifying the repository which contains the templates, in the Cloudformation template used to deploy Weaver.

For more information, see: https://community.atlassian.com/t5/Data-Center-articles/Using-custom-AWS-Cloudformation-templates-with-Weaver/ba-p/1781455

Support

Atlassian Labs DC Weaver is not supported by Atlassian. It is provided as is.

More information

More detailed information about Weaver (previously called Forge), and migrating to AWS CloudFormation in general, is provided in this video.

[Christian Reichert _resolution_]

Hi @Denise Unterwurzacher [Atlassian],

we just talked down in the Lobby great to meet you here - just reading your post, seems like your link under the SAML Section goes nowhere. Ends up in a "Forbidden".

Cheers,
Christian

[Denise Unterwurzacher [Atlassian]]

Hi @Christian Reichert _resolution_, great to meet you yesterday! That link should be open now, give it another go.

[Craig Castle-Mead]

Hi Denise,

Thanks for taking the time at Summit to demo this great tool. Definitely planning on checking it out and seeing how it can benefit our team.

CCM

[Corey Shields]

@Denise Unterwurzacher [Atlassian]Excited to try this!

Ran into an issue using the cf template to deploy forge, the CfnStackR53NodesSelfupdater policy does not exist - do we need to set that up first?  I see mention of it in the templates repo but it is no longer in the template, did it need removing from forge as well?

cheers!

[Bryan Robison]

@Denise Unterwurzacher [Atlassian] There's a change in itsdangerous that prevents Flask/Forge from running. You can get around it by installing a version of itsdangerous<1.0.0

https://stackoverflow.com/questions/52900312/flask-session-cant-import-want-bytes-from-itsdangerous

[Timothy Johnson]

Can anyone help me locate a cloudformation template for Mysql Jira?

[Dan Barnes]

@Denise Unterwurzacher [Atlassian]

When running locally (from the README)

FORGE.PROPERTIES

Used for configuring which AWS regions are available for stack creation/management, analytics collection, and S3 bucket definitions. A version of Forge deployed with the provided CloudFormation template will create this file on the EC2 node, but to run locally you'll need to create one on your local system.

--

There is no sample forge.properties included in the project for running locally. While I could tell what it wants from acforge.py code, it wasn't readily apparent what format to put it in.  I was able to reverse engineer the CloudFormation template to create the forge.properties file. There is no permissions on the repository currently to submit a PR against the project so Ill include an example of the contents here in case anyone else has this problem:

--

# Regions in format 'aws_region: region_name'

# Names are for reference only, so can be AWS region names or Staging/Production etc

# Enter default region first"

[analytics]

enabled: <true|false>

[s3]

bucket: <bucket-name>

[regions]

# aws_region: region_name
us-east-1: N.Virginia

[BaM]

Hi Dan. Thanks very much for bringing the forge.properties issue to our attention. A sample has now been added to the repo.

[Dan Barnes]

Thanks for adding this, perhaps one other request is to open up PRs to the community? I'd like to add a more friendly experience around the properties file. The tool should exit if a properties file is not present and direct the user to the example if its missing, something like the following:

diff --git a/acforge.py b/acforge.py

index 66d6bdb..447737c 100755

--- a/acforge.py

+++ b/acforge.py

@@ -46,6 +46,13 @@ args = parser.parse_args()

 # using dict of dicts called stackstate to track state of a stack's actions

 stackstate = defaultdict(dict)

 

+# if local forge.properties is missing exit

+try:

+    file = open('forge.properties', 'r')

+except Exception:

+    print('forge.properties file is missing. See forge.properties.example')

+    sys.exit(1)

+

 # create and initialize app

 print(f'Starting Atlassian CloudFormation Forge v{__version__}')

 app = Flask(__name__).  

[Rajat Gupta]

hi,

Can anyone please help me out how to pass s3 keys to forge. I am getting following error when I try to deploy jira stack

Traceback (most recent call last):
  File "/home/forge/atl-cfn-forge/forge/aws_cfn_stack/stack.py", line 968, in create
    Tags=tags,
  File "/home/forge/.local/share/virtualenvs/atl-cfn-forge-Sg4iAl_8/local/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/forge/.local/share/virtualenvs/atl-cfn-forge-Sg4iAl_8/local/lib/python3.6/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationError) when calling the CreateStack operation: S3 error: Access Denied

Thanks.

[Jarrett Goetz]

Hi @Denise Unterwurzacher [Atlassian]This looks like some great work.  Still completely maintained right?  (Fully understand it's not officially supported by Atlassian.)

Thanks.

Jarrett

[Denise Unterwurzacher [Atlassian]]

@Jarrett Goetz Absolutely it's still maintained :) Though yes, not officially supported.