Upgrade Best Practices

[#1]

Hello!

My name is Mark Askew and I am a Premier Support Engineer for products Bitbucket Server/Data Center, Fisheye & Crucible.

Today, I want to bring the discussion that Jennifer, Matt, and myself had on the sparsely updated Admins of Atlassian podcast to Community: Upgrade Best Practices. Below you will find an accumulation of our tips and best practices for upgrading your environments.

This is a big post and while not originally conceived for a written format, I took some highlights from the episode and a presentation I gave to the Austin Atlassian User Group to share here.

And away we go!

Disclaimer: The below info is not product specific and does not replace what we may include in our upgrade guides or other pieces of official documentation. It’s more meant to share from a few of our collective experiences to add to your own. This is certainly not a comprehensive guide as I know there are some awesome tips and best practices that you all have as well.

Why You Should Upgrade

We HAVE to start with this. All environments are not the same. New admins may have a vague idea of why they should upgrade, others may take an “if it’s not broke, don’t fix” approach. (I’ve personally had to battle this in my past careers) Some may want to upgrade but need a little help in providing that business justification to their management. All in all, there are four main reasons you should upgrade your applications:

Bugs, Features, Security & Support

Bugs - Each release contains bugs we’ve squashed and fixed. These could be bugs that never triggered in your system or bugs that are currently impacting your workflow now.

Features - Not only would you get those bug fixes, but you will also get a slate of new features that add more functionality to the app. For example; maybe you’ve always wanted to squash commits on your pull request merge in Bitbucket Server OR maybe you’ve been counting down the days for issue priority schemes in Jira. When you upgrade, those must-have features and more are now yours.

Security - We release security vulnerability fixes that will require you to upgrade to receive the fix. Staying up-to-date will keep your application all the more secure.

Support - Atlassian provides support with your license for anything that is two years or newer. Meaning, if the version of the product you are running is 2.5 years, or 4 years old, you are now out of support and will need to upgrade to a supported version of the application.

Putting in the care to maintain the application for your users and yourself can help things go smoothly as:

• you are always within the software versions that Atlassian supports

• you are delivering new features and functionality to your users

• you are making your life as an admin much easier when you ‘tend to the garden’

Bonus Reason - It keeps your environment and tools up-to-date.

If you installed a piece of software and decided to never touch it, you may be in for a surprise when you are forced to upgrade your database due to DBA requirements, OR migrated to a new server that has a more recent OS version and find that the application no longer supports the environment that you’re in or other tools needed to support the application are no longer supported.

Keeping up with software upgrades have the added benefit of keeping everything else in your environment updated as well.

I’m Convinced, What Next?

Start planning.

If you don’t have a plan, you’re going to create one. This plan will be your foundation for every upgrade going forward. So, what is in it?

#1 Upgrade Cadence

Upgrade cadence is asking “how often you upgrade your applications?” Most probably upgrade whenever they want. If this works for you, great! If you are possibly in an environment where maintenance activities like this are shelved for other projects, then this can help set the stage in getting your upgrade done.

Decide:

• Can you upgrade yearly?

• Twice a year?

• Every three months?

• What does your cadence look like?

The more specific the better.

For a while, I upgraded twice a year and then included special upgrades for any bugs, security fixes, or features I wanted. As my instances grew, it then became one major upgrade per year. In fact, my Jira production upgrade happened in either mid-October or November. And Confluence occurred in the spring. This way, my users, and management knew that every October or November Jira was being upgraded and all activities such as team projects, and development releases could be planned around it.

In this step you will:

• Choose when and how often you upgrade. The more specific, the better.

#2 Identify Interested Parties

You want to document the users that are in your system; are they Finance? Developers? Conversion teams? Do you have a way to communicate with them all? Who runs your servers? You? Great. A VM team? Awesome!

You will need to identify contacts and on-call support. This should already be documented when the environment was built, so you will want to ensure that you have them noted, and the information is current.

Knowing this information will not only help you with upgrades, but it can greatly assist when you are troubleshooting issues in your environment for non-upgrade attempts and it helps you have a better understanding of who is using the applications you manage and how they are using the applications you manage.

Example:

My Interested Parties (IP) Are:

• Users: All users of the application.

• Infrastructure:

  • John Doe built my VM

  • “Page Us” is the emergency contact.

• DBA - Contacts who support my database.

  • Jane Doe built my database

  • “Awesome DBA’s” is the messaging room for help.

  • “We’re Not DBA’s” is the emergency contact group.

• Networking: Team that supports my network.

  • Nate Doe originally worked my configuration

  • Sally Doe is the emergency contact

• Integration Developers: Users and or teams that built integration or automation around my applications.

  • Internal App Team A: Wrote custom scripts to pull from Jira.

  • Jared Doe: Wrote a custom plugin for Confluence.

  • Internal App Team B: App linked their Jira instance to ours.

In this step you will:

• Create your own IP Doc. If you don’t have one, create one!

#3 Audit Environment

If not documented, you should perform an audit of your environment to know what and where everything is:

• What version is my app, database, and OS?

• What is the disk space on my app, database, and web server?

• What is the backup schedule?

• What apps integrate to mine?

• What customizations have I made? (heap, SSL, property configs)

• What versions are the apps (plugins) in my Atlassian software?

Supported Platforms

After performing the audit, ask yourself: “Can I run the latest version of Atlassian software and continue to be supported?”

Atlassian provides a Supported Platforms page for each of our applications. You select the version of the software you are interested in, and see what it does and does not support.

This is key as you can reference this section of your upgrade plan to see what, if any, other platforms would need to be upgraded now OR should be upgraded in the future.

• Jira Supported Platforms

• Bitbucket Server/Data Center Supported Platforms

• Fisheye Supported Platforms

• Crucible Supported Platforms

• Bamboo Supported Platforms

• Confluence Supported Platforms

Release Notes / Upgrade Guide

Bookmark the release note pages for each of the products in your environment. Also, subscribe to release notifications. This way you can see what features and fixes are being released over time and get details on what is needed for that particular upgrade. For example, a particular upgrade could require to reindex of your Fisheye repositories, or of your Jira instance. There could also be API changes. This will be critical to know for your planning purposes.

• Crucible upgrade guide

• Fisheye upgrade guide

• Bamboo upgrade guide

• Bitbucket Server upgrade guide

• Jira Software upgrade hub

• Confluence upgrade guide

In this step, you will:

• Audit your environment. Document everything about your instance and environment.

• Update your mail settings to stay up-to-date https://my.atlassian.com/email

#4 Build Your Test Environment

Dev/Test/CAT/Stage

Do you have a staging or test environment? What about a dev environment?

I used to hear that setting up such environments was overdoing it. I very much disagree.

Setting up these environments or at the very least a testing environment will allow you to test your upgraded components before making them in production. When creating this environment, it should match as close to possible as your production environment. I say “close as possible” as some larger enterprise environments won’t allow, from a budgetary, or logistical perspective, you to have the same setup as production.

In addition to replicating the production setup where possible, you should populate this platform from a copy of your production. For our applications, we include instructions on how one can set up a staging platform. All include steps in using a backup of your production environment to populate these additional platforms.

When I was managing Jira and Confluence it took a while to obtain all the environments I needed: Dev. Test & Prod HA (pre-Data Center) but when I received them, I was such a happy admin. For those that have yet to establish such environments, here is how I utilized the platforms.

To be clear, you are not testing the application. This was already tested. What you are testing is your environment and data against the application. This is small change in how to think about your testing. Once you change to that mindset, recreating the production environment as close as possible and refreshing your test platforms with production data will become a necessary part of your testing.

How I used my environments:

Dev - This platform was pretty much anything you wanted to do, you could. It was primarily restricted to my fellow admins. We tested early look at plugins, custom configuration changes, as well as application upgrades, database upgrades, and more. A lot of things we tested never migrated to Test. And everything that migrated to Production HAD to hit Dev first unless it was a production hotfix (customization).

Test - This platform was controlled. We kept a calendar on what was placed on this platform. Workflow changes, plugin evaluations, etc. My team would test and verify first, and then open it up for user acceptance. Later I would implement load & performance testing on this platform.

Production HA - This was a cold-standby environment in the event production had a hard down due to the application, or other environmental impacts. I rotated between servers every quarter to ensure all configuration worked, replication worked, and we knew the process for hot swaps. For example; we encountered a disk latency issue on our primary production server. Given the time our infrastructure team needed to resolve this issue, I performed emergency maintenance and swapped to our standby server which was not affected. Convincing management to give me such a platform paid off big time.

In this step you will:

• If you don’t have testing platforms, think about how you could use them and how they can benefit your process.

• If you already have such environments, are you using them fully to test and prep changes before moving to prod?

#5 Backups

I have a few questions for you:

• Do you have backups?

• Do you know when they run?

• Are the backups for the App and Database taken close together? Or are they hours apart?

• Have you tested your backups?

Having a backup is key to your upgrade plan. Most Atlassian tools do have built-in automatic backups which are great for smaller instances but as you grow it is best to look into using an external system or any backup utilities we provide.

Now, you may be wondering “Duh, who doesn’t have backups?” There are many that find that they do not have a backup. Not just of the application but of the database and anything else they’ve modified. So, if you’re thinking to yourself “I don’t have a backup”, your number 1 priority is to get your systems backed up and your number 2 priority is to define your backup strategy.

I also recommend keeping your application and database backups as close together as possible so that there aren’t any gaps in data. This is critical when it comes to applications such as Bitbucket Server.

P.S. VM snapshots are not backups and your VM provider agrees.

In this step you will:

• Document your backup strategy (time, location)

#6 Testing Tips

There is a lot to testing as you can imagine so here are some high-level things for you to think about:

Backups - Your first test is to test your backup. This is restoring your production back up to your Testing or Dev platform. Does the backup work? Do you understand the process? It is not just enough to take a backup, but you have to make sure that the backup and the recovery process works.

Application - Once you’ve established your platform. Review the release notes and supported platforms for the targeted version you are upgrading to. Will your plugins support this version? Is your database supported?

Application Pro-tip - Use the application for your test scripts. In Confluence, create a page that has third-party plugins active on the page. When you look at it, is it broken? Do they all work? In Jira, create a testing project that contains a mix of your custom workflows, screens, fields, etc. Are they functional?

Plugins/Apps - Plugins should be reviewed and upgraded throughout the year to provide new features, fixes, and functionality to your users, therefore, they should follow the same process as if you are upgrading the application itself. Don’t forget, these plugins could be writing data to the database as well, so you should always test plugins before placing in Production and reduce your risk by upgrading them throughout the year as opposed to all at once when you upgrade the application.

Reduce Change - Reduce as much change as possible. When you are planning your upgrade and find that you need to upgrade your database or operating system, silo those changes unto themselves. Meaning, don’t upgrade your operating system and application at the same time. If you encounter an issue, what caused it? The OS upgrade? Or the app upgrade? Or maybe it was the migration to a newly built server? How do I find out which change caused my issue?

Ensnare Your Admins - On the Admins of Atlassian Podcast, I talked about using your administrators to expand your testing team. For example; all of my Jira admins had to sign off that they tested their Jira project and encountered no issues. Same as my Confluence administrators. This allowed a two-person team to expand to a mighty 30+ users.

Rollback - Don’t think that we are done with testing our backup. You’ve upgraded and tested BUT, let’s pretend something went horribly wrong. How do you revert back? This is where testing your backup comes back. Restore your application and database to the point before the upgrade and verify the application. This helps you understand the rollback process so that it isn’t a foreign process to you when the time comes. And if that time comes, you’re already under enough stress.

Rollback Pro-tip - Define your cutoff date for rolling back. If you upgraded over the weekend and run into issues on Monday - when do you decide to rollback your upgrade, migration, or customization? Is it Monday at 5 pm? Tuesday at Noon? The earlier you determine this, the less impact to your users.

Takeaways

• Test your backups & your recovery plan

• Know your environment & your users

• Communicate every step of the way

• Build your testing within the tool

• Silo your upgrade components

• Set a rollback date

• Save yourself time and upgrade plugins throughout the year

And that’s it! Whew! Thanks for sticking with me to the end. I hope that the above got you thinking about your environments and upgrade process.

Did I miss anything? Have you developed your own best practices for managing and executing your upgrades? Please share below! I’d love to hear what others may do to help their upgrades go smoothly.

[Kat Warner]

This is a great outline of the process. Thanks!

[Mark A.]

Thanks, Kat!

[Gonchik Tsymzhitov]

Thanks for systemizing of the upgrade process :)

[stidler]

Can you please look into this issue:

https://community.atlassian.com/t5/Bitbucket-questions/Push-502-BAD-GATEWAY/qaq-p/946620#U947156

[Mark A.]

^^ Jack was able to get his issue resolved - https://community.atlassian.com/t5/Sourcetree-questions/502-bad-gateway/qaq-p/947466

[abhikbin]

Hi 

Can you have look at this

https://community.atlassian.com/t5/Bitbucket-questions/we-are-planning-to-host-bitbucket-server-on-Azure-Cloud-Do-we/qaq-p/1006404?utm_source=atlcomm&utm_medium=email&utm_campaign=kudos_question&utm_content=topic

I am specifically looking for bitbucket server

[Mark A.]

^^ Abhik's question has been answered in the linked post.

[Alexandra Micallef]

A useful list of tasks to overview when planning an upgrade

[Ravi GH]

Thanks for the comprehensive info.

[Gustavo Chaves]

Since our Atlassian tools are installed on AWS EC2 servers, it's easy to create a Test environment by simply "cloning" the production server.

Someone questioned me recently if we could be violating Atlassian's EULA by cloning the production server. But even if we restore a backup of our production in another machine, as you suggest in step #4, the situation would be the same: we would have two instances of one Atlassian tool running with the same license.

Should I be concerned with this?

I just read section 3.3 (Number of instances) of Atlassian's Software License Agreement (https://www.atlassian.com/legal/software-license-agreement) and it says explicitly that I "may install one (1) production instance of the Software". It also says that I can request "developer" licenses for non-production instances, though. Does this mean that I should request developer licenses for our products and install them in our Test environments? Even if we just use them for a few hours or days to exercise and test the upgrade procedure and nobody else uses them for service?

Anyways, this is such a useful post. I'll make sure to update our own upgrade procedure with some of your tips. Thanks!

[Mark A.]

Hey, @Gustavo Chaves ,

Thanks for commenting.

We do note, regarding the license, what steps to take when setting up a stage environment. With each production license, you have access to a developer license. Once your stage environment is setup, you’ll need to apply the developer license. You can access the license on your  my.atlassian .com profile.

Resource - https://confluence.atlassian.com/bitbucketserverkb/how-to-establish-staging-server-environments-for-bitbucket-server-779171160.html

To answer your question:

Does this mean that I should request developer licenses for our products and install them in our Test environments? 

Yes. It will be best practice to apply the developer license when making the necessary changes after standing up your Stage/Test/Dev server. This will ensure that things are in proper order, especially if you reach out for any upgrade assistance.

I hope that helps!

Cheers,

Mark