My name is Mark Askew and I am a Premier Support Engineer for products Bitbucket Server/Data Center, Fisheye & Crucible.
Today, I want to bring the discussion that Jennifer, Matt, and myself had on the sparsely updated Admins of Atlassian podcast to Community: Upgrade Best Practices. Below you will find an accumulation of our tips and best practices for upgrading your environments.
This is a big post and while not originally conceived for a written format, I took some highlights from the episode and a presentation I gave to the Austin Atlassian User Group to share here.
And away we go!
Disclaimer: The below info is not product specific and does not replace what we may include in our upgrade guides or other pieces of official documentation. It’s more meant to share from a few of our collective experiences to add to your own. This is certainly not a comprehensive guide as I know there are some awesome tips and best practices that you all have as well.
We HAVE to start with this. All environments are not the same. New admins may have a vague idea of why they should upgrade, others may take an “if it’s not broke, don’t fix” approach. (I’ve personally had to battle this in my past careers) Some may want to upgrade but need a little help in providing that business justification to their management. All in all, there are four main reasons you should upgrade your applications:
Bugs - Each release contains bugs we’ve squashed and fixed. These could be bugs that never triggered in your system or bugs that are currently impacting your workflow now.
Features - Not only would you get those bug fixes, but you will also get a slate of new features that add more functionality to the app. For example; maybe you’ve always wanted to squash commits on your pull request merge in Bitbucket Server OR maybe you’ve been counting down the days for issue priority schemes in Jira. When you upgrade, those must-have features and more are now yours.
Security - We release security vulnerability fixes that will require you to upgrade to receive the fix. Staying up-to-date will keep your application all the more secure.
Support - Atlassian provides support with your license for anything that is two years or newer. Meaning, if the version of the product you are running is 2.5 years, or 4 years old, you are now out of support and will need to upgrade to a supported version of the application.
Putting in the care to maintain the application for your users and yourself can help things go smoothly as:
you are always within the software versions that Atlassian supports
you are delivering new features and functionality to your users
you are making your life as an admin much easier when you ‘tend to the garden’
Bonus Reason - It keeps your environment and tools up-to-date.
If you installed a piece of software and decided to never touch it, you may be in for a surprise when you are forced to upgrade your database due to DBA requirements, OR migrated to a new server that has a more recent OS version and find that the application no longer supports the environment that you’re in or other tools needed to support the application are no longer supported.
Keeping up with software upgrades have the added benefit of keeping everything else in your environment updated as well.
If you don’t have a plan, you’re going to create one. This plan will be your foundation for every upgrade going forward. So, what is in it?
Upgrade cadence is asking “how often you upgrade your applications?” Most probably upgrade whenever they want. If this works for you, great! If you are possibly in an environment where maintenance activities like this are shelved for other projects, then this can help set the stage in getting your upgrade done.
Can you upgrade yearly?
Twice a year?
Every three months?
What does your cadence look like?
The more specific the better.
For a while, I upgraded twice a year and then included special upgrades for any bugs, security fixes, or features I wanted. As my instances grew, it then became one major upgrade per year. In fact, my Jira production upgrade happened in either mid-October or November. And Confluence occurred in the spring. This way, my users, and management knew that every October or November Jira was being upgraded and all activities such as team projects, and development releases could be planned around it.
In this step you will:
Choose when and how often you upgrade. The more specific, the better.
You want to document the users that are in your system; are they Finance? Developers? Conversion teams? Do you have a way to communicate with them all? Who runs your servers? You? Great. A VM team? Awesome!
You will need to identify contacts and on-call support. This should already be documented when the environment was built, so you will want to ensure that you have them noted, and the information is current.
Knowing this information will not only help you with upgrades, but it can greatly assist when you are troubleshooting issues in your environment for non-upgrade attempts and it helps you have a better understanding of who is using the applications you manage and how they are using the applications you manage.
My Interested Parties (IP) Are:
Users: All users of the application.
John Doe built my VM
“Page Us” is the emergency contact.
DBA - Contacts who support my database.
Jane Doe built my database
“Awesome DBA’s” is the messaging room for help.
“We’re Not DBA’s” is the emergency contact group.
Networking: Team that supports my network.
Nate Doe originally worked my configuration
Sally Doe is the emergency contact
Integration Developers: Users and or teams that built integration or automation around my applications.
Internal App Team A: Wrote custom scripts to pull from Jira.
Jared Doe: Wrote a custom plugin for Confluence.
Internal App Team B: App linked their Jira instance to ours.
In this step you will:
Create your own IP Doc. If you don’t have one, create one!
If not documented, you should perform an audit of your environment to know what and where everything is:
What version is my app, database, and OS?
What is the disk space on my app, database, and web server?
What is the backup schedule?
What apps integrate to mine?
What customizations have I made? (heap, SSL, property configs)
What versions are the apps (plugins) in my Atlassian software?
After performing the audit, ask yourself: “Can I run the latest version of Atlassian software and continue to be supported?”
Atlassian provides a Supported Platforms page for each of our applications. You select the version of the software you are interested in, and see what it does and does not support.
This is key as you can reference this section of your upgrade plan to see what, if any, other platforms would need to be upgraded now OR should be upgraded in the future.
Release Notes / Upgrade Guide
Bookmark the release note pages for each of the products in your environment. Also, subscribe to release notifications. This way you can see what features and fixes are being released over time and get details on what is needed for that particular upgrade. For example, a particular upgrade could require to reindex of your Fisheye repositories, or of your Jira instance. There could also be API changes. This will be critical to know for your planning purposes.
In this step you will:
Audit your environment. Document everything about your instance and environment.
Update your mail settings to stay up-to-date https://my.atlassian.com/email
Do you have a staging or test environment? What about a dev environment?
I used to hear that setting up such environments was overdoing it. I very much disagree.
Setting up these environments or at the very least a testing environment will allow you to test your upgraded components before making them in production. When creating this environment, it should match as close to possible as your production environment. I say “close as possible” as some larger enterprise environments won’t allow, from a budgetary, or logistical perspective, you to have the same setup as production.
In addition to replicating the production setup where possible, you should populate this platform from a copy of your production. For our applications, we include instructions on how one can set up a staging platform. All include steps in using a backup of your production environment to populate these additional platforms.
When I was managing Jira and Confluence it took a while to obtain all the environments I needed: Dev. Test & Prod HA (pre-Data Center) but when I received them, I was such a happy admin. For those that have yet to establish such environments, here is how I utilized the platforms.
How I used my environments:
Dev - This platform was pretty much anything you wanted to do, you could. It was primarily restricted to my fellow admins. We tested early look at plugins, custom configuration changes, as well as application upgrades, database upgrades, and more. A lot of things we tested never migrated to Test. And everything that migrated to Production HAD to hit Dev first unless it was a production hotfix (customization).
Test - This platform was controlled. We kept a calendar on what was placed on this platform. Workflow changes, plugin evaluations, etc. My team would test and verify first, and then open it up for user acceptance. Later I would implement load & performance testing on this platform.
Production HA - This was a cold-standby environment in the event production had a hard down due to the application, or other environmental impacts. I rotated between servers every quarter to ensure all configuration worked, replication worked, and we knew the process for hot swaps. For example; we encountered a disk latency issue on our primary production server. Given the time our infrastructure team needed to resolve this issue, I performed emergency maintenance and swapped to our standby server which was not affected. Convincing management to give me such a platform paid off big time.
In this step you will:
If you don’t have testing platforms, think about how you could use them and how they can benefit your process.
If you already have such environments, are you using them fully to test and prep changes before moving to prod?
I have a few questions for you:
Do you have backups?
Do you know when they run?
Are the backups for the App and Database taken close together? Or are they hours apart?
Have you tested your backups?
Having a backup is key to your upgrade plan. Most Atlassian tools do have built-in automatic backups which are great for smaller instances but as you grow it is best to look into using an external system or any backup utilities we provide.
Now, you may be wondering “Duh, who doesn’t have backups?” There are many that find that they do not have a backup. Not just of the application but of the database and anything else they’ve modified. So, if you’re thinking to yourself “I don’t have a backup”, your number 1 priority is to get your systems backed up and your number 2 priority is to define your backup strategy.
I also recommend keeping your application and database backups as close together as possible so that there aren’t any gaps in data. This is critical when it comes to applications such as Bitbucket Server.
P.S. VM snapshots are not backups and your VM provider agrees.
In this step you will:
Document your backup strategy (time, location)
There is a lot to testing as you can imagine so here are some high-level things for you to think about:
Backups - Your first test is to test your backup. This is restoring your production back up to your Testing or Dev platform. Does the backup work? Do you understand the process? It is not just enough to take a backup, but you have to make sure that the backup and the recovery process works.
Application - Once you’ve established your platform. Review the release notes and supported platforms for the targeted version you are upgrading to. Will your plugins support this version? Is your database supported?
Application Pro-tip - Use the application for your test scripts. In Confluence, create a page that has third-party plugins active on the page. When you look at it, is it broken? Do they all work? In Jira, create a testing project that contains a mix of your custom workflows, screens, fields, etc. Are they functional?
Plugins/Apps - Plugins should be reviewed and upgraded throughout the year to provide new features, fixes, and functionality to your users, therefore, they should follow the same process as if you are upgrading the application itself. Don’t forget, these plugins could be writing data to the database as well, so you should always test plugins before placing in Production and reduce your risk by upgrading them throughout the year as opposed to all at once when you upgrade the application.
Reduce Change - Reduce as much change as possible. When you are planning your upgrade and find that you need to upgrade your database or operating system, silo those changes unto themselves. Meaning, don’t upgrade your operating system and application at the same time. If you encounter an issue, what caused it? The OS upgrade? Or the app upgrade? Or maybe it was the migration to a newly built server? How do I find out which change caused my issue?
Ensnare Your Admins - On the Admins of Atlassian Podcast, I talked about using your administrators to expand your testing team. For example; all of my Jira admins had to sign off that they tested their Jira project and encountered no issues. Same as my Confluence administrators. This allowed a two-person team to expand to a mighty 30+ users.
Rollback - Don’t think that we are done with testing our backup. You’ve upgraded and tested BUT, let’s pretend something went horribly wrong. How do you revert back? This is where testing your backup comes back. Restore your application and database to the point before the upgrade and verify the application. This helps you understand the rollback process so that it isn’t a foreign process to you when the time comes. And if that time comes, you’re already under enough stress.
Rollback Pro-tip - Define your cutoff date for rolling back. If you upgraded over the weekend and run into issues on Monday - when do you decide to rollback your upgrade, migration, or customization? Is it Monday at 5 pm? Tuesday at Noon? The earlier you determine this, the less impact to your users.
Test your backups & your recovery plan
Know your environment & your users
Communicate every step of the way
Build your testing within the tool
Silo your upgrade components
Set a rollback date
Save yourself time and upgrade plugins throughout the year
And that’s it! Whew! Thanks for sticking with me to the end. I hope that the above got you thinking about your environments and upgrade process.
Did I miss anything? Have you developed your own best practices for managing and executing your upgrades? Please share below! I’d love to hear what others may do to help their upgrades go smoothly.
Mark AskewAtlassian Team
Announced in this blog, this holiday season we’re celebrating all things CI/CD and between now and the end of 2019 we’ll be showcasing content, use cases, feature announcements and more. One featur...
Connect with like-minded Atlassian users at free events near you!Find an event
Connect with like-minded Atlassian users at free events near you!
Unfortunately there are no Community Events near you at the moment.Host an event
You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events