Our path to merging Cloud Data into a Jira & Confluence Server Environment
Merging Cloud Data into a Jira & Confluence Server Environment
What & why?
Merging data from one Jira/Confluence server into another Jira/Confluence server is one thing. Merging data from Jira/Confluence Cloud into a Jira/Confluence server environment is another. However, I was tasked with this fun & challenging project at one of our customers, DPG Media.
At our customer, DPG Media, several teams had started using both Jira & Confluence over the last couple of years, resulting in a big Jira and Confluence server environment setup with over 2000 active users all logging in with their active directory credentials. However, some teams had been working on a different Jira or Confluence environment being it due to company merges or other causes, and some of these instances were living in the cloud.
The downsides here are obvious. Certain users had to go to different URLs to work on different projects in different Jira/Confluence instances. On top of this, there are some major differences in Cloud vs Server deployments (think UI and plugins) which made it unclear to some users why they saw things differently in what they thought was the same tool. Oh yeah, and don't forget the double license cost of course.
In the vision of bringing the teams that used Jira or Confluence closer together, it was time to merge the different environments.
Cloud to Server? Or Server to Cloud?
Each scenario is different, each company is different, but also each migration path is different. However, there are some questions you need to ask yourself when starting a migration or merger project like this.
- Which default functionalities do we lose when migrating to Cloud/Server?
- Which of our installed plugins are (not) available on Cloud/Server?
These 2 are major factors. There are plenty of plugins available for each deployment option, but not every plugin that you currently use, is available for both Cloud and Server.
Other things to worry about is the impact of the data. In most cases it will make more sense to migrate the smallest environment into the bigger one. If however, you choose to merge a bigger Cloud instance into a 'smaller' Server instance, because of certain plugins/functionalities for example, you may need to think about the extra load on the server. Do you have to upgrade your server hardware as well for example?
And of course, it's definitely worth comparing the license cost for both. In our case, we had enough licenses left on our server instance to absorb all the extra users that came from cloud (some users already had an account in the server instance AND the cloud instance).
Requirements & problems
Before developing a migration plan, we first had to fully understand how our requirements could impact the merging and had to tackle some problems before starting the process.
- Next Gen projects currently don't exist on server instances. The cloud instance did use this type of project. In some cases, we were able to delete the projects because we didn't need them anymore, but for others we first had to migrate the issues to a default software project. Keep in mind that Next Gen projects will simply block the option to export cloud data for server instances.
- We made decisions on certain plugins used on cloud regarding whether or not we would use their server version (if it existed) or how we could possibly migrate data from one plugin to a different (alternative) plugin on server.
- Cloud usernames did not match the usernames defined in Active Directory. So simply importing the data as is, would mean that internal jira users would be created and the data wouldn't be linked to the correct AD accounts. We wanted to link all data to the same user as on cloud. All users were either available or created in the AD before we started the migration.
- Jira macro's on Confluence would break as the data wouldn't be configured with the correct application link.
- If a project key already existed on the server instance, we had to update the project key on the cloud data. However this would mean that imported filters would possibly still refer to the old project key and therefor show the wrong issues after being imported on the server instance.
- Duplicate configuration or scheme names would overwrite the existing server configuration, so all scheme names or configuration names had to be unique.
- Statuses, issue types, resolutions, etc are all risky things to import. A status named "BACKLOG" on server, but "Backlog" on the cloud, didn't correctly match while importing at the time, but the lower case status couldn't be created as Jira sees statuses as unique and ignores casing in that regards.
Many other different problems occurred, such as status categories being overwritten, issue linking descriptions, etc. All this had to be fixed for the eventual production migration. Simply put:
- All usernames should be correctly matching the usernames in AD
- No existing production data or configuration should be overwritten
These two requirements were simple, but required a lot of testing and attention.
Our path to merging
Our migration path was actually pretty simple, but required some extra work. Jira was obviously the biggest challenge as we could simply use Confluence's default space export and import. But for Jira, we chose to use the amazing Project Configurator plugin by Adaptavist. The plugin is amazing, the support is extraordinary!
"But the plugin is server only?"
-Yes, correct!
"You migrated from cloud to server?"
-Yes, correct!
"Que?"
Our secret in the migration process, is to simply provide a middle server which hosts a clean Jira installation, a clean database and a clean Confluence installation. Look at the picture below to better understand our process.
Why the temporary middle server?
Having a server in the middle was a helpful tool for several reasons. Let it first of all be clear that the middle server was a clean install of Jira and Confluence and didn't contain any production data, configuration or schemes.
On this middle server we imported the full site backup that was generated from cloud (both for Jira and for Confluence). We used the developer license for Jira and Confluence. On the middle server, we created a Jira User server and linked it to the Confluence instance. This allowed us to re-use the exact same user base as existed on Cloud. The Jira User server was put to the top of the list in the Confluence user directories, so Confluence would use the usernames from Jira's directory rather than its own internal directory. Remember that both Jira and Confluence's internal directory hold the same users and usernames as they are included in the cloud export. Linking them together allows you to change usernames in Jira and automatically apply that to Confluence.
This should ring a bell and remind you of the mismatching username problem. We created a script that ran over a spreadsheet of usernames and email addresses and applied changes through the Jira API. Via the API, each email address was put in a search query and returned the available user on the middle server (which came from cloud) and then updated this cloud-imported username to the username defined in the spreadsheet (which were the usernames used in the active directory).
Because of this change, all data in Jira was mapped correctly to the new username. And as the cherry on top, because of the Jira user server setup, all Confluence data was also automatically updated to the new username.
Cool! But is that the only reason for the setup?
Nope! We also used the middle server to update all configuration that came from cloud. We matched the (from cloud imported) configuration to the configuration that was used on the production instance. Think of changing project keys, issue types, statuses, resolutions, priorities, etc.
It was important to change these settings here for many reasons. First, it allowed us to apply the configuration changes without interrupting the cloud instance setup (as they continued to use the cloud instance until the actual merge). But next to this, it also allowed us to see that for example a priority scheme doesn't exist on cloud yet, and we therefor had to create a default one on the server (so that the existing default on production wouldn't be overwritten).
And of course, remember that the Project Configurator plugin was only available on server.
On top of this, we had a safe way to use database queries to fix Jira macro's that pointed to the wrong application links (they still pointed to cloud).
Last but definitely not least, we got to install the same plugin versions as the production server to increase the chances of a clean plugin data transfer.
What after the middle server changes?
Now it was time to export all projects (data and configurations) and upload these configurations to production (which should have no impact on existing configurations as we updated everything on the middle server). You'll get an overview by Project configurator of the changes that are being applied to existing configuration, go through these carefully to see if you missed something.
In the first batch, we choose not to import boards, filters and dashboards as they need extra attention. For example, usernames that had been changed, would break filters. Same goes for renamed projects or project keys. We fixed these by extracting the config.xml file out of the full export zip and updating all references that were used in the <queries> tags with the find & replace tool which is available in most text editors.
Import the updated XML file and you're done... with Jira.
Confluence
Confluence at this point is easy (if you don't have corrupted spaces or duplicate space key problems). Simply go and export the XML for each space individually and import them, one by one, on the production Confluence instance.
For duplicate space keys, we used the ScriptRunner for Confluence plugin's "copy space" functionality. This allowed us to copy the entire space to a new key.
Conclusions
The merging of the cloud data was successful for our instance. However, no merge or migration goes without some bumps in the road. The advise here is to plan enough time for testing and provide some buffer for unexpected errors or problems that you didn't think of before hand. And again, test, test and test. (You can do a full test migration if you have a staging server that is a copy of your production instance.) We did a full migration test multiple times as we had a staging server available.
I couldn't include all problems or attention points in this article (you see it's already quite long). A migration or merger is a big thing, never expect this to go smoothly from the first try. If you need help with migrations, have any questions questions or would like to learn more, feel free to post something in the comments.
If you are interested in migrating, but would rather leave this to an Atlassian Solution Partner (I work for Solution Partner Abano), you can always send us an email via contact@abano.be
Thanks for reading!
1 comment