This article is another part of a story related to JIRA and Confluence split. If you are interested in how we prepare for that and actually made this migration go to previous parts (Part 1 and Part 2). If you are already done with migration or simply what to know what to do after then continue reading since this is the best place to learn more about it from practice.
Preparing and Doing a migration is one thing but what we should do later? We are in the situation that we have now two separate instances that was a one some time ago. Now it is a question what would Monday morning introduce when everyone would start using a new system..
Having experience with a lot of maintenance work done over the weekend it is not hard to know that real test is coming. Only normal day to day traffic could show us how things are working and how migration went. We did know that we have to pay attention to everything and there would be a lot of support coming. People would have problems with everything.. Some of them would try to continue to use old system, some would notice new problems.. Overall after such a huge change for everyone always mean that we put all hand on deck.
Step 9: Support
Support.. This was something that you cannot prepare.. We started a brand new system with old data and it is now a mystery how it would behave. We had many questions.. How good would be performance, would everyone be able to log in, would everyone see what saw before migration. We actually needed to create a new support project in Jira (since old were left in source Jira), so people had to get used to new rules of creating a tickets.. Overall so many changes..
Business wake up on 7 AM, but most of people start on 9 AM.. On 10 AM we were pretty shocked since it was.. quiet! This mean that either people have problems but they do not know how to report them and accumulation would be ever email or something or actually everything is fine and it was fully transparent to all, goal reached!
We continue to use our monitoring tools (we use Zabbix and Grafana) for that. Noticed that performance is sometimes on the edge but never actually crossed it.. suddenly 1000 users would have to get all pages cached so we should not touch yet anything until it would be really bad. It was a good decision from hour to hour it was reducing so it mean that the heaviest day of the week finally passed!
Of course we handled more tickets than usual mostly related to migration. People (of course as usual) tried to log in to old system.. Some links hard coded into our documentation need to be changed now by owners but those were things that we know would happen. We were happy that nothing really was critical. We continue to monitor for whole week until we notice that people finally fully transitioned and stop asking question about it. This is when we finally could announce that this project was successful!
But it was not the end for us.. Instance is running, people continue to use projects (just on new URL) but we have to do something with data that co-exist now on both instances.
Step 10: Cleanup
Yes, this is where very tired but happy start to think about cleanup. Split was successful, but currently we temporally run two instances with same data. We no longer need projects / spaces that are not used by migrated users and opposite.. Other users do not care about those migrated ones..
Someone could say that this step is not required, but actually that is a very important step. We cannot leave two huge instance. It is required to remove what is not used and optimize resource so that we do not generate additional cost. It was also a matter of data safety. Keeping data which do not belongs to you is a responsibility, so cleaning this ASAP is the last things that we have to do.
From a helicopter view it is easy.. We have a list of projects, simply delete them and both sides and that is all. Well, it depends on the instance.. If splitting small instance nobody probably cares much, but if we take care of bigger one then it is starting to be a challenge.
First we need to determine overall how things behave .. how long data is removed (so that we know how much potential downtime we need), is this affecting performance and what could potentially surprise us. Using "delete" functions is also more risky than "add" or even "edit" since one single incorrect step could cause damage.
Cleanup started by removing everything on a test system. Why? Because then we would have all information about size of the projects/spaces, before and after state and we would be able to validate if anything is missing. We recorded also how much time each object is deleted and based on this we were able to plan downtime.
In our case data removing time was more than 8h so we decided to split it into 2-3 parts and start with less important projects and spaces. Thanks to that we would know how things behave on PRD and verify the time. After removing first group If ended that all where 1.5x slower on PRD than on TST. We automatically adjusted all times and avoided potential problems
Deleting project/space was pretty smooth. We written a simple Python script just for that task. And good that we did that since removing all of those project/spaces manually would take more than we wanted to spent. At the end we also had a report from whole deletion which might be also handy in the future when someone ask us about it - we would know what happened.
Step 11: Research and Cleanup (again)
Deleting projects/spaces is only beginning. It is obvious that now we have to now check what is nor used after that. Many objects where connected to the project/space and now it is matter of identifying them one by one, verify if those are not used and delete. So this phase started again by doing good research so that we do not delete something that is still important.
This step is not easy.. There are no native features that would give us 100% accurate answer what could be deleted and what need to stay. Here we have to get a list of all schemes and objects and simply try to check if there are any "orphans". In case of Jira good thing is to start with schemes that are directly connected with a project. Then removing those would "release" all objects connected to those schemes.. And we can go deeper and deeper in the dependency tree..
For example..
Project A was using ABC Issue Type Scheme that contain Bug, Action and Improvement issue type.. Deleting Project A showed that ABC Issue Type Scheme is no longer used so we deleted it.. After that we were able to check Issue Types and we noticed that Action issue type was only used in Project A.. So at the end we also were able to delete Action..
So we followed this rule (lets call it from Top to Bottom) for other objects as well and after we weeks of work we have this cleaned up. The biggest problem was of course with Custom Fields..
Determine if a custom field is used or not requires more effort since we might think that something is not used now but it was actually used in the past and some values are there in old tickets.. So I recommend leaving Custom Fields to the end and double check everything few times..
Step 12: Final touch
After removing those objects we finally were able to reduce that backup to an optimal level. So far we had to keep two instances with that same huge database and disk space..
Speaking about disk space. Did you know that removing a project does not remove the folder from a disk? Yeah, so additional cleanup was required on the server side also.. We had to identify which projects were deleted and then (sadly) manually remove also attachment directories. Of course there was junk left also (some thumbnails.. Etc.) so removing this helped also optimize disk space usage.
On thing to remember when doing a report and marking something for deletion we have to pay attention the the fact that sometimes you think that folder does not match to any project key, but actually the name of the folder is always connected to the initial key - overtime someone could rename the project but it would not rename a folder on server. So in order to make sure it is good to check in the database old project keys where this information is stored until project is deleted. If you do not do it you might delete an used folder..
On of the steps that were left is to cleanup all plugin data.. Yes, again we had to start with research to determine which group of people used a specific plugin. Sometimes it was obvious since plugin was restricted to a specific group (not open for use for everyone) but for plugins that were adding something to the UI on a global level.. things get more complicated but after painful analysis we made a complete list of plugin that we would disable first and later (if nobody complains) delete.
BTW - Here I have a request to all plugin vendors - always add a feature to restrict ability to use a plugin to a specific group or individuals. Thanks to that it is not only possible to identify who is using it but also we (admins) are able to install something to someone not affecting everyone on the instance. And please add separate plugin audit logs! :)
Step 13: Monitoring and Optimization
Finally, after all of that preparation, migration, and post migration steps we reached to a point that we started to get some breath. Now it was visible how instances (source and target) behave. How much resources we need. We started with one huge instance that used more RAM, CPU.. As you remember we did a complete copy of a system, so now it was a time to check do we still need so many "power".
Good thing was that everything was stable, but overall after few weeks we optimized everything a little bit so that we can save some money on the infrastructure. Of course may things needed also to be reorganized, scripts duplicated and adopted to new environments, new IPs , etc. That was checked one by one and started to work like before.
Few weeks of monitoring both environments everything did the job. We did not get notice more issues, so were able to finally say that whole migration and transition is fully completed!
Summary
Just to quickly summarize I would like to say that this split overall was a success. It shows that everything is possible but of course not everything is simple as we initially think about this. When looking at the documentation it was only few step with not many details.. This series of articles were created to show more details that are also important and specific phases and required to be done. Of course every environment is different. One is small, one huge.. Some are not customized heavily other are integrated from all sides.. So every migration would be different but overall I hope that this information would be helpful to someone. At least to better plan and imagine how much effort might be required. I wish you successful splits / migrations.
Here I would like to thank you for taking time and reading though the series of articles and I am of course if anything is unclear happy to answer any question related to this topic. Please leave a comment and I would definitively reply :)
Mirek
Atlassian Certified Professional
324 accepted answers
2 comments