A Solitary Admin with Cloudy Thinking - A Journey of Hope, Despair, and Triumph - Episode 3

[#1]

Preface (redux)

The idea for this series of articles came to me while deep in the throes of making the journey from my warm and friendly Server environment to the wispy world of the Clouds. I am the solitary admin referenced in the title. 

I had originally thought this would be a single, longish article. However, as the journey progresses, material for such an article continues to pile up, and it is indeed a pile. The naivete at an effort like this needing but one article to do more than brush the surface now causes a rueful chuckle. At least, in the prefaces for the first two episodes, I hedged my bets that it felt like 5 episodes. However, here I am at episode 3 and I now know that 2 more won't cover it given the current pace of articles. I had originally intended to, should I find a "magic bullet" to solve <insert thing here>, perhaps share that bit of technical trivia. I have run across some of that but there are lots of articles about <insert thing here> found whilst grepping Google for the latest thing that broke. As such, I will continue with the life of the journey with enough tech in it to keep the theme alive.  My hope that this might help others in my shoes see that they are not the only ones living it seems to have been borne out. The commiserations from those I consider to be Supreme Beings of Atlassian Skills have helped immensely. As such, from them to me, and me to whomever reads this, we're not alone.

Through this, I have developed the ability to evoke longer and longer streams of non-repeating invective. The emotional roller coaster has become a mashup of the well known, "it was better than cats" quote and the immortal quote from John Cleese, as Brian Simpson in the movie "Clockwise", "It's not the despair, Laura. I can take the despair. It's the hope I can't stand."

So, without further adieu....

The Journey Persists

In the previous episode, the descent into Despair was chronicled. At the end of that episode, I spoke of some glimmers of light. Since then, they have insisted on being only glimmers, albeit enough to keep doggedly forging on but with a ray of light combined with some trepidation.

Opportunity Lost?

I could have said cloud is a Bad Idea, it would've been accepted, and I would've been done building out the resilient AWS environment for Data Center months ago. I could be sleeping peacefully through the night by now.

But no. I had to say that Cloud is The Way.

I still (sometimes surprisingly) think so, but, when looking in the Migrations dashboard at the many, many "FAILED", and "INCOMPLETE" lozenges, I occasionally question my eroding sanity. I suspect that others in my situation will be fixated on the Jira migration as I am. It is the elephant in the room. While it is easy to get fixated on said elephant, there is also an elephant in the room but under a blanket; Confluence. It's maybe not as big of an elephant, but an elephant nonetheless, and the blanket doesn't disguise it. Tentative space migrations were less than... satisfying.

Data Center would have been so much easier.

Forging on...

Since I have been a bit under water, as it were, since the last episode, the amount of "forging" has gotten to where it's almost several episodes on its own. However, I shan't recount all the festivities; just the biggest faceplants on the journey since episode 2.

There is a meme floating around with the caption, "I don't always test my code, but when I do, I do it in production". In my Server environment, I have a mechanism to automagically clone off the latest production snapshot to a test environment. This has long been useful for real world potentially destructive testing. This turned out to be a Bad Idea for migration testing.

My intentions were good. Keep the migration database on production nice and clean until I had it working and then do it in production for real. If things went badly, I could refresh the test instance to start over and also get newer data snapshots.Things went badly and kept going badly. Jira Cloud Migration Assistant, in an abundance of caution, will take a given entity and move it over with a "(migrated)" label on its name rather than risk overwriting stuff one would rather not overwrite. Salutary but perhaps a flag to tell it to merge or add might be good.

The migration process was once typified to me as:

1. Migrate
2. Fix what broke
3. Migrate 20X more fixing what broke each time
4. Production migrate
5. Fix what broke after it's in production

Step 3 is where things seem to have gone very badly by doing the ostensible "right" thing by using my test instance. Aside from never getting a consistent failure to solve against, the worst of it was the custom fields.

Starting with a shiny, new Jira Cloud Premium instance, I did my first Jira Software migration. From test because one tests in test since it's a test. These early tests went reasonably well, albeit with the failed and incomplete results in JCMA. I fixed the most egregious problems and tried again. Better. Since it was late in the day, non-invasive fixes were replicated in production and planned to refresh my test instance the next day with the night's snapshot, which would bring across the fixes to test.

After my initial migration test, I ended up with 863 custom fields (yes, I know....) which included 6 having the now dreaded "(migrated)" suffix after the field name. 6 fields aren't bad to fix and include a now system field "Start Date" that I had as a custom field in the Old World. Using my shiny clean test instance with the shiny clean migration DB, I retried the initial project after the fixes were in. 

This went bad.... ish. The project now migrated just fine. However, I now found I had 1,276 custom fields. Every custom field duplicated with the original "(migrated)" fields now joined by "(migrated 2)". ScriptRunner for Jira has a lovely built-in script that would help me move values from the spurious fields to the real ones but it felt like days of work to sort it out. Fortunately, my MOVE Yoda had provided me a blank Jira import file to restart things. It suspect it should've been foreboding that it was so handy. That said, my MOVE Yoda suggested my tenant was pretty jacked with something called "migration drift" and best to totally destroy the tenant and start over from... nothing.

This is not a fast process, between waiting for Cloud to accomplish its task, and then getting all the base items back to start again with it taking between 18 and 36 hours to do All the Things. Why do I have a range of time, you ask (assuming you asked)? Because things went bad... ish... more than once and needed to be done.

In discussions with MOVE Yoda, something he said caused the immortal penny to drop and it became obvious that using the test system to test things was a bad idea. While I didn't dig into it since I don't actually care all that much other than knowing it exists, each reconstitution looked like a brand new environment to Cloud and, with an abundance of caution, coupled with no "throw caution to the wind and overwrite" button on JCMA, all the fields (and other things) were duplicated. I guess I am testing in production.

After pushing through the icky feeling of testing in production, it turned out to be not that bad in this case. The 20X migrations don't touch anything but the migration database and I get the advantage of migrated data is freshest; it doesn't have the delay induced by nightly snapshot to test and then test to Cloud. However, since production with no way to clean out past failed attempts, a rigorous Migration naming convention is required. What is working for me looks like:

• <project/project group>: <cycle state> <sequence number> where
  • <project/project group> is the key of the project or a key-like project group name
  • <cycle state> is how you might want to typify what you are doing like TEST, POC, UAT, PROD, etc.
  • <sequence number> is purely because you can't delete migration sets and it just isn't likely to work the first time.

Now that I am through the "this is just wrong" feeling of this kind of testing on production and using the above methodology to keep track of it all, I am at least failing consistently. This Sounds Bad but it was a Good Thing as the consistent failures let me find consistent solutions. 

The proverbial first step on the thousand mile journey...

For those who have been following along, you might wonder whether I have not been on the thousand mile journey. I have. This is the new thousand mile journey because....

All the forging, as it were, has gotten me to where I have a project in production on Jira Cloud. It's just one out of so, so many, but it's live. Further migration tests are solid enough that I am using those migrated projects to mitigate for cloud and thence to UAT. I have a way forward at least for Jira Software (JSW) projects. Jira Service Management (JSM) from Jira Service Desk (JSD) isn't there yet as the migration tool is a bit myopic in the JSD to JSM migration. I have 7 service desks to move so it's kind of a big deal. My MOVE Yoda is keeping me abreast of development there and gets me beta releases of the migration assistant to try. One of these days, it will viably move JSD to JSM.

In Our Next Episode

We are deep in the throes of migration with all the pitfalls and pratfalls that can ensue from an effort of this size and complexity.

[Taranjeet Singh]

Great to learn from your experience , @Mike Rathwell ! Thanks for sharing!

[Luzia Mendes [SmartBear]]

That's a great article to read. Thank you for sharing.
I'm taking you're trying the phased Strategy, right? Combined with what method? The JCMA?

If that's right, you're also saying that JCMA will not overwrite data whatsoever never-ever? I mean...

1) Will it import new issues to current project?

2) Will it always duplicate the custom fields? What about issue types?

[Mike Rathwell]

I am indeed taking the phased strategy, @Luzia Mendes [SmartBear] . I literally have no choice but to do that. The one galling thing that is in that as a "when to do this" is the "over 5000" users. Nope. I keep trying to get across to them that number of users is not directly related to complexity. I might have only ~600 users but every group in the company lives on Jira, not just the technicals. With that, there are the needs of each group to be built for. 

I have to touch, or at least review, every set of conditions, validations, and post functions in every workflow I move over. There is no way, with only one of me, that I could do all that in a mass migration and not have a massive outage.

JCMA is... a Thing. It can do a migration to an empty hole. It cannot update an already migrated project. One ends up deleting and re-importing the same project over and over until you clean out the errors that keep it from migrating. Even better, when it hits ENOUGH errors, it just bails. Fix those and then it will bail on the NEXT set of errors. So much fun.

It doesn't duplicate things now that I am coming ONLY from production and not from a regularly refreshed test instance. It is stable enough now that I can take the time to clean stuff up. It does duplicate project roles and priorities (at least priorities are easy to fix). Issue types... it didn't duplicate those but it also didn't move over the icons. I ended up tarring off the icon directory in the Server HOME so I could find them all and reapply them all. Manually. ~350 of them. 2 days effort for that festival. 

[Luzia Mendes [SmartBear]]

wow