Seamless Data Movement: Expanding Cloud-to-Cloud Capabilities

[#1]

Hi Atlassian Community!

I’m Adithya Ramesh, a Product Manager responsible for building products and features that allow you to control how your data is structured even after it’s been migrated to the cloud.

Why do organizations need to move their data in cloud?

As your organizations change, they often need to centralize or separate their data across different Atlassian cloud sites. While the site concept isn’t unique to Atlassian, our implementation offers organizations flexibility, robust security and compliance, and tailored configurations for different teams, departments, regions, or subsidiaries. However, end-users are generally indifferent to the specific location of their data. Their primary concern is accomplishing their tasks with the least friction. Therefore, it falls upon the admins to oversee the management of user and product data and the effectiveness of the organization's cloud blueprint, especially during periods of change.

So, what triggers the need for change? We’ve observed three key reasons why organizations change in cloud:

1. Seeking efficiency gains: Some customers use our cloud-to-cloud capabilities to improve team dynamics and output by reorganizing their data to foster better collaboration.

2. Organizational changes: Others, whether from mergers, acquisitions, divestitures, or internal reorganizations, want to align with their new organizational setup.

3. Optimizing for performance and cost: The least common but still prevalent customer wants to optimize site sizes for efficiency and optimal use of apps and licenses.

What tools do admins have to aid them in moving cloud data?

The Copy Product Data tool within the Atlassian admin hub allows admins to refine or change their cloud blueprint to better align with their organizational needs and strategic objectives. These native capabilities facilitate smoother transitions during business changes, such as optimizing operations or integrating new teams or organizations. The tool is designed to minimize downtime by reducing the complexity associated with moving data, allowing admins to focus more on their core business activities, the “why” of data movement, rather than technical, logistical challenges, which is the “how.”

Copy Product Data sports a streamlined UI, letting admins select which specific data to copy, such as projects, issues, and configurations in Jira or spaces and pages in Confluence. The tool enhances efficiency and reliability by minimizing potential errors proactively and reducing manual work during data migration, a noticeable change from what you may have experienced initially migrating to cloud. Furthermore, it gives administrators enhanced control over their cloud environments, simplifying data management and enabling organizations to adapt to changes swiftly. This ensures consistency and continuity across all Atlassian products being utilized.

Our tool's capabilities are constantly advancing to meet customer needs. Here's a quick recap of what's new: in the past year, Copy Product Data has expanded to support:

• Confluence space migrations between cloud sites, introduced in Q1 2024.

• Both Jira Software (JSW) company-managed and team-managed projects are now Generally Available (GA), with releases in Q3 2023 and Q4 2024, respectively.

• Jira Service Management form, team-managed, and company-managed projects between cloud sites were released in Q2 of 2024.

• A streamlined user experience that gives a consistent plan creation experience, regardless of the specific product for which you create a data copy plan.

These enhancements, and future ones, are backed by significant investments in our enhanced data movement platform, ensuring a reliable and scalable experience. Today, 97% of our customers complete copying their data within a week. We will continue investing in core and advanced capabilities, enabling you to handle complex cloud transitions and effectively support your growing user base.

But more importantly, back to you

Over the past year, we have been diligently enhancing Copy Product Data’s capabilities. Have you noticed the difference? We are eager to learn about your experience with these recent upgrades and how they have benefited your business. We also want your opinion on the name "Copy Product Data tool." Does it resonate with you, or do you have a more creative suggestion for me? Please share your thoughts with us. The best name and most innovative or interesting use of the Copy Product Data Tool will each receive a $25.00 gift card!

[Steven Rhodes]

We primarily move data from our cloud to another in order hand our customers back the data we have been collaborating on, on our instance. We find that we can get up-to speed very quickly once a contract is signed and work together to develop a solution. Once that is completed, or once the customer instance is ready to migrate over to their instance, then we can initiate a migration from our side.

While this all sounds smooth in theory, the practice is that the data copy tool makes a big mess (though it has significantly improved in the last year), and our clients are sometimes unwilling to make us Org admins on their instance so we need to have a workaround.

That workaround is to create a middle instance where both parties can be Org admin, and where we can cleanup the project(s) being copied, and then give them org admin rights and migrate the cleaned project to their instance.

The list of things that are either a mess or just should not be copied is long:

1. Copies every custom field in your instance to the other whether its used in the project or not
2. Adds a (migrated on) + timestamp to every custom field migrated. 
3. Copies seemingly random extra issue types over
4. Icons for all issue types are missing
5. Copies all groups used by the project but also those with zero members not used.
6. All permission, notification, priority schemes that are default will get the name + (migrated)
7. Administrator and atlassian-addons-project-access roles are moved over + (migrated)
8. Duplicate Issue links with (migrated)
9. And lots of (migrated on) added to the descriptions of Statuses, Custom Fields, Screens, Screen Schemes, Workflow Schemes, Issue Type Screen Schemes, Field Configurations and Field Configuration Schemes

So, as you can see, this is far from seamless. Manually deleting (in my case) 500 custom fields, dozens of issue types, replacing issue type icons, deleting hundreds of groups and then removing "migrated on" or "(migrated)" from all the schemes and fields is also very time consuming. Needing to do two migrations in order that you don't pollute the customer instance also adds to the stress. 

I can say that its a much better experience than last year so yes I have noticed the difference, as users that are not part of a project being migrated do not get copied anymore, which was a huge security nightmare for us, and a pain to remove 4000 users at a time using BulkOps (more and it would fail), and having the migration tool crash if there were comments/descriptions with more 32768 characters in was not fun to clean up at all. So, thank you to Atlassian for making these fixes.

Despite the list above seeming very long, I did manage to get it down to about 1 hour of cleaning the middle instance before performing the final migration to the client instance, after which they would clean up all the scheme/field descriptions manually, and replace the issue type icons on their side. All in all, its a working system but requires a lot of patience and is still very broken.

[Adithya Ramesh]

Hey @Steven Rhodes , apologies for the delayed response, the notification slipped my notice. Thank you for the candid feedback! This is extremely helpful for us to constantly evaluate what we are building.
* We are aware that the current issue of entities with the “migrated” tag being created is a major pain point in the tool - we are actively working on a solution to remediate this issue, and you can expect an update regarding that soon.
* The fix for missing issue type icons is being rolled out gradually and should be available to you in the next couple of weeks.
* The ability to move custom fields related the only the projects being moved is currently on our backlog, please vote on this https://jira.atlassian.com/browse/CLOUD-11779 ticket to help us prioritise this.

If you have additional feedback about the tool, I’d love to hear that, please feel free to schedule a meeting with me here: https://calendly.com/aramesh-ep0/30min

Thanks!