Migrate your own data with Jira Cloud Migration Assistant

[#1]

Hello!

In this article I will show you how you can migrate your own data with Jira Cloud Migration Assistant (JCMA).

JCMA helps you to migrate Jira Software and Jira Core data from Server or Data Center to Cloud.

When you develop your app for Jira Server you can store your data in multiple places: issue properties, project properties, Jira database using Active Objects, file system. JCMA will migrate data from issue properties but it will not be able to migrate data from AO tables or file system. In this case your app will not be able to function properly in Cloud after migration.

That is why in order to migrate all the data for you Jira Server app to Cloud correctly you can extend JCMA to migrate your data. You can find more information about it here and you can find example source code for extensions here.

In this article I will explain how it all works in detail with a ready for use code. You can see this code here.

You can also watch a video over here.

Theory

JCMA extension app consists of two apps: one for Jira Server/Data Center and one for Jira Cloud.

Migration starts in Jira Server/Data Center. After JCMA has finished with its own data migration, JCMA executes all classes which implement CloudConnectionListener interface. And that is what we have to do. In our Jira Server app we need to create a class which implements this interface.

Then JCMA sends events to migration webhooks which are registered for the Jira Cloud instance where we migrate data. That is why in our Jira Cloud app we need to register our webhooks which will be triggered by the events.

Ok. That is all for theory. Let's have a look at the code.

Jira Server/Data Center app

We have two classes.

src/main/java/ru/matveev/alexey/atlassian/server/accessor/LocalCloudMigrationAccessor.java

@Named("cloudMigrationAccessor")
public class LocalCloudMigrationAccessor implements ApplicationContextAware {

    private CloudMigrationAccessor registrar;

    @Override
    public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
        try {
            this.registrar = (CloudMigrationAccessor) applicationContext.getAutowireCapableBeanFactory().
                    createBean(getClass().getClassLoader().loadClass("com.atlassian.migration.app.tracker.CloudMigrationAccessor"), AutowireCapableBeanFactory.AUTOWIRE_CONSTRUCTOR, false);
        } catch (Exception e) {
            throw new RuntimeException("Failed to initialise CloudMigrationAccessor", e);
        }
    }

    public CloudMigrationAccessor getCloudMigrationAccessor() {
        return registrar;
    }
}

This class just gets a reference to the CloudMigrationAccessor service. We need this service to execute our migration.

src/main/java/ru/matveev/alexey/atlassian/server/impl/MyPluginComponentImpl.java

@Slf4j
@Named ("myPluginComponent")
public class MyPluginComponentImpl implements CloudMigrationListener, InitializingBean, DisposableBean {

    private final CloudMigrationAccessor accessor;

    @Inject
    public MyPluginComponentImpl(LocalCloudMigrationAccessor accessor) {
        // It is not safe to save a direct reference to the gateway as that can change over time
        this.accessor = accessor.getCloudMigrationAccessor();
    }

    @Override
    public void onRegistrationAccepted() {
        log.info("Nice! The migration listener is ready to take migrations events");
    }

    /**
     * Just a collection of example operations that you can run as part of a migration. None of them are actually required
     */
    @Override
    public void onStartAppMigration(String transferId, MigrationDetails migrationDetails) {
        ObjectMapper objectMapper = new ObjectMapper();
        try {
            log.info("Migration context summary: " + objectMapper.writeValueAsString(migrationDetails));
            PaginatedMapping paginatedMapping = accessor.getCloudMigrationGateway().getPaginatedMapping(transferId, "identity:user", 5);
            while (paginatedMapping.next()) {

                Map<String, String> mappings = paginatedMapping.getMapping();
                log.info("mappings = {}", objectMapper.writeValueAsString(mappings));
            }
        } catch (IOException e) {
            log.error("Error while running app migration", e);
        }

        // You can also upload one or more files to the cloud. You'll be able to retrieve them through Atlassian Connect
        try {
            OutputStream firstDataStream = accessor.getCloudMigrationGateway().createAppData(transferId);
            // You can even upload big files in here
            firstDataStream.write("Your binary data goes here".getBytes());
            firstDataStream.close();

            // You can also apply labels to distinguish files or to add meta data to support your import process
            OutputStream secondDataStream = accessor.getCloudMigrationGateway().createAppData(transferId, "some-optional-label");
            secondDataStream.write("more bytes".getBytes());
            secondDataStream.close();
        } catch (IOException e) {
            log.error("Error uploading files to the cloud", e);
        }
    }

    @Override
    public void onRegistrarRemoved() {
        log.info("The listener is no longer active");
    }

    @Override
    public String getCloudAppKey() {
        return "ru.matveev.alexey.atlassian.server.jcma-cloud";
    }

    @Override
    public String getServerAppKey() {
        return "ru.matveev.alexey.atlassian.server.jcma-server";
    }

    @Override
    public Set<AccessScope> getDataAccessScopes() {
        return Stream.of(
                AccessScope.APP_DATA_OTHER,
                AccessScope.PRODUCT_DATA_OTHER,
                AccessScope.MIGRATION_TRACING_IDENTITY)
                .collect(Collectors.toCollection(HashSet::new));
    }

    @Override
    public void afterPropertiesSet() {
        this.accessor.registerListener(this);
    }

    @Override
    public void destroy() {
        this.accessor.deregisterListener(this);
    }
}

Here we implement our class from CloudMigrationListener, register and unregister our listener in the afterPropertiesSet and destroy methods.

The onRegistrationAccepted method is called by JCMA after our listener was successfully registered by JCMA.

The getCloudAppKey and getServerAppKey methods return the keys of our apps in Server and Cloud.

And the onStartAppMigration method is the method which is called when JCMA wants your app to migrate data.

 @Override
    public void onStartAppMigration(String transferId, MigrationDetails migrationDetails) {
        ObjectMapper objectMapper = new ObjectMapper();
        try {
            log.info("Migration context summary: " + objectMapper.writeValueAsString(migrationDetails));
            PaginatedMapping paginatedMapping = accessor.getCloudMigrationGateway().getPaginatedMapping(transferId, "identity:user", 5);
            while (paginatedMapping.next()) {

                Map<String, String> mappings = paginatedMapping.getMapping();
                log.info("mappings = {}", objectMapper.writeValueAsString(mappings));
            }
        } catch (IOException e) {
            log.error("Error while running app migration", e);
        }

        // You can also upload one or more files to the cloud. You'll be able to retrieve them through Atlassian Connect
        try {
            OutputStream firstDataStream = accessor.getCloudMigrationGateway().createAppData(transferId);
            // You can even upload big files in here
            firstDataStream.write("Your binary data goes here".getBytes());
            firstDataStream.close();

            // You can also apply labels to distinguish files or to add meta data to support your import process
            OutputStream secondDataStream = accessor.getCloudMigrationGateway().createAppData(transferId, "some-optional-label");
            secondDataStream.write("more bytes".getBytes());
            secondDataStream.close();
        } catch (IOException e) {
            log.error("Error uploading files to the cloud", e);
        }
    }

As you can see there are two parameters passed to the function: transferId and migrationDetails. The transferId parameter helps your Server and Cloud app work with the same migrated data. You can run multiple migrations by JCMA and transferId is the id for those migrations. The migrationDetails contains migration context. I would like to pay your attention that you can get mappings in this method. Mappings provide you with mapping for the server object id with the cloud object id.

For example, let's say you migrated users. In Jira Server the user is called admin but in Cloud it will have a different id. And by mappings you can retrieve this new id. Also you can retrieve ids for such objects as issues, workflows, issue types and so on.

So in our implementation of the onStartAppMigration method we create two files, which will be put into Cloud storage by JCMA for us and which we will later read in our Cloud app. In real application those files should contain data which should be migrate to Cloud.

Of course, we could start and finish migration in the onStartAppMigration method itself. For example, let's say, we have REST API in our Cloud app which will accept our data and put everything as it should be in Cloud. We could call this REST API from this method and pass all data we are migrating. But it is not always the case, that is why I made this example more complicated and created a Jira Cloud app which will be triggered upon migration.

Jira Cloud app

First we need to tell Atlassian Migration API which of our endpoints to call upon migration. In other words - to register our webhooks. I will register our webhooks in the installed lifecycle event and I will unregister those webhooks in the uninstalled lifecycle event.

I use atlassian-connect-spring-boot that is why I need to implement two listeners for those lifecycles events.

But to register my webhooks I need to know the base url of my app and I will get it from the application.yml file with src/main/java/ru/matveev/alexey/atlassian/cloud/util/YAMLConfig.java:

@Configuration
@EnableConfigurationProperties
@ConfigurationProperties(prefix="addon")
@Data
public class YAMLConfig {

    private String key;
    private String baseUrl;


}

The code is simple.

Then I can create my listeners.

src/main/java/ru/matveev/alexey/atlassian/cloud/listener/AddonInstalledEventListener.java

@Component
@Slf4j
public class AddonInstalledEventListener implements ApplicationListener<com.atlassian.connect.spring.AddonInstalledEvent> {
    @Autowired
    private YAMLConfig yamlConfig;
    @Autowired
    private AtlassianHostRestClients atlassianHostRestClients;

    @Override
    public void onApplicationEvent(AddonInstalledEvent addonInstalledEvent) {
        log.info("install event");
        JSONArray endpoints = new JSONArray();
        endpoints.put(String.format("%s/migration-started" , yamlConfig.getBaseUrl()));
        JSONObject webhook = new JSONObject();
        webhook.put("endpoints", endpoints);
        HttpHeaders headers = new HttpHeaders();
        headers.setContentType(MediaType.APPLICATION_JSON);
        HttpEntity<String> request = new HttpEntity<>(webhook.toString(), headers);
        atlassianHostRestClients
                .authenticatedAsAddon()
                .put(String.format("%s/rest/atlassian-connect/1/migration/webhook", addonInstalledEvent.getHost().getBaseUrl()), request);

    }
}

As you can see I inject YamlConfig and AtlassianHostRestClients. YamlConfig will give me the base url of my application and AtlassianHostRestClient will let me call Jira Cloud Rest Api with proper authentication.

The unistalled event has about the same code. I will not paste it here.

So we registered our webhook. Now let's write the endpoint for our webhook src/main/java/ru/matveev/alexey/atlassian/cloud/listener/MigrationStartedListener.java:

@Controller
@Slf4j
public class MigrationStartedListener {
    @Autowired
    private AtlassianHostRestClients atlassianHostRestClients;

    @PostMapping("/migration-started")
    @ResponseBody
    public String migrationStarted(@RequestBody JiraMigration body) {
        log.info("migration started");
        if ( "APP_DATA_UPLOADED".equals(body.getWebhookEventType())) {
            ResponseEntity<String> response = atlassianHostRestClients.authenticatedAsAddon()
                    .getForEntity(String.format("%s/rest/atlassian-connect/1/migration/data/%s/all", body.getMigrationDetails().getCloudUrl(), body.getTransferId()), String.class);
            log.info(response.getBody());
        }
        return "ok";
    }
}

We defined the migration-started endpoint. This endpoint will be called by JCMA. We accept the JiraMigration class as the body for the request. This class contains useful methods which will let us perform our migration. In my case I get the tranferId and after it I get the files which I created in my Jira Server app.

That is a pretty simple example. But you can figure out how JCMA extensions work.

[Alina Cucean]

Hello!

Nice article! You said that JCMA will migrate the entity properties (issue properties?).

I tested the migration app and could not find a way to migrate the entity properties from server to cloud.

Do you have an article about that?

Many thanks! :)