Merge previously imported components

[tobiassjosten]

After a previous discussion here, I had to reconnect the GitHub app to get the import functionality working again. It does work now but, after the reconnection, all previously imported components now seem to be disassociated with the GitHub app, resulting in the app wanting to import every single GitHub repository again.

So now it seems we can re-import all components again and manually reconfigure the new ones before deleting the old ones. That's quite a lot of work. Or we continuously check each component manually to see if it's already been imported. Again, over time, a lot of work.

Is there a third option, where we can mark a repository as already imported so that it won't be listed as pending?

[Enrique Serrano Valle]

@tobiassjosten thank you so much for reaching out to us and sharing your experience with the re-importing of components and repositories. We appreciate your feedback, as it helps us improve the product and serve you better.

The system is designed to prevent duplicate imports of repositories that were originally created via GitHub import. So manually selecting repositories that were imported originally from GitHub with either of these options isn't expected to create a duplicate:

Ideally, when you manually select repositories that were originally imported from GitHub, they should not appear as pending for import again. Instead, these should be clearly labeled as "Created" on the manual selection screen:

However, we understand that there might be specific scenarios, such as repositories initially imported from Jira projects during onboarding, where duplicates could potentially arise if imported once more from GitHub. If this situation resonates with your experience, please let us know. Rest assured, reconnecting GitHub to Compass should not impact this process.

To ensure we can offer you the best possible support, could you kindly provide more details about where exactly the system is prompting you to import these components again? Additionally, understanding how the process is generating both new and existing components after the app reinstallation will greatly assist us in pinpointing any issues with the import selection menu not marking them as already "Created."

Your insights are invaluable to us, and we are committed to making this process as seamless as possible. Please feel free to share any additional details or questions you might have. We are here to help!

[tobiassjosten]

Thanks, Enrique! I appreciate the response.

I have no idea what's happened but I'm now in a state where we have 441 components in Compass, from ~220 in GitHub. So they have all been duplicated and I'm forced to go through all of these manually, one by one.

One set of these components — the newer ones — seems to be "connected" with the GitHub app, as they are recognized by Compass as already imported. The other set does not have this connection but has all the data we manually entered during our inventory.

So I'm at a loss for how best to proceed and it feels a little overwhelming. Please do advise.

[Enrique Serrano Valle]

@tobiassjosten got it. I am not sure either about how we arrived to the original situation (I suspect the original mechanism to create the components might not have been a manual repository import), yet we're working on solutions that would prevent this situation coming up in the next quarters.

For your specific case at hand, and assuming that only the last batch of components was created via repository import, and that they're duplicates, there are a couple of mechanisms in Compass (one of them relatively recently introduced) that could help you keep only the original components.

When a manual import from GitHub is completed, the component is added a label with the text `source:github`. I'm assuming that only 50% of the components (the last import) has this label?

If that's the case, you can use the Compass search feature to find them all (filtering by label).

And it is now possible to select all the components in the page of matching results, and issue a bulk archive or delete. Let me know if this helps simplifying the duplicates that you detected.

[tobiassjosten]

I'm afraid all repositories have that label. :/ Both the original and the duplicate of each pair.

One thing that differs, however, is that the newly created components, the duplicates, all have the tag "synced-with-jsm". Does that help in any way?

[Enrique Serrano Valle]

@tobiassjosten the "synced-with-jsm" label indicates that there's a corresponding component created in JSM that matches its Compass counterpart. They both should be linked.

The creation of a new Compass component should be creating these new corresponding components in JSM.

The "source:github" label is applying when importing from GitHub.

I'm missing something about how we got to this situation.

• Is the repository associated to the "original" and the "duplicate" exactly the same?
• Why didn't the original components also sync with JSM? Was the label manually removed, and do they still have a "Jira Service Management" link like what I included above?

In any case, a quick patch might be filtering by the combination of labels "source:github" and "synced-with-jsm" if that's the label combination that is marking the duplicates. (But the removal of a Compass component doesn't automatically remove its JSM counterpart, so those may have to be pruned manually.)

[tobiassjosten]

Thanks! That's weird, though, as we don't use JSM. Maybe that happens because of some new functionality in the GitHub importer? Because I have a growing suspicion that something happened to that importer, that got our components out of sync and forced us to reinstall the importer — maybe a major version update?

As for your questions (and I'm grateful for all your help!):

• The repositories are exactly the same.
• We haven't touched the labels, manually or automatically. Only the newly created, duplicated components (with the synced-with-jsm tag) have that JSM link you screenshotted.

That's a good idea, filtering out the new, duplicate components like that. My worry, however, if I manually prune them is that the GitHub importer will again prompt me to import those repositories.

Ideally, I want new repositories in GitHub to be automatically imported (pending manual approval or not). It's not feasible to remember which of our repos have been imported or not when going through the list of potential new imports.

[Enrique Serrano Valle]

@tobiassjosten thank you for the additional information. There may be a couple of fronts going on.

JSM syncing

Theoretically, the `synced-with-jsm` tag is meant to appear when a Compass component is created and a synced copy is added to JSM, or in the opposite situation (a JSM service is created, and then a synced Compass component is created).

If you're seeing Compass components with the `synced-with-jsm` tag that also have a JSM link, have you tried clicking that link to see where it would take you? Theoretically, that is meant to load the synced service in JSM so wondering whether a JSM instance was spinned up for your domain too.

Sometimes the creation of services in JSM leads to duplicates in Compass if similar components already existed there.

However, that wouldn't have added the `source:github` label. Theoretically, it's only added when a component is imported from GitHub. Which takes me to the second front...

GitHub importing

When a component is imported from GitHub, the repository UUID is used to create an external alias for that component (external source: `github-importer`). If there's already a Compass component associated to that repository UUID, it will not be imported again.

Based on this, I think the following experiment would give us a meaningful answer. If you select two of those components that are duplicated, and run a GraphQL query (I think you might have yoursite.atlassian.net/gateway/api/graphql accessible for faster queries) with:

query MyQuery {
  compass {
    component(
      id: "ari:cloud:compass:...:component/.../..."
    ) {
      ... on CompassComponent {
        id
        name
        externalAliases {
          externalAliasId
          externalSource
          url
        }
      }
    }
  }
}

Then you'll be able to see what external aliases are associated with every one of the duplicated components. Depending on what we find there, it may open some new solution paths for us (and also give some peace of mind on duplicate prevention moving forward).

[tobiassjosten]

That sounds great, to get that kind of insight! I didn't even know there was a GQL playground. :)

Looking around, however, I couldn't find any information on how to format that ID. I've tried with this, which didn't work:

ari:cloud:compass:<ourteam>.atlassian.net:component/<component-uuid-1>/<component-uuid-2>

Sorry but could you please advise?

[Enrique Serrano Valle]

@tobiassjosten yes; crafting a full resource identifier from scratch isn't easy. Fortunately, we have some alternatives.

If you open the component overview page of each one of those components in Compass, you'll notice this meatballs-style menu in the top-right corner of the center area. If you click it, you'll see an option to copy this ID.

Under the hood, the full ID is a combination of:

ari:cloud:compass:<site-id-here>:component/<workspace-id-here>/<component-id-here>

In any case, if you copy that ID from each component, and paste it in the query from my previous message, you'll get the alias associated to each component.

[tobiassjosten]

Thank you so much! That was indeed informative.

Both repositories have "atl_jsm" as their externalSource, while the new ones have an additional externalAlias with "github-importer" as their externalSource (using the correct GitHub ID for externalAliasId). So I guess the new ones are correctly imported and will be properly ignored for future imports. That's good!

Following either link in the url property of the "atl_jsm" externalAlias leads nowhere but my dashboard in Jira.

Does that tell you anything? I imported both using the GitHub importer (although the first one was almost a year ago).

I'm inclined to chalk this up to growing pains and just manually move over whatever we've configured in the old components, to the new components, and then delete the old ones. If you have any recourse to fix it in a more automated fashion, please do advise, else I'll consider it par for the course for us early adopters. ;)

[Enrique Serrano Valle]

@tobiassjosten I think your theory is correct, and that the mechanism to create external alias might not have been available during the first versions. I can confirm that right now, this would prevent duplicates in GitHub imports.

There's a situation where deactivated and reactivated workspaces could also trigger new syncs adding duplication (because workspace Ids change on reactivation and the system sees them as different components / services). But given that you don't currently have JSM active, this might not have been your case. We do have an item in our roadmap to manage these situations too.

I would agree with your plan - the latest components are protected against importing duplication. There are GraphQL operations that could be used, but I think the easiest approach is probably searching, filtering and bulk deletion using the GUI as I outlined in this comment.

Please don't hesitate to reach out if there's something more that we can help with!

[tobiassjosten]

Then we understand the situation and have a way forward. 👍 Thank you so much for your help figuring this out!