Retaining Bugzilla ID when importing

[Fabian Meier]

I am importing 60,000 issues from Bugzilla and for the previous test runs the issue key were identical with the external issue id. However, for the latest test run, they are no longer in sync. I know that this is actually not supported, but I would still like to understand what the order of the import query depends on. Since it worked before I am pretty sure that if I understand the underlying design of the importer better, I can get them in sync again.

It would look a lot better than:

|*Key*|*Summary*|

|BUG-57453|[BUG 59301] Out of memory in web resource module|

[Fabian Meier · Accepted solution]

I ended up doing a CSV import, because the order the Bugzilla importer uses is not deterministic.

[Pawel Niewiadomski]

Sure, totally understand you.

[Pawel Niewiadomski]

Fabian, it's not that strange. You haven't mentioned that you use GreenHopper. So to clarify and shed some light - GreenHopper will automatically upgrade JIRA Importers Plugin to at least 4.4 because GH uses it internally.

That's why you weren't able to downgrade it. You'd have to disable GH before downgrading JIRA Importers Plugin.

[Fabian Meier]

Well, even after disabling Greenhopper, the version of the upgraded / downgraded JIM plugin would not match. For example: I upgraded to JIM 5.0.2, but the screen popping up said I am now running 4.3.2. There is something wrong with the bundled and installed plugins.

But that was actually just one of many problems. The Bugzilla importer behaved everything but deterministic. I can provide you some of the strange logs if you want. I ended up preparing a CSV loadfile just hours before the go live and got most of the information ready "just in time". The CSV load work without any flaws.

I am working with data migration since 10 years now and I have created many highly customized JIRA based sytems that go beyond the current limitations of the software. We even develop cutting edge plugins by ourselves, so I would say that I know what I am doing, but the Bugzilla importer, I don't get...

I would love to give you more details and point out the issues I had with the help of some log files and notes I took.

[Fabian Meier]

Sorry, but I am overwhelmed with work after that weekend and the likeliness that we will do another import from Bugzilla is practically 0%. So I hope you understand that my motivation to follow up on this is pretty low. I had already filed a support request regarding the slow import performance and I have since referenced this topic.

If they ever decide to fix the Bugzilla Importer, they can find information in this topic as well as contact me for more information.

Thanks for your help even though I had to use the CSV import in the end =)

Fabian

[Pawel Niewiadomski]

Sure, totally understand you.

[Pawel Niewiadomski]

Yeah, I don't get it too. Strange things you witnessed.

Please raise an issue on http://studio.atlassian.com/browse/JIM and add details there. Whenever we have time we'll see what we can improve. But don't expect it to be fixed soon. JIM is now in maintenance mode.

[Pawel Niewiadomski]

Depends on JIRA Importers Plugin version you use. In older versions issues were not sorted, so the order was database depenedant. But from JIRA Importers Plugin version 4.1.3 issues are sorted by bug_id (in ascending order, so oldest are first one to be imported).

You can obtain JIRA Importers Plugin source code with JIRA Source code which is accessible through my.atlassian.com

[Fabian Meier]

That would make sense, but I don't think this is the case:

BUG-1 ended up having External Issue ID 6536

BUG-2 ended up having External Issue ID 6915

BUG-3 ended up having External Issue ID 7100

...

Why would that happen?

[Pawel Niewiadomski]

The simplest explanation would be that you use the version older than 4.1.3. If you use a newer version I don't have an answer for you. No clue.

[Fabian Meier]

This was the latest version (JIRA Importers Plugin Version 5.0.2) but now that you mention it, I think the first import I did was in fact with version 4.4. They probably broke it in version 5.x. This is good news for me - downgrading to 4.4...

Will post the results =)

[Pawel Niewiadomski]

They is actually me, I'm the lead developer of JIRA Importers Plugin.

You can try, I'm curious. There weren't many changes between 4.4 and 5.0.2, but we did upgrade Guava, which hypothetically could influence this. But I won't believe until I see it ;-)

[Fabian Meier]

Great to see that you are answering questions here on answers.atlassian.com!

I have just tried to use JIM 4.4 and the sort order is yet another:

BUG-1 ended up having External Issue ID 32444

BUG-1 ended up having External Issue ID 32478

BUG-1 ended up having External Issue ID 32479

Do you have any idea why it is not sorted by bug_id? I will try earlier versions than 4.4 as I just remembered that the plugin might have been an even earlier version when I did the first test run. I will try some versions between 4.1.3 and 4.4.

Tomorrow I will have to run the import into the production system and I would really like to get the IDs in sync because it is somehow confusing to have a BUG-45324 that actually used to be bug #54213 (values are from within the same domain of numbers).

Were there any critical fixes in the last couple of versions?

[Fabian Meier]

I have found the log of the first test import and it was actually JIM 4.3.2. I have installed that particular version and run another Bugzilla import but it still starts with bug_id 6536,6915,7100,...

I am not sure what else has changed in the system. What could cause this kind of behavior? I will try a couple of more imports with the same version and see if it always sorts the same way.

[Fabian Meier]

Found it, even though I installed version 4.3.2 (and it showed as 4.3.2 under managing plugins), version 4.4 was used because it was in the installed and or bundled folder in JIRA\plugins. I have deleted them manually and reinstalled version 4.3.2. It now identifies itself as 4.3.2 when running the import and the bugs are in correct order! *yay*

So "they" broke it after version 4.3.2. Do you need a bug report for that?

[Fabian Meier]

Turns out I was wrong again. The weirdest thing is that even though I got it running after switching the version a couple of times, it works for the import into Project 1 but if I cancel that one and try to import into a new project, it fails again...

The only valuable observation I made is that before a "sorted" import, it takes a long time (about 20 minutes) to fetch the list of issues from the database. For all the "unsorted" imports, it starts importing almost immediately after I see the Importing Issues headline in the log.

[Fabian Meier]

I have finally got a "sorted" import going again and it only took me about 100 tries. I have a list of permutations of clearing plugin caches, upgrading JIM, downgrading JIM, disabling Greenhopper, enabling logging and so on in front of me. This is a really awkward problem - definitely the strangest things I have very experienced because completely unlogical and undeterministic.

My final observations:

- even though I downgrade, it shows with another version number after the installation. however the version number that shows when entering the import parameters matches the one I downgraded to

- even when I upgraded the plugin to 5.0.2, it would sometimes still show e.g. 4.3.3

- I had to disable Greenhopper, because it would always upgrade the plugin to 4.4

- The bundled version seems to somehow sometimes overwrite the manually installed version

- The plugins folder contains several version numbers of the plugin (bundled,installed,osgi)

- The plugin shows under custom plugins as well as Atlassian plugins

- Sometimes the import of the versions is really fast, sometimes very slow

- once I had a "sorted" import and tried it again, it would be "unsorted" again

- every time it works, I get the following additional log lines:

Version does not exist for project: 'ExternalProject[jiraId=11200,id=1,externalName=product1,name=Bugzilla,key=BUG,url=<null>,lead=Fabian,description=Product1,projectCategoryName=<null>,avatarUrl=<null>,assigneeType=<null>,versions=<null>,components=<null>,issues=<null>,workflowSchemeName=Default workflow scheme]' : and version: '---'.

You see, I spend quite some time with this today (about 10 hours) and it feels like it all came down to luck and amount of tries. I hope my observations help to figure this out. I actually believe that there are multiple bugs here (JIM, Plugin Management, Caching, etc.)

Hope to hear back from you =)

[Pawel Niewiadomski]

Each project has it's own issue id counter, if you import multiple times to the same project the last value is reused and increased, so you're not able to go back to 1 unless you delete the project.

Also you need to keep in mind that some issues could had been moved from project to project in Bugzilla (or deleted if you enabled issue deletetion in Bugzilla) so you'll still can end up with holes in the numbering.

If you really want to have issue key in sync, you can edit the source code for JIRA Importers Plugin and force the issue key manually by setting the key field on ExternalIssue in IssueTransformer).

But mind you that's something not supported and you're on your own.

[Fabian Meier]

I am importing into an empty project and I do not want to force set the issuekey to external issue id but I want to control the sort order of the bugzilla issues. For one test run it was in sync (meaning bug #1 resulted in ABC-1 and so on) but then something changed and for a second run they are all messy again (bug #1 resulted in ABC-36725).

So the acutal question is: What is the criteria that defines the sort order in which the bugzilla issues are imported?

Besides that I am still interested in how to modify the source code of the importer plugin. Where can I obtain it and do I have to recompile it?

[Bob Swift OSS (Bob Swift Atlassian Apps)]

When you say empty project it is not clear that you deleted the project and re-created it. It must be deleted and recreated to start a 1. If you need more control of the import process, then you can use JIRA Command Line Interface and a script to preserve numbers (holes in the original numbers can be handled by create and the delete of JIRA issue) but it will be more work and slower than the import.

[Fabian Meier]

Empty project means I made sure that it start counting from BUG-1 by creating a new project.

I saw that JCLI can import the issues from Bugzilla, but the import is already taking almost a complete weekend and I cannot afford that it takes any longer...