External app import intermittently skips data chunks

Question

Hello community,

I set up an external app configuration to import user and group objects (custom object types) from our ID management system (IdM). IdM sends one data chunk for user object type and one data chunk for group object type (currently max 250KB per chunk). Several imports finished successfully.

Later, import service in Insight skipped processing some chunks intermittently (for example, IdM sent 10 users in a chunk and 2 groups, Insight reported 0 user and 2 groups in the input in the results) but did not throw any errors during the upload or in the final summary. As the result, the object were deleted from Insight. If I retried the import with the same data input a moment later, all the users were imported just fine. I am not able to reproduce the issue consistently, it just happens sometimes.

As a workaround, I set the missing object threshold value to 2, so the objects are not deleted right away.

Has anyone encountered this kind of issues?

Are there any limits on the import data chunk size/insight connection?

Are there any detailed logs for an import run available?

Jacques · Answer

Hello,

Have you checked that there are no identical IDs in the import? Like in a database table (with a primary key), Insight/Assets also uses a unique identifier for each asset that it imports. Very often it takes the name of the object. If your imported data has 2 objects with the same name, it will skip one of the objects to be imported as it is considered "the same object".

To prevent this, you have to make sure that in the import definition you use a couple of attributes that uniquely identify each object to be imported.

This might be a reason.

Hope this helps,
--- Jacques.

Victor Parera · Answer

Hello again,

I have written an integration which import 5300 objects to Insight. However only 1920 are populated.

I believe there is a limit somewhere because I tried different ID's to ensure there wasn't a duplication issue but I have had the same result, only 1920 imported.

Victor Parera · Answer

Hello,

I am facing the same issue with the external app import. I am trying to import 5000 objects and when everything finishes with ok status and I go to insight, only 895 are imported.

If I reduce the amount of imported objects to 1000 I only get 461 objects in insight.

If I put 100 only 43 are populated.

It seems like some chunks are skipped but there are no errors.

Could someone give us some light on this topic?

Edit: I log the data i was importing and I saw some id's were repeated, changing the label to another ID solved the problem