Insight - Multiple import sources for same Object type and lack of data manipulation capabilities

[Igor Muravsky]

Hi,

I'm trying to build comprehensive CMDB with Insight, that'd consolidate information from different sources. I've encountered two main problem along the way and would like to hear community opinion on who you approach such cases.

Case 1.   There are multiple data sources, that contains some information about asset. For example, your AV system, Intune and SCCM, Monitoring and discovery tools. Each of this tools can enrich data about asset or update specific attribute. As each tool has its own information and attributes inside, it's close to impossible to use external datasources directly to update and enrich asset information. You can't specify only one attribute to update and leave other mandatory attributes intact. Import force you to specify all mandatory fields during import, assuming, that we'd add objects to DB, but there is no "attribute update from external source" mode. The only way I was able to partially compensate for this - is to build my own data collection and aggregation service, that presents data in required way. But it require constant maintenance and brake whole idea of having all your imports in one place.

Case 2. Somewhat related to Case 1. It's impossible to manipulate incoming data from external sources on Import. It's possible to apply some simple regex. But it's impossible to define static value for non-reference field or use any transformation functions. My workaround is the same as in Case 1.

Do you have any other ways to overcome such problems? Maybe some thoughts on CMDB design that could ease such use cases?

[Paul Twomey]

Hi @Igor Muravsky may I ask if you ever resolved these difficulties and if so how please?

[Igor Muravsky]

Hi, @Paul Twomey 

To be honest, original issues from 2.5 years ago, when I started to use Insight are still in place. Overall pace of development and bug fixes close to 0. 

Said that, I was able to build quite a successful CMDB with hundreds of object types, hundreds of thousands of items and over 50 imports. But to achieve this - I had to use some external tools and ugly hacks.

1. External tools - I'm using an external automation engine that is doing all required data manipulation on source data and presenting it in the correct format for import. So effectively between most data sources and Insight I have middleware. This middleware also handles any additional formating, authentication (Insight still supports only basic auth), and some much more complicated cross-referencing and data enrichment.

2. Hacks. If I need to make sure that multiple data sources will not create and remove each other's objects, but only enrich existing objects (like for example Computer objects, that will be updated by statuses in different systems, by last seen date, encryption status and so on) - I'm trying to add faulty values to existing mandatory select lists (like status) and prohibit to add new values to the list during import. Due to the specifics of how imports work - I'm able to update values in existing objects, but no new objects will be created, as mandatory field can not be populated.

Combersome, but it works. 

[Paul Twomey]

Thanks for the info @Igor Muravsky .

May I ask what the external automation solution is?

[Igor Muravsky]

Used a few over the years. Azure Automation, Azure function, Powershell Universal.