Trouble importing users from OpenLDAP using Insight Discovery

[Dee Heffemm]

I'm trying to setup an LDAP import from Insight Discovery but keep getting the error below "Unable to create LDAP structure". What does this mean?

If I use my LDAP admin account (with write access to the entire DIT) this process succeeds however I cannot use that account in production for this.

I can create a service account in LDAP just for this, but that does not seem to work from I.D. even though it tests OK from other applications and the "Test Configuration" button.

[Alexander Sundström]

Hi Dave,

I think you are confusing two different import types, as LDAP and Discovery are separate. But I think you mean that you're LDAP import is not setup correctly, is that right?

What the error means is that based on the information received from the LDAP using the credentials, there is no matching structure that can be created. 

When using the predefined structure and configuration, Insight tries to build up an representation of the data from the LDAP. As one example, If an entry from the LDAP has any objects under it, it's counted as an object type and not an object. Also, the other way around, if you get back a flat structure, with no underlying objects, that will be imported as object types only.
In some cases, it's better to create the mapping manually, as it's not possible for Insight to interpret the LDAP structure correctly.

Best Regards
Alexander

[Dee Heffemm]

Hi Alexander,

Thanks for the reply. Yes, I've confused the Insight (Jira) plugin with Insight Discovery.  Apologies for that.

The Insight LDAP import only succeeds if I use my rootDN for my LDAP directory.  Using rootDN credentials, I can pull all my users into Insight however I want to be able to create an "insight-import" user in LDAP and use that account on Insight to perform the import. This does not work and gives me the error above about "Invalid credentials".

I only included the screenshot of the structure creation because this is the same error I get when testing my import setup. The structure creation does succeed if I use my rootDN credentials as well. As a regular user, it does not.

If I perform an LDAP bind at the command line, as the insight-import user, the password I type is accepted and the bind completes. There seems to be something with Insight that is not allowing me to use a regular user account for the Import. I am wondering if this is maybe a bug or if a workaround exists? I have tried defining the insight-import user in different OUs and it seems to make no difference. Insight user import only works if I use rootDN.

It's possible something is mis-configured with LDAP but we have close to 1000 users authenticating daily against this directory (from many different applications) and haven't seen an issue like this before.

Thanks again, and apologies for the confusion.

[Alexander Sundström]

Hi Dave,

As all we do with the LDAP import, is to use the credentials given, and perform a search based on the filter in the configuration, the error with Invalid Credentials are returned from LDAP, it's not an message that we create or edit, it's straight from the LDAP.
If you try to use as an example ApacheDircectoryStudio (freeware), and use the same credentials, and search filter, what's the result? Does it work or do you encounter similar problems?

Best Regards
Alexander

[Dee Heffemm]

Alexander,

Thanks! As you suggested, I used ApacheDirectoryStudio to test my account and found I was using the wrong DN for the user. I have corrected this (I am now using "uid=jiraldap" instead of "cn=jiraldap") and can now login to the LDAP server through ApacheDirectoryStudio and view my entire LDAP tree without use of the rootDN.

There is still something wrong however. I see an error in the Jira logs about sizelimit when trying to perform the import. Is this due to our LDAP server limiting un-paged results to 500 entries?

2017-11-28 14:43:00,216 insight-DefaultImportService-11-thread WARN myjiraadminuser 883x9408x1 1euc6bu 10.222.1.5,127.0.0.1 /rest/insight/1.0/import/start/8 [c.r.j.p.i.s.imports.common.DefaultImportService] No parallelism configured, will fall back to default (the available amount of cores)
2017-11-28 14:43:00,301 insight-DefaultImportService-11-thread ERROR myjiraadminuser 883x9408x1 1euc6bu 10.222.1.5,127.0.0.1 /rest/insight/1.0/import/start/8 [c.r.j.p.i.s.i.modules.ldap.LdapImportModule] Unable to fetch data locators for configuration com.riadalabs.jira.plugins.insight.services.imports.modules.ldap.LdapConfiguration@612471e0
javax.naming.SizeLimitExceededException: [LDAP: error code 4 - Sizelimit Exceeded]; remaining name 'ou=wusers,dc=my,dc=org'
 at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3188)
 at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:3082)
 at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2888)
 at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.getNextBatch(AbstractLdapNamingEnumeration.java:148)
 at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMoreImpl(AbstractLdapNamingEnumeration.java:217)
 at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMore(AbstractLdapNamingEnumeration.java:189)
 at com.riadalabs.jira.plugins.insight.services.imports.modules.ldap.LdapImportModule.parseResultToBean(LdapImportModule.java:623)
 at com.riadalabs.jira.plugins.insight.services.imports.modules.ldap.LdapImportModule.searchLDAPObjects(LdapImportModule.java:558)
 at com.riadalabs.jira.plugins.insight.services.imports.modules.ldap.LdapImportModule.searchLDAPObjects(LdapImportModule.java:517)
 at com.riadalabs.jira.plugins.insight.services.imports.modules.ldap.LdapImportModule.fetchDataLocators(LdapImportModule.java:170)
 at com.riadalabs.jira.plugins.insight.services.imports.modules.ldap.LdapImportModule.fetchDataLocators(LdapImportModule.java:67)
 at com.riadalabs.jira.plugins.insight.services.jira.module.ImportModuleDelegator.fetchDataLocators(ImportModuleDelegator.java:83)
 at com.riadalabs.jira.plugins.insight.services.imports.common.DefaultImportService.getDataLocators(DefaultImportService.java:300)
 at com.riadalabs.jira.plugins.insight.services.imports.common.DefaultImportService.fetchDataAndMapToImportObjects(DefaultImportService.java:415)
 at com.riadalabs.jira.plugins.insight.services.imports.common.DefaultImportService.fetchDataAndPrepareImportObjects(DefaultImportService.java:1451)
 at com.riadalabs.jira.plugins.insight.services.imports.common.DefaultImportService.doImportFromSource(DefaultImportService.java:560)
 at com.riadalabs.jira.plugins.insight.services.imports.common.DefaultImportService.lambda$_startImport$157(DefaultImportService.java:517)
 at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
 at com.atlassian.sal.core.executor.ThreadLocalDelegateRunnable.run(ThreadLocalDelegateRunnable.java:34)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)

[Dee Heffemm]

This afternoon, I have given the user above 'unlimited' results for an LDAP query. I verified with ldapsearch on the command line that it will return all 1000+ results when querying as the jiraldap user. I can also use ApacheDirectoryStudio to run a query returning 1000+ users (paged or un-paged). The error I get when trying to set the data locator in Insight is attached and the server message is below. Please advise, thank you.

2017-11-28 16:31:53,323 http-nio-8080-exec-18 ERROR tmb384 991x11170x1 1euc6bu 10.222.1.5,127.0.0.1 /rest/insight/1.0/import/datalocators/33 [c.r.j.p.i.s.i.modules.ldap.LdapImportModule] Unable to fetch data locators for configuration com.riadalabs.jira.plugins.insight.services.imports.modules.ldap.LdapConfiguration@51c94ee8
javax.naming.LimitExceededException: [LDAP: error code 11 - illegal pagedResults page size]; remaining name 'ou=wusers,dc=my,dc=org'
 at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3221)
 at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:3082)
 at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2888)
...

[Alexander Sundström]

Hi Dave,

Thanks for the updates. The error caught is not something internal that is created by us, but this is returned from the LDAP itself. Looking at https://docs.oracle.com/javase/tutorial/jndi/ldap/exceptions.html the error is related to Administrative limit exceeded, in this case LimitExceededException that's related to illegal pagedResults page size.

I'm not sure what is different from the command line and our import, but for some reason, the credentials and configuration set result in the LDAP returning this kind of error.

Best Regards
Alexander