LDAP Connection Sync issues - Users can't login often

[SaSu]

Hello,

we have a Jira Server Instance with a connected Microsoft Active Directory user directory. 

Since we upgraded to Jira 8.* we have massive problems with the LDAP sync.

Very often users are not able to login. Most often the automatic sync times out. 

When users complain about not being able to login we try to trigger the sync manually. Sometimes it works then. 

Maybe someone has an idea of what to do?

We found this error message in our logs:

---

atlassian-jira.log

Caesium-1-2 ERROR ServiceRunner [c.a.crowd.directory.DbCachingRemoteDirectory] Incremental synchronisation for directory [ 10000 ] was unexpectedly interrupted, falling back to a full synchronisation

org.springframework.ldap.UncategorizedLdapException: Uncategorized exception occured during LDAP processing; nested exception is javax.naming.NamingException: LDAP response read timed out, timeout used: 600000 ms.; remaining name '/'

at org.springframework.ldap.support.LdapUtils.convertLdapException(LdapUtils.java:228)

at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:397)

at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:440)

at com.atlassian.crowd.directory.ldap.SpringLdapTemplateWrapper$2.timedGet(SpringLdapTemplateWrapper.java:128)

at com.atlassian.crowd.directory.ldap.monitoring.TimedSupplier.get(TimedSupplier.java:37)

at com.atlassian.crowd.directory.ldap.SpringLdapTemplateWrapper.invokeWithContextClassLoader(SpringLdapTemplateWrapper.java:85)

at com.atlassian.crowd.directory.ldap.SpringLdapTemplateWrapper.lookup(SpringLdapTemplateWrapper.java:117)

at com.atlassian.crowd.directory.MicrosoftActiveDirectory.fetchInvocationId(MicrosoftActiveDirectory.java:714)

at com.atlassian.crowd.directory.synchronisation.cache.UsnChangedCacheRefresher.synchroniseChanges(UsnChangedCacheRefresher.java:113)

at com.atlassian.crowd.directory.DbCachingRemoteDirectory.synchroniseCache(DbCachingRemoteDirectory.java:1080)

at com.atlassian.crowd.manager.directory.DirectorySynchroniserImpl.lambda$synchronise$0(DirectorySynchroniserImpl.java:82)

at com.atlassian.crowd.audit.NoOpAuditLogContext.withAuditLogSource(NoOpAuditLogContext.java:17)

at com.atlassian.crowd.manager.directory.DirectorySynchroniserImpl.synchronise(DirectorySynchroniserImpl.java:80)

at com.atlassian.crowd.directory.DbCachingDirectoryPoller.pollChanges(DbCachingDirectoryPoller.java:48)

at com.atlassian.crowd.manager.directory.monitor.poller.DirectoryPollerJobRunner.runJob(DirectoryPollerJobRunner.java:92)

at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)

at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)

at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)

at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)

at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)

at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)

at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)

at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)

at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)

at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)

at java.base/java.lang.Thread.run(Unknown Source)

Caused by: javax.naming.NamingException: LDAP response read timed out, timeout used: 600000 ms.; remaining name '/'

at java.naming/com.sun.jndi.ldap.LdapRequest.getReplyBer(Unknown Source)

at java.naming/com.sun.jndi.ldap.Connection.readReply(Unknown Source)

at java.naming/com.sun.jndi.ldap.LdapClient.getSearchReply(Unknown Source)

at java.naming/com.sun.jndi.ldap.LdapClient.search(Unknown Source)

at java.naming/com.sun.jndi.ldap.LdapCtx.doSearch(Unknown Source)

at java.naming/com.sun.jndi.ldap.LdapCtx.c_search(Unknown Source)

at java.naming/com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(Unknown Source)

at java.naming/com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(Unknown Source)

at java.naming/javax.naming.directory.InitialDirContext.search(Unknown Source)

at java.base/jdk.internal.reflect.GeneratedMethodAccessor746.invoke(Unknown Source)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.base/java.lang.reflect.Method.invoke(Unknown Source)

at org.springframework.ldap.transaction.compensating.manager.TransactionAwareDirContextInvocationHandler.invoke(TransactionAwareDirContextInvocationHandler.java:90)

at com.sun.proxy.$Proxy4116.search(Unknown Source)

at com.atlassian.crowd.directory.ldap.SpringLdapTemplateWrapper$2.lambda$timedGet$0(SpringLdapTemplateWrapper.java:124)

at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:363)

... 24 more

---

catalina.out

WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [http-nio-8090-exec-11 url: /plugins/servlet/embedded-crowd/directories/troubleshoot/; user: xxx.xxx] (id=[29]) has been active for [127,419] milliseconds (since [6/11/22 1:58 PM]) to serve the same request for [xxx/plugins/servlet/embedded-crowd/directories/troubleshoot/] and may be stuck (configured threshold for this StuckThreadDetectionValve is [120] seconds). There is/are [1] thread(s) in total that are monitored by this Valve and may be stuck.

java.lang.Throwable

at java.base@11.0.13/jdk.internal.misc.Unsafe.park(Native Method)

at java.base@11.0.13/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)

at java.base@11.0.13/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)

at java.base@11.0.13/java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)

at java.naming@11.0.13/com.sun.jndi.ldap.LdapRequest.getReplyBer(Unknown Source)

at java.naming@11.0.13/com.sun.jndi.ldap.Connection.readReply(Unknown Source)

at java.naming@11.0.13/com.sun.jndi.ldap.LdapClient.getSearchReply(Unknown Source)

at java.naming@11.0.13/com.sun.jndi.ldap.LdapClient.search(Unknown Source)

at java.naming@11.0.13/com.sun.jndi.ldap.LdapCtx.doSearch(Unknown Source)

at java.naming@11.0.13/com.sun.jndi.ldap.LdapCtx.searchAux(Unknown Source)

at java.naming@11.0.13/com.sun.jndi.ldap.LdapCtx.c_search(Unknown Source)

[Elias Brattli Sørensen - Kantega SSO]

Hi,

I believe what you are seeing is your LDAP is in periods overloaded and not able to answer queries and gives “connect timed out”. This could be caused by your LDAP directory settings where it by default is set to "Update group memberships on each login". Since Jira queries the LDAP server upon each login of all your users, this will put a lot of traffic towards the LDAP server.

A solution we have found to avoiding such problems is changing the user directory configuration “Update group memberships when logging in” and setting this to either “for newly added users only” or “Never”. This will mean that user login will not update group memberships. Memberships will then only be updated on each sync interval (which is by default every 60 minutes). We have seen that this is a satisfiable solution for several of our clients.

Please try this for your user directory that is failing and report back. Hopefully, this will make logins more stable.

Regards,
Elias Brattli Sørensen
Kantega SSO

[SaSu]

Hi Elias, 

thanks for your response. We already did what you wrote and there is no real change.  We are now in communication with the atlassian support team. Thanks a lot for your answer.

[Rilwan Ahmed]

Hi @SaSu 

I am afraid you are hit with a existing bug https://jira.atlassian.com/browse/JRASERVER-71465. But I could give you some suggestion and you can try if it fixes.

1. Increase read timeout

Go to Administration > Users > User Directories
Edit the LDAP directory
Increase the value of Read Timeout

2.  Disable the Follow Referral option
Go to Administration > Users > User Directories
Edit the LDAP directory
Disable the Follow Referral option

Please note: If you are logged using the same AD, then I would suggest you to create a internal user, grant system admin access, log in using internal user and then perform the above two actions. 

3. Restart Jira and check if issue still exists. 

[SaSu]

Hi @Rilwan Ahmed 

thanks for your answer and tips. 

We already did the two steps you mentioned. Thatd oes not change something.

Are you sure that it is the bug that is described in the issue? There is a different error message.

[Rilwan Ahmed]

Hi @SaSu

I would suggest, raise an Atlassian Support ticket as your users have the impact. Run the sync and then attach the zip file for the ticket. 

[SaSu]

Hi @Rilwan Ahmed 

we raised a support ticket. Thanks for helping!

[Waqar Mustafa]

Hi Sasu,

Did you able to resolve this issue?

Would you please guide me, what was the issue and how it resolved.

Thanks in advance