Confluence failing to sync user directory from Crowd

Rizwan Basha October 23, 2017

We have had Confluence syncing user directory from Crowd for the last several months, and all of a sudden it started to fail synchronization. We have 2 directories defined in Crowd - an Active Directory and a custom directory implemented using the RemoteDirectory interface of Crowd. It had been working with no issues until recently. We have JIRA and Confluence syncing user directories from the same instance of Crowd. JIRA is still functioning with no problems, but Confluence started to fail in synchronization. in every attempt of syncing, it takes over 10 minutes and fail. The fact that JIRA is not affected tells me that the problem may lie on Confluence end.  We are using Confluence version 6.1.2.

Can anyone please help me resolve this.  Here is the log information with stack trace:

2017-10-23 08:17:01,698 INFO [Caesium-1-1] [atlassian.crowd.directory.DbCachingRemoteDirectory] synchroniseCache INCREMENTAL synchronisation for directory [ 3604481 ] starting
2017-10-23 08:17:01,703 INFO [Caesium-1-1] [atlassian.crowd.directory.DbCachingRemoteDirectory] synchroniseCache Attempting INCREMENTAL synchronisation for directory [ 3604481 ]
2017-10-23 08:17:01,703 INFO [Caesium-1-1] [atlassian.crowd.directory.DbCachingRemoteDirectory] synchroniseCache Incremental synchronisation for directory [ 3604481 ] was not completed, falling back to a full synchronisation
2017-10-23 08:17:01,703 INFO [Caesium-1-1] [atlassian.crowd.directory.DbCachingRemoteDirectory] synchroniseCache INCREMENTAL synchronisation for directory [ 3604481 ] was not successful, attempting FULL
2017-10-23 08:17:06,576 INFO [Caesium-1-1] [directory.ldap.cache.RemoteDirectoryCacheRefresher] findAllRemoteUsers found [ 268 ] remote users in [ 4812 ms ]
2017-10-23 08:17:06,589 INFO [Caesium-1-1] [atlassian.crowd.directory.DbCachingRemoteChangeOperations] deleteCachedUsersNotIn scanned and compared [ 268 ] users for delete in DB cache in [ 8ms ]
2017-10-23 08:17:06,589 INFO [Caesium-1-1] [atlassian.crowd.directory.DbCachingRemoteChangeOperations] deleteCachedUsersNotIn scanned for deleted users in [ 8ms ]
2017-10-23 08:17:06,607 INFO [Caesium-1-1] [atlassian.crowd.directory.DbCachingRemoteChangeOperations] getUsersToAddAndUpdate scanning [ 268 ] users to add or update
2017-10-23 08:17:06,614 INFO [Caesium-1-1] [atlassian.crowd.directory.DirectoryCacheImplUsingChangeOperations] addOrUpdateCachedUsers scanned and compared [ 268 ] users for update in DB cache in [ 19ms ]
2017-10-23 08:17:06,630 INFO [Caesium-1-1] [atlassian.crowd.directory.DirectoryCacheImplUsingChangeOperations] addOrUpdateCachedUsers synchronised [ 268 ] users in [ 35ms ]
2017-10-23 08:17:06,959 INFO [Caesium-1-1] [directory.ldap.cache.RemoteDirectoryCacheRefresher] findAllRemoteGroups found [ 157 ] remote groups in [ 329 ms ]
2017-10-23 08:17:06,969 INFO [Caesium-1-1] [atlassian.crowd.directory.DbCachingRemoteChangeOperations] deleteCachedGroupsNotIn scanned and compared [ 157 ] groups for delete in DB cache in [ 6ms ]
2017-10-23 08:17:06,975 INFO [Caesium-1-1] [atlassian.crowd.directory.DirectoryCacheImplUsingChangeOperations] addOrUpdateCachedGroups scanning [ 157 ] groups to add or update
2017-10-23 08:17:06,986 INFO [Caesium-1-1] [atlassian.crowd.directory.DbCachingRemoteChangeOperations] findGroupsToUpdate scanned and compared [ 157 ] groups for update in DB cache in [ 6ms ]
2017-10-23 08:17:07,017 INFO [Caesium-1-1] [atlassian.crowd.directory.DirectoryCacheImplUsingChangeOperations] addOrUpdateCachedGroups synchronized [ 157 ] groups in [ 42ms ]
2017-10-23 08:17:10,072 INFO [Caesium-1-1] [directory.ldap.cache.AbstractCacheRefresher] synchroniseAllGroupAttributes finished group attribute sync with 0 failures in [ 3054ms ]
2017-10-23 08:27:10,109 INFO [Caesium-1-1] [atlassian.crowd.directory.DbCachingRemoteDirectory] synchroniseCache failed synchronisation complete for directory [ 3604481 ] in [ 608411ms ]
2017-10-23 08:27:10,150 ERROR [Caesium-1-1] [atlassian.crowd.directory.DbCachingDirectoryPoller] pollChanges Error occurred while refreshing the cache for directory [ 3604481 ].
com.atlassian.crowd.exception.OperationFailedException: java.net.SocketTimeoutException: Read timed out
at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:425)
at com.atlassian.crowd.integration.rest.service.RestCrowdClient.getMemberships(RestCrowdClient.java:1022)
at com.atlassian.crowd.directory.RemoteCrowdDirectory.getMemberships(RemoteCrowdDirectory.java:1096)
at com.atlassian.crowd.directory.ldap.cache.AbstractCacheRefresher.getMemberships(AbstractCacheRefresher.java:72)
at com.atlassian.crowd.directory.ldap.cache.AbstractCacheRefresher.synchroniseMemberships(AbstractCacheRefresher.java:185)
at com.atlassian.crowd.directory.ldap.cache.AbstractCacheRefresher.synchroniseAll(AbstractCacheRefresher.java:63)
at com.atlassian.crowd.directory.ldap.cache.EventTokenChangedCacheRefresher.synchroniseAll(EventTokenChangedCacheRefresher.java:69)
at com.atlassian.crowd.directory.DbCachingRemoteDirectory.synchroniseCache(DbCachingRemoteDirectory.java:1186)
at com.atlassian.crowd.manager.directory.DirectorySynchroniserImpl.synchronise(DirectorySynchroniserImpl.java:74)
at com.atlassian.crowd.directory.DbCachingDirectoryPoller.pollChanges(DbCachingDirectoryPoller.java:50)
at com.atlassian.crowd.manager.directory.monitor.poller.DirectoryPollerJobRunner.runJob(DirectoryPollerJobRunner.java:96)
at com.atlassian.confluence.impl.schedule.caesium.JobRunnerWrapper.doRunJob(JobRunnerWrapper.java:114)
at com.atlassian.confluence.impl.schedule.caesium.JobRunnerWrapper.lambda$runJob$0(JobRunnerWrapper.java:90)
at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:87)
at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:71)
at com.atlassian.confluence.impl.schedule.caesium.JobRunnerWrapper.runJob(JobRunnerWrapper.java:90)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:153)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:118)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:97)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:443)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeLocalJob(CaesiumSchedulerService.java:410)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:388)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:285)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:282)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:65)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:59)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:34)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
at org.apache.http.impl.client.cache.CachingExec.callBackend(CachingExec.java:592)
at org.apache.http.impl.client.cache.CachingExec.handleCacheMiss(CachingExec.java:356)
at org.apache.http.impl.client.cache.CachingExec.execute(CachingExec.java:275)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.executeCrowdServiceMethod(RestExecutor.java:574)
at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:413)
... 27 more

1 answer

1 accepted

1 vote
Answer accepted
Rizwan Basha October 25, 2017

We have resolved this issue.  This was a timeout being caused by a very slow method in the implementation of RemoteDirectory interface in Crowd.  We have implemented RemoteDirectory interface in Crowd in order to implement a custom user directory from our database.  A recent change in one of the methods made it too slow.  We fixed that method and the timeout issue is fixed.  However, JIRA was not affected by this because JIRA, for some unknown reason, was not making the calls to the slow method, and Confluence was.

Deleted user October 20, 2020

Hi Rizwan,
Can you please provide more information about your fix (the method) and what means "remoteDirectory interface". Is it related to the time out of the user/admin session in Crowd interface.? Because I am facing sync error whith a similar configuration that yours and the time out in Crowd is very very short.

Thanks in advance

Arnold

Om Joshi September 14, 2022

Where did you find the logs? i am unable to locate the logs and which file to refer

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events