Confluence failing to sync user directory from Crowd

[jasper connery]

We have confluence using a crowd directory for user authentication and management, and on crowd this is using a delegated autentication directory connected to our active directory domain. This has been working fine, up until we added a new delegated authentication directory for a new domain, and swapped connectors (in crowd) that confluence uses to the newly created connector. This has all the current users migrated over and everything, however we are now getting a sync error everytime the sync runs. The error logs show:

2023-03-17 04:31:08,594 DEBUG [scheduler_Worker-2] [crowd.manager.directory.DirectorySynchroniser] synchronise request to synchronise directory [ 13870875 ] in INCREMENTAL mode
2023-03-17 04:31:08,599 INFO [scheduler_Worker-2] [atlassian.crowd.directory.DbCachingRemoteDirectory] synchroniseCache INCREMENTAL synchronisation for directory [ 13870875 ] starting
2023-03-17 04:31:08,600 INFO [scheduler_Worker-2] [atlassian.crowd.directory.DbCachingRemoteDirectory] synchroniseCache Attempting INCREMENTAL synchronisation for directory [ 13870875 ]
2023-03-17 04:31:08,600 INFO [scheduler_Worker-2] [atlassian.crowd.directory.DbCachingRemoteDirectory] synchroniseCache Incremental synchronisation for directory [ 13870875 ] was not completed, falling back to a full synchronisation
2023-03-17 04:31:08,600 INFO [scheduler_Worker-2] [atlassian.crowd.directory.DbCachingRemoteDirectory] synchroniseCache INCREMENTAL synchronisation for directory [ 13870875 ] was not successful, attempting FULL
2023-03-17 04:31:08,704 WARN [scheduler_Worker-2] [directory.ldap.cache.EventTokenChangedCacheRefresher] synchroniseAll Incremental synchronisation is not available. Falling back to full synchronisation
com.atlassian.crowd.event.IncrementalSynchronisationNotAvailableException
at com.atlassian.crowd.integration.rest.service.RestCrowdClient.getCurrentEventToken(RestCrowdClient.java:1154)
at com.atlassian.crowd.directory.RemoteCrowdDirectory.getCurrentEventToken(RemoteCrowdDirectory.java:782)
at com.atlassian.crowd.directory.ldap.cache.EventTokenChangedCacheRefresher.synchroniseAll(EventTokenChangedCacheRefresher.java:50)
at com.atlassian.crowd.directory.DbCachingRemoteDirectory.synchroniseCache(DbCachingRemoteDirectory.java:1122)
at com.atlassian.crowd.manager.directory.DirectorySynchroniserImpl.synchronise(DirectorySynchroniserImpl.java:76)
at com.atlassian.crowd.directory.DbCachingDirectoryPoller.pollChanges(DbCachingDirectoryPoller.java:50)
at com.atlassian.crowd.manager.directory.monitor.poller.DirectoryPollerJobRunner.runJob(DirectoryPollerJobRunner.java:93)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:153)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:118)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:97)
at com.atlassian.scheduler.quartz1.Quartz1Job.execute(Quartz1Job.java:32)
at org.quartz.core.JobRunShell.run(JobRunShell.java:223)
at com.atlassian.confluence.schedule.quartz.ConfluenceQuartzThreadPool.lambda$runInThread$0(ConfluenceQuartzThreadPool.java:16)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
2023-03-17 04:33:17,191 INFO [scheduler_Worker-2] [atlassian.crowd.directory.DbCachingRemoteDirectory] synchroniseCache failed synchronisation complete for directory [ 13870875 ] in [ 128592ms ]
2023-03-17 04:33:17,196 ERROR [scheduler_Worker-2] [atlassian.crowd.directory.DbCachingDirectoryPoller] pollChanges Error occurred while refreshing the cache for directory [ 13870875 ].
com.atlassian.crowd.exception.OperationFailedException: javax.xml.bind.UnmarshalException
- with linked exception:
[org.apache.http.ConnectionClosedException: Premature end of chunk coded message body: closing chunk expected]
at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:497)
at com.atlassian.crowd.integration.rest.service.RestCrowdClient.getMemberships(RestCrowdClient.java:916)
at com.atlassian.crowd.directory.RemoteCrowdDirectory.getMemberships(RemoteCrowdDirectory.java:1068)
at com.atlassian.crowd.directory.ldap.cache.AbstractCacheRefresher.getMemberships(AbstractCacheRefresher.java:107)
at com.atlassian.crowd.directory.ldap.cache.AbstractCacheRefresher.synchroniseMemberships(AbstractCacheRefresher.java:185)
at com.atlassian.crowd.directory.ldap.cache.AbstractCacheRefresher.synchroniseAll(AbstractCacheRefresher.java:98)
at com.atlassian.crowd.directory.ldap.cache.EventTokenChangedCacheRefresher.synchroniseAll(EventTokenChangedCacheRefresher.java:67)
at com.atlassian.crowd.directory.DbCachingRemoteDirectory.synchroniseCache(DbCachingRemoteDirectory.java:1122)
at com.atlassian.crowd.manager.directory.DirectorySynchroniserImpl.synchronise(DirectorySynchroniserImpl.java:76)
at com.atlassian.crowd.directory.DbCachingDirectoryPoller.pollChanges(DbCachingDirectoryPoller.java:50)
at com.atlassian.crowd.manager.directory.monitor.poller.DirectoryPollerJobRunner.runJob(DirectoryPollerJobRunner.java:93)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:153)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:118)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:97)
at com.atlassian.scheduler.quartz1.Quartz1Job.execute(Quartz1Job.java:32)
at org.quartz.core.JobRunShell.run(JobRunShell.java:223)
at com.atlassian.confluence.schedule.quartz.ConfluenceQuartzThreadPool.lambda$runInThread$0(ConfluenceQuartzThreadPool.java:16)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Caused by: javax.xml.bind.UnmarshalException
- with linked exception:
[org.apache.http.ConnectionClosedException: Premature end of chunk coded message body: closing chunk expected]
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:213)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:190)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:241)
at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:489)
... 17 more
Caused by: org.apache.http.ConnectionClosedException: Premature end of chunk coded message body: closing chunk expected
at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:268)
at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:227)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:186)
at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
at java.io.SequenceInputStream.read(SequenceInputStream.java:207)
at org.apache.xerces.impl.XMLEntityManager$RewindableInputStream.read(Unknown Source)
at org.apache.xerces.impl.io.UTF8Reader.read(Unknown Source)
at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)
at org.apache.xerces.impl.XMLEntityScanner.skipString(Unknown Source)
at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:211)
... 20 more

The sync shows in the web gui that it takes 127s before it fails, which speaks to me of a timeout issue of 2 minutes potentially causing this, but I have updated the timeout at the crowd end to no avail, and dont see a place where it can be done on the confluence end. Any thoughts?

I did also spot this question: Solved: Confluence failing to sync user directory from Cro... (atlassian.com) but they do not seem to have gone into much detail around the solution, or i am not seeing it. 

Confluence version: 5.9.10

Crowd version: 2.9.1

Yes, they are old versions but we cannot upgrade them at this stage

[Fabio Racobaldo _Herzum_]

Hi @jasper connery ,

welcome to the Atlassian community!

Did u try to disable the existing directory and to create a new one with the same parameters? I had the same problem in the past and this workaround solved my issue.

Hope this helps,

Fabio