User directory syncing

[Kevin O'Brien ADMIN]

Hello, 

For a long time we had the user directory syncing to an external OVD (oracle virtual directory) without incident. Lately, the sync has been failing, which results in users not being able to log in, new users not able to be aded to groups, etc. 

Our config didn't change, no settings were changed on OVD and no one seems to know why the sync fails. 

Here is the error from the logs. Thoughts? This is driving me bananas. 

2015-12-02 16:59:45,227 WARN [http-nio-127.0.0.101-8090-exec-25] [org.springframework.beans.TypeConverterDelegate] findDefaultEditor PropertyEditor [com.sun.beans.editors.EnumEditor] found through deprecated global PropertyEditorManager fallback - consider using a more isolated form of registration, e.g. on the BeanWrapper/BeanFactory!
-- referer: https://confluence.umassonline.net/plugins/servlet/embedded-crowd/configure/ldap/?directoryId=101089281&xsrfTokenValue=e6d67d8cf5ea2caab9ee1407ae13ead594033190&xsrfTokenName=atl_token | url: /plugins/servlet/embedded-crowd/configure/ldap/ | userName: cfadmin
2015-12-02 16:59:45,233 INFO [http-nio-127.0.0.101-8090-exec-25] [crowd.embedded.admin.ConfigurationController] doInTransaction User directory updated: [ UMPO LDAP ], type: [ CONNECTOR ]
2015-12-02 16:59:50,243 INFO [scheduler_Worker-4] [atlassian.crowd.directory.DbCachingRemoteDirectory] synchroniseCache INCREMENTAL synchronisation for directory [ 101089281 ] starting
2015-12-02 16:59:50,244 INFO [scheduler_Worker-4] [atlassian.crowd.directory.DbCachingRemoteDirectory] synchroniseCache Attempting INCREMENTAL synchronisation for directory [ 101089281 ]
2015-12-02 16:59:50,244 INFO [scheduler_Worker-4] [atlassian.crowd.directory.DbCachingRemoteDirectory] synchroniseCache Incremental synchronisation for directory [ 101089281 ] was not completed, falling back to a full synchronisation
2015-12-02 16:59:50,244 INFO [scheduler_Worker-4] [atlassian.crowd.directory.DbCachingRemoteDirectory] synchroniseCache INCREMENTAL synchronisation for directory [ 101089281 ] was not successful, attempting FULL
2015-12-02 17:00:45,691 WARN [http-nio-127.0.0.101-8090-exec-1] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1449093585507, threadId=144, threadName='http-nio-127.0.0.101-8090-exec-1', userId='cfadmin', type='web-request', summary='/plugins/servlet/embedded-crowd/directories/troubleshoot?directoryId=101089281&forceTest=true&xsrfTokenValue=e6d67d8cf5ea2caab9ee1407ae13ead594033190&xsrfTokenName=atl_token'}
-- referer: https://confluence.umassonline.net/plugins/servlet/embedded-crowd/configure/ldap/?directoryId=101089281&xsrfTokenValue=e6d67d8cf5ea2caab9ee1407ae13ead594033190&xsrfTokenName=atl_token | url: /plugins/servlet/embedded-crowd/directories/troubleshoot | userName: cfadmin
2015-12-02 17:00:50,282 INFO [scheduler_Worker-4] [crowd.directory.ldap.SpringLdapTemplateWrapper] call Timed call for search with dircontext on o=umassp,dc=ucollaborate,dc=net took 60037ms
2015-12-02 17:00:50,282 INFO [scheduler_Worker-4] [atlassian.crowd.directory.DbCachingRemoteDirectory] synchroniseCache failed synchronisation complete for directory [ 101089281 ] in [ 60039ms ]
2015-12-02 17:00:50,288 ERROR [scheduler_Worker-4] [atlassian.crowd.directory.DbCachingDirectoryPoller] pollChanges Error occurred while refreshing the cache for directory [ 101089281 ].
com.atlassian.crowd.exception.OperationFailedException: org.springframework.ldap.CommunicationException: sm-prd-ovd.ucollaborate.net:636; nested exception is javax.naming.CommunicationException: sm-prd-ovd.ucollaborate.net:636 [Root exception is java.net.SocketTimeoutException: Read timed out]
at com.atlassian.crowd.directory.SpringLDAPConnector.searchEntitiesWithRequestControls(SpringLDAPConnector.java:506)
at com.atlassian.crowd.directory.SpringLDAPConnector.searchEntities(SpringLDAPConnector.java:459)
at com.atlassian.crowd.directory.SpringLDAPConnector.searchUserObjects(SpringLDAPConnector.java:679)
at com.atlassian.crowd.directory.SpringLDAPConnector.searchUsers(SpringLDAPConnector.java:1076)
at com.atlassian.crowd.directory.ldap.cache.RemoteDirectoryCacheRefresher.findAllRemoteUsers(RemoteDirectoryCacheRefresher.java:55)
at com.atlassian.crowd.directory.ldap.cache.RemoteDirectoryCacheRefresher.synchroniseAllUsers(RemoteDirectoryCacheRefresher.java:90)
at com.atlassian.crowd.directory.ldap.cache.AbstractCacheRefresher.synchroniseAll(AbstractCacheRefresher.java:89)
at com.atlassian.crowd.directory.DbCachingRemoteDirectory.synchroniseCache(DbCachingRemoteDirectory.java:1122)
at com.atlassian.crowd.manager.directory.DirectorySynchroniserImpl.synchronise(DirectorySynchroniserImpl.java:76)
at com.atlassian.crowd.directory.DbCachingDirectoryPoller.pollChanges(DbCachingDirectoryPoller.java:50)
at com.atlassian.crowd.manager.directory.monitor.poller.DirectoryPollerJobRunner.runJob(DirectoryPollerJobRunner.java:93)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:135)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:101)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:80)
at com.atlassian.scheduler.quartz1.Quartz1Job.execute(Quartz1Job.java:32)
at org.quartz.core.JobRunShell.run(JobRunShell.java:223)
at com.atlassian.confluence.schedule.quartz.ConfluenceQuartzThreadPool.lambda$runInThread$151(ConfluenceQuartzThreadPool.java:19)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Caused by: org.springframework.ldap.CommunicationException: sm-prd-ovd.ucollaborate.net:636; nested exception is javax.naming.CommunicationException: sm-prd-ovd.ucollaborate.net:636 [Root exception is java.net.SocketTimeoutException: Read timed out]

[Bruno Raymundo]

Hi Kevin,

I also see you are having errors related to Quartz. Quartz is a third-party library used by Confluence for scheduling system tasks, such as flushing the index queue or syncing the directories. In summary, it works like this:

1. Individual jobs are submitted to one of 10 worker threads
2. When a job starts running, it locks an object which all other threads will look for (only one thread runs at any given time; the others wait until the running thread is free)
3. Once the job completes, the running thread releases the lock
4. Every 500ms, the remaining threads check if they are able to start

If some problem occurs during the job execution that doesn't necessarily causes an error (such as a database timeout), the running thread stalls and never releases the lock -- and the other others never get a chance to run. This could be affecting the LDAP connection; if the connection between your server and the OVD port is OK, as per the check suggested by Steve above, please contact Atlassian Support.

Regards,

Bruno

[Steve Behnke [DiscoverEquip.com]]

It says the read timed out. That's clear.

You need to test the connection between your Confluence Server and OVD port that is supposed to be available.