Active Directory group members do not update after AD changes

[Rondel Ward]

I'm using a Microsoft Active Directory (read Only) as my default Directory in Confluence 3.5.11. Since it is read only, I have the group membership maintained from AD. However, though the connection to AD is solid and users can log in with their domain credentials, the group memberships do NOT update. e.g I had a user added to a group over 10 days ago and the change has not been reflected in the Confluence group. Here is what I have tried:

• Manual Synchronise- Takes about 10 minutes or so but doesn't update the group members
• Flushing Cahce ->Manual Sync- Still doesn't work
• Making Internal default->Clicking Edit on AD connection->Test->Save->Sync->Make default again: Group members still have not updated

I have verified that the changes have been made in AD but they do not reflect in Confluence. In fact, Confluence claims to sync every hour but still have not picked up the changes. How can I resolve this issue?

EDIT

I've been looking through the logs and I found that I was getting some errors midway though the group membership synchornisation process:

Log details:

2011-12-31 06:42:50,893 ERROR [QuartzScheduler_Worker-4] [sf.hibernate.util.JDBCExceptionReporter] logExceptions Violation of UNIQUE KEY constraint 'cwd_unique_membership'. Cannot insert duplicate key in object 'dbo.cwd_membership'. The duplicate key value is (3606017, <NULL>, 3575236).

2011-12-31 06:42:50,893 ERROR [QuartzScheduler_Worker-4] [sf.hibernate.impl.SessionImpl] execute Could not synchronize database state with session

2011-12-31 06:42:50,893 WARN [QuartzScheduler_Worker-4] [persistence.hibernate.batch.AbstractBatchProcessor] processBatch batch failed falling back to individual processing

java.lang.RuntimeException: could not flush session

at ...

Caused by: net.sf.hibernate.exception.ConstraintViolationException: could not insert: [com.atlassian.crowd.embedded.hibernate2.HibernateMembership#3661670]

...

Caused by: java.sql.SQLException: Violation of UNIQUE KEY constraint 'cwd_unique_membership'. Cannot insert duplicate key in object 'dbo.cwd_membership'. The duplicate key value is (3606017, <NULL>, 3575236).

...

EDIT:

The membership errors did not seem to be the problem. I ran the fixcwdmembership.jsp repair for Confluence but apparently from my reading those errors are harmless in Confluence versions with the patch (so any after 3.5.5). I was piecing through the logs and I found some other possible errors. I keep seeing this WARNING/ERROR combination:

ERROR 1:

2012-01-05 05:59:53,282 WARN [QuartzScheduler_Worker-8] [atlassian.crowd.directory.DbCachingRemoteChangeOperations] getUsersToAddAndUpdate remote username [ craghavendra ] casing differs from local username [ CRaghavendra ]. User details will be kept updated, but the username cannot be updated

2012-01-05 06:01:26,698 ERROR [QuartzScheduler_Worker-8] [crowd.embedded.hibernate2.HibernateMembershipDao] addAllUsersToGroup The following group memberships could not be processed:

2012-01-05 06:01:26,698 ERROR [QuartzScheduler_Worker-8] [crowd.embedded.hibernate2.HibernateMembershipDao] addAllUsersToGroup CRaghavendra into SDG Development

2012-01-05 06:01:26,698 ERROR [QuartzScheduler_Worker-8] [crowd.embedded.hibernate2.HibernateMembershipDao] addAllUsersToGroup Please try to resolve any errors with these users and groups, and try again.

Could it be that this one username is causing problems for the entire collection of groups? This seems like an unlikely problem. I feel like it should keep processing groups after running into one error.

ERROR 2:

WARN [QuartzScheduler_Worker-9] [persistence.hibernate.batch.AbstractBatchProcessor] processBatch batch failed falling back to individual processing

java.lang.RuntimeException: could not flush session

...

ERROR [QuartzScheduler_Worker-9] [persistence.hibernate.batch.AbstractBatchProcessor] processIndividual Could not process class com.atlassian.crowd.embedded.hibernate2.HibernateMembership: com.atlassian.crowd.embedded.hibernate2.HibernateMembership@1439cb5

java.lang.RuntimeException: Confluence does not support individual processing

Is this `individual processing` something that I should try to set up? Could this be the issue?

ERROR 3:

2012-01-04 18:33:43,869 INFO [QuartzScheduler_Worker-1] [atlassian.crowd.directory.DbCachingRemoteDirectory] synchroniseCache full synchronisation complete in [ 45201ms ]

2012-01-04 18:33:43,869 ERROR [QuartzScheduler_Worker-1] [atlassian.crowd.directory.DbCachingDirectoryPoller] pollChanges Error occurred while refreshing the cache for directory [ 3506177 ].

...

Caused by: java.util.concurrent.ExecutionException: com.atlassian.crowd.exception.OperationFailedException: org.springframework.ldap.PartialResultException: nested exception is javax.naming.PartialResultException [Root exception is javax.naming.CommunicationException: DomainDnsZones.ldap.server.local:389 [Root exception is java.net.SocketTimeoutException: connect timed out]]

...

Caused by: com.atlassian.crowd.exception.OperationFailedException: org.springframework.ldap.PartialResultException: nested exception is javax.naming.PartialResultException [Root exception is javax.naming.CommunicationException: DomainDnsZones.ldap.server.local:389 [Root exception is java.net.SocketTimeoutException: connect timed out]]

...

Caused by: org.springframework.ldap.PartialResultException: nested exception is javax.naming.PartialResultException [Root exception is javax.naming.CommunicationException: DomainDnsZones.pclc0.merkle.local:389 [Root exception is java.net.SocketTimeoutException: connect timed out]]

...

Caused by: javax.naming.PartialResultException [Root exception is javax.naming.CommunicationException: DomainDnsZones.ldap.server.local:389 [Root exception is java.net.SocketTimeoutException: connect timed out]]

...

... 13 more

Caused by: javax.naming.CommunicationException: DomainDnsZones.ldap.server .local:389 [Root exception is java.net.SocketTimeoutException: connect timed out]

...

... 15 more

Caused by: java.net.SocketTimeoutException: connect timed out

at java.net.PlainSocketImpl.socketConnect(Native Method)

... 18 more

This seems to say that the synchronization completed successfully but fails to refresh the cache. Then I get a Partial Result Exception because it looks like the connection times out. I'm not sure if the timeout is the cause of the errors or if the errors result in the timeout. In any case, is this the most likely cause and is there a workaround for this?

[Jeremy Largman · Accepted solution]

I think you've hit this bug: https://jira.atlassian.com/browse/CONF-22541

Since you're on 3.5.11, you've already got the patch. You should be able to log in as a confluence administrator, navigate to <confluence base url>/admin/fixcwdmemberships.jsp, and follow the prompts to correct it.

[Rondel Ward]

I checked this out and I tried the fix and I got a "Membership repair completed successfully". However, I still get the same issue with the groups not updating. I looked into that issue some more and it looks like the cwd_membership errors are harmless and can be ignored.

[Jeremy Largman]

Anything in the logs, similar to what you found with the groups?

[Rondel Ward]

I appreciate your help with this. I updated the question again with a couple more errors from the log. I tried to filter out unnecessary details to make it more reasonable but let me know if there is any extra information that could help.

[Rondel Ward]

Turns out that it was the duplicate membership issue. The problem is that this 'fix' introduces it's own problems which cancel the sync under certain circumstances. https://jira.atlassian.com/browse/CONF-22631 The suggested fix is to upgrade to Confluence 3.5.12. Thanks

[Rondel Ward]

Turns out that it was the duplicate membership issue. The problem is that this 'fix' introduces it's own problems which cancel the sync under certain circumstances. https://jira.atlassian.com/browse/CONF-22631 The suggested fix is to upgrade to COnfluence 3.5.12. I'll try to get the ok to do the upgrade. Thanks

[Jeremy Largman]

You beat me to it. I was in the midst of looking at it but couldn't find that buried comment. I'll try to make it a bit more prominent. Thanks for updating your find.

[Rondel Ward]

Thanks for the help. I meant to enter this as a comment to your answer not a new answer altogether.