Confluence hang when trying to sync user directory

Problem 

Sometimes when a user tries to log in to confluence the system gets stuck loading for a while. There seems to be a problem with synchronising users from the active directory. 

 

Firstly it tries to perform an incremental synchronisation of the cache with the active directory, and try to add 100 users (who have seemingly been added recently):

 

2017-06-29 09:09:39,563 INFO [Caesium-1-2] [atlassian.crowd.directory.DbCachingRemoteDirectory] synchroniseCache INCREMENTAL synchronisation for directory [ 76480513 ] starting

...

2017-06-29 09:09:43,161 INFO [Caesium-1-2] [atlassian.crowd.directory.DbCachingRemoteChangeOperations] addUsers adding [ 100 ] users

 

Then it will throw a duplicate key error for  a user (alias xxx.xxx)

 

2017-06-29 09:09:43,246 ERROR [Caesium-1-2] [engine.jdbc.spi.SqlExceptionHelper] logExceptions Violation of UNIQUE KEY constraint 'cwd_user_name_dir_id'. Cannot insert duplicate key in object 'dbo.cwd_user'. The duplicate key value is (xxx.xxx, 0513).

2017-06-29 09:09:43,248 WARN [Caesium-1-2] [persistence.hibernate.batch.AbstractBatchProcessor] processBatch batch failed falling back to individual processing

java.lang.RuntimeException: could not flush session

 

It can't seem to process users in a batch because of single user xxx.xxx? And so it tries to process indivduals - but confluence does not support individual processing.

2017-06-29 09:09:43,250 ERROR [Caesium-1-2] [persistence.hibernate.batch.AbstractBatchProcessor] processIndividual Could not process class com.atlassian.crowd.util.persistence.hibernate.batch.TransactionGroup: com.atlassian.crowd.util.persistence.hibernate.batch.TransactionGroup@7faa647e[primaryObject=com.atlassian.crowd.model.user.InternalUser@7c8e78d8[id=130457012,name=yyy.yyy,createdDate=Thu Jun 29 09:09:43 BST 2017,updatedDate=Thu Jun 29 09:09:43 BST

java.lang.RuntimeException: Confluence does not support individual processing

 

Then notifies that those users cannot be processed:

 

2017-06-29 09:09:43,314 ERROR [Caesium-1-2] [crowd.embedded.hibernate2.HibernateUserDao] addAll The following users could not be processed:

list of the users

 

 

Then confluence goes unresponsive for a few minutes (see time stamps) where confluence is essentially stuck in loading and then this is reported:

2017-06-29 09:12:55,713 WARN [http-nio-8443-exec-3] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1498723405374, threadId=226, threadName='http-nio-8443-exec-3', userId='alex.Perrin', type='web-request', summary='/'}

 -- url: / | traceId: 8ff5ec5656cd6ef0 | userName: alex.Perrin

 

Then it seems to work fine for a while until it tries to do this again.

 

Any help/comments much appreciated!

2 answers

1 accepted

Update - This was actually an issue with Active Directory - we had two duplicates of that particular user. Deleting one of them solved the issue. However it seems that the Confluence hang is unrelated to the user issue, and it still occurs.

Ann Worley Atlassian Team Jul 28, 2017

Glad to hear that fixing the user in AD fixed the sync issue.

I understand Confluence still hangs, is that still during the LDAP sync or it hangs periodically and the LDAP sync is healthy now?

This article has some good advice for troubleshooting hangs/crashes: Troubleshooting Confluence hanging or crashing

I look forward to hearing what you find.

0 vote
Ann Worley Atlassian Team Jul 21, 2017

Please let me know what version of Confluence you are running. There have been changes to the way user management works over different versions.

As you noticed, the unique constraint error you cited is telling us that some record is already in the cwd_user table and the sync process is trying to add it again.

  • If you create a new LDAP User directory (Confluence Admin>User Directories) with the same settings it will probably work fine, as it will create a new cache.
  • After you are sure the new directory is working (test by disabling the old one and logging in with LDAP credentials) you can remove the current User Directory.
  • If you are using "read-only with local groups" and added a lot of users to local groups, these memberships will not exist in the new directory, so it's not an optimal solution in that case.

Another strategy that sometimes works in similar cases is to temporarily turn off incremental sync (edit the User Directory and uncheck it) and run a full sync.

Note: If you have Follow Referrals checked in the User Directory it could be causing the unexpected 100 users every time you sync as it will refer the sync to look for users on other AD servers than the one you designated in the settings.

 

Thanks for the reply Ann!

So this problem is actually ocurring in our test environment in which we upgraded our test instance, as a test for this operation in production. The problem started ocurring after this upgrade. 

We upgraded from version 5.4.4 to 6.2.1. I had to change the collation of the database as per https://confluence.atlassian.com/confkb/how-to-fix-the-collation-of-a-microsoft-sql-server-confluence-database-687216725.html. Not sure if this could have caused this problem somehow?

I've re-created the user directory multiple times, and deleted the old directories, and still get the same issue...

I've also tried turning off incremental sync, but with no improvement. Follow referals is not checked.

We have the same settings as in our production environment, which works fine.

Another peculiar feature is that user, if I click "Edit details" on her profile in confluence, her name has a + symbol after it, which is not present in the production version... Could that plus symbol be causing Confluence to try and add the user again perhaps?

Suggest an answer

Log in or Sign up to answer
How to earn badges on the Atlassian Community

How to earn badges on the Atlassian Community

Badges are a great way to show off community activity, whether you’re a newbie or a Champion.

Learn more
Community showcase
Posted Jul 10, 2018 in Confluence

We want to see the templates you've created in Confluence!

Hi Community, Jessica here from the Confluence Product Marketing team!  July’s community challenge is all about sharing pictures  — and as an extension of our first post on what ...

716 views 21 12
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you