Duplicate key when synchronising with Azure AD on Crowd server

[Ariel Perez]

This has been setup as a new directory within our Crowd instance (v3.3.3). It always fails with the following duplicate key error.

Have tried enabling/disabling the directory, removing and re-adding but it always returns the same error.

2019-02-19 21:08:41,257 Caesium-2-2 INFO [microsoft.aad.adal4j.AuthenticationAuthority] [Correlation ID: 74f825cd-1424-4f55-ab18-62dbe8bb88f3] Instance discovery was successful

2019-02-19 21:08:41,922 Caesium-2-2 INFO [atlassian.crowd.directory.DbCachingRemoteDirectory] FULL synchronisation for directory [ 1540098 ] starting

2019-02-19 21:08:44,079 Caesium-2-2 INFO [atlassian.crowd.directory.DbCachingRemoteChangeOperations] scanned and compared [ 655 ] users for delete in DB cache in [ 20ms ]

2019-02-19 21:08:44,079 Caesium-2-2 INFO [atlassian.crowd.directory.DbCachingRemoteChangeOperations] scanned for deleted users in [ 20ms ]

2019-02-19 21:08:44,095 Caesium-2-2 INFO [atlassian.crowd.directory.DbCachingRemoteChangeOperations] scanning [ 655 ] users to add or update

2019-02-19 21:08:44,100 Caesium-2-2 INFO [atlassian.crowd.directory.DirectoryCacheImplUsingChangeOperations] scanned and compared [ 655 ] users for update in DB cache in [ 17ms ]

2019-02-19 21:08:44,101 Caesium-2-2 INFO [atlassian.crowd.directory.DirectoryCacheImplUsingChangeOperations] synchronised [ 655 ] users in [ 18ms ]

2019-02-19 21:08:49,533 Caesium-2-2 INFO [atlassian.crowd.directory.DbCachingRemoteChangeOperations] scanned and compared [ 379 ] groups for delete in DB cache in [ 33ms ]

2019-02-19 21:08:49,535 Caesium-2-2 INFO [atlassian.crowd.directory.DirectoryCacheImplUsingChangeOperations] scanning [ 379 ] groups to add or update

2019-02-19 21:08:49,544 Caesium-2-2 INFO [atlassian.crowd.directory.DbCachingRemoteChangeOperations] scanned and compared [ 379 ] groups for update in DB cache in [ 9ms ]

2019-02-19 21:08:49,553 Caesium-2-2 ERROR [jdbc.batch.internal.BatchingBatch] HHH000315: Exception executing batch [java.sql.BatchUpdateException: Duplicate entry 'group 4-1540098' for key 'uk_group_name_dir_id'], SQL: insert into cwd_group (group_name, lower_group_name, active, is_local, created_date, updated_date, description, group_type, directory_id, external_id, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)

2019-02-19 21:08:49,553 Caesium-2-2 WARN [engine.jdbc.spi.SqlExceptionHelper] SQL Error: 1062, SQLState: 23000

2019-02-19 21:08:49,553 Caesium-2-2 ERROR [engine.jdbc.spi.SqlExceptionHelper] Duplicate entry 'group 4-1540098' for key 'uk_group_name_dir_id'

2019-02-19 21:08:49,553 Caesium-2-2 WARN [hibernate.batch.hibernate5.Hibernate5BatchProcessor] batch failed falling back to individual processing

javax.persistence.PersistenceException: org.hibernate.exception.ConstraintViolationException: could not execute batch




Caused by: org.hibernate.exception.ConstraintViolationException: could not execute batch




Caused by: java.sql.BatchUpdateException: Duplicate entry 'group 4-1540098' for key 'uk_group_name_dir_id'




Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry 'group 4-1540098' for key 'uk_group_name_dir_id'




2019-02-19 21:08:49,574 Caesium-2-2 ERROR [jdbc.batch.internal.BatchingBatch] HHH000315: Exception executing batch [java.sql.BatchUpdateException: Duplicate entry 'group 4-1540098' for key 'uk_group_name_dir_id'], SQL: insert into cwd_group (group_name, lower_group_name, active, is_local, created_date, updated_date, description, group_type, directory_id, external_id, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)

2019-02-19 21:08:49,574 Caesium-2-2 WARN [engine.jdbc.spi.SqlExceptionHelper] SQL Error: 1062, SQLState: 23000

2019-02-19 21:08:49,574 Caesium-2-2 ERROR [engine.jdbc.spi.SqlExceptionHelper] Duplicate entry 'group 4-1540098' for key 'uk_group_name_dir_id'

2019-02-19 21:08:49,576 Caesium-2-2 ERROR [hibernate.batch.hibernate5.Hibernate5BatchProcessor] Could not process class com.atlassian.crowd.util.persistence.hibernate.batch.TransactionGroup: com.atlassian.crowd.util.persistence.hibernate.batch.TransactionGroup@362f01ee[primaryObject=com.atlassian.crowd.model.group.InternalGroup@33141109[id=1442593,name=group 4,type=GROUP,active=true,description=group 4,lowerName=group 4,createdDate=Tue Feb 19 21:08:49 GMT 2019,updatedDate=Tue Feb 19 21:08:49 GMT 2019,directoryId=1540098,externalId=ac27893f-8eaf-4d84-9f50-f8554e38d6f3]]

javax.persistence.PersistenceException: org.hibernate.exception.ConstraintViolationException: could not execute batch




Caused by: org.hibernate.exception.ConstraintViolationException: could not execute batch




Caused by: java.sql.BatchUpdateException: Duplicate entry 'group 4-1540098' for key 'uk_group_name_dir_id'




Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry 'group 4-1540098' for key 'uk_group_name_dir_id'




2019-02-19 21:08:49,580 Caesium-2-2 ERROR [jdbc.batch.internal.BatchingBatch] HHH000315: Exception executing batch [java.sql.BatchUpdateException: Duplicate entry 'group 3-1540098' for key 'uk_group_name_dir_id'], SQL: insert into cwd_group (group_name, lower_group_name, active, is_local, created_date, updated_date, description, group_type, directory_id, external_id, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)

2019-02-19 21:08:49,580 Caesium-2-2 WARN [engine.jdbc.spi.SqlExceptionHelper] SQL Error: 1062, SQLState: 23000

2019-02-19 21:08:49,580 Caesium-2-2 ERROR [engine.jdbc.spi.SqlExceptionHelper] Duplicate entry 'group 3-1540098' for key 'uk_group_name_dir_id'

2019-02-19 21:08:49,581 Caesium-2-2 ERROR [hibernate.batch.hibernate5.Hibernate5BatchProcessor] Could not process class com.atlassian.crowd.util.persistence.hibernate.batch.TransactionGroup: com.atlassian.crowd.util.persistence.hibernate.batch.TransactionGroup@f115b5c[primaryObject=com.atlassian.crowd.model.group.InternalGroup@48a6597[id=1442594,name=group 3,type=GROUP,active=true,description=<null>,lowerName=group 3,createdDate=Tue Feb 19 21:08:49 GMT 2019,updatedDate=Tue Feb 19 21:08:49 GMT 2019,directoryId=1540098,externalId=2d75321d-406f-4447-a020-522513e6c059]]

javax.persistence.PersistenceException: org.hibernate.exception.ConstraintViolationException: could not execute batch




Caused by: org.hibernate.exception.ConstraintViolationException: could not execute batch

Caused by: java.sql.BatchUpdateException: Duplicate entry 'group 3-1540098' for key 'uk_group_name_dir_id'




Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry 'group 3-1540098' for key 'uk_group_name_dir_id'

2019-02-19 21:08:49,586 Caesium-2-2 ERROR [jdbc.batch.internal.BatchingBatch] HHH000315: Exception executing batch [java.sql.BatchUpdateException: Duplicate entry 'group 1-1540098' for key 'uk_group_name_dir_id'], SQL: insert into cwd_group (group_name, lower_group_name, active, is_local, created_date, updated_date, description, group_type, directory_id, external_id, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)

2019-02-19 21:08:49,586 Caesium-2-2 WARN [engine.jdbc.spi.SqlExceptionHelper] SQL Error: 1062, SQLState: 23000

2019-02-19 21:08:49,586 Caesium-2-2 ERROR [engine.jdbc.spi.SqlExceptionHelper] Duplicate entry 'group 1-1540098' for key 'uk_group_name_dir_id'

2019-02-19 21:08:49,587 Caesium-2-2 ERROR [hibernate.batch.hibernate5.Hibernate5BatchProcessor] Could not process class com.atlassian.crowd.util.persistence.hibernate.batch.TransactionGroup: com.atlassian.crowd.util.persistence.hibernate.batch.TransactionGroup@66565585[primaryObject=com.atlassian.crowd.model.group.InternalGroup@4d750357[id=1442595,name=group 1,type=GROUP,active=true,description=group 1,lowerName=group 1,createdDate=Tue Feb 19 21:08:49 GMT 2019,updatedDate=Tue Feb 19 21:08:49 GMT 2019,directoryId=1540098,externalId=ed6126a0-fc84-4e58-a4a8-5da260485f9a]]

javax.persistence.PersistenceException: org.hibernate.exception.ConstraintViolationException: could not execute batch

Caused by: org.hibernate.exception.ConstraintViolationException: could not execute batch




Caused by: java.sql.BatchUpdateException: Duplicate entry 'group 1-1540098' for key 'uk_group_name_dir_id'




Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry 'group 1-1540098' for key 'uk_group_name_dir_id'




2019-02-19 21:08:49,590 Caesium-2-2 ERROR [jdbc.batch.internal.BatchingBatch] HHH000315: Exception executing batch [java.sql.BatchUpdateException: Duplicate entry 'group 2-1540098' for key 'uk_group_name_dir_id'], SQL: insert into cwd_group (group_name, lower_group_name, active, is_local, created_date, updated_date, description, group_type, directory_id, external_id, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)

2019-02-19 21:08:49,590 Caesium-2-2 WARN [engine.jdbc.spi.SqlExceptionHelper] SQL Error: 1062, SQLState: 23000

2019-02-19 21:08:49,590 Caesium-2-2 ERROR [engine.jdbc.spi.SqlExceptionHelper] Duplicate entry 'group 2-1540098' for key 'uk_group_name_dir_id'

2019-02-19 21:08:49,591 Caesium-2-2 ERROR [hibernate.batch.hibernate5.Hibernate5BatchProcessor] Could not process class com.atlassian.crowd.util.persistence.hibernate.batch.TransactionGroup: com.atlassian.crowd.util.persistence.hibernate.batch.TransactionGroup@65457c12[primaryObject=com.atlassian.crowd.model.group.InternalGroup@41fd202b[id=1442596,name=group 2,type=GROUP,active=true,description=group 2,lowerName=group 2,createdDate=Tue Feb 19 21:08:49 GMT 2019,updatedDate=Tue Feb 19 21:08:49 GMT 2019,directoryId=1540098,externalId=5bb07fc7-f714-45fc-9af4-fadab2412899]]

javax.persistence.PersistenceException: org.hibernate.exception.ConstraintViolationException: could not execute batch

Caused by: org.hibernate.exception.ConstraintViolationException: could not execute batch




Caused by: java.sql.BatchUpdateException: Duplicate entry 'group 2-1540098' for key 'uk_group_name_dir_id'




Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry 'group 2-1540098' for key 'uk_group_name_dir_id'




2019-02-19 21:08:49,593 Caesium-2-2 WARN [atlassian.crowd.directory.DbCachingRemoteChangeOperations] Could not add the following entity to the directory [ Crowd Internal Directory ]: group 4

2019-02-19 21:08:49,593 Caesium-2-2 WARN [atlassian.crowd.directory.DbCachingRemoteChangeOperations] Could not add the following entity to the directory [ Crowd Internal Directory ]: group 3

2019-02-19 21:08:49,593 Caesium-2-2 WARN [atlassian.crowd.directory.DbCachingRemoteChangeOperations] Could not add the following entity to the directory [ Crowd Internal Directory ]: group 1

2019-02-19 21:08:49,593 Caesium-2-2 WARN [atlassian.crowd.directory.DbCachingRemoteChangeOperations] Could not add the following entity to the directory [ Crowd Internal Directory ]: group 2

2019-02-19 21:08:49,593 Caesium-2-2 INFO [atlassian.crowd.directory.DbCachingRemoteChangeOperations] added [ 0 ] groups successfully in [ 45ms ]

2019-02-19 21:08:49,598 Caesium-2-2 INFO [atlassian.crowd.directory.DbCachingRemoteChangeOperations] updated [ 1 ] groups successfully in [ 2ms ]

2019-02-19 21:08:49,600 Caesium-2-2 INFO [atlassian.crowd.directory.DirectoryCacheImplUsingChangeOperations] synchronized [ 379 ] groups in [ 65ms ]

2019-02-19 21:08:49,601 Caesium-2-2 INFO [atlassian.crowd.directory.DbCachingRemoteDirectory] failed synchronisation complete for directory [ 1540098 ] in [ 7679ms ]

2019-02-19 21:08:49,612 Caesium-2-2 ERROR [atlassian.crowd.directory.DbCachingDirectoryPoller] Error occurred while refreshing the cache for directory [ 1540098 ].

java.lang.IllegalStateException: Found groups with duplicate name 'group 2', ids: 'd0946814-ec21-4e2e-953d-0448fcddbaf2', '5bb07fc7-f714-45fc-9af4-fadab2412899'

at com.atlassian.crowd.directory.rest.util.ThrowingMapMergeOperatorUtil.lambda$duplicateMappedGroupsThrowingMerger$4(ThrowingMapMergeOperatorUtil.java:37)

at java.util.HashMap.merge(HashMap.java:1254)

at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320)

at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)

at java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1625)

at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)

at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)

at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)

at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)

at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)

at com.atlassian.crowd.directory.cache.DeltaQueryCacheRefresher.synchroniseAllMemberships(DeltaQueryCacheRefresher.java:188)

at com.atlassian.crowd.directory.cache.DeltaQueryCacheRefresher.synchroniseAll(DeltaQueryCacheRefresher.java:92)

at com.atlassian.crowd.directory.DbCachingRemoteDirectory.synchroniseCache(DbCachingRemoteDirectory.java:1022)

at com.atlassian.crowd.manager.directory.DirectorySynchroniserImpl.lambda$synchronise$0(DirectorySynchroniserImpl.java:80)

at com.atlassian.crowd.audit.AuditLogContextInternalImpl.withAuditLogSource(AuditLogContextInternalImpl.java:52)

at com.atlassian.crowd.manager.directory.DirectorySynchroniserImpl.synchronise(DirectorySynchroniserImpl.java:78)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)

at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)

at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)

at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)

at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)

at com.sun.proxy.$Proxy68.synchronise(Unknown Source)

at com.atlassian.crowd.directory.DbCachingDirectoryPoller.pollChanges(DbCachingDirectoryPoller.java:45)

at com.atlassian.crowd.manager.directory.monitor.poller.DirectoryPollerJobRunner.runJob(DirectoryPollerJobRunner.java:92)

at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:153)

at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:118)

at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:97)

at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:443)

at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:438)

at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:462)

at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:390)

at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:285)

at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:282)

at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:65)

at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:59)

at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:34)

at java.lang.Thread.run(Thread.java:748)

[Bruno Vincent]

Hi @Ariel Perez 

Did you check that you don't have multiple groups with that same name in Azure AD?

Quoting Atlassian's documentation:

In Azure AD, you can have multiple groups with the same name (displayName), but it's not supported in Crowd and results in a failing synchronization. Make sure you change your Azure AD group names to unique ones.