MYSQL database corruption in Crowd

We are running Crowd 2.5.1 with a local MYSQL database on our Red Hat server.

Crowd's sync to our AD has been failing with the following error:

=======

2014-11-19 21:31:54,537 scheduler_Worker-6 INFO [atlassian.crowd.directory.DbCachingRemoteDirectory] failed synchronisation complete in [ 9334ms ]

2014-11-19 21:31:54,541 scheduler_Worker-6 ERROR [atlassian.crowd.directory.DbCachingDirectoryPoller] Error occurred while refreshing the cache for directory [ 393217 ].

org.springframework.dao.IncorrectResultSizeDataAccessException: query did not return a unique result: 2; nested exception is org.hibernate.NonUniqueResultException: query did not return a unique result: 2

=======

In practice, what this means is that JIRA and Confluence get incomplete group info, and often what is missing will be certain users will not be in the jira-users group or confluence-users group, which are the groups we use for login authentication for those apps.  We have run database queries such as:

SELECT id,directory_id, lower_parent_name, lower_child_name, membership_type, COUNT(*) as cnt FROM cwd_membership GROUP BY directory_id, lower_parent_name, lower_child_name, membership_type HAVING COUNT(*) > 1;

... and have found some users with duplicate entries.  We removed those users from AD and deleted the dupes in the Crowd MYSQL, and that allowed the sync to complete successfully.  But once we re-added those users to AD, the problem came right back.

Some of this looks like it may be related to case sensitivity:

[root@atlassian01 logs]# grep -i mahler catalina.out.2014-11-20 | tail -1

2014-11-19 21:04:09,209 QuartzWorker-1 WARN ServiceRunner     [atlassian.crowd.directory.DbCachingRemoteChangeOperations] remote username [ Mahler.Brian ] casing differs from local username [ mahler.brian ]. User details will be kept updated, but the username cannot be updated

From https://confluence.atlassian.com/pages/viewpage.action?pageId=254739329 I see that:

Cause

MySQL collation for the Crowd database is set to be case insensitive.

Resolution

Do not use COLLATE utf8_general_ci (e.g. case insensitive). Use COLLATE utf8_bin (e.g. case sensitive). The Crowd schema has keys on VARCHARs which will cause you issues if you use collation ending in CI.

 

However, it looks like we use utf8_bin already:

 

[root@atlassian01 crowd]# pwd

/var/lib/mysql/crowd

[root@atlassian01 crowd]# cat db.opt

default-character-set=utf8

default-collation=utf8_bin

 

I note that we have a TEST instance of Crowd 2.5.1 on an Ubuntu box, and the same corruption does NOT appear in the logs (and the sync works successfully there).

 

My question is, can I shut down Crowd and MYSQL, and blow away (rename) the corrupt database and then recreate it, hopefully making a new, uncorrupted MYSQL database that will sync properly?  Is this reasonable?  Are there risks to doing this?  And how exactly would I do this (what files/directories would I delete/rename)?

3 answers

Hi Thomas. I'm sorry to hear that you're having issues with Crowd. I recommend you open a support ticket at https://support.atlassian.com/ so we can collect all the information and help you better.

0 votes

Hi Thomas, 

Based on your logs, I found this link which can be helpful. 

So, could you please have a look on this link?

 

Also, it may be a collation issue, so, could you please run the queries below?

SELECT * FROM information_schema.COLUMNS WHERE table_schema = '<databasename>' AND collation_name != 'utf8_bin';
SELECT * FROM information_schema.TABLES WHERE table_schema = '<databasename>' AND table_collation != 'utf8_bin’;

If you get any row on this queries, please follow this link to fix the collations.

 

Hope it helps, 

 

Regards 

Renato Rudnicki

0 votes

Hey Tom, 

 

I would like to share the procedures performed as a troubleshooting in our support channel, that have a ticket opened by you:

Hey Tom,

You did a good research in order to find the possible main cause here, which clearly seems to be indicating the collation used in your instance. Given that, since you have already confirmed that the database is using the recommended collation, perhaps we have some specific tables that are still not using it which could fit with the questions that you have to have it fixed in your side.

So, for that, before doing some manual modifications, I will suggest you to take a look at the following KB article, that will point you to the right recommendations in order to have the collation change and verification for all columns inside the DB:

https://confluence.atlassian.com/display/CONFKB/How+to+Fix+the+Collation+and+Character+Set+of+a+MySQL+Database

Once you take a look at this, let us know how is it going and also if you need some specific recommendations in order to proceed with the modification of the collation.

 

Hope it's helping you at the moment!

 

Cheers, 

Giuliano

Suggest an answer

Log in or Join to answer
Community showcase
Teodora [Botron]
Published Thursday in Marketplace Apps

Jira Inferno: The Nine Circles of Jira Administration Hell

If you spend enough time as a Jira admin - whether you are managing a single, mid-sized instance, a large enterprise one or juggling multiple instances at once - you will eventually find yourself in ...

274 views 0 12
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you
Atlassian Team Tour

Join us on the Team Tour

We're bringing product updates and pro tips on teamwork to ten cities around the world.

Save your spot