MYSQL database corruption in Crowd

[Tom]

We are running Crowd 2.5.1 with a local MYSQL database on our Red Hat server.

Crowd's sync to our AD has been failing with the following error:

=======

2014-11-19 21:31:54,537 scheduler_Worker-6 INFO [atlassian.crowd.directory.DbCachingRemoteDirectory] failed synchronisation complete in [ 9334ms ]

2014-11-19 21:31:54,541 scheduler_Worker-6 ERROR [atlassian.crowd.directory.DbCachingDirectoryPoller] Error occurred while refreshing the cache for directory [ 393217 ].

org.springframework.dao.IncorrectResultSizeDataAccessException: query did not return a unique result: 2; nested exception is org.hibernate.NonUniqueResultException: query did not return a unique result: 2

=======

In practice, what this means is that JIRA and Confluence get incomplete group info, and often what is missing will be certain users will not be in the jira-users group or confluence-users group, which are the groups we use for login authentication for those apps.  We have run database queries such as:

SELECT id,directory_id, lower_parent_name, lower_child_name, membership_type, COUNT(*) as cnt FROM cwd_membership GROUP BY directory_id, lower_parent_name, lower_child_name, membership_type HAVING COUNT(*) > 1;

... and have found some users with duplicate entries.  We removed those users from AD and deleted the dupes in the Crowd MYSQL, and that allowed the sync to complete successfully.  But once we re-added those users to AD, the problem came right back.

Some of this looks like it may be related to case sensitivity:

[root@atlassian01 logs]# grep -i mahler catalina.out.2014-11-20 | tail -1

2014-11-19 21:04:09,209 QuartzWorker-1 WARN ServiceRunner     [atlassian.crowd.directory.DbCachingRemoteChangeOperations] remote username [ Mahler.Brian ] casing differs from local username [ mahler.brian ]. User details will be kept updated, but the username cannot be updated

From https://confluence.atlassian.com/pages/viewpage.action?pageId=254739329 I see that:

Cause

MySQL collation for the Crowd database is set to be case insensitive.

Resolution

Do not use COLLATE utf8_general_ci (e.g. case insensitive). Use COLLATE utf8_bin (e.g. case sensitive). The Crowd schema has keys on VARCHARs which will cause you issues if you use collation ending in CI.

However, it looks like we use utf8_bin already:

[root@atlassian01 crowd]# pwd

/var/lib/mysql/crowd

[root@atlassian01 crowd]# cat db.opt

default-character-set=utf8

default-collation=utf8_bin

I note that we have a TEST instance of Crowd 2.5.1 on an Ubuntu box, and the same corruption does NOT appear in the logs (and the sync works successfully there).

My question is, can I shut down Crowd and MYSQL, and blow away (rename) the corrupt database and then recreate it, hopefully making a new, uncorrupted MYSQL database that will sync properly?  Is this reasonable?  Are there risks to doing this?  And how exactly would I do this (what files/directories would I delete/rename)?

[Giuliano C_]

Hey Tom, 

I would like to share the procedures performed as a troubleshooting in our support channel, that have a ticket opened by you:

Hey Tom,

You did a good research in order to find the possible main cause here, which clearly seems to be indicating the collation used in your instance. Given that, since you have already confirmed that the database is using the recommended collation, perhaps we have some specific tables that are still not using it which could fit with the questions that you have to have it fixed in your side.

So, for that, before doing some manual modifications, I will suggest you to take a look at the following KB article, that will point you to the right recommendations in order to have the collation change and verification for all columns inside the DB:

https://confluence.atlassian.com/display/CONFKB/How+to+Fix+the+Collation+and+Character+Set+of+a+MySQL+Database

Once you take a look at this, let us know how is it going and also if you need some specific recommendations in order to proceed with the modification of the collation.

Hope it's helping you at the moment!

Cheers, 

Giuliano

[rrudnicki]

Hi Thomas, 

Based on your logs, I found this link which can be helpful. 

So, could you please have a look on this link?

Also, it may be a collation issue, so, could you please run the queries below?

SELECT * FROM information_schema.COLUMNS WHERE table_schema = '<databasename>' AND collation_name != 'utf8_bin';
SELECT * FROM information_schema.TABLES WHERE table_schema = '<databasename>' AND table_collation != 'utf8_bin’;

If you get any row on this queries, please follow this link to fix the collations.

Hope it helps, 

Regards 

Renato Rudnicki

[Diego Berrueta]

Hi Thomas. I'm sorry to hear that you're having issues with Crowd. I recommend you open a support ticket at https://support.atlassian.com/ so we can collect all the information and help you better.