"ERROR: relation "cwd_cluster_safety" does not exist" after upgrade to 3.0.1

[elij]

After upgrading from 2.x (most recent) to 3.0.1, I see the following error in the logs:

ERROR: relation "cwd_cluster_safety" does not exist

This results in another error:

Scheduled job with ID 'com.atlassian.crowd.manager.cluster.ClusterSafetyManager-checkSafety' failed

java.lang.RuntimeException: org.hibernate.exception.SQLGrammarException: could not perform addBatch

[AnnWorley · Accepted solution]

Hi Eli,

The cwd_cluster_safety table was added to the Crowd database in version 3.0 and caused the issue in this bug report: In some cases Crowd might erroneously detect another instance is accessing the Crowd database and shut down

The issue was fixed in Crowd 3.0.1 but you seem to be running into a situation where the cluster safety job is still running even though the table is not there. The errors are benign, but if you want to reduce the noise in your logs you can add:

-Dcrowd.check.concurrent.instances.interval=0

to your CATALINA_OPTS environment variable to disable the job. The Catalina options are in the file:<Crowd_installation_dir>/apache-tomcat/bin/setenv.sh

Thanks,

Ann

[elij]

Thanks Ann. I added that to the JAVA_OPTS (catalina_opts wasn't in that file), and it seems to have stopped filling up the logs.

Thanks!

[Marcin Kempa]

Hi @elij

Could you check if there were no other errors just after starting Crowd 3.0.1? What database are you using and in which version?

Best Regards,

Marcin Kempa

[elij]

Database is PostgreSQL 9.6.5.

hmm. Looks like there was indeed an error in the upgrade in catalina.out.

2017-09-07 11:50:19,963 localhost-startStop-1 ERROR [engine.jdbc.spi.SqlExceptionHelper] ERROR: column am.amcanorder does not exist

  Position: 407

2017-09-07 11:50:19,964 localhost-startStop-1 ERROR [hibernate.tool.hbm2ddl.SchemaUpdate] HHH000299: Could not complete schema update

java.lang.NullPointerException

Not a very solid upgrade process?

Any way to re-run the failed migration?

[Marcin Kempa]

This is what I thought... Indeed the upgrade process could behave better here, however please mind that Postgres 9.6 is not yet supported by Crowd.

Here is a list of supported platforms for Crowd https://confluence.atlassian.com/crowd/supported-platforms-191851.html

Please also mind that the cwd_cluster_safety table might not be the only one that was not created, therefore there might be other problems later on related to missing relations in database.

Regarding re-running the migration, if possible I would recommend to use supported Postgres database and to run the upgrade again after restoring the database from backup. This way we can ensure that all the upgrade tasks are run and migrate the data in a proper way.
Please let me know if that suits you.

[elij]

hmm. For some reason I thought 9.6 was supported (out for almost a year already), but it looks like only 9.5 is. Well that is certainly a bummer. ;_;

Maybe instead I will just hope for the best, keep running on 9.6 for now, and migrate our company off crowd entirely as soon as we can.

[Marcin Kempa]

Sorry to hear that you are experiencing problems with Crowd.

Are there any other issues that are driving you into decision of migrating off Crowd completly?

I would like to understand the problems and would like to help as best as I can. We can move this discussion off the community.atlassian.com.

My email address is: mkempa@atlassian.com

Best Regards,

Marcin Kempa

[elij]

I have been looking for documentation on migrating from crowd to internal jira user management, and have only found the doc that cover that step for confluence.

It looks like you have to either manually recreate all users, or modify the directory_id for the users in the tables manually?

As to why we want to migrate away from crowd... it isn't really doing us much good. We have a confluence and jira instance, and would be better served simply having confluence use jira as the directory server. The use of Crowd was probably just an overzealous initial configuration most likely.