nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Crowd Bootstrap Failed - The Crowd database is being updated by another instance

Mystech

I've recently upgraded to Crowd 3.0.0 and upon starting crowd, crowd fully loads but shortly after it gives the error:

ERROR [atlassian.scheduler.core.JobLauncher] Scheduled job with ID 'com.atlassian.crowd.manager.cluster.ClusterSafetyManager-checkSafety' failed
java.lang.RuntimeException: The Crowd database (jdbc:postgresql://localhost:5432/crowd) is being updated by another instance. The instance IP is 127.0.0.1. Please make sure all the instances connected to this database are Data Center instances and have clustering enabled.

I have a single server deployment, not a clustered deployment. I did find an article on Confluence that has a similar issue but the resolutions weren't helpful to Crowd.

I also note that I'm connecting via localhost in my database strings, but that is giving the instance IP of 127.0.01...of course both of those are the same. I tried updating my connection string to replace localhost with 127.0.0.1 in case there was an issue there, but either way I get the same error (the difference being localhost above is replaced with 127.0.0.1).

This seems to be a situation where the Crowd Server is blocking itself, but I'm not sure where to go or how to troubleshoot.

Accepted answers

All comments

AnnWorley

All my test instances are using localhost in the JDBC connection string so I think we are ok there.

Crowd thinks another instance is connecting to it's database. We call this state a "cluster panic".

Please make sure there is only one entry in the table cwd_cluster_safety by running this query:

select * from cwd_cluster_safety;

There should only be one record in that table, if there are two, shutdown Crowd, backup the database and delete one of the rows. If you would like syntax to delete a row, please post the results of the query.

In my testing, I only had one record in the cwd_cluster_safety table and when I shut down Crowd it persisted. Because it is a test instance I experimented by deleting the only row and Crowd re-populated it on startup and seems to be running fine. I have to emphasize, since this seems to be your Production instance please back up the database before running any SQL against it.

Mystech

Thanks Ann. There's only one entry in the cwd_cluster_safety table. I have a backup of the database from before the upgrade, so I tried the same thing yesterday before posting -- I shut down Crowd and just deleted the one entry from the cwd_cluster_safety table. On startup it repopulated, but I was still getting the same error.

AnnWorley

If this is on Linux, please check that there is no other instance running (or not quite shut down):

ps aux | grep java

In Windows the processes can be listed by running TASKLIST

The way I am reading the error message it sounds like the other instance updating the database is on 127.0.0.1 (the local machine).

Mystech

It's a Linux server. I do have other java applications on that server (I have other Atlassian products), but crowd is not running. Running the grep for java apps shows confluence & jira, but not crowd. Running the grep for crowd only returns the grep command.

When I perform the upgrade for Crowd my first step is to stop previous instances and move the install directory so I don't inadvertently start it up or start up the wrong instance.

I'm also running Crowd in the foreground for testing so I can make sure nothing else is starting up (it happens whether I run it in the foreground or not though).

lpater

Hi,

As Ann mentioned this is triggered by a check, that tries to determine that multiple Crowd instances are trying to access the same database, and shuts down Crowd, in order to prevent data corruption.

This check was introduced in 3.0.0, the way it works is that every 2 minutes, the instance reads a value from the 'cwd_cluster_safety' table, compares it with the value it expects to find, and writes a new value there.

If the value read from the table differs from the one previously written the instance shuts down with the message you mentioned. This shouldn't be happening if only a single instance is running.

I'd advise double-checking that there's not a leftover instance running still, or if any other processes might be accessing the database and changing it in this manner. Try running `ps aux` as a superuser, to verify the process list, and search for Crowd instances (ther should be exactly one). Also enabling database query logging, and checking the queries made on the table might help you discover the culprit.

If you still have issues, please reach out to Atlassian Support, so that we can investigate further.

Mystech

Yesterday while testing I rebooted the server just to make sure there was nothing in memory. I have confirmed by running ps aux as speruser that there is only one process in the list, but I did not go so far as enabling database query logging.

Just wanted to make a note here that it's definitely not an issue of multiple instance or the same instance running twice.

Sistemas Startic

Same problem here.

I have this topology

APPS----ssl-----[NGINX------8090----CROWD]

nginx, crowd, and postgres are in the same server. Postgres listening in localhost only, and upgrading from 2.7.3 to 3.0 I receive this error some seconds after crowd startup

The Crowd database (jdbc:postgresql://127.0.0.1:5432/crowddb) is being updated by another instance. The instance IP is 127.0.0.1. Please make sure all the instances connected to this database are Data Center instances and have clustering enabled.

After that error I check cwd_cluster_safety table and there is only one record from 127.0.0.1. Of course there is only one crowd instance.

Maybe the reverse proxy from nginx is causing this?

I just upgraded my crowd instance following Automatic Database Upgrade method, but I had to roll back because of that bug.

Please tell me how to disable that scheduled job aswell.

Thank you.

mimacom International GmbH

Hi there,

I'm facing the same issue but I can fix it by simply rebooting the instance.

It seems to occur after I run my ansible playbook which is used to setup Crowd and other servers.

I checked this by printing the contents of "cwd_cluster_safety" before and after the playbook run.

Before:

crowd=# select * from cwd_cluster_safety;
 entry_key | entry_value | node_id | ip_address | entry_timestamp
--------------------+--------------------------------------+---------------+---------------+-----------------
 clusterSafetyToken | ca6bea73-5a84-45ba-a825-27093f91a204 | NOT_CLUSTERED | 172.18.136.20 | 1503562271442
(1 row)

After:

crowd=# select * from cwd_cluster_safety;
 entry_key | entry_value | node_id | ip_address | entry_timestamp
--------------------+--------------------------------------+---------------+---------------+-----------------
 clusterSafetyToken | 55137567-37d8-4811-8c3b-4002bf3cd2b7 | NOT_CLUSTERED | 172.18.136.20 | 1503562391439
(1 row)

The value "entry_value" immediately changes after the playbook run, but the instance is running without any error messages.

When I run my playbook again, the entry in the database changes again:

crowd=# select * from cwd_cluster_safety;
 entry_key | entry_value | node_id | ip_address | entry_timestamp
--------------------+--------------------------------------+---------------+---------------+-----------------
 clusterSafetyToken | e58872e6-30fc-4b44-99da-48e01041de6d | NOT_CLUSTERED | 172.18.136.20 | 1503562918315
(1 row)

Additionally, when I tail the logs I can see somehow an error occuring and Crowd seems to reboot in itself (without raising a new linux process.

Filtered logs:

2017-08-24 10:22:09,640 ContainerBackgroundProcessor[StandardEngine[Catalina]] ERROR [atlassian.event.internal.AsynchronousAbleEventDispatcher] There was an exception thrown trying to dispatch event [com.atlassian.plugin.event.events.PluginFrameworkShutdownEvent@256fdd8e] from the invoker [com.atlassian.plugin.event.impl.MethodSelectorListenerHandler$1$1@68ff6095]
java.lang.RuntimeException: java.lang.NullPointerException

[...]

2017-08-24 10:22:09,664 ContainerBackgroundProcessor[StandardEngine[Catalina]] INFO [com.atlassian.crowd.startup] Stopping Crowd
2017-08-24 10:23:11,433 Caesium-1-2 ERROR [crowd.manager.cluster.ClusterSafetyManager] The Crowd database (jdbc:postgresql://localhost:5432/crowd) is being updated by another instance. The instance IP is 172.18.136.20. Please make sure all the instances connected to this database are Data Center instances and have clustering enabled.

Linux processes, there is always one:

[root@mima-crowdtest-01 rewe]# ps -ef | grep java | grep -v grep | wc -l
1

And it is the same as before I ran my playbook, because it's running since 30mins:

[root@mima-crowdtest-01 rewe]# ps -o etime= -p 23986
 30:20

Important to note that ansible reports that it didn't change anything, on all three runs!

This brings me to the question, what value "entry_value" is exactly and where it comes from, why it seems to change during the runtime and why Crowd seems to kinda reboot itself?

Cheers,
Remo

lpater

Hi all,

I've raised CWD-4964 to track this issue. We'll release Crowd 3.0.1 shortly with a fix.

In the meantime you can disable the check using the system property documented in the CWD-4964 - this should prevent the problem from occuring.

lpater

A new version of Crowd, 3.0.1 is now available, and contains a fix for this problem.

Marcin Kempa

@Mystech, @mimacom International GmbH, @Michael Schneider, @Sistemas Startic

Could you tell a little bit more about how the upgrade process looks like in your organisation? Are you using Crowd WAR distribution by any chance?

After analysing one of the cases with exact same problem, I noticed that the Crowd web context is started two times but for different path (context root):

localhost-startStop-1 INFO [ContainerBase.[Catalina].[localhost].[/]] Initializing Spring root WebApplicationContext

and after some time later in logs:

localhost-startStop-1 INFO [ContainerBase.[Catalina].[localhost].[/crowd]] Initializing Spring root WebApplicationContext

which is then followed up with two (or more) log entires such as:

Starting Crowd Server, Version: 3.0.0 (Build:#865 - 2017-08-14)

occuring shortly after each other (there are no entries about Crowd being stopped in between).

This indicates that even though there is only one java Crowd process as returned by

$ ps -ef | grep java | grep -v grep | wc -l

there are two (or possibly more) Crowd applications deployed on the same application server (tomcat by default in Crowd standalone distribution).

Therefore, as this might turn into a problem (running more than one instance of Crowd on the samedatabase), I would like to ask you if you could check for aforementioned log entries and check if by any chance you are running more then one Crowd application deployed to the same application server (one can check that by connecting via jconsole to the tomcat java process)

I would grately appreciate your reply.

Marcin Kempa

This problem may be related with the following issue CWD-4974,

@Mystech, @mimacom International GmbH, @Sistemas Startic, @Michael Schneider could you check if this is the case in your setup?

Thanks,

Marcin Kempa

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

crowd