Crowd temporarily unavailable status

[Jon Broker]

Hello,

We're receiving a message in the crowd.log
http-nio-8095-exec-X INFO [atlassian.johnson.filters.Jonson503Filter] The application is unavailable, or there are errors. Returning a temporarily unavailable status.

Any idea where to start on this issue? We're unable to log into any Atlassian products.

Thank you

[Jon Broker · Accepted solution]

We are using CrowdID to log into our Atlassian products. 
 
From the webpage we're also seeing a bootstrap error:

Fatal, Crowd has encountered errors while upgrading. Please resolve these errors and restart Crowd.

An error occurred when querying the database <ERROR: relation "REMOTEGROUP" does not exist Position: 26 Location: File: src\backend\parser\parse_relation.c, Routine: parserOpenTable, Line 984 Server SQLStateL 42P01>

[Bruno Vincent]

Were you trying to upgrade Crowd when you first got the error or did it suddenly happen? 

https://confluence.atlassian.com/crowdkb/error-table-crowd-remotegroup-doesn-t-exist-after-migrating-crowd-database-from-another-server-726564970.html

[Jon Broker]

We upgraded maybe a month ago, this suddenly happened. We're running Crowd 3.1.2 now. I do see that entry in our config still, should that be removed?

[Bruno Vincent]

I think you should not remove anything from the database without the advice of Atlassian support. What about the buildnumber in crowd.home/shared/crowd.cfg.xml ? Has it got the right value? (891 for Crowd 3.1.2)

[Jon Broker]

That worked! Thank you so much for your time Bruno, we really appreciate it.
I find it strange that this problem manifested now, it's been quite some time since we updated.

[Bruno Vincent]

You're welcome. I find it strange too, anyway I'm glad you get access back to your applications!

[Jon Broker]

Bruno,

I'm not seeing any errors for exceeding user count, we are seeing,

WARN [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [openidserver] appears to have started a thread named [opened-spray-akka.actor.default-dispatcher-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of threads:
  sun.misc.Unsafe.park(Native Method)
  scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.javaL2075)
  scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java1979)
  scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
SEVERE [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks the web application [openidserver] created a ThreadLocal with key of type [scala.concurrent.forkjoin.ThreadLocalRandom$1] (value [scala.concurrent.forkjoin.ThreadLocalRandom$1@7c844568) and a value of type [scala.concurrent.forkjoin.ThreadLocalRandom] (value [scala.concurrent.forkjoin.ThreadLocalRandom@311307ed) but fiale dto remove it when the web application was stopped. Thread are going to be renewed over time to try and avoid a probable memory leak.
SEVERE [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks the web application [openidserver] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3e53cf2e]) and value of type pscala.concurrent.forkjoin.ForkJoinPool.Submitter] (value [scala.concurrent.forkjoin.ForkJoinPool$Submitter@3e829fca]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
SEVERE [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks the web application [openidserver] created a ThreadLocal with key of type [akka.actor.ActorCell$$anon$1] (value [akka.actor.ActorCell$$anon$1@6766f81c]) and a value of type [scala.collection.immutable.Nil$] (value [List()]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

However, Catalina continues to startup

[Bruno Vincent]

Are you actually using openidserver (CrowdID) or is it deployed just because you kept the standard settings while installing Crowd? If you're not using it, disable it:

• Rename openidserver.xml (and openidclient.xml) as openidserver.xml.inactivated (and openidclient.xml.inactivated) in apache-tomcat/conf/Catalina/localhost. Then stop Tomcat/Crowd.
• Make sure that there is no Java process remaining
• Then start Crowd again and let's see what you get.

[Mandia]

Hi @Bruno Vincent 

When I upgrade Crowd 3.2.3. I have the same error. But I am not find the buildnumber in crowd.home/shared/crowd.cfg.xml. Could you please help me?

Thanks,

Mandia

[Bruno Vincent]

Hi @Mandia

Crowd v3.2.3's build number is 954

Hope this helps.

[Bruno Vincent]

Hello @Jon Broker

Any other errors or warnings in catalina.out? What do you get when you try to connect to Crowd's console in a browser?

(My guess is that you might have exceeded the number of users allowed by your license.)