Jira core server won't start after upgrade

[Sokom]

Hello,

we have upgraded our jira core server instance from 8.2.x to 9.3.0 version. Upgrade procedure seems to have been carried out correctly, but we are no longer able to start the jira service correctly.

catalina.out log reports some strange error like:

2022-09-30 17:48:15,581+0200 ThreadPoolAsyncTaskExecutor::Thread 18 INFO anonymous [c.a.ratelimiting.internal.RateLimitingConfiguration] Rate Limited Request Handler ENABLED
Cannot run program "/bin/java": error=2, File or directory not found
Cannot run program "/opt/atlassian/jira/jre//jre/bin/java": error=2, File or directory not found

Furthermore, two plugin have problems to load:

'com.atlassian.jira.plugins.jira-bitbucket-connector-plugin' - 'Jira DVCS Connector Plugin'
'com.atlassian.jira.plugins.jira-development-integration-plugin'

We are using Debian 11 and PostgreSQL on our vm.

[Sokom · Accepted solution]

up

[Peter Van de Voorde]

Looks like Jira is unable to find your Java installation. 
Can you check if Java is properly installed under /bin/java ?

[Sokom]

Hi, thank you for your answer.

Nope, looks like there is no /bin/java

Pretty strange, how can be gone?

[Peter Van de Voorde]

I don't know, but you can reinstall it by following this article: https://www.digitalocean.com/community/tutorials/how-to-install-java-with-apt-on-debian-11 

Cheers,

Peter

[Sokom]

Thank you, it turns out that java was there but in another location, just create a symlink to the right location and that line: "Cannot run program "/bin/java": error=2, File or directory not found " is gone.

But nothing seems changed

It seems to me that there is a wrong path here: Cannot run program "/opt/atlassian/jira/jre//jre/bin/java": error=2, File or directory not found

But I don't know where to find it

[Peter Van de Voorde]

Hey Sokom,

Can you try to change the Java version being used by Jira using this article: https://confluence.atlassian.com/jirakb/how-to-change-the-java-version-used-by-jira-765594330.html ?

Cheers,
Peter

[Sokom]

Hi, yes, done and set the JAVA_HOME dir in setenv.sh to my actual location: "/usr/lib/jvm/java-11-openjdk-amd64", then I add the folder "/usr/lib/jvm/java-11-openjdk-amd64/bin" to my PATH env variable.

But logs show that it keeps to concat something to that folder path:

"Cannot run program "/usr/lib/jvm/java-11-openjdk-amd64/jre/bin/java": error=2"

There is no /usr/lib/jvm/java-11-openjdk-amd64/jre/ folder. I really don't understand why it does it

[Peter Van de Voorde]

I see, you installed just Java, to run Jira you need to install the Java Runtime Environment too (the JRE as they call it). 

Check this page from Oracle to see how it's done: https://docs.oracle.com/goldengate/1212/gg-winux/GDRAD/java.htm#BGBFJHAB 

Cheers,
Peter

[Sokom]

Sorry for the late and thanks again for your support, I was checking all the config.
This is the output of my java -version:
openjdk version "11.0.16" 2022-07-19
OpenJDK Runtime Environment (build 11.0.16+8-post-Debian-1deb10u1)
OpenJDK 64-Bit Server VM (build 11.0.16+8-post-Debian-1deb10u1, mixed mode, sharing)

BTW, I have a version of Java even in jira installation folder so I set up JAVA_HOME in setenv.sh to use that "/opt/atlassian/jira/jre" but when I start jira.service in catalina.out I see "/opt/atlassian/jira/jre/jre/bin/java"

[Sokom]

Hi, can someone help us?

We have fixed some info in setenv.sh explicitly declaring the home folder etc.. but we keep seeing those errors with the wrong java path in the log.

this is our netstat output regarding java

tcp 0 0 0.0.0.0:44659 0.0.0.0:* LISTEN 8462/java
tcp 0 0 0.0.0.0:39261 0.0.0.0:* LISTEN 8462/java
tcp 0 0 127.0.0.1:39144 127.0.0.1:5432 ESTABLISHED 8462/java
tcp 0 0 127.0.0.1:39132 127.0.0.1:5432 ESTABLISHED 8462/java
tcp 0 0 127.0.0.1:54626 127.0.0.1:5432 ESTABLISHED 8462/java
tcp 0 0 127.0.0.1:47266 127.0.0.1:5432 ESTABLISHED 8462/java
tcp 86 0 158.255.193.26:48276 158.255.193.26:443 CLOSE_WAIT 8462/java
tcp6 0 0 :::8090 :::* LISTEN 693/java
tcp6 0 0 :::8091 :::* LISTEN 1070/java
tcp6 0 0 127.0.0.1:8000 :::* LISTEN 693/java
tcp6 0 0 127.0.0.1:8005 :::* LISTEN 8138/java
tcp6 0 0 :::8080 :::* LISTEN 8138/java
tcp6 0 0 127.0.0.1:57356 127.0.0.1:8091 ESTABLISHED 693/java
tcp6 0 0 127.0.0.1:38552 127.0.0.1:5432 ESTABLISHED 693/java
tcp6 0 0 127.0.0.1:36282 127.0.0.1:5432 ESTABLISHED 693/java
tcp6 0 0 127.0.0.1:45074 127.0.0.1:5432 ESTABLISHED 8138/java
tcp6 0 0 127.0.0.1:44986 127.0.0.1:5432 ESTABLISHED 8138/java
tcp6 0 0 127.0.0.1:35882 127.0.0.1:5432 ESTABLISHED 693/java
tcp6 0 0 127.0.0.1:60262 127.0.0.1:5432 ESTABLISHED 693/java
tcp6 0 0 127.0.0.1:45068 127.0.0.1:5432 ESTABLISHED 8138/java
tcp6 0 0 127.0.0.1:37942 127.0.0.1:5432 ESTABLISHED 693/java
tcp6 0 0 127.0.0.1:44980 127.0.0.1:5432 ESTABLISHED 8138/java
tcp6 0 0 127.0.0.1:51234 127.0.0.1:5432 ESTABLISHED 693/java
tcp6 0 0 127.0.0.1:45052 127.0.0.1:5432 ESTABLISHED 8138/java
tcp6 0 0 127.0.0.1:52020 127.0.0.1:5432 ESTABLISHED 693/java
tcp6 0 0 127.0.0.1:51980 127.0.0.1:5432 ESTABLISHED 693/java
tcp6 0 0 127.0.0.1:35870 127.0.0.1:5432 ESTABLISHED 693/java
tcp6 0 0 127.0.0.1:39808 127.0.0.1:5432 ESTABLISHED 693/java
tcp6 25 0 158.255.193.26:58268 185.166.143.35:443 CLOSE_WAIT 693/java
tcp6 0 0 127.0.0.1:8091 127.0.0.1:57356 ESTABLISHED 1070/java
tcp6 0 0 127.0.0.1:44920 127.0.0.1:5432 ESTABLISHED 8138/java
tcp6 0 0 127.0.0.1:46400 127.0.0.1:5432 ESTABLISHED 693/java
tcp6 0 0 127.0.0.1:45038 127.0.0.1:5432 ESTABLISHED 8138/java
tcp6 0 0 127.0.0.1:33952 127.0.0.1:5432 ESTABLISHED 8138/java
tcp6 32 0 158.255.193.26:50744 18.158.131.75:443 CLOSE_WAIT 8138/java
tcp6 0 0 127.0.0.1:45008 127.0.0.1:5432 ESTABLISHED 8138/java
tcp6 25 0 158.255.193.26:43604 185.166.143.34:443 CLOSE_WAIT 693/java
tcp6 0 0 127.0.0.1:45796 127.0.0.1:5432 ESTABLISHED 693/java
tcp6 0 0 127.0.0.1:46642 127.0.0.1:5432 ESTABLISHED 693/java
tcp6 0 0 127.0.0.1:44968 127.0.0.1:5432 ESTABLISHED 8138/java
tcp6 0 0 127.0.0.1:33956 127.0.0.1:5432 ESTABLISHED 8138/java
tcp6 0 0 127.0.0.1:52018 127.0.0.1:5432 ESTABLISHED 693/java
tcp6 0 0 127.0.0.1:44954 127.0.0.1:5432 ESTABLISHED 8138/java
tcp6 25 0 158.255.193.26:59962 185.166.143.35:443 CLOSE_WAIT 693/java
tcp6 0 0 127.0.0.1:33938 127.0.0.1:5432 ESTABLISHED 8138/java
tcp6 0 0 127.0.0.1:45048 127.0.0.1:5432 ESTABLISHED 8138/java
tcp6 0 0 127.0.0.1:45026 127.0.0.1:5432 ESTABLISHED 8138/java
tcp6 0 0 127.0.0.1:51962 127.0.0.1:5432 ESTABLISHED 693/java
tcp6 0 0 127.0.0.1:45802 127.0.0.1:5432 ESTABLISHED 693/java
tcp6 0 0 127.0.0.1:51988 127.0.0.1:5432 ESTABLISHED 693/java
tcp6 0 0 127.0.0.1:45028 127.0.0.1:5432 ESTABLISHED 8138/java
tcp6 0 0 127.0.0.1:46650 127.0.0.1:5432 ESTABLISHED 693/java
tcp6 0 0 127.0.0.1:44992 127.0.0.1:5432 ESTABLISHED 8138/java
tcp6 0 0 127.0.0.1:52004 127.0.0.1:5432 ESTABLISHED 693/java
tcp6 0 0 127.0.0.1:45024 127.0.0.1:5432 ESTABLISHED 8138/java
tcp6 0 0 127.0.0.1:44932 127.0.0.1:5432 ESTABLISHED 8138/java
tcp6 0 0 127.0.0.1:45912 127.0.0.1:5432 ESTABLISHED 693/java
tcp6 0 0 127.0.0.1:44944 127.0.0.1:5432 ESTABLISHED 8138/java

Jira service is running but not responding

We don't understand how an update could have caused this

[Sokom]

I can attach a complete catalina.out file if it's needed