Issues during updating JIRA from 8.2.1 to 8.5.1

Joshua Kuchenbäcker November 20, 2019

Hello,

i'm trying to upgrade our JIRA Datacenter (Node 1 & Node 2) from 8.2.1 to 8.5.1 but it wont come up after the Upgrade of Node 1.

 

The farm we run is build of 7 Servers (CentOS Linux 7). The Apache Proxy is using a valid, external signed, Certificate but the node use a self signed certificate for the communication between the Servers.

 

Server:

  • Apache Proxy Node 1
  • Jira Node 1
  • Jira Node 2
  • Confluence Node 1
  • Confluence Node 2
  • Database 1
  • File Share 1

 

Now i've put Jira into upgrade mod, stop jira with "systemctl stop jira", decompressed the *.tar.gz to our "/opt/jira" location and reconfigured the "server.xml", "setenv.sh", "jira-application.properties", "seraph-config.xml" and our sso plugin. The dbconfig.xml is unchanged.

After all that I started JIRA with "systemctl start jira" and opened the URL. I Only got the following error:

jira_server_fehler.png

The "/var/jira/log/atlassian-jira.log" is giving me this:

=========================================================

2019-11-20 10:16:14,878 localhost-startStop-1 INFO [c.a.jira.startup.JiraHomeStartupCheck] The jira.home directory '/var/jira' is validated and locked for exclusive use by this instance.
2019-11-20 10:16:14,949 localhost-startStop-1 INFO [c.a.jira.startup.JiraStartupLogger]

****************
JIRA starting...
****************

2019-11-20 10:16:15,209 localhost-startStop-1 INFO [c.a.jira.startup.JiraStartupLogger]

___ Environment _____________________________

JIRA Build : 8.5.1#805001-sha1:c5b54a7317f297e17ddad5bb52a1850cf2bdb4bc
Build Date : Mon Nov 04 00:00:00 CET 2019
JIRA Installation Type : Standalone
Application Server : Apache Tomcat/8.5.42 - Servlet API 3.1
Java Version : 1.8.0_212 - Oracle Corporation
Current Working Directory : /
Maximum Allowable Memory : 3925MB
Total Memory : 3925MB
Free Memory : 3879MB
Used Memory : 46MB
Memory Pool: Code Cache : Code Cache: init = 33554432(32768K) used = 10542528(10295K) committed = 33554432(32768K) max = 536870912(524288K)
Memory Pool: Metaspace : Metaspace: init = 0(0K) used = 21239888(20742K) committed = 22069248(21552K) max = -1(-1K)
Memory Pool: Compressed Class Space : Compressed Class Space: init = 0(0K) used = 2291280(2237K) committed = 2490368(2432K) max = 1073741824(1048576K)
Memory Pool: PS Eden Space : PS Eden Space: init = 1073741824(1048576K) used = 20186008(19712K) committed = 1073741824(1048576K) max = 1073741824(1048576K)
Memory Pool: PS Survivor Space : PS Survivor Space: init = 178782208(174592K) used = 0(0K) committed = 178782208(174592K) max = 178782208(174592K)
Memory Pool: PS Old Gen : PS Old Gen: init = 2863661056(2796544K) used = 27581824(26935K) committed = 2863661056(2796544K) max = 2863661056(2796544K)
JVM Input Arguments : -Djava.util.logging.config.file=/opt/jira/latest/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms4096m -Xmx4096m -XX:InitialCodeCacheSize=32m -XX:ReservedCodeCacheSize=512m -DjvmRoute=node1 -Djava.awt.headless=true -Datlassian.standalone=JIRA -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Dmail.mime.decodeparameters=true -Dorg.dom4j.factory=com.atlassian.core.xml.InterningDocumentFactory -XX:-OmitStackTraceInFastThrow -Djava.locale.providers=COMPAT -Datlassian.plugins.startup.options= -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xloggc:/opt/jira/latest/logs/atlassian-jira-gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=20M -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCCause -Dignore.endorsed.dirs= -Dcatalina.base=/opt/jira/latest -Dcatalina.home=/opt/jira/latest -Djava.io.tmpdir=/opt/jira/latest/temp
Java Compatibility Information : JIRA version = 8.5.1, Java Version = 1.8.0_212

___ Java System Properties _________________

atlassian.plugins.startup.options :
atlassian.standalone : JIRA
awt.toolkit : sun.awt.X11.XToolkit
catalina.base : /opt/jira/atlassian-jira-software-8.5.1-standalone
catalina.home : /opt/jira/atlassian-jira-software-8.5.1-standalone
catalina.useNaming : true
common.loader : "${catalina.base}/lib",
"${catalina.base}/lib/*.jar",
"${catalina.home}/lib",
"${catalina.home}/lib/*.jar"
file.encoding : UTF-8
file.encoding.pkg : sun.io
ignore.endorsed.dirs :
java.awt.graphicsenv : sun.awt.X11GraphicsEnvironment
java.awt.headless : true
java.awt.printerjob : sun.print.PSPrinterJob
java.class.version : 52.0
java.home : /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/jre
java.io.tmpdir : /opt/jira/latest/temp
java.locale.providers : COMPAT
java.naming.factory.initial : org.apache.naming.java.javaURLContextFactory
java.naming.factory.url.pkgs : org.apache.naming
java.protocol.handler.pkgs : org.apache.catalina.webresources
java.runtime.name : OpenJDK Runtime Environment
java.runtime.version : 1.8.0_212-b04
java.specification.name : Java Platform API Specification
java.specification.vendor : Oracle Corporation
java.specification.version : 1.8
java.util.concurrent.ForkJoinPool.common.threadFactory : org.apache.catalina.startup.SafeForkJoinWorkerThreadFactory
java.util.logging.config.file : /opt/jira/latest/conf/logging.properties
java.util.logging.manager : org.apache.juli.ClassLoaderLogManager
java.vendor : Oracle Corporation
java.vendor.url : http://java.oracle.com/
java.vendor.url.bug : http://bugreport.sun.com/bugreport/
java.version : 1.8.0_212
java.vm.info : mixed mode
java.vm.name : OpenJDK 64-Bit Server VM
java.vm.specification.name : Java Virtual Machine Specification
java.vm.specification.vendor : Oracle Corporation
java.vm.specification.version : 1.8
java.vm.vendor : Oracle Corporation
java.vm.version : 25.212-b04
jdk.tls.ephemeralDHKeySize : 2048
jvmRoute : rme_jira_node1
mail.mime.decodeparameters : true
org.apache.catalina.security.SecurityListener.UMASK : 0027
org.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER : true
org.dom4j.factory : com.atlassian.core.xml.InterningDocumentFactory
os.arch : amd64
os.name : Linux
os.version : 3.10.0-957.12.2.el7.x86_64
package.access : sun.,
org.apache.catalina.,
org.apache.coyote.,
org.apache.jasper.,
org.apache.tomcat.
package.definition : sun.,
java.,
org.apache.catalina.,
org.apache.coyote.,
org.apache.jasper.,
org.apache.naming.,
org.apache.tomcat.
server.loader :
shared.loader :
sun.arch.data.model : 64
sun.boot.library.path : /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/jre/lib/amd64
sun.cpu.endian : little
sun.cpu.isalist :
sun.io.unicode.encoding : UnicodeLittle
sun.java.command : org.apache.catalina.startup.Bootstrap start
sun.java.launcher : SUN_STANDARD
sun.jnu.encoding : UTF-8
sun.management.compiler : HotSpot 64-Bit Tiered Compilers
sun.os.patch.level : unknown
tomcat.util.buf.StringCache.byte.enabled : true
tomcat.util.scan.StandardJarScanFilter.jarsToScan : log4j-taglib*.jar,
log4j-web*.jar,
log4javascript*.jar,
slf4j-taglib*.jar
tomcat.util.scan.StandardJarScanFilter.jarsToSkip : annotations-api.jar,
ant-junit*.jar,
ant-launcher.jar,
ant.jar,
asm-*.jar,
aspectj*.jar,
bootstrap.jar,
catalina-ant.jar,
catalina-ha.jar,
catalina-jmx-remote.jar,
catalina-storeconfig.jar,
catalina-tribes.jar,
catalina-ws.jar,
catalina.jar,
cglib-*.jar,
cobertura-*.jar,
commons-beanutils*.jar,
commons-codec*.jar,
commons-collections*.jar,
commons-daemon.jar,
commons-dbcp*.jar,
commons-digester*.jar,
commons-fileupload*.jar,
commons-httpclient*.jar,
commons-io*.jar,
commons-lang*.jar,
commons-logging*.jar,
commons-math*.jar,
commons-pool*.jar,
dom4j-*.jar,
easymock-*.jar,
ecj-*.jar,
el-api.jar,
geronimo-spec-jaxrpc*.jar,
h2*.jar,
hamcrest-*.jar,
hibernate*.jar,
httpclient*.jar,
icu4j-*.jar,
jasper-el.jar,
jasper.jar,
jaspic-api.jar,
jaxb-*.jar,
jaxen-*.jar,
jdom-*.jar,
jetty-*.jar,
jmx-tools.jar,
jmx.jar,
jsp-api.jar,
jstl.jar,
jta*.jar,
junit-*.jar,
junit.jar,
log4j*.jar,
mail*.jar,
objenesis-*.jar,
oraclepki.jar,
oro-*.jar,
servlet-api-*.jar,
servlet-api.jar,
slf4j*.jar,
taglibs-standard-spec-*.jar,
tagsoup-*.jar,
tomcat-api.jar,
tomcat-coyote.jar,
tomcat-dbcp.jar,
tomcat-i18n-*.jar,
tomcat-jdbc.jar,
tomcat-jni.jar,
tomcat-juli-adapters.jar,
tomcat-juli.jar,
tomcat-util-scan.jar,
tomcat-util.jar,
tomcat-websocket.jar,
tools.jar,
websocket-api.jar,
wsdl4j*.jar,
xercesImpl.jar,
xml-apis.jar,
xmlParserAPIs-*.jar,
xmlParserAPIs.jar,
xom-*.jar
user.country : US
user.dir : /
user.home : /home/jira
user.language : en
user.name : jira
user.timezone : Europe/Berlin

2019-11-20 10:16:15,480 localhost-startStop-1 INFO [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from /var/jira/dbconfig.xml
2019-11-20 10:16:15,543 localhost-startStop-1 INFO [c.a.j.config.database.DatabaseConfigHandler] Trying to get encrypted password from xml and decrypt it
2019-11-20 10:16:15,545 localhost-startStop-1 INFO [c.a.j.config.database.DatabaseConfigHandler] Database password decryption not performed.
2019-11-20 10:16:15,844 localhost-startStop-1 ERROR [c.a.config.bootstrap.DefaultAtlassianBootstrapManager] Could not successfully test your database:
org.postgresql.util.PSQLException: Could not open SSL root certificate file /home/jira/.postgresql/root.crt.
at org.postgresql.ssl.LibPQFactory.<init>(LibPQFactory.java:120)
at org.postgresql.core.SocketFactoryFactory.getSslSocketFactory(SocketFactoryFactory.java:61)
at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:33)
at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:441)
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:94)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:458)
at org.postgresql.Driver.connect(Driver.java:260)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at com.atlassian.config.bootstrap.DefaultAtlassianBootstrapManager.getTestDatabaseConnection(DefaultAtlassianBootstrapManager.java:347)
at com.atlassian.jira.config.database.JdbcDatasource.getConnection(JdbcDatasource.java:219)
at com.atlassian.jira.config.database.DatabaseConfig.testConnection(DatabaseConfig.java:88)
at com.atlassian.jira.health.checks.DbConfigurationAndConnectionCheck.doPerform(DbConfigurationAndConnectionCheck.java:60)
at com.atlassian.jira.health.HealthCheckTemplate.perform(HealthCheckTemplate.java:23)
at com.atlassian.jira.health.DefaultHealthCheckExecutor.runCheck(DefaultHealthCheckExecutor.java:74)
at com.atlassian.jira.health.DefaultHealthCheckExecutor.lambda$applyAndCollectExceptions$1(DefaultHealthCheckExecutor.java:53)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at com.atlassian.jira.health.DefaultHealthCheckExecutor.applyAndCollectExceptions(DefaultHealthCheckExecutor.java:53)
at com.atlassian.jira.health.DefaultHealthCheckExecutor.performHealthChecks(DefaultHealthCheckExecutor.java:42)
at com.atlassian.jira.health.HealthChecks.executeChecksAndRecordResults(HealthChecks.java:164)
at com.atlassian.jira.health.HealthChecks.runHealthChecks(HealthChecks.java:154)
at com.atlassian.jira.health.HealthChecks.runHealthChecks(HealthChecks.java:66)
at com.atlassian.jira.startup.BootstrapContainerLauncher.start(BootstrapContainerLauncher.java:48)
at com.atlassian.jira.startup.DefaultJiraLauncher.preDbLaunch(DefaultJiraLauncher.java:117)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:103)
at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:102)
at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:154)
at com.atlassian.jira.startup.LauncherContextListener.initSlowStuffInBackground(LauncherContextListener.java:139)
at com.atlassian.jira.startup.LauncherContextListener.contextInitialized(LauncherContextListener.java:101)
... 5 filtered
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: /home/jira/.postgresql/root.crt (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at org.postgresql.ssl.LibPQFactory.<init>(LibPQFactory.java:117)
2019-11-20 10:16:15,887 localhost-startStop-1 ERROR [c.a.jira.health.HealthChecks] JIRA couldn't connect to your database
2019-11-20 10:16:15,887 localhost-startStop-1 ERROR [c.a.jira.health.HealthChecks] JIRA failed to establish a connection to your database.
This could be because:
- Your database isn't running
- The configuration of your dbconfig.xml file is incorrect (user, password, or database URL etc.)
- There is a network issue between JIRA and your database (e.g. firewall, database doesn't allow remote access etc.)

There are several other solutions you can try, review our documentation and see what works for you.

2019-11-20 10:16:15,891 localhost-startStop-1 INFO [c.a.jira.startup.JiraStartupLogger] Running JIRA startup checks.
2019-11-20 10:16:15,892 localhost-startStop-1 FATAL [c.a.jira.startup.JiraStartupLogger] Startup check failed. JIRA will be locked.
2019-11-20 10:16:15,966 localhost-startStop-1 INFO [c.a.jira.startup.LauncherContextListener] Memory Usage:
---------------------------------------------------------------------------------
Heap memory : Used: 138 MiB. Committed: 3926 MiB. Max: 3926 MiB
Non-heap memory : Used: 39 MiB. Committed: 62 MiB. Max: 1536 MiB
---------------------------------------------------------------------------------
TOTAL : Used: 177 MiB. Committed: 3987 MiB. Max: 5462 MiB
---------------------------------------------------------------------------------

=========================================================

 

This is our "dbconfig.xml"

=========================================================

<?xml version="1.0" encoding="UTF-8"?>

<jira-database-config>
<name>defaultDS</name>
<delegator-name>default</delegator-name>
<database-type>postgres72</database-type>
<schema-name>public</schema-name>
<jdbc-datasource>
<url>jdbc:postgresql://database.de:5432/jira_db?ssl=true</url>
<driver-class>org.postgresql.Driver</driver-class>
<username>jira</username>
<password>password</password>
<pool-min-size>20</pool-min-size>
<pool-max-size>20</pool-max-size>
<pool-max-wait>30000</pool-max-wait>
<validation-query>select 1</validation-query>
<min-evictable-idle-time-millis>60000</min-evictable-idle-time-millis>
<time-between-eviction-runs-millis>300000</time-between-eviction-runs-millis>
<pool-max-idle>20</pool-max-idle>
<pool-remove-abandoned>true</pool-remove-abandoned>
<pool-remove-abandoned-timeout>300</pool-remove-abandoned-timeout>
<pool-test-on-borrow>false</pool-test-on-borrow>
<pool-test-while-idle>true</pool-test-while-idle>
</jdbc-datasource>
</jira-database-config>

=========================================================

 

Why is the there now a "org.postgresql.util.PSQLException" and how can i solve this issue?

1 answer

0 votes
Petr Vaníček
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
November 20, 2019

Hi,

for you is important this line

Caused by: java.io.FileNotFoundException: /home/jira/.postgresql/root.crt (No such file or directory)

So probably there is no certificate or Jira user doesn't have access to this file.

Joshua Kuchenbäcker November 20, 2019

Hello Petr,


yes, this file don't exist, but this wasn't a problem before the update. As a workaround i've tried this:

============

sudo -s

cp /root/cert.crt /home/jira/.postgresql/root.crt

chown jira:jira /home/jira/.postgresql/root.crt

============


Now i have this error:

image2019-11-19_12-30-47.png

 

I don't understand what is changed in 8.5.1 that i can't connect to the database anymore.

Petr Vaníček
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
November 20, 2019

So you are using probably self-signed cert. If it's your case then you need import cert into keystore.

Do you need SSL for DB connection (like it's not in same network)? 

Vickey Palzor Lepcha
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
November 20, 2019

So your Database is enabled to accept SSL connections right ?

Joshua Kuchenbäcker November 20, 2019

The certificate is imported to the system keystore and java keystore.

And yes, the system worked until the update and I changed nothing in the configurations. 

Suggest an answer

Log in or Sign up to answer