Startup check failed. Jira will be locked

[geoffbland]

I am running Jira 8.20.3 running on Ubuntu Server 20.04 against a MySQL database, OpenJDK Java 11. All was working fine until the server restarted last night, from the logs a small Ubuntu patch was applied last night and the server rebooted. This patch is probably unrelated but after the reboot the server will locks on start up.

I am getting the "Startup check failed. Jira will be locked" error:

2022-01-20 18:49:11,669+0000 JIRA-Bootstrap FATAL [c.a.jira.startup.JiraStartupLogger] Startup check failed. Jira will be locked.

I have check no other instance is running. I have tried removing the /var/atlassian/application-data/jira/.jira-home.lock file but JIRA server still will not start up.

No other errors or warnings are shown in the catalina.log. What should I look at next to resolve this?

Full start-up catalina.log:

NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
20-Jan-2022 18:49:07.205 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/8.5.65
20-Jan-2022 18:49:07.208 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Mar 30 2021 12:28:40 UTC
20-Jan-2022 18:49:07.208 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 8.5.65.0
20-Jan-2022 18:49:07.208 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
20-Jan-2022 18:49:07.208 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 5.4.0-96-generic
20-Jan-2022 18:49:07.210 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
20-Jan-2022 18:49:07.210 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /opt/atlassian/jira/jre
20-Jan-2022 18:49:07.210 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 11.0.11+9
20-Jan-2022 18:49:07.210 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: AdoptOpenJDK
20-Jan-2022 18:49:07.210 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /opt/atlassian/jira
20-Jan-2022 18:49:07.210 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /opt/atlassian/jira
20-Jan-2022 18:49:07.211 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
20-Jan-2022 18:49:07.211 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
20-Jan-2022 18:49:07.211 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util=ALL-UNNAMED
20-Jan-2022 18:49:07.211 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
20-Jan-2022 18:49:07.212 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
20-Jan-2022 18:49:07.212 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/opt/atlassian/jira/conf/logging.properties
20-Jan-2022 18:49:07.212 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
20-Jan-2022 18:49:07.212 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms384m
20-Jan-2022 18:49:07.212 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx2048m
20-Jan-2022 18:49:07.212 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:InitialCodeCacheSize=32m
20-Jan-2022 18:49:07.212 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:ReservedCodeCacheSize=512m
20-Jan-2022 18:49:07.212 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.awt.headless=true
20-Jan-2022 18:49:07.212 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Datlassian.standalone=JIRA
20-Jan-2022 18:49:07.212 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true
20-Jan-2022 18:49:07.212 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dmail.mime.decodeparameters=true
20-Jan-2022 18:49:07.212 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.dom4j.factory=com.atlassian.core.xml.InterningDocumentFactory
20-Jan-2022 18:49:07.212 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:-OmitStackTraceInFastThrow
20-Jan-2022 18:49:07.212 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.locale.providers=COMPAT
20-Jan-2022 18:49:07.212 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Datlassian.plugins.startup.options=
20-Jan-2022 18:49:07.213 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
20-Jan-2022 18:49:07.213 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
20-Jan-2022 18:49:07.213 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
20-Jan-2022 18:49:07.213 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xlog:gc*:file=/opt/atlassian/jira/logs/atlassian-jira-gc-%t.log:tags,time,uptime,level:filecount=5,filesize=20M
20-Jan-2022 18:49:07.213 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+ExplicitGCInvokesConcurrent
20-Jan-2022 18:49:07.215 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
20-Jan-2022 18:49:07.216 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/opt/atlassian/jira
20-Jan-2022 18:49:07.216 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/opt/atlassian/jira
20-Jan-2022 18:49:07.217 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/opt/atlassian/jira/temp
20-Jan-2022 18:49:07.217 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]
20-Jan-2022 18:49:07.491 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
20-Jan-2022 18:49:07.564 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 964 ms
20-Jan-2022 18:49:07.602 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
20-Jan-2022 18:49:07.602 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/8.5.65]
2022-01-20 18:49:11,003+0000 localhost-startStop-1 INFO [c.a.jira.startup.JiraHomeStartupCheck] The jira.home directory '/var/atlassian/application-data/jira' is validated and locked for exclusive use by this instance.
2022-01-20 18:49:11,097+0000 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]

****************
Jira starting...
****************

2022-01-20 18:49:11,197+0000 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]

___ Environment _____________________________

JIRA Build : 8.20.3#820003-sha1:e9c1f3b400a5545d56ec3d8e0414de7a0e75ddb3
Build Date : Thu Dec 16 00:00:00 UTC 2021
JIRA Installation Type : Standalone
Application Server : Apache Tomcat/8.5.65 - Servlet API 3.1
Java Version : 11.0.11 - AdoptOpenJDK
Current Working Directory : /
Maximum Allowable Memory : 2048MB
Total Memory : 384MB
Free Memory : 140MB
Used Memory : 244MB
Memory Pool: CodeHeap 'non-nmethods' : CodeHeap 'non-nmethods': init = 5832704(5696K) used = 1173888(1146K) committed = 5832704(5696K) max = 5832704(5696K)
Memory Pool: Metaspace : Metaspace: init = 0(0K) used = 28642792(27971K) committed = 29671424(28976K) max = -1(-1K)
Memory Pool: CodeHeap 'profiled nmethods' : CodeHeap 'profiled nmethods': init = 33554432(32768K) used = 9038336(8826K) committed = 33554432(32768K) max = 265519104(259296K)
Memory Pool: Compressed Class Space : Compressed Class Space: init = 0(0K) used = 3143616(3069K) committed = 3538944(3456K) max = 1073741824(1048576K)
Memory Pool: G1 Eden Space : G1 Eden Space: init = 20971520(20480K) used = 217055232(211968K) committed = 238026752(232448K) max = -1(-1K)
Memory Pool: G1 Old Gen : G1 Old Gen: init = 381681664(372736K) used = 23196928(22653K) committed = 148897792(145408K) max = 2147483648(2097152K)
Memory Pool: G1 Survivor Space : G1 Survivor Space: init = 0(0K) used = 15728640(15360K) committed = 15728640(15360K) max = -1(-1K)
Memory Pool: CodeHeap 'non-profiled nmethods' : CodeHeap 'non-profiled nmethods': init = 33554432(32768K) used = 2831104(2764K) committed = 33554432(32768K) max = 265519104(259296K)
JVM Input Arguments : --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED -Djava.util.logging.config.file=/opt/atlassian/jira/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms384m -Xmx2048m -XX:InitialCodeCacheSize=32m -XX:ReservedCodeCacheSize=512m -Djava.awt.headless=true -Datlassian.standalone=JIRA -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Dmail.mime.decodeparameters=true -Dorg.dom4j.factory=com.atlassian.core.xml.InterningDocumentFactory -XX:-OmitStackTraceInFastThrow -Djava.locale.providers=COMPAT -Datlassian.plugins.startup.options= -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xlog:gc*:file=/opt/atlassian/jira/logs/atlassian-jira-gc-%t.log:tags,time,uptime,level:filecount=5,filesize=20M -XX:+ExplicitGCInvokesConcurrent -Dignore.endorsed.dirs= -Dcatalina.base=/opt/atlassian/jira -Dcatalina.home=/opt/atlassian/jira -Djava.io.tmpdir=/opt/atlassian/jira/temp
Java Compatibility Information : JIRA version = 8.20.3, Java Version = 11.0.11

___ Java System Properties _________________

atlassian.plugins.startup.options :
atlassian.standalone : JIRA
awt.toolkit : sun.awt.X11.XToolkit
catalina.base : /opt/atlassian/jira
catalina.home : /opt/atlassian/jira
catalina.useNaming : true
common.loader : "${catalina.base}/lib",
"${catalina.base}/lib/*.jar",
"${catalina.home}/lib",
"${catalina.home}/lib/*.jar"
file.encoding : UTF-8
ignore.endorsed.dirs :
java.awt.graphicsenv : sun.awt.X11GraphicsEnvironment
java.awt.headless : true
java.awt.printerjob : sun.print.PSPrinterJob
java.class.version : 55.0
java.home : /opt/atlassian/jira/jre
java.io.tmpdir : /opt/atlassian/jira/temp
java.locale.providers : COMPAT
java.naming.factory.initial : org.apache.naming.java.javaURLContextFactory
java.naming.factory.url.pkgs : org.apache.naming
java.protocol.handler.pkgs : org.apache.catalina.webresources
java.runtime.name : OpenJDK Runtime Environment
java.runtime.version : 11.0.11+9
java.specification.name : Java Platform API Specification
java.specification.vendor : Oracle Corporation
java.specification.version : 11
java.util.logging.config.file : /opt/atlassian/jira/conf/logging.properties
java.util.logging.manager : org.apache.juli.ClassLoaderLogManager
java.vendor : AdoptOpenJDK
java.vendor.url : https://adoptopenjdk.net/
java.vendor.url.bug : https://github.com/AdoptOpenJDK/openjdk-support/issues
java.vendor.version : AdoptOpenJDK-11.0.11+9
java.version : 11.0.11
java.version.date : 2021-04-20
java.vm.compressedOopsMode : 32-bit
java.vm.info : mixed mode
java.vm.name : OpenJDK 64-Bit Server VM
java.vm.specification.name : Java Virtual Machine Specification
java.vm.specification.vendor : Oracle Corporation
java.vm.specification.version : 11
java.vm.vendor : AdoptOpenJDK
java.vm.version : 11.0.11+9
jdk.debug : release
jdk.tls.ephemeralDHKeySize : 2048
mail.mime.decodeparameters : true
org.apache.catalina.security.SecurityListener.UMASK : 0027
org.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER : true
org.dom4j.factory : com.atlassian.core.xml.InterningDocumentFactory
os.arch : amd64
os.name : Linux
os.version : 5.4.0-96-generic
package.access : sun.,
org.apache.catalina.,
org.apache.coyote.,
org.apache.jasper.,
org.apache.tomcat.
package.definition : sun.,
java.,
org.apache.catalina.,
org.apache.coyote.,
org.apache.jasper.,
org.apache.naming.,
org.apache.tomcat.
server.loader :
shared.loader :
sun.arch.data.model : 64
sun.boot.library.path : /opt/atlassian/jira/jre/lib
sun.cpu.endian : little
sun.cpu.isalist :
sun.io.unicode.encoding : UnicodeLittle
sun.java.command : org.apache.catalina.startup.Bootstrap start
sun.java.launcher : SUN_STANDARD
sun.jnu.encoding : UTF-8
sun.management.compiler : HotSpot 64-Bit Tiered Compilers
sun.os.patch.level : unknown
tomcat.util.buf.StringCache.byte.enabled : true
tomcat.util.scan.StandardJarScanFilter.jarsToScan : log4j-taglib*.jar,
log4j-web*.jar,
log4javascript*.jar,
slf4j-taglib*.jar
tomcat.util.scan.StandardJarScanFilter.jarsToSkip : annotations-api.jar,
ant-junit*.jar,
ant-launcher.jar,
ant.jar,
asm-*.jar,
aspectj*.jar,
bootstrap.jar,
catalina-ant.jar,
catalina-ha.jar,
catalina-jmx-remote.jar,
catalina-storeconfig.jar,
catalina-tribes.jar,
catalina-ws.jar,
catalina.jar,
cglib-*.jar,
cobertura-*.jar,
commons-beanutils*.jar,
commons-codec*.jar,
commons-collections*.jar,
commons-daemon.jar,
commons-dbcp*.jar,
commons-digester*.jar,
commons-fileupload*.jar,
commons-httpclient*.jar,
commons-io*.jar,
commons-lang*.jar,
commons-logging*.jar,
commons-math*.jar,
commons-pool*.jar,
dom4j-*.jar,
easymock-*.jar,
ecj-*.jar,
el-api.jar,
geronimo-spec-jaxrpc*.jar,
h2*.jar,
hamcrest-*.jar,
hibernate*.jar,
httpclient*.jar,
icu4j-*.jar,
jasper-el.jar,
jasper.jar,
jaspic-api.jar,
jaxb-*.jar,
jaxen-*.jar,
jdom-*.jar,
jetty-*.jar,
jmx-tools.jar,
jmx.jar,
jsp-api.jar,
jstl.jar,
jta*.jar,
junit-*.jar,
junit.jar,
log4j*.jar,
mail*.jar,
objenesis-*.jar,
oraclepki.jar,
oro-*.jar,
servlet-api-*.jar,
servlet-api.jar,
slf4j*.jar,
taglibs-standard-spec-*.jar,
tagsoup-*.jar,
tomcat-api.jar,
tomcat-coyote.jar,
tomcat-dbcp.jar,
tomcat-i18n-*.jar,
tomcat-jdbc.jar,
tomcat-jni.jar,
tomcat-juli-adapters.jar,
tomcat-juli.jar,
tomcat-util-scan.jar,
tomcat-util.jar,
tomcat-websocket.jar,
tools.jar,
websocket-api.jar,
wsdl4j*.jar,
xercesImpl.jar,
xml-apis.jar,
xmlParserAPIs-*.jar,
xmlParserAPIs.jar,
xom-*.jar
user.country : GB
user.dir : /
user.home : /home/jira
user.language : en
user.name : jira
user.timezone : Etc/UTC

20-Jan-2022 18:49:11.252 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
20-Jan-2022 18:49:11.260 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
20-Jan-2022 18:49:11.273 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 3707 ms
2022-01-20 18:49:11,351+0000 JIRA-Bootstrap INFO [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from /var/atlassian/application-data/jira/dbconfig.xml
2022-01-20 18:49:11,376+0000 JIRA-Bootstrap INFO [c.a.j.config.database.DatabaseConfigHandler] Trying to get encrypted password from xml and decrypt it
2022-01-20 18:49:11,377+0000 JIRA-Bootstrap INFO [c.a.j.config.database.DatabaseConfigHandler] Database password decryption not performed.
2022-01-20 18:49:11,668+0000 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger] Running Jira startup checks.
2022-01-20 18:49:11,669+0000 JIRA-Bootstrap FATAL [c.a.jira.startup.JiraStartupLogger] Startup check failed. Jira will be locked.
2022-01-20 18:49:11,715+0000 JIRA-Bootstrap INFO [c.a.jira.startup.LauncherContextListener] Startup is complete. Jira is ready to serve.
2022-01-20 18:49:11,717+0000 JIRA-Bootstrap INFO [c.a.jira.startup.LauncherContextListener] Memory Usage:
---------------------------------------------------------------------------------
Heap memory : Used: 55 MiB. Committed: 384 MiB. Max: 2048 MiB
Non-heap memory : Used: 53 MiB. Committed: 110 MiB. Max: 1536 MiB
---------------------------------------------------------------------------------
TOTAL : Used: 107 MiB. Committed: 494 MiB. Max: 3584 MiB
---------------------------------------------------------------------------------

[geoffbland]

Answering my own issue here.

The issue was that the MySQL database jira user had lost access rights to the jira database. Re-assigning access rights and restarting the jira service fixed the issue.

Why are there no warnings in the logs that the problem is that the database is not accessible?

[Paul Pelland]

I have same issue.  Where did you get access rights info?  (thanks)