JIRA Fails to start - We haven't been able to start all the required system plugins

Snezana Bozovic May 28, 2019

As of yesterday, we have an issue with our self hosted instance of JIRA. It fails to start, with well known message "System plugins failed to start: We haven't been able to start all the required system plugins"

So far we have tried, checking in the pluginenabled if the plugins are disabled, increasing max memory for JIRA, increasing the plugin start timeout to 300s, moving instance to an SSD backed KVM server, the issue still remains.

here is the (almost) complete log of startup:

28-May-2019 13:01:17.197 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version: Apache Tomcat/8.5.35
28-May-2019 13:01:17.200 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Nov 3 2018 17:39:20 UTC
28-May-2019 13:01:17.201 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number: 8.5.35.0
28-May-2019 13:01:17.201 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
28-May-2019 13:01:17.201 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 4.4.0-146-generic
28-May-2019 13:01:17.201 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
28-May-2019 13:01:17.205 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /opt/atlassian/jira/jre
28-May-2019 13:01:17.205 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 1.8.0_181-b13
28-May-2019 13:01:17.206 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Oracle Corporation
28-May-2019 13:01:17.206 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /opt/atlassian/jira
28-May-2019 13:01:17.206 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /opt/atlassian/jira
28-May-2019 13:01:17.206 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/opt/atlassian/jira/conf/logging.properties
28-May-2019 13:01:17.206 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
28-May-2019 13:01:17.207 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms384m
28-May-2019 13:01:17.207 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx1024m
28-May-2019 13:01:17.207 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:InitialCodeCacheSize=32m
28-May-2019 13:01:17.207 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:ReservedCodeCacheSize=512m


28-May-2019 13:01:17.210 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Datlassian.plugins.enable.wait=300

28-May-2019 13:01:17.213 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
28-May-2019 13:01:17.213 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/opt/atlassian/jira
28-May-2019 13:01:17.213 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/opt/atlassian/jira
28-May-2019 13:01:17.213 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/opt/atlassian/jira/temp
28-May-2019 13:01:17.213 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]
28-May-2019 13:01:17.471 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
28-May-2019 13:01:17.483 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 861 ms
28-May-2019 13:01:17.497 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
28-May-2019 13:01:17.497 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.35
2019-05-28 13:01:24,316 localhost-startStop-1 INFO [c.a.jira.startup.JiraHomeStartupCheck] The jira.home directory '/var/atlassian/application-data/jira' is validated and locked for exclusive use by this instance.
2019-05-28 13:01:24,374 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]

****************
JIRA starting...
****************

2019-05-28 13:01:24,441 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]

___ Environment _____________________________

JIRA Build : 8.1.0#801000-sha1:2e1cd1bb771978cda2c5e8f3f10539ab180613f6
Build Date : Thu Apr 04 00:00:00 CEST 2019
JIRA Installation Type : Standalone
Application Server : Apache Tomcat/8.5.35 - Servlet API 3.1
Java Version : 1.8.0_181 - Oracle Corporation
Current Working Directory : /opt/atlassian/jira/bin
Maximum Allowable Memory : 910MB
Total Memory : 551MB
Free Memory : 484MB
Used Memory : 67MB

___ Java System Properties _________________

atlassian.plugins.enable.wait : 300
atlassian.plugins.startup.options :
atlassian.standalone : JIRA
awt.toolkit : sun.awt.X11.XToolkit
catalina.base : /opt/atlassian/jira
catalina.home : /opt/atlassian/jira
catalina.useNaming : true

2019-05-28 13:01:24,591 JIRA-Bootstrap INFO [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from /var/atlassian/application-data/jira/dbconfig.xml
2019-05-28 13:01:24,795 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger] Running JIRA startup checks.
2019-05-28 13:01:24,795 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger] JIRA pre-database startup checks completed successfully.
2019-05-28 13:01:24,977 JIRA-Bootstrap INFO [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from /var/atlassian/application-data/jira/dbconfig.xml
2019-05-28 13:01:31,019 JIRA-Bootstrap INFO [c.a.j.config.database.DatabaseConfigurationManagerImpl] The database is configured. Now running Database Checklist Launcher
2019-05-28 13:01:31,022 JIRA-Bootstrap INFO [c.a.jira.startup.DatabaseChecklistLauncher] JIRA database startup checks completed successfully.
2019-05-28 13:01:31,022 JIRA-Bootstrap INFO [c.a.j.config.database.DatabaseConfigurationManagerImpl] The database is configured. Now running Post database-configuration launchers
2019-05-28 13:01:31,994 JIRA-Bootstrap INFO [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from /var/atlassian/application-data/jira/dbconfig.xml
2019-05-28 13:01:34,213 JIRA-Bootstrap INFO [c.a.jira.startup.DatabaseLauncher] Database transactions enabled: true
2019-05-28 13:01:34,213 JIRA-Bootstrap INFO [c.a.jira.startup.DatabaseLauncher] Using JIRA's default for database transaction isolation level: 2
2019-05-28 13:01:34,221 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]

Database configuration OK

___ Database Configuration _________________

Loading entityengine.xml from : file:/opt/atlassian/jira/atlassian-jira/WEB-INF/classes/entityengine.xml
Entity model field type name : postgres72
Entity model schema name : public
Database Version : PostgreSQL - 9.5.16
Database Driver : PostgreSQL Native Driver - PostgreSQL 9.4.1212
Database URL : jdbc:postgresql://localhost:5432/jiradb
Database JDBC config : postgres72 jdbc:postgresql://localhost:5432/jiradb

2019-05-28 13:01:34,280 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]

___ Starting the JIRA Plugin System _________________

28-May-2019 13:01:44.877 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
28-May-2019 13:01:44.892 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
28-May-2019 13:01:44.955 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 26399 ms
2019-05-28 13:06:46,211 JIRA-Bootstrap ERROR [c.a.jira.startup.SystemPluginsEnabledHealthCheck] We couldn't start the following plugins:
- Gadget Directory Plugin (com.atlassian.gadgets.directory)
- Atlassian Jira - Plugins - Gadgets Plugin (com.atlassian.jira.gadgets)
- Atlassian Jira - Plugins - Global Issue Navigator (com.atlassian.jira.jira-issue-nav-plugin)
- Atlassian JIRA - Plugins - Quick Edit Plugin (com.atlassian.jira.jira-quick-edit-plugin)
- Atlassian Jira - Plugins - REST Plugin (com.atlassian.jira.rest)

2019-05-28 13:06:46,643 JIRA-Bootstrap INFO [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from /var/atlassian/application-data/jira/dbconfig.xml
2019-05-28 13:06:47,034 JIRA-Bootstrap ERROR [c.a.jira.startup.DefaultJiraLauncher] JIRA has failed to start because of the following errors: [(Event: Level = (EventLevel: fatal) , Key = (EventType: system-plugins) , Desc = We haven't been able to start all the required system plugins , Exception = We couldn't start the following plugins:<br/><ul><li>Gadget Directory Plugin (com.atlassian.gadgets.directory)</li><li>Atlassian Jira - Plugins - Gadgets Plugin (com.atlassian.jira.gadgets)</li><li>Atlassian Jira - Plugins - Global Issue Navigator (com.atlassian.jira.jira-issue-nav-plugin)</li><li>Atlassian JIRA - Plugins - Quick Edit Plugin (com.atlassian.jira.jira-quick-edit-plugin)</li><li>Atlassian Jira - Plugins - REST Plugin (com.atlassian.jira.rest)</li></ul>)]
2019-05-28 13:06:47,038 JIRA-Bootstrap INFO [c.a.jira.startup.LauncherContextListener] Memory Usage:
---------------------------------------------------------------------------------
Heap memory : Used: 280 MiB. Committed: 770 MiB. Max: 993 MiB
Non-heap memory : Used: 210 MiB. Committed: 230 MiB. Max: 1536 MiB
---------------------------------------------------------------------------------
TOTAL : Used: 490 MiB. Committed: 1000 MiB. Max: 2529 MiB
---------------------------------------------------------------------------------

 

4 answers

0 votes
Michael Thompson
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
May 28, 2019

I just went back through my notes on when I deployed Jira 8 to our test environment. Initially I got the same error as you did. Solution was actually kind of easy.

  1. Start Jira in safe mode, all apps turned off.
  2. Confirm that the Jira UI is accessible.
  3. Stop Jira, then restart it as you normally would.

This got Jira up and running and i was able to update all apps. I did need to go in to the System Apps section and turn on several apps that were in a disabled state, but they all started without any issue.

There seems to be something wrong with starting in standard mode right after the upgrade. When I deployed to my production environment I specifically started in safe mode first, then stopped and restarted. By doing that I did not experience any problems with apps.

Şükrü TURGUT August 28, 2020

Hello Micheal ,

How to start JIRA in safe mode? Could you tell please?

 

Thank You

Michael Thompson
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
August 31, 2020

Hello Sukru,

While logged in to your Jira user as the account that owns the app, move into the working directory (/opt/atlassian/jira/bin) and enter the command:

./start-jira.sh --disable-all-addons
Like Şükrü TURGUT likes this
Şükrü TURGUT September 1, 2020

Hello Micheal ,

I am new getting started with atlassian-plugin-sdk tutorial.

I am following the instructions in  https://developer.atlassian.com/server/framework/atlassian-sdk/create-a-helloworld-plugin-project/

in cmd window 

atlas-run 

jira server is not started because of the plugins not enabled.

there is no jira software installed extra. 

So there is no .sh file in my pc.

 

Is there another way ?

Thank You.

Michael Thompson
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
September 1, 2020

I'm sorry, I'm not at all familiar with the SDK, and I don't even have an atlas-run script in the directory structure. Hopefully someone else will be able to assist you. 

Good luck!

Predrag Tasevski May 11, 2021

 @Şükrü TURGUT any workaround? I'm also having the same issue.

Predrag Tasevski May 16, 2021

Hi,

I'm having the same problem following the same tutorial: https://developer.atlassian.com/server/framework/atlassian-sdk/create-a-helloworld-plugin-project/

 

Here is my output from JIRA:

System plugins failed to start: We haven't been able to start all the required system plugins

We couldn't start the following plugins:

  • Jira Projects Plugin (com.atlassian.jira.jira-projects-plugin)
  • Atlassian OAuth Consumer Plugin (com.atlassian.oauth.consumer)
  • Atlassian Navigation Links Plugin (com.atlassian.plugins.atlassian-nav-links-plugin)
  • Atlassian Jira - Plugins - OAuth Service Provider SPI (com.atlassian.jira.oauth.serviceprovider)
  • Atlassian Jira - Plugins - Project Config Plugin (com.atlassian.jira.jira-project-config-plugin)
  • Applinks - Plugin - Core (com.atlassian.applinks.applinks-plugin)
  • Atlassian OAuth Service Provider Plugin (com.atlassian.oauth.serviceprovider)
  • Embedded Gadgets Plugin (com.atlassian.gadgets.embedded)
  • Gadget Directory Plugin (com.atlassian.gadgets.directory)
  • Atlassian Plugins - JavaScript libraries (com.atlassian.plugin.jslibs)
  • Atlassian Jira - Plugins - Gadgets Plugin (com.atlassian.jira.gadgets)
  • Atlassian Jira - Plugins - Global Issue Navigator (com.atlassian.jira.jira-issue-nav-plugin)
  • Atlassian JIRA - Plugins - File viewer plugin (com.atlassian.jira.jira-fileviewer-plugin)
  • Atlassian Pretty URLs Plugin (com.atlassian.prettyurls.atlassian-pretty-urls-plugin)
  • Atlassian JIRA - Plugins - Quick Edit Plugin (com.atlassian.jira.jira-quick-edit-plugin)
  • Atlassian OAuth Admin Plugin (com.atlassian.oauth.admin)
  • Atlassian REST - Module Types (com.atlassian.plugins.rest.atlassian-rest-module)
  • Atlassian Jira - Plugins - REST Plugin (com.atlassian.jira.rest)
  • Opensocial Plugin (com.atlassian.gadgets.opensocial)
  • atlassian-servlet-plugin (com.atlassian.web.atlassian-servlet-plugin)
  • Atlassian Jira - Plugins - Look And Feel Logo Upload Plugin (com.atlassian.jira.lookandfeel)
  • Gadget Spec Publisher Plugin (com.atlassian.gadgets.publisher)
  • Atlassian Jira - Plugins - OAuth Consumer SPI (com.atlassian.jira.oauth.consumer)
  • Atlassian Jira - Plugins - SAL Plugin (com.atlassian.sal.jira)
  • Gadget Dashboard Plugin (com.atlassian.gadgets.dashboard)

Any solution or workaround?

Like Sara Santini likes this
Şükrü TURGUT May 21, 2021

Hello 

The solution for my case was very simple. I am using windows system language on turkish so when i changed the system language to english and clear all files and restart the same steps on tutorial, the problem solved. 

Maybe this solution help you.

0 votes
Michael Thompson
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
May 28, 2019

Are you able to start Jira in safe mode? This would at least tell you that plugins are the only problem point for you.

./start-jira.sh --disable-all-addons
Snezana Bozovic May 29, 2019

Tried that, didn't help. I still get the same error, no access to UI.

0 votes
PVS
Banned
May 28, 2019

Hi Snezana Bozovic

 

First you shutdown the jira and You can Delete "lock  file"It is In   Program Files\Atlassian\Application Data\JIRA  and  the Plugin cache  Clear and Delete  .bundled-plugins and  .osgi-plugins  in  Plugin data After restart the jira  If You want more info check the below link 

 

https://confluence.atlassian.com/jirakb/jira-applications-system-plugin-timeout-while-waiting-for-add-ons-to-enable-212173447.html

Snezana Bozovic May 28, 2019

That didn't help, there is no lock file. Already tried deleting plugins caches.

Also, this is a Linux Server.

0 votes
Snezana Bozovic May 28, 2019

Also tried deleting plugin cache in

JIRA_HOME/plugins/.osgi-plugins

but that didn't help.

Suggest an answer

Log in or Sign up to answer