SQL server went down unexpectedly, now JIRA won't start.

Question

The database went down, without my jira instance expecting it.

Now when I try to start jira it doesn't work. I've found multiple solutions that aren't working. (things like deleting plugin caches and temporary files. None of that is working. )

Here is the log output, it is really frustrating to debug with just this but this is all i'm getting. The installation process just hangs.

```

sudo tail -f /opt/atlassian/jira/logs/catalina.out
2018-04-11 19:26:11,666 Caesium-1-2 INFO [c.a.j.upgrade.util.BuildNumberDao] Setting downgrade version to 7.1.2
2018-04-11 19:26:11,676 Caesium-1-2 INFO [c.a.jira.upgrade.LoggingUpgradeService] run upgrades has finished successfully, and took 313 milliseconds to process.
2018-04-11 19:26:11,677 Caesium-1-2 INFO [c.a.jira.upgrade.UpgradeScheduler] JIRA upgrades completed successfully
2018-04-11 19:26:11,683 Caesium-1-2 INFO [c.a.s.bootstrap.upgrade.ServiceDeskUpgradeTaskFactory] Converting JSD Async Upgrade tasks to the Upgrade Task Framework format.
2018-04-11 19:26:11,683 Caesium-1-2 INFO [c.a.jira.upgrade.UpgradeScheduler] Plugins upgrades completed successfully
2018-04-11 19:26:11,684 Caesium-1-2 INFO [c.a.jira.upgrade.UpgradeIndexManager] Reindexing is not allowed after this upgrade and there is no immediate reindex requests
2018-04-11 19:26:13,271 Caesium-1-3 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.servicedesk.servicedesk-notifications-plugin]
2018-04-11 19:26:13,369 Caesium-1-4 DEBUG ServiceRunner [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.jira.jira-email-processor-plugin]
2018-04-11 19:26:17,604 Caesium-1-4 DEBUG ServiceRunner [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.troubleshooting.plugin-jira]
2018-04-11 19:35:23,396 http-nio-8080-exec-1 DEBUG - [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.whisper.atlassian-whisper-plugin]

```

I also have this in the security log :

```

2018-04-11 19:09:27,953 Caesium-1-3 ERROR [c.a.jira.plugin.JiraPluginManager] Could not enable plugin with key 'tac.jira software.languages.ja_JP'
2018-04-11 19:24:09,394 localhost-startStop-1 ERROR [webwork.config.DefaultConfiguration] Skipping configuration for class 'com.atlassian.jira.config.webwork.JiraPluginsConfiguration'
2018-04-11 19:24:09,434 localhost-startStop-1 ERROR [webwork.config.DefaultConfiguration] Skipping configuration for class 'com.atlassian.jira.config.webwork.JiraPluginsConfiguration'
jira.projectkey.warning : admin.errors.must.specify.unique.project.key
2018-04-11 19:26:07,254 Caesium-1-2 ERROR [c.a.jira.plugin.JiraPluginManager] Could not enable plugin with key 'tac.jira software.languages.ja_JP'

```

My question is what has caused this problem. What plugin is causing it or what setting can I adjust so that I can start JIRA on this server again.

Thanks.

unknown · Answer

Hey @Sergey It looks like that didn't work. All the plugins are enabled according to the db.

```

jiradb=> select * from pluginstate;
pluginkey | pluginenabled
-----------+---------------
(0 rows)

```

Here are the logs from the current state of jira:

```

$ sudo tail -f /opt/atlassian/jira/logs/catalina.out
2018-04-19 19:12:48,927 Caesium-1-1 INFO [c.a.jira.upgrade.UpgradeScheduler] JIRA upgrades completed successfully
2018-04-19 19:12:48,935 Caesium-1-1 INFO [c.a.s.bootstrap.upgrade.ServiceDeskUpgradeTaskFactory] Converting JSD Async Upgrade tasks to the Upgrade Task Framework format.
2018-04-19 19:12:48,936 Caesium-1-1 INFO [c.a.jira.upgrade.UpgradeScheduler] Plugins upgrades completed successfully
2018-04-19 19:12:48,937 Caesium-1-1 INFO [c.a.jira.upgrade.UpgradeIndexManager] Reindexing is not allowed after this upgrade and there is no immediate reindex requests
2018-04-19 19:12:50,447 Caesium-1-2 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.jira.jira-email-processor-plugin]
2018-04-19 19:12:51,612 Caesium-1-3 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.troubleshooting.plugin-jira]
2018-04-19 19:12:57,610 http-nio-8080-exec-1 DEBUG - [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.whisper.atlassian-whisper-plugin]
2018-04-19 19:13:19,702 http-nio-8080-exec-1 INFO - [c.a.j.plugin.navigation.HeaderFooterRendering] Bigpipe taking longer than 5s. Exact time:20024
2018-04-19 19:13:19,702 http-nio-8080-exec-3 INFO anonymous 1152x3x3 - 213.138.162.155 / [c.a.j.plugin.navigation.HeaderFooterRendering] Bigpipe taking longer than 5s. Exact time:20022
2018-04-19 19:13:19,703 http-nio-8080-exec-2 INFO anonymous 1152x2x2 - 213.138.162.155 / [c.a.j.plugin.navigation.HeaderFooterRendering] Bigpipe taking longer than 5s. Exact time:20025

```

Currently getting back 521 errors and 502 errors when accessing my jira instance through my cloudflare managed subdomain.

Do you have any other ideas on what might be going on?

Thanks.

Sergey_k6s8t8 · Answer

Hi David, Sorry to hear you're experiencing this issue. It's odd to see in the logs that JIRA is performing upgrade tasks... Have you upgraded JIRA recently? For "could not enable plugin" messages -- actually the plugins might be disabled in the actual database. This is actually a documented issue. Check out this article: https://confluence.atlassian.com/jirakb/jira-application-does-not-start-due-to-disabled-bundled-plugin-254738702.html Please run the following SQL query to verify that: SELECT * FROM pluginstate where pluginenabled = 'false'; If there are any plugins that are disabled there, you can enable them by running the following query (WARNING -- STOP JIRA first before executing the query and backup the database by means of native backup tools in case you need to revert back to the previous state) UPDATE pluginstate SET pluginenabled='true' WHERE pluginkey=''; Hope this helps, Regards, Sergey