Jira application is down

[Kurasi Vishnu Priya]

Hi,

.jira.startup.IndexRecoveryLauncher] Issue index is out of date with the database by more than 24 hours. Automatic recovery is not attempted

Facing this error after upgrading my jira from 8.4 to 8.7,please let me know how to resolve this

We have upgraded our DB to postgres 11.10 after the jira upgrade

Kindly help me out on resolving this issue.

Thanks

Priya K

[Nic Brough -Adaptavist-]

So, the "You can now access JIRA through your web browser." belies your original "Jira is down" question.  It's not, according to the log, it is running.

The rest of the log is not showing any errors that would halt Jira either, so you'll need to do some debugging.

What exactly do you mean by "Jira is down"?  What happens when you try to browse to the site?  Churning?  Errors?  Warnings?  Corrupted Jira login screen?

On the server side, is it actually running, as the log file implies?  Is the log really just stopping at that point?

[Kurasi Vishnu Priya]

@Nic Brough -Adaptavist-

@Nic Brough -Adaptavist- @Jonas Ekström 

Hi,

This is the screen when i try to access the jira site.Am not sure whats blocking my site.

[Nic Brough -Adaptavist-]

Ok, a 408 is a timeout, which implies there is something there, but it's not responding to anything after the initial "yep, give me a sec to get some data for you" that a page sends back to your browser when first contacted (you'd get a 404 if there was no response to the first contact)

The thing that is running there will be the Tomcat (which will be asking the Jira it is running for the actual content to serve up).

So you'll need to look at why Tomcat or Jira are not serving up.

I would start with the application log, where you've looked already, but at the actual errors that are a result of it crashing.  If it really is stopping dead where you've cut it off in your post, then there is a fault happening in the application server, not the application, and you'll want to read the Tomcat log at <jira install>/logs/catalina.out to find out why it's hard-killing Jira.

[Kurasi Vishnu Priya]

@Nic Brough -Adaptavist- 

Please find the complete log details,still i couldnt access my site

2021-07-21 08:03:37,417-0400 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]

___ System-level feature flags ______________

com.atlassian.jira.agile.darkfeature.editable.detailsview : true
jira.zdu.admin-updates-ui : true
nps.survey.inline.dialog : true
jira.zdu.jmx-monitoring : true
jira.plugin.devstatus.phasetwo : true
jira.frother.reporter.field : true
atlassian.rest.xsrf.legacy.enabled : true
jira.issue.status.lozenge : true
com.atlassian.jira.config.BIG_PIPE : true
sd.new.settings.sidebar.location.disabled : true
jira.zdu.cluster-upgrade-state : true
com.atlassian.jira.config.PDL : true
jira.plugin.devstatus.phasetwo.enabled : true
atlassian.aui.raphael.disabled : true
app-switcher.new : true
frother.assignee.field : true
jira.onboarding.cyoa : true
com.atlassian.jira.config.CoreFeatures.LICENSE_ROLES_ENABLED : true
com.atlassian.jira.config.ProjectConfig.MENU : true
com.atlassian.jira.projects.sidebar.DEFER_RESOURCES : true
jira.export.csv.enabled : true

2021-07-21 08:03:37,418-0400 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]

***********************************************************************************
JIRA 8.7.0 build: 807000 started. You can now access JIRA through your web browser.
***********************************************************************************

2021-07-21 08:03:38,069-0400 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]

___ Plugin System Started _________________

2021-07-21 08:03:38,166-0400 JIRA-Bootstrap INFO [c.a.jira.upgrade.UpgradeScheduler] Scheduling upgrades to run in 1 minute(s)
2021-07-21 08:03:40,627-0400 Modification Check:thread-1 INFO [c.a.jira.startup.JiraStartupLogger]

___ Modifications ___________________________

Modified Files : jira-application.properties
Removed Files : None

2021-07-21 08:03:43,246-0400 JIRA-Bootstrap INFO [c.a.jira.startup.LauncherContextListener] Memory Usage:
---------------------------------------------------------------------------------
Heap memory : Used: 610 MiB. Committed: 1781 MiB. Max: 4020 MiB
Non-heap memory : Used: 279 MiB. Committed: 304 MiB. Max: 1536 MiB
---------------------------------------------------------------------------------
TOTAL : Used: 889 MiB. Committed: 2084 MiB. Max: 5556 MiB
---------------------------------------------------------------------------------
2021-07-21 08:03:43,305-0400 Caesium-1-1 INFO [c.a.r.internal.configuration.ConfigurationLoggerJob] Periodic rate limiting configuration log. System rate limiting settings: [SystemRateLimitingSettings(mode=OFF, bucketSettings=TokenBucketSettings(capacity=50, fillRate=10, intervalFrequency=1, intervalTimeUnit=Seconds), jobControlSettings=SystemJobControlSettings(reportingDbArchivingJobFrequencyDuration=PT1M10S, reportingDbRetentionPeriodDuration=PT24H, bucketCollectionJobFrequencyDuration=PT5M, bucketCleanupJobFrequencyDuration=PT15M, settingsReloadJobFrequencyDuration=PT30S))]
2021-07-21 08:03:43,664-0400 Caesium-1-3 INFO [c.a.jira.startup.IndexRecoveryLauncher] Running start-index-recovery
2021-07-21 08:03:43,700-0400 Caesium-1-1 INFO [c.a.jira.startup.CacheWarmerLauncher] Warming up 3 caches.
2021-07-21 08:03:44,674-0400 Caesium-1-3 ERROR [c.a.jira.startup.IndexRecoveryLauncher] Issue index is out of date with the database by more than 24 hours. Automatic recovery is not attempted
2021-07-21 08:03:53,375-0400 Caesium-1-1 INFO [c.a.jira.startup.CacheWarmerLauncher] Warmed cache(s) in 9674 ms.
2021-07-21 08:04:38,169-0400 Caesium-1-1 INFO ServiceRunner [c.a.jira.upgrade.UpgradeScheduler] Running scheduled upgrades
2021-07-21 08:04:38,194-0400 Caesium-1-1 INFO ServiceRunner [c.a.jira.upgrade.LoggingUpgradeService] run upgrades has started
2021-07-21 08:04:38,416-0400 Caesium-1-1 INFO ServiceRunner [c.a.j.upgrade.util.BuildNumberDao] Setting current build number to 807000
2021-07-21 08:04:38,419-0400 Caesium-1-1 INFO ServiceRunner [c.a.jira.upgrade.UpgradeIndexManager] There are no reindex requests of type [IMMEDIATE, DELAYED] so none will be run
2021-07-21 08:04:38,419-0400 Caesium-1-1 INFO ServiceRunner [c.a.j.upgrade.util.BuildNumberDao] Setting current version to 8.7.0
2021-07-21 08:04:38,419-0400 Caesium-1-1 INFO ServiceRunner [c.a.j.upgrade.util.BuildNumberDao] Setting downgrade version to 7.1.2
2021-07-21 08:04:38,423-0400 Caesium-1-1 INFO ServiceRunner [c.a.jira.upgrade.LoggingUpgradeService] run upgrades has finished successfully, and took 230 milliseconds to process.
2021-07-21 08:04:38,423-0400 Caesium-1-1 INFO ServiceRunner [c.a.jira.upgrade.UpgradeScheduler] JIRA upgrades completed successfully
2021-07-21 08:04:38,430-0400 Caesium-1-1 INFO ServiceRunner [c.a.jira.upgrade.UpgradeScheduler] Plugins upgrades completed successfully
2021-07-21 08:04:38,431-0400 Caesium-1-1 INFO ServiceRunner [c.a.jira.upgrade.UpgradeIndexManager] Reindexing is not allowed after this upgrade and there is no immediate reindex requests
2021-07-21 20:03:43,304-0400 Caesium-1-3 INFO anonymous My reminders Service [c.a.r.internal.configuration.ConfigurationLoggerJob] Periodic rate limiting configuration log. System rate limiting settings: [SystemRateLimitingSettings(mode=OFF, bucketSettings=TokenBucketSettings(capacity=50, fillRate=10, intervalFrequency=1, intervalTimeUnit=Seconds), jobControlSettings=SystemJobControlSettings(reportingDbArchivingJobFrequencyDuration=PT1M10S, reportingDbRetentionPeriodDuration=PT24H, bucketCollectionJobFrequencyDuration=PT5M, bucketCleanupJobFrequencyDuration=PT15M, settingsReloadJobFrequencyDuration=PT30S))]
2021-07-22 08:03:43,304-0400 Caesium-1-3 INFO ServiceRunner [c.a.r.internal.configuration.ConfigurationLoggerJob] Periodic rate limiting configuration log. System rate limiting settings: [SystemRateLimitingSettings(mode=OFF, bucketSettings=TokenBucketSettings(capacity=50, fillRate=10, intervalFrequency=1, intervalTimeUnit=Seconds), jobControlSettings=SystemJobControlSettings(reportingDbArchivingJobFrequencyDuration=PT1M10S, reportingDbRetentionPeriodDuration=PT24H, bucketCollectionJobFrequencyDuration=PT5M, bucketCleanupJobFrequencyDuration=PT15M, settingsReloadJobFrequencyDuration=PT30S))]
2021-07-22 20:03:43,311-0400 Caesium-1-1 INFO ServiceRunner [c.a.r.internal.configuration.ConfigurationLoggerJob] Periodic rate limiting configuration log. System rate limiting settings: [SystemRateLimitingSettings(mode=OFF, bucketSettings=TokenBucketSettings(capacity=50, fillRate=10, intervalFrequency=1, intervalTimeUnit=Seconds), jobControlSettings=SystemJobControlSettings(reportingDbArchivingJobFrequencyDuration=PT1M10S, reportingDbRetentionPeriodDuration=PT24H, bucketCollectionJobFrequencyDuration=PT5M, bucketCleanupJobFrequencyDuration=PT15M, settingsReloadJobFrequencyDuration=PT30S))]
2021-07-23 08:03:43,303-0400 Caesium-1-2 INFO anonymous My reminders Service [c.a.r.internal.configuration.ConfigurationLoggerJob] Periodic rate limiting configuration log. System rate limiting settings: [SystemRateLimitingSettings(mode=OFF, bucketSettings=TokenBucketSettings(capacity=50, fillRate=10, intervalFrequency=1, intervalTimeUnit=Seconds), jobControlSettings=SystemJobControlSettings(reportingDbArchivingJobFrequencyDuration=PT1M10S, reportingDbRetentionPeriodDuration=PT24H, bucketCollectionJobFrequencyDuration=PT5M, bucketCleanupJobFrequencyDuration=PT15M, settingsReloadJobFrequencyDuration=PT30S))]
2021-07-23 20:03:43,305-0400 Caesium-1-2 INFO ServiceRunner [c.a.r.internal.configuration.ConfigurationLoggerJob] Periodic rate limiting configuration log. System rate limiting settings: [SystemRateLimitingSettings(mode=OFF, bucketSettings=TokenBucketSettings(capacity=50, fillRate=10, intervalFrequency=1, intervalTimeUnit=Seconds), jobControlSettings=SystemJobControlSettings(reportingDbArchivingJobFrequencyDuration=PT1M10S, reportingDbRetentionPeriodDuration=PT24H, bucketCollectionJobFrequencyDuration=PT5M, bucketCleanupJobFrequencyDuration=PT15M, settingsReloadJobFrequencyDuration=PT30S))]
2021-07-24 08:03:43,303-0400 Caesium-1-1 INFO anonymous My reminders Service [c.a.r.internal.configuration.ConfigurationLoggerJob] Periodic rate limiting configuration log. System rate limiting settings: [SystemRateLimitingSettings(mode=OFF, bucketSettings=TokenBucketSettings(capacity=50, fillRate=10, intervalFrequency=1, intervalTimeUnit=Seconds), jobControlSettings=SystemJobControlSettings(reportingDbArchivingJobFrequencyDuration=PT1M10S, reportingDbRetentionPeriodDuration=PT24H, bucketCollectionJobFrequencyDuration=PT5M, bucketCleanupJobFrequencyDuration=PT15M, settingsReloadJobFrequencyDuration=PT30S))]
[jira@jira logs]$

Thanks

Priya 

[Jonas Ekström]

If you have issues with indexing after upgrade of Postgres to 11.10 you can look at https://confluence.atlassian.com/jirakb/reindex-progresses-very-slowly-after-postgres-upgrade-or-restore-1070073091.html

I had to vacuum and reindex the database after the upgrade to make Jira work properly

[Kurasi Vishnu Priya]

@Jonas Ekström @Nic Brough -Adaptavist- @a.gowri82@gmail.com

Hi ,

We have performed the vaccum full and reindex in database, but still the jira site is down,

kindly see the error am facing out.

JIRA 8.7.0 build: 807000 started. You can now access JIRA through your web browser.
    ***********************************************************************************2021-07-21 05:53:57,206-0400 JIRA-Bootstrap INFO      [c.a.jira.startup.JiraStartupLogger]    ___ Plugin System Started _________________2021-07-21 05:53:57,286-0400 JIRA-Bootstrap INFO      [c.a.jira.upgrade.UpgradeScheduler] Scheduling upgrades to run in 1 minute(s)
2021-07-21 05:53:59,143-0400 Modification Check:thread-1 INFO      [c.a.jira.startup.JiraStartupLogger]    ___ Modifications ___________________________         Modified Files                                : jira-application.properties
         Removed Files                                 : None2021-07-21 05:54:02,799-0400 JIRA-Bootstrap INFO      [c.a.jira.startup.LauncherContextListener] Memory Usage:
    ---------------------------------------------------------------------------------
      Heap memory     :  Used:  757 MiB.  Committed: 1817 MiB.  Max: 4004 MiB
      Non-heap memory :  Used:  276 MiB.  Committed:  302 MiB.  Max: 1536 MiB
    ---------------------------------------------------------------------------------
      TOTAL           :  Used: 1033 MiB.  Committed: 2119 MiB.  Max: 5540 MiB
    ---------------------------------------------------------------------------------
2021-07-21 05:54:02,895-0400 Caesium-1-4 INFO anonymous    My reminders Service [c.a.r.internal.configuration.ConfigurationLoggerJob] Periodic rate limiting configuration log. System rate limiting settings: [SystemRateLimitingSettings(mode=OFF, bucketSettings=TokenBucketSettings(capacity=50, fillRate=10, intervalFrequency=1, intervalTimeUnit=Seconds), jobControlSettings=SystemJobControlSettings(reportingDbArchivingJobFrequencyDuration=PT1M10S, reportingDbRetentionPeriodDuration=PT24H, bucketCollectionJobFrequencyDuration=PT5M, bucketCleanupJobFrequencyDuration=PT15M, settingsReloadJobFrequencyDuration=PT30S))]
2021-07-21 05:54:02,967-0400 Caesium-1-1 INFO ServiceRunner     [c.a.jira.startup.IndexRecoveryLauncher] Running start-index-recovery
2021-07-21 05:54:03,276-0400 Caesium-1-4 INFO anonymous    My reminders Service [c.a.jira.startup.CacheWarmerLauncher] Warming up 3 caches.
2021-07-21 05:54:03,976-0400 Caesium-1-1 ERROR ServiceRunner     [c.a.jira.startup.IndexRecoveryLauncher] Issue index is out of date with the database by more than 24 hours. Automatic recovery is not attempted
2021-07-21 05:54:13,163-0400 Caesium-1-4 INFO anonymous    My reminders Service [c.a.jira.startup.CacheWarmerLauncher] Warmed cache(s) in 9886 ms.
2021-07-21 05:54:57,291-0400 Caesium-1-2 INFO      [c.a.jira.upgrade.UpgradeScheduler] Running scheduled upgrades
2021-07-21 05:54:57,308-0400 Caesium-1-2 INFO      [c.a.jira.upgrade.LoggingUpgradeService] run upgrades has started
2021-07-21 05:54:57,529-0400 Caesium-1-2 INFO      [c.a.j.upgrade.util.BuildNumberDao] Setting current build number to 807000
2021-07-21 05:54:57,532-0400 Caesium-1-2 INFO      [c.a.jira.upgrade.UpgradeIndexManager] There are no reindex requests of type [IMMEDIATE, DELAYED] so none will be run
2021-07-21 05:54:57,532-0400 Caesium-1-2 INFO      [c.a.j.upgrade.util.BuildNumberDao] Setting current version to 8.7.0
2021-07-21 05:54:57,533-0400 Caesium-1-2 INFO      [c.a.j.upgrade.util.BuildNumberDao] Setting downgrade version to 7.1.2
2021-07-21 05:54:57,539-0400 Caesium-1-2 INFO      [c.a.jira.upgrade.LoggingUpgradeService] run upgrades has finished successfully, and took 230 milliseconds to process.
2021-07-21 05:54:57,539-0400 Caesium-1-2 INFO      [c.a.jira.upgrade.UpgradeScheduler] JIRA upgrades completed successfully
2021-07-21 05:54:57,545-0400 Caesium-1-2 INFO      [c.a.jira.upgrade.UpgradeScheduler] Plugins upgrades completed successfully
2021-07-21 05:54:57,546-0400 Caesium-1-2 INFO      [c.a.jira.upgrade.UpgradeIndexManager] Reindexing is not allowed after this upgrade and there is no immediate reindex requests

to perform the reindex through the application my site is not up.

Kindly suggest the method to overcome this.

Thanks

Priya K

[Nic Brough -Adaptavist-]

That error will not be stopping Jira from running, it's telling you there are data inconsistencies, and you really need to re-index your Jira in full.

What error messages are you getting that are stopping Jira?