JIRA Service Desk access problem

[Murthy Namagiri]

Unable to access the JIRA Servie Desk, 

Error Message Shows "Unexpected exception during JIRA startup. This JIRA instance will not be able to recover".

[Nic Brough -Adaptavist-]

Ok, there's something broken in your database.

There is a fix, see https://confluence.atlassian.com/jirakb/jira-fails-to-start-due-to-multiple-entries-with-same-key-800702874.html

[Murthy Namagiri]

I run the below SQL query,

select * from [JiraSDSchema].[clusteredjob] where job_id = 'AutomationRule-2';

Result:

Message: 0 rows affected.

[Murthy Namagiri]

How to check the multiple entries with the same key values..

[Murthy Namagiri]

Hi Nic Brough,

2017-12-18 17:56:29,926 JIRA-Bootstrap INFO      [atlassian.servicedesk.lifecycle]
    
    JIRA ServiceDesk Run History
    
        *JIRA ServiceDesk v3.5.0-REL-0008 (DevVersion) started at 26 May 2017 04:46
    
    
2017-12-18 17:56:29,942 JIRA-Bootstrap INFO      [atlassian.servicedesk.lifecycle] There does not appear to be any JIRA ServiceDesk upgrade history
2017-12-18 17:56:29,942 JIRA-Bootstrap INFO      [atlassian.servicedesk.lifecycle]
    Description        : JIRA Service Desk (Server) 3 Agents: Starter License
    Type               : license type <COMMERCIAL>
    Valid              : true
    Maintenance Expiry : 23-05-2018
    Maximum Users      : 3
    Organisation       : Techminds Group Llc
    SEN                : SEN-9714726
2017-12-18 17:56:29,942 JIRA-Bootstrap INFO      [atlassian.servicedesk.lifecycle] started. How can we help you?
2017-12-18 17:56:30,004 JIRA-Bootstrap INFO      [c.a.j.i.m.processor.bootstrap.MailPluginLifeCycleAware] JIRA Email Processor is initializing...
2017-12-18 17:56:30,020 JIRA-Bootstrap INFO      [c.a.j.i.m.processor.bootstrap.MailPluginLifeCycleAware] JIRA Email Processor is initialized.
2017-12-18 17:56:30,176 JIRA-Bootstrap DEBUG      [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.labs.hipchat.hipchat-for-jira-plugin]
2017-12-18 17:56:30,176 JIRA-Bootstrap INFO      [c.a.j.p.hipchat.service.HipChatPluginLauncher] HipChat Plugin launched successfully
2017-12-18 17:56:30,176 JIRA-Bootstrap INFO      [c.a.j.p.h.service.scheduling.AbstractTaskScheduler] On plugin event: com.atlassian.jira.plugins.hipchat.model.event.PluginStartedEvent
2017-12-18 17:56:30,192 JIRA-Bootstrap INFO      [c.a.j.p.h.service.scheduling.AbstractTaskScheduler] Registering a new hipchatUpdateGlancesDataJob task.
2017-12-18 17:56:30,192 JIRA-Bootstrap INFO      [c.a.j.p.h.service.scheduling.AbstractTaskScheduler] On plugin event: com.atlassian.jira.plugins.hipchat.model.event.PluginStartedEvent
2017-12-18 17:56:30,192 JIRA-Bootstrap INFO      [c.a.j.p.h.service.scheduling.AbstractTaskScheduler] Registering a new hipchatInstallGlancesJob task.
2017-12-18 17:56:30,192 JIRA-Bootstrap INFO      [c.a.j.p.h.service.scheduling.AbstractTaskScheduler] On plugin event: com.atlassian.jira.plugins.hipchat.model.event.PluginStartedEvent
2017-12-18 17:56:30,192 JIRA-Bootstrap INFO      [c.a.j.p.h.service.scheduling.AbstractTaskScheduler] Registering a new hipchatRefreshConnectionStatusJob task.
2017-12-18 17:56:30,254 JIRA-Bootstrap INFO      [c.a.p.internal.bootstrap.Launcher] PSMQ is initializing...
2017-12-18 17:56:30,254 JIRA-Bootstrap INFO      [c.a.p.internal.bootstrap.Launcher] PSMQ is initialized.
2017-12-18 17:56:38,270 JIRA-Bootstrap INFO      [c.a.s.p.a.internal.bootstrap.AutomationPluginLauncher] AutomationPlugin onStart() - Service Desk Automation Plugin is starting...
2017-12-18 17:56:38,301 JIRA-Bootstrap DEBUG      [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.psmq]
2017-12-18 17:56:38,895 JIRA-Bootstrap INFO      [c.a.s.p.a.internal.bootstrap.AutomationPluginLauncher] AutomationPlugin onStart() - Service Desk Automation Plugin is started.
2017-12-18 17:56:38,895 JIRA-Bootstrap INFO      [c.a.j.p.w.i.bootstrap.lifecycle.WorkingHoursPluginLauncher] JIRA (SD) Working Hours Plugin starting...
2017-12-18 17:56:38,895 JIRA-Bootstrap INFO      [c.a.j.p.w.i.bootstrap.lifecycle.WorkingHoursPluginLauncher] JIRA (SD) Working Hours Plugin started
2017-12-18 17:56:38,895 JIRA-Bootstrap INFO      [c.a.s.t.internal.bootstrap.TimedPromiseLauncher] Timed Promises is initializing...
2017-12-18 17:56:38,926 JIRA-Bootstrap INFO      [c.a.s.t.internal.bootstrap.TimedPromiseLauncher] Timed Promises is initialized.
2017-12-18 17:56:39,286 JIRA-Bootstrap INFO      [c.a.jira.scheduler.JiraSchedulerLauncher] Starting the JIRA Scheduler....
2017-12-18 17:56:39,301 JIRA-Bootstrap ERROR      [c.a.jira.scheduler.JiraSchedulerLauncher] A RuntimeException occurred during JiraSchedulerLauncher servlet context initialisation - Multiple entries with same key: sd.custom.notification.batch.send=Thu Dec 14 17:02:52 IST 2017 and sd.custom.notification.batch.send=Thu Dec 14 17:02:52 IST 2017.
java.lang.IllegalArgumentException: Multiple entries with same key: sd.custom.notification.batch.send=Thu Dec 14 17:02:52 IST 2017 and sd.custom.notification.batch.send=Thu Dec 14 17:02:52 IST 2017
    at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:150)
    at com.google.common.collect.RegularImmutableMap.checkNoConflictInBucket(RegularImmutableMap.java:104)
    at com.google.common.collect.RegularImmutableMap.<init>(RegularImmutableMap.java:70)
    at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:254)
    at com.atlassian.jira.scheduler.OfBizClusteredJobDao.refresh(OfBizClusteredJobDao.java:111)
    at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobsFromDao(SchedulerQueueImpl.java:139)
    at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobsUnderLock(SchedulerQueueImpl.java:130)
    at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobs(SchedulerQueueImpl.java:117)
    at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.refreshClusteredJobs(CaesiumSchedulerService.java:359)
    at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.startImpl(CaesiumSchedulerService.java:275)
    at com.atlassian.jira.scheduler.JiraCaesiumSchedulerService.startImpl(JiraCaesiumSchedulerService.java:36)
    at com.atlassian.scheduler.core.AbstractSchedulerService.start(AbstractSchedulerService.java:180)
    at com.atlassian.jira.scheduler.JiraSchedulerLauncher.proceedIfAllClear(JiraSchedulerLauncher.java:38)
    at com.atlassian.jira.scheduler.JiraSchedulerLauncher.start(JiraSchedulerLauncher.java:24)
    at com.atlassian.jira.startup.ActiveServicesLauncher.start(ActiveServicesLauncher.java:55)
    at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postTenantArrived$4(DefaultJiraLauncher.java:184)
    at com.atlassian.jira.startup.DefaultInstantUpgradeManager$StartupTask.run(DefaultInstantUpgradeManager.java:139)
    at com.atlassian.jira.startup.DefaultInstantUpgradeManager.runTask(DefaultInstantUpgradeManager.java:55)
    at com.atlassian.jira.startup.DefaultInstantUpgradeManager.doNowOrWhenInstanceBecomesActive(DefaultInstantUpgradeManager.java:42)
    at com.atlassian.jira.startup.DefaultJiraLauncher.postTenantArrived(DefaultJiraLauncher.java:177)
    at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDBActivated$3(DefaultJiraLauncher.java:165)
    at com.atlassian.jira.tenancy.DefaultTenantManager.doNowOrWhenTenantArrives(DefaultTenantManager.java:55)
    at com.atlassian.jira.startup.DefaultJiraLauncher.postDBActivated(DefaultJiraLauncher.java:162)
    at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:151)
    at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:298)
    at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:194)
    at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:141)
    at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:103)
    at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
    at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:101)
    at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:149)
    at java.lang.Thread.run(Thread.java:745)
2017-12-18 17:56:39,348 Caesium-1-2 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.user.UserHistoryDatabaseCompactor; runDetails=RunDetailsImpl[startTime=1513599999301,runOutcome=SUCCESS,durationInMillis=47,message=]
2017-12-18 17:56:39,348 JIRA-Bootstrap ERROR      [c.a.jira.startup.LauncherContextListener] Unable to start JIRA.
java.lang.IllegalArgumentException: Multiple entries with same key: sd.custom.notification.batch.send=Thu Dec 14 17:02:52 IST 2017 and sd.custom.notification.batch.send=Thu Dec 14 17:02:52 IST 2017
    at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:150)
    at com.google.common.collect.RegularImmutableMap.checkNoConflictInBucket(RegularImmutableMap.java:104)
    at com.google.common.collect.RegularImmutableMap.<init>(RegularImmutableMap.java:70)
    at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:254)
    at com.atlassian.jira.scheduler.OfBizClusteredJobDao.refresh(OfBizClusteredJobDao.java:111)
    at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobsFromDao(SchedulerQueueImpl.java:139)
    at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobsUnderLock(SchedulerQueueImpl.java:130)
    at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobs(SchedulerQueueImpl.java:117)
    at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.refreshClusteredJobs(CaesiumSchedulerService.java:359)
    at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.startImpl(CaesiumSchedulerService.java:275)
    at com.atlassian.jira.scheduler.JiraCaesiumSchedulerService.startImpl(JiraCaesiumSchedulerService.java:36)
    at com.atlassian.scheduler.core.AbstractSchedulerService.start(AbstractSchedulerService.java:180)
    at com.atlassian.jira.scheduler.JiraSchedulerLauncher.proceedIfAllClear(JiraSchedulerLauncher.java:38)
    at com.atlassian.jira.scheduler.JiraSchedulerLauncher.start(JiraSchedulerLauncher.java:24)
    at com.atlassian.jira.startup.ActiveServicesLauncher.start(ActiveServicesLauncher.java:55)
    at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postTenantArrived$4(DefaultJiraLauncher.java:184)
    at com.atlassian.jira.startup.DefaultInstantUpgradeManager$StartupTask.run(DefaultInstantUpgradeManager.java:139)
    at com.atlassian.jira.startup.DefaultInstantUpgradeManager.runTask(DefaultInstantUpgradeManager.java:55)
    at com.atlassian.jira.startup.DefaultInstantUpgradeManager.doNowOrWhenInstanceBecomesActive(DefaultInstantUpgradeManager.java:42)
    at com.atlassian.jira.startup.DefaultJiraLauncher.postTenantArrived(DefaultJiraLauncher.java:177)
    at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDBActivated$3(DefaultJiraLauncher.java:165)
    at com.atlassian.jira.tenancy.DefaultTenantManager.doNowOrWhenTenantArrives(DefaultTenantManager.java:55)
    at com.atlassian.jira.startup.DefaultJiraLauncher.postDBActivated(DefaultJiraLauncher.java:162)
    at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:151)
    at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:298)
    at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:194)
    at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:141)
    at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:103)
    at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
    at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:101)
    at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:149)
    at java.lang.Thread.run(Thread.java:745)
2017-12-18 17:56:39,348 Caesium-1-1 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.cache.monitor.CacheStatisticsMonitor; runDetails=RunDetailsImpl[startTime=1513599999301,runOutcome=SUCCESS,durationInMillis=47,message=]
2017-12-18 17:56:39,380 Caesium-1-1 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=CompatibilityPluginScheduler.JobId.BundledUpdateCheckJob-job; runDetails=RunDetailsImpl[startTime=1513599999364,runOutcome=SUCCESS,durationInMillis=16,message=]
2017-12-18 17:56:39,380 JIRA-Bootstrap INFO      [c.a.jira.startup.LauncherContextListener] Memory Usage:
    ---------------------------------------------------------------------------------
      Heap memory     :  Used:  323 MiB.  Committed:  684 MiB.  Max:  731 MiB
      Non-heap memory :  Used:  252 MiB.  Committed:  274 MiB.  Max: 1264 MiB
    ---------------------------------------------------------------------------------
      TOTAL           :  Used:  575 MiB.  Committed:  957 MiB.  Max: 1995 MiB
    ---------------------------------------------------------------------------------
2017-12-18 17:56:39,395 Caesium-1-4 DEBUG      [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.jira.jira-email-processor-plugin]
2017-12-18 17:56:39,395 Caesium-1-1 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.scheduler.timedpromise.TimedPromiseHistoryPruner; runDetails=RunDetailsImpl[startTime=1513599999380,runOutcome=SUCCESS,durationInMillis=15,message=]
2017-12-18 17:56:39,473 Caesium-1-1 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.scheduler.timedpromise.internal.TimedPromisePruner; runDetails=RunDetailsImpl[startTime=1513599999411,runOutcome=SUCCESS,durationInMillis=62,message=]
2017-12-18 17:56:39,567 Caesium-1-2 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailProcessorJobRunner; runDetails=RunDetailsImpl[startTime=1513599999348,runOutcome=SUCCESS,durationInMillis=219,message=]
2017-12-18 17:56:39,567 Caesium-1-3 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailPullerJobRunner; runDetails=RunDetailsImpl[startTime=1513599999301,runOutcome=SUCCESS,durationInMillis=266,message=]
2017-12-18 17:56:39,567 Caesium-1-4 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailCleanerJobRunner; runDetails=RunDetailsImpl[startTime=1513599999301,runOutcome=SUCCESS,durationInMillis=266,message=]
2017-12-18 17:56:40,520 Caesium-1-1 DEBUG      [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.jira.plugins.workinghours]
2017-12-18 17:56:41,911 Caesium-1-1 DEBUG      [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.servicedesk.approvals-plugin]
2017-12-18 17:56:42,817 Caesium-1-1 DEBUG      [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.servicedesk.plugins.automation.servicedesk-automation-plugin]
2017-12-18 17:56:43,239 Caesium-1-1 WARN rajkumarr     [c.a.jira.security.GlobalPermissionsCache] Could not add permission Global Permission: [group=jira-administrators][permissionKey=ADMINISTER] - it already existed?
2017-12-18 17:56:43,239 Caesium-1-1 WARN rajkumarr     [c.a.jira.security.GlobalPermissionsCache] Could not add permission Global Permission: [group=jira-administrators][permissionKey=SYSTEM_ADMIN] - it already existed?
2017-12-18 17:56:44,082 Caesium-1-1 WARN anonymous     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.scheduler.timedpromise.internal.TimedPromiseExecutor.job; runDetails=RunDetailsImpl[startTime=1513599999473,runOutcome=SUCCESS,durationInMillis=4609,message=]
2017-12-18 17:56:44,567 PsmmqAsyncExecutors-job:thread-1 DEBUG rajkumarr     [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.jira.plugins.webhooks.jira-webhooks-plugin]
2017-12-18 17:56:44,739 PsmmqAsyncExecutors-job:thread-1 DEBUG rajkumarr     [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.plugins.base-hipchat-integration-plugin-api]
2017-12-18 17:56:47,067 PsmmqAsyncExecutors-job:thread-1 DEBUG rajkumarr     [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.servicedesk.servicedesk-lingo-plugin]

[Murthy Namagiri]

I found the log "multiple entries with same key", but the SQL Query didn't show any result..

[Murthy Namagiri]

But when i ran the below SQL Query, shows the result as

SQL Query Result:

ID    JOB_ID    JOB_RUNNER_KEY    SCHED_TYPE    INTERVAL_MILLIS    FIRST_RUN    CRON_EXPRESSION    TIME_ZONE    NEXT_RUN    VERSION    PARAMETERS
13089    sd.custom.notification.batch.send    sd.custom.notification.batch.job.runner    I    60000    1513251172433    NULL    NULL    1513251172433    1    NULL
13090    sd.custom.notification.batch.send    sd.custom.notification.batch.job.runner    I    60000    1513251172433    NULL    NULL    1513251172433    1    NULL

SQL Query:

select * from [JiraSDSchema].[clusteredjob] where job_id = 'sd.custom.notification.batch.send';

[Murthy Namagiri]

Issue resolved, when i deleted the ID: 13089,

Thanks you  Nic Brough.

[Nic Brough -Adaptavist-]

Could you look at what the log says?  <jira home>/logs/atlassian-jira.log

[Murthy Namagiri]

192.168.1.4 0x24922x1 - [04/Jan/2018:00:00:04 +0530] "GET /secure/errors.jsp HTTP/1.1" 503 760 4 "http://192.168.1.4:8085/secure/errors.jsp" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36" "9h1435"
192.168.1.4 0x24923x1 - [04/Jan/2018:00:00:34 +0530] "GET /secure/errors.jsp HTTP/1.1" 503 760 4 "http://192.168.1.4:8085/secure/errors.jsp" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36" "9h1435"
192.168.1.4 1x24924x1 - [04/Jan/2018:00:01:04 +0530] "GET /secure/errors.jsp HTTP/1.1" 503 760 5 "http://192.168.1.4:8085/secure/errors.jsp" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36" "9h1435"
192.168.1.4 1x24925x1 - [04/Jan/2018:00:01:34 +0530] "GET /secure/errors.jsp HTTP/1.1" 503 760 5 "http://192.168.1.4:8085/secure/errors.jsp" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36" "9h1435"
192.168.1.4 2x24926x1 - [04/Jan/2018:00:02:04 +0530] "GET /secure/errors.jsp HTTP/1.1" 503 760 4 "http://192.168.1.4:8085/secure/errors.jsp" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36" "9h1435"
192.168.1.4 2x24927x1 - [04/Jan/2018:00:02:35 +0530] "GET /secure/errors.jsp HTTP/1.1" 503 760 5 "http://192.168.1.4:8085/secure/errors.jsp" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36" "9h1435"
192.168.1.4 3x24928x1 - [04/Jan/2018:00:03:05 +0530] "GET /secure/errors.jsp HTTP/1.1" 503 760 4 "http://192.168.1.4:8085/secure/errors.jsp" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36" "9h1435"
192.168.1.4 3x24929x1 - [04/Jan/2018:00:03:35 +0530] "GET /secure/errors.jsp HTTP/1.1" 503 760 5 "http://192.168.1.4:8085/secure/errors.jsp" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36" "9h1435"

[Murthy Namagiri]

192.168.1.79 952x84x1 - [15/Jan/2018:15:52:43 +0530] "GET /static-assets/metal-all.css HTTP/1.1" 200 24353 0 "http://192.168.1.4:8085/secure/errors.jsp" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0" "4x2qh8"
192.168.1.79 952x85x2 - [15/Jan/2018:15:52:43 +0530] "GET /static-assets/metal-all.js HTTP/1.1" 200 37523 16 "http://192.168.1.4:8085/secure/errors.jsp" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0" "4x2qh8"
192.168.1.79 953x86x1 - [15/Jan/2018:15:53:33 +0530] "GET /secure/errors.jsp HTTP/1.1" 503 759 16 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0" "4x2qh8"
192.168.1.79 953x87x1 - [15/Jan/2018:15:53:33 +0530] "GET /static-assets/metal-all.css HTTP/1.1" 200 24353 15 "http://192.168.1.4:8085/secure/errors.jsp" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0" "4x2qh8"
192.168.1.79 953x88x2 - [15/Jan/2018:15:53:33 +0530] "GET /static-assets/metal-all.js HTTP/1.1" 200 37523 16 "http://192.168.1.4:8085/secure/errors.jsp" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0" "4x2qh8"
192.168.1.79 954x89x1 - [15/Jan/2018:15:54:23 +0530] "GET /secure/errors.jsp HTTP/1.1" 503 759 16 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0" "4x2qh8"
192.168.1.79 954x90x1 - [15/Jan/2018:15:54:23 +0530] "GET /static-assets/metal-all.css HTTP/1.1" 200 24353 0 "http://192.168.1.4:8085/secure/errors.jsp" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0" "4x2qh8"
192.168.1.79 954x91x2 - [15/Jan/2018:15:54:23 +0530] "GET /static-assets/metal-all.js HTTP/1.1" 200 37523 15 "http://192.168.1.4:8085/secure/errors.jsp" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0" "4x2qh8"
192.168.1.79 955x92x1 - [15/Jan/2018:15:55:14 +0530] "GET /secure/errors.jsp HTTP/1.1" 503 759 15 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0" "4x2qh8"
192.168.1.79 955x93x1 - [15/Jan/2018:15:55:14 +0530] "GET /static-assets/metal-all.css HTTP/1.1" 200 24353 0 "http://192.168.1.4:8085/secure/errors.jsp" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0" "4x2qh8"
192.168.1.79 955x94x1 - [15/Jan/2018:15:55:14 +0530] "GET /static-assets/metal-all.js HTTP/1.1" 200 37523 62 "http://192.168.1.4:8085/secure/errors.jsp" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0" "4x2qh8"
127.0.0.1 984x95x1 - [15/Jan/2018:16:24:25 +0530] "GET / HTTP/1.1" 302 - 16 "-" "Apache-HttpClient/4.4.1 (Java/1.8.0_102)" "-"
127.0.0.1 984x96x1 - [15/Jan/2018:16:24:25 +0530] "GET /secure/errors.jsp HTTP/1.1" 503 759 0 "-" "Apache-HttpClient/4.4.1 (Java/1.8.0_102)" "-"
127.0.0.1 1044x97x1 - [15/Jan/2018:17:24:57 +0530] "GET / HTTP/1.1" 302 - 3953 "-" "Apache-HttpClient/4.4.1 (Java/1.8.0_102)" "-"
127.0.0.1 1044x98x1 - [15/Jan/2018:17:24:58 +0530] "GET /secure/errors.jsp HTTP/1.1" 503 759 999 "-" "Apache-HttpClient/4.4.1 (Java/1.8.0_102)" "-"

[Murthy Namagiri]

Below are the exact logs

2018-01-15 17:57:00,005 Caesium-1-1 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.service.JiraService:10000; runDetails=RunDetailsImpl[startTime=1516019220005,runOutcome=SUCCESS,durationInMillis=0,message=]
2018-01-15 17:57:29,177 Caesium-1-2 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailProcessorJobRunner; runDetails=RunDetailsImpl[startTime=1516019249161,runOutcome=SUCCESS,durationInMillis=16,message=]
2018-01-15 17:57:29,208 Caesium-1-4 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailPullerJobRunner; runDetails=RunDetailsImpl[startTime=1516019249177,runOutcome=SUCCESS,durationInMillis=31,message=]
2018-01-15 17:57:52,949 Caesium-1-3 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=CompatibilityPluginScheduler.JobId.hipchatUpdateGlancesDataJob; runDetails=RunDetailsImpl[startTime=1516019272933,runOutcome=SUCCESS,durationInMillis=16,message=]
2018-01-15 17:58:00,004 Caesium-1-3 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.service.JiraService:10000; runDetails=RunDetailsImpl[startTime=1516019280004,runOutcome=SUCCESS,durationInMillis=0,message=]
2018-01-15 17:58:23,583 Caesium-1-1 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.servicedesk.plugins.automation.internal.execution.engine.executors.PsmqResyncExecutorsJobsScheduler; runDetails=RunDetailsImpl[startTime=1516019303255,runOutcome=SUCCESS,durationInMillis=328,message=]
2018-01-15 17:58:29,176 Caesium-1-2 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailProcessorJobRunner; runDetails=RunDetailsImpl[startTime=1516019309176,runOutcome=SUCCESS,durationInMillis=0,message=]
2018-01-15 17:58:29,239 Caesium-1-1 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailPullerJobRunner; runDetails=RunDetailsImpl[startTime=1516019309192,runOutcome=SUCCESS,durationInMillis=47,message=]
2018-01-15 17:58:52,942 Caesium-1-2 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=CompatibilityPluginScheduler.JobId.hipchatUpdateGlancesDataJob; runDetails=RunDetailsImpl[startTime=1516019332942,runOutcome=SUCCESS,durationInMillis=0,message=]
2018-01-15 17:59:00,002 Caesium-1-3 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.service.JiraService:10000; runDetails=RunDetailsImpl[startTime=1516019340002,runOutcome=SUCCESS,durationInMillis=0,message=]
2018-01-15 17:59:29,189 Caesium-1-1 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailProcessorJobRunner; runDetails=RunDetailsImpl[startTime=1516019369189,runOutcome=SUCCESS,durationInMillis=0,message=]
2018-01-15 17:59:29,299 Caesium-1-4 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailPullerJobRunner; runDetails=RunDetailsImpl[startTime=1516019369205,runOutcome=SUCCESS,durationInMillis=94,message=]
2018-01-15 17:59:52,971 Caesium-1-4 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=CompatibilityPluginScheduler.JobId.hipchatUpdateGlancesDataJob; runDetails=RunDetailsImpl[startTime=1516019392955,runOutcome=SUCCESS,durationInMillis=16,message=]
2018-01-15 18:00:00,002 Caesium-1-1 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.service.JiraService:10000; runDetails=RunDetailsImpl[startTime=1516019400002,runOutcome=SUCCESS,durationInMillis=0,message=]
2018-01-15 18:00:29,534 Caesium-1-2 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailProcessorJobRunner; runDetails=RunDetailsImpl[startTime=1516019429190,runOutcome=SUCCESS,durationInMillis=344,message=]
2018-01-15 18:00:29,549 Caesium-1-1 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailPullerJobRunner; runDetails=RunDetailsImpl[startTime=1516019429206,runOutcome=SUCCESS,durationInMillis=343,message=]
2018-01-15 18:00:53,047 Caesium-1-2 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=CompatibilityPluginScheduler.JobId.hipchatUpdateGlancesDataJob; runDetails=RunDetailsImpl[startTime=1516019452970,runOutcome=SUCCESS,durationInMillis=77,message=]
2018-01-15 18:01:00,002 Caesium-1-1 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.service.JiraService:10000; runDetails=RunDetailsImpl[startTime=1516019460000,runOutcome=SUCCESS,durationInMillis=2,message=]
2018-01-15 18:01:29,219 Caesium-1-1 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailProcessorJobRunner; runDetails=RunDetailsImpl[startTime=1516019489203,runOutcome=SUCCESS,durationInMillis=16,message=]
2018-01-15 18:01:29,234 Caesium-1-2 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailPullerJobRunner; runDetails=RunDetailsImpl[startTime=1516019489219,runOutcome=SUCCESS,durationInMillis=15,message=]
2018-01-15 18:01:53,000 Caesium-1-2 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=CompatibilityPluginScheduler.JobId.hipchatUpdateGlancesDataJob; runDetails=RunDetailsImpl[startTime=1516019512983,runOutcome=SUCCESS,durationInMillis=17,message=]
2018-01-15 18:02:00,009 Caesium-1-2 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.service.JiraService:10000; runDetails=RunDetailsImpl[startTime=1516019520009,runOutcome=SUCCESS,durationInMillis=0,message=]
2018-01-15 18:02:29,328 Caesium-1-1 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailProcessorJobRunner; runDetails=RunDetailsImpl[startTime=1516019549203,runOutcome=SUCCESS,durationInMillis=125,message=]
2018-01-15 18:02:29,344 Caesium-1-4 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailPullerJobRunner; runDetails=RunDetailsImpl[startTime=1516019549235,runOutcome=SUCCESS,durationInMillis=109,message=]
2018-01-15 18:02:53,002 Caesium-1-1 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=CompatibilityPluginScheduler.JobId.hipchatUpdateGlancesDataJob; runDetails=RunDetailsImpl[startTime=1516019572998,runOutcome=SUCCESS,durationInMillis=4,message=]
2018-01-15 18:03:00,011 Caesium-1-1 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.service.JiraService:10000; runDetails=RunDetailsImpl[startTime=1516019580001,runOutcome=SUCCESS,durationInMillis=10,message=]
2018-01-15 18:03:29,233 Caesium-1-4 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailProcessorJobRunner; runDetails=RunDetailsImpl[startTime=1516019609218,runOutcome=SUCCESS,durationInMillis=15,message=]
2018-01-15 18:03:29,280 Caesium-1-1 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailPullerJobRunner; runDetails=RunDetailsImpl[startTime=1516019609249,runOutcome=SUCCESS,durationInMillis=31,message=]
2018-01-15 18:03:53,001 Caesium-1-4 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=CompatibilityPluginScheduler.JobId.hipchatUpdateGlancesDataJob; runDetails=RunDetailsImpl[startTime=1516019632998,runOutcome=SUCCESS,durationInMillis=3,message=]
2018-01-15 18:04:00,012 Caesium-1-4 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.service.JiraService:10000; runDetails=RunDetailsImpl[startTime=1516019640000,runOutcome=SUCCESS,durationInMillis=12,message=]
2018-01-15 18:04:29,247 Caesium-1-1 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailProcessorJobRunner; runDetails=RunDetailsImpl[startTime=1516019669231,runOutcome=SUCCESS,durationInMillis=16,message=]
2018-01-15 18:04:29,293 Caesium-1-1 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailPullerJobRunner; runDetails=RunDetailsImpl[startTime=1516019669262,runOutcome=SUCCESS,durationInMillis=31,message=]

[Murthy Namagiri]

2018-01-15 15:23:25,285 JIRA-Bootstrap INFO      [c.a.p.internal.bootstrap.Launcher] PSMQ is initializing...
2018-01-15 15:23:25,285 JIRA-Bootstrap INFO      [c.a.p.internal.bootstrap.Launcher] PSMQ is initialized.
2018-01-15 15:23:25,363 JIRA-Bootstrap INFO      [c.a.jira.scheduler.JiraSchedulerLauncher] Starting the JIRA Scheduler....
2018-01-15 15:23:25,395 JIRA-Bootstrap ERROR      [c.a.jira.scheduler.JiraSchedulerLauncher] A RuntimeException occurred during JiraSchedulerLauncher servlet context initialisation - Multiple entries with same key: sd.custom.notification.batch.send=Thu Dec 14 17:02:52 IST 2017 and sd.custom.notification.batch.send=Thu Dec 14 17:02:52 IST 2017.
java.lang.IllegalArgumentException: Multiple entries with same key: sd.custom.notification.batch.send=Thu Dec 14 17:02:52 IST 2017 and sd.custom.notification.batch.send=Thu Dec 14 17:02:52 IST 2017
    at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:150)
    at com.google.common.collect.RegularImmutableMap.checkNoConflictInBucket(RegularImmutableMap.java:104)
    at com.google.common.collect.RegularImmutableMap.<init>(RegularImmutableMap.java:70)
    at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:254)
    at com.atlassian.jira.scheduler.OfBizClusteredJobDao.refresh(OfBizClusteredJobDao.java:111)
    at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobsFromDao(SchedulerQueueImpl.java:139)
    at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobsUnderLock(SchedulerQueueImpl.java:130)
    at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobs(SchedulerQueueImpl.java:117)
    at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.refreshClusteredJobs(CaesiumSchedulerService.java:359)
    at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.startImpl(CaesiumSchedulerService.java:275)
    at com.atlassian.jira.scheduler.JiraCaesiumSchedulerService.startImpl(JiraCaesiumSchedulerService.java:36)
    at com.atlassian.scheduler.core.AbstractSchedulerService.start(AbstractSchedulerService.java:180)
    at com.atlassian.jira.scheduler.JiraSchedulerLauncher.proceedIfAllClear(JiraSchedulerLauncher.java:38)
    at com.atlassian.jira.scheduler.JiraSchedulerLauncher.start(JiraSchedulerLauncher.java:24)
    at com.atlassian.jira.startup.ActiveServicesLauncher.start(ActiveServicesLauncher.java:55)
    at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postTenantArrived$4(DefaultJiraLauncher.java:184)
    at com.atlassian.jira.startup.DefaultInstantUpgradeManager$StartupTask.run(DefaultInstantUpgradeManager.java:139)
    at com.atlassian.jira.startup.DefaultInstantUpgradeManager.runTask(DefaultInstantUpgradeManager.java:55)
    at com.atlassian.jira.startup.DefaultInstantUpgradeManager.doNowOrWhenInstanceBecomesActive(DefaultInstantUpgradeManager.java:42)
    at com.atlassian.jira.startup.DefaultJiraLauncher.postTenantArrived(DefaultJiraLauncher.java:177)
    at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDBActivated$3(DefaultJiraLauncher.java:165)
    at com.atlassian.jira.tenancy.DefaultTenantManager.doNowOrWhenTenantArrives(DefaultTenantManager.java:55)
    at com.atlassian.jira.startup.DefaultJiraLauncher.postDBActivated(DefaultJiraLauncher.java:162)
    at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:151)
    at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:298)
    at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:194)
    at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:141)
    at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:103)
    at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
    at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:101)
    at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:149)
    at java.lang.Thread.run(Thread.java:745)
2018-01-15 15:23:25,410 JIRA-Bootstrap ERROR      [c.a.jira.startup.LauncherContextListener] Unable to start JIRA.
java.lang.IllegalArgumentException: Multiple entries with same key: sd.custom.notification.batch.send=Thu Dec 14 17:02:52 IST 2017 and sd.custom.notification.batch.send=Thu Dec 14 17:02:52 IST 2017
    at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:150)
    at com.google.common.collect.RegularImmutableMap.checkNoConflictInBucket(RegularImmutableMap.java:104)
    at com.google.common.collect.RegularImmutableMap.<init>(RegularImmutableMap.java:70)
    at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:254)
    at com.atlassian.jira.scheduler.OfBizClusteredJobDao.refresh(OfBizClusteredJobDao.java:111)
    at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobsFromDao(SchedulerQueueImpl.java:139)
    at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobsUnderLock(SchedulerQueueImpl.java:130)
    at com.atlassian.scheduler.caesium.impl.SchedulerQueueImpl.refreshClusteredJobs(SchedulerQueueImpl.java:117)
    at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.refreshClusteredJobs(CaesiumSchedulerService.java:359)
    at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.startImpl(CaesiumSchedulerService.java:275)
    at com.atlassian.jira.scheduler.JiraCaesiumSchedulerService.startImpl(JiraCaesiumSchedulerService.java:36)
    at com.atlassian.scheduler.core.AbstractSchedulerService.start(AbstractSchedulerService.java:180)
    at com.atlassian.jira.scheduler.JiraSchedulerLauncher.proceedIfAllClear(JiraSchedulerLauncher.java:38)
    at com.atlassian.jira.scheduler.JiraSchedulerLauncher.start(JiraSchedulerLauncher.java:24)
    at com.atlassian.jira.startup.ActiveServicesLauncher.start(ActiveServicesLauncher.java:55)
    at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postTenantArrived$4(DefaultJiraLauncher.java:184)
    at com.atlassian.jira.startup.DefaultInstantUpgradeManager$StartupTask.run(DefaultInstantUpgradeManager.java:139)
    at com.atlassian.jira.startup.DefaultInstantUpgradeManager.runTask(DefaultInstantUpgradeManager.java:55)
    at com.atlassian.jira.startup.DefaultInstantUpgradeManager.doNowOrWhenInstanceBecomesActive(DefaultInstantUpgradeManager.java:42)
    at com.atlassian.jira.startup.DefaultJiraLauncher.postTenantArrived(DefaultJiraLauncher.java:177)
    at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDBActivated$3(DefaultJiraLauncher.java:165)
    at com.atlassian.jira.tenancy.DefaultTenantManager.doNowOrWhenTenantArrives(DefaultTenantManager.java:55)
    at com.atlassian.jira.startup.DefaultJiraLauncher.postDBActivated(DefaultJiraLauncher.java:162)
    at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:151)
    at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:298)
    at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:194)
    at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:141)
    at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:103)
    at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
    at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:101)
    at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:149)
    at java.lang.Thread.run(Thread.java:745)
2018-01-15 15:23:25,426 Caesium-1-2 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.user.UserHistoryDatabaseCompactor; runDetails=RunDetailsImpl[startTime=1516010005395,runOutcome=SUCCESS,durationInMillis=31,message=]
2018-01-15 15:23:25,457 JIRA-Bootstrap INFO      [c.a.jira.startup.LauncherContextListener] Memory Usage:
    ---------------------------------------------------------------------------------
      Heap memory     :  Used:  360 MiB.  Committed:  672 MiB.  Max:  733 MiB
      Non-heap memory :  Used:  253 MiB.  Committed:  275 MiB.  Max: 1264 MiB
    ---------------------------------------------------------------------------------
      TOTAL           :  Used:  613 MiB.  Committed:  947 MiB.  Max: 1997 MiB
    ---------------------------------------------------------------------------------
2018-01-15 15:23:25,457 Caesium-1-1 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.cache.monitor.CacheStatisticsMonitor; runDetails=RunDetailsImpl[startTime=1516010005395,runOutcome=SUCCESS,durationInMillis=62,message=]
2018-01-15 15:23:25,488 Caesium-1-1 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=CompatibilityPluginScheduler.JobId.BundledUpdateCheckJob-job; runDetails=RunDetailsImpl[startTime=1516010005457,runOutcome=SUCCESS,durationInMillis=31,message=]
2018-01-15 15:23:25,488 Caesium-1-4 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.scheduler.timedpromise.internal.TimedPromiseExecutor.job; runDetails=RunDetailsImpl[startTime=1516010005395,runOutcome=SUCCESS,durationInMillis=93,message=]
2018-01-15 15:23:25,504 Caesium-1-3 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.scheduler.timedpromise.TimedPromiseHistoryPruner; runDetails=RunDetailsImpl[startTime=1516010005395,runOutcome=SUCCESS,durationInMillis=109,message=]
2018-01-15 15:23:25,520 Caesium-1-3 DEBUG      [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.jira.jira-email-processor-plugin]
2018-01-15 15:23:25,551 Caesium-1-2 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.scheduler.timedpromise.internal.TimedPromisePruner; runDetails=RunDetailsImpl[startTime=1516010005426,runOutcome=SUCCESS,durationInMillis=125,message=]
2018-01-15 15:23:25,645 Caesium-1-4 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailProcessorJobRunner; runDetails=RunDetailsImpl[startTime=1516010005504,runOutcome=SUCCESS,durationInMillis=141,message=]
2018-01-15 15:23:25,660 Caesium-1-1 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailPullerJobRunner; runDetails=RunDetailsImpl[startTime=1516010005488,runOutcome=SUCCESS,durationInMillis=172,message=]
2018-01-15 15:23:25,660 Caesium-1-3 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailCleanerJobRunner; runDetails=RunDetailsImpl[startTime=1516010005504,runOutcome=SUCCESS,durationInMillis=156,message=]
2018-01-15 15:23:35,960 UpmScheduler:thread-1 WARN anonymous     [c.a.upm.pac.PacClientImpl] Timed out while checking for addon updates: com.atlassian.util.concurrent.RuntimeTimeoutException: com.atlassian.util.concurrent.TimedOutException: Timed out after: 10000000000 NANOSECONDS
2018-01-15 15:23:38,256 Caesium-1-3 INFO      [c.a.j.p.h.service.connect.InstallGlancesJobHandler] Running InstallGlancesJobHandler...
2018-01-15 15:23:38,256 Caesium-1-3 DEBUG      [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.plugins.base-hipchat-integration-plugin-api]
2018-01-15 15:23:38,271 Caesium-1-3 INFO      [c.a.j.p.h.service.connect.InstallGlancesJobHandler] There is no link to HipChat, no need to install glances.
2018-01-15 15:23:38,271 Caesium-1-3 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=CompatibilityPluginScheduler.JobId.hipchatInstallGlancesJob; runDetails=RunDetailsImpl[startTime=1516010018256,runOutcome=SUCCESS,durationInMillis=15,message=]
2018-01-15 15:23:40,410 Caesium-1-1 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=JiraPluginScheduler:com.atlassian.analytics.client.upload.RemoteFilterRead:job; runDetails=RunDetailsImpl[startTime=1516010013114,runOutcome=SUCCESS,durationInMillis=7296,message=]
2018-01-15 15:23:49,895 Caesium-1-3 INFO      [c.a.j.p.h.service.ping.RefreshConnectionStatusJobHandler] Running RefreshConnectionStatusJobHandler...
2018-01-15 15:23:49,895 Caesium-1-3 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=CompatibilityPluginScheduler.JobId.hipchatRefreshConnectionStatusJob; runDetails=RunDetailsImpl[startTime=1516010029895,runOutcome=SUCCESS,durationInMillis=0,message=]
2018-01-15 15:23:51,285 Caesium-1-3 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=CompatibilityPluginScheduler.JobId.hipchatUpdateGlancesDataJob; runDetails=RunDetailsImpl[startTime=1516010031285,runOutcome=SUCCESS,durationInMillis=0,message=]
2018-01-15 15:24:00,004 Caesium-1-2 WARN ServiceRunner     [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.service.JiraService:10000; runDetails=RunDetailsImpl[startTime=1516010040004,runOutcome=SUCCESS,durationInMillis=0,message=]
2018-01-15 15:24:17,192 Caesium-1-4 INFO      [c.a.jira.upgrade.UpgradeScheduler] Running schedules upgrades
2018-01-15 15:24:17,223 Caesium-1-4 INFO      [c.a.jira.upgrade.LoggingUpgradeService] run upgrades has started
2018-01-15 15:24:18,583 Caesium-1-4 INFO      [c.a.j.upgrade.util.BuildNumberDao] Setting current build number to 73017
2018-01-15 15:24:18,615 Caesium-1-4 INFO      [c.a.jira.upgrade.UpgradeIndexManager] There is no reindex requests of type [IMMEDIATE, DELAYED] so none will be run
2018-01-15 15:24:18,615 Caesium-1-4 INFO      [c.a.j.upgrade.util.BuildNumberDao] Setting current version to 7.3.6
2018-01-15 15:24:18,630 Caesium-1-4 INFO      [c.a.j.upgrade.util.BuildNumberDao] Setting downgrade version to 7.1.2
2018-01-15 15:24:18,646 Caesium-1-4 INFO      [c.a.jira.upgrade.LoggingUpgradeService] run upgrades has finished successfully, and took 1423 milliseconds to process.
2018-01-15 15:24:18,646 Caesium-1-4 INFO      [c.a.jira.upgrade.UpgradeScheduler] Schedules upgrades completed successfully
2018-01-15 15:24:18,740 Caesium-1-4 WARN      [c.a.j.cluster.zdu.DefaultClusterUpgradeStateManager] Unexpected JiraUpgradedEvent. Cluster wasn't in RUNNING_UPGRADE_TASKS state
2018-01-15 15:24:18,755 Caesium-1-4 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.upgrade.UpgradeService; runDetails=RunDetailsImpl[startTime=1516010057192,runOutcome=SUCCESS,durationInMillis=1563,message=]
2018-01-15 15:24:25,535 Caesium-1-3 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailPullerJobRunner; runDetails=RunDetailsImpl[startTime=1516010065503,runOutcome=SUCCESS,durationInMillis=32,message=]
2018-01-15 15:24:25,535 Caesium-1-4 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=com.atlassian.jira.internal.mail.services.MailProcessorJobRunner; runDetails=RunDetailsImpl[startTime=1516010065519,runOutcome=SUCCESS,durationInMillis=16,message=]
2018-01-15 15:24:26,550 Caesium-1-1 DEBUG      [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.support.healthcheck.support-healthcheck-plugin]
2018-01-15 15:24:28,243 LuceneSupportHealthCheck.Searcher: WARN      [c.a.jira.security.GlobalPermissionsCache] Could not add permission Global Permission: [group=jira-administrators][permissionKey=ADMINISTER] - it already existed?
2018-01-15 15:24:28,243 LuceneSupportHealthCheck.Searcher: WARN      [c.a.jira.security.GlobalPermissionsCache] Could not add permission Global Permission: [group=jira-administrators][permissionKey=SYSTEM_ADMIN] - it already existed?
2018-01-15 15:24:29,254 Caesium-1-1 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=JiraPluginScheduler:com.atlassian.support.healthcheck.scheduler.HealthCheckSchedulerImpl:job; runDetails=RunDetailsImpl[startTime=1516010064864,runOutcome=SUCCESS,durationInMillis=4390,message=]
2018-01-15 15:24:51,285 Caesium-1-1 WARN      [c.a.jira.scheduler.OfBizRunDetailsDao] Cannot record run details when the scheduling system is not started: jobId=CompatibilityPluginScheduler.JobId.hipchatUpdateGlancesDataJob; runDetails=RunDetailsImpl[startTime=1516010091285,runOutcome=SUCCESS,durationInMillis=0,message=]
2018-01-15 15:24:53,660 Modification Check:thread-1 INFO      [c.a.jira.startup.JiraStartupLogger]
    
    ___ Modifications ___________________________
    
         Modified Files                                : jira-application.properties
         Removed Files                                 : None