JIRA Startup Failed - Unable to either create or replace clustered job

Hi,

I have run JIRA using docker image ahaasler/jira:7.3.5 together with PostgresDB.
After crash of one machnie I have problem with JIRA Startup.

on WebGUI i received

Error occurred while starting component 'com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor'. caused by: com.atlassian.scheduler.SchedulerServiceException: Unable to either create or replace clustered job: ImmutableClusteredJob[jobId=com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor,jobRunnerKey=com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor,schedule=Schedule[type=INTERVAL,intervalScheduleInfo=IntervalScheduleInfo[firstRunTime=Fri Dec 08 22:49:34 GMT 2017,intervalInMillis=3600000]],nextRunTime=Fri Dec 08 22:49:34 GMT 2017,version=1,rawParameters=(null)] 

in atlassian-jira.log

    ___ FAILED PLUGIN REPORT _____________________

    1 plugin failed to load during JIRA startup.

        'com.atlassian.plugins.authentication.atlassian-authentication-plugin' - 'SAML for Atlassian Data Center'  failed to load.
                Error creating bean with name 'statisticsCollectionService': Invocation of init method failed; nested exception is com.atlassian.scheduler.SchedulerServiceException: Unable to either create or replace clustered job: ImmutableClusteredJob[jobId=analytics-collection,jobRunnerKey=com.atlassian.plugins.authentication.impl.analytics.StatisticsCollectionService,schedule=Schedule[type=CRON_EXPRESSION,cronScheduleInfo=CronScheduleInfo[cronExpression='0 0 23 * * ?',timeZone=null]],nextRunTime=Fri Dec 08 23:00:00 GMT 2017,version=1,rawParameters=(null)]
                        Unable to either create or replace clustered job: ImmutableClusteredJob[jobId=analytics-collection,jobRunnerKey=com.atlassian.plugins.authentication.impl.analytics.StatisticsCollectionService,schedule=Schedule[type=CRON_EXPRESSION,cronScheduleInfo=CronScheduleInfo[cronExpression='0 0 23 * * ?',timeZone=null]],nextRunTime=Fri Dec 08 23:00:00 GMT 2017,version=1,rawParameters=(null)]

                It was loaded from /opt/jira/atlassian-jira/WEB-INF/atlassian-bundled-plugins/atlassian-authentication-plugin-2.0.3.jar

    ********************************************************************************************************************************************************************************************************

 Even I disabled atlassian-authentication-plugin by sql on SB

INSERT into pluginstate (pluginkey, pluginenabled) VALUES ('com.atlassian.plugins.authentication.atlassian-authentication-plugin','false');

I got:

    ___ Plugin System Started _________________

2017-12-08 22:17:23,107 JIRA-Bootstrap ERROR      [c.a.jira.upgrade.UpgradeLauncher] Skipping, JIRA is locked.
2017-12-08 22:17:23,107 JIRA-Bootstrap INFO      [c.a.jira.scheduler.JiraSchedulerLauncher] JIRA Scheduler not started: JIRA startup checklist failed.
2017-12-08 22:17:23,188 JIRA-Bootstrap ERROR      [c.a.jira.startup.AnalyticsLauncher] Start analytics not scheduled
com.atlassian.scheduler.SchedulerServiceException: Unable to either create or replace clustered job: ImmutableClusteredJob[jobId=c3cb8b50-4fb3-4f7b-a2a1-9928cb5e9198,jobRunnerKey=com.atlassian.jira.startup.AnalyticsLauncher.Start,schedule=Schedule[type=INTERVAL,intervalScheduleInfo=IntervalScheduleInfo[firstRunTime=Fri Dec 08 22:20:23 GMT 2017,intervalInMillis=0]],nextRunTime=Fri Dec 08 22:20:23 GMT 2017,version=1,rawParameters=(null)]
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.createOrReplaceWithRetry(CaesiumSchedulerService.java:202)
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.scheduleClusteredJob(CaesiumSchedulerService.java:189)
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.scheduleJob(CaesiumSchedulerService.java:137)
        at com.atlassian.scheduler.core.AbstractSchedulerService.scheduleJobWithGeneratedId(AbstractSchedulerService.java:114)
        at com.atlassian.scheduler.core.DelegatingSchedulerService.scheduleJobWithGeneratedId(DelegatingSchedulerService.java:77)
        at com.atlassian.jira.startup.AnalyticsLauncher.runStartAnalyticsJob(AnalyticsLauncher.java:72)
        at com.atlassian.jira.startup.AnalyticsLauncher.start(AnalyticsLauncher.java:40)
        at com.atlassian.jira.startup.ActiveServicesLauncher.start(ActiveServicesLauncher.java:56)
        at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postTenantArrived$4(DefaultJiraLauncher.java:184)
        at com.atlassian.jira.startup.DefaultInstantUpgradeManager$StartupTask.run(DefaultInstantUpgradeManager.java:139)
        at com.atlassian.jira.startup.DefaultInstantUpgradeManager.runTask(DefaultInstantUpgradeManager.java:55)
        at com.atlassian.jira.startup.DefaultInstantUpgradeManager.doNowOrWhenInstanceBecomesActive(DefaultInstantUpgradeManager.java:42)
        at com.atlassian.jira.startup.DefaultJiraLauncher.postTenantArrived(DefaultJiraLauncher.java:177)
        at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDBActivated$3(DefaultJiraLauncher.java:165)
        at com.atlassian.jira.tenancy.DefaultTenantManager.doNowOrWhenTenantArrives(DefaultTenantManager.java:55)
        at com.atlassian.jira.startup.DefaultJiraLauncher.postDBActivated(DefaultJiraLauncher.java:162)
        at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:151)
        at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:298)
        at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:194)
        at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:141)
        at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:103)
        at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
        at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:101)
        at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:149)
        at java.lang.Thread.run(Thread.java:745)
2017-12-08 22:17:23,256 JIRA-Bootstrap ERROR      [c.a.jira.startup.IndexRecoveryLauncher] Index recovery not scheduled
com.atlassian.scheduler.SchedulerServiceException: Unable to either create or replace clustered job: ImmutableClusteredJob[jobId=fe6042ca-eadc-4b92-804d-a945d3e45b4f,jobRunnerKey=com.atlassian.jira.DefaultJiraLauncher.IndexRecovery,schedule=Schedule[type=INTERVAL,intervalScheduleInfo=IntervalScheduleInfo[firstRunTime=Fri Dec 08 22:17:23 GMT 2017,intervalInMillis=0]],nextRunTime=Fri Dec 08 22:17:23 GMT 2017,version=1,rawParameters=(null)]
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.createOrReplaceWithRetry(CaesiumSchedulerService.java:202)
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.scheduleClusteredJob(CaesiumSchedulerService.java:189)
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.scheduleJob(CaesiumSchedulerService.java:137)
        at com.atlassian.scheduler.core.AbstractSchedulerService.scheduleJobWithGeneratedId(AbstractSchedulerService.java:114)
        at com.atlassian.scheduler.core.DelegatingSchedulerService.scheduleJobWithGeneratedId(DelegatingSchedulerService.java:77)
        at com.atlassian.jira.startup.IndexRecoveryLauncher.runIndexRecoveryJob(IndexRecoveryLauncher.java:102)
        at com.atlassian.jira.startup.IndexRecoveryLauncher.start(IndexRecoveryLauncher.java:49)
        at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postTenantArrived$4(DefaultJiraLauncher.java:187)
        at com.atlassian.jira.startup.DefaultInstantUpgradeManager$StartupTask.run(DefaultInstantUpgradeManager.java:139)
        at com.atlassian.jira.startup.DefaultInstantUpgradeManager.runTask(DefaultInstantUpgradeManager.java:55)
        at com.atlassian.jira.startup.DefaultInstantUpgradeManager.doNowOrWhenInstanceBecomesActive(DefaultInstantUpgradeManager.java:42)
        at com.atlassian.jira.startup.DefaultJiraLauncher.postTenantArrived(DefaultJiraLauncher.java:177)
        at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDBActivated$3(DefaultJiraLauncher.java:165)
        at com.atlassian.jira.tenancy.DefaultTenantManager.doNowOrWhenTenantArrives(DefaultTenantManager.java:55)
        at com.atlassian.jira.startup.DefaultJiraLauncher.postDBActivated(DefaultJiraLauncher.java:162)
        at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:151)
        at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:298)
        at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:194)
        at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:141)
        at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:103)
        at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
        at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:101)
        at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:149)
        at java.lang.Thread.run(Thread.java:745)
2017-12-08 22:17:23,263 JIRA-Bootstrap INFO      [c.a.jira.startup.DefaultInstantUpgradeManager] Late startup launchers took 0s
2017-12-08 22:17:23,274 JIRA-Bootstrap ERROR      [c.a.jira.startup.LauncherContextListener] Unable to start JIRA.
java.lang.IllegalStateException: Abnormal system startup detected
        at com.atlassian.jira.startup.LauncherContextListener.initDone(LauncherContextListener.java:180)
        at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:162)
        at java.lang.Thread.run(Thread.java:745)
2017-12-08 22:17:23,281 JIRA-Bootstrap INFO      [c.a.jira.startup.LauncherContextListener] Memory Usage:
    ---------------------------------------------------------------------------------
      Heap memory     :  Used:  328 MiB.  Committed:  638 MiB.  Max:  747 MiB
      Non-heap memory :  Used:  189 MiB.  Committed:  206 MiB.  Max: 1264 MiB
    ---------------------------------------------------------------------------------
      TOTAL           :  Used:  516 MiB.  Committed:  844 MiB.  Max: 2011 MiB
    ---------------------------------------------------------------------------------
2017-12-08 22:17:28,111 Modification Check:thread-1 INFO      [c.a.jira.startup.JiraStartupLogger]

    ___ Modifications ___________________________

         Modified Files                                : None
         Removed Files                                 : None

Eny sugestions?

2 answers

1 accepted

Unfortunately,

jiradb=> SELECT * FROM clusteredjob WHERE job_id in (SELECT job_id FROM clusteredjob GROUP BY job_id HAVING COUNT(*)>1);
 id | job_id | job_runner_key | sched_type | interval_millis | first_run | cron_expression | time_zone | next_run | version | parameters
----+--------+----------------+------------+-----------------+-----------+-----------------+-----------+----------+---------+------------
(0 rows)

 Even after removeing all records from clustereddjob (I have made a backup) still the same.

2017-12-08 22:09:07,926 JIRA-Bootstrap ERROR      [c.atlassian.jira.ComponentManager] Error occurred while starting component 'com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor'.
java.lang.RuntimeException: com.atlassian.scheduler.SchedulerServiceException: Unable to either create or replace clustered job: ImmutableClusteredJob[jobId=com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor,jobRunnerKey=com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor,schedule=Schedule[type=INTERVAL,intervalScheduleInfo=IntervalScheduleInfo[firstRunTime=Fri Dec 08 23:09:07 GMT 2017,intervalInMillis=3600000]],nextRunTime=Fri Dec 08 23:09:07 GMT 2017,version=1,rawParameters=(null)]
        at com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor.scheduleCleanup(DefaultTemporaryWebAttachmentsMonitor.java:75)
        at com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor.start(DefaultTemporaryWebAttachmentsMonitor.java:61)
        at com.atlassian.jira.ComponentManager.runStartable(ComponentManager.java:311)
        at com.atlassian.jira.ComponentManager.quickStart(ComponentManager.java:224)
        at com.atlassian.jira.ComponentManager.start(ComponentManager.java:164)
        at com.atlassian.jira.upgrade.PluginSystemLauncher.start(PluginSystemLauncher.java:43)
        at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:150)
        at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:298)
        at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:194)
        at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:141)
        at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:103)
        at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
        at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:101)
        at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:149)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.atlassian.scheduler.SchedulerServiceException: Unable to either create or replace clustered job: ImmutableClusteredJob[jobId=com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor,jobRunnerKey=com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor,schedule=Schedule[type=INTERVAL,intervalScheduleInfo=IntervalScheduleInfo[firstRunTime=Fri Dec 08 23:09:07 GMT 2017,intervalInMillis=3600000]],nextRunTime=Fri Dec 08 23:09:07 GMT 2017,version=1,rawParameters=(null)]
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.createOrReplaceWithRetry(CaesiumSchedulerService.java:202)
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.scheduleClusteredJob(CaesiumSchedulerService.java:189)
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.scheduleJob(CaesiumSchedulerService.java:137)
        at com.atlassian.scheduler.core.DelegatingSchedulerService.scheduleJob(DelegatingSchedulerService.java:70)
        at com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor.scheduleCleanup(DefaultTemporaryWebAttachmentsMonitor.java:73)
2017-12-08 22:09:07,934 JIRA-Bootstrap ERROR      [c.a.jira.upgrade.PluginSystemLauncher] A fatal error occured during initialisation. JIRA has been locked.

My apologies, while your error is similar to that bug, you seem to have two different errors that are unable to update/create new entries to that table.  But from looking at the errors, your instance looks to have two completely different keys that cannot be added.  Also it doesn't appear so far that there exist duplicate keys as the reason these can't be added. 

You also mentioned that there was a crash in one the machines.  Could you elaborate on that?  I'm interested to learn more about the event that first caused this problem.

So I think it would be best to jump back and make sure that the SQL user account Jira is using to connect to this database still has the needed permissions to be able to create/update the database as needed.   I would recommend tracing back through the steps in Connecting JIRA applications to PostgreSQL

It might be helpful here to try to create a new empty database, and a new sql user.   You could then shutdown Jira, edit the dbconfig.xml file to tell Jira to use this new empty database, and from there you could then start up Jira.   When Jira starts up connected to an empty database, it launches the setup wizard.  In this setup wizard there is an option to import from a backup.  By default Jira is still creating these xml backup zip files in the $JIRAHOME/export/ directory.   Provided you have a backup zip file here, you could then just copy this zip file to your $JIRAHOME/import/ directory and then in the setup wizard you can import this backup as a means to copy your Jira data into this new empty database.   If the old database or user has in some way become corrupted, these steps should allow you to still maintain your data and get around that problem.

Thanks Andrew for your comments.

I have made reindaxation of database on vocuum on clusteredjob table.

jiradb=# reindex database "jiradb";
jiradb=# vacuum analyze public.clusteredjob

After restarting, JIRA works.

Of course additional consistancy check and fix were dane by me but fortunately using JIRA GUI.

Thx

Hi Lukasz,

Thanks for letting us know what the solution here was.  I would not have expected that specific procedure to fix this problem, but you can't argue with results.

Cheers,

Andy

Hi Lukasz,

Sorry to hear that you are having this problem.  However I believe this is a known bug in Jira we are tracking in https://jira.atlassian.com/browse/JRASERVER-64325

As such I would recommend following the KB JIRA Fails to Start due to Too many rows found for query on ClusteredJob.  You will need to run this SQL query to find the duplicate rows in that table first:

SELECT * FROM clusteredjob WHERE job_id in (SELECT job_id FROM clusteredjob GROUP BY job_id HAVING COUNT(*)>1)

And from there you would have to manually remove one of these duplicates.  In the case of the KB the duplicate had an id of 11106.

Resolution

  1. Shut down JIRA and back up the database properly

  2. Delete either of the entries:

    delete from clusteredjob where id = 11106;
  3. For Oracle DB, ensure run commit; after the above.
  4. Restart JIRA

Also you will need to re-enable that plugin in the pluginstate table.  I would recommend doing both SQL steps at the same time while Jira is shutdown.  Then you should just be able to start Jira up again in order to resolve this problem.

Regards,
Andy

Suggest an answer

Log in or Sign up to answer
How to earn badges on the Atlassian Community

How to earn badges on the Atlassian Community

Badges are a great way to show off community activity, whether you’re a newbie or a Champion.

Learn more
Community showcase
Published Thursday in Marketplace Apps

Tips on how to choose the best estimation method for your planning

Planning and grooming sessions all come with their own sets of rules. Team members meet to estimate stories or other work items, all according to an agreed-upon process. And with every session comes ...

67 views 0 11
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you