JIRA Startup Failed - Unable to either create or replace clustered job

Lukasz Zalewski December 8, 2017

Hi,

I have run JIRA using docker image ahaasler/jira:7.3.5 together with PostgresDB.
After crash of one machnie I have problem with JIRA Startup.

on WebGUI i received

Error occurred while starting component 'com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor'. caused by: com.atlassian.scheduler.SchedulerServiceException: Unable to either create or replace clustered job: ImmutableClusteredJob[jobId=com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor,jobRunnerKey=com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor,schedule=Schedule[type=INTERVAL,intervalScheduleInfo=IntervalScheduleInfo[firstRunTime=Fri Dec 08 22:49:34 GMT 2017,intervalInMillis=3600000]],nextRunTime=Fri Dec 08 22:49:34 GMT 2017,version=1,rawParameters=(null)] 

in atlassian-jira.log

    ___ FAILED PLUGIN REPORT _____________________

    1 plugin failed to load during JIRA startup.

        'com.atlassian.plugins.authentication.atlassian-authentication-plugin' - 'SAML for Atlassian Data Center'  failed to load.
                Error creating bean with name 'statisticsCollectionService': Invocation of init method failed; nested exception is com.atlassian.scheduler.SchedulerServiceException: Unable to either create or replace clustered job: ImmutableClusteredJob[jobId=analytics-collection,jobRunnerKey=com.atlassian.plugins.authentication.impl.analytics.StatisticsCollectionService,schedule=Schedule[type=CRON_EXPRESSION,cronScheduleInfo=CronScheduleInfo[cronExpression='0 0 23 * * ?',timeZone=null]],nextRunTime=Fri Dec 08 23:00:00 GMT 2017,version=1,rawParameters=(null)]
                        Unable to either create or replace clustered job: ImmutableClusteredJob[jobId=analytics-collection,jobRunnerKey=com.atlassian.plugins.authentication.impl.analytics.StatisticsCollectionService,schedule=Schedule[type=CRON_EXPRESSION,cronScheduleInfo=CronScheduleInfo[cronExpression='0 0 23 * * ?',timeZone=null]],nextRunTime=Fri Dec 08 23:00:00 GMT 2017,version=1,rawParameters=(null)]

                It was loaded from /opt/jira/atlassian-jira/WEB-INF/atlassian-bundled-plugins/atlassian-authentication-plugin-2.0.3.jar

    ********************************************************************************************************************************************************************************************************

 Even I disabled atlassian-authentication-plugin by sql on SB

INSERT into pluginstate (pluginkey, pluginenabled) VALUES ('com.atlassian.plugins.authentication.atlassian-authentication-plugin','false');

I got:

    ___ Plugin System Started _________________

2017-12-08 22:17:23,107 JIRA-Bootstrap ERROR      [c.a.jira.upgrade.UpgradeLauncher] Skipping, JIRA is locked.
2017-12-08 22:17:23,107 JIRA-Bootstrap INFO      [c.a.jira.scheduler.JiraSchedulerLauncher] JIRA Scheduler not started: JIRA startup checklist failed.
2017-12-08 22:17:23,188 JIRA-Bootstrap ERROR      [c.a.jira.startup.AnalyticsLauncher] Start analytics not scheduled
com.atlassian.scheduler.SchedulerServiceException: Unable to either create or replace clustered job: ImmutableClusteredJob[jobId=c3cb8b50-4fb3-4f7b-a2a1-9928cb5e9198,jobRunnerKey=com.atlassian.jira.startup.AnalyticsLauncher.Start,schedule=Schedule[type=INTERVAL,intervalScheduleInfo=IntervalScheduleInfo[firstRunTime=Fri Dec 08 22:20:23 GMT 2017,intervalInMillis=0]],nextRunTime=Fri Dec 08 22:20:23 GMT 2017,version=1,rawParameters=(null)]
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.createOrReplaceWithRetry(CaesiumSchedulerService.java:202)
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.scheduleClusteredJob(CaesiumSchedulerService.java:189)
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.scheduleJob(CaesiumSchedulerService.java:137)
        at com.atlassian.scheduler.core.AbstractSchedulerService.scheduleJobWithGeneratedId(AbstractSchedulerService.java:114)
        at com.atlassian.scheduler.core.DelegatingSchedulerService.scheduleJobWithGeneratedId(DelegatingSchedulerService.java:77)
        at com.atlassian.jira.startup.AnalyticsLauncher.runStartAnalyticsJob(AnalyticsLauncher.java:72)
        at com.atlassian.jira.startup.AnalyticsLauncher.start(AnalyticsLauncher.java:40)
        at com.atlassian.jira.startup.ActiveServicesLauncher.start(ActiveServicesLauncher.java:56)
        at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postTenantArrived$4(DefaultJiraLauncher.java:184)
        at com.atlassian.jira.startup.DefaultInstantUpgradeManager$StartupTask.run(DefaultInstantUpgradeManager.java:139)
        at com.atlassian.jira.startup.DefaultInstantUpgradeManager.runTask(DefaultInstantUpgradeManager.java:55)
        at com.atlassian.jira.startup.DefaultInstantUpgradeManager.doNowOrWhenInstanceBecomesActive(DefaultInstantUpgradeManager.java:42)
        at com.atlassian.jira.startup.DefaultJiraLauncher.postTenantArrived(DefaultJiraLauncher.java:177)
        at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDBActivated$3(DefaultJiraLauncher.java:165)
        at com.atlassian.jira.tenancy.DefaultTenantManager.doNowOrWhenTenantArrives(DefaultTenantManager.java:55)
        at com.atlassian.jira.startup.DefaultJiraLauncher.postDBActivated(DefaultJiraLauncher.java:162)
        at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:151)
        at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:298)
        at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:194)
        at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:141)
        at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:103)
        at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
        at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:101)
        at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:149)
        at java.lang.Thread.run(Thread.java:745)
2017-12-08 22:17:23,256 JIRA-Bootstrap ERROR      [c.a.jira.startup.IndexRecoveryLauncher] Index recovery not scheduled
com.atlassian.scheduler.SchedulerServiceException: Unable to either create or replace clustered job: ImmutableClusteredJob[jobId=fe6042ca-eadc-4b92-804d-a945d3e45b4f,jobRunnerKey=com.atlassian.jira.DefaultJiraLauncher.IndexRecovery,schedule=Schedule[type=INTERVAL,intervalScheduleInfo=IntervalScheduleInfo[firstRunTime=Fri Dec 08 22:17:23 GMT 2017,intervalInMillis=0]],nextRunTime=Fri Dec 08 22:17:23 GMT 2017,version=1,rawParameters=(null)]
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.createOrReplaceWithRetry(CaesiumSchedulerService.java:202)
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.scheduleClusteredJob(CaesiumSchedulerService.java:189)
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.scheduleJob(CaesiumSchedulerService.java:137)
        at com.atlassian.scheduler.core.AbstractSchedulerService.scheduleJobWithGeneratedId(AbstractSchedulerService.java:114)
        at com.atlassian.scheduler.core.DelegatingSchedulerService.scheduleJobWithGeneratedId(DelegatingSchedulerService.java:77)
        at com.atlassian.jira.startup.IndexRecoveryLauncher.runIndexRecoveryJob(IndexRecoveryLauncher.java:102)
        at com.atlassian.jira.startup.IndexRecoveryLauncher.start(IndexRecoveryLauncher.java:49)
        at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postTenantArrived$4(DefaultJiraLauncher.java:187)
        at com.atlassian.jira.startup.DefaultInstantUpgradeManager$StartupTask.run(DefaultInstantUpgradeManager.java:139)
        at com.atlassian.jira.startup.DefaultInstantUpgradeManager.runTask(DefaultInstantUpgradeManager.java:55)
        at com.atlassian.jira.startup.DefaultInstantUpgradeManager.doNowOrWhenInstanceBecomesActive(DefaultInstantUpgradeManager.java:42)
        at com.atlassian.jira.startup.DefaultJiraLauncher.postTenantArrived(DefaultJiraLauncher.java:177)
        at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDBActivated$3(DefaultJiraLauncher.java:165)
        at com.atlassian.jira.tenancy.DefaultTenantManager.doNowOrWhenTenantArrives(DefaultTenantManager.java:55)
        at com.atlassian.jira.startup.DefaultJiraLauncher.postDBActivated(DefaultJiraLauncher.java:162)
        at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:151)
        at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:298)
        at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:194)
        at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:141)
        at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:103)
        at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
        at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:101)
        at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:149)
        at java.lang.Thread.run(Thread.java:745)
2017-12-08 22:17:23,263 JIRA-Bootstrap INFO      [c.a.jira.startup.DefaultInstantUpgradeManager] Late startup launchers took 0s
2017-12-08 22:17:23,274 JIRA-Bootstrap ERROR      [c.a.jira.startup.LauncherContextListener] Unable to start JIRA.
java.lang.IllegalStateException: Abnormal system startup detected
        at com.atlassian.jira.startup.LauncherContextListener.initDone(LauncherContextListener.java:180)
        at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:162)
        at java.lang.Thread.run(Thread.java:745)
2017-12-08 22:17:23,281 JIRA-Bootstrap INFO      [c.a.jira.startup.LauncherContextListener] Memory Usage:
    ---------------------------------------------------------------------------------
      Heap memory     :  Used:  328 MiB.  Committed:  638 MiB.  Max:  747 MiB
      Non-heap memory :  Used:  189 MiB.  Committed:  206 MiB.  Max: 1264 MiB
    ---------------------------------------------------------------------------------
      TOTAL           :  Used:  516 MiB.  Committed:  844 MiB.  Max: 2011 MiB
    ---------------------------------------------------------------------------------
2017-12-08 22:17:28,111 Modification Check:thread-1 INFO      [c.a.jira.startup.JiraStartupLogger]

    ___ Modifications ___________________________

         Modified Files                                : None
         Removed Files                                 : None

Eny sugestions?

5 answers

1 accepted

0 votes
Answer accepted
Lukasz Zalewski December 12, 2017

Unfortunately,

jiradb=> SELECT * FROM clusteredjob WHERE job_id in (SELECT job_id FROM clusteredjob GROUP BY job_id HAVING COUNT(*)>1);
 id | job_id | job_runner_key | sched_type | interval_millis | first_run | cron_expression | time_zone | next_run | version | parameters
----+--------+----------------+------------+-----------------+-----------+-----------------+-----------+----------+---------+------------
(0 rows)

 Even after removeing all records from clustereddjob (I have made a backup) still the same.

2017-12-08 22:09:07,926 JIRA-Bootstrap ERROR      [c.atlassian.jira.ComponentManager] Error occurred while starting component 'com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor'.
java.lang.RuntimeException: com.atlassian.scheduler.SchedulerServiceException: Unable to either create or replace clustered job: ImmutableClusteredJob[jobId=com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor,jobRunnerKey=com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor,schedule=Schedule[type=INTERVAL,intervalScheduleInfo=IntervalScheduleInfo[firstRunTime=Fri Dec 08 23:09:07 GMT 2017,intervalInMillis=3600000]],nextRunTime=Fri Dec 08 23:09:07 GMT 2017,version=1,rawParameters=(null)]
        at com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor.scheduleCleanup(DefaultTemporaryWebAttachmentsMonitor.java:75)
        at com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor.start(DefaultTemporaryWebAttachmentsMonitor.java:61)
        at com.atlassian.jira.ComponentManager.runStartable(ComponentManager.java:311)
        at com.atlassian.jira.ComponentManager.quickStart(ComponentManager.java:224)
        at com.atlassian.jira.ComponentManager.start(ComponentManager.java:164)
        at com.atlassian.jira.upgrade.PluginSystemLauncher.start(PluginSystemLauncher.java:43)
        at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:150)
        at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:298)
        at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:194)
        at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:141)
        at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:103)
        at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
        at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:101)
        at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:149)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.atlassian.scheduler.SchedulerServiceException: Unable to either create or replace clustered job: ImmutableClusteredJob[jobId=com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor,jobRunnerKey=com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor,schedule=Schedule[type=INTERVAL,intervalScheduleInfo=IntervalScheduleInfo[firstRunTime=Fri Dec 08 23:09:07 GMT 2017,intervalInMillis=3600000]],nextRunTime=Fri Dec 08 23:09:07 GMT 2017,version=1,rawParameters=(null)]
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.createOrReplaceWithRetry(CaesiumSchedulerService.java:202)
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.scheduleClusteredJob(CaesiumSchedulerService.java:189)
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.scheduleJob(CaesiumSchedulerService.java:137)
        at com.atlassian.scheduler.core.DelegatingSchedulerService.scheduleJob(DelegatingSchedulerService.java:70)
        at com.atlassian.jira.web.action.issue.DefaultTemporaryWebAttachmentsMonitor.scheduleCleanup(DefaultTemporaryWebAttachmentsMonitor.java:73)
2017-12-08 22:09:07,934 JIRA-Bootstrap ERROR      [c.a.jira.upgrade.PluginSystemLauncher] A fatal error occured during initialisation. JIRA has been locked.
Andy Heinzer
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
December 12, 2017

My apologies, while your error is similar to that bug, you seem to have two different errors that are unable to update/create new entries to that table.  But from looking at the errors, your instance looks to have two completely different keys that cannot be added.  Also it doesn't appear so far that there exist duplicate keys as the reason these can't be added. 

You also mentioned that there was a crash in one the machines.  Could you elaborate on that?  I'm interested to learn more about the event that first caused this problem.

So I think it would be best to jump back and make sure that the SQL user account Jira is using to connect to this database still has the needed permissions to be able to create/update the database as needed.   I would recommend tracing back through the steps in Connecting JIRA applications to PostgreSQL

It might be helpful here to try to create a new empty database, and a new sql user.   You could then shutdown Jira, edit the dbconfig.xml file to tell Jira to use this new empty database, and from there you could then start up Jira.   When Jira starts up connected to an empty database, it launches the setup wizard.  In this setup wizard there is an option to import from a backup.  By default Jira is still creating these xml backup zip files in the $JIRAHOME/export/ directory.   Provided you have a backup zip file here, you could then just copy this zip file to your $JIRAHOME/import/ directory and then in the setup wizard you can import this backup as a means to copy your Jira data into this new empty database.   If the old database or user has in some way become corrupted, these steps should allow you to still maintain your data and get around that problem.

Lukasz Zalewski December 12, 2017

Thanks Andrew for your comments.

I have made reindaxation of database on vocuum on clusteredjob table.

jiradb=# reindex database "jiradb";
jiradb=# vacuum analyze public.clusteredjob

After restarting, JIRA works.

Of course additional consistancy check and fix were dane by me but fortunately using JIRA GUI.

Thx

Andy Heinzer
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
December 12, 2017

Hi Lukasz,

Thanks for letting us know what the solution here was.  I would not have expected that specific procedure to fix this problem, but you can't argue with results.

Cheers,

Andy

0 votes
Manuel Bähnisch September 25, 2020

Hi Andy,

I also came across this problem while upgrading my Jira 8.5.x to 8.12.x in order to switch from MySQL 5.7 to 8.0. I still need to verify this, but am fairly sure this is due to MySQL8 being case-sensitive regarding tablenames. During the data migration process by hand, tables were created with lowercase table names where Jira expects uppercase.

 

How did I find out.

Step 1: When realizing that the clusteredJob entry could not be created in the database I reassured my database user actually had right to do that - which he had.

Step 2: I turned on Debug logging for the following classes in the <jira-installation>/WEB-INF/classes/log4j.properties - this created about 40 Megs of logs during startup:

  • log4j.logger.com.atlassian
  • log4j.logger.com.atlassian.jira

Step 3: I had a closer look in the logs

Already in the first 100+ lines of the logs I found that during startup jira tries to fix its database schema to the new layout used in 8.12:

2020-09-25 08:37:19,613+0200 JIRA-Bootstrap INFO [o.o.c.entity.jdbc.DatabaseUtil] Database Driver Version is mysql-connector-java-8.0.21 (Revision: 33f65445a1bcc544eb0120491926484da168f199)
2020-09-25 08:37:20,035+0200 JIRA-Bootstrap ERROR [o.o.c.entity.jdbc.DatabaseUtil] Could not widen column "OWNER" in table "OS_CURRENTSTEP" to size: VARCHAR(255).
2020-09-25 08:37:20,035+0200 JIRA-Bootstrap ERROR [o.o.c.entity.jdbc.DatabaseUtil] SQL Exception while executing the following:
ALTER TABLE OS_CURRENTSTEP MODIFY OWNER VARCHAR(255)
Error was: java.sql.SQLSyntaxErrorException: Table 'jira_dev_db.OS_CURRENTSTEP' doesn't exist

Since I knew the database user had permissions on the database I set of the sql command manually in a dbtool. There I got the same exception regarding the non-existant table. Looking at all tables in a tree view, I found that the table did exist - in lowercase though. Changing the sql command to a lowercase tablename then worked.

I requested our db admins to configure the jira-db to be case-insensitive now which helped in migrating other applications. Not sure though, if Postgres has similar options but it makes sense that problems disappear when recreating the database and then migrating data to the freshly created one.

Maybe a case-sensitivity health check when starting up jira could be implemented. Drilling down the problem took half a day even though it may be fairly simple.

Hope that helps.
Manuel

Manuel Bähnisch September 29, 2020

As I expected: MySQL8 is per default case-sensitive regarding table names. Some sql statements set of by JIRA use table names with uppercase, some lowercase. Thus MySQL8 must be set to be case-insensitive. Then JIRA will start properly.

Manuel

0 votes
JakubWoo October 3, 2019

We experienced the same issue - generally speaking it was clustered jobs start failure causing Jira start failure and nothing from above suggestions did work.

We managed to solve the problem by changing database type declared in dbconfig.xml - using postgres72 instead of postgres. The root cause was invalid  and not supported byte-array field type. We had to increase log verbosity level to DEBUG for com.atlassian.jira log4j appender to see root error in logs.


Caused by: org.ofbiz.core.entity.GenericModelException: GenericDAO.getValue: definition fieldType byte-array not found, cannot setValue for field ClusteredJob.parameters.
at org.ofbiz.core.entity.jdbc.SqlJdbcUtil.setValue(SqlJdbcUtil.java:761)
at org.ofbiz.core.entity.jdbc.SqlJdbcUtil.setValue(SqlJdbcUtil.java:753)
at org.ofbiz.core.entity.jdbc.SqlJdbcUtil.setValues(SqlJdbcUtil.java:549)
at org.ofbiz.core.entity.GenericDAO.singleInsert(GenericDAO.java:197)
... 66 more

Cheers! 

0 votes
Henry Auffahrt July 23, 2018

We have exactly the same issue as you have. But the solutions you present here are not working for us.

We want to update from Jira 7.1.4 to 7.2.15. After running the installer and startup the instance the instance failed with a very similar error message in the logs.

 

2018-07-23 12:11:46,571 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger] 

***********************************************************************************
JIRA 7.2.15 build: 72018 started. You can now access JIRA through your web browser.
***********************************************************************************

2018-07-23 12:11:46,588 JIRA-Bootstrap INFO [c.a.plugin.manager.DefaultPluginManager] Plugin system lateStartup begun
2018-07-23 12:11:46,591 JIRA-Bootstrap INFO [c.a.plugin.manager.DefaultPluginManager] Plugin system lateStartup ended
2018-07-23 12:11:46,594 JIRA-Bootstrap DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.plugins.atlassian-whitelist-api-plugin]
2018-07-23 12:11:46,625 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]

___ Plugin System Started _________________

2018-07-23 12:11:46,625 JIRA-Bootstrap ERROR [c.a.jira.upgrade.UpgradeLauncher] Skipping, JIRA is locked.
2018-07-23 12:11:46,626 JIRA-Bootstrap INFO [c.a.jira.scheduler.JiraSchedulerLauncher] JIRA Scheduler not started: JIRA startup checklist failed.
2018-07-23 12:11:46,678 JIRA-Bootstrap ERROR [c.a.jira.startup.AnalyticsLauncher] Start analytics not scheduled
com.atlassian.scheduler.SchedulerServiceException: Unable to either create or replace clustered job: ImmutableClusteredJob[jobId=03def26b-b3ec-4a6d-afc7-a7b0e102d241,jobRunnerKey=com.atlassian.jira.startup.AnalyticsLauncher.Start,schedule=Schedule[type=INTERVAL,intervalScheduleInfo=IntervalScheduleInfo[firstRunTime=Mon Jul 23 12:14:46 CEST 2018,intervalInMillis=0]],nextRunTime=Mon Jul 23 12:14:46 CEST 2018,version=1,rawParameters=(null)]
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.createOrReplaceWithRetry(CaesiumSchedulerService.java:202)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.scheduleClusteredJob(CaesiumSchedulerService.java:189)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.scheduleJob(CaesiumSchedulerService.java:137)
at com.atlassian.scheduler.core.AbstractSchedulerService.scheduleJobWithGeneratedId(AbstractSchedulerService.java:114)
at com.atlassian.scheduler.core.DelegatingSchedulerService.scheduleJobWithGeneratedId(DelegatingSchedulerService.java:77)
at com.atlassian.jira.startup.AnalyticsLauncher.runStartAnalyticsJob(AnalyticsLauncher.java:72)
at com.atlassian.jira.startup.AnalyticsLauncher.start(AnalyticsLauncher.java:40)
at com.atlassian.jira.startup.ActiveServicesLauncher.start(ActiveServicesLauncher.java:56)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postTenantArrived$4(DefaultJiraLauncher.java:178)
at com.atlassian.jira.startup.DefaultInstantUpgradeManager$StartupTask.run(DefaultInstantUpgradeManager.java:139)
at com.atlassian.jira.startup.DefaultInstantUpgradeManager.runTask(DefaultInstantUpgradeManager.java:55)
at com.atlassian.jira.startup.DefaultInstantUpgradeManager.doNowOrWhenInstanceBecomesActive(DefaultInstantUpgradeManager.java:42)
at com.atlassian.jira.startup.DefaultJiraLauncher.postTenantArrived(DefaultJiraLauncher.java:171)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDBActivated$3(DefaultJiraLauncher.java:159)
at com.atlassian.jira.tenancy.DefaultTenantManager.doNowOrWhenTenantArrives(DefaultTenantManager.java:55)
at com.atlassian.jira.startup.DefaultJiraLauncher.postDBActivated(DefaultJiraLauncher.java:156)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:145)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:298)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:194)
at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:135)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:101)
at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:99)
at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:149)
at java.lang.Thread.run(Thread.java:745)
2018-07-23 12:11:46,745 JIRA-Bootstrap ERROR [c.a.jira.startup.IndexRecoveryLauncher] Index recovery not scheduled
com.atlassian.scheduler.SchedulerServiceException: Unable to either create or replace clustered job: ImmutableClusteredJob[jobId=260b755d-0820-42db-9726-3e1d91c91d50,jobRunnerKey=com.atlassian.jira.DefaultJiraLauncher.IndexRecovery,schedule=Schedule[type=INTERVAL,intervalScheduleInfo=IntervalScheduleInfo[firstRunTime=Mon Jul 23 12:11:46 CEST 2018,intervalInMillis=0]],nextRunTime=Mon Jul 23 12:11:46 CEST 2018,version=1,rawParameters=(null)]
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.createOrReplaceWithRetry(CaesiumSchedulerService.java:202)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.scheduleClusteredJob(CaesiumSchedulerService.java:189)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.scheduleJob(CaesiumSchedulerService.java:137)
at com.atlassian.scheduler.core.AbstractSchedulerService.scheduleJobWithGeneratedId(AbstractSchedulerService.java:114)
at com.atlassian.scheduler.core.DelegatingSchedulerService.scheduleJobWithGeneratedId(DelegatingSchedulerService.java:77)
at com.atlassian.jira.startup.IndexRecoveryLauncher.runIndexRecoveryJob(IndexRecoveryLauncher.java:102)
at com.atlassian.jira.startup.IndexRecoveryLauncher.start(IndexRecoveryLauncher.java:49)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postTenantArrived$4(DefaultJiraLauncher.java:181)
at com.atlassian.jira.startup.DefaultInstantUpgradeManager$StartupTask.run(DefaultInstantUpgradeManager.java:139)
at com.atlassian.jira.startup.DefaultInstantUpgradeManager.runTask(DefaultInstantUpgradeManager.java:55)
at com.atlassian.jira.startup.DefaultInstantUpgradeManager.doNowOrWhenInstanceBecomesActive(DefaultInstantUpgradeManager.java:42)
at com.atlassian.jira.startup.DefaultJiraLauncher.postTenantArrived(DefaultJiraLauncher.java:171)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDBActivated$3(DefaultJiraLauncher.java:159)
at com.atlassian.jira.tenancy.DefaultTenantManager.doNowOrWhenTenantArrives(DefaultTenantManager.java:55)
at com.atlassian.jira.startup.DefaultJiraLauncher.postDBActivated(DefaultJiraLauncher.java:156)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:145)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:298)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:194)
at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:135)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:101)
at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:99)
at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:149)
at java.lang.Thread.run(Thread.java:745)
2018-07-23 12:11:46,755 JIRA-Bootstrap INFO [c.a.jira.startup.DefaultInstantUpgradeManager] Late startup launchers took 0s
2018-07-23 12:11:46,760 JIRA-Bootstrap ERROR [c.a.jira.startup.LauncherContextListener] Unable to start JIRA.
java.lang.IllegalStateException: Abnormal system startup detected
at com.atlassian.jira.startup.LauncherContextListener.initDone(LauncherContextListener.java:180)
at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:162)
at java.lang.Thread.run(Thread.java:745)
2018-07-23 12:11:46,775 JIRA-Bootstrap INFO [c.a.jira.startup.LauncherContextListener] Memory Usage:
---------------------------------------------------------------------------------
Heap memory : Used: 418 MiB. Committed: 1602 MiB. Max: 1946 MiB
Non-heap memory : Used: 224 MiB. Committed: 244 MiB. Max: 1264 MiB
---------------------------------------------------------------------------------
TOTAL : Used: 642 MiB. Committed: 1846 MiB. Max: 3210 MiB
---------------------------------------------------------------------------------
2018-07-23 12:11:48,826 Modification Check:thread-1 INFO [c.a.jira.startup.JiraStartupLogger]

___ Modifications ___________________________

Modified Files : jira-application.properties
Removed Files : None

I run these to commands on our database:

jiradb=# reindex database "jiradb";
jiradb=# vacuum analyze public.clusteredjob

This did not work, then I delete the "clustereddjob" table complete, also this didn't work. (There were no duplicate rows in this table)

I tried also the following things:

Henry Auffahrt July 29, 2018

I solved it by creating a new database and then import the backup to the new database like Andrew Heinzer mentioned it above.

The crucial point was:

It might be helpful here to try to create a new empty database, and a new sql user.   You could then shutdown Jira, edit the dbconfig.xml file to tell Jira to use this new empty database, and from there you could then start up Jira.   When Jira starts up connected to an empty database, it launches the setup wizard.  In this setup wizard there is an option to import from a backup.  By default Jira is still creating these xml backup zip files in the $JIRAHOME/export/ directory.   Provided you have a backup zip file here, you could then just copy this zip file to your $JIRAHOME/import/ directory and then in the setup wizard you can import this backup as a means to copy your Jira data into this new empty database.   If the old database or user has in some way become corrupted, these steps should allow you to still maintain your data and get around that problem.

0 votes
Andy Heinzer
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
December 11, 2017

Hi Lukasz,

Sorry to hear that you are having this problem.  However I believe this is a known bug in Jira we are tracking in https://jira.atlassian.com/browse/JRASERVER-64325

As such I would recommend following the KB JIRA Fails to Start due to Too many rows found for query on ClusteredJob.  You will need to run this SQL query to find the duplicate rows in that table first:

SELECT * FROM clusteredjob WHERE job_id in (SELECT job_id FROM clusteredjob GROUP BY job_id HAVING COUNT(*)>1)

And from there you would have to manually remove one of these duplicates.  In the case of the KB the duplicate had an id of 11106.

Resolution

  1. Shut down JIRA and back up the database properly

  2. Delete either of the entries:

    delete from clusteredjob where id = 11106;
  3. For Oracle DB, ensure run commit; after the above.
  4. Restart JIRA

Also you will need to re-enable that plugin in the pluginstate table.  I would recommend doing both SQL steps at the same time while Jira is shutdown.  Then you should just be able to start Jira up again in order to resolve this problem.

Regards,
Andy

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events