JIRA offline after changing custom field key VERY URGENT!

[Adriana Stancu]

Hello,

I needed to change a custom field type and updated the customfield table with a new type:

UPDATE customfield SET CUSTOMFIELDTYPEKEY = 'com.atlassian.jira.plugin.sysme', CUSTOMFIELDSEARCHERKEY = 'com.atlassian.jira.plugin.system.customfieldtypes:multiselectsearcher' WHERE cfname = '%SOMENAME%'

Now after I log in into Jira I get 500 Invalid module key specified.

It is very urgent as this is a production Jira and I don t have any database backups.

[Adriana Stancu]

If someone has the same problem, here is what I did to restore Jira:

Check if JIRA has exported automated backups in the $JIRA_HOME/export location. Then I have followed the procedure below:

1. Stop JIRA;
2. Place the exported XML Backup file into $JIRA_HOME/import directory;
3. Create an empty database by strictly following Connecting JIRA to a Databasedocumentation.
4. Edit the $JIRA_HOME/dbconfig.xml file to point JIRA to use the empty database;
5. Rename the caches folder in $JIRA_HOME and create an empty folder named indexes inside.
6. Start JIRA - Since JIRA will detect an empty database it will bring the setup wizard dialog, select to import the data on the first configuration screen and type the filename placed on $JIRA_HOME/import directory.

[Nic Brough -Adaptavist-]

So, going back to the backup you said you didn't have is the solution.

[Adriana Stancu]

I didn't have a backup of my database, but I found out that Jira was saving backups in JIRA_HOME/export folder.

[Adriana Stancu]

.

[Nic Brough -Adaptavist-]

That's because you set it to that.  It is wrong. 

Even if it weren't misspelt, it's being truncated down to a module that definitely does not exist.

You need to set back ALL of the custom fields customfieldtypekey to what they were before.  The searchers are less important because they can be fixed in the UI, and wouldn't cause massive crashing like this, just piles of wrong-type warnings.

[Adriana Stancu]

Thanks a lot for that hint, but the update query only modified one row, the others have the same key. What is the default key value for Jira 3.15.3 ?

[Adriana Stancu]

It was a drop down list, I think the right type would be com.atlassian.jira.plugin.system.customfieldtypes:select right?

[Adriana Stancu]

Tried with that and the same:
 com.atlassian.cache.CacheException: java.lang.IllegalArgumentException: Invalid module key specified:
    at com.atlassian.cache.memory.DelegatingCachedReference.get(DelegatingCachedReference.java:91)
    at com.atlassian.jira.issue.search.managers.DefaultSearchHandlerManager.getHelper(DefaultSearchHandlerManager.java:254)
    at com.atlassian.jira.issue.search.managers.DefaultSearchHandlerManager.getAllSearchers(DefaultSearchHandlerManager.java:234)
    at com.atlassian.jira.issue.search.managers.DefaultIssueSearcherManager.getAllSearchers(DefaultIssueSearcherManager.java:26)
    at com.atlassian.jira.issue.index.managers.FieldIndexerManagerImpl.loadIndexers(FieldIndexerManagerImpl.java:83)
    at com.atlassian.cache.memory.MemoryCacheManager$1$1.load(MemoryCacheManager.java:101)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
    at com.atlassian.cache.memory.DelegatingCachedReference.getUnderLock(DelegatingCachedReference.java:101)
    at com.atlassian.cache.memory.DelegatingCachedReference.get(DelegatingCachedReference.java:86)

[Nic Brough -Adaptavist-]

There is no default key - it is different for each custom field.

You can take a guess by looking at a custom field value record for that custom field though - if the data says number or date, you know it's one of those and might be able to work out what a matching key is by looking at another field of a similar type.

(and even if I had a Jira database to hand, mine is not a 3.x one and things have changed.  Also, there is no Jira 3.15, the last 3.x version was 3.13)

[Adriana Stancu]

I understanda that I have to restore the values from the column CUSTOMFILEDTYPEKEY to the right ones, but how can I find the correct values?

[Adriana Stancu]

They were all systme type I just copied the values when I updated my field. I there a documentation where I can find the corresponding types?

[Adriana Stancu]

I found this, https://docs.atlassian.com/software/jira/docs/api/3.13.4/com/atlassian/jira/issue/customfields/CustomFieldType.html can I find the corresponding field type from this?

[Nic Brough -Adaptavist-]

No, they were not, it was one of your SQL updates that trashed them.

There is no way to know what they might have been - you are going to have to guess based on the field name and the contents.  You've said you were on Jira 3.15, which doesn't exist, so I can't begin to tell you what the customfieldtypekey might have to be restored to.

If you were on Jira 5 or above, I could start to tell you things like:

For "Satisfaction Date", check that the values for that field are all in the "datevalue" column in the customfieldvalue table - if they are, then you can set the type key to m.atlassian.jira.plugin.system.customfieldtypes:datetimefield and that will at least make it readable again.

[Nic Brough -Adaptavist-]

Our messages crossed.  Yes, that doc should give you the keys like the "com.atlassian.jira.plugin.system.customfieldtypes:datetimefield" I used in my example - many of them might be the same as the Jira 7 I'm looking at.  I just know that some of the fields definitely changed in version 4, and that might have affected the storage and the key

[Adriana Stancu]

Regarding the version I just pasted the name of the installer and the folder created which is atlassian-servicedesk-3.15.3, it was bought in october I think is the last version. I found that you helped someone with a similar problem here https://community.atlassian.com/t5/Answers-Developer-Questions/Need-a-list-of-custom-field-types/qaq-p/467438  I used the mappings to update some of hte fileds but I have to get them all to the correct values in order to work? We don t use much of them.

[Adriana Stancu]

The searcher types are
? com.atlassian.servicedesk:sd-request-participants-searcher
? com.atlassian.servicedesk:vp-origin-searcher
? com.atlassian.servicedesk:sd-customer-organizations-searcher
? com.atlassian.servicedesk:sd-request-feedback-searcher
? com.atlassian.servicedesk:sd-request-feedback-date-searcher
? com.atlassian.servicedesk.approvals-plugin:sd-approvals-searcher
multiselect - com.atlassian.jira.plugin.system.customfieldtypes:multiselectsearcher
? NULL  Approvers
cascadingselect - com.atlassian.jira.plugin.system.customfieldtypes:cascadingselectsearcher
textfield - com.atlassian.jira.plugin.system.customfieldtypes:textsearcher
userpicker - com.atlassian.jira.plugin.system.customfieldtypes:userpickergroupsearcher
sd-sla-field - com.atlassian.servicedesk:sd-sla-field-searcher
float - com.atlassian.jira.plugin.system.customfieldtypes:exactnumber

Can you help me with the corresponding values for the ones with question mark? Also take a look for the others if there may be somthing wrong.

[Adriana Stancu]

It is important to make it work again, I can delete some of the fields from Jira if I cannot find the correct type for them. I think the system field cannot be deleted. Do I have to correct them or it will work without these fields?

[Nic Brough -Adaptavist-]

Right.  "Jira Service Desk" is not Jira, and it only works on Jira 6 and 7, so you can at least rely on the recent docs for the types.

However, it's quite likely that a load of your fields are Service Desk fields which have their own keys again.

The thing here is that you don't need to get them totally right, just right enough to make them work again.  I gave the example of a date/time field, but that type of field would also rescue plain date fields.   You'd see the fields working, but with all the times showing as 00:00:00. 

Another one would be user fields - participants is a very specific user type of field, but if you just set it to "multi user picker", you will get that working again, as it has the same sort of data in it.

[Adriana Stancu]

I updated the table as follows:

[Adriana Stancu]

But the same error:
Caused by: java.lang.IllegalArgumentException: Invalid module key specified:
    at com.atlassian.plugin.ModuleCompleteKey.<init>(ModuleCompleteKey.java:49)
    at com.atlassian.plugin.ModuleCompleteKey.<init>(ModuleCompleteKey.java:29)
    at com.atlassian.plugin.manager.ProductPluginAccessorBase.getEnabledPluginModule(ProductPluginAccessorBase.java:180)
    at com.atlassian.plugin.manager.ForwardingPluginAccessor.getEnabledPluginModule(ForwardingPluginAccessor.java:54)
    at com.atlassian.plugin.manager.EnabledModuleCachingPluginAccessor.getEnabledPluginModule(EnabledModuleCachingPluginAccessor.java:39)
    at com.atlassian.plugin.manager.ProductPluginAccessor.getEnabledPluginModule(ProductPluginAccessor.java:19)
    at com.atlassian.extension.plugins.ExtensionAccessingPluginAccessor.getEnabledPluginModule(ExtensionAccessingPluginAccessor.java:169)
    at com.atlassian.jira.plugin.customfield.CustomFieldTypeModuleDescriptorsImpl.getCustomFieldType(CustomFieldTypeModuleDescriptorsImpl.java:51)
    at com.atlassian.jira.issue.fields.ImmutableCustomField.loadCustomFieldType(ImmutableCustomField.java:1794)
    at io.atlassian.fugue.Suppliers$MemoizingSupplier.get(Suppliers.java:175)
    at com.atlassian.jira.issue.fields.ImmutableCustomField.getCustomFieldType(ImmutableCustomField.java:1542)
    at com.atlassian.jira.issue.managers.CachingCustomFieldManager$CustomFieldInMemoryStore.lambda$getAllSorted$2(CachingCustomFieldManager.java:559)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
    at java.util.Iterator.forEachRemaining(Iterator.java:116)
    at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
    at com.atlassian.jira.issue.managers.CachingCustomFieldManager$CustomFieldInMemoryStore.getAllSorted(CachingCustomFieldManager.java:560)
    at com.atlassian.jira.issue.managers.CachingCustomFieldManager$CustomFieldInMemoryStore.access$300(CachingCustomFieldManager.java:519)
    at com.atlassian.jira.issue.managers.CachingCustomFieldManager.getCustomFieldObjects(CachingCustomFieldManager.java:341)
    at com.atlassian.jira.issue.search.managers.DefaultSearchHandlerManager.createHelper(DefaultSearchHandlerManager.java:276)
    at com.atlassian.jira.issue.search.managers.DefaultSearchHandlerManager.access$000(DefaultSearchHandlerManager.java:77)
    at com.atlassian.jira.issue.search.managers.DefaultSearchHandlerManager$1.get(DefaultSearchHandlerManager.java:102)
    at com.atlassian.jira.issue.search.managers.DefaultSearchHandlerManager$1.get(DefaultSearchHandlerManager.java:99)
    at com.atlassian.cache.memory.MemoryCacheManager$1$1.load(MemoryCacheManager.java:125)
    at com.atlassian.cache.memory.MemoryCacheManager$1$1.load(MemoryCacheManager.java:101)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
    at com.atlassian.cache.memory.DelegatingCachedReference.getUnderLock(DelegatingCachedReference.java:101)
    at com.atlassian.cache.memory.DelegatingCachedReference.get(DelegatingCachedReference.java:86)
    ... 32 more
2018-11-29 19:29:41,555 JiraTaskExectionThread-1 ERROR admin 1169x139x1 ylmqcl 0:0:0:0:0:0:0:1 /secure/admin/IndexReIndex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex All FAILED.  Indexer: DefaultIndexManager: paths: [D:\Application Data\caches\indexes\comments, D:\Application Data\caches\indexes\issues, D:\Application Data\caches\indexes\changes, D:\Application Data\caches\indexes\worklogs, D:\Application Data\caches\indexes\plugins\servicedeskcannedresponses]
com.atlassian.jira.index.IndexingFailureException: Indexing completed with 24 errors
    at com.atlassian.jira.index.AccumulatingResultBuilder$CompositeResult.await(AccumulatingResultBuilder.java:150)
    at com.atlassian.jira.issue.index.DefaultIndexManager.doIndexIssuesInBatchMode(DefaultIndexManager.java:1051)
    at com.atlassian.jira.issue.index.DefaultIndexManager.doStopTheWorldReindex(DefaultIndexManager.java:1026)
    at com.atlassian.jira.issue.index.DefaultIndexManager.lambda$reIndexAll$0(DefaultIndexManager.java:328)
    at com.atlassian.jira.issue.index.DefaultIndexManager.withReindexLock(DefaultIndexManager.java:377)
    at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:328)
    at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:261)
    ... 3 filtered
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:38)
    at com.sun.proxy.$Proxy12.reIndexAll(Unknown Source)
    at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:66)
    at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:49)
    at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.doReindex(ReIndexAsyncIndexerCommand.java:25)
    at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:63)
    at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.call(ReIndexAsyncIndexerCommand.java:16)
    at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:26)
    at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:458)
    at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:426)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at com.atlassian.jira.task.ForkedThreadExecutor$ForkedRunnableDecorator.run(ForkedThreadExecutor.java:216)
    at java.lang.Thread.run(Thread.java:748)
2018-11-29 19:29:41,570 JiraTaskExectionThread-1 INFO admin 1169x139x1 ylmqcl 0:0:0:0:0:0:0:1 /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 0% complete. Current index: SearchRequest
2018-11-29 19:29:41,580 JiraTaskExectionThread-1 INFO admin 1169x139x1 ylmqcl 0:0:0:0:0:0:0:1 /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 2% complete. Current index: SearchRequest
2018-11-29 19:29:41,586 JiraTaskExectionThread-1 INFO admin 1169x139x1 ylmqcl 0:0:0:0:0:0:0:1 /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 5% complete. Current index: SearchRequest
2018-11-29 19:29:41,625 JiraTaskExectionThread-1 INFO admin 1169x139x1 ylmqcl 0:0:0:0:0:0:0:1 /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 8% complete. Current index: SearchRequest
2018-11-29 19:29:41,632 JiraTaskExectionThread-1 INFO admin 1169x139x1 ylmqcl 0:0:0:0:0:0:0:1 /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 11% complete. Current index: SearchRequest
2018-11-29 19:29:41,633 JiraTaskExectionThread-1 INFO admin 1169x139x1 ylmqcl 0:0:0:0:0:0:0:1 /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 14% complete. Current index: SearchRequest
2018-11-29 19:29:41,633 JiraTaskExectionThread-1 INFO admin 1169x139x1 ylmqcl 0:0:0:0:0:0:0:1 /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 17% complete. Current index: SearchRequest
2018-11-29 19:29:41,634 JiraTaskExectionThread-1 INFO admin 1169x139x1 ylmqcl 0:0:0:0:0:0:0:1 /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 20% complete. Current index: SearchRequest
2018-11-29 19:29:41,733 JiraTaskExectionThread-1 INFO admin 1169x139x1 ylmqcl 0:0:0:0:0:0:0:1 /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 20% complete. Current index: PortalPage
2018-11-29 19:29:41,736 JiraTaskExectionThread-1 INFO admin 1169x139x1 ylmqcl 0:0:0:0:0:0:0:1 /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 23% complete. Current index: PortalPage
2018-11-29 19:29:41,744 JiraTaskExectionThread-1 INFO admin 1169x139x1 ylmqcl 0:0:0:0:0:0:0:1 /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 26% complete. Current index: PortalPage
2018-11-29 19:29:41,746 JiraTaskExectionThread-1 INFO admin 1169x139x1 ylmqcl 0:0:0:0:0:0:0:1 /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 29% complete. Current index: PortalPage
2018-11-29 19:29:41,751 JiraTaskExectionThread-1 INFO admin 1169x139x1 ylmqcl 0:0:0:0:0:0:0:1 /secure/admin/IndexReIndex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex took: 171ms. Indexer: SharedEntityIndexManager: paths: []
2018-11-29 19:29:41,751 JiraTaskExectionThread-1 INFO admin 1169x139x1 ylmqcl 0:0:0:0:0:0:0:1 /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing finished
2018-11-29 19:29:41,752 JiraTaskExectionThread-1 ERROR admin 1169x139x1 ylmqcl 0:0:0:0:0:0:0:1 /secure/admin/IndexReIndex.jspa [c.a.jira.task.TaskManagerImpl] Task 'Jira Indexing' failed.
com.atlassian.jira.index.IndexingFailureException: Indexing completed with 1 errors
    at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:83)
    at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:49)
    at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.doReindex(ReIndexAsyncIndexerCommand.java:25)
    at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:63)
    at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.call(ReIndexAsyncIndexerCommand.java:16)
    at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:26)
    at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:458)
    at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:426)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at com.atlassian.jira.task.ForkedThreadExecutor$ForkedRunnableDecorator.run(ForkedThreadExecutor.java:216)
    at java.lang.Thread.run(Thread.java:748)

[Adriana Stancu]

Sorry for the long post but I thought that there may be some hint that I missed:
stopped Jira, update the tables, start Jira and perform reindex. Maybe some cache files or something else?

[Nic Brough -Adaptavist-]

I'd guess that at least one of the fields is still wrong, but it is encouraging that the indexing is completing without huge numbers of errors.  That suggests it might be just a searcher.

Your stop and start of Jira should be cleaning out the caches, so I would not worry about those.

I would try a simple JQL search on each of the custom fields to see if that errors - if they work and pull back good looking data, the indexing is ok.  If they error, focus on that custom field!

[Adriana Stancu]

Thank you for your help, I have contacted Atlassian support and recovered the database. I will post the resolution in case that someone does the same.

[Nic Brough -Adaptavist-]

Stop Jira, change the keys back to what they were before, restart it and run a full locking re-index.

This SQL only works when the field types hold data that are 100% compatible, and I suspect

• They are not
• sysme is not a valid custom field type

[Adriana Stancu]

I tried but I got this

[Adriana Stancu]

Where can I find the logs file that reports this error?

[Nic Brough -Adaptavist-]

Start with the logs under <jira home>/log - look for "atlassian-jira.log".

[Adriana Stancu]

com.atlassian.jira.index.IndexingFailureException: Indexing completed with 1 errors
    at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:83)
    at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:49)
    at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.doReindex(ReIndexAsyncIndexerCommand.java:25)
    at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:63)
    at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.call(ReIndexAsyncIndexerCommand.java:16)
    at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:26)
    at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:458)
    at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:426)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at com.atlassian.jira.task.ForkedThreadExecutor$ForkedRunnableDecorator.run(ForkedThreadExecutor.java:216)
    at java.lang.Thread.run(Thread.java:748)

[Adriana Stancu]

Thanks for getting back so fast. When I try to access issue related content I get the following error. Maybe this is more relevant:
018-11-29 17:08:19,229 http-nio-8080-exec-12 ERROR horia.panait 1028x398x1 pt61n0 10.17.2.52 /rest/gadget/1.0/issueTable/jql [c.a.p.r.c.error.jersey.ThrowableExceptionMapper] Uncaught exception thrown by REST service: java.lang.IllegalArgumentException: Invalid module key specified:
com.atlassian.cache.CacheException: java.lang.IllegalArgumentException: Invalid module key specified:
    at com.atlassian.cache.memory.DelegatingCache.get(DelegatingCache.java:211)
    at com.atlassian.cache.memory.DelegatingCache.get(DelegatingCache.java:162)
    at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.getRelevantFieldLayout(AbstractFieldLayoutManager.java:264)
    at com.atlassian.jira.issue.fields.layout.field.DefaultFieldLayoutManager.getFieldLayout(DefaultFieldLayoutManager.java:702)

[Adriana Stancu]

I performed a integrity check and there is another error. If any of them seems to mean something please let me know. We need to get Jira back and running fast.
018-11-29 17:43:55,691 http-nio-8080-exec-14 ERROR      [c.a.j.web.servlet.InternalServerErrorServlet] {errorId=7bc960a4-914f-47cf-9729-8e1441dcd58e, interpretedMsg=, cause=com.atlassian.jira.appconsistency.integrity.exception.IntegrityException: Error occurred while performing check., stacktrace=com.atlassian.jira.appconsistency.integrity.exception.IntegrityException: Error occurred while performing check.
        at com.atlassian.jira.appconsistency.integrity.check.FieldLayoutCheck.doCheck(FieldLayoutCheck.java:82) [classes/:?]
        at com.atlassian.jira.appconsistency.integrity.check.FieldLayoutCheck.preview(FieldLayoutCheck.java:31) [classes/:?]
        at com.atlassian.jira.appconsistency.integrity.IntegrityChecker.preview(IntegrityChecker.java:52) [classes/:?]
        at com.atlassian.jira.appconsistency.integrity.IntegrityChecker.previewWithIds(IntegrityChecker.java:29) [classes/:?]
        at com.atlassian.jira.web.action.admin.IntegrityChecker.doExecute(IntegrityChecker.java:56) [classes/:?]
        at webwork.action.ActionSupport.execute(ActionSupport.java:165) [webwork-1.4-atlassian-30.jar:?]
        at com.atlassian.jira.action.JiraActionSupport.execute(JiraActionSupport.java:63) [jira-api-7.12.3.jar:?]
        at webwork.interceptor.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:39) [webwork-1.4-atlassian-30.jar:?]
        at webwork.interceptor.NestedInterceptorChain.proceed(NestedInterceptorChain.java:31) [webwork-1.4-atlassian-30.jar:?]
        at webwork.interceptor.ChainedInterceptor.intercept(ChainedInterceptor.java:16) [webwork-1.4-atlassian-30.jar:?]
        at webwork.interceptor.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:35) [webwork-1.4-atlassian-30.jar:?]
        at webwork.dispatcher.GenericDispatcher.executeAction(GenericDispatcher.java:225) [webwork-1.4-atlassian-30.jar:?]

[Adriana Stancu]

This is the error that  generate all the fuss. Please point me some workarounds:
Caused by: java.lang.IllegalArgumentException: Invalid module key specified:
    at com.atlassian.plugin.ModuleCompleteKey.<init>(ModuleCompleteKey.java:49)
    at com.atlassian.plugin.ModuleCompleteKey.<init>(ModuleCompleteKey.java:29)
    at com.atlassian.plugin.manager.ProductPluginAccessorBase.getEnabledPluginModule(ProductPluginAccessorBase.java:180)
    at com.atlassian.plugin.manager.ForwardingPluginAccessor.getEnabledPluginModule(ForwardingPluginAccessor.java:54)
    at com.atlassian.plugin.manager.EnabledModuleCachingPluginAccessor.getEnabledPluginModule(EnabledModuleCachingPluginAccessor.java:39)
    at com.atlassian.plugin.manager.ProductPluginAccessor.getEnabledPluginModule(ProductPluginAccessor.java:19)
    at com.atlassian.extension.plugins.ExtensionAccessingPluginAccessor.getEnabledPluginModule(ExtensionAccessingPluginAccessor.java:169)
    at com.atlassian.jira.plugin.customfield.CustomFieldTypeModuleDescriptorsImpl.getCustomFieldType(CustomFieldTypeModuleDescriptorsImpl.java:51)
    at com.atlassian.jira.issue.fields.ImmutableCustomField.loadCustomFieldType(ImmutableCustomField.java:1794)
    at io.atlassian.fugue.Suppliers$MemoizingSupplier.get(Suppliers.java:175)
    at com.atlassian.jira.issue.fields.ImmutableCustomField.getCustomFieldType(ImmutableCustomField.java:1542)
    at com.atlassian.jira.issue.managers.CachingCustomFieldManager$CustomFieldInMemoryStore.lambda$get$0(CachingCustomFieldManager.java:542)
    at io.atlassian.fugue.Option.filter(Option.java:288)
    at com.atlassian.jira.issue.managers.CachingCustomFieldManager$CustomFieldInMemoryStore.get(CachingCustomFieldManager.java:542)
    at com.atlassian.jira.issue.managers.CachingCustomFieldManager$CustomFieldInMemoryStore.access$100(CachingCustomFieldManager.java:519)
    at com.atlassian.jira.issue.managers.CachingCustomFieldManager.getCustomFieldObject(CachingCustomFieldManager.java:282)
    at java.util.Optional.map(Optional.java:215)
    at com.atlassian.jira.issue.managers.CachingCustomFieldManager.getCustomFieldObject(CachingCustomFieldManager.java:293)
    at com.atlassian.jira.issue.managers.CachingCustomFieldManager.exists(CachingCustomFieldManager.java:303)
    at com.atlassian.jira.issue.fields.DefaultFieldManager.isExistingCustomField(DefaultFieldManager.java:518)
    at com.atlassian.jira.issue.fields.DefaultFieldManager.isCustomField(DefaultFieldManager.java:508)
    at com.atlassian.jira.issue.fields.DefaultFieldManager.isOrderableField(DefaultFieldManager.java:557)
    at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.loadFieldLayoutItems(AbstractFieldLayoutManager.java:305)
    at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.loadFieldLayout(AbstractFieldLayoutManager.java:334)
    at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager.access$100(AbstractFieldLayoutManager.java:46)
    at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager$FieldLayoutCacheLoader.load(AbstractFieldLayoutManager.java:426)
    at com.atlassian.jira.issue.fields.layout.field.AbstractFieldLayoutManager$FieldLayoutCacheLoader.load(AbstractFieldLayoutManager.java:423)
    at com.atlassian.cache.memory.MemoryCacheManager$3$1.load(MemoryCacheManager.java:207)
    at com.atlassian.cache.memory.DelegatingCache.lambda$get$0(DelegatingCache.java:162)
    at com.atlassian.cache.memory.DelegatingCache.lambda$get$1(DelegatingCache.java:190)
    at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
    at com.atlassian.cache.memory.DelegatingCache.get(DelegatingCache.java:177)

[Adriana Stancu]

Can I just delete the field from the database? It seems to be refferenced in multiple tabels.

[Nic Brough -Adaptavist-]

No.  That will not help in the slightest.  You need to undo all the SQL damage you've done, putting the fields back to what they were before.