JIRA CORE: After upgrade, search and "all issues and filters" broken

[Randall Williams]

After an upgrade to JIRA core 8.14.0, responses to queries don't display in the browser. The responses to the query DO appear in the browser debugger - I see them in the JSON response to the "issueTable" request, but the actual page where the response does not render the results. Right after upgrading we did install Jira Service Manager. I've performed a full reindex already. In the browser console, I see a requested rejected due to bad origin. 

Is this a known issue? Everything was good before the upgrade. 

[Patrice Eber]

In our case, the problem might be related to a not overwritten jpm.xml file during the upgrade process

(see https://confluence.atlassian.com/jirakb/issue-page-shows-up-blank-after-upgrade-1041075386.html)

a brand new jpm.xml, a restart, it's working again

[Daniel Ebers]

Hi @Randall Williams

I'd also be very interested if anything relating to this error is visible from logfile (atlassian-jira.log) from your instance.

Have you had the chance to check this already?
If so could you please share a probably WARNING or ERROR that is logged?

Thanks and regards,
Daniel

[Randall Williams]

@Daniel Ebers Thanks - I didn't look here previously. 

2021-01-04 07:37:07,368-0800 periodic-de-index-fixer-0 ERROR [c.a.jira.versioning.VersioningCleanupService] [VERSIONING] NumberFormatException during parsing entity version retention days, falling back to the default value. Property key: jira.entity.version.cleanup.deleted.retention.days, property value: null, default value: PT48H
2021-01-04 07:37:17,041-0800 https-openssl-nio-8443-exec-4 ERROR my-user-name 457x9082x1 sj3m3u 10.212.x.xxx:50380,10.44.x.xxx /secure/QuickSearch.jspa [c.atlassian.velocity.DefaultVelocityManager] MethodInvocationException occurred getting message body from Velocity: java.lang.NullPointerException
java.lang.NullPointerException
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1838)
at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
at java.lang.Double.parseDouble(Double.java:538)
at java.lang.Double.valueOf(Double.java:502)
at com.atlassian.jira.plugin.issuenav.properties.ParsingApplicationProperties.getFromProperty(ParsingApplicationProperties.java:49)
at com.atlassian.jira.plugin.issuenav.properties.ParsingApplicationProperties.getDouble(ParsingApplicationProperties.java:30)
at com.atlassian.jira.plugin.issuenav.IssueNavAction.getJqlAutocompleteQueryDelay(IssueNavAction.java:838)
... 2 filtered
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.velocity.runtime.parser.node.PropertyExecutor.execute(PropertyExecutor.java:142)
at org.apache.velocity.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:529)
at org.apache.velocity.runtime.parser.node.ASTIdentifier.execute(ASTIdentifier.java:198)
at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:262)
at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:342)
at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:336)
at org.apache.velocity.Template.merge(Template.java:328)
at org.apache.velocity.Template.merge(Template.java:235)
at org.apache.velocity.app.VelocityEngine.mergeTemplate(VelocityEngine.java:381)
at com.atlassian.velocity.DefaultVelocityManager.writeEncodedBodyImpl(DefaultVelocityManager.java:117)
at com.atlassian.velocity.DefaultVelocityManager.writeEncodedBody(DefaultVelocityManager.java:97)
at com.atlassian.jira.template.velocity.DefaultVelocityTemplatingEngine$DefaultRenderRequest.toWriterImpl(DefaultVelocityTemplatingEngine.java:122)
at com.atlassian.jira.template.velocity.DefaultVelocityTemplatingEngine$DefaultRenderRequest.asHtml(DefaultVelocityTemplatingEngine.java:113)
... 4 filtered
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
... 48 filtered
at com.atlassian.servicedesk.internal.web.ExternalCustomerLockoutFilter.doFilter(ExternalCustomerLockoutFilter.java:55)
... 4 filtered
at com.atlassian.diagnostics.internal.platform.monitor.http.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:55)
... 3 filtered
at com.atlassian.web.servlet.plugin.request.RedirectInterceptingFilter.doFilter(RedirectInterceptingFilter.java:21)
... 41 filtered
at com.atlassian.jira.security.JiraSecurityFilter.lambda$doFilter$0(JiraSecurityFilter.java:66)
... 1 filtered
at com.atlassian.jira.security.JiraSecurityFilter.doFilter(JiraSecurityFilter.java:64)
... 20 filtered
at com.atlassian.jira.servermetrics.CorrelationIdPopulatorFilter.doFilter(CorrelationIdPopulatorFilter.java:30)
... 5 filtered
at com.atlassian.servicedesk.internal.web.CustomerContextSettingFilter.doFilter(CustomerContextSettingFilter.java:118)
... 11 filtered
at com.atlassian.jira.servermetrics.MetricsCollectorFilter.doFilter(MetricsCollectorFilter.java:25)
... 7 filtered
at com.atlassian.jira.plugin.issuenav.IssueNavRewriteFilter.doFilter(IssueNavRewriteFilter.java:111)
... 4 filtered
at com.atlassian.web.servlet.plugin.request.RedirectInterceptingFilter.doFilter(RedirectInterceptingFilter.java:21)
... 53 filtered
at com.atlassian.jira.security.JiraSecurityFilter.lambda$doFilter$0(JiraSecurityFilter.java:66)
... 1 filtered
at com.atlassian.jira.security.JiraSecurityFilter.doFilter(JiraSecurityFilter.java:64)
... 20 filtered
at com.atlassian.jira.servermetrics.CorrelationIdPopulatorFilter.doFilter(CorrelationIdPopulatorFilter.java:30)
... 5 filtered
at com.atlassian.servicedesk.internal.web.CustomerContextSettingFilter.doFilter(CustomerContextSettingFilter.java:118)
... 11 filtered
at com.atlassian.jira.servermetrics.MetricsCollectorFilter.doFilter(MetricsCollectorFilter.java:25)
... 7 filtered
at com.atlassian.jira.plugin.issuenav.KickassRedirectFilter.doFilter(KickassRedirectFilter.java:133)
... 11 filtered
at com.atlassian.jira.security.JiraSecurityFilter.lambda$doFilter$0(JiraSecurityFilter.java:66)
... 1 filtered
at com.atlassian.jira.security.JiraSecurityFilter.doFilter(JiraSecurityFilter.java:64)
... 20 filtered
at com.atlassian.jira.servermetrics.CorrelationIdPopulatorFilter.doFilter(CorrelationIdPopulatorFilter.java:30)
... 5 filtered
at com.atlassian.jira.plugin.issuenav.LegacyTextQueryUrlRedirectFilter.doFilter(LegacyTextQueryUrlRedirectFilter.java:39)
... 4 filtered
at com.atlassian.servicedesk.internal.web.CustomerContextSettingFilter.doFilter(CustomerContextSettingFilter.java:118)
... 11 filtered
at com.atlassian.jira.servermetrics.MetricsCollectorFilter.doFilter(MetricsCollectorFilter.java:25)
... 7 filtered
at com.atlassian.jira.web.action.issue.QuickSearch.sendInternalRedirect(QuickSearch.java:50)
at com.atlassian.jira.web.action.issue.QuickSearch.doExecute(QuickSearch.java:43)
... 1 filtered
at com.atlassian.jira.action.JiraActionSupport.execute(JiraActionSupport.java:63)
... 7 filtered
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
... 48 filtered
at com.atlassian.servicedesk.internal.web.ExternalCustomerLockoutFilter.doFilter(ExternalCustomerLockoutFilter.java:55)
... 4 filtered
at com.atlassian.greenhopper.jira.filters.ClassicBoardRouter.doFilter(ClassicBoardRouter.java:56)
... 7 filtered
at com.atlassian.jira.plugin.mobile.web.filter.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:59)
... 4 filtered
at com.atlassian.jira.plugin.mobile.login.MobileLoginSuccessFilter.doFilter(MobileLoginSuccessFilter.java:54)
... 3 filtered
at com.atlassian.diagnostics.internal.platform.monitor.http.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:55)
... 8 filtered
at com.atlassian.web.servlet.plugin.request.RedirectInterceptingFilter.doFilter(RedirectInterceptingFilter.java:21)
... 62 filtered
at com.atlassian.jira.security.JiraSecurityFilter.lambda$doFilter$0(JiraSecurityFilter.java:66)
... 1 filtered
at com.atlassian.jira.security.JiraSecurityFilter.doFilter(JiraSecurityFilter.java:64)
... 39 filtered
at com.atlassian.jira.servermetrics.CorrelationIdPopulatorFilter.doFilter(CorrelationIdPopulatorFilter.java:30)
... 5 filtered
at com.atlassian.servicedesk.internal.web.CustomerContextSettingFilter.lambda$invokeFilterChain$0(CustomerContextSettingFilter.java:215)
at com.atlassian.servicedesk.internal.api.util.context.ReentrantThreadLocalBasedCodeContext.rteInvoke(ReentrantThreadLocalBasedCodeContext.java:136)
at com.atlassian.servicedesk.internal.api.util.context.ReentrantThreadLocalBasedCodeContext.runOutOfContext(ReentrantThreadLocalBasedCodeContext.java:89)
at com.atlassian.servicedesk.internal.utils.context.CustomerContextServiceImpl.runOutOfCustomerContext(CustomerContextServiceImpl.java:47)
at com.atlassian.servicedesk.internal.web.CustomerContextSettingFilter.outOfCustomerContext(CustomerContextSettingFilter.java:206)
at com.atlassian.servicedesk.internal.web.CustomerContextSettingFilter.doFilterImpl(CustomerContextSettingFilter.java:134)
at com.atlassian.servicedesk.internal.web.CustomerContextSettingFilter.doFilter(CustomerContextSettingFilter.java:123)
... 4 filtered
at com.atlassian.jwt.internal.servlet.JwtAuthFilter.doFilter(JwtAuthFilter.java:37)
... 8 filtered
at com.atlassian.web.servlet.plugin.request.RedirectInterceptingFilter.doFilter(RedirectInterceptingFilter.java:21)
... 4 filtered
at com.atlassian.web.servlet.plugin.LocationCleanerFilter.doFilter(LocationCleanerFilter.java:36)
... 29 filtered
at com.atlassian.jira.servermetrics.MetricsCollectorFilter.doFilter(MetricsCollectorFilter.java:25)
... 24 filtered
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

[Randall Williams]

ran the integrity checker, no issues there

[Daniel Ebers]

The integrity checker would not be able to diagnose this kind of issue - to my knowledge.
Have you had a look into atlassian-jira.log if anything related to this can be spotted?

[Daniel Ebers]

There are some errors which I were not able to find a root cause for using knowledge base.

Also this one: https://confluence.atlassian.com/jirakb/jira-throws-nullpointerexception-in-issue-navigator-867013154.html
makes sense but in your case Integrity Checker is not able to find anything related.

1.) Have you tried to restart the Jira instance - is it after that also occurring?
2.) Have you disabled any Apps for a test and/or have all updated to the latest available version after upgrading Jira?

Regards,
Daniel

[Christian Schopf]

Same here, came from Jira Server 8.8.1 to 8.14.1
Test instance worked fine
Production instance shows this issue about 12h past the upgrade.
Reindexed which fixed the issue for a few hours before it came back.

Jira support working with me on it right now.

Integrity Checker -> some custom field thing for one issue, fixed but no changes.
Instance restart -> no effect
Safe Mode -> no effect
delete plugin cache -> no effect

Beside this error 

periodic-de-index-fixer-0 ERROR [c.a.jira.versioning.VersioningCleanupService] [VERSIONING] NumberFormatException during parsing entity version retention days, falling back to the default value. Property key: jira.entity.version.cleanup.deleted.retention.days, property value: null, default value: PT48H

 I have these:

https-openssl-nio-443-exec-22 ERROR USERNAME 1013x4871x1 ezcz7q 127.0.0.1 /secure/QueryComponent!Jql.jspa [c.a.j.issue.fields.DefaultFieldManager] FieldLayoutItem for field named Archiver with ID archivedby is null. Usually this is a transient race condition due to concurrent modification. Please contact Atlassian support if you see this log message frequently.

and these:

https-openssl-nio-443-exec-22 ERROR USERNAME 1013x4871x1 ezcz7q 127.0.0.1 /secure/QueryComponent!Jql.jspa [c.a.j.issue.fields.DefaultFieldManager] FieldLayoutItem for field named Archived with ID archiveddate is null. Usually this is a transient race condition due to concurrent modification. Please contact Atlassian support if you see this log message frequently.

I have no custom fields Archiver or Archivedby.
We use Jira Software Server, not datacenter, therefor there is no archival feature.

Also when I upgraded on Sat, there was 8.14.1 and 8.15
Now there is a version 8.14.2 

If you were able to solve the issue, it would be great, if you could share what worked for you. @Randall Williams 

greets

[Jacek Zarzycki]

Hi,
I have exactly the same problem. There are over
5000 such errors in Logs every day.

[Tim Theis]

Have you identified anything on this error? Having the same logs reported repeatedly after upgrading to 8.14.

[Jacek Zarzycki]

Hi
If you have this problem, join "Affected customers:" here

https://jira.atlassian.com/browse/JRASERVER-72181

[Brant Schroeder]

Randall,

  Is this happening for all users?  Does it happen in all browsers?  Have you cleared your browser cache since you upgraded?

[Randall Williams]

@Brant Schroeder 

A. Yes, all users

B. Yes, all browsers.

C. Yes, cache cleared. 

[Brant Schroeder]

What version did you upgrade from?