Intermediate confluence 7.4.1 crash

[Pandiyan Muthuraman]

recently production server is up dated to 7.4.1 version.

The confluence service crashes very frequently with below message

ERROR [http-nio-8090-exec-11] [[Standalone].[localhost].[/].[file-server]] log Servlet.service() for servlet [file-server] threw exception
-- url: /s/-guhr2o/8401/758836457240a7983be3b3db3463da30107b780e/1.0.9/_/download/resources/com.atlassian.confluence.plugins.confluence-icons:confluence-icon-resources/assets/icons/confluence-icon-video.svg |
traceId: 1f5312c88f0c2307 | userName: anonymous

ERROR [http-nio-8090-exec-11] [[Standalone].[localhost].[/].[noop]] log Servlet.service() for servlet [noop] in context with path [] threw exception
java.lang.RuntimeException: Cannot read resource assets/icons/confluence-icon-video.svg
at com.atlassian.plugin.webresource.impl.snapshot.Resource$1.writeTo(Resource.java:204)
at com.atlassian.plugin.webresource.transformer.CompileTimeTransformer$1.writeTo(CompileTimeTransformer.java:59)
at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$8.writeTo(ResourceServingHelpers.java:465)
at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$9.writeTo(ResourceServingHelpers.java:477)
at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$3.writeTo(ResourceServingHelpers.java:264)
at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$2.lambda$writeTo$0(ResourceServingHelpers.java:187)
at com.atlassian.plugin.cache.filecache.impl.PassThroughCache.cache(PassThroughCache.java:18)
at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$2.writeTo(ResourceServingHelpers.java:187)
at com.atlassian.plugin.webresource.impl.http.Controller.lambda$sendCachedInProduction$0(Controller.java:325)

--------------------------
JVM Stats
--------------------------
xmx = 4294967296
usedNonHeap = 504520088
availableHeap = 2915605528
availableNonHeap = -1
allocatedHeap = 2147483648
freeAllocatedHeap = 768121880
maxNonHeap = -1
availablePermGen = 0
maxPermGen = -1
maxHeap = 4294967296
usedHeap = 1379361768
usedPermGen = -1
xms = 2147483648
--------------------------
Request Information
--------------------------
URL: https://dashboard/500page.jsp
Scheme: https
Server: dashboard
Port: 443
URI: /500page.jsp
Context Path:
Servlet Path: /500page.jsp
Path Info: null
Query String: null

--------------------------
Attributes
--------------------------
javax.servlet.forward.request_uri: /s/-guhr2o/8401/758836457240a7983be3b3db3463da30107b780e/1.0.9/_/download/resources/com.atlassian.confluence.plugins.confluence-icons:confluence-icon-resources/assets/icons/confluence-icon-video.svg
javax.servlet.forward.context_path:
javax.servlet.forward.servlet_path: /s
javax.servlet.forward.path_info: /-guhr2o/8401/758836457240a7983be3b3db3463da30107b780e/1.0.9/_/download/resources/com.atlassian.confluence.plugins.confluence-icons:confluence-icon-resources/assets/icons/confluence-icon-video.svg
javax.servlet.forward.mapping: org.apache.catalina.core.ApplicationMapping$MappingImpl@6d5bc8d1
javax.servlet.error.status_code: 500
_statichash: -guhr2o/8401/758836457240a7983be3b3db3463da30107b780e/1.0.9
com.atlassian.confluence.impl.profiling.DecoratorTimings: com.atlassian.confluence.impl.profiling.DecoratorTimings@317e1276
com.opensymphony.sitemesh.APPLIED_ONCE: true
__cleanup_recursion_counter: 0
javax.servlet.error.message:
javax.servlet.error.servlet_name: noop
com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter_already_filtered: true
atlassian.core.seraph.original.url: /500page.jsp
com.atlassian.labs.botkiller.BotKillerFilter: true
com.atlassian.gzipfilter.GzipFilter_already_filtered: true
Confluence-Request-Time: 1611417825474
B3-TraceId: 779fa9ec1b773c
loginfilter.already.filtered: true
javax.servlet.error.request_uri: /s/-guhr2o/8401/758836457240a7983be3b3db3463da30107b780e/1.0.9/_/download/resources/com.atlassian.confluence.plugins.confluence-icons:confluence-icon-resources/assets/icons/confluence-icon-video.svg
com.atlassian.core.filters.HeaderSanitisingFilter_already_filtered: true
com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFixupFilter: true
com.atlassian.confluence.web.ConfluenceJohnsonFilter_already_filtered: true
javax.servlet.error.exception: java.lang.RuntimeException: Cannot read resource assets/icons/confluence-icon-video.svg
os_securityfilter_already_filtered: true
com.atlassian.seraph.auth.LoginReason: OUT
com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFilter: true
--------------------------
Parameters
--------------------------
caused by: java.lang.RuntimeException: Cannot read resource assets/icons/confluence-icon-video.svg
at com.atlassian.plugin.webresource.impl.snapshot.Resource$1.writeTo(Resource.java:204)

2021-01-23 16:04:29,854 WARN [http-nio-8090-exec-30] [common.security.jersey.XsrfResourceFilter] passesAllXsrfChecks XSRF checks failed for request: https://dashboard/rest/spacetree/1.0/spaces , origin: https://dashboard , referrer: https://dashboard/display/HotSpot/VirtualCalls
-- referer: https://dashboard/display/HotSpot/VirtualCalls | url: /rest/spacetree/1.0/spaces | traceId: 933b5d509e080cf0 | userName: anonymous

[Gonchik Tsymzhitov]

Hi @Pandiyan Muthuraman !  

Did you clean plugin cache ? https://confluence.atlassian.com/confkb/how-to-clear-confluence-plugins-cache-297664846.html

Check the rights ?

And double the logs?

https://jira.atlassian.com/browse/CONFSERVER-55062

[Pandiyan Muthuraman]

Cleaning plugin cache is workaround and temporary solution. Its happening frequently even after cleaning up.

what was the root cause and how it can be fixed permanently.  

Do you think JDK need to be updated ? currently am using jdk 8u271 and solaris x86 machine ( JDK 11 is not supported). 

[Gonchik Tsymzhitov]

@Pandiyan Muthuraman As I understood, it works after cleanup is that right?

But the problem it happened again ?

Could you check log in catalina.out please? 

[Pandiyan Muthuraman]

Yes, the issue is happening after cleanup plugin-cache in a day or two. Here is the  catalina.out captured during issue. 

23-Jan-2021 18:32:51.547 WARNING [http-nio-8090-exec-15] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://dashboard/rest/spacetree/1.0/spaces, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
23-Jan-2021 18:32:56.614 WARNING [http-nio-8090-exec-6] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://dashboard/rest/spacetree/1.0/spaces, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
23-Jan-2021 18:33:16.630 WARNING [http-nio-8090-exec-16] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://dashboard/rest/spacetree/1.0/spaces, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
23-Jan-2021 18:33:48.879 WARNING [http-nio-8090-exec-24] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://dashboard/rest/spacetree/1.0/spaces, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
23-Jan-2021 18:34:47.663 WARNING [http-nio-8090-exec-24] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://dashboard/rest/spacetree/1.0/spaces, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
23-Jan-2021 18:34:47.931 SEVERE [http-nio-8090-exec-10] com.sun.jersey.spi.container.ContainerResponse.mapException Exception mapper com.atlassian.plugins.rest.common.security.jersey.SecurityExceptionMapper@5e770952 for Throwable com.atlassian.plugins.rest.common.security.AuthenticationRequiredException: Client must be authenticated to access this resource. threw a RuntimeException when attempting to obtain the response





23-Jan-2021 18:34:47.932 SEVERE [http-nio-8090-exec-10] com.sun.jersey.spi.container.ContainerResponse.logException Mapped exception to response: 500 (Internal Server Error)
javax.ws.rs.WebApplicationException: java.text.ParseException: String, es-419, is not a valid language tag
at com.sun.jersey.server.impl.model.HttpHelper.clientError(HttpHelper.java:273)
at com.sun.jersey.server.impl.model.HttpHelper.getAcceptLanguage(HttpHelper.java:228)
at com.sun.jersey.server.impl.VariantSelector.selectVariant(VariantSelector.java:290)
at com.sun.jersey.spi.container.ContainerRequest.selectVariant(ContainerRequest.java:696)
at com.sun.jersey.spi.container.AdaptingContainerRequest.selectVariant(AdaptingContainerRequest.java:286)
at sun.reflect.GeneratedMethodAccessor1564.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jersey.server.impl.application.WebApplicationImpl$2.invoke(WebApplicationImpl.java:301)
at com.sun.proxy.$Proxy2846.selectVariant(Unknown Source)
at com.atlassian.plugins.rest.common.Status.variantFor(Status.java:304)
at com.atlassian.plugins.rest.common.security.jersey.SecurityExceptionMapper.toResponse(SecurityExceptionMapper.java:22)
at com.atlassian.plugins.rest.common.security.jersey.SecurityExceptionMapper.toResponse(SecurityExceptionMapper.java:16)
at com.sun.jersey.spi.container.ContainerResponse.mapException(ContainerResponse.java:480)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1479)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at com.atlassian.plugins.rest.module.RestDelegatingServletFilter$JerseyOsgiServletContainer.doFilter(RestDelegatingServletFilter.java:162)
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
at com.atlassian.plugins.rest.module.RestDelegatingServletFilter.doFilter(RestDelegatingServletFilter.java:72)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:55)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:43)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
at com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilterInternal(RestServletUtilsUpdaterFilter.java:22)
at com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilter(RestServletUtilsUpdaterFilter.java:34)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)

[Daniel Ebers]

The error message you have in logs

com.sun.jersey.spi.container.ContainerResponse.logException Mapped exception to response: 500 (Internal Server Error)
javax.ws.rs.WebApplicationException: java.text.ParseException: String, es-419, is not a valid language tag

might be related to this Bug: https://jira.atlassian.com/browse/CONFSERVER-59540
But I doubt this is the reason why the Confluence instances crashes.
For me there is nothing obvious visible from logs why I only could refer to a more general guide:
https://confluence.atlassian.com/confkb/troubleshooting-confluence-hanging-or-crashing-163415613.html

[Pandiyan Muthuraman]

I have checked atlassian and catalina old logs, where could see below reported issues are occurring for very long time (even before upgrade) and having work around to fix solution. So this couldn't be cause of issue

• not a valid language tag
• Cannot read resource assets/icons/confluence-icon-video.svg .

I have gone through checklist  https://confluence.atlassian.com/confkb/troubleshooting-confluence-hanging-or-crashing-163415613.html everything looks normal.

I suspect some plugins (atlassian, other vendors) causing issue. Is there any open source / free atlassian plugin to track plugin and generate reports ?

[Daniel Ebers]

Not that I would be aware of - but to start with basic things, if you suspect an App being responsible for the crashed you could check if they are up to date. If not, you would need to upgrade them. In case one behaves suspicious you also could check with it's vendor.

[Gonchik Tsymzhitov]

@Pandiyan Muthuraman  Just curious did you raised a ticket to getsupport.atlassian.com. 

Or we need to review all logs from support.zip. 

Also, during crash, any problem on GC level ?