Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Intermediate confluence 7.4.1 crash

Edited

recently production server is up dated to 7.4.1 version.

The confluence service crashes very frequently with below message

 

ERROR [http-nio-8090-exec-11] [[Standalone].[localhost].[/].[file-server]] log Servlet.service() for servlet [file-server] threw exception
-- url: /s/-guhr2o/8401/758836457240a7983be3b3db3463da30107b780e/1.0.9/_/download/resources/com.atlassian.confluence.plugins.confluence-icons:confluence-icon-resources/assets/icons/confluence-icon-video.svg |
traceId: 1f5312c88f0c2307 | userName: anonymous


ERROR [http-nio-8090-exec-11] [[Standalone].[localhost].[/].[noop]] log Servlet.service() for servlet [noop] in context with path [] threw exception
java.lang.RuntimeException: Cannot read resource assets/icons/confluence-icon-video.svg
at com.atlassian.plugin.webresource.impl.snapshot.Resource$1.writeTo(Resource.java:204)
at com.atlassian.plugin.webresource.transformer.CompileTimeTransformer$1.writeTo(CompileTimeTransformer.java:59)
at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$8.writeTo(ResourceServingHelpers.java:465)
at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$9.writeTo(ResourceServingHelpers.java:477)
at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$3.writeTo(ResourceServingHelpers.java:264)
at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$2.lambda$writeTo$0(ResourceServingHelpers.java:187)
at com.atlassian.plugin.cache.filecache.impl.PassThroughCache.cache(PassThroughCache.java:18)
at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$2.writeTo(ResourceServingHelpers.java:187)
at com.atlassian.plugin.webresource.impl.http.Controller.lambda$sendCachedInProduction$0(Controller.java:325)


--------------------------
JVM Stats
--------------------------
xmx = 4294967296
usedNonHeap = 504520088
availableHeap = 2915605528
availableNonHeap = -1
allocatedHeap = 2147483648
freeAllocatedHeap = 768121880
maxNonHeap = -1
availablePermGen = 0
maxPermGen = -1
maxHeap = 4294967296
usedHeap = 1379361768
usedPermGen = -1
xms = 2147483648
--------------------------
Request Information
--------------------------
URL: https://dashboard/500page.jsp
Scheme: https
Server: dashboard
Port: 443
URI: /500page.jsp
Context Path:
Servlet Path: /500page.jsp
Path Info: null
Query String: null


--------------------------
Attributes
--------------------------
javax.servlet.forward.request_uri: /s/-guhr2o/8401/758836457240a7983be3b3db3463da30107b780e/1.0.9/_/download/resources/com.atlassian.confluence.plugins.confluence-icons:confluence-icon-resources/assets/icons/confluence-icon-video.svg
javax.servlet.forward.context_path:
javax.servlet.forward.servlet_path: /s
javax.servlet.forward.path_info: /-guhr2o/8401/758836457240a7983be3b3db3463da30107b780e/1.0.9/_/download/resources/com.atlassian.confluence.plugins.confluence-icons:confluence-icon-resources/assets/icons/confluence-icon-video.svg
javax.servlet.forward.mapping: org.apache.catalina.core.ApplicationMapping$MappingImpl@6d5bc8d1
javax.servlet.error.status_code: 500
_statichash: -guhr2o/8401/758836457240a7983be3b3db3463da30107b780e/1.0.9
com.atlassian.confluence.impl.profiling.DecoratorTimings: com.atlassian.confluence.impl.profiling.DecoratorTimings@317e1276
com.opensymphony.sitemesh.APPLIED_ONCE: true
__cleanup_recursion_counter: 0
javax.servlet.error.message:
javax.servlet.error.servlet_name: noop
com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter_already_filtered: true
atlassian.core.seraph.original.url: /500page.jsp
com.atlassian.labs.botkiller.BotKillerFilter: true
com.atlassian.gzipfilter.GzipFilter_already_filtered: true
Confluence-Request-Time: 1611417825474
B3-TraceId: 779fa9ec1b773c
loginfilter.already.filtered: true
javax.servlet.error.request_uri: /s/-guhr2o/8401/758836457240a7983be3b3db3463da30107b780e/1.0.9/_/download/resources/com.atlassian.confluence.plugins.confluence-icons:confluence-icon-resources/assets/icons/confluence-icon-video.svg
com.atlassian.core.filters.HeaderSanitisingFilter_already_filtered: true
com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFixupFilter: true
com.atlassian.confluence.web.ConfluenceJohnsonFilter_already_filtered: true
javax.servlet.error.exception: java.lang.RuntimeException: Cannot read resource assets/icons/confluence-icon-video.svg
os_securityfilter_already_filtered: true
com.atlassian.seraph.auth.LoginReason: OUT
com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFilter: true
--------------------------
Parameters
--------------------------
caused by: java.lang.RuntimeException: Cannot read resource assets/icons/confluence-icon-video.svg
at com.atlassian.plugin.webresource.impl.snapshot.Resource$1.writeTo(Resource.java:204)

2021-01-23 16:04:29,854 WARN [http-nio-8090-exec-30] [common.security.jersey.XsrfResourceFilter] passesAllXsrfChecks XSRF checks failed for request: https://dashboard/rest/spacetree/1.0/spaces , origin: https://dashboard , referrer: https://dashboard/display/HotSpot/VirtualCalls
-- referer: https://dashboard/display/HotSpot/VirtualCalls | url: /rest/spacetree/1.0/spaces | traceId: 933b5d509e080cf0 | userName: anonymous

1 answer

Cleaning plugin cache is workaround and temporary solution. Its happening frequently even after cleaning up.

what was the root cause and how it can be fixed permanently.  

Do you think JDK need to be updated ? currently am using jdk 8u271 and solaris x86 machine ( JDK 11 is not supported). 

Like Gonchik Tsymzhitov likes this

@Pandiyan Muthuraman As I understood, it works after cleanup is that right?

But the problem it happened again ?

Could you check log in catalina.out please? 

Yes, the issue is happening after cleanup plugin-cache in a day or two. Here is the  catalina.out captured during issue. 

23-Jan-2021 18:32:51.547 WARNING [http-nio-8090-exec-15] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://dashboard/rest/spacetree/1.0/spaces, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
23-Jan-2021 18:32:56.614 WARNING [http-nio-8090-exec-6] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://dashboard/rest/spacetree/1.0/spaces, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
23-Jan-2021 18:33:16.630 WARNING [http-nio-8090-exec-16] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://dashboard/rest/spacetree/1.0/spaces, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
23-Jan-2021 18:33:48.879 WARNING [http-nio-8090-exec-24] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://dashboard/rest/spacetree/1.0/spaces, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
23-Jan-2021 18:34:47.663 WARNING [http-nio-8090-exec-24] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://dashboard/rest/spacetree/1.0/spaces, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
23-Jan-2021 18:34:47.931 SEVERE [http-nio-8090-exec-10] com.sun.jersey.spi.container.ContainerResponse.mapException Exception mapper com.atlassian.plugins.rest.common.security.jersey.SecurityExceptionMapper@5e770952 for Throwable com.atlassian.plugins.rest.common.security.AuthenticationRequiredException: Client must be authenticated to access this resource. threw a RuntimeException when attempting to obtain the response





23-Jan-2021 18:34:47.932 SEVERE [http-nio-8090-exec-10] com.sun.jersey.spi.container.ContainerResponse.logException Mapped exception to response: 500 (Internal Server Error)
javax.ws.rs.WebApplicationException: java.text.ParseException: String, es-419, is not a valid language tag
at com.sun.jersey.server.impl.model.HttpHelper.clientError(HttpHelper.java:273)
at com.sun.jersey.server.impl.model.HttpHelper.getAcceptLanguage(HttpHelper.java:228)
at com.sun.jersey.server.impl.VariantSelector.selectVariant(VariantSelector.java:290)
at com.sun.jersey.spi.container.ContainerRequest.selectVariant(ContainerRequest.java:696)
at com.sun.jersey.spi.container.AdaptingContainerRequest.selectVariant(AdaptingContainerRequest.java:286)
at sun.reflect.GeneratedMethodAccessor1564.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jersey.server.impl.application.WebApplicationImpl$2.invoke(WebApplicationImpl.java:301)
at com.sun.proxy.$Proxy2846.selectVariant(Unknown Source)
at com.atlassian.plugins.rest.common.Status.variantFor(Status.java:304)
at com.atlassian.plugins.rest.common.security.jersey.SecurityExceptionMapper.toResponse(SecurityExceptionMapper.java:22)
at com.atlassian.plugins.rest.common.security.jersey.SecurityExceptionMapper.toResponse(SecurityExceptionMapper.java:16)
at com.sun.jersey.spi.container.ContainerResponse.mapException(ContainerResponse.java:480)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1479)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at com.atlassian.plugins.rest.module.RestDelegatingServletFilter$JerseyOsgiServletContainer.doFilter(RestDelegatingServletFilter.java:162)
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
at com.atlassian.plugins.rest.module.RestDelegatingServletFilter.doFilter(RestDelegatingServletFilter.java:72)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:55)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:43)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
at com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilterInternal(RestServletUtilsUpdaterFilter.java:22)
at com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilter(RestServletUtilsUpdaterFilter.java:34)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)

 

Daniel Ebers Community Leader Jan 25, 2021

The error message you have in logs

com.sun.jersey.spi.container.ContainerResponse.logException Mapped exception to response: 500 (Internal Server Error)
javax.ws.rs.WebApplicationException: java.text.ParseException: String, es-419, is not a valid language tag

might be related to this Bug: https://jira.atlassian.com/browse/CONFSERVER-59540
But I doubt this is the reason why the Confluence instances crashes.
For me there is nothing obvious visible from logs why I only could refer to a more general guide:
https://confluence.atlassian.com/confkb/troubleshooting-confluence-hanging-or-crashing-163415613.html

I have checked atlassian and catalina old logs, where could see below reported issues are occurring for very long time (even before upgrade) and having work around to fix solution. So this couldn't be cause of issue

  • not a valid language tag
  • Cannot read resource assets/icons/confluence-icon-video.svg .

I have gone through checklist  https://confluence.atlassian.com/confkb/troubleshooting-confluence-hanging-or-crashing-163415613.html everything looks normal.

I suspect some plugins (atlassian, other vendors) causing issue. Is there any open source / free atlassian plugin to track plugin and generate reports ?

Daniel Ebers Community Leader Jan 25, 2021

Not that I would be aware of - but to start with basic things, if you suspect an App being responsible for the crashed you could check if they are up to date. If not, you would need to upgrade them. In case one behaves suspicious you also could check with it's vendor.

@Pandiyan Muthuraman  Just curious did you raised a ticket to getsupport.atlassian.com. 

 

Or we need to review all logs from support.zip. 

Also, during crash, any problem on GC level ?

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Confluence

Confluence Mythbusters: Does Atlassian even use Confluence?

Hi, Confluence collaborators! As part of #Confluence-Collaboratory month, we’ve created a very special Mythsbusters segment, where we're dive into an interesting myth and uncover the truth behind i...

1,364 views 7 28
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you