Errors loading images/CSS/JS after upgrading to Crowd 3

[Laurens Blankers]

After upgrading to Crowd 3.0.1 from version 2.11.1 loading images, CSS, and JS files stopped working. It seems that the response header indicates that GZIP compression is used, but the content is considered invalid by the browser.

At the same time the following error is logged:

WARN [webresource] error in `Content.writeTo` for com.atlassian.plugins.atlassian-nps-plugin:nps-btf-admin-resources/btf-admin.js
java.lang.RuntimeException: com.atlassian.plugin.servlet.DownloadException: Unable to stream to the output
        at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$6.writeTo(ResourceServingHelpers.java:360)
        at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$2.lambda$writeTo$0(ResourceServingHelpers.java:184)
        at com.atlassian.plugin.cache.filecache.impl.PassThroughCache.cache(PassThroughCache.java:18)
        at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$2.writeTo(ResourceServingHelpers.java:184)
        at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$8.writeTo(ResourceServingHelpers.java:384)
[..]
Caused by: com.atlassian.plugin.servlet.DownloadException: Unable to stream to the output
        at com.atlassian.plugin.webresource.transformer.TransformerUtils.transformAndStreamResource(TransformerUtils.java:44)
        at com.atlassian.plugin.webresource.transformer.CharSequenceDownloadableResource.streamResource(CharSequenceDownloadableResource.java:42)
        at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$6.writeTo(ResourceServingHelpers.java:358)
        ... 114 more
Caused by: org.apache.catalina.connector.ClientAbortException: java.net.SocketException: Broken pipe (Write failed)
        at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:367)
        at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:331)
        at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:101)
        at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297)
        at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
        at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
        at org.apache.commons.io.IOUtils.copy(IOUtils.java:2128)
        at org.apache.commons.io.IOUtils.copy(IOUtils.java:2158)
        at com.atlassian.plugin.webresource.transformer.TransformerUtils.transformAndStreamResource(TransformerUtils.java:40)
        ... 116 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
        at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
        at org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:119)
        at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:800)
        at org.apache.coyote.Response.action(Response.java:172)
        at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:363)
        ... 124 more

Note that connecting using localhost does work because GZIP compression is then automatically disabled.

I found a workaround by disabling compression as described here:

https://confluence.atlassian.com/crowdkb/how-to-disable-http-compression-on-crowd-2-9-and-above-856714638.html

Why did this stop working? What is the nps plugin? And why is it messing with output streams?

[Ann Worley · Accepted solution]

I am happy to hear you already found a workaround.

I discussed your case with a Crowd developer and he suggested that the nps plugin was an innocent bystander, and we were running into this bug: Tomcat7 produces corrupted gzip response with newest zlib (1.2.11) library

The bug is fixed in Crowd 3.1, but should have also impacted your Crowd 2.11.1 instance. Is it possible that the zlib library was also upgraded recently?

[Laurens Blankers]

Looking at my logs I did indeed upgrade zlib (from version 1.2.8.dfsg-2+b1 to 1.2.8.dfsg-5), although this is not 1.2.11 I think some (security related) fixes where backported by Debian into 1.2.8, causing the issue as described in CWD-4968.

Nice to hear that this was already fixed, looking forward to Crowd 3.1!