upload file to jira issues fails

[ddraveh]

when uploading file to jira issues (by browse or drag&drop, i get this error in a popup:

File "xxxx" was not uploaded

An internal error has occurred. Please contact your administrator.

i see the following exception in /var/atlassian/application-data/jira/log/atlassian-jira.log:

2019-09-19 18:53:14,211 http-nio-8080-exec-21 ERROR draveh 1133x4009x1 miwemq 10.100.102.3 /rest/internal/2/AttachTemporaryFile [c.a.p.r.c.error.jersey.ThrowableExceptionMapper] Uncaught exception thrown by REST service: null
java.lang.NullPointerException
at com.atlassian.plugins.whitelist.core.matcher.MatcherUtils.compare(MatcherUtils.java:59)
at com.atlassian.plugins.whitelist.core.matcher.MatcherUtils.compare(MatcherUtils.java:78)
at com.atlassian.plugins.whitelist.core.matcher.SelfUrlMatcher.apply(SelfUrlMatcher.java:27)
at com.atlassian.plugins.whitelist.core.AbstractWhitelist.isAllowed(AbstractWhitelist.java:41)
at com.atlassian.plugins.cors.WhitelistBasedCorsDefaults.allowsOrigin(WhitelistBasedCorsDefaults.java:49)
... 2 filtered
at com.google.common.collect.Iterators$5.computeNext(Iterators.java:639)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
at com.google.common.collect.Iterables.isEmpty(Iterables.java:988)
... 7 filtered
at com.atlassian.plugins.rest.module.RestDelegatingServletFilter$JerseyOsgiServletContainer.doFilter(RestDelegatingServletFilter.java:159)
... 1 filtered
at com.atlassian.plugins.rest.module.RestDelegatingServletFilter.doFilter(RestDelegatingServletFilter.java:69)
... 36 filtered
at com.atlassian.jira.plugin.mobile.web.filter.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:31)
... 4 filtered
at com.atlassian.jira.plugin.mobile.login.MobileLoginSuccessFilter.doFilter(MobileLoginSuccessFilter.java:54)
... 8 filtered
at com.atlassian.web.servlet.plugin.request.RedirectInterceptingFilter.doFilter(RedirectInterceptingFilter.java:21)
... 57 filtered
at com.atlassian.jira.security.JiraSecurityFilter.lambda$doFilter$0(JiraSecurityFilter.java:66)
... 1 filtered
at com.atlassian.jira.security.JiraSecurityFilter.doFilter(JiraSecurityFilter.java:64)
... 16 filtered
at com.atlassian.plugins.rest.module.servlet.RestSeraphFilter.doFilter(RestSeraphFilter.java:37)
... 19 filtered
at com.atlassian.jira.servermetrics.CorrelationIdPopulatorFilter.doFilter(CorrelationIdPopulatorFilter.java:30)
... 10 filtered
at com.atlassian.web.servlet.plugin.request.RedirectInterceptingFilter.doFilter(RedirectInterceptingFilter.java:21)
... 4 filtered
at com.atlassian.web.servlet.plugin.LocationCleanerFilter.doFilter(LocationCleanerFilter.java:36)
... 26 filtered
at com.atlassian.jira.servermetrics.MetricsCollectorFilter.doFilter(MetricsCollectorFilter.java:25)
... 25 filtered
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

[Andy Heinzer]

Hi @ddraveh 

Thanks for including the stacktrace here.  Sorry to hear that you are having problems uploading attachments to Jira.   I have an idea of what might be happening here.

In my search of this issue, I came across a this ticket: JRASERVER-45006 This is bug ticket for Jira that was closed as Won't Do, but the interesting thing about it is that it appears to show the same kind of error message and then also details a specific proxy configuration of Apache that appears to be causing this.

Are you by chance using an Apache server as a reverse proxy for Jira? 

The bug ticket indicates that

Add the following parameter in the proxy configuration (such as httpd.conf, httpd-vhosts.conf or jira.conf):

RequestHeader add Origin "Intranet"

can be the factor that causes this kind of CORS related error. On the comment thread there, David Black explains:

... In short the Origin header is required to not be tampered with for CORS and in some cases CSRF protection mechanisms.

In turn it looks like Jira's own security mechanisms are likely generating this error because of your current proxy configuration.

If this describes your environment, I could see two viable options to correct this:

1. Remove this Origin header alteration from the reverse proxy configuration
2. Guido Wischrop found he could alter some of the code in Jira in such a way to accept this altered header.  He details it in a comment https://jira.atlassian.com/browse/JRASERVER-45006?focusedCommentId=784240&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-784240

I hope this helps.  Please let me know if we can be of any additional help here.

Andy

[ddraveh]

Hi,

I don’t think we are using Apache as a reverse proxy server (is there a way to tell for sure?)

I didn’t find any of the files you mentioned on our machine. I looked at /opt/atlassian/jira/conf/server.xml but proxy setting there is marked.

Note that the issue started after upgrading Jira (and Confluence) to the new version. Before the upgrade file attachment works fine.

Thanks,

Danny

[Andy Heinzer]

Hi Danny,

It can be tricky to figure out in some cases.  I would try to use a utility like curl and then a command such as

curl -I http://jira.example.com

where the http://jira.example.com is the address you are using to serve up that Jira content, in order to see what comes back.  Basically I would be looking to see if there is a server header that is returned that might identify a proxy server. 

We can also look the server.xml to get some clues.  The proxyName and proxyPort values indicate the publicly facing proxy address.  Maybe that is apache, nginx, IIS, or some other proxy, OR possibly if you're using something like port redirection, for example iptables, you might still need a proxyPort/proxyName so that the Jira application can properly expect it's traffic to be received from a different address.

[yogev ch]

Hi @Andy Heinzer.

I don't think we have any proxy server configured.

1. This is the result of curl command (abbreviated). Pay attention to ajs-behind-proxy part:

<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=Edge"/>
<meta name="application-name" content="JIRA" data-name="jira" data-version="8.3.2"><meta name="ajs-server-scheme" content="http">
<meta name="ajs-server-port" content="8080">
<meta name="ajs-server-name" content="192.168.201.11">
<meta name="ajs-behind-proxy" content="null">
<meta name="ajs-base-url" content="http://192.268.201.11:8080">
<meta name="ajs-dev-mode" content="false">
<meta name="ajs-context-path" content="">
<meta name="ajs-version-number" content="8.3.2">
<meta name="ajs-build-number" content="803003">
<meta name="ajs-is-beta" content="false">
<meta name="ajs-is-rc" content="false">
<meta name="ajs-is-snapshot" content="false">
<meta name="ajs-is-milestone" content="false">
<meta name="ajs-remote-user" content="">
<meta name="ajs-remote-user-fullname" content="">
<meta name="ajs-user-locale" content="en_US">

And everywhere in our server.xml where proxy is explained, it's in comments.
For example:

<!--
==============================================================================================================
HTTP - Proxying Jira via Apache or Nginx over HTTP

If you're proxying traffic to Jira over HTTP, uncomment the below connector and comment out the others.
Ensure the proxyName and proxyPort are updated with the appropriate information if necessary as per the docs.

See the following for more information:

Apache - https://confluence.atlassian.com/x/4xQLM
nginx - https://confluence.atlassian.com/x/DAFmGQ
==============================================================================================================
-->

<!--
<Connector port="8080" relaxedPathChars="[]|" relaxedQueryChars="[]|{}^&#x5c;&#x60;&quot;&lt;&gt;"
maxThreads="150" minSpareThreads="25" connectionTimeout="20000" enableLookups="false"
maxHttpHeaderSize="8192" protocol="HTTP/1.1" useBodyEncodingForURI="true" redirectPort="8443"
acceptCount="100" disableUploadTimeout="true" bindOnInit="false" scheme="http"
proxyName="<subdomain>.<domain>.com" proxyPort="80"/>

-->

Same for HTTPS.

$ iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

$ iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination

Are there any other suspections ?

I saw this:

But on disk under that path we have already 4GB, and Jira provides maximum configuration of 2GB. Is it related? or this setting is per attachment?

Thanks,

Yogev

[Andy Heinzer]

The attachment size screenshot you posted is in regards to how large an individual attachment can be in Jira, not the total sum size of all attachments.  This is explained in Configuring file attachments.

Are you accessing the Jira site on the same address listed in the curl output? 

http://192.268.201.11:8080

Or are you reaching that site using a different address/port number?   If so, then it could explain the error we are seeing here. If not, then I'd be curious to see if we can see any problems in a browser console both when loading a Jira page, and when trying to upload a file.  Perhaps these can give us some additional clues as to the source of the problem here.  In addition to these browser logs, it might also be helpful to generate a HAR file.  These can contain additional information in regards to the browser/server interaction.

[yogev ch]

Are you accessing the Jira site on the same address listed in the curl output? 

Yes, this is also the Base URL:

Hmm,
I wanted to attach console/network logs, but cannot attach files nor paste them here due to this error:

So I don't know what to do.

Yogev

[Andy Heinzer]

I created a support case on your behalf in https://getsupport.atlassian.com/servicedesk/customer/portal/22/JSP-410045

You can upload these files to that case.  I have requested some more detailed log files you can upload there as well.  Together these files should better help us to understand your environment so that we can offer some assistance here.

Andy