JIRA Some gadgets broken after upgrade to 7.1.1

Matt Gillan March 8, 2016

I have just upgraded our sandbox install from JIRA 6.3.15 to JIRA 7.1.1, and am experiencing problem very similar to this issue: https://answers.atlassian.com/questions/24649645

I have pinpointed the issue to SSL, the issue appears and disappears when switching between the two security modes for JIRA 7.1.1 outlined below:

  • Tested JIRA 6.3.15 using HTTP only - Gadgets function correctly
  • Tested JIRA 6.3.15 using HTTP and HTTPS (redirect HTTP to HTTPs), set port to 8090 and using our own keystore - Gadgets function correctly
  • Tested JIRA 7.1.1 using HTTP only - Gadgets function correctly
  • Tested JIRA 7.1.1 using HTTP and HTTPS (redirect HTTP to HTTPs), set port to 8090 and using our own keystore - Gadgets are broken

 

All installed add-ons are up to date.

 

Here's what I'm seeing:

Some gadgets have the correct titles, other don't.

Here's the list of "broken" gadgets:

  • __MSG_gadget.activity.stream.title__

  • __MSG_gadget.filter.results.title__

  • __MSG_gadget.watched.title__

Gadgets.jpg

My focus was on the Activity Stream gadget:

  • My existing Activity Stream gadget displays most information, the status information at the bottom of each entry (xx minutes age, Comment, Watch) is missing.
  • Trying to add it to my dashboard leaves it hanging on the loading bar.  

Gadget2.jpg

  • I can select some options while it is in this state, and choosing configure or reload opens a very broken window:Gadget3.jpg

 

Looking at the Catalina logs

  • On startup of JIRA:

    09-Mar-2016 15:12:37.374 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version: Apache Tomcat/8.0.17
    09-Mar-2016 15:12:37.374 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Jan 9 2015 15:58:59 UTC
    09-Mar-2016 15:12:37.375 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number: 8.0.17.0
    09-Mar-2016 15:12:37.375 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
    09-Mar-2016 15:12:37.375 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 3.2.0-4-amd64
    09-Mar-2016 15:12:37.375 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
    09-Mar-2016 15:12:37.375 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JAVA_HOME: /opt/atlassian/jira/jre
    09-Mar-2016 15:12:37.375 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 1.8.0_51-b16
    09-Mar-2016 15:12:37.375 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Oracle Corporation
    09-Mar-2016 15:12:37.375 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /opt/atlassian/jira
    09-Mar-2016 15:12:37.376 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /opt/atlassian/jira
    09-Mar-2016 15:12:37.376 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/opt/atlassian/jira/conf/logging.properties
    09-Mar-2016 15:12:37.376 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
    09-Mar-2016 15:12:37.376 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms384m
    09-Mar-2016 15:12:37.376 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx768m
    09-Mar-2016 15:12:37.376 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.awt.headless=true
    09-Mar-2016 15:12:37.377 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Datlassian.standalone=JIRA
    09-Mar-2016 15:12:37.377 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true
    09-Mar-2016 15:12:37.380 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dmail.mime.decodeparameters=true
    09-Mar-2016 15:12:37.381 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.dom4j.factory=com.atlassian.core.xml.InterningDocumentFactory
    09-Mar-2016 15:12:37.381 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Datlassian.mail.senddisabled=true
    09-Mar-2016 15:12:37.381 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Datlassian.mail.fetchdisabled=true
    09-Mar-2016 15:12:37.382 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Datlassian.mail.popdisabled=true
    09-Mar-2016 15:12:37.382 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+PrintGCDateStamps
    09-Mar-2016 15:12:37.382 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:-OmitStackTraceInFastThrow
    09-Mar-2016 15:12:37.382 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.endorsed.dirs=/opt/atlassian/jira/endorsed
    09-Mar-2016 15:12:37.382 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/opt/atlassian/jira
    09-Mar-2016 15:12:37.382 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/opt/atlassian/jira
    09-Mar-2016 15:12:37.382 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/opt/atlassian/jira/temp
    09-Mar-2016 15:12:37.383 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    09-Mar-2016 15:12:38.695 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
    09-Mar-2016 15:12:38.856 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
    09-Mar-2016 15:12:38.858 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-bio-8090"]
    09-Mar-2016 15:12:40.204 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 4580 ms
    09-Mar-2016 15:12:40.335 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Catalina
    09-Mar-2016 15:12:40.335 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.17
    09-Mar-2016 15:13:06.644 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
    09-Mar-2016 15:13:06.666 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-bio-8090"]
    09-Mar-2016 15:13:06.675 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 26466 ms

    09-Mar-2016 15:12:37.336 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'maxSpareThreads' to '75' did not find a matching property.

  • On JIRA Stop:
    09-Mar-2016 15:31:18.873 INFO [main] org.apache.catalina.core.StandardServer.await A valid shutdown command was received via the shutdown port. Stopping the Server instance.
    09-Mar-2016 15:31:18.873 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8080"]
    09-Mar-2016 15:31:18.925 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-bio-8090"]
    09-Mar-2016 15:31:18.975 INFO [main] org.apache.catalina.core.StandardService.stopInternal Stopping service Catalina
    09-Mar-2016 15:31:33.057 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
    java.lang.Object.wait(Native Method)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
    com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)
    09-Mar-2016 15:31:33.058 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [JonasBatch] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:502)
    org.objectweb.jonas_timer.TimerManager.batch(TimerManager.java:193)
    org.objectweb.jonas_timer.Batch.run(TimerManager.java:65)
    09-Mar-2016 15:31:33.058 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [JonasClock] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
    java.lang.Thread.sleep(Native Method)
    org.objectweb.jonas_timer.TimerManager.clock(TimerManager.java:142)
    org.objectweb.jonas_timer.Clock.run(TimerManager.java:46)
    09-Mar-2016 15:31:33.069 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8080"]
    09-Mar-2016 15:31:33.157 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-bio-8090"]
    09-Mar-2016 15:31:33.157 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8080"]
    09-Mar-2016 15:31:33.161 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-bio-8090"]

 

5 answers

1 accepted

1 vote
Answer accepted
Matt Gillan March 15, 2016

I solved my problem by adding the following to /opt/atlassian/jira/bin/catalina.sh

JAVA_OPTS="$JAVA_OPTS -Djavax.net.ssl.trustStore=$CATALINA_HOME/certificates/truststore.ks -Djavax.net.ssl.trustStorePassword=truststorePassword -server"

Chris Stamp March 16, 2016

I would say the setenv.sh file as that is the enviroment variable scrip apparently. Thanks for this. It gives some sort of clarity. This seems  like a major problem if it was such a pervasive issue.

Matt Gillan March 16, 2016

I looked into this a little further and found that the solution I used (regardless of where you change the Djavax.net.ssl.trustStore breaks connectivity to the Atlassian Marketplace.

There's two ways around this:

1) Don't change the Djavax.net.ssl.trustStore location at all.  Instead import your SSL certs into Jira's default cacerts keystore.

2) Change the Djavax.net.ssl.trustStore to point at your custom keystore, and import Atlassian's certs into your custom keystore (either download them or export from cacerts)

1 vote
Matt Gillan March 10, 2016

I was pointed to this link by Atlassian support, I hope it helps some of you:

https://confluence.atlassian.com/jirakb/how-to-fix-gadget-titles-showing-as-__msg_gadget-813697086.html

1 vote
Chris Stamp March 10, 2016

Its like they don't care... Its really fustrating

0 votes
trackhe May 5, 2016

I have a similar problem on the Profile:

002.png

and on the Gadget:

001.png

0 votes
Andrew McKay March 10, 2016

Having a similar issue with a similar configuration, any help that could be given would be great.

 

Thanks

Suggest an answer

Log in or Sign up to answer