NoClassDefFoundError, ClassNotFoundExcception or NoSuchMethodException due to Incompatible Plugin

[Sue Bridges]

Had problems with JIRA Stability this week, despite it being fine since I upgraded it in April to 8.1. So, I've been running the Log Analyser more than once a day since! Initially I saw Java Heap issues so I've made sure that the -Xmx parameter in the JIRA Service is 2048MB as seen in docs. I had plugin errors in the analyser so checked all my plugins were up to date (we don't have many and they hadn't changed).

The docs also said to check for S/W compatibility. We are running Oracle 12.1.0.2 (12c R1) and ojdbc8.jar is in \lib. Java 8 which should be fine.

To start afresh to log this issue on the Community, I have restarted JIRA and run the Log Analyser and have two errors immediately that I have been seeing over the last couple of dates after restart:-

ERROR NoClassDefFoundError, ClassNotFoundExcception or NoSuchMethodException due to Incompatible Plugin

Checking atlassian-jira.log and searching for ClassNotFoundException.  Just before it is the following:-

2019-06-19 14:38:21,867 JIRA-Bootstrap ERROR [c.a.jira.event.DefaultListenerManager] Could not configure listener: GreenHopper Listener className:com.pyxis.greenhopper.jira.customfields.GreenHopperCTFIndexer
com.atlassian.jira.event.ListenerException: Could not find class: com.pyxis.greenhopper.jira.customfields.GreenHopperCTFIndexer

and

WARN Installation Failed with 'Unable to enable web fragment' Error

 JIRA Welcome Plugin : com.atlassian.jira.welcome.jira-welcome-plugin
Version : 1.1.43-3
Status : disabled
Unloadable Reason : There was a problem loading the module descriptor: java.lang.Void.<br/>Unable to enable web fragment
Vendor : Atlassian
Description : Displays a welcome dialog for first-time usage of JIRA, as well as a whats new dialog for users just having upgraded to this version of JIRA.

...
2019-06-19 14:38:23,602 JIRA-Bootstrap WARN [c.a.jira.startup.JiraStartupLogger]

***************************************************************************************************************************************************************
The following plugins failed to load:
JIRA Welcome Plugin: There was a problem loading the module descriptor: java.lang.Void.<br/>Unable to enable web fragment
***************************************************************************************************************************************************************

...

Having seen the errors before, I had uninstalled the JIRA Welcome Plugin as I have no idea how old it is, but there wasn't anything in the Marketplace for an update.

Before the restart above, it said the Atlassian Troubleshooting and Software Tools plugin needed updating, so I did that. It now gives an Instance Health Check incompatibility with the database indicating the 12.2.0.1 driver (which is ojdbc8 as far as I can see)...
Checking the atlassian-jira.log for that I get the following:-

Failed to find version number in Oracle version string 'Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options'

If I leave JIRA, I think it will fall over again tomorrow.  Originally it was due to memory (complained about the Mail Queue which might have been a red herring) but if it isn't happy about compatibility that doesn't seem good either.  Regarding the e-mails, we are not big users.  A couple of us use Automation which sends out a few e-mails and when issues are re-assigned or created that triggers e-mails, but it's not volume.

Any help appreciated with the log and the Instance Health (which has only been flagged up since I updated the Plugin).  I update Plugins as required, which may have caused an issue, but they are standard free ones which I can list if you think it could be one of those.

[Sue Bridges]

OK, still having problems - JIRA ran out of Java Help memory again.  My gut feel is that I suspect that this is to do with all of the all of the Instance Health issues that are being thrown.  Even though I had updated the ATST plugin, it's only today that it requested another updated, and version 1.17.2 I notice contains a fix for the Oracle DB driver checking.  My instance health check now appears OK.  However, still get the initial error about on the log analyzer, but will check again on Monday after it's had a reboot after cold backups.

Hope this helps someone else re the instance health check.

[Sue Bridges]

I have reindexed the DB about three times this week in case of an index error due to the CTFIndex error seen above.

So nothing much has happened in the DB since the above I think as we are not busy in it.  Just updated and reassigned an Issue though and noticed the following in atlassian-jira.log:-

2019-06-19 15:20:27,922 JIRA-INFORM-Thread-0 WARN anonymous 920x752x1 1j0p7r5 192.100.100.132 /secure/AssignIssue.jspa [o.a.commons.dbcp2.BasicDataSource] An internal object pool swallowed an Exception.
java.sql.SQLException: Protocol violation: [ 1, ]

...

2019-06-19 15:20:27,922 JIRA-INFORM-Thread-0 ERROR anonymous 920x752x1 1j0p7r5 192.100.100.132 /secure/AssignIssue.jspa [c.a.jira.transaction.TransactionSupportImpl] Unable to rollback transaction : Error occurred while rolling back transaction. ( )
org.ofbiz.core.entity.GenericTransactionException: Error occurred while rolling back transaction. ( )

...

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException:
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:450)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:392)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:385)
at oracle.jdbc.driver.T4C7Ocommoncall.processError(T4C7Ocommoncall.java:91)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:655)
at oracle.jdbc.driver.T4C7Ocommoncall.doOROLLBACK(T4C7Ocommoncall.java:66)
at oracle.jdbc.driver.T4CConnection.doRollback(T4CConnection.java:906)
at oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:4539)
at org.apache.commons.dbcp2.DelegatingConnection.rollback(DelegatingConnection.java:488)
at org.apache.commons.dbcp2.DelegatingConnection.rollback(DelegatingConnection.java:488)
at org.ofbiz.core.entity.jdbc.interceptors.connection.DelegatingConnection.rollback(DelegatingConnection.java:61)
at org.ofbiz.core.entity.TransactionUtil.rollbackLocalTransaction(TransactionUtil.java:383)
... 53 more
2019-06-19 15:20:27,922 JIRA-INFORM-Thread-0 ERROR anonymous 920x752x1 1j0p7r5 192.100.100.132 /secure/AssignIssue.jspa [c.a.j.p.i.events.managers.EventManagerImpl] Exception occured
java.lang.ArrayIndexOutOfBoundsException: 10
at oracle.jdbc.driver.Accessor.setOffset(Accessor.java:651)
at

...

2019-06-19 15:20:27,922 JIRA-INFORM-Thread-0 ERROR anonymous 920x752x1 1j0p7r5 192.100.100.132 /secure/AssignIssue.jspa [c.a.j.p.i.events.managers.EventManagerImpl] Exception occured
java.lang.ArrayIndexOutOfBoundsException: 10
at oracle.jdbc.driver.Accessor.setOffset(Accessor.java:651)
at oracle.jdbc.driver.T4CNumberAccessor.unmarshalBytes(T4CNumberAccessor.java:177)
at oracle.jdbc.driver.T4CNumberAccessor.unmarshalOneRow(T4CNumberAccessor.java:161)

...

2019-06-19 15:20:53,880 JiraIndexCommitThread-18 INFO [c.a.jira.index.MonitoringIndexWriter] [lucene-stats] flush stats: snapshotCount=3, totalCount=3, periodSec=2527, flushIntervalMillis=842533, indexDirectory=D:\apps\Atlassian\Application Data\JIRA\caches\indexesV1\changes, indexWriterId=com.atlassian.jira.index.MonitoringIndexWriter@50477d45, indexDirectoryId=MMapDirectory@D:\apps\Atlassian\Application Data\JIRA\caches\indexesV1\changes lockFactory=org.apache.lucene.store.NativeFSLockFactory@5573dae9
2019-06-19 15:20:53,880 JiraIndexCommitThread-17 INFO [c.a.jira.index.MonitoringIndexWriter] [lucene-stats] flush stats: snapshotCount=3, totalCount=3, periodSec=2550, flushIntervalMillis=850062, indexDirectory=D:\apps\Atlassian\Application Data\JIRA\caches\indexesV1\comments, indexWriterId=com.atlassian.jira.index.MonitoringIndexWriter@761ce80f, indexDirectoryId=MMapDirectory@D:\apps\Atlassian\Application Data\JIRA\caches\indexesV1\comments lockFactory=org.apache.lucene.store.NativeFSLockFactory@5573dae9
2019-06-19 15:20:53,958 JiraIndexCommitThread-9 INFO [c.a.jira.index.MonitoringIndexWriter] [lucene-stats] flush stats: snapshotCount=3, totalCount=3, periodSec=2527, flushIntervalMillis=842637, indexDirectory=D:\apps\Atlassian\Application Data\JIRA\caches\indexesV1\issues, indexWriterId=com.atlassian.jira.index.MonitoringIndexWriter@6b39989d, indexDirectoryId=MMapDirectory@D:\apps\Atlassian\Application Data\JIRA\caches\indexesV1\issues lockFactory=org.apache.lucene.store.NativeFSLockFactory@5573dae9
2019-06-19 15:23:36,337 indexing-stats-0 INFO [c.a.j.issue.index.IndexingStatsManager] [indexing-stats] Top 10 addIndex total (noFieldIndexers: 69, sum: 209ms): [{field: customfield_10166 (Rank), addIndex: {sum/allSum:82.7%, sum:173ms, avg:86.7ms, max:172ms, count:2}}, {field: customfield_10164 (Sprint), addIndex: {sum/allSum:2.2%, sum:4ms, avg:2.3ms, max:3ms, count:2}}, {field: customfield_10165 (Epic Link), addIndex: {sum/allSum:2.0%, sum:4ms, avg:2.1ms, max:3ms, count:2}}, {field: customfield_10660 (Closed Date), addIndex: {sum/allSum:1.0%, sum:2ms, avg:1.0ms, max:2ms, count:2}}, {field: watcher, addIndex: {sum/allSum:1.0%, sum:2ms, avg:1.0ms, max:1ms, count:2}}, {field: environment, addIndex: {sum/allSum:0.7%, sum:1ms, avg:0.7ms, max:1ms, count:2}}, {field: timespent, addIndex: {sum/allSum:0.7%, sum:1ms, avg:0.7ms, max:1ms, count:2}}, {field: customfield_10040 (Module), addIndex: {sum/allSum:0.4%, sum:0ms, avg:0.4ms, max:0ms, count:2}}, {field: attachment, addIndex: {sum/allSum:0.4%, sum:0ms, avg:0.4ms, max:0ms, count:2}}, {field: customfield_10022 (User Manual), addIndex: {sum/allSum:0.3%, sum:0ms, avg:0.4ms, max:0ms, count:2}}]
2019-06-19 15:23:36,337 indexing-stats-0 INFO [c.a.j.issue.index.IndexingStatsManager] [indexing-stats] Top 10 addIndex snapshot (noFieldIndexers: 69, sum: 209ms): [{field: customfield_10166 (Rank), addIndex: {sum/allSum:82.7%, sum:173ms, avg:86.7ms, max:172ms, count:2}}, {field: customfield_10164 (Sprint), addIndex: {sum/allSum:2.2%, sum:4ms, avg:2.3ms, max:3ms, count:2}}, {field: customfield_10165 (Epic Link), addIndex: {sum/allSum:2.0%, sum:4ms, avg:2.1ms, max:3ms, count:2}}, {field: customfield_10660 (Closed Date), addIndex: {sum/allSum:1.0%, sum:2ms, avg:1.0ms, max:2ms, count:2}}, {field: watcher, addIndex: {sum/allSum:1.0%, sum:2ms, avg:1.0ms, max:1ms, count:2}}, {field: environment, addIndex: {sum/allSum:0.7%, sum:1ms, avg:0.7ms, max:1ms, count:2}}, {field: timespent, addIndex: {sum/allSum:0.7%, sum:1ms, avg:0.7ms, max:1ms, count:2}}, {field: customfield_10040 (Module), addIndex: {sum/allSum:0.4%, sum:0ms, avg:0.4ms, max:0ms, count:2}}, {field: attachment, addIndex: {sum/allSum:0.4%, sum:0ms, avg:0.4ms, max:0ms, count:2}}, {field: customfield_10022 (User Manual), addIndex: {sum/allSum:0.3%, sum:0ms, avg:0.4ms, max:0ms, count:2}}]