standalone Jira on debian stops unexpectedly

cardpros

Hi, we have a standalone JIRA version on a debian server. after few days, we find JIRA server stopped and we have to restart it manually.

in the logs we see the last messages before stopping are these:

ties to export in memory.
2017-12-27 12:00:00,082 Caesium-1-4 INFO anonymous Backup Service [c.a.j.bc.dataimport.DefaultExportService] Attempting to save the Active Objects Backup
2017-12-27 12:00:00,512 Caesium-1-4 INFO anonymous Backup Service [c.a.j.bc.dataimport.DefaultExportService] Finished saving the Active Objects Backup
2017-12-27 12:39:12,383 Caesium-1-4 INFO ServiceRunner [c.a.j.p.h.service.ping.RefreshConnectionStatusJobHandler] Running RefreshConnectionStatusJobHandler...
2017-12-27 12:39:14,824 Caesium-1-2 INFO ServiceRunner [c.a.j.p.h.service.connect.InstallGlancesJobHandler] Running InstallGlancesJobHandler...
2017-12-27 12:39:14,824 Caesium-1-2 INFO ServiceRunner [c.a.j.p.h.service.connect.InstallGlancesJobHandler] There is no link to HipChat, no need to install glances.
2017-12-27 13:39:19,016 Caesium-1-3 INFO ServiceRunner [c.a.j.p.h.service.ping.RefreshConnectionStatusJobHandler] Running RefreshConnectionStatusJobHandler...
2017-12-27 13:39:33,588 Caesium-1-1 INFO ServiceRunner [c.a.j.p.h.service.connect.InstallGlancesJobHandler] Running InstallGlancesJobHandler...
2017-12-27 13:39:34,087 Caesium-1-1 INFO ServiceRunner [c.a.j.p.h.service.connect.InstallGlancesJobHandler] There is no link to HipChat, no need to install glances.
2017-12-27 14:46:34,611 Caesium-1-1 INFO ServiceRunner [c.a.j.p.h.service.ping.RefreshConnectionStatusJobHandler] Running RefreshConnectionStatusJobHandler...
2017-12-27 14:46:38,594 Caesium-1-3 INFO ServiceRunner [c.a.j.p.h.service.connect.InstallGlancesJobHandler] Running InstallGlancesJobHandler...
2017-12-27 14:46:40,243 Caesium-1-3 INFO ServiceRunner [c.a.j.p.h.service.connect.InstallGlancesJobHandler] There is no link to HipChat, no need to install glances.
2017-12-27 15:04:22,994 HealthCheck:thread-1 ERROR ServiceRunner [c.a.t.j.healthcheck.support.BaseUrlHealthCheck] An error occurred when performing the Base URL healthcheck:
org.apache.http.conn.ConnectTimeoutException: Connect to 192.168.1.100:8777 [/192.168.1.100] timed out
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at com.atlassian.troubleshooting.jira.healthcheck.support.BaseUrlHealthCheck.doCheck(BaseUrlHealthCheck.java:58)
at com.atlassian.troubleshooting.jira.healthcheck.support.AbstractSupportHealthCheck.check(AbstractSupportHealthCheck.java:23)
at com.atlassian.troubleshooting.healthcheck.impl.PluginSuppliedSupportHealthCheck.check(PluginSuppliedSupportHealthCheck.java:49)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException
at java.net.SocksSocketImpl.remainingMillis(SocksSocketImpl.java:111)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
2017-12-27 15:19:15,630 HealthCheckWatchdog:thread-5 WARN ServiceRunner [c.a.t.healthcheck.concurrent.SupportHealthCheckTask] Health check Lucene index files location was unable to complete within the timeout of 20000.
2017-12-27 15:19:15,631 HealthCheckWatchdog:thread-8 WARN ServiceRunner [c.a.t.healthcheck.concurrent.SupportHealthCheckTask] Health check End of Life was unable to complete within the timeout of 20000.
2017-12-27 15:19:15,635 HealthCheckWatchdog:thread-1 WARN ServiceRunner [c.a.t.healthcheck.concurrent.SupportHealthCheckTask] Health check Lucene was unable to complete within the timeout of 20000.
2017-12-27 15:19:15,636 HealthCheckWatchdog:thread-2 WARN ServiceRunner [c.a.t.healthcheck.concurrent.SupportHealthCheckTask] Health check Base URL for gadgets was unable to complete within the timeout of 20000.
2017-12-27 15:46:27,096 Caesium-1-4 INFO ServiceRunner [c.a.j.p.h.service.ping.RefreshConnectionStatusJobHandler] Running RefreshConnectionStatusJobHandler...
2017-12-27 15:46:36,226 Caesium-1-3 INFO ServiceRunner [c.a.j.p.h.service.connect.InstallGlancesJobHandler] Running InstallGlancesJobHandler...
2017-12-27 15:46:36,226 Caesium-1-3 INFO ServiceRunner [c.a.j.p.h.service.connect.InstallGlancesJobHandler] There is no link to HipChat, no need to install glances.
2017-12-27 16:46:27,096 Caesium-1-3 INFO ServiceRunner [c.a.j.p.h.service.ping.RefreshConnectionStatusJobHandler] Running RefreshConnectionStatusJobHandler...
2017-12-27 16:46:36,227 Caesium-1-3 INFO ServiceRunner [c.a.j.p.h.service.connect.InstallGlancesJobHandler] Running InstallGlancesJobHandler...
2017-12-27 16:46:36,227 Caesium-1-3 INFO ServiceRunner [c.a.j.p.h.service.connect.InstallGlancesJobHandler] There is no link to HipChat, no need to install glances.
2017-12-27 17:46:27,096 Caesium-1-3 INFO ServiceRunner [c.a.j.p.h.service.ping.RefreshConnectionStatusJobHandler] Running RefreshConnectionStatusJobHandler...
2017-12-27 17:46:36,226 Caesium-1-2 INFO ServiceRunner [c.a.j.p.h.service.connect.InstallGlancesJobHandler] Running InstallGlancesJobHandler...
2017-12-27 17:46:36,227 Caesium-1-2 INFO ServiceRunner [c.a.j.p.h.service.connect.InstallGlancesJobHandler] There is no link to HipChat, no need to install glances.
2017-12-27 18:53:12,635 Caesium-1-4 INFO ServiceRunner [c.a.j.p.h.service.ping.RefreshConnectionStatusJobHandler] Running RefreshConnectionStatusJobHandler...
2017-12-27 18:53:12,657 Caesium-1-3 INFO ServiceRunner [c.a.j.p.h.service.connect.InstallGlancesJobHandler] Running InstallGlancesJobHandler...
2017-12-27 18:53:12,720 Caesium-1-3 INFO ServiceRunner [c.a.j.p.h.service.connect.InstallGlancesJobHandler] There is no link to HipChat, no need to install glances.
27-Dec-2017 18:57:05.021 INFO [Thread-4] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8777"]
l glances.
27-Dec-2017 18:57:05.021 INFO [Thread-4] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8777"]
27-Dec-2017 18:57:05.092 INFO [Thread-4] org.apache.catalina.core.StandardService.stopInternal Stopping service Catalina
2017-12-27 18:57:05,157 localhost-startStop-2 INFO [c.a.jira.startup.DefaultJiraLauncher] Stopping launchers
2017-12-27 18:57:06,036 localhost-startStop-2 INFO [c.a.plugin.manager.DefaultPluginManager] Preparing to shut down the plugin system
2017-12-27 18:57:06,060 localhost-startStop-2 INFO [c.a.plugin.manager.DefaultPluginManager] Shutting down the plugin system
2017-12-27 18:57:06,679 Gemini Blueprint context shutdown thread2 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.pyxis.greenhopper.jira]
2017-12-27 18:57:07,222 Gemini Blueprint context shutdown thread2 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.troubleshooting.plugin-jira]
2017-12-27 18:57:07,289 Gemini Blueprint context shutdown thread1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.streams.thirdparty-plugin]
2017-12-27 18:57:07,571 Gemini Blueprint context shutdown thread1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.plugins.authentication.atlassian-authentication-plugin]
2017-12-27 18:57:08,287 Gemini Blueprint context shutdown thread2 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.labs.hipchat.hipchat-for-jira-plugin]
2017-12-27 18:57:08,709 Gemini Blueprint context shutdown thread1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.plugins.base-hipchat-integration-plugin-api]
2017-12-27 18:57:08,807 Gemini Blueprint context shutdown thread2 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.jira.plugins.webhooks.jira-webhooks-plugin]
2017-12-27 18:57:09,767 Gemini Blueprint context shutdown thread1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.whisper.atlassian-whisper-plugin]
2017-12-27 18:57:10,109 Gemini Blueprint context shutdown thread2 WARN [o.s.context.annotation.CommonAnnotationBeanPostProcessor] Invocation of destroy method failed on bean with name 'devtoolsWorkflowEventBridge': org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(&(objectClass=com.atlassian.event.remote.RemoteEventListenerRegistrar)(objectClass=com.atlassian.event.remote.RemoteEventListenerRegistrar))] unavailable
2017-12-27 18:57:10,895 Gemini Blueprint context shutdown thread2 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.jira.jira-invite-user-plugin]
2017-12-27 18:57:11,152 Gemini Blueprint context shutdown thread1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.plugins.atlassian-whitelist-api-plugin]
2017-12-27 18:57:13,379 Gemini Blueprint context shutdown thread1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.jira.plugins.jira-dvcs-connector-plugin]
2017-12-27 18:57:13,379 Gemini Blueprint context shutdown thread1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] destroy
2017-12-27 18:57:15,162 Gemini Blueprint context shutdown thread2 INFO [c.a.j.plugins.monitor.MonitoringScheduler] Unscheduling metrics collector...
2017-12-27 18:57:15,163 Gemini Blueprint context shutdown thread2 INFO [c.a.j.plugins.monitor.MonitorLauncher] Stopped JIRA monitoring
2017-12-27 18:57:15,255 FelixStartLevel WARN [c.a.p.osgi.factory.OsgiBundlePlugin] Cannot disable Bundle 'rome.rome-1.0', not ACTIVE
2017-12-27 18:57:15,256 FelixStartLevel WARN [c.a.p.osgi.factory.OsgiBundlePlugin] Cannot disable Bundle 'org.osgi.service.cm-1.5.0.201505202024', not ACTIVE
2017-12-27 18:57:15,256 FelixStartLevel WARN [c.a.p.osgi.factory.OsgiBundlePlugin] Cannot disable Bundle 'org.apache.httpcomponents.httpcore-4.4.1', not ACTIVE
2017-12-27 18:57:15,260 FelixStartLevel WARN [c.a.p.osgi.factory.OsgiBundlePlugin] Cannot disable Bundle 'org.apache.httpcomponents.httpclient-4.5.3', no

2017-12-27 18:57:15,260 FelixStartLevel WARN [c.a.p.osgi.factory.OsgiBundlePlugin] Cannot disable Bundle 'org.apache.httpcomponents.httpclient-4.5.3', not ACTIVE
2017-12-27 18:57:15,260 FelixStartLevel WARN [c.a.p.osgi.factory.OsgiBundlePlugin] Cannot disable Bundle 'io.atlassian.fugue.scala-4.5.0', not ACTIVE
2017-12-27 18:57:15,260 FelixStartLevel WARN [c.a.p.osgi.factory.OsgiBundlePlugin] Cannot disable Bundle 'io.atlassian.fugue.retry-4.5.0', not ACTIVE
2017-12-27 18:57:15,260 FelixStartLevel WARN [c.a.p.osgi.factory.OsgiBundlePlugin] Cannot disable Bundle 'io.atlassian.fugue.optics-4.5.0', not ACTIVE
2017-12-27 18:57:15,260 FelixStartLevel WARN [c.a.p.osgi.factory.OsgiBundlePlugin] Cannot disable Bundle 'io.atlassian.fugue.guava-4.5.0', not ACTIVE
2017-12-27 18:57:15,260 FelixStartLevel WARN [c.a.p.osgi.factory.OsgiBundlePlugin] Cannot disable Bundle 'io.atlassian.fugue-4.5.0', not ACTIVE
2017-12-27 18:57:15,261 FelixStartLevel WARN [c.a.p.osgi.factory.OsgiBundlePlugin] Cannot disable Bundle 'com.springsource.org.jdom-1.0.0', not ACTIVE
2017-12-27 18:57:15,261 FelixStartLevel WARN [c.a.p.osgi.factory.OsgiBundlePlugin] Cannot disable Bundle 'com.springsource.com.sun.syndication-0.9.0', not ACTIVE
2017-12-27 18:57:15,612 FelixStartLevel WARN [c.a.p.osgi.factory.OsgiBundlePlugin] Cannot disable Bundle 'com.atlassian.plugin.atlassian-spring-scanner-runtime', not ACTIVE
2017-12-27 18:57:15,612 FelixStartLevel WARN [c.a.p.osgi.factory.OsgiBundlePlugin] Cannot disable Bundle 'com.atlassian.plugin.atlassian-spring-scanner-annotation', not ACTIVE
2017-12-27 18:57:16,142 FelixStartLevel WARN [c.a.p.osgi.factory.OsgiBundlePlugin] Cannot disable Bundle 'com.atlassian.bundles.nekohtml-1.9.12.1', not ACTIVE
2017-12-27 18:57:16,143 FelixStartLevel WARN [c.a.p.osgi.factory.OsgiBundlePlugin] Cannot disable Bundle 'com.atlassian.bundles.json-20070829.0.0.1', not ACTIVE
2017-12-27 18:57:16,143 FelixStartLevel WARN [c.a.p.osgi.factory.OsgiBundlePlugin] Cannot disable Bundle 'com.atlassian.bundles.icu4j-3.8.0.1', not ACTIVE
2017-12-27 18:57:16,226 localhost-startStop-2 WARN [c.a.p.osgi.factory.OsgiBundlePlugin] OSGi container not running or undefined: Will not remove bundle listener and will not close package admin service
2017-12-27 18:57:16,227 localhost-startStop-2 WARN [c.a.p.osgi.factory.OsgiBundlePlugin] OSGi container not running or undefined: Will not remove bundle listener and will not close package admin service
2017-12-27 18:57:16,234 localhosl glances.
27-Dec-2017 18:57:05.021 INFO [Thread-4] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8777"]
27-Dec-2017 18:57:05.092 INFO [Thread-4] org.apache.catalina.core.StandardService.stopInternal Stopping service Catalina
2017-12-27 18:57:05,157 localhost-startStop-2 INFO [c.a.jira.startup.DefaultJiraLauncher] Stopping launchers
2017-12-27 18:57:06,036 localhost-startStop-2 INFO [c.a.plugin.manager.DefaultPluginManager] Preparing to shut down the plugin system
2017-12-27 18:57:06,060 localhost-startStop-2 INFO [c.a.plugin.manager.DefaultPluginManager] Shutting down the plugin system
2017-12-27 18:57:06,679 Gemini Blueprint context shutdown thread2 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.pyxis.greenhopper.jira]
2017-12-27 18:57:07,222 Gemini Blueprint context shutdown thread2 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.troubleshooting.plugin-jira]
2017-12-27 18:57:07,289 Gemini Blueprint context shutdown thread1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.streams.thirdparty-plugin]
2017-12-27 18:57:07,571 Gemini Blueprint context shutdown thread1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.plugins.authentication.atlassian-authentication-plugin]
2017-12-27 18:57:08,287 Gemini Blueprint context shutdown thread2 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.labs.hipchat.hipchat-for-jira-plugin]
2017-12-27 18:57:08,709 Gemini Blueprint context shutdown thread1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.plugins.base-hipchat-integration-plugin-api]
2017-12-27 18:57:08,807 Gemini Blueprint context shutdown thread2 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.jira.plugins.webhooks.jira-webhooks-plugin]
2017-12-27 18:57:09,767 Gemini Blueprint context shutdown thread1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.whisper.atlassian-whisper-plugin]
2017-12-27 18:57:10,109 Gemini Blueprint context shutdown thread2 WARN [o.s.context.annotation.CommonAnnotationBeanPostProcessor] Invocation of destroy method failed on bean with name 'devtoolsWorkflowEventBridge': org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(&(objectClass=com.atlassian.event.remote.RemoteEventListenerRegistrar)(objectClass=com.atlassian.event.remote.RemoteEventListenerRegistrar))] unavailable
2017-12-27 18:57:10,895 Gemini Blueprint context shutdown thread2 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.jira.jira-invite-user-plugin]
2017-12-27 18:57:11,152 Gemini Blueprint context shutdown thread1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.plugins.atlassian-whitelist-api-plugin]
2017-12-27 18:57:13,379 Gemini Blueprint context shutdown thread1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] ungetService bundle [com.atlassian.jira.plugins.jira-dvcs-connector-plugin]
2017-12-27 18:57:13,379 Gemini Blueprint context shutdown thread1 DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] destroy
2017-12-27 18:57:15,162 Gemini Blueprint context shutdown thread2 INFO [c.a.j.plugins.monitor.MonitoringScheduler] Unscheduling metrics collector...
2017-12-27 18:57:15,163 Gemini Blueprint context shutdown thread2 INFO [c.a.j.plugins.monitor.MonitorLauncher] Stopped JIRA monitoring
2017-12-27 18:57:15,255 FelixStartLevel WARN [c.a.p.osgi.factory.OsgiBundlePlugin] Cannot disable Bundle 'rome.rome-1.0', not ACTIVE
2017-12-27 18:57:15,256 FelixStartLevel WARN [c.a.p.osgi.factory.OsgiBundlePlugin] Cannot disable Bundle 'org.osgi.service.cm-1.5.0.201505202024', not ACTIVE
2017-12-27 18:57:15,256 FelixStartLevel WARN [c.a.p.osgi.factory.OsgiBundlePlugin] Cannot disable Bundle 'org.apache.httpcomponents.httpcore-4.4.1', not ACTIVE
2017-12-27 18:57:15,260 FelixStartLevel WARN [c.a.p.osgi.factory.OsgiBundlePlugin] Cannot disable Bundle 'org.apache.httpcomponents.httpclient-4.5.3', not-startStop-2 WARN [c.a.p.osgi.factory.OsgiBundlePlugin] OSGi container not running or undefined: Will not remove bundle listener and will not close package admin service
2017-12-27 18:57:16,235 localhost-startStop-2 WARN [c.a.p.osgi.factory.OsgiBundlePlugin] OSGi container not running or undefined: Will not remove bundle listener and will not close package admin service

Andy Heinzer

I can see that the Jira process is getting a command telling it to shutdown.  This is much more different than the scenario we can see in some linux environments where the kernel can kill the java process that is running Jira. When that happens, Jira stops so quickly, it does not log that the plugin system is being shutdown.  If you were seeing that, then I would recommend the KB How to Resolve JIRA Process being Terminated Unexpectedly in Linux.

 

But since we have a different scenario here, there must be something else that is causing the process to stop.   Jira has two sets of logs you can look at:

1. $JIRAINSTALL/logs/catalina.out
2. $JIRAHome/log/atlassian-jira.log

while these two files tend to contain a lot of overlapping information, it might be worth checking into both of these to see if we can understand more about why this process stopped.

Is Jira running as a service here?  I'd be interested to see if the OS has any other logging around who starts/stop this process.   It could be another admin to the box could be starting/stopping this process, or it could be that there is something else happening to Jira to cause this.  But with the section of log I can see so far, it is not clear to me if this shutdown was caused by a process, a user, or some other unexpected problem for JIra.