Existing Crucible instance not starting properly

[James O_Connell]

fisheye.out log file indicates Crucible starts up ok, but then nothing seems to work.  Until this service went down, this was an operational Production instance of Crucible.  Now we have no idea why it won't come back up.

2019-03-04 11:59:53,854 INFO - Using log4j configuration file: /opt/applocal/crucible/fecru-4.5.1/log4j.xml
2019-03-04 11:59:55,514 INFO - FishEye arguments: []
2019-03-04 11:59:55,831 INFO - FishEye/Crucible 4.5.1 (20171127103437), Built on 2017-11-27
2019-03-04 11:59:55,831 INFO - Logging started (Oracle Corporation|25.73-b02|/opt/applocal/vendor/java/jdk1.8.0_73/jre|Linux|amd64|3.10.0-693.21.1.el7.x86_64|maxmem=910MB)
2019-03-04 11:59:56,391 INFO - This program contains InfinityDB Version 3.0.21 2013.08.04_07.20 Deployment edition
2019-03-04 11:59:56,391 INFO - See infinitydb.com - Boiler Bay Inc
2019-03-04 11:59:56,391 INFO - InfinityDB is Copyight (C) 2001-2013 Roger L. Deran, all rights reserved.
2019-03-04 11:59:56,392 INFO - InfinityDB has been Licensed to Atlassian
2019-03-04 11:59:56,392 INFO - for distribution in FishEye and Crucible.
2019-03-04 11:59:56,392 INFO - Atlassian Pty Ltd (ABN 53 102 443 916)
2019-03-04 11:59:56,392 INFO - of 341 George St., Sydney, New South Wales, 2000 Australia
2019-03-04 12:00:01,794 INFO - UpgradeManager started.
2019-03-04 12:00:02,782 INFO - loading config from file: /opt/applocal/crucible/fisheye_home/config.xml
2019-03-04 17:00:02 UTC [85dec8b6] info [native] License = license ok; no uem volume available;
2019-03-04 12:00:17,355 INFO - Crucible (Server) Unlimited Users: Developer License License registered to Fidelity Investments.
2019-03-04 12:00:30,646 INFO - Adding secondary content dir of /opt/applocal/crucible/fisheye_home/content
2019-03-04 12:02:24,173 INFO - Starting database...
2019-03-04 12:02:34,734 INFO - BoneCP - tracking statements enabled for pool [mainPool]
2019-03-04 12:03:17,560 INFO - BoneCP - tracking statements enabled for pool [retriablePool]
2019-03-04 12:03:18,572 INFO - Database started.
2019-03-04 12:03:34,849 INFO - Using synchronous global permission cache reload policy
2019-03-04 12:03:36,499 INFO - Starting plugin system...
2019-03-04 12:04:11,849 INFO - Starting mail system...
2019-03-04 12:04:12,099 INFO - Mail system started.
2019-03-04 12:04:37,758 INFO - Active spring profiles: []
log4j:WARN No appenders could be found for logger (com.amazonaws.jmx.spi.SdkMBeanRegistry).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
2019-03-04 12:08:34,185 INFO - Plugin system started.
2019-03-04 12:09:12,774 INFO - Server started on :8060 (http) (control port on 127.0.0.1:8059)
2019-03-04 12:10:58,237 WARN - No indexed parents found for commit 80d387f7c63b59f71cda677931d3a36c02af9c95, parent 4c81cad6d779593812ad7f93201e1eb22944f4c4
2019-03-04 12:10:58,269 WARN - Diff Information is missing parent changeset id for pm-svc-macs-targets-db/pom.xml. Could not resolve parent changeset 4c81cad6d779593812ad7f93201e1eb22944f4c4
2019-03-04 12:10:59,712 WARN - Diff Information is missing parent changeset id for pm-svc-macs-targets-db/src/main/sql/install/AM_TARGETS/Types/TARGET_OBJ.sql. Could not resolve parent changeset 4c81cad6d779593812ad7f93201e1eb22944f4c4
2019-03-04 12:10:59,794 ERROR - Exception "javax.el.ELException: Problems calling function 'cru:renderHeader'" (javax.servlet.ServletException) while processing "/404.jsp" (Referer:"null")
javax.servlet.ServletException: javax.el.ELException: Problems calling function 'cru:renderHeader'
at org.apache.jasper.runtime.PageContextImpl.doHandlePageException(PageContextImpl.java:944) [org.apache.jasper.glassfish-2.2.2.v201112011158.jar:?]
at org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:879) [org.apache.jasper.glassfish-2.2.2.v201112011158.jar:?]
at org.apache.jsp._404_jsp._jspService(_404_jsp.java:113) [classes/:?]
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111) [org.apache.jasper.glassfish-2.2.2.v201112011158.jar:?]

[Ben Stuart · Accepted solution]

Hello, 

I would recommend making sure you version of Java is supported and also the following steps: 

1. Stop Fisheye/Crucible
2. Delete FISHEYE_INST/var/osgi-cache
3. Start Fisheye/Crucible

If that doesn't help, can you please post the full stack trace of the error message? 

2019-03-04 12:10:59,794 ERROR - Exception "javax.el.ELException: Problems calling function 'cru:renderHeader'" (javax.servlet.ServletException) while processing "/404.jsp" (Referer:"null")

There may be a caused by similar to:

Caused by: com.atlassian.soy.renderer.SoyException: Unable to compile Soy templates at: com.atlassian.plugins.atlassian-nav-links-plugin:rotp-menu

It could be one of the core plugins,  in this case (Atlassian Navigation Link Plugin) was somehow not enabled. It could be either corrupted or missing. The easiest way to test this theory is to get a fresh copy of FishEye's instance and try starting with that. 

I hope this helps!