FishEye/Crucible keeps crashing

[FPA Support]

Our FishEye/Crucible instance crashes about once a day and will not recover even with the windows service saying to restart on crash. Manually restarting the service will bring it back up and it will work fine for about another day before it crashes. I cannot figure out what it is having issues with. We have JIRA/Confluence and BitBucket Server all running on the same machine with no issues. It is a Windows 2008 R2 server 64-bit w/ 128GB of RAM. I have configured it with 10gb initial and 20gb max incase it was having problems with memory limitations. When it is running fine it sits around 7.2gb of ram.

 This is what I see in the thread dump section. The support tools show no errors.

 [336] qtp2144838275-336: TIMED_WAITING (Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@71250406 ) [335] qtp2144838275-335: TIMED_WAITING (Waiting on org.eclipse.jetty.io.nio.SelectChannelEndPoint@36af9d0f ) [334] qtp2144838275-334: TIMED_WAITING (Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@71250406 ) [333] qtp2144838275-333: TIMED_WAITING (Waiting on org.eclipse.jetty.io.nio.SelectChannelEndPoint@66a03324 ) [332] qtp2144838275-332: TIMED_WAITING (Waiting on org.eclipse.jetty.io.nio.SelectChannelEndPoint@5adb7cbe ) [331] qtp2144838275-331: RUNNABLE [323] qtp2144838275-323: TIMED_WAITING (Waiting on org.eclipse.jetty.io.nio.SelectChannelEndPoint@78055866 ) [321] qtp2144838275-321: TIMED_WAITING (Waiting on org.eclipse.jetty.io.nio.SelectChannelEndPoint@7c91e192 ) [316] qtp2144838275-316: TIMED_WAITING (Waiting on org.eclipse.jetty.io.nio.SelectChannelEndPoint@e584dfe ) [308] qtp2144838275-308: TIMED_WAITING (Waiting on org.eclipse.jetty.io.nio.SelectChannelEndPoint@4ef8ce7f ) [302] ThreadPool1: WAITING (Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3898ef6e ) [300] AtlassianEvent::pool-3-thread-12: WAITING (Waiting on java.util.concurrent.SynchronousQueue$TransferStack@6561b6fe ) [299] AtlassianEvent::pool-3-thread-11: WAITING (Waiting on java.util.concurrent.SynchronousQueue$TransferStack@6561b6fe ) [298] AtlassianEvent::pool-3-thread-10: WAITING (Waiting on java.util.concurrent.SynchronousQueue$TransferStack@6561b6fe ) [297] AtlassianEvent::pool-3-thread-9: WAITING (Waiting on java.util.concurrent.SynchronousQueue$TransferStack@6561b6fe ) [296] AtlassianEvent::pool-3-thread-8: WAITING (Waiting on java.util.concurrent.SynchronousQueue$TransferStack@6561b6fe ) [295] AtlassianEvent::pool-3-thread-7: WAITING (Waiting on java.util.concurrent.SynchronousQueue$TransferStack@6561b6fe ) [294] AtlassianEvent::pool-3-thread-6: WAITING (Waiting on java.util.concurrent.SynchronousQueue$TransferStack@6561b6fe ) [293] AtlassianEvent::pool-3-thread-5: WAITING (Waiting on java.util.concurrent.SynchronousQueue$TransferStack@6561b6fe ) [292] AtlassianEvent::pool-3-thread-4: WAITING (Waiting on java.util.concurrent.SynchronousQueue$TransferStack@6561b6fe ) [291] AtlassianEvent::pool-3-thread-3: WAITING (Waiting on java.util.concurrent.SynchronousQueue$TransferStack@6561b6fe ) [290] AtlassianEvent::pool-3-thread-2: WAITING (Waiting on java.util.concurrent.SynchronousQueue$TransferStack@6561b6fe ) [289] AtlassianEvent::pool-3-thread-1: WAITING (Waiting on java.util.concurrent.SynchronousQueue$TransferStack@6561b6fe ) [284] Failure Cache Plugin Executor:thread-2: WAITING (Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@d1a191a ) [283] Failure Cache Plugin Executor:thread-1: WAITING (Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@d1a191a ) [274] UpmAsynchronousTaskManager:thread-1: WAITING (Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59d8bb2f ) [260] Navlink Plugin Executor:thread-12: WAITING (Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2261b12d )

Thank you.

[Rodrigo M]

Hello Chet

We can't know for sure the root cause of thje issue just with the information provided.

You can deeper investigate the issue from your side by following the itens below

• Did you enable DEBUG logs? It will write more information on the logs to look for errors;
• You may want to check for any errors related to this in the O.S logs;
• You can also generate external thread dumps while reproducing the issue in order to further investigate.
• Please review the Best practices for FishEye configuration, especially item 10.
• Be Sure to meet all the requirements to run Fisheye on the version you are running by following Fisheye Supported Platforms.

Regards
Rodrigo