Help with Recent FE/Crucible Upgrade from 3.1.5 to 3.4.5

[Patrick Flannery]

I upgraded our prod instance last week from 3.1.5 to 3.4.5. We use a mysql database and our SVN is served up via NFS share on linux. Each day FE/Crucible. This error is in the log just before the instance becomes un-stable and requires a bounce. Any help would be appreciated. -->Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.writev0(Native Method)
at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
at sun.nio.ch.IOUtil.write(IOUtil.java:122)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:369)
at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
at org.eclipse.jetty.io.nio.ChannelEndPoint.gatheringFlush(ChannelEndPoint.java:376)
at org.eclipse.jetty.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:335)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:336)
at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:838)
... 138 more

[Patrick Flannery · Accepted solution]

You can close this question.

[Patrick Flannery]

At this point. We believe the bounce of FE/Crucible is due to an automated script that does a wget to the FE/Crucible web page. We do not think FE/Crucible itself is the cause of the bounce. We added code to our automated script to catch the error on the wget and will pursue this to see why FE/Crucible gets bounced every few days. At this point, i'd like to close this issue. Thanks for all your help. We will upgrade to 3.5 at the earliest convenience. Please close the ticket and thanks for all your help.

[Patrick Flannery]

This question can be closed.

[Patrick Flannery]

Hi Nick. I wanted to follow-up as to where we are with our FE/Crucible 3.4.5 issue. I downloaded 3.5.2 to my test server and testing now. Not sure when i will upgrade our prod node. Our production node, before the upgrade was running fine with 4gb of memory. Now with 3.4.5 the instance was requiring a bounce every 2-3 days. I changed the memory from 4gb to 6gb and the insance has been up since yesterday - 8/12/14 at 9:30 am est. I would like to understand why we keep getting these errors on our SVN but it may be unrelated as to why the instance becomes unresponsive. Do you have any insight as to why we see these errors ??? -->

2014-08-13 07:49:35,226 ERROR - Problem processing revisions from repo merlins_idmtestclients due to class com.cenqua.fisheye.rep.RepositoryClientException - org.apache.subversion.javahl.ClientException: svn: E180001: Unable to open an ra_local session to URL
svn: E180001: Unable to open repository 'file:///nfsmounts/svnap01exdu/svnprod/repos/merlins/idmtestclients'
svn: E204899: Can't open file '/nfsmounts/svnap01exdu/svnprod/repos/merlins/db/current'
com.cenqua.fisheye.rep.RepositoryClientException: org.apache.subversion.javahl.ClientException: svn: E180001: Unable to open an ra_local session to URL
svn: E180001: Unable to open repository 'file:///nfsmounts/svnap01exdu/svnprod/repos/merlins/idmtestclients'
svn: E204899: Can't open file '/nfsmounts/svnap01exdu/svnprod/repos/merlins/db/current'
at com.cenqua.fisheye.svn.SvnThrottledClient.executeNoThrottle(SvnThrottledClient.java:176)
at com.cenqua.fisheye.svn.SvnThrottledClient.execute(SvnThrottledClient.java:145)
at com.cenqua.fisheye.svn.SvnThrottledClient.info(SvnThrottledClient.java:110)
at com.atlassian.fisheye.svn.Svn2RepositoryClient.getLatestRevision(Svn2RepositoryClient.java:97)
at com.atlassian.fisheye.svn.Svn2Scanner.doChangesetScan(Svn2Scanner.java:248)
at com.atlassian.fisheye.svn.Svn2Scanner.slurpRepository(Svn2Scanner.java:202)
at com.atlassian.fisheye.svn.Svn2Scanner.doSlurpTransaction(Svn2Scanner.java:179)
Caused by: org.apache.subversion.javahl.ClientException: svn: E180001: Unable to open an ra_local session to URL
svn: E180001: Unable to open repository 'file:///nfsmounts/svnap01exdu/svnprod/repos/merlins/idmtestclients'
svn: E204899: Can't open file '/nfsmounts/svnap01exdu/svnprod/repos/merlins/db/current'
at org.tmatesoft.svn.core.javahl17.SVNClientImpl.getClientException(SVNClientImpl.java:1492)
at org.tmatesoft.svn.core.javahl17.SVNClientImpl.info2(SVNClientImpl.java:1673)
at org.apache.subversion.javahl.SVNClient.info2(SVNClient.java:270)
at com.cenqua.fisheye.svn.SvnThrottledClient$1.call(SvnThrottledClient.java:114)
at com.cenqua.fisheye.svn.SvnThrottledClient$1.call(SvnThrottledClient.java:111)
at com.cenqua.fisheye.svn.SvnTask.access$101(SvnTask.java:14)
at com.cenqua.fisheye.svn.SvnTask$1.run(SvnTask.java:36)
at com.cenqua.fisheye.svn.SvnTask.run(SvnTask.java:31)
Caused by: org.tmatesoft.svn.core.SVNException: svn: E180001: Unable to open an ra_local session to URL
svn: E180001: Unable to open repository 'file:///nfsmounts/svnap01exdu/svnprod/repos/merlins/idmtestclients'
svn: E204899: Can't open file '/nfsmounts/svnap01exdu/svnprod/repos/merlins/db/current'
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
at org.tmatesoft.svn.core.internal.io.fs.FSRepository.openRepository(FSRepository.java:796)
at org.tmatesoft.svn.core.internal.io.fs.FSRepository.getLatestRevision(FSRepository.java:121)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgRepositoryAccess.getRevisionNumber(SvnNgRepositoryAccess.java:118)
at org.tmatesoft.svn.core.internal.wc2.SvnRepositoryAccess.getLocations(SvnRepositoryAccess.java:182)
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgRepositoryAccess.createRepositoryFor(SvnNgRepositoryAccess.java:45)
at org.tmatesoft.svn.core.internal.wc2.remote.SvnRemoteGetInfo.run(SvnRemoteGetInfo.java:46)
at org.tmatesoft.svn.core.internal.wc2.remote.SvnRemoteGetInfo.run(SvnRemoteGetInfo.java:31)
at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1258)
at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
at org.tmatesoft.svn.core.javahl17.SVNClientImpl.info2(SVNClientImpl.java:1671)

[Nick]

Sorry - that error looks completely unrelated to anything FishEye could be doing, especially if you only see it sporadically.

It is coming directly from SVN, and if you were using the svn command line to run the same command, it is very likely you would be seeing the same error.

[Nick]

Do you have any monitoring on your NFS mount?

What would be causing you the following ERROR ?

svn: E180001: Unable to open repository 'file:///nfsmounts/svnap01exdu/svnprod/repos/merlins/audit'
svn: E204899: Can't open file '/nfsmounts/svnap01exdu/svnprod/repos/merlins/db/current'

[Patrick Flannery]

Hi Nick. Thank for respsonse. I read in notes about client qutting the connection but seems to be many of these since we upgraded, but i truly didn't watch the logs until the instance started this behavior. I can send more log info. When we upgraded last week, i did not change any java settings. We have max mem in jvm set to 4gb and max perm size set to 2gb. The machine is a physical server and has 16gb of memory. It seems to happen each day or at least every other day since the upgrade. The way we determine if Fisheye is responsive is to run a monitor via a wget to the fisheye url - on same network vlan. we run the wget with a timeout of 20 and try 5 times. if the wget fails, it bounces the fisheye instance. This was always effective in the past. We see many of these in the logs. I can turn on debug because i'd like to know what is causing it as opposed to upgrading to 3.5 but i will consider a upgrade if we cant resolve.

We see these in the log just before the bounce is issued.

2014-08-05 08:47:33,239 ERROR - Exception "null" (org.eclipse.jetty.io.EofException) while processing "/avatar/weilandd" (Referer:"http://qualityap04qxdu.dev.oclc.org:15000/cru")
org.eclipse.jetty.io.EofException
at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:914)
at org.eclipse.jetty.http.AbstractGenerator.blockForOutput(AbstractGenerator.java:523)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:170)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)
at java.util.zip.GZIPOutputStream.finish(GZIPOutputStream.java:91)
at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:140)
at org.eclipse.jetty.http.gzip.AbstractCompressedStream.close(AbstractCompressedStream.java:153)
at org.eclipse.jetty.server.Dispatcher.commitResponse(Dispatcher.java:315)
at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:279)
at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:103)
at com.cenqua.fisheye.web.filters.TotalityFilter.doFilter(TotalityFilter.java:200)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502)
at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:100)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:69)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
at com.cenqua.fisheye.web.filters.CustomIncludableGzipFilter.doFilter(CustomIncludableGzipFilter.java:27)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502)
at com.cenqua.fisheye.web.filters.ProductInfoFilter.doFilter(ProductInfoFilter.java:44)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502)
at com.cenqua.fisheye.web.filters.UpfrontFilter.doFilter(UpfrontFilter.java:59)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502)
at com.atlassian.fecru.profiling.ProfilingServletFilter.doFilter(ProfilingServletFilter.java:62)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
at sun.nio.ch.IOUtil.write(IOUtil.java:40)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:336)
at org.eclipse.jetty.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:293)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:362)
at org.eclipse.jetty.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:341)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:336)
at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:841)
... 77 more
2014-08-05 09:01:58,522 INFO - Using log4j configuration file: /nfsmounts/devnas3s4/fe_data/fecru-3.4.5/log4j.xml
2014-08-05 09:01:58,590 INFO - FishEye arguments: []
2014-08-05 09:01:58,593 INFO - FishEye/Crucible 3.4.5 (20140701133253), Built on 2014-07-01
2014-08-05 09:01:58,593 INFO - Logging started (Sun Microsystems Inc.|20.8-b03|/usr/java/jdk1.6.0_33/jre|Linux|amd64|2.6.16.60-0.93.1-smp|maxmem=3640MB)
2014-08-05 09:01:58,635 INFO - This program contains InfinityDB Version 3.0.21 2013.08.04_07.20 Deployment edition
2014-08-05 09:01:58,636 INFO - See infinitydb.com- Boiler Bay Inc
2014-08-05 09:01:58,636 INFO - InfinityDB is Copyight (C) 2001-2013 Roger L. Deran, all rights reserved.
2014-08-05 09:01:58,636 INFO - InfinityDB has been Licensed to Atlassian
2014-08-05 09:01:58,636 INFO - for distribution in FishEye and Crucible.
2014-08-05 09:01:58,636 INFO - Atlassian Pty Ltd (ABN 53 102 443 916)
2014-08-05 09:01:58,636 INFO - of 341 George St., Sydney, New South Wales, 2000 Australia
2014-08-05 09:01:58,944 INFO - UpgradeManager started.
2014-08-05 09:01:59,004 INFO - loading config from file: /fe_data/fisheye_data/config.xml
2014-08-05 09:02:00,441 INFO - Crucible: Community License registered to OCLC.
2014-08-05 09:02:11,682 INFO - Adding secondary content dir of /nfsmounts/devnas3s4/fe_data/fisheye_data/content
2014-08-05 09:02:17,842 INFO - Starting database...
2014-08-05 09:02:21,848 INFO - Database started.
2014-08-05 09:02:30,125 INFO - Starting plugin system...
2014-08-05 09:05:50,415 WARN - Invalid LDAP read timeout interval: 'null', using default '1m'
2014-08-05 09:05:51,280 INFO - Starting mail system...
2014-08-05 09:05:51,292 INFO - Mail system started.
2014-08-05 09:05:53,217 INFO - Now checking for new software releases every 1 week
2014-08-05 09:05:53,608 INFO - You are not running the latest version of FishEye and Crucible. Latest version 3.5.0 can be downloaded at: http://www.atlassian.com/software/fisheye/CrucibleDownloadCenter.jspa<br< a="">>2014-08-05 09:06:36,571 INFO - Plugin system started.
2014-08-05 09:06:38,622 INFO - Server started on :15000 (http)

svn: E180001: Unable to open repository 'file:///nfsmounts/svnap01exdu/svnprod/repos/merlins/audit'
svn: E204899: Can't open file '/nfsmounts/svnap01exdu/svnprod/repos/merlins/db/current'
com.cenqua.fisheye.rep.RepositoryClientException: org.apache.subversion.javahl.ClientException: svn: E180001: Unable to open an ra_local session to URL
svn: E180001: Unable to open repository 'file:///nfsmounts/svnap01exdu/svnprod/repos/merlins/audit'
svn: E204899: Can't open file '/nfsmounts/svnap01exdu/svnprod...

[Nick]

Have you got any more information in the logs about which request was being serviced when you got the Broken pipe IOException. That generally means that a client has quit the connection, before the server fully finished writing the response.

It is unlikely that this is the cause of the instability you are seeing, however it could be another symptom.

Is this happening at a similar time each day ? Is anything happening with your network, or server at that time ?

Can you run with DEBUG logging enabled (can enable via the admin UI) and capture some more information. Please also specifiy what you mean by "becomes unstable".

Do you have enough RAM allocated to the FishEye/Crucible JVM ?

You should also consider upgrading to the latest 3.5.x release, since the memory consumption has been vastly improved especially for instances with many repositories.