Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

FishEye leaving defunct processes

Edited

Since upgrading FishEye from 4.8.2 to 4.8.3 it started creating lots and lots of defunct processes.

root@fisheye:/tmp# ps -Af |grep defunct 
[...]
root 32624 12727 0 09:44 ? 00:00:00 [sh] <defunct>
root@fisheye:/tmp# ps -Af |grep defunct |wc -l
49

Currently there are 49 processes after running for 23 minutes. Before I restarted the container there were more than 8500 defunct processes (running for 3 days), all with the PID of the Fisheye server as the parent PID.

I can't find anything suspicious in the log files.

Is there any way I can find out where these processes come from?

Update

I tried to find out more about the defunct processes by looking at the corresponding files in /proc/<PID>, but I couldn't find anything helpful. The cmd file for example is empty (it isn't for still running processes).

I looked at the Fisheye log files at the timestamps where the defunct processes were started. The number of defunct processes from that 1 minute  timeframe matches the number of nonfunctional repositories Fisheye encountered in that timeframe. For example, today at 09:58 two defunct processes were created, in the log file I find two WARN messages within that minute:

2020-08-06 09:58:14,349 WARN [IncrPing2 REPO01 ] fisheye BaseRepositoryScanner-handleSlurpException - Problem processing revisions from repository REPO01 (REPO01) due to class com.cenqua.fisheye.rep.RepositoryClientException - org.apache.subversion.javahl.ClientException: svn: E175002: PROPFIND of '/scm/svn/repo01/!svn/ver/4300/branches/branch/path/to/file.msh': 400 Bad Request (https://scm.example.com)
com.cenqua.fisheye.rep.RepositoryClientException: org.apache.subversion.javahl.ClientException: svn: E175002: PROPFIND of '/scm/svn/repo01/!svn/ver/4300/branches/branch/path/to/file.msh': 400 Bad Request (https://scm.example.com)
at com.cenqua.fisheye.svn.SvnThrottledClient.executeNoThrottle(SvnThrottledClient.java:189) [fisheye.jar:?]
at com.cenqua.fisheye.svn.SvnThrottledClient.execute(SvnThrottledClient.java:158) [fisheye.jar:?]
at com.cenqua.fisheye.svn.SvnThrottledClient.diffSummarize(SvnThrottledClient.java:228) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Infill2Processor.summarizeDiff(Svn2Infill2Processor.java:314) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Infill2Processor.createDiffSummary(Svn2Infill2Processor.java:296) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Infill2Processor.infillChangeSet(Svn2Infill2Processor.java:212) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Infill2Processor.access$100(Svn2Infill2Processor.java:92) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Infill2Processor$1.perform(Svn2Infill2Processor.java:143) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Infill2Processor$1.perform(Svn2Infill2Processor.java:127) [fisheye.jar:?]
at com.cenqua.fisheye.cache.BaseRevisionCache.withDbWriteLock(BaseRevisionCache.java:1086) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Infill2Processor.process(Svn2Infill2Processor.java:127) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Scanner.infillRepository(Svn2Scanner.java:233) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Scanner.slurpRepository(Svn2Scanner.java:219) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Scanner.doSlurpTransaction(Svn2Scanner.java:184) [fisheye.jar:?]
at com.cenqua.fisheye.rep.BaseRepositoryScanner.ping(BaseRepositoryScanner.java:73) [fisheye.jar:?]
at com.cenqua.fisheye.rep.BaseRepositoryEngine.doSlurp(BaseRepositoryEngine.java:85) [fisheye.jar:?]
at com.cenqua.fisheye.rep.RepositoryEngine.slurp(RepositoryEngine.java:419) [fisheye.jar:?]
at com.cenqua.fisheye.rep.ping.IndexingPingRequest.doRequest(IndexingPingRequest.java:28) [fisheye.jar:?]
at com.cenqua.fisheye.rep.ping.IncrementalPingRequest.doRequest(IncrementalPingRequest.java:30) [fisheye.jar:?]
at com.cenqua.fisheye.rep.ping.PingRequest$1.run(PingRequest.java:55) [fisheye.jar:?]
at com.cenqua.fisheye.util.NamedExecution.run(NamedExecution.java:27) [fisheye.jar:?]
at com.cenqua.fisheye.rep.ping.PingRequest.process(PingRequest.java:52) [fisheye.jar:?]
at com.cenqua.fisheye.rep.RepositoryHandle.processPingRequests(RepositoryHandle.java:211) [fisheye.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Caused by: org.apache.subversion.javahl.ClientException: svn: E175002: PROPFIND of '/scm/svn/repo01/!svn/ver/4300/branches/branch/path/to/file.msh': 400 Bad Request (https://scm.example.com)
at org.apache.subversion.javahl.ClientException.fromException(ClientException.java:117) [svnkit-javahl16-1.10.1.jar:?]
at org.tmatesoft.svn.core.javahl17.SVNClientImpl.getClientException(SVNClientImpl.java:1535) [svnkit-javahl16-1.10.1.jar:?]
at org.tmatesoft.svn.core.javahl17.SVNClientImpl.diffSummarize(SVNClientImpl.java:1249) [svnkit-javahl16-1.10.1.jar:?]
at org.apache.subversion.javahl.SVNClient.diffSummarize(SVNClient.java:211) [svnkit-javahl16-1.10.1.jar:?]
at com.cenqua.fisheye.svn.SvnThrottledClient$4.call(SvnThrottledClient.java:234) [fisheye.jar:?]
at com.cenqua.fisheye.svn.SvnThrottledClient$4.call(SvnThrottledClient.java:229) [fisheye.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_252]
at com.cenqua.fisheye.svn.SvnTask.access$101(SvnTask.java:13) [fisheye.jar:?]
at com.cenqua.fisheye.svn.SvnTask$1.run(SvnTask.java:35) [fisheye.jar:?]
at com.cenqua.fisheye.util.NamedExecution.run(NamedExecution.java:27) [fisheye.jar:?]
at com.cenqua.fisheye.svn.SvnTask.run(SvnTask.java:30) [fisheye.jar:?]
... 3 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: PROPFIND of '/scm/svn/repo01/!svn/ver/4300/branches/branch/path/to/file.msh': 400 Bad Request (https://scm.example.com)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:112) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:96) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:765) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:910) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1361) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.diff(DAVRepository.java:839) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgDiffSummarize.doDiffReposRepos(SvnNgDiffSummarize.java:273) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgDiffSummarize.doDiff(SvnNgDiffSummarize.java:86) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgDiffSummarize.run(SvnNgDiffSummarize.java:59) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgDiffSummarize.run(SvnNgDiffSummarize.java:23) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.javahl17.SVNClientImpl.diffSummarize(SVNClientImpl.java:1247) [svnkit-javahl16-1.10.1.jar:?]
... 11 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: PROPFIND of '/scm/svn/repo01/!svn/ver/4300/branches/branch/path/to/file.msh': 400 Bad Request (https://scm.example.com)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.DAVUtil.getResourceProperties(DAVUtil.java:79) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.handlers.DAVEditorHandler.addNodeProperties(DAVEditorHandler.java:659) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.handlers.DAVEditorHandler.endElement(DAVEditorHandler.java:504) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.handlers.BasicDAVHandler.endElement(BasicDAVHandler.java:103) [svnkit-1.10.1.jar:?]
at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source) [xercesImpl-2.12.0.jar:?]
at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown Source) [xercesImpl-2.12.0.jar:2.12.0]
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) [xercesImpl-2.12.0.jar:2.12.0]
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) [xercesImpl-2.12.0.jar:2.12.0]
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) [xercesImpl-2.12.0.jar:?]
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) [xercesImpl-2.12.0.jar:?]
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) [xercesImpl-2.12.0.jar:?]
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) [xercesImpl-2.12.0.jar:?]
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) [xercesImpl-2.12.0.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:898) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:863) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPRequest.dispatch(HTTPRequest.java:220) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:451) [svnkit-1.10.1.jar:?]
... 26 more
2020-08-06 09:58:55,115 WARN [IncrPing2 REPO02 ] fisheye IndexingPingRequest-doRequest - Exception during Fisheye Incremental Indexing of REPO02 (REPO02): com.cenqua.fisheye.config.ConfigException: com.atlassian.fisheye.dvcs.handler.DvcsProcessException: Error while communicating with VCS: fatal: '/svn/git/REPO02/' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

So I guess the defunct processes are left behind after Fisheye encountered an error.

Of course these errors in the repositories shouldn't happen and need to be fixed, but Fisheye still shouldn't leave defunct processes behind when it encounters one. It doesn't on my second Fisheye instance, just on this server.

Update 2

The problem persists after updating to FishEye 4.8.4.

Update 3

I'm on FishEye 4.8.5 now and the problem still persists. Am I the only one with this problem? I currently have a cron job running that restarts FishEye every night.

Update 4 (2021-02-02)

I modified the Docker container, basing it on the adoptopenjdk images with Ubuntu 20.04 instead 18.04, hoping the newer package versions might help. The did not. I also modified the container to start FishEye as an ordinary user instead of root, like the official images for JIRA, Confluence etc. It didn't change a thing, I still get lots and lots of defunct processes.

0 answers

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
SERVER
VERSION
4.8.5
TAGS
Community showcase
Published in Jira Service Management

Security Advisory for Jira Service Management

On October 20, 2021, Atlassian published a security advisory for Jira Service Management. The full advisory is available at this link.  We've seen a number of questions already asking for...

54 views 0 1
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you