FishEye leaving defunct processes

[Gerald Schneider]

Since upgrading FishEye from 4.8.2 to 4.8.3 it started creating lots and lots of defunct processes.

root@fisheye:/tmp# ps -Af |grep defunct 
[...]
root 32624 12727 0 09:44 ? 00:00:00 [sh] <defunct>
root@fisheye:/tmp# ps -Af |grep defunct |wc -l
49

Currently there are 49 processes after running for 23 minutes. Before I restarted the container there were more than 8500 defunct processes (running for 3 days), all with the PID of the Fisheye server as the parent PID.

I can't find anything suspicious in the log files.

Is there any way I can find out where these processes come from?

Update

I tried to find out more about the defunct processes by looking at the corresponding files in /proc/<PID>, but I couldn't find anything helpful. The cmd file for example is empty (it isn't for still running processes).

I looked at the Fisheye log files at the timestamps where the defunct processes were started. The number of defunct processes from that 1 minute  timeframe matches the number of nonfunctional repositories Fisheye encountered in that timeframe. For example, today at 09:58 two defunct processes were created, in the log file I find two WARN messages within that minute:

2020-08-06 09:58:14,349 WARN [IncrPing2 REPO01 ] fisheye BaseRepositoryScanner-handleSlurpException - Problem processing revisions from repository REPO01 (REPO01) due to class com.cenqua.fisheye.rep.RepositoryClientException - org.apache.subversion.javahl.ClientException: svn: E175002: PROPFIND of '/scm/svn/repo01/!svn/ver/4300/branches/branch/path/to/file.msh': 400 Bad Request (https://scm.example.com)
com.cenqua.fisheye.rep.RepositoryClientException: org.apache.subversion.javahl.ClientException: svn: E175002: PROPFIND of '/scm/svn/repo01/!svn/ver/4300/branches/branch/path/to/file.msh': 400 Bad Request (https://scm.example.com)
at com.cenqua.fisheye.svn.SvnThrottledClient.executeNoThrottle(SvnThrottledClient.java:189) [fisheye.jar:?]
at com.cenqua.fisheye.svn.SvnThrottledClient.execute(SvnThrottledClient.java:158) [fisheye.jar:?]
at com.cenqua.fisheye.svn.SvnThrottledClient.diffSummarize(SvnThrottledClient.java:228) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Infill2Processor.summarizeDiff(Svn2Infill2Processor.java:314) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Infill2Processor.createDiffSummary(Svn2Infill2Processor.java:296) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Infill2Processor.infillChangeSet(Svn2Infill2Processor.java:212) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Infill2Processor.access$100(Svn2Infill2Processor.java:92) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Infill2Processor$1.perform(Svn2Infill2Processor.java:143) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Infill2Processor$1.perform(Svn2Infill2Processor.java:127) [fisheye.jar:?]
at com.cenqua.fisheye.cache.BaseRevisionCache.withDbWriteLock(BaseRevisionCache.java:1086) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Infill2Processor.process(Svn2Infill2Processor.java:127) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Scanner.infillRepository(Svn2Scanner.java:233) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Scanner.slurpRepository(Svn2Scanner.java:219) [fisheye.jar:?]
at com.atlassian.fisheye.svn.Svn2Scanner.doSlurpTransaction(Svn2Scanner.java:184) [fisheye.jar:?]
at com.cenqua.fisheye.rep.BaseRepositoryScanner.ping(BaseRepositoryScanner.java:73) [fisheye.jar:?]
at com.cenqua.fisheye.rep.BaseRepositoryEngine.doSlurp(BaseRepositoryEngine.java:85) [fisheye.jar:?]
at com.cenqua.fisheye.rep.RepositoryEngine.slurp(RepositoryEngine.java:419) [fisheye.jar:?]
at com.cenqua.fisheye.rep.ping.IndexingPingRequest.doRequest(IndexingPingRequest.java:28) [fisheye.jar:?]
at com.cenqua.fisheye.rep.ping.IncrementalPingRequest.doRequest(IncrementalPingRequest.java:30) [fisheye.jar:?]
at com.cenqua.fisheye.rep.ping.PingRequest$1.run(PingRequest.java:55) [fisheye.jar:?]
at com.cenqua.fisheye.util.NamedExecution.run(NamedExecution.java:27) [fisheye.jar:?]
at com.cenqua.fisheye.rep.ping.PingRequest.process(PingRequest.java:52) [fisheye.jar:?]
at com.cenqua.fisheye.rep.RepositoryHandle.processPingRequests(RepositoryHandle.java:211) [fisheye.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Caused by: org.apache.subversion.javahl.ClientException: svn: E175002: PROPFIND of '/scm/svn/repo01/!svn/ver/4300/branches/branch/path/to/file.msh': 400 Bad Request (https://scm.example.com)
at org.apache.subversion.javahl.ClientException.fromException(ClientException.java:117) [svnkit-javahl16-1.10.1.jar:?]
at org.tmatesoft.svn.core.javahl17.SVNClientImpl.getClientException(SVNClientImpl.java:1535) [svnkit-javahl16-1.10.1.jar:?]
at org.tmatesoft.svn.core.javahl17.SVNClientImpl.diffSummarize(SVNClientImpl.java:1249) [svnkit-javahl16-1.10.1.jar:?]
at org.apache.subversion.javahl.SVNClient.diffSummarize(SVNClient.java:211) [svnkit-javahl16-1.10.1.jar:?]
at com.cenqua.fisheye.svn.SvnThrottledClient$4.call(SvnThrottledClient.java:234) [fisheye.jar:?]
at com.cenqua.fisheye.svn.SvnThrottledClient$4.call(SvnThrottledClient.java:229) [fisheye.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_252]
at com.cenqua.fisheye.svn.SvnTask.access$101(SvnTask.java:13) [fisheye.jar:?]
at com.cenqua.fisheye.svn.SvnTask$1.run(SvnTask.java:35) [fisheye.jar:?]
at com.cenqua.fisheye.util.NamedExecution.run(NamedExecution.java:27) [fisheye.jar:?]
at com.cenqua.fisheye.svn.SvnTask.run(SvnTask.java:30) [fisheye.jar:?]
... 3 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: PROPFIND of '/scm/svn/repo01/!svn/ver/4300/branches/branch/path/to/file.msh': 400 Bad Request (https://scm.example.com)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:112) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:96) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:765) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:910) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1361) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.diff(DAVRepository.java:839) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgDiffSummarize.doDiffReposRepos(SvnNgDiffSummarize.java:273) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgDiffSummarize.doDiff(SvnNgDiffSummarize.java:86) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgDiffSummarize.run(SvnNgDiffSummarize.java:59) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgDiffSummarize.run(SvnNgDiffSummarize.java:23) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.javahl17.SVNClientImpl.diffSummarize(SVNClientImpl.java:1247) [svnkit-javahl16-1.10.1.jar:?]
... 11 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: PROPFIND of '/scm/svn/repo01/!svn/ver/4300/branches/branch/path/to/file.msh': 400 Bad Request (https://scm.example.com)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.DAVUtil.getResourceProperties(DAVUtil.java:79) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.handlers.DAVEditorHandler.addNodeProperties(DAVEditorHandler.java:659) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.handlers.DAVEditorHandler.endElement(DAVEditorHandler.java:504) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.handlers.BasicDAVHandler.endElement(BasicDAVHandler.java:103) [svnkit-1.10.1.jar:?]
at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source) [xercesImpl-2.12.0.jar:?]
at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown Source) [xercesImpl-2.12.0.jar:2.12.0]
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) [xercesImpl-2.12.0.jar:2.12.0]
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) [xercesImpl-2.12.0.jar:2.12.0]
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) [xercesImpl-2.12.0.jar:?]
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) [xercesImpl-2.12.0.jar:?]
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) [xercesImpl-2.12.0.jar:?]
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) [xercesImpl-2.12.0.jar:?]
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) [xercesImpl-2.12.0.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:898) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:863) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPRequest.dispatch(HTTPRequest.java:220) [svnkit-1.10.1.jar:?]
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:451) [svnkit-1.10.1.jar:?]
... 26 more
2020-08-06 09:58:55,115 WARN [IncrPing2 REPO02 ] fisheye IndexingPingRequest-doRequest - Exception during Fisheye Incremental Indexing of REPO02 (REPO02): com.cenqua.fisheye.config.ConfigException: com.atlassian.fisheye.dvcs.handler.DvcsProcessException: Error while communicating with VCS: fatal: '/svn/git/REPO02/' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

So I guess the defunct processes are left behind after Fisheye encountered an error.

Of course these errors in the repositories shouldn't happen and need to be fixed, but Fisheye still shouldn't leave defunct processes behind when it encounters one. It doesn't on my second Fisheye instance, just on this server.

Update 2

The problem persists after updating to FishEye 4.8.4.

Update 3

I'm on FishEye 4.8.5 now and the problem still persists. Am I the only one with this problem? I currently have a cron job running that restarts FishEye every night.

Update 4 (2021-02-02)

I modified the Docker container, basing it on the adoptopenjdk images with Ubuntu 20.04 instead 18.04, hoping the newer package versions might help. The did not. I also modified the container to start FishEye as an ordinary user instead of root, like the official images for JIRA, Confluence etc. It didn't change a thing, I still get lots and lots of defunct processes.

[Gerald Schneider · Accepted solution]

The cause of the defunct processes are failed repository queries.

Below is the Zabbix graph of the last seven days of one of my FishEye instances:

I had a daily restart scheduled at 3 o'clock for the FishEye container, which also removed all defunct processes.

On Dec 7th I stopped and disabled all repositories that threw errors (because the repo didn't exist anymore, or was defective, or contained problematic files). The creation of defunct processes stopped instantly. After the next scheduled restart the number of processes stayed low.

So, the cause is clear now. Why those repository queries leave a defunct process behind when they fail is still not answered.