"IOException: Stream closed" error when running 2 Git commands in parallel

[Alexander Kitaev]

I develop an add-on for Bitbucket Server and for our purposes I want to run

{code}
  git fast-export | git fast-import
{code}

commands connected with a pipe.

 

For these purposes I use GitCommandBuilderFactory, start 'git fast-export' command asynchronously, get the output from it. Then asynchronously start 'git fast-import' and feed the output of the first command as the input of the second one.

But once the second command starts, the output of the first command gets closed and only contains buffered data.

{code}
Caused by: java.io.IOException: Stream closed
        at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:336)
        at java.io.FilterInputStream.read(FilterInputStream.java:133)
        at com.atlassian.utils.process.WatchdogAwareInputStream.read(WatchdogAwareInputStream.java:44)
        at java.io.FilterInputStream.read(FilterInputStream.java:107)
        at com.atlassian.utils.process.WatchdogAwareInputStream.read(WatchdogAwareInputStream.java:37)
        at org.tmatesoft.subgit.stash.mirror.tasks.GitCommandBuilderFactoryBug.run(GitCommandBuilderFactoryBug.java:93)
        ... 12 common frames omitted
{code}
Note that this buffering feature could be the reason why the problem can be unnoticed if the command output is not as big.

 

I conducted experiments with inserting sleep() and just reading the output of the first command without starting the second one. Everything works fine unless and until the second command is started. So it's not a problem of idleness or timeouts. The second command is the only cause of the problem.

 

There's also a similar issue: BSERV-11330 it has absolutely the same stack trace and the same symptoms (several Git processes running simultaneously). The issue is closed as fixed at some versions including 5.16.0, but the issue I'm reporting is reproducible with 5.16.1 as well. So probably 5.16.1 was maybe fixed by avoiding running parallel processes rather than by making them work properly.

 

I wonder: are 2 or more Git processes supposed to work in parallel at all? Or are they executed one after another by design and is the behaviour I see really expected?

 

If yes I would be great if you could fix the issue before releasing 6.0 version because otherwise it would break our add-on: the old API is not available and we can't use the new one because of this bug.

 

I'm including a class that can be used to reproduce the problem (construct the object a call run() from it).
{code:java}
public class GitCommandBuilderFactoryBug implements CommandOutputHandler<Void>, CommandInputHandler {

    private final GitCommandBuilderFactory gitCommandBuilderFactory;
    private final Repository repository;
    private final Logger log;

    private InputStream fastExportInputStream;
    private OutputStream fastImportOutputStream;

    public GitCommandBuilderFactoryBug(GitCommandBuilderFactory gitCommandBuilderFactory, Repository repository, Logger log) {
        this.gitCommandBuilderFactory = gitCommandBuilderFactory;
        this.repository = repository;
        this.log = log;
    }

    @SuppressWarnings("TryWithIdenticalCatches")
    public void run() {
        final GitScmCommandBuilder fastExportCommandBuilder = gitCommandBuilderFactory.builder(repository)
                .command("fast-export");
        final GitScmCommandBuilder fastImportCommandBuilder = gitCommandBuilderFactory.builder(repository)
                .command("fast-import")
                .inputHandler(this);

        final GitCommand<Void> fastExportCommand = fastExportCommandBuilder.build(this);
        fastExportCommand.setExecutionTimeout(Integer.MAX_VALUE);
        fastExportCommand.setIdleTimeout(Integer.MAX_VALUE);

        final GitCommand<Void> fastImportCommand = fastImportCommandBuilder.build(new DummyCommandOutputHandler());
        fastImportCommand.setExecutionTimeout(Integer.MAX_VALUE);
        fastImportCommand.setIdleTimeout(Integer.MAX_VALUE);


        final Future<Void> fastExportFuture;
        final Future<Void> fastImportFuture;
        synchronized (this) {
            log.info("About to execute 'git fast-export'");
            fastExportFuture = fastExportCommand.asynchronous().start();

            log.info("'git fast-export' command started");
            try {
                log.info("Waiting for the command output");
                wait();
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }

            log.info("'git fast-export' command output object is " + this.fastExportInputStream);

            //at this moment reading from the stream has no problems
            //but once any other command is started
            //the input has only buffered data
            //but the original 'git fast-export' command either dies
            //or closes its output, so no new output of 'git fast-export' can
            //be obtained

            log.info("About to execute 'git fast-import'");
            fastImportFuture = fastImportCommand.asynchronous().start();

            log.info("'git fast-import' command started");
            try {
                log.info("Waiting for the command input");
                wait();
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }

            log.info("'git fast-import' command input object is " + this.fastImportOutputStream);

            //now redirect fastExportInputStream to fastImportOutputStream
        }

        final byte[] buffer = new byte[65536];
        try {
            for (int readBytes = fastExportInputStream.read(buffer); readBytes > 0; readBytes = fastExportInputStream.read(buffer)) {
                fastImportOutputStream.write(buffer, 0, readBytes);
            }
            fastImportOutputStream.flush();
        } catch (IOException e) {
            throw new RuntimeException(e);
        }

        try {
            fastExportFuture.get();
            fastImportFuture.get();
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        } catch (ExecutionException e) {
            throw new RuntimeException(e);
        }

        //we should get here without errors

        log.info("Fast-export/fast-import completed");
    }

    @Nullable
    @Override
    public Void getOutput() {
        return null;
    }

    @Override
    public void process(InputStream inputStream) throws ProcessException {
        synchronized (this) {
            fastExportInputStream = inputStream;
            notify();
        }
    }

    @Override
    public void process(OutputStream outputStream) {
        synchronized (this) {
            fastImportOutputStream = outputStream;
            notify();
        }
    }
    
    @Override
    public void complete() {
    }

    @Override
    public void setWatchdog(Watchdog watchdog) {
    }

    private static class DummyCommandOutputHandler implements CommandOutputHandler<Void> {
        public Void getOutput() {return null;}
        public void process(InputStream inputStream) throws ProcessException {}
        public void complete() throws ProcessException {}
        public void setWatchdog(Watchdog watchdog) {}
    }
}
{code}

[Pramodh M]

Hi @Alexander Kitaev 

Did you solve the error

Thanks,
Pramodh