Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

"IOException: Stream closed" error when running 2 Git commands in parallel

I develop an add-on for Bitbucket Server and for our purposes I want to run

{code}
  git fast-export | git fast-import
{code}

commands connected with a pipe.

 

For these purposes I use GitCommandBuilderFactory, start 'git fast-export' command asynchronously, get the output from it. Then asynchronously start 'git fast-import' and feed the output of the first command as the input of the second one.

But once the second command starts, the output of the first command gets closed and only contains buffered data.

{code}
Caused by: java.io.IOException: Stream closed
        at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:336)
        at java.io.FilterInputStream.read(FilterInputStream.java:133)
        at com.atlassian.utils.process.WatchdogAwareInputStream.read(WatchdogAwareInputStream.java:44)
        at java.io.FilterInputStream.read(FilterInputStream.java:107)
        at com.atlassian.utils.process.WatchdogAwareInputStream.read(WatchdogAwareInputStream.java:37)
        at org.tmatesoft.subgit.stash.mirror.tasks.GitCommandBuilderFactoryBug.run(GitCommandBuilderFactoryBug.java:93)
        ... 12 common frames omitted
{code}
Note that this buffering feature could be the reason why the problem can be unnoticed if the command output is not as big.

 

I conducted experiments with inserting sleep() and just reading the output of the first command without starting the second one. Everything works fine unless and until the second command is started. So it's not a problem of idleness or timeouts. The second command is the only cause of the problem.

 

There's also a similar issue: BSERV-11330 it has absolutely the same stack trace and the same symptoms (several Git processes running simultaneously). The issue is closed as fixed at some versions including 5.16.0, but the issue I'm reporting is reproducible with 5.16.1 as well. So probably 5.16.1 was maybe fixed by avoiding running parallel processes rather than by making them work properly.

 

I wonder: are 2 or more Git processes supposed to work in parallel at all? Or are they executed one after another by design and is the behaviour I see really expected?

 

If yes I would be great if you could fix the issue before releasing 6.0 version because otherwise it would break our add-on: the old API is not available and we can't use the new one because of this bug.

 

I'm including a class that can be used to reproduce the problem (construct the object a call run() from it).
{code:java}
public class GitCommandBuilderFactoryBug implements CommandOutputHandler<Void>, CommandInputHandler {

    private final GitCommandBuilderFactory gitCommandBuilderFactory;
    private final Repository repository;
    private final Logger log;

    private InputStream fastExportInputStream;
    private OutputStream fastImportOutputStream;

    public GitCommandBuilderFactoryBug(GitCommandBuilderFactory gitCommandBuilderFactory, Repository repository, Logger log) {
        this.gitCommandBuilderFactory = gitCommandBuilderFactory;
        this.repository = repository;
        this.log = log;
    }

    @SuppressWarnings("TryWithIdenticalCatches")
    public void run() {
        final GitScmCommandBuilder fastExportCommandBuilder = gitCommandBuilderFactory.builder(repository)
                .command("fast-export");
        final GitScmCommandBuilder fastImportCommandBuilder = gitCommandBuilderFactory.builder(repository)
                .command("fast-import")
                .inputHandler(this);

        final GitCommand<Void> fastExportCommand = fastExportCommandBuilder.build(this);
        fastExportCommand.setExecutionTimeout(Integer.MAX_VALUE);
        fastExportCommand.setIdleTimeout(Integer.MAX_VALUE);

        final GitCommand<Void> fastImportCommand = fastImportCommandBuilder.build(new DummyCommandOutputHandler());
        fastImportCommand.setExecutionTimeout(Integer.MAX_VALUE);
        fastImportCommand.setIdleTimeout(Integer.MAX_VALUE);


        final Future<Void> fastExportFuture;
        final Future<Void> fastImportFuture;
        synchronized (this) {
            log.info("About to execute 'git fast-export'");
            fastExportFuture = fastExportCommand.asynchronous().start();

            log.info("'git fast-export' command started");
            try {
                log.info("Waiting for the command output");
                wait();
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }

            log.info("'git fast-export' command output object is " + this.fastExportInputStream);

            //at this moment reading from the stream has no problems
            //but once any other command is started
            //the input has only buffered data
            //but the original 'git fast-export' command either dies
            //or closes its output, so no new output of 'git fast-export' can
            //be obtained

            log.info("About to execute 'git fast-import'");
            fastImportFuture = fastImportCommand.asynchronous().start();

            log.info("'git fast-import' command started");
            try {
                log.info("Waiting for the command input");
                wait();
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }

            log.info("'git fast-import' command input object is " + this.fastImportOutputStream);

            //now redirect fastExportInputStream to fastImportOutputStream
        }

        final byte[] buffer = new byte[65536];
        try {
            for (int readBytes = fastExportInputStream.read(buffer); readBytes > 0; readBytes = fastExportInputStream.read(buffer)) {
                fastImportOutputStream.write(buffer, 0, readBytes);
            }
            fastImportOutputStream.flush();
        } catch (IOException e) {
            throw new RuntimeException(e);
        }

        try {
            fastExportFuture.get();
            fastImportFuture.get();
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        } catch (ExecutionException e) {
            throw new RuntimeException(e);
        }

        //we should get here without errors

        log.info("Fast-export/fast-import completed");
    }

    @Nullable
    @Override
    public Void getOutput() {
        return null;
    }

    @Override
    public void process(InputStream inputStream) throws ProcessException {
        synchronized (this) {
            fastExportInputStream = inputStream;
            notify();
        }
    }

    @Override
    public void process(OutputStream outputStream) {
        synchronized (this) {
            fastImportOutputStream = outputStream;
            notify();
        }
    }
    
    @Override
    public void complete() {
    }

    @Override
    public void setWatchdog(Watchdog watchdog) {
    }

    private static class DummyCommandOutputHandler implements CommandOutputHandler<Void> {
        public Void getOutput() {return null;}
        public void process(InputStream inputStream) throws ProcessException {}
        public void complete() throws ProcessException {}
        public void setWatchdog(Watchdog watchdog) {}
    }
}
{code}

1 answer

0 votes
Pramodh M Community Leader Dec 20, 2021

Hi @Alexander Kitaev 

Did you solve the error

Thanks,
Pramodh

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket

📣 Calling Bitbucket Data Center customers to participate in research

Hi everyone, Are you Bitbucket DC customer? If so, we'd love to talk to you! Our team wants to dive deep to understand your long-term plans regarding Bitbucket DC and Atlassian Cloud. Do you plan...

108 views 2 4
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you