nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Need to disable the ssh proxy for git checkouts

Joe Ardent

The standard Checkout Source task for git repos, for some insane reason I've never fathomed, proxies the ssh connection through the Bamboo master node. In addition to this becoming a network chokepoint when we have a bunch of builds going, therefore causing the checkout times to be on the order of 20 minutes instead of thirty seconds, it's fragile, and breaks all the time. When this happens, the build breaks.

Here's the stacktrace, with possibly sensitive information, like usernames, hostnames, and repo names, replaced with metavalues or bland equivalent strings:

{noformat}

java.lang.RuntimeException: com.atlassian.bamboo.plugins.git.GitCommandException: command /usr/bin/git, fetch, ssh://<randomish numerical="" username="">@127.0.0.1:47141/repo.git, +refs/heads/branch:refs/heads/branch, --update-head-ok, --progress, --verbose failed. Working directory was `/home/bamboo/bamboo-agent-home/xml-data/build-dir/PLAN_NAME'. stderr: SSH Proxy error: Can't connect session, connectFuture is not connected... stdout: <username>@<hostname>/<ip address="">:22 Can't connect session, connectFuture is not connected...fatal: The remote end hung up unexpectedly

at com.atlassian.bamboo.executor.RetryingTaskExecutor.rerun(RetryingTaskExecutor.java:119)

at com.atlassian.bamboo.executor.RetryingTaskExecutor.runTask(RetryingTaskExecutor.java:79)

at com.atlassian.bamboo.executor.RetryingTaskExecutor.retry(RetryingTaskExecutor.java:174)

at com.atlassian.bamboo.plugins.vcs.task.VcsCheckoutTask.execute(VcsCheckoutTask.java:97)

at com.atlassian.bamboo.task.TaskExecutorImpl.executeTasks(TaskExecutorImpl.java:180)

at com.atlassian.bamboo.task.TaskExecutorImpl.executePreparationTasks(TaskExecutorImpl.java:64)

at com.atlassian.bamboo.build.pipeline.tasks.PrepareBuildTask.call(PrepareBuildTask.java:66)

at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent.build(DefaultBuildAgent.java:202)

at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl.waitAndPerformBuild(BuildAgentControllerImpl.java:103)

at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent$1.run(DefaultBuildAgent.java:109)

at com.atlassian.bamboo.build.pipeline.concurrent.NamedThreadFactory$2.run(NamedThreadFactory.java:50)

at java.lang.Thread.run(Thread.java:662)

Caused by: com.atlassian.bamboo.plugins.git.GitCommandException: command /usr/bin/git, fetch, ssh://<randomish username="">@127.0.0.1:47141/repo.git, +refs/heads/branch:refs/heads/branch, --update-head-ok, --progress, --verbose failed. Working directory was `/home/bamboo/bamboo-agent-home/xml-data/build-dir/PLAN_NAME'. stderr: SSH Proxy error: Can't connect session, connectFuture is not connected... stdout: <username>@<hostname>/<ip address="">:22 Can't connect session, connectFuture is not connected...fatal: The remote end hung up unexpectedly

at com.atlassian.bamboo.plugins.git.GitCommandProcessor.runCommand(GitCommandProcessor.java:223)

at com.atlassian.bamboo.plugins.git.GitCommandProcessor.runFetchCommand(GitCommandProcessor.java:119)

at com.atlassian.bamboo.plugins.git.NativeGitOperationHelper.doFetch(NativeGitOperationHelper.java:56)

at com.atlassian.bamboo.plugins.git.GitOperationHelper.fetch(GitOperationHelper.java:162)

at com.atlassian.bamboo.plugins.git.GitRepository.retrieveSourceCode(GitRepository.java:345)

at com.atlassian.bamboo.plugins.vcs.task.VcsCheckoutTask.checkoutCode(VcsCheckoutTask.java:153)

at com.atlassian.bamboo.plugins.vcs.task.VcsCheckoutTask.access$100(VcsCheckoutTask.java:41)

at com.atlassian.bamboo.plugins.vcs.task.VcsCheckoutTask$1.call(VcsCheckoutTask.java:102)

at com.atlassian.bamboo.plugins.vcs.task.VcsCheckoutTask$1.call(VcsCheckoutTask.java:98)

at com.atlassian.bamboo.executor.RetryingTaskExecutor.rerun(RetryingTaskExecutor.java:100)

... 11 more

Caused by: com.atlassian.utils.process.ProcessException: Non-zero exit code: 128

at com.atlassian.utils.process.PluggableProcessHandler.complete(PluggableProcessHandler.java:83)

at com.atlassian.utils.process.ExternalProcess.finish(ExternalProcess.java:366)

at com.atlassian.utils.process.ExternalProcess.execute(ExternalProcess.java:409)

at com.atlassian.bamboo.plugins.git.GitCommandProcessor.runCommand(GitCommandProcessor.java:218)

... 20 more

{noformat}

So, how can we disable this? We've actually replaced a lot of our jobs' source checkout tasks with a custom script that just calls git natively through bash, but that doesn't work very well for our Windows hosts. Also, because there's no programmatic access to the Bamboo admin functionality, there's no way to not have to click click click on the slow-ass web UI in order to update jobs to use our custom task, which makes it hard to motivate to apply that workaround.

Please help. We're planning on upgrading to Bamboo 4.4 as soon as possible, but that is likely several months away, and if it still has the proxy-based design, we'll still be screwed. Thank you.

Accepted answers

PiotrA

Hi Joe,

Are you feeling comfortable with developing plugins for Atlassian products? Because I remember that once in the past I've experimented with cutting out the ssh proxy out of the bamboo git plugin: https://bitbucket.org/atlassian/bamboo-git-plugin/commits/f797004d39ac8b152948e24d93e017011d51b1e6 - but it was long time ago (10 months ago), so one would probably have to use the above-linked changeset to patch his/her current bamboo git plugin version and rebuild the plugin (and install it on his/her Bamboo instance).

It's not something very easy, but it's not a rocket science too - so if you think it is a good idea to cut the ssh proxy you might try this approach (-> patch your bamboo git plugin).

Is it helpful to you?

regards,

P. S.
4.4 is still using proxy-based design.

All comments

PiotrA

Joe Ardent

That is awesome; thank you, Piotr!

Oleg Semyonov

The main idea around ssh proxy is don't pass ssh keys or password to remote agents. It is ok, but in some cases I'd like to have a direct connection if I trust my remote agents. Maybe some configuration option...

I tried to build latest git plugin from the repository with this patch, but Bamboo 4.4 didn't like it (with an exception caught).

At the same time I see that some remote agent git caching is added. I hope it will solve the problem being published in the next release of plugin (and apparently it will require Bamboo 5.x if I am right looking at plugin source).

PiotrA

Can you post the exception here? I'm curious why Bamboo 4.4 didn't worked with the patched git plugin.

ad 'remote agent git caching' - it doesn't solve the problem of accessing remote ssh'ish repositories directly.

Oleg Semyonov

Sorry, I've removed the plugin and have no time yet to rebuild and install it. Besides, it is a public OpenSource project server accessible 24/7 worldwide, and I can't experiment a lot. Maybe I will try to use evaluation version later. It was something like "unable to check repositories", the same as after I returned back bundled version 2.5.6 and until server was restarted (it didn't help with 3.0.4-SNAPSHOT). Might be it was because I used ssh access instead of anonymous, so it was unable to update repositories w/o ssh key. But it even didn't want to show past build results.

As for remote git cache, it may (should) solve the primary customer problem - " the checkout times to be on the order of 20 minutes instead of thirty seconds".

PiotrA

Ah, I see. That make more sense now.

Thanks for taking time to explain the things - and if you ever happen to test that patched plugin against evaluation version, please don't hesitate to post the exception/errors here. I'm keen to review them - maybe it's something trivial to fix.

regards!

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

bamboo

git

ssh