Need to disable the ssh proxy for git checkouts

[Joe Ardent]

The standard Checkout Source task for git repos, for some insane reason I've never fathomed, proxies the ssh connection through the Bamboo master node. In addition to this becoming a network chokepoint when we have a bunch of builds going, therefore causing the checkout times to be on the order of 20 minutes instead of thirty seconds, it's fragile, and breaks all the time. When this happens, the build breaks.

Here's the stacktrace, with possibly sensitive information, like usernames, hostnames, and repo names, replaced with metavalues or bland equivalent strings:

{noformat}

java.lang.RuntimeException: com.atlassian.bamboo.plugins.git.GitCommandException: command /usr/bin/git, fetch, ssh://<randomish numerical="" username="">@127.0.0.1:47141/repo.git, +refs/heads/branch:refs/heads/branch, --update-head-ok, --progress, --verbose failed. Working directory was `/home/bamboo/bamboo-agent-home/xml-data/build-dir/PLAN_NAME'. stderr: SSH Proxy error: Can't connect session, connectFuture is not connected... stdout: <username>@<hostname>/<ip address="">:22 Can't connect session, connectFuture is not connected...fatal: The remote end hung up unexpectedly

at com.atlassian.bamboo.executor.RetryingTaskExecutor.rerun(RetryingTaskExecutor.java:119)

at com.atlassian.bamboo.executor.RetryingTaskExecutor.runTask(RetryingTaskExecutor.java:79)

at com.atlassian.bamboo.executor.RetryingTaskExecutor.retry(RetryingTaskExecutor.java:174)

at com.atlassian.bamboo.plugins.vcs.task.VcsCheckoutTask.execute(VcsCheckoutTask.java:97)

at com.atlassian.bamboo.task.TaskExecutorImpl.executeTasks(TaskExecutorImpl.java:180)

at com.atlassian.bamboo.task.TaskExecutorImpl.executePreparationTasks(TaskExecutorImpl.java:64)

at com.atlassian.bamboo.build.pipeline.tasks.PrepareBuildTask.call(PrepareBuildTask.java:66)

at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent.build(DefaultBuildAgent.java:202)

at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl.waitAndPerformBuild(BuildAgentControllerImpl.java:103)

at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent$1.run(DefaultBuildAgent.java:109)

at com.atlassian.bamboo.build.pipeline.concurrent.NamedThreadFactory$2.run(NamedThreadFactory.java:50)

at java.lang.Thread.run(Thread.java:662)

Caused by: com.atlassian.bamboo.plugins.git.GitCommandException: command /usr/bin/git, fetch, ssh://<randomish username="">@127.0.0.1:47141/repo.git, +refs/heads/branch:refs/heads/branch, --update-head-ok, --progress, --verbose failed. Working directory was `/home/bamboo/bamboo-agent-home/xml-data/build-dir/PLAN_NAME'. stderr: SSH Proxy error: Can't connect session, connectFuture is not connected... stdout: <username>@<hostname>/<ip address="">:22 Can't connect session, connectFuture is not connected...fatal: The remote end hung up unexpectedly

at com.atlassian.bamboo.plugins.git.GitCommandProcessor.runCommand(GitCommandProcessor.java:223)

at com.atlassian.bamboo.plugins.git.GitCommandProcessor.runFetchCommand(GitCommandProcessor.java:119)

at com.atlassian.bamboo.plugins.git.NativeGitOperationHelper.doFetch(NativeGitOperationHelper.java:56)

at com.atlassian.bamboo.plugins.git.GitOperationHelper.fetch(GitOperationHelper.java:162)

at com.atlassian.bamboo.plugins.git.GitRepository.retrieveSourceCode(GitRepository.java:345)

at com.atlassian.bamboo.plugins.vcs.task.VcsCheckoutTask.checkoutCode(VcsCheckoutTask.java:153)

at com.atlassian.bamboo.plugins.vcs.task.VcsCheckoutTask.access$100(VcsCheckoutTask.java:41)

at com.atlassian.bamboo.plugins.vcs.task.VcsCheckoutTask$1.call(VcsCheckoutTask.java:102)

at com.atlassian.bamboo.plugins.vcs.task.VcsCheckoutTask$1.call(VcsCheckoutTask.java:98)

at com.atlassian.bamboo.executor.RetryingTaskExecutor.rerun(RetryingTaskExecutor.java:100)

... 11 more

Caused by: com.atlassian.utils.process.ProcessException: Non-zero exit code: 128

at com.atlassian.utils.process.PluggableProcessHandler.complete(PluggableProcessHandler.java:83)

at com.atlassian.utils.process.ExternalProcess.finish(ExternalProcess.java:366)

at com.atlassian.utils.process.ExternalProcess.execute(ExternalProcess.java:409)

at com.atlassian.bamboo.plugins.git.GitCommandProcessor.runCommand(GitCommandProcessor.java:218)

... 20 more

{noformat}

So, how can we disable this? We've actually replaced a lot of our jobs' source checkout tasks with a custom script that just calls git natively through bash, but that doesn't work very well for our Windows hosts. Also, because there's no programmatic access to the Bamboo admin functionality, there's no way to not have to click click click on the slow-ass web UI in order to update jobs to use our custom task, which makes it hard to motivate to apply that workaround.

Please help. We're planning on upgrading to Bamboo 4.4 as soon as possible, but that is likely several months away, and if it still has the proxy-based design, we'll still be screwed. Thank you.

[PiotrA · Accepted solution]

Hi Joe,

Are you feeling comfortable with developing plugins for Atlassian products? Because I remember that once in the past I've experimented with cutting out the ssh proxy out of the bamboo git plugin: https://bitbucket.org/atlassian/bamboo-git-plugin/commits/f797004d39ac8b152948e24d93e017011d51b1e6 - but it was long time ago (10 months ago), so one would probably have to use the above-linked changeset to patch his/her current bamboo git plugin version and rebuild the plugin (and install it on his/her Bamboo instance).

It's not something very easy, but it's not a rocket science too - so if you think it is a good idea to cut the ssh proxy you might try this approach (-> patch your bamboo git plugin).

Is it helpful to you?

regards,

P. S.
4.4 is still using proxy-based design.

[Joe Ardent]

That is awesome; thank you, Piotr!

[Oleg Semyonov]

The main idea around ssh proxy is don't pass ssh keys or password to remote agents. It is ok, but in some cases I'd like to have a direct connection if I trust my remote agents. Maybe some configuration option...

I tried to build latest git plugin from the repository with this patch, but Bamboo 4.4 didn't like it (with an exception caught).

At the same time I see that some remote agent git caching is added. I hope it will solve the problem being published in the next release of plugin (and apparently it will require Bamboo 5.x if I am right looking at plugin source).

[PiotrA]

Can you post the exception here? I'm curious why Bamboo 4.4 didn't worked with the patched git plugin.

ad 'remote agent git caching' - it doesn't solve the problem of accessing remote ssh'ish repositories directly.

[Oleg Semyonov]

Sorry, I've removed the plugin and have no time yet to rebuild and install it. Besides, it is a public OpenSource project server accessible 24/7 worldwide, and I can't experiment a lot. Maybe I will try to use evaluation version later. It was something like "unable to check repositories", the same as after I returned back bundled version 2.5.6 and until server was restarted (it didn't help with 3.0.4-SNAPSHOT). Might be it was because I used ssh access instead of anonymous, so it was unable to update repositories w/o ssh key. But it even didn't want to show past build results.

As for remote git cache, it may (should) solve the primary customer problem - " the checkout times to be on the order of 20 minutes instead of thirty seconds".

[PiotrA]

Ah, I see. That make more sense now.

Thanks for taking time to explain the things - and if you ever happen to test that patched plugin against evaluation version, please don't hesitate to post the exception/errors here. I'm keen to review them - maybe it's something trivial to fix.

regards!

[Oleg Semyonov]

Righ now I am more concerned with this error, and not fetch time:

Push command error: Warning: Permanently added '[127.0.0.1]:60287' (RSA) to the list of known hosts.
error: unpack failed: eof before pack header was fully read
To ssh://d09ed13d-78e9-44c2-bc33-359a5f1f189f@127.0.0.1:60287/project.git
 ! [remote rejected] next -> next (n/a (unpacker error))
error: failed to push some refs to 'ssh://d09ed13d-78e9-44c2-bc33-359a5f1f189f@127.0.0.1:60287/project.git'

It is on original 4.4 and bundled git plugin (ssh keypair used for auth, gitolite on the other side). And it fails with 50% probability. I can restart it and it succeeds. This really makes me go nuts... And that's why I need to fix it somehow.

[Oleg Semyonov]

Now building plugin on linux (was on Windows). Meanwhile you may want to fix the plugin README file, the SDK download page is moved and link is wrong now.

[Oleg Semyonov]

I installed just built UNPATCHED plugin to the local copy of Bamboo to play. It shows past builds w/o problem, but it can't start new builds, and here is the error message:

Plan 'XXXX-YYYY0' could not be started. Exception: java.lang.NoSuchMethodError: com.atlassian.utils.process.ExternalProcessBuilder.idleTimeout(J)Lcom/atlassian/utils/process/ExternalProcessBuilder;
(com.google.common.util.concurrent.ExecutionError : java.lang.NoSuchMethodError: com.atlassian.utils.process.ExternalProcessBuilder.idleTimeout(J)Lcom/atlassian/utils/process/ExternalProcessBuilder;)

[Oleg Semyonov]

Bamboo: 4.4.0 build 3501.

Plugin: latest master https://bitbucket.org/atlassian/bamboo-git-plugin/commits/0f0e0834614fe4b2df8d89244d2fd0d7b2e1ecd0

After build I have two plugin jar files:

1773702 2013-02-27 12:41 atlassian-bamboo-plugin-git-3.0.2-SNAPSHOT.jar
1554084 2013-02-27 12:41 original-atlassian-bamboo-plugin-git-3.0.2-SNAPSHOT.jar

I used the 1st one (according to README and comparing 2.5.6 and 3.0.2 sizes).

[PiotrA]

Mhm... please remind me - which Bamboo version are you running, and which revision of bamboo-git-plugin have you built? reference: https://bitbucket.org/atlassian/bamboo-git-plugin/commits/all

[PiotrA]

Hm... I think you need to build&patch against "atlassian-bamboo-plugin-git-2.5.6-snapshot" tag, not the latest master. So to be precise: https://bitbucket.org/atlassian/bamboo-git-plugin/src/0c0df45966041d81d184bbeb1aae2d29c51d8ead/pom.xml?at=atlassian-bamboo-plugin-git-2.5.6-SNAPSHOT

The master is incompatibile with 4.4.0 and the error message seems to complain about changes between 4.4.0 -> now.

Can you try this?

git checkout atlassian-bamboo-plugin-git-2.5.6-snapshot
build plugin (atlas-mvn package ?)
install it into test 4.4.0 instance

[Oleg Semyonov]

Yes, I will do that. But I built master not because I wanted to get rid of ssh proxy. My problem is listed above - it can't push merged branches for unknown reason with 50% probability. I expected that it may be fixed somehow in 3.x plugin.

[Oleg Semyonov]

Reporting: yes, your fix works. It was tested using Bamboo 4.4.0 and git plugin 2.5.6 patched similar to your fix.

To make it working one has to install ssh keys on Bamboo server (it uses the plugin to poll repositories for changes), and on every agent (to fetch sources and optionally push merged branches). And of course, allow these agents (keys) to read/write on the gitolite side. I did not change the auth method, it still is "uploaded private key", but it obviously is not used in this case.

NOTE: you should put ssh keys on agents to those accounts under which you run bamboo agents. Say, if you run Bamboo on Windows using Local System account, it won't find the key put to an user account.

Now I have instant fetch and instant push, no more problems with timeouts or empty headers... So the ssh proxy is seriously faulty, it doesn't work in two installations for me even if one of them uses all machines in one LAN (no Internet delays).

Unfortunately, I can't install this on production site - it requires all agents to run the same (patched) plugin, and not all agent machines are trusted to do so.

So please pay attention to the ssh proxy. While the idea is good, the implementation is bad. And I'd like to have an option to bypass the proxy channel, and allow Bamboo server to pass passwords/keys to selected agents using encrypted channel, then allow them to authenticate to gitolite directly, removing the keys/password afterwards (or not storing them to disk at all). Currently I provide 4 remote agents on 4 operating systems to our OpenSource project (www.openpilot.org), and git fetch/push is just one trouble I have. Too many builds fail due to this, or fetch is never ending...

[PiotrA]

Hm... I must admit that to some extend I agree with you, and from some point I have doubts as to what is possible in a reasonable time. And whether the result would be comfortable with majority of Bamboo customers. I don't know.

Anyway, you have points with adding options for different authentication strategy in Bamboo (like bypassing the proxy channel). Can I ask you to raise Bamboo issues in https://jira.atlassian.com for:
* adding an option to bypass the proxy channel
* adding an option to pass passwords/keys to selected agents to allow them to auth to gitolite directly

Of course if you want I can do that for you, but if you report them directly I'm sure they would be more precise and would have more impact when reviewed by PM - after all they would be raised by a real customer instead of Atlassian's developer.

Does it sound OK to you?

regards,

[Oleg Semyonov]

https://jira.atlassian.com/browse/BAM-12925