timeout error when accesing stash from jenkins

[Aditya Rajan Agrawal]

EnvInject] - Loading node environment variables.
Building remotely on jxlprdbld01 (reports snapshot admin compile junit distributions) in workspace /opt/reference/deployments/integration/QAP/availity-web
 > git rev-parse --is-inside-work-tree
Fetching changes from the remote Git repository
 > git config remote.integration.url ssh://git@git.xxxxx.com:7xxx/ss/xxxxxx-web.git
Fetching upstream changes from ssh://git@git.xxxxx.com:7xxx/ss/xxxxxx-web.git
 > git --version
 > git fetch --tags --progress ssh://git@git.xxxxx.com:7xxx/ss/xxxxxx-web.git +refs/heads/*:refs/remotes/integration/*
ERROR: Timeout after 10 minutes
FATAL: Failed to fetch from ssh://git@git.xxxxx.com:7xxx/ss/xxxxxx-web.git
hudson.plugins.git.GitException: Failed to fetch from ssh://git@git.xxxxx.com:7xxx/ss/xxxxxx-web.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:622)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:854)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:879)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:615)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:524)
	at hudson.model.Run.execute(Run.java:1710)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:234)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress ssh://git@git.xxxxx.com:7xxx/ss/xxxxxx-web.git +refs/heads/*:refs/remotes/integration/*" returned status code 143:
stdout: 
stderr: 
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1325)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1186)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(CliGitAPIImpl.java:87)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:257)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)

[Jeff Thomas]

Hi,

The timeout appears to be coming from the Git command below:

git fetch --tags --progress ssh://git@git.xxxxx.com:7xxx/ss/xxxxxx-web.git +refs/heads/*:refs/remotes/integration/*

Could you try running that command manually on the server to see how long it's taking? You can find the exact time with:

time git fetch --tags --progress ssh://git@git.xxxxx.com:7xxx/ss/xxxxxx-web.git +refs/heads/*:refs/remotes/integration/*

By default, the Git client plugin for Jenkins has a 10 minute timeout set for Git operations. When Git Plugin 2.3 is released, this can be configured in the settings. Until then, it can be increased by changing the system property:

java -Dorg.jenkinsci.plugins.gitclient.Git.timeOut=120 -jar jenkins.war

[Gattupalli Rahul Chakradhar]

Hi Jeff, Can you please let me know where do we change the system property and run the script that you have given?

[Gontran SEGUE]

Hi Aditya,

copy your ssh keys inside $HOME/.ssh

Hope it has helped.

Gontran. S

[Sabareesh]

Is there any solution that helped you to by-pass this error? If so please help me.

[Phoung-Dinh Tieu]

Sadly, I have run across this error, as well.  No resolution?

I did time the command, and it only took 3.5 minutes.  Also, I increased the timeout some time ago, so I am now consistently seeing this with a 60 minute timeout.  Any other ideas what may be causing this?

One note, the jenkins job does build several configurations at once (matrix job).  Could this be affecting the ability to fetch, and why I don't see it when I clone directly on the machine?

[Phoung-Dinh Tieu]

Aha!  I finally realized that the jenkins service wasn't running as who I thought it was.  E.g. I missed a step to change the login of the jenkins service to the expected user whose ssh keys had been added to Stash.

So, the lesson I learned is, if Jenkins times out trying to connect to Stash:

1. Verify that Jenkins is running as the user you expect by checking Jenkin's node system information (user.name).
2. Verify that ssh is properly setup for that user, on the node, by logging in and performing the clone.  You might need to add the ssh key to Stash, or fix the ssh setup.
3. Time the above clone operation to ensure the timeout is set appropriately.