HELP: Build plan could not be started

[Eazie Enibe]

I am having the below error:

"ulo-ui-v2 - ulo-ui : Plan 'ULUI-ULOUI' could not be started. Exception: com.atlassian.bamboo.plugins.git.GitCommandException: command 'C:\Program Files\Git\cmd\git.exe' ls-remote ssh://c4d11824-8068-4af9-86f8-63983770dd14@127.0.0.1:55734/uw/admin-portal-v2.git failed with code 128. Working directory was [D:\Bamboo\temp]., stderr:
Warning: Permanently added '[127.0.0.1]:55734' (RSA) to the list of known hosts.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

This error may have occurred due to an authentication issue with Bitbucket Server. Save the repository configuration to refresh the authentication details.(11 Apr 2018, 6:08:53 AM)"

I have tried everything i know to resolve this exception with no success, kindly help to resolve this.

I am using Bamboo 6.3.1 Server.

Regards,

Eazie

[Foong]

Do you mean that you have tried re-saving the Repository settings at Bamboo Plan Configuration > Repositories and/or Bamboo Administration > Build Resources > Linked repositories ?

Is there any error when those repository settings are edited?

[Eazie Enibe]

Yes I have tried multiple times, it still same issue.

My build plans all of a sudden started failing at bitbucket repository checkout task, so I dropped the Linked repository and re-created and since then I have been having this issue. Not sure if its from bamboo or bitbucket, perhaps both.

Please need some urgent assist with this, it kind-off crippled our CICD. We are back to manual build and deploy.

[Foong]

Since the Linked repository is successfully created, there shouldn't be any permission error that cause the plan fail to start build. Can you check again if there is any error when those repository settings are edited at Bamboo Administration > Build Resources > Linked repositories ?

If no error, can you confirm if the latest error is the plan fail to start build or the checkout task is failing?

If it is the checkout task, is it using remote agent?

[Eazie Enibe]

I checked at all my Linked Repositories, I noticed that it gives the error in the image attached below.

I believe there is a problem with the linked repositories, probably issue with authentication with bitbucket.

I am not using a remote agent

[Eazie Enibe]

Hi, my problem is still not solved, could you assist further.

Thanks

[Foong]

What happen after clicking on the Save repository button?

Does editing the repository still show the error? Does the build works fine after saving it?

[Eazie Enibe]

Yes it still shows the same error in the image posted after clicking the Save repository button.

The job fails afterwards when I ignore the error and proceed with the build, it fails because its not able to checkout from bitbucket.

[Foong]

Check bamboo-home/logs/atlassian-bamboo.log when clicking on the Save button to look for the error that make it fail

[Foong]

Yes it still shows the same error in the image posted after clicking the Save repository button.

Is there any error in the UI when clicking on the Save button? Does the error only happens during the edit view and the Save button does not throw any error?

[Eazie Enibe]

See Log content below:

2018-04-13 22:43:09,184 ERROR [6-DelayedChangeDetectionThread:pool-12-thread-11] [ChainExecutionManagerImpl] Errors getting changes for ULUI-ULOUI-147
com.atlassian.bamboo.plugins.stash.repository.StashRepositoryException: com.atlassian.bamboo.plugins.git.GitCommandException: command 'C:\Program Files\Git\cmd\git.exe' ls-remote ssh://95b5b791-03c5-465f-ad75-378b2fe588db@127.0.0.1:55734/uw/admin-portal-v2.git failed with code 128. Working directory was [D:\Bamboo\temp]., stderr:
Authenticating remote session failed
stdout:
Warning: Permanently added '[127.0.0.1]:55734' (RSA) to the list of known hosts.
BAMBOO-SSH-PROXY: [SSH proxy tried to connect to [git@/192.227.65.100:7999]: and failed due to the following error: [Authenticating remote session failed]: No more authentication methods available]
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
at com.atlassian.bamboo.plugins.stash.v2.BitbucketServerChangeDetector.collectChangesSinceRevision(BitbucketServerChangeDetector.java:30)
at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectionChangesWithRetry(DefaultChangeDetectionManager.java:481)
at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.lambda$createBuildRepositoryChanges$1(DefaultChangeDetectionManager.java:382)
at com.atlassian.bamboo.variable.CustomVariableContextImpl.withVariableSubstitutor(CustomVariableContextImpl.java:185)
at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectChangesSinceLastBuildInternal(DefaultChangeDetectionManager.java:344)
at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectChangesSinceRevisions(DefaultChangeDetectionManager.java:290)
at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectChangesSinceRevisions(DefaultChangeDetectionManager.java:199)
at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectAllChangesSinceLastBuild(DefaultChangeDetectionManager.java:152)
at com.atlassian.bamboo.v2.trigger.ManualBuildDetectionAction.performDelayedChangeDetection(ManualBuildDetectionAction.java:154)
at com.atlassian.bamboo.chains.ChainExecutionManagerImpl$1.getChainState(ChainExecutionManagerImpl.java:240)
at com.atlassian.bamboo.chains.ChainExecutionManagerImpl.tryStartChainState(ChainExecutionManagerImpl.java:315)
at com.atlassian.bamboo.chains.ChainExecutionManagerImpl.delayedStart(ChainExecutionManagerImpl.java:232)
at sun.reflect.GeneratedMethodAccessor2315.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy198.delayedStart(Unknown Source)
at sun.reflect.GeneratedMethodAccessor2315.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy198.delayedStart(Unknown Source)
at com.atlassian.bamboo.plan.PlanExecutionManagerImpl$1$1$1.lambda$call$0(PlanExecutionManagerImpl.java:374)
at com.atlassian.bamboo.util.CacheAwareness$3.call(CacheAwareness.java:136)
at com.atlassian.bamboo.util.CacheAwareness$3.call(CacheAwareness.java:133)
at com.atlassian.bamboo.util.CacheAwareness.withValuesOlderThanTimestampReloaded(CacheAwareness.java:162)
at com.atlassian.bamboo.util.CacheAwareness.withValuesOlderThanTimestampReloaded(CacheAwareness.java:133)
at com.atlassian.bamboo.util.CacheAwareness.withValuesOlderThanTimestampReloaded(CacheAwareness.java:187)
at com.atlassian.bamboo.plan.PlanExecutionManagerImpl$1$1$1.call(PlanExecutionManagerImpl.java:373)
at com.atlassian.bamboo.plan.PlanExecutionManagerImpl$1$1$1.call(PlanExecutionManagerImpl.java:366)
at io.atlassian.util.concurrent.ManagedLocks$ManagedLockImpl.withLock(ManagedLocks.java:293)
at com.atlassian.bamboo.plan.PlanExecutionLockServiceImpl.lock(PlanExecutionLockServiceImpl.java:75)
at com.atlassian.bamboo.plan.PlanExecutionManagerImpl.doWithProcessLock(PlanExecutionManagerImpl.java:655)
at com.atlassian.bamboo.plan.PlanExecutionManagerImpl.access$400(PlanExecutionManagerImpl.java:130)
at com.atlassian.bamboo.plan.PlanExecutionManagerImpl$1$1.run(PlanExecutionManagerImpl.java:362)
at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:48)
at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:26)
at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:17)
at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:41)
at java.lang.Thread.run(Unknown Source)

[Foong]

This is the error when Bamboo checking for new changes.

Is there any error when you click on the Save button?

[Eazie Enibe]

By new change, are you referring to change in configuration at bitbucket? If so none.

[Foong]

No. It is the error when Bamboo checking for new commits in Bitbucket. The Git command fail probably due to SSH key not working.

Is there any error when you click on the Save button in Bamboo Linked repository?

[Eazie Enibe]

When I click on click on "Save Button" see image below:

But when I select same repository after this, it shows me the image below:

If this is because of SSH please can you advise on how to get bamboo to use http(s) 

[Foong]

It is not possible to use HTTP(s) when creating a Bitbucket Server / Stash repository type in Bamboo. You can only create a Git repository type and configure the URL manually.

What is your Bamboo and Bitbucket Server version?

Is SSH access key enabled at Bitbucket Administration > Settings > Server settings > SSH access ? Is the SSH base URL correctly configured?

[Eazie Enibe]

My Bamboo version is 6.3.1

Below is a a snapshot of my SSH configuration on bitbucket:

The SSH base url was not set so it uses the default.

[Foong]

1. Check what is/are the existing Access Key at Bitbucket Server > Repository settings > Security > Access Key
2. Click on the Save button in Bamboo Linked repository
3. Check if there is a new Access Key created at Bitbucket Server > Repository settings > Security > Access Key
4. Check if there is error in Bamboo Linked repository
5. Confirm if the Access key still exist at Bitbucket Server > Repository settings > Security > Access Key

What is the version of Bitbucket Server?

When the Save button is clicked in Bamboo Linked repository, does it show the following INFO in atlassian-bamboo.log?

2018-04-13 23:54:47,579 INFO [http-nio-8085-exec-21] [BitbucketServerServerConfigurator] Public key was added to Bitbucket Server (Bamboo user: foong, application: Bitbucket [http://localhost:7990], location: repository TST/stash): ssh-rsa xxxxx http://localhost:8085

[Eazie Enibe]

My Bitbucket version is 5.7.0

I followed the steps, I had 6 keys at the Access Keys:

When I click on Save button, there are no new keys added all six still there as before, the I checked the logs and see this:

"2018-03-28 11:40:04,264 INFO [http-apr-8085-exec-21] [AuthenticatorImpl] Authenticated app 'Bitbucket:0291115601' as user 'null' successfully
2018-03-28 11:40:04,264 INFO [http-apr-8085-exec-21] [AccessLogFilter] 192.227.65.100 GET http://192.227.65.100:8085/rest/remote-event/1/status 169522kb"

Not the like the log entry above

[Eazie Enibe]

Hi Foong,

Is it possible for us to have a Skype call on Monday 16th April, I could share my screen and go through my current bamboo setup with you to help me troubleshoot on the root cause.

Please advise on the best time that suits you, I will be available.

Regards,

Eazie.

[Foong]

1. Add debug logging for the Classpath org.apache.http.headers at Bamboo Administration > System > Log settings
2. Enable debug logging at Bitbucket Server Administration > Support > Logging and profiling
3. Confirm that there are still error when editing Bamboo Linked repository
4. Click on the Save button in Bamboo Linked repository
5. Check Bamboo atlassian-bamboo.log for the following example logs: 

   2018-04-20 22:25:58,356 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 >> POST /rest/keys/1.0/projects/TST/repos/stashreponame/ssh HTTP/1.1
   2018-04-20 22:25:58,356 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 >> Content-Type: application/json
   2018-04-20 22:25:58,356 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 >> X-Atlassian-Token: no-check
   2018-04-20 22:25:58,356 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 >> Authorization: OAuth oauth_token="NfEL1TldqQkX5HDXQLLfgMev2rtxU8PN", oauth_consumer_key="Bamboo%3A0415430879", oauth_signature_method="RSA-SHA1", oauth_timestamp="1524234358", oauth_nonce="26592300712664", oauth_version="1.0", oauth_signature="wDlkHdkmBDWeIC9s%2FB01P%2FIX8Bq9M86fPFcLpdzf3x0gusrkUEtZrKIf7MQlBGR2n2MiGRp9iW4KaS884CznNKfe%2FmpA6M8Vb5A80bhwF0LBPtLkSPDcdz4RrTOkfj1R7a7a8S7ZZ4nFEHlFZ2BeZscSq%2BgwKN8w6Gbhd7zMdjplhvL0eeG8Nk1yMHGtJ7vuftZrOzy8D9HoKxF7cKLg9O1vuIEO3gp5dKG%2BppZ23TTLYFovA4YWNGiXrrkmEmR96mgsoHBrAGbrH%2FIY08fl%2BLPsXQl3p4YnaPPKDnP%2BNNZmIH7DLuLLvuOrdAiah2mMlnTsUs4FwvhsyqqXgt%2FGIg%3D%3D"
   2018-04-20 22:25:58,356 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 >> Content-Length: 523
   2018-04-20 22:25:58,356 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 >> Host: localhost:7990
   2018-04-20 22:25:58,356 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 >> Connection: Keep-Alive
   2018-04-20 22:25:58,356 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 >> User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_73)
   2018-04-20 22:25:58,356 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 >> Accept-Encoding: gzip,deflate
   2018-04-20 22:25:58,378 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 << HTTP/1.1 201 
   2018-04-20 22:25:58,378 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 << X-AREQUESTID: @21YD6Mx1345x386x0
   2018-04-20 22:25:58,378 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 << X-ASEN: SEN-L4201929
   2018-04-20 22:25:58,378 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 << X-AUSERID: 1
   2018-04-20 22:25:58,378 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 << X-AUSERNAME: kahloun.foong
   2018-04-20 22:25:58,378 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 << Location: http://localhost:7990/rest/keys/1.0/projects/TST/repos/stashreponame/ssh/0
   2018-04-20 22:25:58,378 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 << Cache-Control: no-cache, no-transform
   2018-04-20 22:25:58,378 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 << Vary: X-AUSERNAME,Accept-Encoding
   2018-04-20 22:25:58,378 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 << Content-Encoding: gzip
   2018-04-20 22:25:58,378 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 << Transfer-Encoding: chunked
   2018-04-20 22:25:58,378 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 << X-Content-Type-Options: nosniff
   2018-04-20 22:25:58,378 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 << Content-Type: application/json;charset=UTF-8
   2018-04-20 22:25:58,378 DEBUG [http-nio-8085-exec-1] [headers] http-outgoing-75 << Date: Fri, 20 Apr 2018 14:25:58 GMT

   From the example, Bamboo is sending a POST rest call to Bitbucket Server for the URL http://localhost:7990/rest/keys/1.0/projects/<projectkey>/repos/<reponame>/ssh and get the reply that its successful (code 201)

6. Check Bitbucket Server log atlassian-bitbucket.log for the following example log or if there is any error: 

   2018-04-20 22:25:58,369 DEBUG [http-nio-7990-exec-6] kahloun.foong @21YD6Mx1345x386x0 127.0.0.1 "POST /rest/keys/1.0/projects/TST/repos/stashreponame/ssh HTTP/1.1" c.a.b.i.key.ssh.DefaultSshKeyService kah loun added an ssh key for 456d0a7972073140
   2018-04-20 22:25:58,372 DEBUG [http-nio-7990-exec-6] kahloun.foong @21YD6Mx1345x386x0 127.0.0.1 "POST /rest/keys/1.0/projects/TST/repos/stashreponame/ssh HTTP/1.1" c.a.b.i.k.s.DefaultSshAccessKeyService Service user 456d0a7972073140 added ssh access key for repository "TST/stashreponame[15]"

[Hema]

Hi @Foong ,

I am also getting same issue mentioned in above image, and when i clicked on save repository then another error comes up, New access key required but could not added on bitbucket server. Warning : Permanently added in ip address rsa in known hosts. Could not read from remote repository. I am facing this issue after logon on single sign on.
Here, I am not to able to create plan or repo on bamboo ui also.

Another issue on other bamboo server, bamboo scans for bamboo specs stored on bitbucket repository and it is failing for plan repository with Request to stash failed. Return with empty reaponse body 401. If I removed plan repo from bamboo java spec then it is workimg fine. I can publish plan witj plan repo from local also but not from bitbucket stored repo.

Its urgent, could you please help me

[Hema]

Hi @Foong ,

I am also getting same issue mentioned in above image, and when i clicked on save repository then another error comes up, New access key required but could not added on bitbucket server. Warning : Permanently added in ip address rsa in known hosts. Could not read from remote repository. I am facing this issue after logon on single sign on.
Here, I am not to able to create plan or repo on bamboo ui also.

Another issue on other bamboo server, bamboo scans for bamboo specs stored on bitbucket repository and it is failing for plan repository with Request to stash failed. Return with empty reaponse body 401. If I removed plan repo from bamboo java spec then it is workimg fine. I can publish plan witj plan repo from local also but not from bitbucket stored repo.

Its urgent, could you please help me

[Eazie Enibe]

Hi Hema,

You may want to check the version of bamboo against the bitbucket version you are using, I remembered resolving mine previously by rolling back my version of bamboo. This issue started  after a bamboo update and it didn't occur to me that was an issue. A compatibility issue which I hope Atlassian could have caught.

Try rolling back one of 2 versions back depending on whether it was bamboo or bitbucket you updated.

Hope this helps.

Regards

Eazie.

[Hema]

Hi @Eazie Enibe 

Thanks for reply. It was server issue.
Now I am facing another issue on prod bamboo instance. Plan repository using bitbucketserver repository from bamboo spec is not creating. It is giving error like Request to stash failed. Return with 401. Response body is empty. When i saw log for it it is failinh at OAuthApplinkexecutor.
When I am passing ssh keys and app link for creating plan repo. Same os working from bamboo ui but not from bamboo spec.
When I tried to create plan repository using GitRepository from bamboo spec then it is working but using bitbucket server repo, it is failing.
Please help me.