How to share pulled source code with several remote agent

[yogev ch]

Hi,
In an older Bamboo version there was a local agent which was able to run several "workers" at the same time.
These workers shared resources, one of them is the storage.
So, if I had a "check out" stage in the beginning of a plan that pulled source code from git, any local agent worker was able to run the following stages and use the same source code.

I have a "heavy" source code checkout which takes a lot of time. For that reason I wanted to run it once in the beginning in the 1'st stage.

In today's Bamboo version there aren't local agents anymore.

I can still run a remote agent on the local bamboo device, for example, but that's one "worker" only.

If I start another remote agent, even if it's on the same device as the 1'st one, the don't share the pulled source code because each has a different `bamboo home`.

Also, I don't know which remote agent will run each stage.
So, for my plan which has several stages, a solution might be to add a "check out" job for each for the stages, which is quite cumbersome.

Please advise with a solution to this issue,
Thank you

Plan example:

Stage: checkout

  Job: checkout

Stage: tests

  Job: build1 + test1

  Job: build2 + test2

[Shashank Kumar]

Hello yogev ch,

Welcome to Atlassian community.

You can read Why it is not advisable to share a common build working directory for Bamboo Remote agents to understand why remote agents don't share common working directories.

According to Bamboo architecture Stage runs sequentially which means Stage 2 will only be called when Stage 1 is completed. if your case Stage tests will be run only when Stage checkout is completed.

I can understand the problem here let's say Job : checkout was run by remote agent 1 but Job : build  + test 1 can be run by remote agent 2 and Job : build2 + test 2 can be run by remote agent 3 and they will fails as they won't have access to checkout which contains the test files.

If all your Jobs are dependent upon each other then probably the better idea is to move everything under one stage and One Job which will be executed by the agent at one go, the final state would look like

Stage: checkout + test

  Job: 

   Task checkout

   Task build1 + test1

   Task build2 + test2

This is one way to achieve what you are looking for.

Regards,

Shashank kumar

**please don't forget to Accept the answer if your query was answered**

[yogev ch]

Thank you for your reply.

I tried to simplify my plan.
My actual plan has a lot of stages.

- Stage checkout
  - Job checkout
    - Task checkout
    - Task some commands and scripts
- Stage tests
  - Job test1
    - Task build1
    - Task test1
    - Task test2
    - ...
  - Job test2
    - Task build2
    - Task test2_1
    - ...
Stage Build product 1
  - Job product 1
    - Task build
Stage Build product 2
  - Job product 2
    - Task build
...

 As you can see, trying to move the task checkout will cause this plan to look worse :/ 

[Shashank Kumar]

Hello Yogev,

To confirm your approach on the local agents, to understand the approach properly.

1. In the first stage you were checking out the code into during the Job run into the build working directory of that particular Job or you used a shared build working directory which was accesible for other agents as well ?

2. With the remote agents as I had earlier explained you can't have shared build working directory for multiple agents. Probably you can use dedicated agents for this particular plan so that the same agent runs all the Jobs from checkout until test and can access the source code checkout from the other Job.

3. Have you enabled the caching on the linked repository and see if it makes any difference during the checkout tasks. 

4. Logically on remote agents if anyone of your tasks are depended upon the the output of another task, both of them should be part of the same Job.

Regards,

Shashank Kumar

[yogev ch]

Hi,

1. I am using the regular check out task. I'm pulling the code into the job's directory.

Now, because these are 2 workers of the local agent, and they share the file system, on the next job I could use the directory where I pulled the code to.

That is quite a hack we can't use anymore.

2. Right. That's my problem. I can't run 2 jobs of the same stage in parallel and I have to have a dedicated agent.

3. I didn't. We want each time to pull everything from scratch to make sure it's a clean build. Also, even if I enable caching, and the next step is pull in each job, it means I'll have the git repo multiplied by the number of jobs I have, right? My repos are quite big so I don't really like that.

4. Mm ok. By that you mean that I need to checkout each job, right? That returns me to the issue that our code takes a lot of time to pull, and it's big.

Long time to pull > you day use cache > I say it's big and multiplied by the number of jobs that have a checkout task.

Thank you

[Shashank Kumar]

Hi Yogev,

With the current Design of the remote agent and your requirement of just checking out the code once , the best possible solution I can think of is that you'll have to move all your Jobs and tasks under one stage.

I would let anyone else in the community opine if they have any alternate solution.

Regards,

Shashank Kumar

[yogev ch]

.