Cloning git repo from self-hosted runner

[kristoffer]

Hello!

I've successfully set up a self-hosted Linux runner to serve a Bitbucket Cloud pipeline.

On each build the git repo is cloned from Bitbucket, additionally some git submodules are pulled as part of the build script.

The total amount of data cloned from git in these operations weighs in at 2.4GB, and the number of builds per day, when fully operational, should be in the double digits. That's a lot of data to shuffle around the network.

Questions:

1. Will these git clone operations count towards some data rate limit and make things inaccessible/slower if exceeded?

2. Is there a recommended way to have a local git proxy of sorts that just keep up to date with Bitbucket Cloud and then let the self-hosted runner clone from that (local) location instead?

3. Is there any way to have submodules pulled as part of the built in cloning process?

/K

[Theodora Boudale · Accepted solution]

Hi Kristoffer,

1. Will these git clone operations count towards some data rate limit and make things inaccessible/slower if exceeded?

The rate limits we have in place have to do with the number of requests, and not with the size of the data being transferred.

You can see these limits here:

• https://support.atlassian.com/bitbucket-cloud/docs/api-request-limits/

Clones via HTTPS count towards Git web (HTTPS://) requests - 60,000 requests per hour
Clones via SSH count towards SSH requests - 30,000 requests per hour

If the submodules are private repos and you use authentication to clone them, the calls are measured against the user ID of the user you use to clone.

2. Is there a recommended way to have a local git proxy of sorts that just keep up to date with Bitbucket Cloud and then let the self-hosted runner clone from that (local) location instead?

We don't have any official documentation on such a configuration.

3. Is there any way to have submodules pulled as part of the built in cloning process?

I'm afraid that this is not possible at the moment.

We have a feature request in our issue tracker:

• https://jira.atlassian.com/browse/BCLOUD-12853

If you'd be interested in that feature, I would suggest adding your vote (by selecting the Vote for this issue link) as the number of votes helps the development team and product managers better understand the demand for new features. You are more than welcome to leave any feedback, and you can also add yourself as a watcher (by selecting the Start watching this issue link) if you'd like to get notified via email on updates.

Implementation of new features is done as per our policy here and any updates will be posted in the feature request.

In the meantime, submodules need to be initialized and updated with a command in your script.

Kind regards,
Theodora

[Timo Laru]

Is there way to disable automatic cloning altogether? I'd rather do all that from my build scripts, especially when there's need for more than one repo to be included in the build environment.

[Theodora Boudale]

Hi @Timo Laru,

Yes, this is possible. You can use in your bitbucket-pipelines.yml the following:

clone:
  enabled: false

If you have multiple steps and you want to disable cloning in all of them, you can place this at the beginning of the file, like below:

image: atlassian/default-image:3

clone:
  enabled: false

pipelines:
  default:
    - step:
        name: 'Build and Test'
        runs-on:
          - 'self.hosted'
          - 'linux'
        script:
          - echo "Your build and test goes here..."
    - step:
        name: 'Lint'
        runs-on:
          - 'self.hosted'
          - 'linux'
        script:
          - echo "Your linting goes here..."

Alternatively, if you want to disable cloning only for a specific step you can add this to a certain step:

image: atlassian/default-image:3

pipelines:
  default:
    - step:
        name: 'Build and Test'
        runs-on:
          - 'self.hosted'
          - 'linux'
        script:
          - echo "Your build and test goes here..."
    - step:
        name: 'Lint'
        runs-on:
          - 'self.hosted'
          - 'linux'
        clone:
          enabled: false
        script:
          - echo "Your linting goes here..."

Kind regards,
Theodora

[Timo Laru]

Thank you! That solved my problem :)

[Theodora Boudale]

That's good to hear, you are very welcome!