Bitbucket Linux Docker Runner — Cache upload fails with S3 ReadTimeoutException

Question

Hello Atlassian Community,

We are experiencing a persistent cache-upload failure with Bitbucket Pipelines running on our self-hosted Linux Docker runner.

Environment

* Runner version: 6.1.2
* Repository: mirocksdevops/hostinghome-ci-cache-benchmark
* Affected pipeline: #3
* Cache archive: approximately 226.8 MiB
* Compression completed in approximately 41 seconds

Issue

The pipeline successfully completed dependency installation, security verification and smoke tests. During cache upload, the runner encountered an Amazon S3 ReadTimeoutException. No cache archive was saved, preventing restoration in subsequent pipelines.

Our investigation found no explicit DNS, TLS-handshake, proxy or authentication errors. The precise cause remains unidentified.

Request

Please investigate the cache-storage upload timeout and confirm whether this is related to Bitbucket's storage endpoint, network connectivity, upload timeout behavior or a known runner issue.

Please also advise whether a supported configuration change or runner-version fix is available.

We can provide sanitized runner logs and timestamps for pipeline #3.

Regards,

Mirocks Infotech

Mihai_LeanZero · Answer

Hi @hansiemithun,

There's no runner fix to wait for yet. The changelog for 6.1.3 and 7.0.0 has nothing on cache, S3 or upload timeouts.
https://product-downloads.atlassian.com/software/bitbucket/pipelines/CHANGELOG.md

Trevan and Arek covered the timeout and BCLOUD-23971. For diagnostics, the runner takes its log level from BITBUCKET_PIPELINES_RUNNER_LOG_LEVEL, so -e BITBUCKET_PIPELINES_RUNNER_LOG_LEVEL=DEBUG on docker run gives one test pipeline more detail. I haven't checked what DEBUG prints for a cache upload. Support doesn't cover custom logging.
https://support.atlassian.com/bitbucket-cloud/docs/configure-logging-in-self-hosted-runners/
Where runner.log lives is under Review runner logs here:
https://support.atlassian.com/bitbucket-cloud/kb/troubleshooting-runners/

My own suggestion, not from a doc, is to time a 227 MiB upload from the runner host to a test bucket in us-east-1 and to one in your region. If only us-east-1 crawls, it's probably the cross-region path.

With Premium runners, the cache can go to a bucket in the runner's own region:

- step:
    oidc: true
    caches:
      - your-cache
    script:
      - <your build commands>
    runtime:
      self-hosted:
        storage:
          aws:
            bucket: your-bucket-name
            region: your-aws-region
            oidc-role: my-role-arn
    runs-on:
      - linux

https://support.atlassian.com/bitbucket-cloud/docs/configure-your-runner-in-bitbucket-pipelines-yml/
The role needs Bitbucket set up as an OIDC provider in AWS first:
https://support.atlassian.com/bitbucket-cloud/docs/deploy-on-aws-using-bitbucket-pipelines-openid-connect/

Without Premium runners, smaller split caches and a watch on BCLOUD-23971 are what's left.

hansiemithun · Answer

Hi @Trevan Householder_Isos-Tech-Consulting_,

Thank you for the detailed clarification and for confirming that S3_READ_TIMEOUT_SECONDS is no longer applicable to Runner 6.1.2.

While this explains the timeout configuration behavior, our primary concern remains unresolved: improving pipeline performance and ensuring reliable cache uploads on self-hosted Linux Docker runners.

Our benchmark pipeline successfully completed its dependency installation, security checks, and smoke tests. However, uploading approximately 227 MiB of cache took roughly 10 minutes before failing with ReadTimeoutException, leaving subsequent builds without a reusable cache.

We would appreciate further clarification on the following:

* Root cause and diagnostics: Is there a recommended way to distinguish slow cross-region S3 throughput from runner-side transfer issues, network inspection, or service-side throttling? Are there additional runner logs or metrics we should capture?
* Upload performance: Does Atlassian have a documented throughput baseline or expected upload duration for cache archives of approximately 200–250 MiB?
* Cache optimization: Are there recommended compression, archive-size, or transfer optimizations for Linux Docker Runner 6.1.x? Is compression behavior configurable?
* Regional storage: Apart from Premium runners with custom storage, is Atlassian considering a solution that improves cache-upload performance for standard self-hosted runners outside AWS us-east-1?
* BCLOUD-23971: Is there an estimated resolution timeline or planned improvement? The issue appears directly relevant to our observations.
* Alternative caching: For teams prioritizing faster CI/CD with minimal additional infrastructure costs, what is Atlassian's recommended approach when the overhead of uploading and restoring caches exceeds the time saved by caching dependencies?

We will also verify the runner's network location, check whether an inspecting proxy is involved, and benchmark smaller cache archives.

Our objective is to achieve predictable, reliable, and faster pipeline execution, rather than merely extending the upload timeout or rerunning failed pipelines.

Thank you again for your assistance.

Regards,

Mirocks Infotech

Trevan Householder_Isos-Tech-Consulting_ · Answer

@hansiemithun, thanks for the thorough digging, your reading of the KB is spot on. @Arkadiusz Wroblewski was right to ask where the runner lives. S3_READ_TIMEOUT_SECONDS only mattered for 3.0.0 to 3.16.x. The runner changelog says 3.17.0 removed the need for it, and nothing in 6.1.x touches cache timeouts, so there's no setting to tune on 6.1.2.

Roughly 10 minutes for 227 MiB points to slow throughput to the default S3 bucket in us-east-1. That's tracked in BCLOUD-23971. Its workarounds are to rerun, host the runner closer to us-east-1, or use v5 custom storage with a bucket in your runner's region (that's a Premium runners feature).

I'd also try trimming the cache. Is the runner outside AWS us-east-1, or behind an inspecting proxy?