How to resolve artifact processing errors?

Question

In my pipeline a step fails with the message 'Error occurred whilst processing an artifact'. I see no other message that could be an indication of what the error was so how would i go about solving this?

This step is running on a self-hosted windows runner with version 3.10.0. I'm not surpassing the 1GB artifact size limit.

This is the step in my bitbucket-pipelines.yml:

- step: &BuildWindows
 name: Build Windows
 runs-on:
  - 'self.hosted'
  - 'windows'
 script:
  - python -u ./CICD/BuildWindows.py
 artifacts:
  #- N/A
  #- N/A
  - Qumulus/Qumulus_v*.exe
  - Qumulus/ReleaseNotes.html
  - "**.pdb"

Here's the output for the Build teardown step:

Searching for files matching artifact pattern Qumulus/Qumulus_v*.exe
Artifact pattern Qumulus/Qumulus_v*.exe matched 1 files with a total size of 7.6 MiB
Compressed files matching artifact pattern Qumulus/Qumulus_v*.exe to 7 MiB in 0 seconds
Uploading artifact of 7 MiB
Successfully uploaded artifact in 4 seconds

Searching for files matching artifact pattern Qumulus/ReleaseNotes.html
Artifact pattern Qumulus/ReleaseNotes.html matched 1 files with a total size of 2.4 KiB
Compressed files matching artifact pattern Qumulus/ReleaseNotes.html to 1.4 KiB in 0 seconds
Uploading artifact of 1.4 KiB
Successfully uploaded artifact in 1 seconds

Searching for files matching artifact pattern **.pdb
Artifact pattern **.pdb matched 50 files with a total size of 989.1 MiB
Compressed files matching artifact pattern **.pdb to 246.7 MiB in 31 seconds
Uploading artifact of 246.7 MiB
Searching for test report files in directories named [failsafe-reports, TestResults, surefire-reports, test-reports, test-results] down to a depth of 4
Finished scanning for test reports. Found 0 test report files.
Merged test suites, total number tests is 0, with 0 failures and 0 errors.

The runner.log is riddled with errors and exceptions, i havn't been able to get a stable runner for several months now so im not sure anymore if this is expected or not. Here's a link (i couldn't find the attachment feature): https://gist.github.com/MennoVink/435438a5ebef0e5c9fd0444ba42ec444

Patrik S · Answer

Hello @MennoVink ,

It's not possible to disable it as this is a platform-wise change. All the runners from version 3.0.0 onwards will use the new artifact and cache management that interacts directly with AWS S3.

Since December 2nd, steps running on outdated versions of the Runner (older than 3.0.0)  can no longer interact with the Pipelines file-storage infrastructure.

That being said, since you have confirmed there's no firewall blocking the connection, the timeout in the logs may be caused by a slow connection from the runner's host to the AWS S3 infrastructure.

There's a flag that can be configured in the runner's startup to increase the timeout (which by default is 30 seconds), which could help when using slow connections.

Could you try restarting your runner and providing this flag in the start command to confirm if the builds are more stable with an increased timeout?

* Remove the current runner and start it over providing the flag  -s3ReadTimeoutSeconds 60

.\start.ps1 -s3ReadTimeoutSeconds 60 <rest of the command>

After the runner is restarted with that flag, please run a few builds to monitor whether the issue you are currently facing has improved.

Thank you, @MennoVink !

Patrik S

MennoVink · Answer

Hi @Patrik S

Thank you for your answer. Unfortunately my runner is not operating behind a firewall. There is a Windows Defender in it's default configuration, but i assume if that was an issue there would be many more users with issues.

The other (smaller) artifacts do work. Is there a cutoff when the runner automatically switches between storage backends? Can i change this cutoff value somewhere?

Since my artifacts did work before i dont think i need the 'enhanced artifact and cache management'. Is there a way to disable it for my workspace?

Kind Regards,

Menno

Patrik S · Answer

Hello @MennoVink ,

thank you for reaching out to the Community!

The self-hosted runner's v3.0.0 and later introduced an enhanced artifact and cache management which interacts directly with Pipelines storage infrastructure in AWS S3.

The error message in the logs indicates a connection issue from your runner's host to AWS S3.

If your self-hosted runners are operating behind a firewall that filters connections based on IP addresses or URLs, please ensure that you unblock the following for both incoming and outgoing traffic when upgrading to version 3.0.0 or above:

If Using IP

An exhaustive list of IP addresses that traffic can route to AWS can be found by utilizing the following endpoint, filtering for records where the service equals S3, and specifying the us-east-1 and us-west-2 regions.

To facilitate the filtering process, the following command can be used to retrieve the latest list of AWS S3 IPs used by the self-hosted runners :

curl https://ip-ranges.amazonaws.com/ip-ranges.json | jq -r '.prefixes[] | select((.region=="us-east-1" or .region=="us-west-2") and .service=="S3") | .ip_prefix'

Please note that you have to allowlist all these IPs irrespective of the step size.

If Using URL

micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com

It's necessary to implement these changes to ensure that your pipeline runners can run stable and continue to use artifacts and cache features.

I hope that helps!  Let us know in case you have any questions.

Thank you, @MennoVink !

Patrik S