Bitbucket pipeline test runner failing with internal errors

[Mike Makuch]

 Getting errors when Bitbucket runs my pipeline:

Error
There was an unexpected system error that may have left your step half-completed. To solve this error, check your step progress and rerun it if you see fit. Refer to the Troubleshooting runners document for help resolving the issue.

In the pipeline build log (see attached, on the right) it shows no failed command, no errors it appears to be some internal pipeline problem.

I choose Rerun failed steps and it will move past that error for 1 or 2 more steps, then show same error again. If I keep choosing Rerun failed steps 4 or 6 times, it keeps making progress and eventually completes.

* To be clear Rerun failed steps is rerunning the same instance, not creating a new pipeline instance.

BTW this pipeline has been running for 6 months without a problem, until today. It ran several times yesterday without a problem.

Any ideas appreciated.
 
Thanks
Mike

[Ajay _view26_]

Hi @Mike Makuch 

Welcome to the community!

That error message is almost always tied to self-hosted runners. It means the step's container got evicted mid-execution

A few things I'd check, roughly in order of likelihood:

• Docker disk space buildup.
• Memory pressure
• Timing note

Runner logs are the most direct diagnostic — they'll often show the specific reason for the container eviction that the Pipelines UI doesn't surface. Good luck!

let us know what you find.

[Mike Makuch]

Thanks Ajay

Not using Docker, just running the jar as a daemon.

Plenty of disk.

I had claude analyze the log, says:

### What it says

The runner is doing this:

1. starts a step
2. runs it successfully
3. finishes the step
4. then Bitbucket/runner hits 409 conflict or 403 forbidden while trying to:
- append to the main log
- update step progress
- finalize the step

### The key lines

You’ve got:

- 409 Conflict ... Failed to update stepLogRecord ... updated concurrently
- State transition received for step ... while tracking step ...
- 403 Forbidden ... Runner currently has no pipeline scheduled
- Unexpected error running task: MainLogUploader
- then the runner still says:
- Step was completed with result: COMPLETED

### What that means

This is not a normal script failure.

It means the runner and Bitbucket are getting out of sync during step completion / log upload / progress update.

### Important detail

The failure appears after the step has already basically finished, which is why Bitbucket shows the generic:

│ “There was an unexpected system error that may have left your step half-completed.”

### What stands out in this log

- The first several steps complete fine.
- Then at 00:52:24 the runner hits the first 409 Conflict.
- After that, it starts seeing:
- Runner currently has no pipeline scheduled
- Runner is currently assigned to another step
- That’s exactly the kind of state desync you’ve been seeing.

### So what this is likely to be

Most likely:
- runner/Bitbucket state handoff issue
- log uploader race
- step completion race

Less likely:
- YAML syntax
- a specific command in the step
- a bad script: entry

### One clue that matters

This line is especially telling:

│ Unexpected error running task: MainLogUploader. Cancelling.

That means the runner is not failing on your command itself — it’s failing while trying to upload logs.

[Ben]

Hi @Mike Makuch 

This error message means that the runner was not able to execute the build completely, and the build was abandoned as a result. This is a non-specific error which can occur on rare occasions.

The occurrence of this error often means that the runner directory is not cleaned up as a result, which can lead to further execution failures. This particularly affects Windows runners, where we are tracking an active bug ticket related to this:

• https://jira.atlassian.com/browse/BCLOUD-23347

To resolve this:

• Please proceed with deleting your existing runner, and spin up a fresh runner instance
• Once complete, please execute a fresh build, rather than re-running a failed build
• If this fails again on a fresh build - please let us know

Cheers!

- Ben (Bitbucket Cloud Support)

[Mike Makuch]

Thanks Ben

The occurrence of this error often means that the runner directory is not cleaned

It was not that.

This particularly affects Windows runners

My runner is on linux.

I first had my bitbucket-pipelines.yml broken into several steps. And it would fail in between various steps, never showing a problem with the yml.

I tried many things. Finally AI suggested to make it one single step so I did that and it started working.