How can I split my test execution stage from the test parsing stage?

[Aron Weiler]

Using bamboo-specs, when I attempt to split my unit test execution up from my test-parse task, I always receive this error:

my-test-results.trx was ignored because it was modified (2021-12-02 15:20:32.000) before task started (2021-12-02 15:20:52.626)

This appears to be happening because I have my unit test execution running in a stage before my unit test parsing.

The parser finds the .trx files for my MSTest compatible unit tests, and then just outright fails because the test results were generated before the current parsing task was started.

What is the point of this restriction?  Is it somehow to ensure that the test-parser task is looking at the most current results?

Here is an example of what my bamboo-specs look like:

Stage 1:
  - tasks: 
    # Run the unit tests
    - script:
      interpreter: BINSH_OR_CMDEXE 
      scripts: 
      - <execute my unit tests>

Stage 2:
  - tasks: 
    # ... copy results into current dir somewhere ... #
    # Parse the unit test results
    - test-parser:
      type: mstest
      test-results:
        - "**/*.trx"

[Eduardo Alvarenga · Accepted solution]

Hello @Aron Weiler

As per your description, you are running multiple Jobs within the same Stage in Bamboo. Jobs will run in parallel and will not consider the results of their sibling jobs.

To instruct a Plan to run tasks in a specific order, you need to assign each Job to distinct Stages. You can also assign all tasks sequentially within the same Job, but I feel that might not be your case.

Stages groups (or maps) jobs to individual steps within a plan's build process. For example, you may have an overall build process plan that comprises a compilation step, followed by several test steps, followed by a deployment step. You can create separate Bamboo stages to represent each of these steps.

Here is a graphical representation of how projects in Bamboo contain plans, plans contain stages, stages contain jobs and jobs contain tasks:

Don't forget that whenever you need to refer to resulting artifacts from one Job in the other, you'll need to share the artifact between Jobs and stages.

It may also be the case that the parser is not considering the 'trx' files as they were created by another task. Make sure to enable "Pick up test results that were created outside of this build" under the Task advanced options.

Kind regards,

Eduardo Alvarenga
Atlassian Support APAC

[Aron Weiler]

Sorry for the misunderstanding- but the tasks are happening in different stages, not jobs.  I have updated my question to reflect this.

Does it matter how I share artifacts between stages?  Because as it is right now, the artifacts are located in a docker container that is being accessed across many stages.

I'm using docker this way because of a bug in bamboo that does not allow me to use docker runner in Windows (https://jira.atlassian.com/browse/BAM-20219). 

In that issue, you can even see a workaround answer from me to address the issue, and you can use your imagination to see me splitting up the docker tasks across stages, like this:

• Stage 1:
  • spin up docker
• Stage 2:
  • run build in docker (using docker exec)
• Stage 3:
  • run unit tests in docker (<-- generates .trx results)
• Stage 4:
  • run integration tests in docker (<-- generates .trx results)
• Stage 5:
  • copy test results from docker container to build system (docker cp)
  • attempt to parse test results (<-- this is where I get the error described in the first post)

I can always use a Powershell script or something to modify the timestamp on the .trx files, but that is such a giant hack for an issue that shouldn't even exist.

[Aron Weiler]

Additionally, it is impossible to pick up results outside of the build, because that is not supported in YAML specs.

Pick-up test results outside of this build (Java Specs only)

[Aron Weiler]

Also, one more thing @Eduardo Alvarenga... the image you posted is from a confluence site that I do not have access to, so it's just a broken image on here.

[Eduardo Alvarenga]

Hello @Aron Weiler

As you have not specified which Bamboo release you are running, we assume you are on the latest.

Check the following BAM that addresses the "Pick-up test results outside of this build" limitation:

• BAM-20851: Support "Pick-up test results outside of this build" for YAML Bamboo Specs

The YAML syntax has changed slightly after 7.0. From 7.1 it uses the any-task.

On a recent version of Bamboo you can try this syntax:

  tasks:
  # ... copy results into current dir somewhere ... #
  # Parse the unit test results
  - any-task:
      plugin-key: com.atlassian.bamboo.plugin.dotnet:mstest
      configuration:
        mstestTestResultsDirectory: '**/*.trx'
        pickupOutdatedFiles: 'true' 

About the image, you'll need an Atlassian ID to access it. You can create a new account by checking https://id.atlassian.com/signup.

Kind regards,

Eduardo Alvarenga
Atlassian Support APAC 

[Aron Weiler]

Thank you for the response, @Eduardo Alvarenga

I am running Bamboo Server 7.2.3. 

Apparently the specs documentation that I have been looking at for the past month (6.9) is not the latest documentation.  I didn't even know that a version newer than 6.9 existed until today.

I will try using the YAML you posted here, but I am curious...

1. Where can I find a list of all of the available versions of Bamboo YAML specs? 
2. What is the best way for me to learn about the plugins that are available, and how to use them in YAML?

The information that I am looking for does not appear in any searches, or by perusing the bamboo documentation.  In fact the version of the YAML specs that you are describing above (7.1) was buried under 3 other documents I had to sort through, and I still have yet to find just a simple list of bamboo specs revisions (or better yet a matrix that shows me which version of specs I should use for my server version).

https://docs.atlassian.com/bamboo-specs-docs/  <- that would seem to list all of the versions of the YAML spec, but then there is no version 7.1.X in there at all. 

However, 7.1.2 is available here: https://docs.atlassian.com/bamboo-specs-docs/7.1.2/specs.html

Also, there are errors all over the documentation... such as on the following pages:

• https://confluence.atlassian.com/bamboo0700/bamboo-specs-1014682894.html
• https://confluence.atlassian.com/bamboo0702/bamboo-specs-1031180127.html
• https://confluence.atlassian.com/bamboo0609/bamboo-specs

All of these pages are ostensibly related to different Bamboo Server versions, but the links on it don't take you to the correct specs documentation- in fact, they take you to 8.0.4 documentation (latest).

[Aron Weiler]

Hey, so yesterday I replied here with a bunch of questions... and now my response is gone.

What happened to it?

[Eduardo Alvarenga]

Hello @Aron Weiler,

I was able to see your long post. Thanks for pointing that. I've raised a bug about the missing links on that page so that can be fixed:

• BAM-21534: Fix directory listing in https://docs.atlassian.com/bamboo-specs-docs

The official documentation for YAML specs is provided by that link.

A developer reference can be found at the Atlassian Developer portal. E.g:

• How to extend Bamboo Specs

YAML specs is still an experimental feature and not recommended for advanced Plans. Bamboo YAML Specs is evolving through time and adding more and more features that would match the same feature sets that are available in Java Specs, but that will still take some time.

This filter may give you an idea of the current disparity between Java and YAML specs:

• Java and YAML Specs feature disparity

Our recommendation, for now, is to switch to Java Specs if you can't achieve your intended action through YAML Specs.

I didn't see any errors on the other three links you mentioned.

Kind regards,

Eduardo Alvarenga
Atlassian Support APAC

[Aron Weiler]

Thank you @Eduardo Alvarenga 

If the YAML specs are not the preferred method of constructing build plans, then it would be helpful if that were stated somewhere that allows a developer to choose the right scripting engine before going down a long path of developing for the wrong one.

As it stands right now, I have a number of massive build projects that are written in YAML, which is completely out of the question converting to Java- unless there is a tool to do it automatically (and error free).

---

The last three links that I posted all have the same problem.  Let's take a look at one of the links as an example:

https://confluence.atlassian.com/bamboo0700/bamboo-specs-1014682894.html

If you navigate to that link, and then look at the "Docs and more docs" section, you can see that the links for the specs reference and the specs API are not pointing to the "Data Center and Server 7.0" documentation that is specified in the drop down at the top of the page.

Instead these links are pointing to the "latest" documentation.  They should be pointing at the version of the documentation that is called out on the top of the page.