Where do I view pipeline test reports?

According to the current docs, maven test results are picked up automatically and test reports generated. I can see in the tear down section of my build pipeline that it is picking up the results (output below), but can't see the reports anywhere. Can someone please advise where to view test reports?

 

Finished scanning for test reports.

Found 22 test report files.

All test reports aggregated into 44 test cases

Uploading test results

Finished uploading test results.

6 answers

@Jason Sheedy I have found that you can only view the results of failed tests. When you have a pipeline with failing tests, you will then be able to select a dropdown that appears where the Logs header is and select Test Results. Alternatively, you can click on the x number of failures message on the unit testing step and that will take you to the test results as well.

Hope this helps!

Thanks Andrew. It seems a like a bit of an oversight to not display passing tests. I can't think of any good reason why they wouldn't include it. Maybe that's only available in the Enterprise version. :/

I would also like to see reports where all tests passed.

We now display the successful tests too. But don't generate a full report. Is there anything specific you'd want to see in a larger report for a 100% tests passing?

Hi Philip. Could you tell me where to see passing tests? I see maven test results output in the logs, but nothing appears in the bitbucket user interface.

thanks

Hi @Jason Sheedy,

Were you previously seeing failed test results as well? Or have none shown?

If you have everything already set up, they should be showing on the step like so:

Screen Shot 2018-01-15 at 5.36.39 pm.png

If you don't have it set up, you should have debugging information about this in the 'Build Teardown' section of your pipeline. Note there was a bug that prevented them from displaying, which we have now resolved so you may only seem them on new pipelines.

Screen Shot 2018-01-15 at 5.40.10 pm.png

Which directory are your test outputs going to at the moment? Right now we're only reading from specific test report directory names listed here: https://confluence.atlassian.com/bitbucket/test-reporting-in-pipelines-939708543.html

Thanks,

Phil

@Jason Sheedy I am a bit late to this conversation, but I noted in one of your earlier response you said,

It seems a like a bit of an oversight to not display passing tests. I can't think of any good reason why they wouldn't include it.

I would say that sometimes the most important metric for a fast-paced agile team is to know if we have failures to fix or analyse. If the tests are passing the result details will not change what happens. The details of the tests are only interesting as to what we will test. That occurs in planning the tests. 

That is my humble opinion, and in my experience, it is most useful to those who did not design, implement and execute the tests.

Thanks @Philip HodderLooks like my test reports are being captured in the build teardown, but I don't see anything in the UI to indicate failed or passing tests. 

Found 22 test report files.
All test reports aggregated into 44 test cases
Uploading test results
Finished uploading test results.

When you say "should be showing on the step", is that displayed in the log output of the individual build or the main pipelines screen that shows all builds?

Correction. I do see failing tests, but it's not displayed in the same way as the screenshot you posted. 

FailedBitbucketTest.png

@Jason Sheedy

Sorry, I just showed a subset of the page. Here's the whole result:

Screen Shot 2018-01-16 at 10.36.54 am.pngThe same view as you're looking at.

Thanks,

Phil

Thanks Phil. I'm not seeing the same thing. Could there be some difference because it's an academic team account? (See below)

Screenshot from 2018-01-16 12-45-08.png

@Jason Sheedy

Using an academic account is unlikely to be the issue here. It looks like you just have your log view expanded. Do you have an icon that looks like this on the top right of your screen?

Screen Shot 2018-01-16 at 1.49.59 pm.png

Larger screenshot here. Look at the top right (next to 'Download raw'):

Screen Shot 2018-01-16 at 1.50.15 pm.png

If so, when you click it you should see what I'm seeing.

If not, can you let me know what Browser + Version + OS you're using?

Thanks,

Phil

Oh wow. I feel kind of stupid. I see it now, but I've got to say that it's less than intuitive. Maybe a job for the UX team??

Thanks for your help.

No worries. I'll pass on the feedback. :)

Is there any permission needed to get setup before you can view test reports? base from the log it already created the report but I'm not sure why I wasn't able to access it or maybe it's just permission issue or I'm doing it the wrong way.

 

Screen Shot 2018-02-07 at 2.49.36 PM.png

@Philip Hodder

Hello,

I cannot see the test reports too. I'm on Windows 10, with both Chrome & Firefox (latest versions).

I tried with 1 or 2 steps, but still there is no dropdown next to "Logs" to switch to test reports. The tests should be visible since half of them fail. The screen is almost the same as for the previous post (except it is not the same project).

How can I view the test reports ?

Hi,

What do you see in the logs for the Build Teardown command? Are any tests being discovered?

Which directory are the tests being output to at the moment? We only check for reports in certain directories. Have a look at the documentation here to double check: https://confluence.atlassian.com/bitbucket/test-reporting-in-pipelines-939708543.html

 

Thanks,

Phil

Hi,

I output the tests in the "test-results" directory, and they are found in teardown, but not shown even if half of them have failed... See the image below:

bp.png

Thanks,
TL

Ok, I think I understand: tests are shown only if the result code is not 0 (which is not the same as all tests succeed...)

Hi,

The Test Result view will be shown by default if you have failed tests + failed pipeline.

However, it looks like a bug that you can't navigate to the tests even though you have failures (but a green build). I'll open an internal ticket to fix this.

Thanks,

Phil

Hi @Philip Hodder

I can now see Test Result view but it doesn't show anything on my endScreen Shot 2018-02-14 at 2.25.39 PM.png

Looks like the XML isn't being parsed correctly. What build tool are you using? Are you able to show me the XML that is being output?

@Philip Hodder

I'm using codeception for automation test. Here's an example of the xml file result

 

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="acceptance (chrome-qa)" tests="13" assertions="83" failures="2" errors="0" time="203.458746">
<testcase file="/Users/lovie/Documents/Projects/web/tests/tests/acceptance/Sanity/004-WinePageCest.php" name="loadLoginPage" class="WinePageCest" feature="load login page" assertions="0" time="9.928986"/>
<testcase file="/Users/lovie/Documents/Projects/web/tests/tests/acceptance/Sanity/004-WinePageCest.php" name="loginUser" class="WinePageCest" feature="login user" assertions="0" time="7.722183"/>
<testcase file="/Users/lovie/Documents/Projects/web/tests/tests/acceptance/Sanity/004-WinePageCest.php" name="redirectToMyCollectionsWines" class="WinePageCest" feature="redirect to my collections wines" assertions="0" time="7.060703"/>
<testcase file="/Users/lovie/Documents/Projects/web/tests/tests/acceptance/Sanity/004-WinePageCest.php" name="verifyWinesPage" class="WinePageCest" feature="verify wines page" assertions="18" time="11.594282"/>
<testcase file="/Users/lovie/Documents/Projects/web/tests/tests/acceptance/Sanity/004-WinePageCest.php" name="verifyAddWineModal" class="WinePageCest" feature="verify add wine modal" assertions="12" time="19.244257"/>
<testcase file="/Users/lovie/Documents/Projects/web/tests/tests/acceptance/Sanity/004-WinePageCest.php" name="verifyCellar" class="WinePageCest" feature="verify cellar" assertions="1" time="8.617440"/>
<testcase file="/Users/lovie/Documents/Projects/web/tests/tests/acceptance/Sanity/004-WinePageCest.php" name="verifyLocation" class="WinePageCest" feature="verify location" assertions="1" time="8.631113"/>
<testcase file="/Users/lovie/Documents/Projects/web/tests/tests/acceptance/Sanity/004-WinePageCest.php" name="verifyBin" class="WinePageCest" feature="verify bin" assertions="1" time="8.732308"/>
<testcase file="/Users/lovie/Documents/Projects/web/tests/tests/acceptance/Sanity/004-WinePageCest.php" name="verifyBrowseWinesDetailsCatalogs" class="WinePageCest" feature="verify browse wines details catalogs" assertions="24" time="35.186749">
<failure type="Codeception\Exception\ConditionalAssertionFailed">WinePageCest: Verify browse wines details catalogs
Failed asserting that on page /wines/browse/1789818
--&gt; Wines
Add Asset
WINES SUMMARYWINES COLLECTION131BROWSE WINESDRINK SOONLOCATIONS
2013 "Carlo V" il Rosso dell' Imperatore Merlot and Cabernet Venezie IGT
RED
2016 ”Podere Casa Guidi” di Giuseppe Leonelli Barbera Frizzante Emilia IGT
RED - SPARKLING
N.V. ”Podere Casa Guidi” di Giuseppe Leonell
[Content too long to display. See complete response in '/Users/lovie/Documents/Projects/web/tests/tests/_output/raw/' directory]
--&gt; contains "vintage1".

/Users/lovie/Documents/Projects/web/tests/tests/_support/_generated/AcceptanceTesterActions.php:858
/Users/lovie/Documents/Projects/web/tests/tests/_support/Page/WinePage.php:230
/Users/lovie/Documents/Projects/web/tests/tests/acceptance/Sanity/004-WinePageCest.php:89
</failure>
</testcase>
<testcase file="/Users/lovie/Documents/Projects/web/tests/tests/acceptance/Sanity/004-WinePageCest.php" name="addWine" class="WinePageCest" feature="add wine" assertions="1" time="41.462510"/>
<testcase file="/Users/lovie/Documents/Projects/web/tests/tests/acceptance/Sanity/004-WinePageCest.php" name="browseWinesFromInventory" class="WinePageCest" feature="browse wines from inventory" assertions="1" time="16.380560"/>
<testcase file="/Users/lovie/Documents/Projects/web/tests/tests/acceptance/Sanity/004-WinePageCest.php" name="verifyBrowseWinesDetailsInventory" class="WinePageCest" feature="verify browse wines details inventory" assertions="24" time="23.722190">
<failure type="Codeception\Exception\ConditionalAssertionFailed">WinePageCest: Verify browse wines details inventory
Failed asserting that on page /wines/collection/1772564
--&gt; Wines
Add Asset
WINES SUMMARYWINES COLLECTION132BROWSE WINESDRINK SOONLOCATIONS
1876 ”Podere Casa Guidi” di Giuseppe Leonelli Vino Nouvo Emilia IGT
RED
1999 3 Bridges Sémillon Golden Mist
WHITE - SWEET/DESSERT
1999 Kevin Cabrera Grape Producer
RED
1999 Montecalvi Alta Valle della Greve IGT
RED

[Content too long to display. See complete response in '/Users/lovie/Documents/Projects/web/tests/tests/_output/raw/' directory]
--&gt; contains "vintage1".

/Users/lovie/Documents/Projects/web/tests/tests/_support/_generated/AcceptanceTesterActions.php:858
/Users/lovie/Documents/Projects/web/tests/tests/_support/Page/WinePage.php:230
/Users/lovie/Documents/Projects/web/tests/tests/acceptance/Sanity/004-WinePageCest.php:116
</failure>
</testcase>
<testcase file="/Users/lovie/Documents/Projects/web/tests/tests/acceptance/Sanity/004-WinePageCest.php" name="logoutUser" class="WinePageCest" feature="logout user" assertions="0" time="5.175464"/>
</testsuite>
</testsuites>

@Philip Hodder I have the exact same problem as @John Lovie Too. I can only see the "m/n test failures", but I do not see the individual failed cases.  

The reports are generated by eslint. Neither eslint -f junit, or eslint -f tap | tap-xunit produces files that file show individual failure reports on the test results page.

eslint -f junit example

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
<testsuite package="org.eslint" time="0" tests="2" errors="2" name="/home/daniel/Documents/flying_sheep/code/sheepcore-boilerplate/src/js/index.js">
<testcase time="0" name="org.eslint.no-mixed-operators"><failure message="Unexpected mix of &apos;&amp;&amp;&apos; and &apos;||&apos;."><![CDATA[line 20, col 16, Error - Unexpected mix of &apos;&amp;&amp;&apos; and &apos;||&apos;. (no-mixed-operators)]]></failure></testcase>
<testcase time="0" name="org.eslint.no-mixed-operators"><failure message="Unexpected mix of &apos;&amp;&amp;&apos; and &apos;||&apos;."><![CDATA[line 20, col 25, Error - Unexpected mix of &apos;&amp;&amp;&apos; and &apos;||&apos;. (no-mixed-operators)]]></failure></testcase>
</testsuite>
<testsuite package="org.eslint" time="0" tests="2" errors="2" name="/home/daniel/Documents/flying_sheep/code/sheepcore-boilerplate/src/js/state/ThreeExample.js">
<testcase time="0" name="org.eslint.padded-blocks"><failure message="Block must not be padded by blank lines."><![CDATA[line 15, col 1, Error - Block must not be padded by blank lines. (padded-blocks)]]></failure></testcase>
<testcase time="0" name="org.eslint.indent"><failure message="Expected indentation of 1 tab but found 0."><![CDATA[line 15, col 1, Error - Expected indentation of 1 tab but found 0. (indent)]]></failure></testcase>
</testsuite>
</testsuites>

eslint -f tap | tap-xunit example

<?xml version="1.0"?>
<testsuites>
<testsuite tests="2" failures="2" errors="0" name="Default">
<testcase name="#1 /home/daniel/Documents/flying_sheep/code/sheepcore-boilerplate/src/js/index.js">
<failure>
---
message: Unexpected mix of '&amp;&amp;' and '||'.
severity: error
data: {"line":20,"column":16,"ruleId":"no-mixed-operators"}
messages: [{"message":"Unexpected mix of '&amp;&amp;' and '||'.","severity":"error","data":{"line":20,"column":25,"ruleId":"no-mixed-operators"}}]
...
</failure>
</testcase>
<testcase name="#2 /home/daniel/Documents/flying_sheep/code/sheepcore-boilerplate/src/js/state/ThreeExample.js">
<failure>
---
message: Block must not be padded by blank lines.
severity: error
data: {"line":15,"column":1,"ruleId":"padded-blocks"}
messages: [{"message":"Expected indentation of 1 tab but found 0.","severity":"error","data":{"line":15,"column":1,"ruleId":"indent"}}]
...
</failure>
</testcase>
</testsuite>
</testsuites>

I also am seeing the tests results being collected only in the build teardown.
If there are no errors I don't see anything else in the web interface. Maybe I need some setup. It does says that the test results from junit (Android build) are found and uploaded.

Hi,

At the moment we're only showing full results of failed tests. Can you check that you see them when you have failed tests?

Is there anything in particular you'd be interested in seeing when all your tests have passed?

Thanks,

Phil

Phil, do you have any feedback on why John and I don't see any details for failed tests with the XML we provided?

Nothing immediate. I'll look into that now.

Daniel,

Can you check to see if this is still an issue for you?

We've fixed some bugs in our parser that appear to make your XML parse properly now. I've checked the previously submitted XML files and they all now parse successfully in our parser.

Thanks,

Phil

Thank you, Phil!

I'll check tomorrow at work and get back to you. 

Excellent!

Let me know how it goes.

Phil,

I tested both formats again, eslint's JUinit formatting and eslint's tap output piped through tap-xunit. Unfortunately, I still just have the x/x failures display, but no list of failures.

Is your updated parser rolled out to everybody, or might I just need to wait some more?

Thank you,
Danielfail.jpg

Hi Daniel,

I've opened an internal bug report. As I get the same issue when I use your XML.

Thanks,

Phil

Thks for the quick answer and sorry for misplacing my original reply. I misunderstood the interface ^^

Anyway, failing tests failing the pipeline works like it's supposed to.

In case of full success, I would see a green bar in the log output and/or interface on the left (equivalent to the big red one on failure)

No problem. I'll pass on your feedback to the rest of the team. :)

Thks !

Also, I don't have the test details on the left, close to the link to see the build configuration as per your screenshots.

Maybe I could have these in there too when they succeed ?

 

Anyway, thks for the follow up

Hey guys,

after watching your webinar last week I started using Pipelines on our PHP API.

I was reading this thread carefully but still cant see a way to access the error reports created see:2018-03-22_1648.png

 

I know there is a way upload this to S3 but I cant find the documentation anymore.
Wouldn't it be the easiest way to attach this report to the email you send anyway on failure:

 

2018-03-22_1650.png

Hope you can help me and others here.
I already checked: https://confluence.atlassian.com/bitbucket/test-reporting-in-pipelines-939708543

Cheers from Fishburners Sydney :)

I don't know If I 'm missing sth here but phpunit can fail on deprecation notices(if not configured otherwise) but the tests will pass. As a result you got a failed pipeline with successful tests. Well, I don't know if it is an uncaught use case by your side, maybe it is worth mentioning it.

0 votes

Hi guys.. from https://confluence.atlassian.com/bitbucket/test-reporting-in-pipelines-939708543.html, says pipelines only support XML xUnit-compatible reports... what if i have an ExtentReport that spits out a TestAutomationReport.html at the end of every test run (and/or attaches a screenshot when a bdd step fails)... how do I integrate this in Bitbucket pipelines and actually have a Test Report tab or whatever that business users can jump into to see my colorful dashboard ?? :)

Hey Jackie,

I believe the reports should be strictly in XML format.

Suggest an answer

Log in or Sign up to answer
Community showcase
Published Aug 21, 2018 in Bitbucket

Branch Management with Bitbucket

As a project manager, I have discovered that different developers want to bring their previous branching method with them when they join the team. Some developers are used to performing individual wo...

2,369 views 9 12
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you