We generate NUnit reports in our test runs that are picked up by Bamboo. These nunit.xml files are generated by a custom script, since we don't have a test runner that will generate a report recognized by Bamboo. This is working just fine.
But, we do have tests that are not run every time for some reason or another. Maybe the test is not finished yet or there is a know bug that we can live with for now. In which case we want the test to be listed as skipped, or ignored or not-run. All of which are possible with the NUnit file format.
My question is can Bamboo pick up on different statuses than success or failure? Does it recognize that tests can be skipped and can we get it to report the skipped tests. What would the nunit.xml file look like with a skipped test in it?
I'm not sure. I remember that in Bamboo 3.2 we've added some routines for handling 'skipped/inconclusive' test results, but I think we're currently not displaying in them in UI (if we encounter skipped test we just ignore it, that is). However, I just browsed the code for NUnit test result collecting, and it seems to me it has only stub implementation for the NUnit skipped tests:
So... I think the answer is "no", Bamboo can't pickup 'inconclusive' status for NUnit file format yet. Though I think enabling support for the third test status is easy(TM) task for a person having basic java programming skills (and access to the Bamboo sources... not sure if they are distributed with every license, or only the most expensive ones...). Would you like me to raise a relevant feature request on Bamboo issue tracker (http://jira.atlassian.com/browse/BAM) ?
I can't seem to get skipped to work with JUnit either. Here is a stripped down sample of the file I tried:
<?xml version="1.0" encoding="UTF-8" ?> <testsuite errors="1" failures="1" skips="2" tests="5" time="0.048" timestamp="2007-11-02T23:13:50"> <testcase classname="net.cars.engine.ValveTest" name="checkSpeed" time="0.0080" /> <testcase classname="net.cars.engine.ValveTest" name="moveDown" time="0.0010"> <skipped /> </testcase> <testcase classname="net.cars.engine.ValveTest" name="fjasefisken" time="0.0030"> <skipped message="i am skipped." type="junit.framework.Skipped">junit.framework.Skipped: i am skipped. C:\exp\test.py: 8: Skipped: 1 </skipped> </testcase> <testcase classname="net.cars.engine.ValveTest" name="checkStatus" time="0.0030"> <failure message="Plunger status invalid." type="junit.framework.AssertionFailedError">junit.framework.AssertionFailedError: Plunger status invalid at net.cars.engine.ValveTest.checkStatus(ValveTest.java:42) </failure> </testcase> <testcase classname="net.cars.engine.ValveTest" name="moveUp" time="0.022"> <error message="test timed out" type="java.lang.Exception">java.lang.Exception: test timed out </error> </testcase> <system-out><![CDATA[Slowly moving up]]></system-out> <system-err><![CDATA]></system-err> </testsuite>
What I see is the passing test, the failure and the error. The two skipped are just ignored. We are using an old version of Bamboo though, so maybe we need to upgrade. (Are running 3.4.3 now)
Anyway, Piotr: I would appriciate if you could raise that issue with NUnit.
Hey Everyone! Unfortunately, the venue that was hosting us on the 23rd has pulled out so we're looking for a new venue. If anyone would have a room free that we could use on the ev...
Connect with like-minded Atlassian users at free events near you!Find a group
Connect with like-minded Atlassian users at free events near you!
Unfortunately there are no AUG chapters near you at the moment.Start an AUG
You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs