We recently switched to Bamboo for our CI. One of the issues we've been running into however, is dealing with broken tests. Is there a "proper way" to handle either:
- new tests for features not yet implemented (Test Driven Development)
- broken tests exemplifying a bug
What I was basically thinking would be some way to quaranteen tests that are new and broken, until they pass...and then unquaranteen them. It'd be a shame to have to either @Ignore or comment out your tests if you wanted to commit something, where it wasn't yet complete...or if you wanted to make a ticket and point to a specific bug, that is exemplified by a failing unit test.
Are we going about this the wrong way? Is there some way to do this? We are using Maven / JUnit, and the latest version of Bamboo.
There is no such functionality in Bamboo that would auto-manage the quaranteen tests yet (and I haven't heard of 3rd party plugins doing that). I can't find if we have already raised an issue in Bamboo tracker for your particular use-case (I thought we had), but you might be interested in these, somewhat relevant, issues:
* https://jira.atlassian.com/browse/BAM-12665 (A new type of Quarantine could be made for new test submitted just to illustrate a failure)
* https://jira.atlassian.com/browse/BAM-12406 (Features to help keep quarantined tests in check)
* https://jira.atlassian.com/browse/BAM-12044 (Provide mechanism for Bamboo to auto-quarantine tests based on class annotation)
* https://jira.atlassian.com/browse/BAM-11990 (Option to auto-unleash passing quarantined tests)
These are probably not exactly what you need from Bamboo, but I think these features could be a step-in towards achieving your goal. So it might be a good idea for you to add yourself as a watcher/voter for that particular issues. Or maybe even create a new issue?
Ideally, if it automatically made it, AND automatically un-quaranteened it when the ticket was closed, that'd be awesome.
I feel like once a broken test is "accounted for" by a JIRA ticket, it's a bit silly to continue failing the build because of it -- basically, we already know / are working on it.
Automatically quarantining failed unit tests is definitely not a good approach. A failed unit test needs to be fixed. Immediately. It should be treated just like a compile error in your build.
Note that you haven't specifically mentioned "unit" tests, although you're referring to test-driven development. Component or integration tests have to be handled differently.
An example workflow that I can think of is someone has found a bug in the system. They create a unit/integration test that demestrates that bug. He commits it so another developer can respond to the issue. You wouldn't want that new test to fail the build until it is fixed by the developer. A ticket could signal this or maybe just have it automatically know that as soon as this test past for the first time we should unquarantine it.
I’m a designer on the Jira team. For a long time, I’ve fielded questions from other designers about how they should be using Jira Software with their design team. I’ve also heard feedback from other ...
Connect with like-minded Atlassian users at free events near you!Find a group
Connect with like-minded Atlassian users at free events near you!
Unfortunately there are no AUG chapters near you at the moment.Start an AUG
You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs