Test Driven Development?

[JL]

Hi,

We recently switched to Bamboo for our CI. One of the issues we've been running into however, is dealing with broken tests. Is there a "proper way" to handle either:

- new tests for features not yet implemented (Test Driven Development)
- broken tests exemplifying a bug

What I was basically thinking would be some way to quaranteen tests that are new and broken, until they pass...and then unquaranteen them. It'd be a shame to have to either @Ignore or comment out your tests if you wanted to commit something, where it wasn't yet complete...or if you wanted to make a ticket and point to a specific bug, that is exemplified by a failing unit test.

Are we going about this the wrong way? Is there some way to do this? We are using Maven / JUnit, and the latest version of Bamboo.

Thanks!

[PiotrA · Accepted solution]

Hi Jon,

There is no such functionality in Bamboo that would auto-manage the quaranteen tests yet (and I haven't heard of 3rd party plugins doing that). I can't find if we have already raised an issue in Bamboo tracker for your particular use-case (I thought we had), but you might be interested in these, somewhat relevant, issues:

* https://jira.atlassian.com/browse/BAM-12665 (A new type of Quarantine could be made for new test submitted just to illustrate a failure)
* https://jira.atlassian.com/browse/BAM-12406 (Features to help keep quarantined tests in check)
* https://jira.atlassian.com/browse/BAM-12044 (Provide mechanism for Bamboo to auto-quarantine tests based on class annotation)
* https://jira.atlassian.com/browse/BAM-11990 (Option to auto-unleash passing quarantined tests)

These are probably not exactly what you need from Bamboo, but I think these features could be a step-in towards achieving your goal. So it might be a good idea for you to add yourself as a watcher/voter for that particular issues. Or maybe even create a new issue?

regards,

[JL]

Hi Piotr,

Those are similar. One thing that would be nice would be able to "supress" a test (essentially, quaranteen it) by associating it with a ticket somehow...essentially, that ticket is responsible for fixing it.

[PiotrA]

That reminds me of another, related ticket: https://jira.atlassian.com/browse/BAM-11995 ("Create Issue" button next to "Quarantine") :)

[JL]

Ideally, if it automatically made it, AND automatically un-quaranteened it when the ticket was closed, that'd be awesome.

I feel like once a broken test is "accounted for" by a JIRA ticket, it's a bit silly to continue failing the build because of it -- basically, we already know / are working on it.

[David Karr]

Automatically quarantining failed unit tests is definitely not a good approach. A failed unit test needs to be fixed. Immediately. It should be treated just like a compile error in your build.

Note that you haven't specifically mentioned "unit" tests, although you're referring to test-driven development. Component or integration tests have to be handled differently.

[JL]

I don't know what kind of project you work on, but this would result in our build being continually broken. Just because a test is failing does not mean that it would be a blocking concern.

[Bob Swift OSS (Bob Swift Atlassian Apps)]

There is a quarantine function in the latest Bamboo versions. Run the test and have it fail. Then the quarantine option button shows up.

[JL]

I know there is a quaranteen feature.

Users should not have to manually quaranteen and unquaranteen tests every time though. This isn't really the solution I was looking for...this also has you break your build intentionally, just so that you can quaranteen it next time.

[Dan]

An example workflow that I can think of is someone has found a bug in the system. They create a unit/integration test that demestrates that bug. He commits it so another developer can respond to the issue. You wouldn't want that new test to fail the build until it is fixed by the developer. A ticket could signal this or maybe just have it automatically know that as soon as this test past for the first time we should unquarantine it.