How do you handle defects through multiple environments?

[Ross LeMere]

Hello Atlassian/Jira community!

I have a question for you today...how do you all handle managing defects through environments?

Let me give you some context on the situation. We currently have a QA environment where the majority of our testing takes place. In addition, we have a Pre-Production environment, or a staging environment, that everything gets promoted to once validated and approved in our QA environment. Finally, we have Production when the product goes Live.

The current issue I am trying to figure out is if a defect is found in the QA environment, how do you handle it once it is fixed and validated in the QA environment? Do you close it and create a test case to account for that potential issue going forward? Do you keep the defect open into Pre-Production and only close it before going to Production?

The reason I'm asking this is that I'm trying to provide reports to upper management, but the current way we handle defects makes those reports inaccurate. We are currently leaving the defects open through the environments and are only closing them once a deliverable goes to Production. This skews reports such as Defect Age and Created v. Resolved.

I'm all ears here and would love to hear what others do that have a similar setup. Any advice is greatly appreciated and I can answer any clarifying questions you might have!

[Emre Toptancı _OBSS_ · Accepted solution]

Hello @Ross LeMere ,

My first suggestion would be not to make deployments directly to pre-production or staging. If a bug is discovered on these environments (and decided to be fixed immediately), the source code is modified so it is a new deployment, a new package. This new deployment should start its life from the beginning and go through all the stages (Dev/Test/QA/Staging...). You shouldn't fix it and deploy the fix directly to the same stage.

If you are using test automation, the first step should be to build an automated test case that reproduces the bug. This should be done even before fixing the bug. Build your test case, see it fail, then fix the bug and see it pass. This way, your test automation suite will grow and have more coverage with each new bug.

For the workflow, I suggest using a workflow that ends with Resolved and Closed statuses. When a fix is implemented for a bug, you move the issue to Resolved status. If the tests for the bug fail, you reopen the same issue. You Close that bug issue only when that version (with the fix) is released to production environment.

If you discover a bug about a functionality in an environement other that production, you can resolve and reopen the same bug issue over and over again. If that bug (or some other bug) is discovered in production, it doesn't matter if it is related to the first bug, the version is already released to production, you should now create a new bug issue, new version and new release for the additional fix.

Finally, while working with a bug issue in different environments (except production), that issue might be resolved and reopened multiple times. It is very useful to track in which build the bug was discovered and in which build a fix was implemented (different builds of the same version). Jira's "Affects Version" and "Fix Version" fields just contain the version number, it does not include build numbers so you can't use them for that purpose. The user who discovered the bug and the developer who implemented a fix for the bug should note this build number on the issue. You can create a text custom field for this but most of the time putting the build number in the comments work just fine.

[Ross LeMere]

Hi Emre,

First, apologies if my original post was misleading. If we find an issue in Pre-Production/Staging then yes, we do start from the beginning in Dev and make sure the fix for that specific bug is verified from the beginning and through all of the stages. Very rarely, if ever, would we deploy a fix straight to the Pre-Production environment.

Second, we do have Automation! I think this is a great idea, creating a test case specifically for the bug scenario and then passing that Test Case to our Automation team to cover in our Automation. I will definitely be taking this back to our team to see if it is something we can do.

Third, I agree with you 100% regarding Resolved and Closed. We have multiple teams working on multiple different projects all at the same time and each time runs their Jira differently. This makes it difficult to do this kind of thing, especially as someone on multiple teams at the same time, but I agree that it is something we should start doing as this is the approach I have done previously at other jobs.

Fourth, we do this already for the most part, but looking forward at your last point, we currently don't use Versions or any form of build number in our Jira tickets. I mentioned this when I first started at this job, but everything is always moving so fast it's hard to take a step back and make structural improvements.

Thanks so much for your reply. Very helpful!

[Lou Lowe]

All --- I would still like to expand on Ross's initial question. Let's assume two different cases. 

CASE 1: User Stories:  I have a new story that is currently in development. In JIRA I have the status set to IN PROGRESS until Dev and Unit Testing is complete. I then need to move the story through QA/SIT/UAT/Prod environments. I believe Ross's original question was related to how to best status the story for each environment. Do I simply have a separate status for "In QA Testing", "In SIT Testing", etc.?  Or is there a more elegant way to do this? The issue is that if I have multiple QA Statuses in each environment, the corresponding JIRA board gets crowded but the raw data is more accurate for management readouts. Thoughts?

CASE 2: Bugs/Defects: I am not sure that I would agree that a bug is resolved when it is fixed in a lower environment. We often work on large scale projects where there are 6+ development teams, multiple vendors, and complex technical environments. In this scenario we have noticed that a bug may be environment specific. For example, the bug may not occur in a lower environment but once it gets to SIT or UAT it is discovered. This can be for any number of reasons, such as environment configuration, issues with deployment processes, scripts, more robust (or potentially bad) data, etc.  So for us, we need to track the QA status of a bug through each of the environments as well. 

Does this make sense? 

In Summary: Is the best way to track QA progress in each of the environment to setup a set of status for each environment or is there a more sophisticated way to leverage Jira functionality to accomplish the same results (e.g., I considered trying to use a combination of the environment attribute and the status to arrive at a detailed break down but this seems to make reporting and dashboarding more complex).  

Any thoughts or guidance will be greatly appreciated!!

[Bill Sheboy · Accepted solution]

Hi @Ross LeMere  -- Welcome to the Atlassian Community!

To help with your questions please consider:

• How do you manage defects today, and why?
• You note that some of your "reports to upper management" are inaccurate.  What is inaccurate about them and how is the information used?

You note that you have 3 environments...I am guessing there is a fourth: Dev, QA, Pre-Production, and Production.  And, you seem to be describing defect discovery in the QA environment.

How you manage defects currently and the defect impact may help you decide.

For example, for a newly built feature a discovered defect would be "resolved" once validated as corrected in the lower environment...no need to wait until release to production.  However for a production-impacting defect, "resolved" usually occurs once released to production.

Best regards,

Bill

[Ross LeMere]

Bill, you are correct! We have a Dev environment as well.

Thanks for the advice, it makes sense and is greatly appreciated!