How do you track cloud changes - for validated environments?

[Steven Haworth]

We are preparing several new development projects, using Confluence (premium cloud) and Jira (std cloud), and our quality group has confirmed using cloud is acceptable - as long as we validate our critical control points (ie areas where we depend on the tool for requirements traceability, or approvals, or defect management, for example).

But - cloud evolves of course.  To address that, we will need to perform an impact review as cloud changes roll out - to determine when re-validation of impacted features might be needed.  Which means someone will need to closely track changes as they occur.

How do others do this?

For us, I guess we monitor the weeklyi release notes, and document a monthly review of those notes.  Then, if a change affects something critical (Jira workflow use, for example), we re-execute our validation test cases for workflows.

[Matteo Gubellini _SoftComply_]

Hello @Aaron Morris ,

We have recently released and App for Confluence Cloud to take care of this problem.

The App runs a number of automated tests on your Confluence instance to check its continuous integrity. Then it produces a protocol and results documents.

The App can run up to once a week.

The App was intended to address the software tool validation issue for the medical device industry, so it focuses on the requirements of ISO 13485 and the FDA Guidance documents, including the new Sept 2022 Draft Guidance. But it can be applied to any other sector.

Ref. SoftComply Validation App on the Marketplace.

The other suggestions of subscribing to the release blog, periodic review, backups etc are still valid.

Matteo

[Aaron Morris]

Hi @Matteo Gubellini _SoftComply_ -- This app seems pretty neat.  Thanks for sharing! 

Just to make sure I understood your documentation: the app contains a library (protocol) of off-the-shelf (OTS) scripts that are run weekly.  Is that correct?  If so, do you have a listing of test conditions or maybe a listing of OTS requirements that are verified?  I'm just trying to understand how I would tie the validation results into my traceability documentation.  Thanks!

(I hope this is still on topic. :-) )

[Matteo Gubellini _SoftComply_]

Yes you are correct.

You can try the app for free and it will output a (partial) report, which will allow you to see the test cases. 

Below is the MVP that I'm working on (still work in progress). I can send you an example of a protocol and report by email too if you prefer.

1. Objective

This Master validation plan describes the high level activities required to maintain the validation status of Atlassian Confluence Cloud using SoftComply Validation App.

The software tools in scope are:

1. Atlassian Confluence Cloud (Standard plan or higher), specifically the instance at the address below:

Add instance here (URL)

2. Terms and Definitions

None

3. References

1. General Principles of Software Validation; Final Guidance for Industry and FDA Staff; Document issued on: January 11, 2002

2. Computer Software Assurance for Production and Quality System Software; Draft Guidance for Industry and Food and Drug Administration Staff, September 2022

3. ISO 13485:2016 Medical devices — Quality management systems — Requirements for regulatory purposes

4. SoftComply Validation for Confluence, ref https://marketplace.atlassian.com/apps/1229288/softcomply-validation-for-confluence?hosting=cloud&tab=overview and https://softcomply.com/validation-app-user-guide/ .

4. System Description

4.1 Confluence

Confluence Cloud is a Cloud-based document management system. It manages Pages that are both documents and containers of other pages, and Spaces, equivalent to folders.

5. Validation Approach

5.1 Description

The SoftComply Validation App for Confluence runs integrity tests on the current Confluence instance periodically.

The tests can be run up to once every 7 days.

5.2 Documentation

For each run, the App automatically creates a pair of pages, a Validation Protocol and a Validation Report.

These documents are created in a predefined space:

Add space link here

5.2.1 Validation Protocol

Each Validation Protocol is named using the following convention:

“Validation Protocol DDD MMM dd hh:mm:ss UTC YYYY”, e.g. Validation Protocol Tue Feb 02 12:41:50 UTC 2023.

The protocol contains a list of Test Cases, each identified as “TC-CON-nnn” with nnn as a sequential number starting at 001. Each test case contains one or more steps, with the associated step description which includes the acceptance criteria. Each step is numbered as 1., 2., 3., etc.

For clarity, each validation protocol is nested under the relative result page.

5.2.2 Validation Report

Each Validation report is named using the following convention:

“Validation Report DDD MMM dd hh:mm:ss UTC YYYY”, e.g. Validation Report Tue Feb 02 12:41:50 UTC 2023.

The protocol contains the list of Test Cases and Steps, each identified as per the Validation Protocol. Each step is associated with the actual results (usually in the form of a screenshot with timestamp watermark) and a Pass/Fail statement.

At the end of each report, a summary describes the number of tests run, passed cases, skipped cases and failed cases.

5.2.2.1 Pass

A test case step is marked as Pass if it meets the acceptance criteria.

A test case is marked as Pass if all steps are a Pass.

5.2.2.2 Fail

A test case step is marked as Fail if it does not meet the acceptance criteria.

A test case is marked as Fail if any step is Fail.

5.2.2.3 Skipped

A test case step is skipped if any precondition for its execution is incorrect, in particular if any previous steps / cases it depends on are either skipped or failed.

A test case is marked Skipped if any step is Skipped, unless one or more steps are Fail.

5.3 Features to be validated

5.3.1 Intended use and risk

Confluence is intended to store electronic records and manage their changes. This documents are related to production and quality system. Confluence is considered to be used directly as part of production or the quality system, except for the exclusions listed below.

Features are considered high process risk when its failure to perform as intended may result in a quality problem that foreseeably compromises safety, meaning an increased medical device risk.

Feature	Risk	Rationale
Protection of records to enable their accurate and ready retrieval throughout the records retention period. Confluence will store data persistently. Confluence instances can be backed up and restored. Confluence pages can be stored and attachments can be managed.	The feature does not pose high process risk.	Failure of these feature to perform as intended would impact the integrity of the quality system record but would not foreseeably compromise safety.
Limiting system access and specific operations to authorized individuals. Only Users who an Administrator has invited will have access to Confluence. Confluence will allow administrators to manage plug-ins and other Administrator Settings. Authorized users shall be able to manage pages and spaces. Admins shall be able to set permissions and restrictions at global, space and page level.	These features does not pose high process risk.	Failure of these features to perform as intended would impact the integrity of the quality system record but would not foreseeably compromise safety.
Audit trail, page history. Confluence will retain previous versions of documents after changes have been made.	The feature does not pose high process risk.	Failure of these feature to perform as intended would impact the integrity of the quality system record but would not foreseeably compromise safety.
Reporting. Confluence will be able to pull issue data from JIRA and use them to generate tables in reports.	The feature does not pose high process risk.	Failure of these feature to perform as intended would impact the integrity of the quality system record but would not foreseeably compromise safety.
Concurrent collaboration.	The feature does not pose high process risk.	Failure of these feature to perform as intended would impact the integrity of the quality system record but would not foreseeably compromise safety.
Inline and page comments.	The feature does not pose high process risk.	Failure of these feature to perform as intended would impact the integrity of the quality system record but would not foreseeably compromise safety.

5.3.2 Exclusions

1. Blogs;

2. Minor utilities and macros.

5.4 Means

The test cases consist of automated inputs into the system. The Validation app interfaces with the system simulating actual user inputs.

5.5 Management of Failures

The App developer (SoftComply) is in charge of the initial triage of the failures.

The developer will determine if the issue is either:

1. A glitch in the execution, typically associated with a timeout of Confluence. This is not considered to be a failure, and the test run can be repeated.

2. A minor issue with the App, typically related to minor changes in Confluence such as the id of an element or the position of a button. This does not compromise the integrity of Confluence. SoftComply will upgrade the app an a test run can be repeated.

3. An actual Confluence issue. SoftComply will open a ticket with Atlassian and share it with the Company. An internal assessment will determine the actual impact on the document management system.

[Aaron Morris]

This looks great!

[Aaron Morris]

Hi @Steven Haworth -- I'll try to give a short answer based on my personal experience with this...

Yes, you can monitor Atlassian's change logs to watch for potentially impactful changes to your validated state.  Of course, this isn't a bad practice, but it's extremely time consuming and error prone.  I don't recommend relying on this strategy, at least not primarily.

So, assuming:

1. You've completed an initial validation of Jira/Confluence for intended use (preferably using a risk-based approach)
2. You've identified certain critical functions that your company does not trust the COTS vendor (i.e., Atlassian or app developers) to maintain

Then it's much more realistic and maintainable to rely on mitigations other than re-validating the environment on a (potentially) monthly basis.  For example, you can implement redundant quality checks into your workflows/processes, sync records with another system, export critical records (like traceability summaries) to an EDMS, etc.

Hope this helps!

[Steven Haworth]

Hi @Aaron Morris  Yes, we are taking a risk-based approach, and yes we can't trust the COTS vendors as they don't have a documented QMS / records to reference.

We did consider automating the validation of those critical points, so test automation could run on its own - but of course there's extra initial cost to that approach.

What we really envision is using Comala for approving native Confluence pages for documentation & design reviews, and using Xray and some type of p11-compliant add-on to approve test cases or executions inside Jira - without having to export and approve elsewhere.  But - exporting a snapshop and approving that some other way, could be a better approach.

Thanks for your thoughts - very helpful!

[Aaron Morris]

No problem!

For the Jira/Xray items, you might evaluate Tricentis Vera as a Part 11 signature solution.  It has its pros and cons (like any tool), but it might be helpful for your use case.

Vera is an external system that integrates with Jira. So you can lock your issues inside Jira and route them for approval externally in Vera.  And Vera would be installed to a bespoke cloud environment for your company, where you can control and validate changes.  (At least that's how it used to work.)

If you validate the integrated system (Jira w/ Xray + Vera), then you can have the best of both worlds (in terms of functionality), while mitigating many of your concerns about Atlassian's shared-cloud environment.

Some downsides I can warn you about:

• Vera does not support Confluence, so you'll still need Comala (or etc.)
• Vera supports qTest much better than Xray, because qTest is a Tricentis product.  (And FWIW, I personally prefer Xray.)
• Vera can be pricey compared to embedded signature solutions from the  Atlassian Marketplace.

Normally, I prefer using embedded signature solutions (like you mentioned), but given your use case and concerns, Vera might be worth a look.

Good luck!

Disclaimer:  I am a former employee of Tricentis, but I no longer have any affiliation with them.

[Philippe BARON]

Very interesting approach !

[marc -Collabello--Phase Locked-]

Hi @Steven Haworth ,

In addition to your approach I would recommend also:

1. Perform a periodic review, i.e. not only changelog driven.
2. Backup critical records for long term storage somewhere on different infrastructure.

Many of our customers for Document Control for Confluence Cloud do this, as such a strategy is more resilient than depending on Atlassian as the sole records storage.

[Nic Brough -Adaptavist-]

Your suggestion of monitoring and testing when necessary is the only option.  You can't get out of the service changing.

I subscribed to the release blogs for Cloud, over at https://confluence.atlassian.com/cloud/blog/2023/02/atlassian-cloud-changes-feb-6-to-feb-13-2023 but they're changing, so you'll want to follow the instructions for getting the new ones in a few weeks as well