Feature Preview: Metrics

[Patrick Hill]

Hey everyone, Patrick here from the Compass Product Management team.

We’re currently working on enabling Scorecards to use metrics and data that come from the tools you use as Scorecard rule criteria. We want to bring the data you care about most and the data you need to make decisions into Compass to help you alleviate toil and evaluate the health of your components. For example, maybe the Bitbucket integration, and soon GitHub, could emit a metric for the number of open pull requests waiting to be addressed allowing you to define a Scorecard criteria like “Average PR Cycle time <= 720 minutes”. Or perhaps those integrations could send a computed Metric for how your CI/CD pipelines/workflows associated with a Component perform and if that exceeds a threshold you define then the Scorecard criteria is not met - e.g. “Unit Test Coverage >= 80%”. In addition to the Metrics that will come from integrations, you’ll also be able to define your own custom Metrics and use them as Scorecard criteria as well. Here's a preview of what we're thinking:

What do you think about metrics like these? What other metrics or data would you like to see on Scorecards that would help you reduce friction and alleviate pain points in your organization? What are some examples of Scorecard rules you’d like to be able to have?

[Josh Campbell]

Hey Pat, this is awesome thanks for sharing it with everyone! I’m really excited to start building integrations for Compass that bring these metrics, and many more, to everyone using Compass. We’re hard at work building a GitHub integration and we’ve got a couple other ideas for metrics that can be used with Scorecards that we think are really important to engineers and for helping paint a picture of service health. The two that I’m currently the most excited about are:

• Average build duration on the main branch to help customers understand if their build times are increasing and why. Along with this metric, I think it would be really cool to highlight the build stage or even specific step/job in a pipeline that is taking the longest. This could help with tracking down inefficient integration tests or identifying where caching can improve CI performance.
• Green/Red build ratio, or the percentage of recent builds that were successful (green) versus failed (red), on the main branch as a way to see if there are flaky tests or other frequent transient issues that result in build inconsistencies.

Hey Alpha participants, what do you think about these metrics and what Pat proposed, are they useful to you? What’s your top three most important data/metrics/info you look at regularly for your job?

[Will Le]

My use case would be to utilise the scorecard as a means monitor the health / performance of a product / service. In addition to the items that have already been mentioned for source code / CI some further additions using JSM / software:

- Incidents per month (reliability) measured against a KPI

- Availability per month measured against a KPI / SLA

- Problems / defects created (measure to show prevention to incidents) 

- Percentage of enhancement v defects v incidents (to ensure our devs aren't just focussed on the "shiny" things but rather ensure usability and customer satisfaction is always aren't the forefront - let's face it, our customers just want things to work and are reliable) 

- Cloud compliance of a product / service e.g. CIS benchmark from Azure / AWS

And so on.....

[Will Le]

Hey Team, 

I've been waiting with eagerness on the movement with scorecards so adding this functionality would be very beneficial. Have you considered broadening these metrics beyond repos and against incidents / problems from Jira Service Management (JSM) and issues (defects) from Jira Software? This would then create a 360 degree view of a service not just from a technical pipeline perspective but also from the perception and issues of end users who log incidents / problems via against a service / product in JSM and Software? 

Thanks

Will

[Josh Campbell]

Hey Will thanks for the question! Yes, we are definitely thinking about metrics more broadly than just source code and CI. The team is hard at work building out the underlying platform and the examples above are some of what we are likely to showcase as part of our first release. More will absolutely follow!

That said, we would LOVE to hear what kind of metrics are most important to you and others around incidents, issues, or something else from another tool/source.