Metrics Calculations in Compass

[Pete Capra]

Hi there, 

I'm wondering whether you've considered having support for metrics calculation in Compass? By this I mean the ability for Compass to receive single values via API that are stored and then calculated within Compass. For example - deployment frequency that is calculated from receiving single updates via API for each deploy for that application. 

Having the ability to do this would remove the need for users of Compass needing a service that manages this as well. 

Cheers, 

Pete

[Henry Olson]

Hi @Pete Capra ! Thanks for reaching out. As it happens, we are actually in the process of implementing and releasing a feature that I think will meet your needs. We're calling it "Derived Metrics", and the concept is that you only need to send Compass Events (like Deployments or Builds), and Compass will be able to calculate certain metrics automatically for you.

Deployment Frequency and Build Success Rate are two metrics that are actually able to be created as "derived" today. When you create these metric sources from the component UI, they'll be auto-calculated based on the events present on that component. When creating metric sources via the API, you can supply `derived: true` as part of the input to set it to auto-calculate based on component events.

Look forward to a blog post in the coming weeks that will explain all the details of how to use this feature and which metrics are supported. And of course, let us know if you have suggestions on how the feature can be improved.

Best,
Henry

[Ben Wozniak]

Okay it appears that my build pipeline URL is fine, it was actually the "exampleUrl.com" that was causing issues. It appears the API is expecting the URLs to be prepended with "https://". I think this would be a valuable add to the documentation as well as the error that is returned from the attempted API call.

If this is in fact the case, it looks like the cURL command is working now. It was a little painful to work through but got it working :) 

[Ben Wozniak]

Thanks, Henry. I'm guessing this was a new addition since the Alpha release, I just needed to delete my old Deployment Frequency metric and add another one back. Now it seems to be derived.

[Henry Olson]

Hi @Ben Wozniak , you decide a Metric Source is derived (or not) at creation time, it can't be changed later. When creating a new Deployment Frequency Metric Source on a component, you'll want to supply `derived: true` to the input to set it to be derived, and then the values will only be calculated based on that component's deployment events. It's worth noting that creating a deployment frequency metric through the UI will always create a derived metric.

Here's an example of the GraphQL query and input needed to create a derived metric source:

mutation createMetricSource($input: CompassCreateMetricSourceInput!) {
 compass {
  createMetricSource(input: $input) {
   success
  }
 }
}

{
 "input": {
  "componentId": <your component ID goes here>,
  "metricDefinitionId": "ari:cloud:compass::metric-definition/builtin/weekly-deployment-frequency-28d",
  "externalMetricSourceId": "test123",
  "url": "https://google.com",
  "derived": true
 }
}

Once you have a derived metric source, you won't add metric values manually, they will be auto-calculated by Compass for you. 

[Henry Olson]

Glad you  figured it out @Ben Wozniak ! I agree, that's not clear and I'll add a ticket for us to improve the documentation and validation rules for that field.

[Ben Wozniak]

I think my first issue was that I was using the cURL command for a custom event, it looks like there are different payloads for things like a Deployment event. So I modified my cURL command to match the Deployment event in the documentation.

Now I am running the command and getting a format error for my build pipeline URL: "The value for the url field is invalid. Follow these validation rules: URL is invalid"

My URL is https://dev.azure.com/<ORG>/<PROJECT>/_git/<REPO>

Here is an example of the whole cURL now, with some secret info taken out:

curl \
--request POST \
--url https://example.atlassian.net/gateway/api/compass/v1/events \
--user "EMAIL:PAT" \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--data "{
\"cloudId\": \"ID\",
\"event\": {
\"deployment\": {
\"updateSequenceNumber\": \"1\",
\"displayName\": \"name\",
\"description\": \"description\",
\"url\": \"www.exampleUrl.com\",
\"lastUpdated\": \"$(date -u +'%Y-%m-%dT%H:%M:%SZ')\",
\"externalEventSourceId\": \"ID\",
\"deploymentProperties\": {
\"sequenceNumber\": \"1\",
\"state\": \"SUCCESSFUL\",
\"pipeline\": {
\"pipelineId\": \"ID\",
\"url\": \"https://dev.azure.com/<ORG>/<PROJECT>/_git/<REPO>\",
\"displayName\": \"Display\"
},
\"environment\": {
\"category\": \"DEVELOPMENT\",
\"displayName\": \"Auth Dev\",
\"environmentId\": \"Auth.Dev\"
}
}
}
}
}"

[Henry Olson]

@Ben Wozniak did you create the event source manually or was it created by a Forge app (i.e. Bitbucket)? It might help if you could share:

1. The details of that event source when you query it

2. The input you're providing to the createEvent mutation

Typically, I've seen this error when the wrong type of event is being pushed to the event source (like a Deployment event for an Incident event source) or if the event source was created by a Forge app and I was trying to add events to it manually.

[Ben Wozniak]

When I try to post to my new event, I get this error message:

`We could not create the event because the event source is not found. Please check that the external event source ID and the type of event being submitted match the event source.If a Forge App ID was used during event source creation, please also check you are using the same Forge App ID when accessing the event source.`

I've confirmed that my external ID matches the one in the created event source. I've also supplied the event source ID in the "customEventProperties", although the documentation does not specify that being necessary. I can't seem to figure out what field I have incorrect that Compass cannot find the event source.

[Henry Olson]

Yeah, it was a recent change (we still haven't done an official announcement post with dev documentation). But if you have any feedback on our implementation we'd love to hear it!

[Ben Wozniak]

Hi @Henry Olson ,

Do you have an example of the cURL request that would make a metric derived? I've tried with Deployment Frequency, but it always ends up using the value I supply. If I remove the "value" input then it removes any data.

[Pete Capra]

Oh, that's awesome! Thanks @Henry Olson