Prometheus Alertmanager Alert priority is not being parsed correctly

Question

Hi there,

I am running prometheus community in k8s using helm, and have my alert rules and alertmanager files as values I pass to helm during upgrade.  As such, I am having difficulty using the examples listed here, particularly the following error;

"{\"message\":\"Request body is not processable. Please check the errors.\",\"errors\":{\"priority\":\"should be one of [ P1, P2, │
│ 3, P4, P5 []\"},\"took\":0.001,\"requestId\":\"ca7xxx3a0\"}"

I have tried escaping the curly brackets, using square brackets, single quotes, double quotes and many other syntax variations however I can't seem to get past this hurdle.  What would be really useful is to be able to see how OpsGenie is receiving the payload, so I can get more feedback to my changes.

Is there a way to see what the 'priority' key is being set to?

I have tried using the requests API but it doesn't give me the .Alert payload.

My alerts have this configured;

labels:
severity: info

And the alertmanager config at the mo is;

priority: '{{ range .Alerts }}{{ if eq .Labels.severity "critical" }}P1{{ else }}P2{{- end -}}'

I've also tried setting the label key as 'priority' instead of 'severity', and this label is available in the alertmanager TSDB, but OpsGenie is still not seeing it.

Thanks!

Swapna

Connor Eyles · Answer

Hi @Swapna

That makes a lot more sense here!

Appreciate you sharing your findings here in relation to the grouping and why it's not setting the priority correctly as the range of alerts are being concatenated.

You are correct in your understanding here surrounding grouping, what I can see from the prometheus documentation

To aggregate by all possible labels use the special value '...' as the sole label name, for example:
# group_by: ['...']
# This effectively disables aggregation entirely, passing through all
# alerts as-is.

Thanks,

Connor

Swapna_ivdb4t · Answer

Hi @Connor Eyles

Just an update, I was playing around again trying to get this working, and was about to go down the whole 'Relabel Config' route, when I came across this post, which I had seen before but it made more sense to me now;

https://github.com/prometheus/alertmanager/issues/2088

He is saying that by having the {{ range .Alerts }} at the beginning, it will cycle over the multiple alerts that fired in the given interval, and print the priority level for each alert.. so they will get concatenated, i.e. you'll get a long string like 'P1P1P2P3P2' instead of just P1 etc.

So I tried it with this new config;

priority: '{{ if eq .GroupLabels.severity "critical" }}P1{{ else if eq .GroupLabels.severity "warning" }}P2{{ else }}P3{{- end -}}'

And also I had to group by severity (I was already grouping by alertname and environment);

route:
group_by: [alertname, environment, severity]

And now, I am able to see alerts with priority levels set by the label.severity key value! :D

Do you think that your example code works because of the group labels [...]?  I believe this means that alerts are not grouped, and so each alert comes in individually (thus creating a lot more noise) and so the concatenation doesn't occur?  In which case your code will also probably work without the {{ range .Alerts }}?

Thanks fo all your help!

Swapna

Swapna_ivdb4t · Answer

Hi @Connor Eyles

Thanks for the suggestion, I will give that a go, it looks very useful!

Regards,

Swapna