What is available to troubleshoot where a webhook alert failed (i.e. wasn't fired, rejected, etc)?

[Ece Akturk]

Hello Atlassian Community!

I am working on sorting out an issue with our sole webhook subscriber for our Statuspage and have a primary question about troubleshooting that I couldn't find answers to after scouring the documentation and community Q's and A's. Following the breadcrumbs of course lead to 3 sort of sub-questions that I couldn't find definitive answers on.

My primary question is around troubleshooting (general) of the webhook notification deliveries. As I mentioned I have a customer who has both an email subscription and a webhook subscription and they're not receiving the webhook subscriptions. I checked all the things I could think of (the webhook isn't quarantined, exported subscriber list to see if there was any additional hidden data, webhook subscriptions are enabled, etc). I was curious if there's any additional logging anywhere or how we can verify that the the webhook notifications actually went out. If they're not receiving the notifications I would think it means that either the notification never went out or that there's a problem with the url but if the url is bad then I would imagine it should've gotten quarantined after the retries. 

And now for the questions from deep down the rabbit hole!

1. I found this question where the answer from Atlassian indicated that Statuspage doesn't honor case sensitivity in webhook subscription url's - at least not as of July 2018 - is this still the case? My customer has some caps in their URL and they did confirm that the url is case sensitive.
2. In response to this question about Microsoft Teams the answer indicated that (at least for this integration) direct webhook subscriptions from Microsoft Teams aren't supported but would need to be configured from the third party app integration. Is this true for all third party app integrations, or just this one? And does this only apply to integrating your own MST with your own StatusPage or does the limitation also apply to your customers who want to consume webhooks from your Statuspage into their third party app (I'm especially interested in PagerDuty)?
3. In this article it was indicated that as of November 2019 double opt-in would be required to subscribe to a Statuspage. The title indicates that this is for email, sms and webhooks, but the article really only talks about sms. If so, is the delivery of the email associated with the webhook subscription considered the double opt in? 
4. I noticed that by default you seem to use sendgrid for delivering the alerts - do we have access to the backend sendgrid account used for our notifications (to see delivered status, rejected, spam, opens, clicks, etc)? 

Apologies if this is too much for a post, but I thought other subscribers might be interested in the answers. Thanks in advance for any and all help!

[Jesse Klein · Accepted solution]

Hello Ece,

This is Jesse from the Statuspage support team. Thank you for submitting a community ticket with questions around webhooks. No worries on the block of text. We're happy to make sure you're taken care of and having it all in one place is awesome. Let me see if I can help answer your questions.

Generally speaking, I do all my testing with webhook.site. You can quickly create a webhook to use with Statuspage and test subscriptions. As for your subquestions, I will match the number of your question with my answer:

1. I just tested this case sensitivity with webhook addresses and it appears that the ARE case sensitive now. I am unsure when this change was made but it seems case sensitivity is part of it as I haven't gotten my first request on it after a few mins.

2. As far as webhooks go, in general we will work with everything but we use a basic form of webhook and so things like MS Teams and Slack don't work, which is why we have the apps built out. Unfortunately, we are not entirely sure which applications would take the kinds of webhooks but if we have an app for it, typically that would imply that our webhook is not compatible. My apologies for not having more information on that.

3. The double opt in for this really us mentioning SMS requiring the YES statement when a text is sent to them. The rest of the article talks about the API, which affects the other subscriptions as well but there is not double opt in for webhooks. You just need to provide an email in case the webhook breaks. If it can no longer deliver, it should send a notification and send the webhook to quarantine.

4. Unfortunately, customers do not have access to this at this time but we would be happy to look if you submit a ticket with us.

Hopefully that helps you out with your questions. If you have any others, I would be happy to assist. I hope you have a great day!

Regards,

Jesse

[Ece Akturk]

Hi Jesse - thanks so much for the thorough and quick reply! I really appreciate it! 

I know how to test webhooks myself (like general testing), but I was just curious if there were any more diagnostic tools available that I was missing, like a paper trail of events, logs, diagnostic api calls, etc that I wasn't seeing :) 

My money was on the case sensitivity still being a factor from my search so thanks for confirming this (I needed a little W today!). So, just to confirm, case sensitivity is still not honored so webhook URL's need to be lower case if the tool ingesting the webhook is case sensitive, yes? 

Thanks again Jesse - I really appreciate your help!

Ece

[Ece Akturk]

Jesse - 

I almost forgot! I did have *one* more follow up question (as I always do!). In terms of quarantining a webhook - I read in the documentation that a webhook will get quarantined after 10 failed delivery attempts in an hour and then Statuspage sends an email to the email address associated with the webhook to let the subscriber know.

What qualifies as a failed delivery attempt? I'm guessing that the webhook has to actually provide a response in order to be failed because in the case of our customer whose webhook url turned out to be invalid (wrong) and in the case of your case sensitive test, there was no endpoint so there would be no response, just a timeout.

I'm guessing this doesn't qualify as "failed" delivery? If not, what does? 

Thanks again, Jesse! I really want to tell you that this is my last follow up question but I make no promises! :) 

Ece

[Jesse Klein]

Hi Ece,

Always happy to help for sure! Just to clarify. It is case sensitive. So having a webhook with "test" in the url is different from "TeSt" Hopefully that clears it up!

In regards to the other question, you are correct. This came up in my testing as well. If I entered the webhook and changed a letter, it did not send it to quarantine ever. From some investigating, it seems that webhooks go to quarantine if there is a blocker, like a firewall. 

And no worries if you have more questions. If you have an active paid for page, you can also always feel free to open a ticket with us. I hope you have a wonderful day!

Regards,
Jesse

[Ece Akturk]

Thanks Jesse! This was super helpful!