If you’ve ever been tasked with writing a status update when things are on fire you know that it’s hard. It’s often urgent, stressful, and requires conveying clear and accurate communication (often before you’re clear on the details yourself).
What makes one-to-many communication so much harder than one-to-one communication? And how can we ease this angst for folks tasked with the job of getting the word out to customers when things go wrong?
While (thankfully) incidents are rare for FullStory, Ben and his team have developed helpful practices to ensure downtime doesn’t equal disaster for their support team or their customers. He was kind enough to sit down with us and share their techniques for combating stress and writer’s block during an incident:
When things are on fire, you don’t want to waste time determining whether you should communicate the problem to customers. This ‘should we/shouldn’t we debate is harmful in a couple of ways: 1) It keeps customers in the dark longer than necessary if you end up determining that comms are needed and 2) It can cause internal confusion and debate that could have been sorted out before an incident, saving your team time and strife.
Ben’s team recognized this and decided to create a “Statuspage Constitution.” This constitution is essentially a document that lists out things that must be true in order for the team to post to Statuspage during an incident. The questions focus upon incident severity, expected duration, and customer impact:
For each bullet point, the Statuspage Constitution includes further definition (e.g. what does “large number of customers mean”) and examples of what might qualify or be disqualified. It’s a lot of work up front, but the added clarity lets them move fast if something comes up.
As soon as the team is alerted about an issue, they quickly answer the questions together in Slack. If they determine that communication is necessary, they quickly spring into action and get the right people on the communication front lines.
It’s easy to let communication fall to the wayside during incident response. The dev team is focused on fixing the problem, while the support team is trying to handle the surge in inbound tickets or emails coming in.
The FullStory team ensures communication remains front and center by embedding their support team into their on-call process. There are always support team members on-call alongside the dev team, and at least one ‘Incident Hugger’ gets paged when comms are needed. The Incident Hugger’s job is to be heads down focused on customer communication, allowing engineers to stay focused on resolving the incident.
When you work on a support team, communicating with customers on support tickets becomes second nature. Since incidents (hopefully) aren’t an everyday thing, status update practice is not inherent in the role.
As Ben told us:
To actually sit down and write a status update – even if you’re an expert at customer communication – is such a different type of communication and audience you’re trying to speak to. You can’t rely on your expertise in other types of comms.
That’s why setting aside time to practice writing incident updates is so crucial. They recommend holding mock incidents or fire drills to get your team comfortable with writing under pressure. Ben’s team is even working on a series of playbooks for incident response which will include incident communication fire drills.
Even the most practiced team will encounter stressful situations that they may not feel ready for. In this case, Ben recommends pausing, taking a breath, and refocusing on your goal. For the FullStory team, the goal is to deliver the quick and accurate information to customers, while precluding additional follow-up. If they are trying as hard as they can to meet this goal, the rest should fall into place.
💡How do you ease the pain of incident communication? Have any of Ben’s tips worked well for you? Let us know in the comments.
I’m Serhat Can, Technical Evangelist at Atlassian and the author of “On Call”. I’ve spent years participating in and helping customers build on-call programs. I recently compiled all my insights into...
Connect with like-minded Atlassian users at free events near you!Find an event
Connect with like-minded Atlassian users at free events near you!
Unfortunately there are no Community Events near you at the moment.Host an event
You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events