Scripting the Creation of integration filters, routing rules, etc.

Eric Sletten

We want to use OpsGenie to do what is currently done by some legacy operations functions.  Primarily, this is taking an email with the result of a scheduled job and perform some simple actions based on those results.  For example, if job XYZ123 fails, contact TEAM A's oncall person immediately or if job ABC789 fails, send an email to TEAM B so that it can be addressed in the morning.  Obviously this is easily accomplished on a small scale however, we have a "runbook", compiled over many years, that contains 1600+ individual tasks that may generate an email.  Below is an example of these instructions.  There are large groups of tasks that can be grouped together and be handled in the same way.  If we can come up with a way of condensing and refining these tasks and instructions  is there a way to script inputting into OpsGenie?  For example if group all the "Marketing" tasks that have instructions of "Contact the on-call programmer" together is there a way to script the input?

I may be thinking of it incorrectly too so other suggestions are welcome.  we just want to avoid making changes to the existing jobs (because there is a change control process that needs to be followed) and we don't want to manually input 1600 different sets of instructions.

 

Task Name	Team	Instructions
CRM_Employee_SSIS_Execution_WF	Marketing	1) Leave the task running and contact the on-call programmer.
CRM_CompanyAndContact-SSIS_Execution-WF with MIG_NBR	Marketing	1) Leave the task running and contact the on-call programmer.
PCActuarial MapVar ConnString	Actuarial	If task fails, contact programming according to call list
CHAIN_Task Checks And Balances Monthly Fact Verify	Commercial Health	1) Leave the task running to be reviewed by programmers in the morning 2) Forward failure email from Opswise to XXXXXXXXXX@XXXXXX
DBA_Reorg_Indexes_SQL	Information Services	​​Send an e-mail to DBA mailing list if there is an issue with this task
DataFlux Data Migration-WC7LineSchedule	Policy Processing	1) Do Not Call. 2) Send email to Data Migration programmer group (@POLCNV). 3) Leave the task running to be reviewed by programmers in the morning.

Robert

Hi @Eric Sletten ! 

 

tl;dr: Opsgenie's API and Infrastructure as Code plugins can help input this data. However, it will be difficult to implement at this scale without rethinking how the tasks are performed in Opsgenie first. 

 

In order to automate these tasks in Opsgenie, it'll be helpful to consider how Opsgenie segments these tasks for a given alert.

A team's routing rules, escalation policy, and schedule will determine who to notify when an alert is created. But Opsgenie leaves how they are notified up to individual users via notification rules or a central notification template. 

With that in mind, it may make sense to group notifications into modular "scenarios" where a tag ("email-only", "email-then-call") can be picked up by a notification rule downstream. This would change how your example is formatted out a bit:

 

1) Do Not Call.
2) Send email to Data Migration programmer group (@POLCNV).
3) Leave the task running to be reviewed by programmers in the morning.

could be rewritten as

1. Integration Action - Create alert
   1. Apply 'email-only' tag
   2. Assign POLCNV team as a responder
2. Leave the alert to be reviewed by programmers in the morning

Now the runbook only needs to define properties of the alert instead of the entire workflow. But there are still pieces of the Team and User configurations that this will take advantage of. 

 

Hope this helps!

P.S.: It's also worth mentioning that some of the recovery/diagnostic tasks could be scripted and automatically triggered for these alerts as well using Opsgenie Edge Connector, Opsgenie Actions, or any other outgoing integration.