Customizing alert notifications with Grafana OnCall templates.

Question

Pulumi · Accepted Answer

Customizing alert notifications in Grafana OnCall involves using the Grafana OnCall platform to manage incident response workflows, including defining how alerts are routed, who is notified, and how notifications are formatted. In Pulumi, this can be achieved by using resources such as grafana.OncallRoute, grafana.MessageTemplate, and grafana.ContactPoint to configure these settings as part of your infrastructure code.

Below is a sample program written in TypeScript using Pulumi's Grafana provider. This program sets up a custom message template and an on-call route that uses this template, and a contact point that defines where and how alerts are sent.

Message Template: Defines the format of the alert message.
Oncall Route: Specifies how incoming alerts are routed and which message template to use.
Contact Point: Determines where the alert is sent, like to Slack, email, etc.

Let's set up the program:

import * as grafana from "@pulumi/grafana";

// Create a new message template for alert notifications
const customMessageTemplate = new grafana.MessageTemplate("customMessage", {
    name: "HighPriorityAlert",
    template: "Alert: {{ .Title }} at {{ .OrgName }} is at critical level. Details: {{ .Message }}",
});

// Define an on-call route with a custom message template
const highPriorityRoute = new grafana.OncallRoute("highPriority", {
    routingType: "template",
    routingRegex: "^high$", // Regex to match alert tags/categories for routing
    integrationId: "my-integration-id", // Replace with your integration ID
    escalationChainId: "my-escalation-chain-id", // Replace with your escalation chain ID
    position: 1, // Specifies the order of the route evaluation
    slack: {
        enabled: true,
        channelId: "C1234567890", // Replace with your Slack channel ID
    },
    // Other notification channels like msteams, telegram can also be added here
});

// Create a contact point for sending notifications to a team channel on Slack
const slackContactPoint = new grafana.ContactPoint("slackContact", {
    name: "TeamChannel",
    slacks: [{
        uid: "my-contact-point-uid", // Replace with your contact point UID
        url: "<your_slack_webhook_url>", // Replace with your Slack webhook URL
        // Additional settings like 'token', 'recipient', 'mentionGroups' etc. can also be specified
    }],
});

// Export the IDs of the resources
export const messageTemplateId = customMessageTemplate.id;
export const oncallRouteId = highPriorityRoute.id;
export const contactPointId = slackContactPoint.id;

In the above program:

We create a message template called customMessageTemplate with a simple structure that includes placeholders for alert title, organization name, and message content. These placeholders will be replaced by the actual alert data at runtime.
We then define an on-call route highPriorityRoute that utilizes the custom message template for alerts matching a high priority regex pattern. This is controlled by the routingRegex property.
Finally, a slackContactPoint is defined to send the formatted alert messages to a specified Slack channel.

Remember to replace placeholder values like my-integration-id, my-escalation-chain-id, C1234567890, my-contact-point-uid, and <your_slack_webhook_url> with the actual values from your setup.

After deploying this code with Pulumi, Grafana OnCall will route alerts tagged as high priority through the defined route, format them using the custom message template, and send the notification to the specified Slack channel.

Documentation for the resources used:

Please replace the placeholders with actual data from your infrastructure before running the program. This program is an infrastructure-as-code representation of what you would otherwise set up manually in the Grafana OnCall UI. Using Pulumi allows you to keep track of these configurations in code, version control them, and apply consistent configurations across different environments.