Grafana Alerting integration#

IncidentRelay can receive Grafana Alerting notifications through a dedicated webhook endpoint. Each Grafana alert instance is normalized, routed, grouped, deduplicated, and processed through the standard IncidentRelay alert lifecycle.

Endpoint#

CopyPOST /api/integrations/grafana

The endpoint requires the intake token of an active IncidentRelay route whose source is grafana.

CopyAuthorization: Bearer <route-intake-token>
Content-Type: application/json

Create an IncidentRelay route#

• Open Routes in IncidentRelay.
• Create a route or edit an existing one.
• Select Grafana as the source.
• Select the team that owns the alerts.
• Configure matchers, grouping, and the assignment target.
• Make sure the route is active.
• Copy the generated intake URL and route token.

A typical grouping configuration is:

Copy[
  "alertname",
  "grafana_folder",
  "instance"
]

Choose only labels that identify the logical incident. Avoid grouping by labels whose values change frequently.

Example route matchers:

Copy{
  "environment": "production",
  "team": "sre"
}

Grafana labels are available to route matchers after common labels and per-alert labels are merged.

Configure Grafana#

In Grafana:

• Open Alerts & IRM → Alerting → Notification configuration.
• Open the Contact points tab.
• Add a contact point.
• Select Webhook as the integration.
• Set the URL to the IncidentRelay Grafana endpoint:

``text https://incidentrelay.example.com/api/integrations/grafana ``

• Set the HTTP method to POST.
• In the authorization settings, set:
• Authentication Header Scheme: Bearer
• Authentication Header Credentials: the IncidentRelay route token
• Keep Disable resolved message disabled so IncidentRelay receives recovery notifications.
• Save the contact point.
• Use Grafana's test action to verify delivery.

Attach the contact point to the required Grafana notification policy.

Grafana provisioning example#

Grafana OSS/Enterprise can provision the contact point from a file under provisioning/alerting:

CopyapiVersion: 1

contactPoints:
  - orgId: 1
    name: IncidentRelay
    receivers:
      - uid: incidentrelay-webhook
        type: webhook
        disableResolveMessage: false
        settings:
          url: https://incidentrelay.example.com/api/integrations/grafana
          httpMethod: POST
          authorization_scheme: Bearer
          authorization_credentials: $INCIDENTRELAY_ROUTE_TOKEN

Set INCIDENTRELAY_ROUTE_TOKEN in the Grafana process/container environment instead of storing the token in the provisioning file. Keep disableResolveMessage: false so recovery events reach IncidentRelay.

Place the file, for example, at:

Copy/etc/grafana/provisioning/alerting/incidentrelay.yaml

Then restart Grafana or reload the provisioned alerting resources using Grafana's provisioning API. The contact point must still be selected by an alert rule or notification policy.

Recommended Grafana labels#

Add stable labels to Grafana alert rules so IncidentRelay can route and group them predictably.

Copylabels:
  team: sre
  environment: production
  severity: critical

Useful labels include:

The normalizer also recognizes oncall_team as a fallback team label and priority or level as fallback severity labels.

Alert normalization#

IncidentRelay processes every object in Grafana's alerts array independently.

Status#

The alert instance status takes precedence over the top-level notification status.

Resolved-like values are normalized to:

Copyresolved

Other values are treated as:

Copyfiring

This allows one Grafana notification group to contain alert instances with different states.

Title#

IncidentRelay selects the first available value from:

• annotations.summary
• labels.alertname
• top-level title
• Grafana alert

Message#

IncidentRelay selects the first available value from:

• annotations.description
• annotations.message
• top-level message
• valueString

Severity#

IncidentRelay selects the first available label from:

• severity
• priority
• level

Team#

IncidentRelay selects the first available value from:

• labels.team
• labels.oncall_team
• top-level team

Route matching remains authoritative. The label does not bypass normal route access or matcher checks.

Labels added by IncidentRelay#

When present in the Grafana payload, the integration exposes the following values as labels:

The first available Grafana link is also stored as event_link. The selection order is:

• dashboard URL
• panel URL
• alert rule URL
• silence URL
• Grafana base URL

Deduplication#

When Grafana provides fingerprint, IncidentRelay uses it as the deduplication key.

When fingerprint is missing, IncidentRelay generates a stable key from:

• Grafana source;
• alert rule UID when available;
• alert title;
• stable labels;
• Grafana organization ID.

A firing notification followed by a resolved notification with the same deduplication key updates the existing alert instead of creating another alert.

Stored payload#

IncidentRelay stores the original Grafana group context, but each normalized IncidentRelay alert keeps only its own Grafana alert instance in the stored alerts array.

This preserves fields such as:

• orgId;
• receiver;
• groupKey;
• common labels and annotations;
• dashboard and panel links;
• expression values;
• timestamps.

Example payload#

Copy{
  "receiver": "incidentrelay",
  "status": "firing",
  "orgId": 1,
  "groupKey": "{}:{alertname=\"DiskFull\"}",
  "commonLabels": {
    "team": "sre",
    "environment": "production"
  },
  "commonAnnotations": {
    "runbook_url": "https://example.com/runbooks/disk"
  },
  "externalURL": "https://grafana.example.com/",
  "title": "[FIRING:1] DiskFull",
  "state": "alerting",
  "message": "Grafana notification",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "DiskFull",
        "severity": "critical",
        "instance": "host1",
        "grafana_folder": "Infrastructure",
        "__alert_rule_uid__": "disk-full-rule"
      },
      "annotations": {
        "summary": "Disk is full",
        "description": "/var is 95% full"
      },
      "startsAt": "2026-06-21T10:00:00Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://grafana.example.com/alerting/grafana/disk-full-rule/view",
      "fingerprint": "grafana-disk-full-host1",
      "silenceURL": "https://grafana.example.com/alerting/silence/new",
      "dashboardURL": "https://grafana.example.com/d/system-overview",
      "panelURL": "https://grafana.example.com/d/system-overview?viewPanel=12",
      "values": {
        "A": 95
      },
      "valueString": "[ var='A' value=95 ]"
    }
  ]
}

Manual test#

Copycurl -X POST \
  "https://incidentrelay.example.com/api/integrations/grafana" \
  -H "Authorization: Bearer ROUTE_INTAKE_TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @grafana-payload.json

A successful response contains one result for every object in the Grafana alerts array.

Copy[
  {
    "created": true,
    "alert_id": 123,
    "group_id": 45,
    "status": "firing",
    "team_id": 2,
    "team_slug": "sre",
    "route_id": 7,
    "routing_error": null,
    "trace_id": "..."
  }
]

HTTP responses#

When routing fails, the response includes a trace_id. An administrator can use the alert explain trace to inspect matcher evaluation and the exact routing failure.

Troubleshooting#

Route intake token is required#

Grafana did not send an authorization header.

Verify:

CopyAuthentication Header Scheme: Bearer
Authentication Header Credentials: <route-intake-token>

Do not add the word Bearer to the credentials field when the scheme is configured separately.

Alert did not match any active route#

Check that:

• the route is active;
• the route source is grafana;
• the token belongs to that route;
• the incoming labels satisfy every configured matcher;
• the route's team is active and accessible.

Use the returned trace_id to inspect route evaluation.

Resolved alerts are not received#

Make sure Disable resolved message is not enabled in the Grafana webhook contact point.

IncidentRelay uses the status of each item in the alerts array, not only the top-level status.

Alerts are duplicated#

Check that Grafana sends a stable fingerprint. When the fingerprint is unavailable, keep rule UID and routing labels stable between firing and resolved notifications.

Do not include volatile values in labels used to identify an alert instance.

Dashboard or panel links are empty#

Grafana includes dashboard and panel links only when the alert rule is associated with the relevant dashboard and panel metadata. The alert rule URL remains available through generatorURL.

Test notification does not match production routing#

Grafana test payload labels can differ from real alert-rule labels. Compare the received labels in the explain trace with the route matchers.

Security#

• Use HTTPS for the webhook endpoint.
• Store route intake tokens as secrets.
• Use a separate route token for each integration or trust boundary.
• Rotate a token immediately if it is exposed.
• Do not place the token in the URL.
• Restrict route matchers so a leaked token cannot route unrelated alerts.