Worked examples for IncidentRelay v2.3.1, checked on . Names, identifiers and timings below are illustrative; validate them in a test deployment before using production credentials.
Business example · Setup · Worked example · Acceptance checks · Limits and failure modes
Business example: three client systems, one noisy support chat
Illustrative scenario, not a customer case study. Metrics below are suggestions for your pilot, not measured savings or a claim that IR includes every reporting dashboard.
Situation and risk. A service provider supports three clients, each with application and infrastructure alerts. Everything lands in one chat. Engineers forward messages to other teams while repeated alerts bury the original report. The business risk is delayed ownership and avoidable support effort, even when the monitoring detected the failure promptly.
Workflow with IR. Map each client service to its responsible team and use the intended route token for each sender. Match environment and service labels, choose the on-call target and group repeats only within the intended ownership boundary. Pilot one client's production billing service using the matcher below before extending the pattern.
Operational benefit to validate. Replace routine message forwarding with explicit routing. Keep the first response with the team that can act, while preserving separate child alerts when several hosts report the same problem.
What to measure. Count alerts reassigned after the first routing decision, groups without an owner and notifications per underlying fault for each client. Review rejected or unmatched test events too: fewer pages is not a success if a client's outage becomes invisible.
Boundary. Label matching is not customer data isolation. Review access permissions, credentials and deployment boundaries separately. Inconsistent labels or overly broad grouping can still send work to the wrong team or combine unrelated symptoms; validate those cases before onboarding another client.
Separate intake, ownership and delivery
A production billing service and a staging copy should not wake the same people by accident. For the first pilot, create a Webhook route for the production billing team, select its default service and rotation or policy, and attach channels. Its intake token selects the route; matching labels alone do not authenticate a sender or select an arbitrary route.
A narrow route matcher and an explicit grouping key
Paste this into the route's matcher editor. It is a matcher object, not the incoming webhook body. All conditions use AND semantics. Requiring production and service labels keeps staging and unlabelled events out of this route.
{
"source": "webhook",
"labels": {
"environment": "production",
"service": "billing-api"
}
}For this example, set the route's group_by field to:
["alertname", "severity", "labels.cluster"]Two hosts reporting the same alert in the same cluster can then share an AlertGroup, while keeping separate child identities. For a generic webhook, use stable fingerprints such as billing:node-1:health and billing:node-2:health. Include the expected labels on failure and recovery, and never use a new timestamp as the identity of every retry.
Deduplication and grouping are different: the same fingerprint updates a child; different fingerprints with the same group key create separate children in one group. Route, team, source and service also scope grouping. Add instance to the grouping fields only if each host should be a separate group.
When one sender really serves several teams
Use global orchestration when ownership must be chosen from event content. In the Builder, start with conditions labels.environment equals production and labels.service equals billing-api, then select valid team, route and service objects from the same group. Do not copy numeric object IDs between installations.
Validate and simulate both matches and non-matches, publish, then use shadow with hybrid compatibility. Review real executions before switching to active. Shadow records decisions without changing production behavior; the Simulator does not prove end-to-end delivery.
Use a small routing test matrix
| Input | Expected result for this example |
|---|---|
| Production billing event | Matcher passes; selected service, assignee and channels match the planned ownership. |
| Staging or missing service label | Matcher does not pass. Confirm the sender's intended alternative path; this example does not create a catch-all route. |
| Duplicate fingerprint | The same child is updated. A different host fingerprint creates a second child in the same group when grouping labels match. |
| Recovery for only one of two children | That child resolves; the group stays firing until all firing children recover. An orphan recovery creates no new group. |
Avoid rules that silently erase the evidence
An empty matcher matches everything. A label named severity is not automatically the same field as normalized event severity; use the documented field path and inspect a real normalized event. Do not require status=firing on an intake path that also needs to accept recovery.
active + legacy does not apply orchestration decisions. In hybrid, invalid entity combinations can fall back to the legacy path; authoritative orchestration mode instead blocks processing on evaluation or entity-validation failures. Choose the failure behavior intentionally.
Start without broad drop or suppress rules. Suppression and grouping can reduce notifications, but neither makes a bad monitoring signal useful. Keep a tested previous version and review unexpected non-matches before expanding the rollout.