Worked examples for IncidentRelay v2.3.1, checked on . Names, identifiers and timings below are illustrative; validate them in a test deployment before using production credentials.
Business example · Setup · Worked example · Acceptance checks · Limits and failure modes
Business example: checkout fails after the office closes
Illustrative scenario, not a customer case study. Metrics below are suggestions for your pilot, not measured savings or a claim that IR includes every reporting dashboard.
Situation and risk. An online store has four engineers. At 22:40, its monitoring detects failed payment requests while customers are still checking out. A message in the general chat does not tell support who is handling the problem. Unfinished purchases and a growing support queue are the business risk; the alert itself is only the signal.
Workflow with IR. Send the billing alert to the current primary responder. If nobody acknowledges it, the policy moves to the named backup. The assigned engineer acknowledges the group and investigates; support can use the agreed incident process to find that owner and obtain an update. Test this chain with the failure and recovery pair below.
Operational benefit to validate. Replace the manual search for an available engineer with a predefined ownership chain. Keep responsibility visible through the handoff rather than assuming that a delivered chat message means somebody is working on it.
What to measure. Record the time from the first actionable alert to ACK, the share acknowledged within your internal target and the number of manual reassignments. Separately track failed checkouts and the time until successful payments resume in commerce monitoring.
Boundary. IR does not repair checkout or a payment provider. An ACK does not prove that customers can pay again, and an escalation policy cannot help if neither engineer is reachable. Keep the fallback contact path and a real payment-recovery check.
Four engineers, one billing service, one ownership chain
Suppose four engineers share support for a billing API. Start with one team and one weekly primary rotation. Give the route a default service and one tested delivery channel. Add a backup person explicitly; a shared chat room is not an escalation policy.
| Object | Example setting in the UI |
|---|---|
| Ownership | Team payments; service billing-api; rotation payments-primary with four active members and 24/7 coverage. |
| Intake | Source webhook, default service billing-api, policy escalation mode and attached notification channels. |
| Escalation | Rule 1: primary rotation, delay 300 seconds. Rule 2: a named backup user, delay 600 seconds. Set repeat_count=0 for this pilot. |
Configure channels and responder contacts before sending an alert. Open the full On-call Health diagnostics: the green table summary alone does not scan future coverage or prove that a message reaches a phone.
Send one failure, then its recovery
Send these JSON bodies separately to POST /api/integrations/webhook with Content-Type: application/json and Authorization: Bearer ROUTE_TOKEN. Replace the placeholder with the test route token from the UI; keep it out of source control and shared logs. This is the generic webhook format, not a normalized Simulator event.
{
"status": "firing",
"fingerprint": "billing-api:health:prod",
"title": "Billing API health check failed",
"severity": "critical",
"labels": {
"alertname": "BillingUnavailable",
"environment": "production",
"service": "billing-api"
}
}{
"status": "resolved",
"fingerprint": "billing-api:health:prod",
"title": "Billing API health check failed",
"severity": "critical",
"labels": {
"alertname": "BillingUnavailable",
"environment": "production",
"service": "billing-api"
}
}Keep the same route, fingerprint and grouping labels for recovery. The fingerprint identifies the child alert; changing it to a new random value on every request defeats deduplication. An orphan recovery does not create a new group.
Prove the chain before a real night shift
- Send the firing body twice. Expect one child alert, the intended service and the current primary assignee; confirm actual delivery, not just an HTTP success.
- Leave a separate test alert unacknowledged. Check that the backup becomes the target after the first policy delay. Scheduler cadence, queueing and provider delivery mean this is not an exact five-minute delivery SLA.
- Acknowledge another test group and verify that pending firing-only notifications stop. Send recovery and confirm that the group resolves when no firing children remain.
- Create a temporary rotation override and repeat the test. Confirm that the override is the effective assignee, then remove the test configuration.
Know when the chain stops
In policy mode, team reminder-based escalation is ignored. After all rules and allowed repeats are exhausted, policy reminders stop; the alert can still be firing. A backup person without working contact details is only a name in a database.
ACK is ownership, not recovery of the billing service. IR cannot repair the application or supply a missing engineer. Keep an external way to reach the team if IR or its notification provider is unavailable, and agree on that fallback during onboarding.