Worked examples for IncidentRelay v2.3.1, checked on . Names, identifiers and timings below are illustrative; validate them in a test deployment before using production credentials.
Business example · Setup · Worked example · Acceptance checks · Limits and failure modes
Business example: one nightly backup never finishes
Illustrative scenario, not a customer case study. Metrics below are suggestions for your pilot, not measured savings or a claim that IR includes every reporting dashboard.
Situation and risk. A company backs up its order database on two servers before the working day starts. One job stops silently; the other succeeds. A single shared success signal would hide the gap. The business risk is discovering an old or missing recovery point only when a restore is needed.
Workflow with IR. Define a completion deadline and track each expected server separately. A host sends its heartbeat only after the job and success checks finish. A missing completion creates an overdue alert routed to the platform responder, with a backup escalation path. The engineer investigates or reruns the job and records the outcome.
Operational benefit to validate. Turn a silent scheduled-job failure into an owned task before the next business shift. The same pattern can cover a scheduled export or batch job, provided that its completion signal genuinely represents success.
What to measure. Track missed deadlines per server, time from the deadline to acknowledgement and the age of the latest verified backup. Keep backup verification and restore-drill results in the backup system or operations report; do not infer them from a green heartbeat.
Boundary. A heartbeat proves that a signal arrived, not that the data can be restored. It does not guarantee a recovery point or recovery time. Test restores independently and monitor IR itself from outside the same failure domain.
Treat the paging system as a service you operate
Consider a platform team that owns a production cluster and nightly database backups. It wants IR near its monitoring stack, but that also makes the team responsible for the system that wakes it up. Name an owner for the web service, scheduler, database, secrets and recovery procedure.
The release Docker setup runs web and scheduler services separately. After following that deployment guide in a test environment, inspect them from the release checkout:
docker compose ps
docker compose logs --tail=50 incidentrelay
docker compose logs --tail=50 incidentrelay-schedulerThe web process accepts incoming alerts; the scheduler handles reminders, escalation and periodic jobs. Do not start a scheduler inside every web worker. A working login page does not prove that timed escalation is running.
Detect a missing backup on one host
Create a scheduled-completion heartbeat due daily at 03:00 in UTC, attached to a route whose source is heartbeat. Enable instance tracking; for a fixed fleet, use static expected instances such as db-1 and db-2. Attach the platform rotation or policy and test channels.
After a host's backup and its success checks finish, send this body to POST /api/heartbeats/ping/HEARTBEAT_TOKEN with JSON content type. Use that heartbeat's secret URL, not a route token or an administrative API token.
{
"status": "completed",
"instance": "db-1",
"payload": {
"job": "nightly-database-backup"
}
}Send a distinct instance value from each host. Without instance tracking, a successful ping from one server can hide another server's missing backup. Do not send a success ping unconditionally from a shell cleanup handler after a failed job. Redact secret ping URLs from proxy and job logs.
Test failure, not only startup
- Ping both expected instances, then deliberately omit one in a short test schedule. Expect an overdue alert for that instance while the healthy instance stays OK.
- Resume the missing instance. If auto-resolve is enabled, verify that its overdue alert resolves without resolving another instance's failure.
- Stop the scheduler only in the test deployment. Observe the effect on timed work and prove that an independent monitor notices it; then restore it and verify processing.
- Restore the database, configuration and required secrets into an isolated environment with outbound notifications blocked. Review old firing groups before allowing delivery, so a restore drill cannot page production responders.
A success ping is not a restore test
Heartbeats detect missing signals; they are not active HTTP/TCP probes and cannot prove that a backup is usable. IR also cannot reliably notify you of its own total outage. Keep an external availability check and a separate contact path outside the same failure domain.
Self-hosting does not remove chat, email, voice or network dependencies. Decide which database and deployment model fit the workload, pin the tested release, and rehearse upgrades and rollback. These examples do not establish high availability or a throughput guarantee.
Set retention deliberately: resolved alert history is retained indefinitely by default. Cleanup can also remove linked comments and delivery records, and it does not necessarily shrink the database file. Retention is not a substitute for backups.