IncidentRelay#

IncidentRelay is a self-hosted on-call scheduling, alert routing and notification service.

It provides:

• groups and RBAC;
• teams and rotations;
• service directory with affected systems, links, runbooks, dependencies and impact analytics;
• route-based alert intake tokens;
• Alertmanager, Zabbix and generic webhook intake;
• Mattermost, Telegram, email, webhook-based and voice call notification channels;
• ACK, Resolve and Alert Shelving workflows;
• temporary per-AlertGroup shelving with automatic expiry and resume;
• reminders and escalation;
• silences and rotation overrides;
• calendar view;
• personal API tokens;
• Swagger/OpenAPI documentation.

Core workflow#

CopyMonitoring system -> Route -> Service -> Team -> Rotation -> Notification channels -> ACK / Resolve / Shelve

Routes decide how alerts enter IncidentRelay. Services describe what logical system is affected.

Service and team display order:

Copyname -> slug -> "-"

Installation#

Choose one method:

Runtime services#

IncidentRelay should run as separate services:

Copyincidentrelay           # web API, UI, incoming webhooks
incidentrelay-scheduler # reminders, escalations, periodic jobs

Telegram worker is optional and only needed when Telegram polling/actions are used.

Configuration#

IncidentRelay reads config path from:

CopyINCIDENTRELAY_CONFIG_FILE

Example:

Copyexport INCIDENTRELAY_CONFIG_FILE=/etc/incidentrelay/incidentrelay.conf

Do not use the old ONCALL_CONFIG_FILE name.

Database migrations#

Copypython manage.py migrate

Create first admin#

Copypython manage.py create-admin   --username admin   --password 'change-me-123'   --email admin@example.com

First setup flow#

Copy1. Create a group
2. Create users
3. Add users to the group
4. Create a team
5. Add users to the team
6. Create a rotation
7. Add rotation members
8. Create a service
9. Add service links and runbooks
10. Create notification channels
11. Create a route and select default service
12. Copy the route intake token
13. Configure Alertmanager, Zabbix, or webhook sender
14. Send a test alert
15. Acknowledge or resolve the alert

Services#

A service describes the affected system, for example:

CopyRabbitMQ Cloud
Billing API
PostgreSQL Prod
Frontend Web

A service can have:

• dashboard, logs, traces, repository and documentation links;
• generic runbooks;
• alert-specific runbooks selected by matchers;
• dependencies;
• analytics and impact status.

Runbook matcher behavior:

Copyempty matchers -> generic runbook for all alerts of the service
matchers set   -> runbook only for matching alerts

Email notifications#

Email channel does not store recipients or SMTP transport settings.

• SMTP is configured globally in the config file.
• Emails are sent to the assigned user's profile email address.
• Email channel can optionally override the HTML template.

Reminder intervals#

Reminder interval is configured on rotations:

Copy0 disables reminders
>= 60 enables reminders
1..59 invalid

API#

Swagger UI:

Copy/docs

OpenAPI JSON:

Copy/api/openapi.json

Service API endpoints are available under:

Copy/api/services

Documentation#

License#

MIT