IncidentRelay Documentation#

IncidentRelay is a self-hosted on-call scheduling, alert routing and notification service. It keeps teams, rotations, routes, notification channels, browser push subscriptions, acknowledgements, resolves, reminders and escalations inside your own infrastructure.

Alert flow#

Monitoring system
  -> Incoming integration endpoint
  -> Route intake token
  -> Route match
  -> Service
  -> Team and rotation
  -> Assigned on-call user
  -> Notification channels and profile browser push
  -> ACK / Resolve / Shelve

Browser push notifications are enabled by users in Profile and are delivered automatically to assigned users. They are not route channels.

Installation paths#

Choose one installation method:

Method	Recommended for	Start here
Docker Compose	quick start, testing, simple self-hosted deployments	Docker Installation
RPM package	RHEL, Rocky Linux, AlmaLinux, CentOS Stream	RPM Installation
Manual systemd	source checkout, custom Python environment, classic Linux VM	Manual systemd Installation

All production installations should run two processes:

incidentrelay             # HTTP API, UI, incoming webhooks
incidentrelay-scheduler   # reminders, escalations, periodic jobs

Do not run scheduler jobs inside every web worker.

Configuration#

IncidentRelay reads the config path from:

INCIDENTRELAY_CONFIG_FILE

Example:

export INCIDENTRELAY_CONFIG_FILE=/etc/incidentrelay/incidentrelay.conf

The old ONCALL_CONFIG_FILE name should not be used.

Core concepts#

Concept	Description
Group	Access boundary and group-level administration scope
User	Person who can log in, be on-call, receive notifications, enable browser push or use personal API tokens
Team	Operational unit inside a group
Rotation	On-call schedule for a team
Route	Alert routing rule with its own intake token
Channel	Outgoing notification target such as Mattermost, Telegram, email, webhook or voice call
Browser push	Profile-level browser/PWA notification delivery for assigned users
Alert	IncidentRelay alert created from an incoming integration
Silence	Rule that suppresses notifications for matching new alerts
Shelve	Temporary responder-controlled pause for one existing AlertGroup; technical status and impact continue
Override	Temporary replacement for a rotation member

RBAC summary#

IncidentRelay uses two permission layers:

Layer	Purpose
Group role	Defines access boundary and group-level user administration
Team role	Defines what a user can do inside a specific team

Current role names:

Group roles: viewer, editor, user_admin
Team roles:  viewer, responder, manager

Important rules:

A user must belong to a group before they can be added to a team in that group.
Adding a user to a team does not add the user to the group automatically.
user_admin can create users only inside the selected group boundary.
editor can create teams in a group, but does not automatically manage all teams.
manager is the write role for a specific team.
responder can acknowledge, resolve or temporarily shelve alerts without changing team settings.

Integrations#

IncidentRelay has two integration layers.

Incoming alert sources#

Source	Endpoint	Documentation
Alertmanager	`POST /api/integrations/alertmanager`	Alertmanager
AWS SNS/Cloud watch	`POST /api/integrations/aws-sns`	AWS SNS/Cloud watch
Grafana	`POST /api/integrations/grafana`	Grafana
RMON	`POST /api/integrations/rmon`	Grafana
Zabbix	`POST /api/integrations/zabbix`	Zabbix
Sentry	`POST /api/integrations/sentry/<route_id>`	Sentry
LibreNMS	`POST /api/integrations/librenms`	LibreNMS
Generic webhook	`POST /api/integrations/webhook`	Generic webhook

Incoming integrations use route intake tokens.

Notification delivery#

Delivery method	Documentation
Common channel behavior	Notification channels
Mattermost	Mattermost
Telegram	Telegram
Email	Email
Slack	docs/integrations/slack.md
Discord, Microsoft Teams, custom webhook	Webhook-based channels
Voice call	Voice call
Browser/PWA push	Browser Push Notifications

Notification channels do not have intake tokens. Routes receive alerts, then send notifications to attached channels. Browser push is profile-level and is sent to the assigned user's active browser devices.

Calendar sync#

Sync method	Best for	Documentation
CalDAV	Apple Calendar, Thunderbird, DAVx5 and other CalDAV clients	CalDAV Calendar Sync
ICS subscription feed	Outlook, Google Calendar and web calendar subscriptions	ICS Calendar Feed

CalDAV uses personal API tokens with the calendar:read scope. ICS calendar feeds use secret subscription URLs and do not require login.

Incident management#

Services#

Scheduling#

API and automation#

Swagger UI:

/docs

OpenAPI JSON:

/api/openapi.json

Useful pages:

First setup flow#

1. Install IncidentRelay
2. Configure the service and public_base_url
3. Configure browser push VAPID keys if browser/PWA notifications are required
4. Run migrations
5. Create the first global admin
6. Create a group
7. Create or add users to the group
8. Assign group roles: viewer, editor, user_admin
9. Create a team
10. Add group users to the team
11. Assign team roles: viewer, responder, manager
12. Create a rotation
13. Add rotation members
14. Create a service
15. Add service links and runbooks
16. Create notification channels
17. Ask users to enable browser push in Profile if required
18. Create a route and attach channels
19. Select a default service or configure service match rules
20. Copy the route intake token
21. Configure Alertmanager, Zabbix or webhook sender
22. Send a test alert
23. Acknowledge or resolve the alert

Project links#

Repository: https://github.com/roxy-wi/IncidentRelay
Swagger UI: /docs
OpenAPI JSON: /api/openapi.json