Configuration#

IncidentRelay reads the config file path from:

CopyINCIDENTRELAY_CONFIG_FILE

Example:

Copyexport INCIDENTRELAY_CONFIG_FILE=/etc/incidentrelay/incidentrelay.conf

For systemd:

CopyEnvironment=INCIDENTRELAY_CONFIG_FILE=/etc/incidentrelay/incidentrelay.conf

For Docker Compose:

Copyenvironment:
  INCIDENTRELAY_CONFIG_FILE: /etc/incidentrelay/incidentrelay.conf

The old ONCALL_CONFIG_FILE name should not be used.

Environment overrides#

Any option from the config file can also be set with an environment variable named INCIDENTRELAY__<SECTION>__<OPTION>, with the section and option in upper case. It takes precedence over the file:

Copyexport INCIDENTRELAY__DATABASE__PASSWORD=replace-with-the-database-password
export INCIDENTRELAY__LOGGING__LEVEL=DEBUG

To read the value from a file instead, add __FILE to the name and pass the path. Use this for Docker or Kubernetes secrets and Secrets Store CSI volumes, so secrets never have to be written into incidentrelay.conf:

Copyexport INCIDENTRELAY__DATABASE__PASSWORD__FILE=/run/secrets/db-password

A trailing newline in the file is ignored. Setting both a variable and its __FILE form is an error. For systemd, put the variables into an EnvironmentFile=.

When main.secret_key comes from the environment, empty main.secret_encryption_key, auth.jwt_secret, mattermost.action_secret and voice.callback_secret fall back to it instead of being generated by the Docker entrypoint.

main.secret_encryption_key encrypts the secrets stored in the database, so it must not change once IncidentRelay has used it, whether it comes from the config file, an environment variable or a __FILE. When it is empty, IncidentRelay encrypts with main.secret_key instead, and the same applies to that key. If the encryption key changes, the Docker entrypoint refuses to start: restore the previous key. Changing the key directly is not supported. The entrypoint compares the key with a salted hash it keeps in /var/lib/incidentrelay, so the check only works when that directory persists between restarts.

In [voice_provider], whose options depend on the provider, a variable only overrides an option that is present in the file, and the option keeps its spelling from the file. Variables don't add new options there:

Copy[voice_provider]
apiKey =

Copyexport INCIDENTRELAY__VOICE_PROVIDER__APIKEY=replace-with-the-api-key

Provider options can also reference a variable directly, for example api_token = ${VOICE_API_TOKEN}; see Voice Provider Configuration.

Main and authentication secrets#

Generate two different random values and keep them stable across restarts and upgrades:

Copyopenssl rand -hex 32
openssl rand -hex 32

Copy[main]
secret_key = replace-with-the-first-random-value

[auth]
jwt_secret = replace-with-the-second-random-value
jwt_cookie_secure = true

secret_key belongs to [main], not [server]. jwt_secret belongs to [auth]. Do not reuse the same value for both settings. Set jwt_cookie_secure = true when public_base_url uses HTTPS.

Server section#

Copy[server]
host = 0.0.0.0
port = 8080
public_base_url = https://incidentrelay.example.com

In production, public_base_url must be the real external HTTPS URL.

Database: SQLite#

Copy[database]
type = sqlite
name = /var/lib/incidentrelay/incidentrelay.db

[sqlite]
wal = true
busy_timeout = 5000

SQLite is suitable for small self-hosted installations. Keep one web worker when using SQLite.

Database: PostgreSQL#

Copy[database]
type = postgresql
host = 127.0.0.1
port = 5432
name = incidentrelay
user = incidentrelay
password = change-me

Use PostgreSQL for larger installations, higher alert volume, multiple web workers, or long-term production deployments.

Outbound HTTP network policy#

IncidentRelay protects administrator-configured outbound HTTP requests against server-side request forgery (SSRF). Private, loopback, link-local, multicast, reserved and unspecified destination addresses are blocked by default.

The policy is controlled by the [security] section:

Copy[security]
outbound_private_network_allowlist =
outbound_http_max_redirects = 3
outbound_http_max_response_bytes = 1048576

outbound_private_network_allowlist is a comma- or semicolon-separated list of IPv4/IPv6 addresses and CIDR networks that IncidentRelay is explicitly allowed to contact when they are otherwise considered private or unsafe.

Examples:

Copy# One internal service only.
outbound_private_network_allowlist = 192.168.50.10/32

Copy# Several approved internal networks/addresses.
outbound_private_network_allowlist = 10.20.0.0/16,192.168.50.10/32,fd00:1234::/48

A single IP may also be written without the prefix length, but /32 for IPv4 and /128 for IPv6 make the intended scope explicit. Prefer the narrowest possible entries instead of allowlisting whole private address ranges.

This policy is used by the shared outbound HTTP client, including OIDC metadata and JWKS retrieval and outgoing integrations such as generic/Teams/Discord webhooks, Slack webhooks and Mattermost API requests. The list is not a hostname allowlist: IncidentRelay resolves the hostname first and checks the resolved IP addresses.

For DNS names, every address returned by DNS must be public or explicitly allowlisted. If even one returned address is blocked, the request fails closed. Redirect targets are resolved and checked again before IncidentRelay follows them.

For example, if an internal identity provider resolves to 10.42.7.15:

Copy[security]
outbound_private_network_allowlist = 10.42.7.15/32

After changing this setting, restart every IncidentRelay process that can make outbound requests.

The allowlist changes only the destination network policy. It does not disable HTTPS certificate verification or trust a private certificate authority. Internal HTTPS endpoints using a private CA must also have that CA installed in the operating system/container trust store.

Alert processing trace#

The global Explain Trace detail level is configured in [alerts]:

Copy[alerts]
explain_trace_level = full

Supported values are full, compact and disabled. full preserves the existing detailed trace. compact keeps ordered processing steps but omits input_summary, result payloads and per-step data. disabled stores no Alert Explain Trace rows. A global Event Orchestration rule can override this value for matching events with the set_trace_level action.

Alert event history#

Incoming child-alert history can be controlled independently from alert lifecycle processing:

Copy[alerts]
event_history = full

Supported values are full, initial and disabled. full stores incoming child-alert created, updated and resolved events. initial stores only the initial child-alert created event. disabled stores none of those incoming child-alert history rows. Alert state updates, grouping, notifications, escalation and resolution continue normally in every mode. Operational incident timeline entries such as acknowledgements, comments, reminders, maintenance, correlations, responders and stakeholders are always preserved.

Global and service Event Orchestration can override the configured default for matching events with set_alert_event_history. If several matching actions set a level, the last applied action wins.

Alert shelving scheduler#

Timed AlertGroup shelves are expired by the scheduler:

Copy[alerts]
shelve_lifecycle_check_interval_seconds = 30
shelve_lifecycle_batch_size = 100

check_interval_seconds controls how often due shelves are reconciled. batch_size bounds one scheduler pass. Keep the scheduler service running whenever timed shelving is used. Shelving pauses notifications, reminders and escalation but does not change the AlertGroup technical status or service/business impact. See Alert shelving.

Data retention#

IncidentRelay 2.1 keeps retention settings in one section:

Copy[retention]
alert_days = 30
# explain_trace_days = 30
# orchestration_execution_days = 30
cleanup_interval_seconds = 86400
batch_size = 500

alert_days = 0 is the default and keeps resolved alert history indefinitely. Explain Trace and general Event Orchestration execution retention inherit alert_days unless their optional override is set. See Data Retention for deletion rules and upgrade compatibility.

SMTP section#

Email notification channels use global SMTP settings. SMTP transport is not configured per channel.

Copy[smtp]
host = 127.0.0.1
port = 25
from = incidentrelay@example.com
use_tls = false
user =
password =

For an unauthenticated local relay, leave user and password empty.

For an authenticated SMTP server:

Copy[smtp]
host = smtp.example.com
port = 587
from = incidentrelay@example.com
use_tls = true
user = incidentrelay@example.com
password = change-me

Email notifications are sent to the assigned user's profile email address.

Telegram proxy#

If the environment requires a proxy for Telegram Bot API calls, configure it globally. Keep token values in channel configuration, not in the global config.

Example option names depend on the current service config implementation. Use the same config file for web and Telegram worker processes.

Voice section#

Copy[voice]
provider = stub
providers_dir = /usr/local/lib/incidentrelay/voice_providers
callback_secret =

Voice call notifications are sent to the assigned user's profile phone number.

Browser push section#

Browser push notifications are profile-level PWA/browser notifications. They are not configured as notification channels.

Copy[browser_push]
enabled = true
vapid_public_key = CHANGE_ME_PUBLIC_KEY
vapid_private_key = /etc/incidentrelay/vapid/private_key.pem
vapid_subject = mailto:admin@example.com
action_token_ttl_seconds = 900

After changing browser push settings, restart the web service. Restart the scheduler too if it sends notifications in your installation.

Read more: Browser Push.

Metrics section#

IncidentRelay can serve Prometheus metrics at GET /metrics:

Copy[metrics]
enabled = true
auth_token = replace-with-a-random-value

The endpoint is disabled by default. Enable it only when something actually scrapes it, and protect the port at the network layer the same way you protect /healthz.

Generate a value for auth_token with openssl rand -hex 32. Example Prometheus scrape configuration:

Copyscrape_configs:
  - job_name: web
    scheme: https
    metrics_path: /metrics
    bearer_token_file: /etc/prometheus/incidentrelay.token
    static_configs:
      - targets:
          - incidentrelay.example.com

Exported metrics:

incidentrelay_database_up and incidentrelay_migrations_pending are evaluated at scrape time and repeat the /readyz checks. /metrics keeps answering while the database is down, so both values stay visible during an outage. While the migration state cannot be read (database down or migration check failed), the incidentrelay_migrations_pending sample is absent rather than reading 0 — it is only reported when it is actually known. Alert on incidentrelay_database_up = 0 instead of the pending count alone.

Request counting is active only while enabled = true. After changing metrics settings, restart the web service and the scheduler and worker services too — they pick up the setting when they start. Only the web service serves /metrics; the scheduler and the other workers only record into it.

IncidentRelay normally runs several processes behind one scrape target: multiple Gunicorn web workers plus the scheduler and the Telegram/Slack workers as separate daemons. So that one /metrics scrape covers them all, every IncidentRelay process records its counters into one shared directory configured through the PROMETHEUS_MULTIPROC_DIR environment variable, and the web workers merge all processes' files at scrape time. The packaged deployments set this up already: the systemd units share /run/incidentrelay/metrics, so a scrape covers every unit on the host — web workers, scheduler and chat workers alike. Without the variable — single-process development runs — the counters cover that one process only.

The counter files are keyed by process PID, so a PROMETHEUS_MULTIPROC_DIR may only be shared by processes that see the same PIDs: those on one host, or inside one container. Never point containers at the same directory — independent PID namespaces can reuse the same PIDs and corrupt each other's files. The Docker image therefore gives every service container its own directory under /var/lib/incidentrelay/metrics/; a containerized /metrics scrape aggregates the web container's workers, while scheduler and chat-worker transitions land in their own directories and are not visible to it. To merge them you would need an aggregation mechanism that crosses container boundaries, which this feature does not provide.

If you point PROMETHEUS_MULTIPROC_DIR at your own location, keep it to processes of one host or one container; it must exist and be writable before the processes start. While [metrics] is disabled the directory is not touched, so a broken directory cannot block startup. When a process exits, its live-gauge files are removed the next time an IncidentRelay process starts. Its counter and histogram files are deliberately kept: deleting one would drop the dead process's share from the merged counter, which Prometheus reads as a counter reset that rate() and increase() distort. Stale counter files keep contributing until the shared directory is cleared on a full redeploy, and their number is bounded by how many distinct PIDs the directory has ever seen. Gauge cleanup checks liveness by PID, so a recycled PID keeps a dead process's gauge file until the process that now owns that PID exits.

The database-derived gauges (incidentrelay_database_up, incidentrelay_migrations_pending, incidentrelay_user_notification_deliveries_recent, incidentrelay_alert_notification_errors_recent, incidentrelay_scheduler_last_run_timestamp_seconds, incidentrelay_build_info) are computed once per scrape from the database, so their values never depend on how many processes are running.

incidentrelay_user_notification_deliveries_recent, incidentrelay_alert_notification_errors_recent and incidentrelay_scheduler_last_run_timestamp_seconds are recomputed from the database at scrape time, so they work across processes. The two delivery gauges cover rows updated in the last 24 hours: delivery rows transition in place (pending to sent or failed, new provider errors), so the window follows updated_at, not created_at, and a scrape never aggregates the full delivery history. During a database outage both are absent from the exposition and the heartbeat reads 0; incidentrelay_database_up = 0 explains why. The heartbeat gauge requires a running scheduler worker; the heartbeat job itself only runs while [metrics] is enabled.

incidentrelay_worker_last_seen_timestamp_seconds is persisted in the database, so it works across systemd units and separate Docker/Kubernetes containers. The scheduler is also exposed under incidentrelay_scheduler_last_run_timestamp_seconds for compatibility. Telegram and Slack update their timestamp after a successful worker loop; alert only on workers that your deployment actually runs.

The user-notification queue gauges describe current backlog, not history. state="due" counts only pending deliveries whose scheduled_at has arrived, so intentionally delayed notifications do not look stuck. state="processing" counts claimed deliveries and measures age from their last state update.

incidentrelay_orchestration_pending_events exposes the current paused-event inventory. failed means activation exhausted its automatic retries and needs attention. incidentrelay_orchestration_oldest_due_age_seconds includes only pending rows whose activation/retry time has arrived, so future pauses do not inflate the age.

Scheduler settings#

The scheduler process checks reminders, escalations and periodic jobs.

The scheduler refreshes a heartbeat every heartbeat_interval_seconds (default 30, [scheduler] section); incidentrelay_scheduler_last_run_timestamp_seconds in /metrics reports how recently it ran.

The scheduler wake-up interval is separate from rotation reminder intervals. Rotation reminder intervals are configured per rotation:

Copy0 disables reminders for that rotation
>= 60 sends reminders at that interval in seconds
1..59 invalid

Do not use a global reminder-after setting as a runtime fallback when rotations require an explicit interval.

Logging#

If file logging is enabled, use a writable path:

Copy[main]
log_level = INFO
log_file = /var/log/incidentrelay/incidentrelay.log

For systemd and containers, also check journal or container logs.