Documentation

Configuration

Version v2.5.0 · Updated
Interactive docs View on GitHub

Configuration#

IncidentRelay reads the config file path from:

INCIDENTRELAY_CONFIG_FILE

Example:

export INCIDENTRELAY_CONFIG_FILE=/etc/incidentrelay/incidentrelay.conf

For systemd:

Environment=INCIDENTRELAY_CONFIG_FILE=/etc/incidentrelay/incidentrelay.conf

For Docker Compose:

environment:
  INCIDENTRELAY_CONFIG_FILE: /etc/incidentrelay/incidentrelay.conf

The old ONCALL_CONFIG_FILE name should not be used.

Environment overrides#

Any option from the config file can also be set with an environment variable named INCIDENTRELAY__<SECTION>__<OPTION>, with the section and option in upper case. It takes precedence over the file:

export INCIDENTRELAY__DATABASE__PASSWORD=replace-with-the-database-password
export INCIDENTRELAY__LOGGING__LEVEL=DEBUG

To read the value from a file instead, add __FILE to the name and pass the path. Use this for Docker or Kubernetes secrets and Secrets Store CSI volumes, so secrets never have to be written into incidentrelay.conf:

export INCIDENTRELAY__DATABASE__PASSWORD__FILE=/run/secrets/db-password

A trailing newline in the file is ignored. Setting both a variable and its __FILE form is an error. For systemd, put the variables into an EnvironmentFile=.

When main.secret_key comes from the environment, empty main.secret_encryption_key, auth.jwt_secret, mattermost.action_secret and voice.callback_secret fall back to it instead of being generated by the Docker entrypoint.

main.secret_encryption_key encrypts the secrets stored in the database, so it must not change once IncidentRelay has used it, whether it comes from the config file, an environment variable or a __FILE. When it is empty, IncidentRelay encrypts with main.secret_key instead, and the same applies to that key. If the encryption key changes, the Docker entrypoint refuses to start: restore the previous key. Changing the key directly is not supported. The entrypoint compares the key with a salted hash it keeps in /var/lib/incidentrelay, so the check only works when that directory persists between restarts.

In [voice_provider], whose options depend on the provider, a variable only overrides an option that is present in the file, and the option keeps its spelling from the file. Variables don't add new options there:

[voice_provider]
apiKey =
export INCIDENTRELAY__VOICE_PROVIDER__APIKEY=replace-with-the-api-key

Provider options can also reference a variable directly, for example api_token = ${VOICE_API_TOKEN}; see Voice Provider Configuration.

Main and authentication secrets#

Generate two different random values and keep them stable across restarts and upgrades:

openssl rand -hex 32
openssl rand -hex 32
[main]
secret_key = replace-with-the-first-random-value

[auth]
jwt_secret = replace-with-the-second-random-value
jwt_cookie_secure = true

secret_key belongs to [main], not [server]. jwt_secret belongs to [auth]. Do not reuse the same value for both settings. Set jwt_cookie_secure = true when public_base_url uses HTTPS.

Server section#

[server]
host = 0.0.0.0
port = 8080
public_base_url = https://incidentrelay.example.com
OptionDescription
hostAddress to bind the web service to
portHTTP port
public_base_urlExternal URL used in alert links, buttons and callbacks

In production, public_base_url must be the real external HTTPS URL.

Database: SQLite#

[database]
type = sqlite
name = /var/lib/incidentrelay/incidentrelay.db

[sqlite]
wal = true
busy_timeout = 5000

SQLite is suitable for small self-hosted installations. Keep one web worker when using SQLite.

Database: PostgreSQL#

[database]
type = postgresql
host = 127.0.0.1
port = 5432
name = incidentrelay
user = incidentrelay
password = change-me

Use PostgreSQL for larger installations, higher alert volume, multiple web workers, or long-term production deployments.

Outbound HTTP network policy#

IncidentRelay protects administrator-configured outbound HTTP requests against server-side request forgery (SSRF). Private, loopback, link-local, multicast, reserved and unspecified destination addresses are blocked by default.

The policy is controlled by the [security] section:

[security]
outbound_private_network_allowlist =
outbound_http_max_redirects = 3
outbound_http_max_response_bytes = 1048576

outbound_private_network_allowlist is a comma- or semicolon-separated list of IPv4/IPv6 addresses and CIDR networks that IncidentRelay is explicitly allowed to contact when they are otherwise considered private or unsafe.

Examples:

# One internal service only.
outbound_private_network_allowlist = 192.168.50.10/32
# Several approved internal networks/addresses.
outbound_private_network_allowlist = 10.20.0.0/16,192.168.50.10/32,fd00:1234::/48

A single IP may also be written without the prefix length, but /32 for IPv4 and /128 for IPv6 make the intended scope explicit. Prefer the narrowest possible entries instead of allowlisting whole private address ranges.

This policy is used by the shared outbound HTTP client, including OIDC metadata and JWKS retrieval and outgoing integrations such as generic/Teams/Discord webhooks, Slack webhooks and Mattermost API requests. The list is not a hostname allowlist: IncidentRelay resolves the hostname first and checks the resolved IP addresses.

For DNS names, every address returned by DNS must be public or explicitly allowlisted. If even one returned address is blocked, the request fails closed. Redirect targets are resolved and checked again before IncidentRelay follows them.

Upgrade impact in 2.1

IncidentRelay 2.1 enforces this policy for outbound requests. An installation

upgraded from 1.2 can therefore lose access to an existing internal OIDC

metadata/JWKS endpoint or outgoing integration even though its URL did not

change. Before upgrading, resolve every internal endpoint from the

IncidentRelay host/pod and add only the required IPs or CIDRs.

For example, if an internal identity provider resolves to 10.42.7.15:

[security]
outbound_private_network_allowlist = 10.42.7.15/32

After changing this setting, restart every IncidentRelay process that can make outbound requests.

The allowlist changes only the destination network policy. It does not disable HTTPS certificate verification or trust a private certificate authority. Internal HTTPS endpoints using a private CA must also have that CA installed in the operating system/container trust store.

Alert processing trace#

The global Explain Trace detail level is configured in [alerts]:

[alerts]
explain_trace_level = full

Supported values are full, compact and disabled. full preserves the existing detailed trace. compact keeps ordered processing steps but omits input_summary, result payloads and per-step data. disabled stores no Alert Explain Trace rows. A global Event Orchestration rule can override this value for matching events with the set_trace_level action.

Alert event history#

Incoming child-alert history can be controlled independently from alert lifecycle processing:

[alerts]
event_history = full

Supported values are full, initial and disabled. full stores incoming child-alert created, updated and resolved events. initial stores only the initial child-alert created event. disabled stores none of those incoming child-alert history rows. Alert state updates, grouping, notifications, escalation and resolution continue normally in every mode. Operational incident timeline entries such as acknowledgements, comments, reminders, maintenance, correlations, responders and stakeholders are always preserved.

Global and service Event Orchestration can override the configured default for matching events with set_alert_event_history. If several matching actions set a level, the last applied action wins.

Recovery notification hysteresis#

Incoming webhook and heartbeat recoveries can delay external resolved notifications while technical AlertGroup resolution remains immediate:

[alerts]
recovery_notification_delay_seconds = 300

The default 0 retains existing immediate recovery delivery. Values are clamped to 0–86400 seconds. This option is independent of Event Orchestration set_grouping.reopen_window_seconds: setting one does not enable the other. Delayed recovery is persisted through the normal AlertGroup notification queue, uses the scheduler, and is cancelled on a matching AlertGroup reopen. Duplicate source resolved events do not reschedule it. Manual Resolve remains immediate.

A worker that starts sending exactly as a reopen occurs is serialized with reopen on supported database backends; a message already delivered before reopen cannot be recalled. Delivery uses the existing notifier semantics; providers without an idempotency API can duplicate a message after a crash between sending and committing the queue update. To serialize with reopen, the worker holds a database transaction while delivering recovery; for high-volume installations or slow outbound channels, use PostgreSQL rather than SQLite.

Alert shelving scheduler#

Timed AlertGroup shelves are expired by the scheduler:

[alerts]
shelve_lifecycle_check_interval_seconds = 30
shelve_lifecycle_batch_size = 100

check_interval_seconds controls how often due shelves are reconciled. batch_size bounds one scheduler pass. Keep the scheduler service running whenever timed shelving is used. Shelving pauses notifications, reminders and escalation but does not change the AlertGroup technical status or service/business impact. See Alert shelving.

Data retention#

IncidentRelay 2.1 keeps retention settings in one section:

[retention]
alert_days = 30
# explain_trace_days = 30
# orchestration_execution_days = 30
cleanup_interval_seconds = 86400
batch_size = 500

alert_days = 0 is the default and keeps resolved alert history indefinitely. Explain Trace and general Event Orchestration execution retention inherit alert_days unless their optional override is set. See Data Retention for deletion rules and upgrade compatibility.

SMTP section#

Email notification channels use global SMTP settings. SMTP transport is not configured per channel.

[smtp]
host = 127.0.0.1
port = 25
from = incidentrelay@example.com
use_tls = false
user =
password =

For an unauthenticated local relay, leave user and password empty.

For an authenticated SMTP server:

[smtp]
host = smtp.example.com
port = 587
from = incidentrelay@example.com
use_tls = true
user = incidentrelay@example.com
password = change-me

Email channels default to the assigned user's profile email; optional fallback/fixed recipients can be configured per channel using recipient_mode and recipient_email. SMTP transport remains global.

Telegram proxy#

If the environment requires a proxy for Telegram Bot API calls, configure it globally. Keep token values in channel configuration, not in the global config.

Example option names depend on the current service config implementation. Use the same config file for web and Telegram worker processes.

Voice section#

[voice]
provider = stub
providers_dir = /usr/local/lib/incidentrelay/voice_providers
callback_secret =
OptionDescription
providerVoice provider name
providers_dirDirectory with custom provider modules
callback_secretSecret used for callback validation

Voice call notifications are sent to the assigned user's profile phone number.

Browser push section#

Browser push notifications are profile-level PWA/browser notifications. They are not configured as notification channels.

[browser_push]
enabled = true
vapid_public_key = CHANGE_ME_PUBLIC_KEY
vapid_private_key = /etc/incidentrelay/vapid/private_key.pem
vapid_subject = mailto:admin@example.com
action_token_ttl_seconds = 900
OptionDescription
enabledEnables or disables browser push globally
vapid_public_keyPublic VAPID key returned to the browser for PushManager.subscribe()
vapid_private_keyPrivate VAPID key or PEM file path used by the server to send Web Push messages
vapid_subjectContact URI included in VAPID claims, usually mailto:admin@example.com
action_token_ttl_secondsLifetime of one-time ACK/Resolve/Shelve/Unshelve tokens embedded into push notifications

After changing browser push settings, restart the web service. Restart the scheduler too if it sends notifications in your installation.

Read more: Browser Push.

Metrics section#

IncidentRelay can serve Prometheus metrics at GET /metrics:

[metrics]
enabled = true
auth_token = replace-with-a-random-value
OptionDescription
enabledServes GET /metrics in the Prometheus text exposition format. The default is false and the endpoint answers 404
auth_tokenWhen set, scrapes must send Authorization: Bearer <token> or receive 401

The endpoint is disabled by default. Enable it only when something actually scrapes it, and protect the port at the network layer the same way you protect /healthz.

Generate a value for auth_token with openssl rand -hex 32. Example Prometheus scrape configuration:

scrape_configs:
  - job_name: web
    scheme: https
    metrics_path: /metrics
    bearer_token_file: /etc/prometheus/incidentrelay.token
    static_configs:
      - targets:
          - incidentrelay.example.com

Exported metrics:

MetricTypeDescription
incidentrelay_http_requests_totalcounterHandled HTTP requests, labels method and status
incidentrelay_http_request_duration_secondshistogramRequest latency, label method
incidentrelay_database_upgauge1 when the database answered SELECT 1 at scrape time, otherwise 0
incidentrelay_migrations_pendinggaugeNumber of migration files not applied yet, absent while the migration state is unknown
incidentrelay_build_infogaugeRunning version, label version, value is always 1
incidentrelay_alerts_received_totalcounterAlerts received through the integration API, label source
incidentrelay_alert_group_actions_totalcounterAlert group transitions, label action: created, acknowledged, resolved, reopened
incidentrelay_user_notification_deliveries_recentgaugeUser notification deliveries updated in the last 24 hours, labels method and status
incidentrelay_alert_notification_errors_recentgaugeAlert notification deliveries that ended in an error, updated in the last 24 hours, label provider
incidentrelay_scheduler_last_run_timestamp_secondsgaugeUnix time of the last scheduler heartbeat, 0 when it never ran
incidentrelay_worker_last_seen_timestamp_secondsgaugeLast successful worker-loop heartbeat, label worker: scheduler, telegram, slack
incidentrelay_user_notification_queue_depthgaugeDue or processing user-notification work, label state: due, processing
incidentrelay_user_notification_queue_oldest_age_secondsgaugeAge of the oldest user-notification work item, label state: due, processing
incidentrelay_orchestration_pending_eventsgaugePaused orchestration events, label status: pending, activating, failed
incidentrelay_orchestration_oldest_due_age_secondsgaugeAge of the oldest paused orchestration event already due for activation/retry

incidentrelay_database_up and incidentrelay_migrations_pending are evaluated at scrape time and repeat the /readyz checks. /metrics keeps answering while the database is down, so both values stay visible during an outage. While the migration state cannot be read (database down or migration check failed), the incidentrelay_migrations_pending sample is absent rather than reading 0 — it is only reported when it is actually known. Alert on incidentrelay_database_up = 0 instead of the pending count alone.

Request counting is active only while enabled = true. After changing metrics settings, restart the web service and the scheduler and worker services too — they pick up the setting when they start. Only the web service serves /metrics; the scheduler and the other workers only record into it.

IncidentRelay normally runs several processes behind one scrape target: multiple Gunicorn web workers plus the scheduler and the Telegram/Slack workers as separate daemons. So that one /metrics scrape covers them all, every IncidentRelay process records its counters into one shared directory configured through the PROMETHEUS_MULTIPROC_DIR environment variable, and the web workers merge all processes' files at scrape time. The packaged deployments set this up already: the systemd units share /run/incidentrelay/metrics, so a scrape covers every unit on the host — web workers, scheduler and chat workers alike. Without the variable — single-process development runs — the counters cover that one process only.

The counter files are keyed by process PID, so a PROMETHEUS_MULTIPROC_DIR may only be shared by processes that see the same PIDs: those on one host, or inside one container. Never point containers at the same directory — independent PID namespaces can reuse the same PIDs and corrupt each other's files. The Docker image therefore gives every service container its own directory under /var/lib/incidentrelay/metrics/; a containerized /metrics scrape aggregates the web container's workers, while scheduler and chat-worker transitions land in their own directories and are not visible to it. To merge them you would need an aggregation mechanism that crosses container boundaries, which this feature does not provide.

If you point PROMETHEUS_MULTIPROC_DIR at your own location, keep it to processes of one host or one container; it must exist and be writable before the processes start. While [metrics] is disabled the directory is not touched, so a broken directory cannot block startup. When a process exits, its live-gauge files are removed the next time an IncidentRelay process starts. Its counter and histogram files are deliberately kept: deleting one would drop the dead process's share from the merged counter, which Prometheus reads as a counter reset that rate() and increase() distort. Stale counter files keep contributing until the shared directory is cleared on a full redeploy, and their number is bounded by how many distinct PIDs the directory has ever seen. Gauge cleanup checks liveness by PID, so a recycled PID keeps a dead process's gauge file until the process that now owns that PID exits.

The database-derived gauges (incidentrelay_database_up, incidentrelay_migrations_pending, incidentrelay_user_notification_deliveries_recent, incidentrelay_alert_notification_errors_recent, incidentrelay_scheduler_last_run_timestamp_seconds, incidentrelay_build_info) are computed once per scrape from the database, so their values never depend on how many processes are running.

incidentrelay_user_notification_deliveries_recent, incidentrelay_alert_notification_errors_recent and incidentrelay_scheduler_last_run_timestamp_seconds are recomputed from the database at scrape time, so they work across processes. The two delivery gauges cover rows updated in the last 24 hours: delivery rows transition in place (pending to sent or failed, new provider errors), so the window follows updated_at, not created_at, and a scrape never aggregates the full delivery history. During a database outage both are absent from the exposition and the heartbeat reads 0; incidentrelay_database_up = 0 explains why. The heartbeat gauge requires a running scheduler worker; the heartbeat job itself only runs while [metrics] is enabled.

incidentrelay_worker_last_seen_timestamp_seconds is persisted in the database, so it works across systemd units and separate Docker/Kubernetes containers. The scheduler is also exposed under incidentrelay_scheduler_last_run_timestamp_seconds for compatibility. Telegram and Slack update their timestamp after a successful worker loop; alert only on workers that your deployment actually runs.

The user-notification queue gauges describe current backlog, not history. state="due" counts only pending deliveries whose scheduled_at has arrived, so intentionally delayed notifications do not look stuck. state="processing" counts claimed deliveries and measures age from their last state update.

incidentrelay_orchestration_pending_events exposes the current paused-event inventory. failed means activation exhausted its automatic retries and needs attention. incidentrelay_orchestration_oldest_due_age_seconds includes only pending rows whose activation/retry time has arrived, so future pauses do not inflate the age.

Grafana dashboard#

After Prometheus has successfully scraped the IR web service, import the tools/grafana/incidentrelay-overview.json via Dashboards → New → Import in Grafana. Select the Prometheus data source at import, then choose IR scrape job and Web target on the dashboard. Set the data source's scrape interval to the actual Prometheus interval so the rate panels use the right window.

Use one job per IR deployment sharing a database. Scrape each web replica as a separate target: the dashboard adds request and event rates, but counts a shared database queue only once. Notification outcomes are rolling 24-hour counts regardless of the dashboard time range. Missing data is not zero: for example, a database outage can hide the migration count, and an optional worker has no heartbeat line until it first reports. In Docker and Kubernetes, web scrapes may not include counters from separate scheduler or worker containers; their database-backed queue gauges are still available.

Scheduler settings#

The scheduler process checks reminders, escalations and periodic jobs.

The scheduler refreshes a heartbeat every heartbeat_interval_seconds (default 30, [scheduler] section); incidentrelay_scheduler_last_run_timestamp_seconds in /metrics reports how recently it ran.

The scheduler wake-up interval is separate from rotation reminder intervals. Rotation reminder intervals are configured per rotation:

0 disables reminders for that rotation
>= 60 sends reminders at that interval in seconds
1..59 invalid

Do not use a global reminder-after setting as a runtime fallback when rotations require an explicit interval.

Logging#

If file logging is enabled, use a writable path:

[main]
log_level = INFO
log_file = /var/log/incidentrelay/incidentrelay.log

For systemd and containers, also check journal or container logs.

Change Intelligence experimental intake#

All Change Intelligence settings apply to the entire installation. The system settings design describes inheritance for future Group settings.

Global administrators control Accept Change Events and Background change processing under Administration → System settings, or through GET and PATCH on /api/admin/system-settings. Send only the fields you want to change, for example {"change_worker_enabled": false} to pause processing. The intake field is change_intake_enabled. On a new installation intake is off and background processing is on. Turning intake on allows authorized integrations to send events to POST /api/change-events.

Intake changes take effect on the next request. The scheduler checks background processing at the start of each cycle. A running cycle can finish, and jobs created during a pause stay in the queue until processing resumes. Changes take effect without a restart. The audit log records the administrator, time, old and new values.

Advanced Change Intelligence settings#

Under Administration → System settings → Advanced settings, edit the numbers and select Save. The API accepts the same integers in a partial PATCH /api/admin/system-settings. The database stores the values for all IR processes. If you customized limits in a prerelease configuration file or environment, enter them here after running the migration.

API fieldPurposeDefaultRange
change_max_future_secondsAllowed future event time (seconds)300≥ 0
change_events_per_minuteNew events per integration per minute600≥ 1
change_events_per_changeEvents per change10001–10000
change_worker_interval_secondsWorker interval (seconds)5≥ 1
change_worker_jobs_per_tickJobs per worker tick201–100
change_worker_batch_sizeRows per batch1001–1000
change_worker_lease_secondsJob lease duration (seconds)60≥ 10
change_worker_max_attemptsMaximum attempts per job8≥ 1
change_worker_retry_base_secondsBase retry delay (seconds)5≥ 1
change_worker_retry_max_secondsMaximum retry delay (seconds)300≥ 1
change_pending_ttl_secondsPending reference lifetime (seconds)604800≥ 60
change_pending_poll_secondsPending reference check interval (seconds)60≥ 5

The common integer ceiling is 2,147,483,647. The base retry delay must fit within the maximum. Both can be edited in one request. An integration’s rolling one-minute quota covers all its sources and tokens. Exact event retries return the stored result without consuming quota.

Each scheduler instance checks the worker interval every five seconds, including while processing is paused. When it changes, the next tick is scheduled one new interval from that check. A batch in progress finishes with its original limit. Existing reference expiries, leases and scheduled retries keep their deadlines. The new duration applies the next time a deadline is assigned or a lease is renewed.

The body is limited to 256 KiB before parsing; at most 100 Service references may be submitted, and details are limited to 64 KiB, 10 nesting levels and 2000 nodes. The entire payload is limited to 15 levels and 5000 nodes. Duplicate keys, non-finite numbers, NUL and invalid Unicode are rejected. Timestamps require a timezone and at most six fractional digits. Titles are limited to 500 characters, descriptions to 10000, URLs to 2048; URLs must be HTTP(S) without credentials. Configure the existing WSGI/reverse-proxy request timeouts to bound body transport.

201 confirms durable acceptance; 200 acknowledges an identical retry. An event-ID/content or immutable identity conflict returns 409. Retry 429 after 60 seconds and 503 after at least one second with the same event ID and content. PostgreSQL lock/statement timeouts are 2/5 seconds per attempt; transient database errors are retried up to three times. Other write errors return 500; failures before commit roll back. A connection loss during commit may leave acceptance uncertain: retry the same identity/content to discover the stored result safely. Neither diagnostic responses nor logs include the submitted payload. The stage 5 worker resolves pending references; it does not purge events. Stored identities remain retryable while retained. See OpenAPI and the architecture for ordering, grants, namespace pinning and the remaining stages.

Durable Change worker#

The worker runs independently of event intake. The values below are the defaults.

Each tick starts up to 20 jobs and stops starting jobs after 10 seconds. Reference and discovery batches contain at most 100 rows by default (configurable 1–1000); the job limit is 1–100. A lease lasts 60 seconds (minimum 10). Reference calculation renews it every ten rows. Publication checks the lease token, generation, expiry and Change/auth/graph revisions under locks, including a final fence before acknowledgement. PostgreSQL claims use SKIP LOCKED; SQLite reserves writes before reading. A lost worker cannot publish or alter a replacement worker's retry state.

Successful batches checkpoint their cursor durably and reset their technical failure budget. Transient database failures use exponential backoff with jitter between half and all of the configured delay, capped at 300 seconds. Eight failed attempts, including crashes with expired leases, exhaust the budget. Unexpected errors immediately mark a job dead; diagnostics contain a safe error category. The periodic sweep does not restart dead jobs. A new accepted event or an audited operator action may request a new generation.

Pending TTL is seven days from reference acceptance (minimum 60 seconds), not the sender's occurrence time. Older pending rows without an expiry use their original created_at. Successful resolution preserves the canonical Service ID permanently. An unavailable namespace/Service gives invalid, missing current authority gives denied, and an elapsed pending window gives expired. These states remain until an explicit operator retry or a newly accepted event containing the same reference. An exact duplicate event_id leaves the state and TTL unchanged. Both retry paths check current authority and preserve the pinned identity. Polling unresolved references defaults to 60 seconds (minimum 5); waiting for a Service is not a technical failure. Token revocation or expiry does not cancel already accepted work; the worker uses current integration, source and grants.

Catalog and dependency mutations atomically advance graph revision and request a bounded discovery pass over retained Changes; authorization mutations scan the affected integration. Each pass has a high-water mark and durable cursor. The periodic fallback scans only indexed pending references, not full Change history. It restores missing work without resetting an active sweep or reviving dead jobs. Mapping publication may span batches. A current whole mapping requires a matching input stamp, no remaining batch cursor, and claimed_generation=requested_generation. The job can be done or pending its next missing-Service poll; a fresh request or an unfinished batch is not a completed pass. During a pass, individual canonical links may still carry an earlier Change revision.

Global administrators can inspect GET /api/admin/change-integrations/jobs; Group administrators use GET /api/admin/change-integrations/{integration_id}/jobs for their owned integrations. These endpoints require JWT, return only queue metadata, support state and after, and paginate at 50 records. Retry with POST to the corresponding /jobs/{job_id}/retry, supplying {"expected_generation": <requested_generation>}. A stale request returns 409. Retry is audited, resets the TTL of blocked unresolved references, and retains original submitted identities and resolved IDs. It never bypasses current grants.

Metrics: incidentrelay_change_worker_results_total records fixed claim/outcome categories; incidentrelay_change_jobs, incidentrelay_change_queue_oldest_due_age_seconds, incidentrelay_change_references, and incidentrelay_change_dead_jobs expose queue size, age and errors without source/Team/Service labels. Stage 7 enables impact, correlation and eligible episode discovery. Stage 9 enables safe Service Timeline publication through the same worker.

To pause the entire flow, turn off both switches in System settings. Keep rows and cursors intact; expired leases can be reclaimed after re-enabling. Stage 5 does not delete events, jobs or identity digests, so retries remain possible while history is retained. Retention/tombstone policy remains release work for later stages; the automatic horizon is documented below.

Changes Inbox and read API#

Open Operations → Changes to search and inspect accepted Changes. Filters and pagination are saved in the URL; detail links use the opaque Change UUID. The Team selector includes owned Changes and readable related Services. Owner-only filters (status, source and environment) exclude limited cross-team cards.

Owner readers see lifecycle conflicts, service mapping diagnostics and immutable event history. A reader of a related Service sees a limited card, without the sender's title, source, actor, external IDs, URL or event payload. Stale or revoked cross-team links disappear until revalidated. Refresh reloads processing state; failed loads clear the previous data and offer a retry. Mapping and automatic correlation are separate worker steps. Manual review and Incident links are available in Change Details; Related Changes are also available on AlertGroup and Incident pages.

Read endpoints require a human JWT or an active user's personal token with changes:read, even if global API authentication is optional. Personal token Group/Team restrictions still apply. /api/services/{id}/changes additionally requires services:read. Machine integration tokens cannot read. Responses are Cache-Control: no-store; inaccessible and absent resources share 404.

curl --fail-with-body "$IR_URL/api/changes?kind=deployment&page_size=25" \
  -H "Authorization: Bearer $PERSONAL_TOKEN"
curl --fail-with-body "$IR_URL/api/changes/$CHANGE_UID/events?page=1" \
  -H "Authorization: Bearer $PERSONAL_TOKEN"

Events are owner-only. /services returns scoped canonical links and, for owners, separately paginated submitted-reference diagnostics. Both collections use the requested page/page_size (1–100). Summary safe cards include up to five links; use /services for the complete accessible set. SQL filtering precedes counts and pagination. service_id, or service_slug with exactly one team_id or team_slug, selects a readable Service. scope_team_id selects Team context; owner_team_id selects readable ownership. occurred_from/occurred_to are inclusive ISO timestamps with a timezone. Unknown/duplicate selectors are 400.

For an internal acceptance environment with intake enabled, first create an integration/Source, a changed grant for the target Service, and a machine token. Replace the IDs, source name and event time below with that environment's values:

curl --fail-with-body "$IR_URL/api/change-events" \
  -H "Authorization: Bearer $CHANGE_TOKEN" -H "Content-Type: application/json" \
  --data '{"schema_version":1,"event_id":"deploy-42-started","change_id":"deploy-42","owner_team_id":1,"source":"deploy","kind":"deployment","event_type":"change.started","occurred_at":"2026-11-02T12:00:00Z","title":"Deploy checkout 42","changed_service_ids":[1]}'

Reuse the exact event_id and payload after transport failures (200 for an exact retry). A completion has the same change_id and a new event_id, event type and sender time. Use source_sequence only for a source configured for sequence ordering. Public intake stays disabled until end-to-end acceptance in stage 10.

Opt-in automatic Change suggestions#

Enable Automatic suggestions in the Service editor for Services whose alerts should be checked against changes. The default is off, including after upgrade. Service POST/PUT accept change_correlation_enabled and change_correlation_exclusions; omitted settings are preserved on PUT, and [] clears exclusions.

Example exclusions: [{"source":"heartbeat"},{"labels":{"kind":"synthetic"}}]. Source means the alert source. All clauses of one rule must match one child exactly. Any matching rule excludes the child; the episode participates if a non-excluded child remains, including resolved children observed in that cycle. Existing Service edit permissions apply; settings and audit commit in one transaction.

The history horizon is seven days, including still-running older Changes; the post-operation window is two hours. Old groups/episodes without context are not reconstructed. Stage 7 stores potential-cause suggestions without status or notification effects; manual review is available in Change Details; Related Changes are also available on AlertGroup and Incident pages. Public intake stays disabled until stage 10. Engine limits are depth 5, 200 links/ impact rows, 1,000 edges and five seconds per calculation; existing worker batch, lease and retry settings apply. A graph limit records limit_exceeded in the job.

When rolling back to an application version without correlation-aware retention, also disable alert history cleanup ([retention] alert_days=0). Keep the additive tables and evidence intact.

Manual Change relations#

Change Details supports confirming/rejecting suggestions, manually assigning context/mitigation/cause to existing episodes, and linking Changes to Incidents. Each operation requires Change visibility, the target's responder role and a reason. Manual relations work independently of automatic correlation policy and time windows. Legacy AlertGroups without episodes are not reconstructed; direct Incident links remain available.

On a version conflict, close the dialog and reload. Removed Incident links can be restored through their action menu. These operations do not change alert/Incident status or send notifications. Apply migrations, including 20261104010000_change_reviews, before starting the updated application. Public intake remains disabled until stage 10.

Apply 20261104020000_change_service_timeline before starting the updated application and worker. It preserves legacy Service events and queues revalidation of accepted Changes; it creates no old alert episodes. Failed timeline jobs follow the existing retry/backoff/dead-letter controls and can be retried by integration administrators.

Alert details → Related Changes lets you select the latest or a historical episode and review/link Changes for that episode. Incident workspace provides the same link role/edit/remove/restore actions. Manual links remain available when automatic correlation is disabled. A legacy group without an episode shows an explanation; it gains an episode only on a genuine reopen.

Service Timeline links to the authorized Change page. Its stored projection contains only the Change UUID, kind, relation and event time, even for the owner. Inferred impact is labelled separately. Service details reload history on open; stale derived links disappear after access or graph changes. Public intake remains disabled until stage 10 acceptance.