Self-hosted on-call management: when it makes sense

Teams choose self-hosted on-call management for data control, network locality, customization, predictable licensing or because procurement moves slower than tectonic plates. All valid. The useful question is not "can we run a container?" It is "can we operate the complete notification path with an acceptable recovery time?"

Use a decision matrix, not a philosophical debate

Requirement	Self-hosting is attractive when	Hosted service is attractive when
Data residency	Alert payloads and responder data must stay in your network or jurisdiction.	Vendor regions and controls meet the requirement.
Connectivity	Monitoring sources are private or disconnected from public SaaS endpoints.	Reliable outbound connectivity already exists.
Customization	You need internal channels, payload handling or code-level control.	Standard integrations cover the workflow.
Operations capacity	A team can own upgrades, backups, security and availability.	You want the paging control plane operated for you.
Time to value	Existing platform automation makes deployment routine.	You need reliable paging this week.
Cost shape	Infrastructure and operator time compare favorably at your scale.	Per-user pricing is cheaper than internal ownership.

Regulatory or network requirements can make the decision easy. If they do not, score each row with evidence and include the opportunity cost of the engineers who will own it.

Calculate total cost honestly

Open source removes a license barrier; it does not remove economics. Estimate:

annual cost = compute + database + storage + backups
            + SMS/voice/email provider usage
            + monitoring and security tooling
            + routine operator hours
            + expected incident and upgrade work

Include non-recurring work: initial deployment, identity/access integration, migration from the old tool, game days and documentation. Then compare the result with hosted pricing and the value of control. "The VM already exists" is not a complete capacity or ownership model.

Know the IncidentRelay runtime

A basic deployment has two application processes and persistent state:

Monitoring systems
       |
       v
Reverse proxy / TLS
       |
       v
IncidentRelay web service ----> SQLite or PostgreSQL
       |                              ^
       |                              |
       +---------------- IncidentRelay scheduler
                              |
                              v
                  Slack / Mattermost / Telegram
                  Teams / email / webhook / voice

The web service handles the UI, API and incoming integrations. The standalone scheduler handles reminders, escalations and periodic jobs. If the web UI is healthy but the scheduler is stopped, new alerts may arrive while timed escalation quietly takes a nap. Operate both as first-class services.

Docker Compose is suitable for testing and simple installations. SQLite is a reasonable small-installation starting point. Use PostgreSQL for larger alert volume, multiple web workers or a more conventional production database setup. The Docker installation guide documents the current container layout and paths.

Design the failure domains

Place the platform where it remains reachable during the incidents it must report. Hosting it inside the same Kubernetes cluster, identity provider and DNS zone it monitors creates a neat circular dependency:

cluster fails -> IncidentRelay fails -> cluster failure cannot page

You do not need infinite independence, but identify shared dependencies: compute, database, network, DNS, TLS, identity, secrets manager and notification providers. For critical environments, isolate enough of the control plane to survive the most likely production failures.

The incident system does not need to survive the heat death of the universe. It does need to survive the failure modes already present in your risk register.

Create an out-of-band fallback

No on-call platform can use itself to report that it is completely unavailable. Maintain a small external fallback:

• an external uptime check for the public intake and UI endpoints;
• a separate destination controlled outside IncidentRelay;
• an exported or printed current duty roster/contact tree;
• a documented manual way to notify the primary and backup;
• clear criteria for switching to fallback mode.

Exercise it. A fallback phone tree that has not been updated since three reorgs ago is historical fiction.

Production readiness checklist

Network and TLS

• Expose the web service through Nginx, HAProxy or another maintained reverse proxy.
• Use HTTPS and set public_base_url to the real external URL so generated links and callbacks are correct.
• Restrict administrative access while keeping required integration and callback endpoints reachable.
• Test DNS and certificate renewal before the certificate chooses a holiday weekend.

Database and persistence

• Put SQLite data or PostgreSQL storage on an explicitly persistent volume.
• Document who owns database capacity, integrity checks and retention.
• Monitor free space, connection failures and migration errors.
• Do not assume a container image contains the database just because both disappeared together once.

Process supervision

• Run the web and scheduler processes separately under Compose, systemd or Kubernetes.
• Ensure only the intended standalone scheduler instance performs scheduled work.
• Restart failed processes and retain logs outside ephemeral containers.
• Check /healthz and /readyz where supported, plus an end-to-end test that proves alerts are processed.

Notification providers

• Use production credentials and rotate them through a documented process.
• Test every channel responders depend on, including interactive ACK/resolve actions.
• Monitor provider failures, rejected webhooks and callback signature errors.
• Keep at least one independent delivery path for critical escalation when risk requires it.

Back up what you need to restore

Back up the database, application configuration and the secrets or secret references required to reconnect notification providers. Also preserve the deployment manifest, reverse-proxy configuration and a record of the running IncidentRelay version.

Choose recovery objectives:

• RPO: how much schedule, alert and configuration data can be lost?
• RTO: how long can paging operate in fallback mode before IncidentRelay must return?

A backup job is not evidence of recoverability. Restore into an isolated environment quarterly, run migrations if required, sign in, inspect rotations and routes, then send and acknowledge a test alert. Record elapsed time and missing steps.

Use a boring upgrade runbook

1. Read release notes and identify schema, configuration and integration changes.
2. Back up persistent data and configuration; verify the backup artifact exists.
3. Test the target version and migrations in a non-production copy.
4. Announce the maintenance window and prepare the external paging fallback.
5. Upgrade the web and scheduler components to compatible versions and run the documented migrations.
6. Verify health, login, rotations, routes, channels and scheduler activity.
7. Send a real test alert, let one reminder/escalation occur, acknowledge it and confirm notifications stop.
8. Document the version and close the fallback window.

Define rollback before the upgrade, especially when database migrations are involved. "Put the old image back" may not reverse a changed schema, and hope is not a database tool.

Secure the alert path

Alert payloads may contain hostnames, customer context and internal topology. Treat intake tokens and channel credentials as secrets. Scope route tokens to their intended source, rotate exposed credentials, keep authorization groups small and review who can manage teams, rotations, routes, policies and silences.

Log administrative changes and notification failures without leaking full secrets. Apply operating-system and dependency updates, protect backups, and validate incoming provider callbacks when signature support exists.

Monitor the system from outside and inside

Infrastructure checks should cover web readiness, database connectivity, certificate expiry, disk capacity, scheduler process state and notification-provider errors. Add a low-frequency synthetic alert that travels through an intake route to a dedicated test destination; alert if it does not arrive within the expected window.

The synthetic test should not wake the real primary every hour. That would prove delivery by manufacturing alert fatigue, a wonderfully self-defeating achievement.

Know when self-hosting is not ready

Pause production adoption if nobody owns the service, restores have never been tested, the scheduler is not monitored, all delivery depends on one unverified channel, or the platform shares every failure domain with the system it pages about. Start in a non-critical team, build operational evidence, then expand.

A sensible adoption path

1. Pilot: deploy with persistent storage and one non-critical route; learn the data and notification flow.
2. Harden: add TLS, monitoring, backups, restore testing, access controls and fallback paging.
3. Exercise: run channel tests, missed-ACK escalation, scheduler failure and database restore scenarios.
4. Migrate: move a production team with a defined rollback and parallel fallback window.
5. Operate: assign ownership, patch regularly and review availability and notification metrics.

Self-hosting makes sense when control is valuable and operational ownership is real. IncidentRelay gives you the on-call workflow; your deployment supplies the reliability envelope around it. The incident platform should be boring infrastructure, not the most mysterious service in the building.