On-call game days: test the pager before production does

An on-call plan can look flawless in a diagram and still fail in wonderfully ordinary ways: the route token belongs to an old integration, the current responder never enabled notifications, the backup rotation resolves to the same person, or the escalation worker quietly stopped last Tuesday.

An on-call game day tests the complete path before a real incident does:

monitoring event
  -> intake endpoint and route
  -> team, service and severity
  -> current on-call assignment
  -> notification delivery
  -> acknowledgement
  -> reminder and escalation
  -> resolution and audit trail

This is not an exam for the responder. It is an experiment on the system. If Alice misses the test page because the channel was configured incorrectly, the finding is "delivery is broken", not "Alice lacks pager spirit".

Decide what the exercise must prove

"Test on-call" is too vague. Pick three to six observable questions for one session:

• Does a real monitoring payload reach the intended intake route?
• Does the route select the correct team, service and current responder?
• Does at least one channel produce a notification the responder notices?
• Can the responder acknowledge without searching for permissions?
• Does a missed acknowledgement notify the correct backup at the expected time?
• Do reminders and escalation stop after acknowledgement or resolution?

Write the expected answers before triggering anything. Otherwise, when the alert lands somewhere surprising, the team may decide that was probably the plan all along. Humans are excellent at retrospective configuration design.

Install safety rails first

A game day should be realistic without impersonating a customer-impacting incident. Use these guardrails:

• Create a dedicated test route, team or clearly isolated matcher when possible.
• Put [GAME DAY] in the alert title and add labels such as exercise=true and environment=game-day.
• Tell all possible recipients when the exercise starts and ends.
• Choose an abort phrase and name the person allowed to stop the exercise.
• Do not disable a real production channel merely to simulate failure.
• Never paste a route token, phone number or provider secret into the notes.
• Keep one operator outside the exercise who can restore configuration immediately.

The safest useful game day changes test objects, not production safety controls.

Assign four small roles

Role	Responsibility	Can be combined?
Facilitator	Runs the timeline, reveals scenario details and calls abort.	Yes, with the observer in a small team.
Responder	Acts exactly as they would while on-call.	No; they should not know every injected failure.
Observer	Records timestamps, decisions, confusion and missing context.	Yes, with the facilitator.
Platform operator	Sends test events and safely injects or reverses faults.	Yes, if the facilitator can do both without losing notes.

For the first exercise, let everyone know the scenario. Surprise is not required to find bad permissions. Add uncertainty later, after the basic delivery path works.

Run a preflight check

One day before the session, verify the boring prerequisites. In IncidentRelay:

1. Open the rotation calendar and record who should be primary and backup at exercise time.
2. Open On-call Health for the rotation and team; resolve schedule gaps, inactive members and routes without assignment targets.
3. Confirm the test route source, token, matchers, escalation mode and attached channels.
4. Confirm participants can sign in, open alert details and use ACK and Resolve actions.
5. Confirm the scheduler process is running if the exercise includes reminders or timed escalation.
6. Capture the expected timeline in minutes, including every policy delay.

Do not send the actual test alert during preflight. That is like eating the fire drill because you wanted to inspect it.

Use a scenario ladder

Start with the golden path, then add one failure at a time. When five things are broken simultaneously, you learn that five broken things are confusing — valuable philosophy, weak diagnostics.

Scenario	What you inject	What it proves
1. Golden path	A correctly labelled firing alert.	Routing, assignment, delivery, ACK and Resolve work end to end.
2. Missed ACK	The consenting primary deliberately ignores the test page.	Reminder timing and backup escalation match the policy.
3. Planned absence	A temporary rotation override assigns another test responder.	The final schedule, notification target and calendar agree.
4. Bad matcher	A safe test payload omits or changes one required label.	Routing failure is visible and explainable through the trace.
5. Channel failure	A test-only channel is intentionally invalid.	Failure is recorded and an independent delivery path remains usable.

Run scenarios one and two in the first session. Add the others over time. A game day is not improved by continuing until everybody misses dinner.

A ready-to-run 60-minute script

Time	Action	Evidence to capture
00:00	Brief participants, confirm abort phrase and expected on-call users.	Calendar screenshot and expected escalation timeline.
00:05	Send the golden-path alert through the actual test integration.	HTTP status, response body and trace_id.
00:07	Responder opens the page but thinks aloud before ACK.	Delivery time, available context and first diagnostic choice.
00:12	Responder acknowledges and follows the runbook.	ACK time, actor and confirmation that reminders stop.
00:20	Resolve the alert and reset for scenario two.	Final state and notification updates.
00:25	Send a new alert; primary deliberately does not ACK.	Reminder and escalation timestamps.
00:40	Backup acknowledges, explains ownership and resolves.	Correct target, channel, policy step and stopped reminders.
00:50	Debrief and assign fixes.	Owners, deadlines and a retest date.

Adjust the clock to your configured policy. If the real first delay is fifteen minutes, either reserve more time or use an isolated temporary policy with shorter delays. Do not quietly change production escalation timing and promise to remember it later.

Send a traceable test payload

For an IncidentRelay generic webhook route, the request can look like this. Use a test route token and a unique fingerprint for each scenario:

POST /api/integrations/webhook
Authorization: Bearer ROUTE_TOKEN
Content-Type: application/json

{
  "status": "firing",
  "external_id": "game-day-2026-07-26-01",
  "fingerprint": "game-day-2026-07-26-01",
  "title": "[GAME DAY] Checkout API unavailable",
  "message": "Synthetic incident. No customer impact.",
  "severity": "critical",
  "team": "infra",
  "labels": {
    "service": "checkout-api",
    "environment": "game-day",
    "exercise": "true"
  }
}

Save the response. IncidentRelay includes a trace_id for processed alerts, including stopped paths such as a routing failure. In the UI, open the alert details and its Explain tab; for an orphan routing failure, open the explain trace directly using the returned ID.

The Generic Webhook guide contains the exact authentication and payload formats. Resolve the test alert from the UI or send the appropriate follow-up event for the integration you are testing.

Observe the responder experience

The interesting evidence is not only whether a notification arrived. Watch what happens next:

• Can the responder tell that the event is a test in one glance?
• Do they know which service and environment are affected?
• Is the runbook or event link present and usable?
• Do they understand what ACK means and where to perform it?
• Can they find the current escalation state and contact backup?
• Do expired credentials or missing permissions block the first action?

Record hesitation and workarounds without judging them. If the responder must remember an undocumented URL from a dream they had in 2024, improve the alert.

Score the result with facts

Check	Example pass condition	Failure owner
Intake	Expected HTTP response and trace created.	Monitoring/integration owner.
Routing	Expected route, team, service and severity selected.	IncidentRelay configuration owner.
Assignment	Calendar and actual assignee match.	On-call team manager.
Delivery	Notification reaches the device inside the agreed budget.	Channel/platform owner.
ACK	Ownership is recorded and later reminders stop.	Workflow owner.
Escalation	Correct backup is notified at the expected policy step.	Policy owner.
Resolution	Alert closes and the timeline remains understandable.	Workflow owner.

Use exact timestamps and outcomes. "Notification was a little slow" is difficult to fix; "voice delivery arrived 96 seconds after the escalation step, budget 30 seconds" is an engineering input.

Turn findings into changes

Hold a fifteen-minute debrief immediately. Ask what worked, what surprised the team, what slowed ownership and which safeguard prevented a worse outcome. Convert every important finding into a small owned task:

Finding: backup received chat but no voice call
Impact: P1 escalation has one practical delivery path
Action: repair voice provider configuration and retest
Owner: platform team
Due: 2026-08-02
Evidence required: successful scenario-two timeline

Prioritize broken routing, missing assignments and failed delivery before polishing documentation. Then rerun the failed scenario. Closing a ticket is not proof that the pager has changed its behavior.

Choose a useful cadence

Run a basic golden-path test monthly or after material routing and channel changes. Run a broader game day quarterly, after a major upgrade, when a team adopts a new escalation policy, or before a new responder takes solo duty. Rotate participants so the system is tested by more than the person who configured it.

Keep a compact evidence packet: scenario, expected timeline, actual timestamps, trace IDs, screenshots, findings, owners and retest results. Over time it becomes proof that on-call readiness is maintained, not merely declared.

The copy-paste checklist

• Scope and success criteria written
• Test labels, route and recipients confirmed
• Current primary and backup recorded
• On-call Health checked
• Expected reminder and escalation times recorded
• Participants, observer and abort phrase confirmed
• Golden-path alert sent and trace saved
• ACK stopped reminders
• Missed ACK reached the correct backup
• Alert resolved and temporary configuration removed
• Findings assigned with deadlines
• Failed checks scheduled for retest

A good game day is slightly boring and extremely specific. Nobody earns points for panic. You are proving that the alert can find a qualified human, that ownership is visible, and that the next step still works when the first one does not. Production will eventually run its own exercise; it is considerate like that. Better to rehearse first.