On-call rotation best practices for small SRE teams

Small teams feel every on-call mistake. One vacation can remove a quarter of the rotation, one noisy service can ruin everyone's month, and one unclear handoff will reveal itself at 03:17, scientifically the worst minute to discover process ambiguity.

The goal is not perfect equality. The goal is predictable coverage, comparable burden and fast recovery without depending on a permanent hero.

Start with constraints, not calendar colors

Write down the operating constraints before choosing daily or weekly shifts:

• How many engineers can independently diagnose the production service?
• Is coverage 24/7, business hours, or only during a launch window?
• How many actionable pages arrive during work hours and overnight?
• What response time is promised for P1 and P2 incidents?
• Can the secondary take over immediately, or are some permissions missing?
• Which timezones and local labor rules affect the team?

Count only responders who have access, training and authority to act. A five-person team with two people still shadowing is a three-person rotation. Adding names to a schedule does not grant database permissions through positive thinking.

Estimate whether the rotation is sustainable

For a weekly primary rotation, the rough annual load is:

primary weeks per engineer = 52 / qualified responders

Qualified responders	Primary weeks/person/year	Operational reality
2	26	Extremely fragile; one absence breaks the model.
3	17.3	Possible at low page volume, but vacations need deliberate cover.
4	13	A workable small-team baseline with a secondary path.
6	8.7	More recovery time and easier training capacity.

Weeks alone hide the painful part. Track after-hours interruptions. A week with zero pages and a week with four 03:00 pages are both one week in a spreadsheet and completely different experiences for the nervous system.

Choose shift length deliberately

Shift	Useful when	Cost
One week	Low-to-moderate volume; context takes time to acquire.	A noisy week concentrates fatigue.
Three or four days	Pages are frequent and a full week is too punishing.	More handoffs and uneven weekend accounting.
One day	High-volume operational work or follow-the-sun coverage.	Maximum handoff overhead.
12-hour regional window	Two genuinely capable regions share 24/7 duty.	Requires precise timezone and DST handling.

Weekly rotations are a good default for many small teams because they are predictable and reduce handoff count. Change the duration when evidence says the burden is unsafe, not because a conference slide called something "industry standard".

Separate primary duty from backup duty

The primary handles first response. The secondary covers missed acknowledgements, helps during a major incident and replaces the primary when needed. Keep them as separate duty lines or escalation targets so the same person cannot silently occupy both positions.

10:00 P1 alert -> primary Alice
10:05 no ACK  -> secondary Bob
10:15 no ACK  -> team lead Carol

Make the secondary role explicit: are they expected to keep a laptop nearby, or are they only a management escalation? "Backup" can mean anything from "fully operational responder" to "person who knows somebody's phone number". Those are not equivalent architectures.

Measure fairness in more than hours

Fairness has at least four dimensions:

• scheduled hours: total primary and secondary duty;
• undesirable hours: nights, weekends and public holidays;
• interruptions: pages received, especially after-hours;
• recovery cost: time spent on incidents and lost rest afterward.

Review these over a rolling eight-to-twelve-week window. Do not "fix" one noisy week by manually punishing the quiet person next week; adjust future weekend and holiday allocation, offer time off after serious incidents, and eliminate the noisy alert source.

A useful team dashboard can start with five numbers: primary hours, after-hours pages, acknowledged pages, missed acknowledgements and incident hours per responder. You do not need a fairness blockchain. A small table and an honest conversation will survive.

Make handoff a two-person transaction

A calendar event proves a scheduled transition, not a successful one. Require the incoming responder to accept the shift. The outgoing responder stays responsible until the agreed boundary and raises a missing handoff before leaving.

Use a short template:

Current incidents: [links and owners]
Risky changes next shift: [deploys/migrations]
Known degradation: [service + impact]
Active silences: [scope + expiry]
Temporary access/coverage notes: [details]
Incoming responder: [name + accepted timestamp]

Schedule handoffs during normal working hours when possible. Friday at 18:00 is a surprisingly creative time for calendars, phones and humans to disagree.

Treat swaps as data

Vacations and swaps should be visible in IncidentRelay as rotation overrides, not hidden in a chat thread named #random-but-important. Give every override an exact start, exact end and useful reason. Verify the final calendar after saving it.

Use layers and restrictions for recurring patterns; use overrides for temporary exceptions. If Pat covers every Tuesday forever, Pat needs a schedule rule, not an immortal override.

Onboard responders through shadowing

Do not place a new engineer alone in the primary rotation immediately after they discover where the dashboards live. A practical progression is:

1. Observe: receive notifications without primary ownership and join incident calls.
2. Reverse-shadow: act as primary while an experienced responder watches and can take over.
3. Qualify: demonstrate access, notification delivery, runbook use and escalation knowledge.
4. Join: start a membership period at a clear future handoff boundary.

Run a game day before qualification: send a test alert, acknowledge it, find the dashboard and runbook, explain the severity, escalate for help and resolve it. The goal is not theater. It is finding the expired credential while everyone is awake.

Use health checks before the pager does

IncidentRelay's On-call Health checks for disabled rotations, missing or empty layers, inactive users, single-member risk, future gaps and routes without a usable assignment or delivery path. Open full diagnostics after changing members, layer restrictions, timezones or routes.

Also test the real delivery chain quarterly and after channel changes:

monitoring -> intake token -> route -> target
           -> notification channel -> responder device
           -> acknowledge -> reminders stop

Run a lightweight monthly review

Bring data, then make one or two concrete changes. Review:

• pages per shift and after-hours pages per responder;
• median and 90th-percentile acknowledgement time;
• missed pages and escalations, with causes;
• alerts that fired repeatedly or had no useful action;
• coverage gaps, late swaps and failed notifications;
• whether every responder still has the required access.

The output should be owned work: tune alert X, write runbook Y, repair channel Z, add a trained secondary. "Be more vigilant" is not an engineering action.

A 30-day rollout for a small team

1. Week 1: inventory services, page volume, qualified responders and response objectives.
2. Week 2: create one primary rotation, one backup path, clear handoff rules and narrow paging scope.
3. Week 3: run test alerts and a game day; fix access and notification failures.
4. Week 4: start the rotation, review every page and reduce the first repeat offenders.

Keep the system boring, visible and adjustable. The point of on-call is to protect a service without converting the people who operate it into haunted furniture.