← Blog Incident command

Incident Commander: roles, decisions and communication in IncidentRelay

Three engineers can all be busy fixing an outage and still leave nobody responsible for the next decision. This drill uses IncidentRelay for the technical records and a shared response log for the human agreement.

By Published Updated 4 min read incident commander incident management incident communication

The pager picks an engineer, not a commander

It is 08:55 UTC. Uploads fail in one region. Priya acknowledges the API AlertGroup; Anton sees storage errors in another. Maya opens an Incident, takes operational ownership and links the two groups. Leo agrees to coordinate. Those are four distinct responsibilities even if, on a quieter shift, one person holds three of them.

In IncidentRelay 2.4.0 the Incident has an operational assignee. Technical assignment, responder requests, stakeholder records and technical comments belong to AlertGroups. “Commander” here is a team responsibility: name that person in the response channel or shared log. Do not look for an Incident commander field or assume the next person on the schedule silently takes command.

IncidentRelay on-call calendar showing scheduled coverage
Use the calendar to find coverage. Ask the person to accept the handoff.

Get three answers in the first ten minutes

Time (UTC)QuestionWhat a useful answer sounds like
08:57What is actually broken?New EU uploads fail. Downloads still pass. Previously started uploads have not been checked.
09:00Who is testing which explanation?Priya compares failing requests with the last deploy. Anton checks storage access independently. Both report by 09:07.
09:05Who hears from us next?Jo updates support at 09:15, even if the only change is what we have ruled out.

Write the decision before it becomes folklore

“Probably the deploy” is not a rollback plan. Write what observation would make the team change its mind, who is authorised to act, and what will be checked afterwards. A shared log entry can be short:

09:09 UTC — EU uploads; existing downloads unaffected.
            Evidence: failures began after r42; old configuration passes the same test.
            Decision: restore r41 under our emergency-change procedure.
            Priya executes; Anton watches new uploads and the retry backlog.
            Stop if failures spread. Check at 09:12; Jo updates support at 09:15.

Keep source queries and traces in the appropriate AlertGroup comments; link those groups from the log. The commander coordinates the decision. That title does not grant permission to change production.

Do a handoff, not a disappearing act

At 09:20 new uploads pass, but the retry backlog is draining. Leo asks Nia to take command. Until Nia acknowledges, “someone from the next shift will pick it up” is a hope, not an assignment. Give her the Incident link, current impact, last action, open question and next two deadlines.

09:22 UTC — Nia accepts command; Maya remains Incident assignee.
            Priya verified a new EU upload. Anton checks backlog at 09:25.
            If errors return, reconvene immediately. Jo updates support at 09:30.

Announce acceptance to the responders and update the shared log. If operational ownership or a technical assignment is also changing, update the corresponding IR record separately. A calendar boundary alone proves none of those transfers happened.

Score the exercise by what survives the shift

Ask an engineer who missed the call to find the impact, the decision owner, the recovery check and the next update time in two minutes. Missing links and contradictory owners are useful findings. Fix those before adding another meeting. For the exact object model, see Incidents in 2.4 and AlertGroup responder requests.

Want to make this less theoretical?

IncidentRelay gives you schedules, rotations, routing, escalations and alert actions in one open-source, self-hosted package. The pager may still be rude, but at least it will be organized.