Incident management in IncidentRelay: from the first signal to the follow-up

The pager has done its job. Someone has acknowledged the alert. Three engineers are investigating, support wants an update, a database warning appears, and another team has opened a ticket for what might be the same outage.

The difficult questions have changed. Who coordinates the response? Which signals belong to the same problem? Has customer impact stopped, or has one dashboard merely turned green? What must happen before the incident can be closed?

Incident management in IncidentRelay gives those questions a shared place to live. Technical alerts remain connected to their sources and notification policies. An Incident holds the operational work: ownership, participants, affected services, decisions, external references and the follow-up that makes the next response better.

Start with three useful objects

IncidentRelay distinguishes an Alert, an Alert Group and an Incident. Each answers a different question, and each keeps its own history.

Object	What it represents	Example
Alert	A normalized technical signal occurrence.	A checkout timeout signal from a monitoring integration.
Alert Group	A technical episode containing related alerts, with routing, acknowledgement, escalation and source state.	The checkout timeout group assigned to the current application on-call engineer.
Incident	An operational investigation with its own workflow, owner and response context.	Checkout failures in the EU region, linking application errors and database connection warnings.

An Alert Group can be handled without declaring an Incident. An Incident can exist before monitoring produces any alert. One Incident can link several Alert Groups while each keeps its original signals, timestamps and technical responsibility.

This becomes useful as soon as a problem crosses a team or tool boundary. The application team and database team can retain their own alert workflows while working from one operational account of the outage.

Declare an incident when coordination becomes necessary

Use an Incident when the response needs a shared owner, coordinated investigation, stakeholder communication or tracked recovery. Customer impact, an uncertain failure involving several services, a difficult mitigation or a significant near miss are useful declaration criteria.

In IR, an operator can create an Incident directly, create one from an Alert Group, or link a group to an existing Incident. A direct declaration creates an operational record without manufacturing a hidden alert to justify its existence.

That matters when support reports failed purchases before a threshold fires. The responder can record the impact, service, priority and owner immediately, then attach monitoring evidence as it arrives. Conversely, creating a manual Alert Group remains useful when the task is to put a technical signal through the alert workflow.

Choose a title that another team can understand: “EU checkout requests time out” carries more information than “Database issue”. Describe the affected journey and known scope, then update the explanation as evidence improves.

Make the response state explicit

The Incident workflow describes the team's operational progress. Its normal path is:

declared
  → investigating
  → identified
  → monitoring
  → resolved
  → closed

Declared records that coordinated response is needed. Investigating means the team is working through the problem. Identified records that the team has an explanation it can act on. Monitoring makes the observation period after mitigation visible. Resolved records operational recovery. Closed is an explicit decision to finish the incident workflow.

The response can move back to investigation when new evidence or renewed impact demands it. A declaration that is no longer appropriate can be cancelled. These states describe what is happening; responders still need to record the evidence behind a transition.

Acknowledging an Alert Group records a technical acknowledgement. It does not automatically declare the Incident investigated. Resolution of all linked groups can move the Incident to resolved, but closure remains explicit. Closing the Incident does not resolve or delete its linked groups.

Before marking recovery, check the customer journey and any remaining work such as draining a queue. A quiet alert stream and a functioning checkout are different observations. Record which one you have verified.

Keep ownership stable through the response

The Alert Group assignee identifies the current technical responsibility or paging target. The Incident assignee owns the operational investigation. IR keeps those assignments independent, including when either is reassigned manually.

Suppose Nora acknowledges the checkout alert and Sam owns the Incident. At the next schedule boundary, Eli becomes the application on-call engineer. The schedule change does not silently replace Sam as Incident owner. Transfer that responsibility deliberately, with the current impact, decisions and next checks included in the handoff.

Incident collaboration adds an Incident Commander, responders and stakeholders. The commander coordinates the response; responders investigate and perform mitigation; stakeholders need a clear account of impact and progress. A small team may have one person cover several responsibilities, but it should still be clear which responsibilities they are covering.

On-call Roles provide coverage for different response needs and help identify current candidates for ownership, command and responder participation. When a person is assigned to the Incident, IR stores that concrete person. The operational record does not become a moving pointer to whoever happens to be on a schedule later.

Service-default stakeholders are copied into the Incident as a snapshot. The incident's participant context remains understandable even as service configuration changes.

Connect related signals without losing their boundaries

Linking brings evidence into an Incident while preserving the original Alert Groups. Relations such as primary, related, symptom and duplicate explain why a group is attached. The group selected as primary is a coordination reference; that label alone is not proof of root cause.

For the checkout outage, application timeouts can be the primary group, database connection warnings a related group, and growing request latency a symptom. An unrelated backup failure remains its own operational problem even if it arrives during the same ten-minute window.

IR also supports Incident merge and split when the investigation changes the team's understanding. Merge operational records when they describe the same response; split them when evidence reveals independent problems. Use the resulting history to explain the change. Technical Alert Group merging is a separate choice for signals that truly belong in one technical aggregate.

Group classification records the reviewed outcome: Incident, False Positive, Duplicate, Known Issue, No Action Required, Planned Activity, Test Alert or Other. A duplicate points to its canonical group or Incident. That reference is more useful than a label that merely says “somebody else is dealing with it”.

Classification is independent of state. A firing group can already be classified as Incident; a resolved group can be classified as False Positive. Teams can require review for manual completion. Source-driven recovery is still accepted, with unreviewed groups collected in the review queue when that policy applies.

Give responders context they can use immediately

The Incident workspace brings together status, priority, ownership, participants, linked groups, affected-service context, runbooks, dashboards, dependencies and external tickets. That gives the next responder somewhere to start before asking five people to repeat the last twenty minutes.

Keep operational comments focused on decisions, impact and coordination: “Rollback approved; database team checking connection waits; next update at 14:20 UTC.” Technical observations can remain on the relevant Alert Group, close to its source and lifecycle.

The combined activity view can surface linked groups' comments with their original source identified. A note keeps its author, timestamp and target; it is not silently copied into a second conversation. This makes it possible to reconstruct both what the service was doing and what the team understood at the time.

Team and group permissions still govern access. Linking an Alert Group to an Incident does not make restricted source data visible to people who could not otherwise read it. Timeline and audit records provide the history of status, ownership, classification and relationship changes.

Handle flapping without rewriting history

Some failures recover briefly and return. Repeatedly generating fresh groups and sending recovery/failure pairs can bury the actual response in administrative noise.

Event Orchestration lets a team opt into a reopen window for matching Alert Groups. A new occurrence inside that window can reuse a recently resolved group. It creates a new child Alert; the previous resolved occurrence keeps its original state and timestamps. Outside the window, a new group represents the new episode.

If a linked group becomes unresolved again, an Incident in monitoring or resolved can return to investigating. A closed Incident requires an explicit reopen action. Expiry of a technical reopen window does not close an operational Incident.

Delayed recovery notifications address another part of the problem: a quick reopen cancels the pending recovery message while the stored history still records what happened. Reopen policy and notification delay are separate controls, and the reopen behavior is disabled until configured.

Silence from a webhook source is not recovery by default. If a team configures a stale-signal policy, it should do so deliberately for that source's semantics, rather than assuming that missing traffic means healthy production.

Automate declaration with clear decision rules

Event Orchestration connects signal processing to Incident operations. Rules can declare an Incident, prepare a draft for confirmation, attach a group to an open Incident, set classification and apply operational priority or assignment decisions.

Start with rules that a responder can explain. A critical customer-facing service with sustained impact may justify declaration. A signal reaching a later escalation step may require coordinated investigation. An existing Incident for the same failure scope may be the right attachment target.

Check for similar open Incidents and use repeat-safe actions so retries do not create a new investigation every time the same event is delivered. Similarity deserves care: the same service name does not make two failures the same incident. A draft-and-confirm workflow is useful when the rule can identify candidates but a human should make the operational decision.

Duration-based rules continue to evaluate over time; they cannot depend on receiving another webhook at exactly the right moment. Escalation-triggered actions similarly follow the escalation event. This lets automation respond to a problem that persists after the first signal.

Define what handling an alert means

Incident coordination works better when teams agree on the technical handling contract. IR's Alert Handling Policies provide reusable team settings, with a team default and an optional service override.

A policy can enable claim-on-acknowledgement so the person acknowledging a group also takes its technical assignment. This is opt-in and does not rewrite the Incident owner. Without that policy, acknowledgement and assignment remain separate actions.

Priority-based handling SLA rules make overdue technical work visible. The effective rule is captured when handling starts, so changing policy later does not silently move the target for work already underway. The original acknowledgement remains authoritative; repeated ACKs do not start a fresh history.

Notification Policies can deliver acknowledged, resolved and handling-overdue events, including structured lifecycle webhooks. Teams choose where those events are useful. The on-call schedule answers who is available, escalation determines who is paged next, and the handling policy defines the handling expectations.

These controls concern Alert Groups. They do not turn an Incident's operational timeline into an alert SLA clock or transfer command whenever somebody clicks ACK.

Connect Jira without creating two competing accounts

External references connect an Incident to work in an ITSM system. A simple link-only reference is enough when another system already owns the ticket. The Jira and Jira Service Management integration can create tickets and synchronize selected updates through configured mappings and authenticated incoming events.

Decide which system owns the workflow before enabling synchronization. Use the source-of-truth mode and conflict handling to make that decision explicit. An IR-led response and an externally managed service-desk workflow can use different arrangements; a bare ticket link does not imply full synchronization.

Queued delivery, retries and duplicate protection handle temporary provider failures. Failed synchronization stays visible so operators can distinguish “the Incident was updated” from “Jira received the update”. Loop prevention keeps an incoming update from bouncing between the systems indefinitely.

For example, Sam can coordinate recovery in IR while a linked Jira ticket tracks the lasting configuration fix. The Incident can reach resolved while the engineering ticket remains open. Map statuses according to that meaning instead of assuming that every green state in one system should close everything in the other.

A checkout incident from declaration to closure

Consider this fictional exercise. The names, times and service behavior below illustrate the workflow; they are not a report of a real outage.

Time (UTC)	What the team does	What IR keeps connected
14:00	Support reports EU checkout failures. Sam declares an Incident and records the known impact.	A standalone Incident with an owner, priority and service context.
14:02	Checkout and database alerts arrive. Nora and Dev take the technical investigations.	Two linked Alert Groups with their original routes, assignees and histories.
14:04	Sam coordinates the response and records the next stakeholder update time.	Commander, responders, stakeholders and an operational decision log.
14:09	The team identifies a configuration change and begins a rollback.	The identified state, supporting evidence and mitigation decision.
14:13–14:15	The groups recover briefly, then the checkout symptom returns inside the configured reopen window.	A new alert occurrence in the reused group and a return from monitoring to investigating.
14:24	The rollback is complete. Synthetic purchases and backlog checks confirm recovery.	Recovery evidence, resolved Incident state and the resolution summary.
Later	The owner completes the operational closure checks and assigns review actions.	Explicit closure, a linked Jira ticket, the postmortem and accountable follow-up.

The group reopen window, notification delay and response checks in this exercise are configuration choices. The useful outcome is that a temporary green interval does not erase the earlier occurrence or leave the team coordinating in a record marked finished.

Turn the incident into improvements with owners

IR's configurable Postmortems and Incident Reviews carry the investigation into follow-up. The Incident already provides ownership, linked signals, activity and recovery context. The review adds an agreed account of impact, contributing conditions, decisions and unresolved questions.

Separate the root-cause explanation from the resolution summary. “Rolled back the pool configuration” explains how service was restored. It does not explain why the change passed review, whether a rollout check missed the impact or what would prevent recurrence.

Corrective actions have owners and due dates, with export and reporting to keep the work visible. Make acceptance evidence part of each action's description. For the fictional checkout incident, use actions such as:

• Nora, within five working days: add a representative concurrency check; demonstrate that it rejects the incident configuration and accepts the agreed safe baseline.
• Dev, within seven working days: rehearse the rollback with another responder; record the restored configuration and customer-journey checks.
• Sam, at the next review: confirm the Incident declaration rule attaches repeat signals to the intended open investigation and leaves an independent failure separate.

Operational closure and corrective-action completion are separate milestones. Agree on the closure criteria, keep unfinished work visible, and record verification before considering an improvement complete. “Ticket created” is an excellent description of a ticket being created.

Measure the stages you actually want to improve

Incident and Alert Group analytics expose different parts of the response. Review declaration, investigation, resolution and closure times alongside alert acknowledgement, classification outcomes, group-to-Incident conversion, linked-group counts and external-ticket synchronization failures.

Use the exact timestamp definitions behind each report. Time from declaration to resolved says something different from time between the first customer failure and stable recovery. Time to closed includes operational completion work that may continue after the customer impact ends.

Track Incident reopenings separately from Alert Group reopenings and flapping. Several short technical recurrences may belong to one operational response. Counting every occurrence as a brand-new incident can make one difficult afternoon look like a month of outages.

Classification reports help test whether alert-quality work is effective. A lower page count is useful when independent problems remain visible, responders still detect customer impact, and the review queue is being completed. A quieter graph alone cannot establish those outcomes.

Start with a workflow your team can explain

A practical starting configuration for IR is one service, a small set of declaration criteria, explicit operational ownership and a short list of review outcomes. Exercise the whole path before expanding automation:

1. Declare an Incident from a human report, then link the relevant Alert Groups.
2. Confirm that technical assignment and operational ownership remain independent through a shift change.
3. Record an impact update, mitigation decision and recovery check where the next responder can find them.
4. Exercise a brief recurrence and verify the group and Incident histories.
5. Test external-ticket delivery failure and recovery if synchronization is enabled.
6. Close the operational record explicitly and verify one corrective action from the review.

The value of Incident Management is visible when the next responder can answer who owns the problem, what customers are experiencing, which evidence supports the current explanation and what happens next. IncidentRelay keeps that operational account connected to the technical signals throughout the response.

For the underlying response practice, Google's Incident Response chapter explains role clarity, coordination and maintaining a working record. For related IR workflows, read alert grouping without hidden problems, the on-call handoff checklist and the practical postmortem guide.