Incident data retention without regret: what to keep, what to delete, and when

Incident history has an awkward career path. During an outage it is critical evidence. A week later it feeds the postmortem. A few months later it might support an SLO review, a customer question or a recurring-problem investigation. Eventually, however, it becomes a growing collection of rows that everyone is afraid to delete because nobody wrote down why they were keeping them.

IncidentRelay 2.1 adds a central [retention] policy for resolved alert history, Explain Trace records and general Event Orchestration execution traces. The feature is deliberately conservative: resolved alert cleanup is disabled by default, so upgrading does not silently remove history.

The difficult part is not writing alert_days = 90. It is choosing a number that matches how your team actually investigates incidents. Let us do that before the storage graph develops a personality.

One incident is more than one database row

A resolved alert group can own considerably more than its title and timestamps. Its dependent history may include child alerts, lifecycle events, comments, notification deliveries, responder and stakeholder state, silence or maintenance applications, correlation rows and Explain Trace data.

That is why retention should be discussed as an operational policy, not as a nightly DELETE somebody found on a forum. You need to know which evidence disappears together and which evidence remains.

When IncidentRelay removes an eligible resolved group, its group-owned history is removed through the database relationships. Old standalone resolved alerts are also eligible. Audit log records remain because they store scalar object identifiers rather than owning the deleted alert records.

Retention is not backup, and an audit log is not a postmortem archive. Each solves a different problem. Giving all three the same job is how three disappointed systems meet in one incident.

First decide what question the data must still answer

Start from the longest real investigation window, not from a pleasing round number. Ask the people who use the history:

• How long after an incident do postmortems normally happen?
• Do monthly or quarterly reliability reviews reopen individual incidents?
• How far back can a customer, auditor or security reviewer reasonably ask for evidence?
• How long does it take the team to notice a recurring alert pattern?
• Are full payload details necessary for that work, or are ordered processing steps enough?
• Does an external system already hold the long-term record?

If postmortems happen within two weeks but quarterly reviews need incident counts and ownership, you may keep alert history for a quarter while keeping bulky diagnostic detail for a shorter period. If regulation requires a year of evidence, verify exactly which fields count as evidence; “keep the entire production database forever” is rarely the clause the lawyer intended.

Separate how much you write from how long you keep it

IncidentRelay gives you two independent controls. Explain Trace level controls the amount of detail written for new events. Retention controls how long persisted records remain.

Control	Question	Options
[alerts] explain_trace_level	How much trace detail should a new event write?	full, compact, disabled
[retention]	How long should already persisted history remain?	Days per record family; 0 means indefinite

full keeps the complete Explain Trace, including input summary, result payload and per-step data. compact keeps the ordered processing steps but leaves those detailed JSON fields empty. disabled creates no Explain Trace or Explain Step rows for the event.

A global orchestration can apply set_trace_level to matching traffic. That makes a useful tiered policy possible: full traces for production P1 events, compact traces for ordinary alerts and disabled traces for a trusted high-volume source whose routing is already boring. Boring routing is a compliment.

Understand the three retention clocks

[retention]
alert_days = 90
explain_trace_days = 30
orchestration_execution_days = 30
cleanup_interval_seconds = 86400
batch_size = 500

Setting	What it covers	If omitted
alert_days	Resolved groups and standalone resolved alerts, including group-owned history	Defaults to 0: keep indefinitely
explain_trace_days	Standalone Explain Trace records	Inherits alert_days
orchestration_execution_days	General Event Orchestration execution traces	Inherits alert_days

The word standalone matters. If an Explain Trace belongs to an alert or group, deleting the owning alert history also deletes that trace. Setting explain_trace_days = 365 cannot preserve a linked trace after alert_days = 30 removes its parent.

An explicit trace value of 0 disables cleanup for that standalone trace family. General orchestration retention also has independent special rules for dropped events, terminal pending events and webhook executions; those records may disappear earlier than the general window.

Three starting policies that are easy to explain

These are conversation starters, not universal commandments. Map them to your review and compliance needs.

A small team that wants predictable growth

[retention]
alert_days = 90
explain_trace_days = 30
orchestration_execution_days = 30
cleanup_interval_seconds = 86400
batch_size = 500

Ninety days covers monthly review cycles and gives a postmortem plenty of breathing room. Thirty days of diagnostics is often enough to debug recent routing changes without retaining every JSON detail for a whole quarter.

A regulated or customer-audited service

[retention]
alert_days = 365
explain_trace_days = 90
orchestration_execution_days = 90
cleanup_interval_seconds = 86400
batch_size = 500

Long alert history supports annual evidence requirements while traces remain shorter. Confirm the values with whoever owns compliance, privacy and contracts. Engineers should not reverse-engineer legal retention from vibes and an old ticket.

Keep incident history, aggressively limit diagnostics

[alerts]
explain_trace_level = compact

[retention]
alert_days = 0
explain_trace_days = 14
orchestration_execution_days = 14

This is useful while a team is not ready to delete resolved incidents but wants to cap standalone diagnostic growth. Remember that linked traces remain attached to indefinitely retained alerts unless another lifecycle removes the parent.

Estimate storage from your own traffic

Generic “bytes per incident” numbers are misleading because payload size, orchestration steps, notification fan-out and comments vary wildly. Measure your system instead.

1. Choose a representative period, such as 14 or 30 days.
2. Record database size at the start and end, excluding unrelated bulk changes where possible.
3. Count resolved groups and standalone alerts created during the same period.
4. Note whether tracing was full, compact or mixed.
5. Calculate daily growth and project it across the intended window.

daily growth = database growth during sample / sample days
steady retained working set ~= daily growth x retention days
capacity target = working set + indexes + active data + safety headroom

Use the projection for capacity planning, not as a promise that the file will stop at the exact result. Active incidents, indexes, audit logs and other product data have their own growth. Keep headroom for traffic spikes and maintenance; a database at 99% disk usage is already writing its resignation letter.

Know exactly when an alert becomes eligible

The alert clock starts at resolved_at, not at creation time. A group is eligible only when it is resolved, its resolved_at is older than the cutoff and every alert still attached to it is also resolved.

Active, acknowledged, silenced, reactivated and otherwise non-terminal incidents are not selected by this cleanup. This prevents a long-running incident from vanishing merely because it was created before the retention window.

That safety rule is also a diagnostic clue. If an old group remains after several cleanup cycles, inspect the group status, resolved_at and child alert states before assuming the scheduler is broken. One unresolved child is enough to keep the family together. Database relationships: stricter than most group chats.

Roll it out like a data-deletion feature

Retention is routine maintenance after deployment, but its first run may remove months or years of accumulated history. Treat that first run as a controlled change.

1. Write the policy. Record the chosen windows, their owner and the operational or legal reason.
2. Measure the backlog. Estimate how many resolved groups and standalone alerts are already older than the cutoff.
3. Verify backups. Take a current backup and confirm the restore procedure works. A backup that has never been restored is a hopeful file.
4. Choose a quiet window. The cleanup is online and batched, but the first pass deserves observation.
5. Apply the configuration. Use the normal IncidentRelay configuration deployment path.
6. Watch several runs. Confirm the scheduler completes, database latency remains normal and eligible history decreases.
7. Sample what remains. Check an active incident, a recently resolved incident and an item on either side of the cutoff.
8. Review after one cycle. Ask responders whether any workflow needs a longer window before tightening it further.

batch_size limits how many alert groups or standalone alerts are deleted in one transaction. The default is 500. If the initial backlog creates noticeable database pressure, use a smaller batch and let cleanup make gradual progress. If the database is quiet and the backlog is large, tune carefully from measurements rather than converting production into a benchmark.

Do not judge cleanup by file size alone

On SQLite and PostgreSQL, deleting rows normally makes database pages reusable for future writes. It does not guarantee that the file on disk immediately becomes smaller. That is expected behavior, not proof that retention failed.

Verify logical results first: old eligible records disappear, newer and active records remain, and the cleanup job completes. If you must return allocated space to the operating system, use the database engine's normal maintenance procedure during an appropriate maintenance window. Do not improvise storage surgery while on-call. The irony would be too expensive.

A compact verification checklist

• The policy owner and reason are documented.
• The window covers postmortem, reliability review, customer and compliance needs.
• Backup restore has been tested before the first destructive run.
• alert_days is explicit; trace inheritance is understood.
• Trace level and trace retention are treated as separate controls.
• Linked traces are not expected to outlive their deleted parent alerts.
• Special orchestration trace rules are accounted for.
• Scheduler completion and database latency are monitored.
• Records on both sides of the cutoff are sampled.
• Database file size is not the only success metric.

Upgrading from IncidentRelay 2.0

Use the new [retention] section for all new configuration. IncidentRelay 2.1 can still read the old [alerts] alert_explain_trace_retention_days value when retention.explain_trace_days is absent, and it accepts previous cleanup interval settings as fallbacks. Explicit new retention settings take precedence.

Do not leave the fallback as tribal knowledge. Move the values into [retention], document the intended inheritance and remove obsolete settings through your normal configuration review.

The useful policy is the one the team can defend

A good retention policy can be explained in one paragraph: what is kept, for how long, who needs it and what happens when it expires. It preserves enough history for real investigations while keeping diagnostic detail proportional to its value.

Start conservatively, measure actual growth, confirm your restore path, then shorten windows only with evidence. The goal is not the smallest possible database. The goal is a database that contains the evidence you need and does not become tomorrow's most predictable incident.

For the exact settings and deletion behavior, read the Data Retention guide. The Explain Trace guide covers trace levels, global orchestration overrides and trace access.