Automatic incidents

With incidents.enabled, a check run can open, progress and resolve incidents on its own. Automatic incidents are marked auto: true, and only those are reconciled — an incident you opened by hand is never touched by the monitoring loop.

stateDiagram-v2
[*] --> investigating: failures reach failureThreshold
investigating --> monitoring: a check succeeds
monitoring --> investigating: it fails again
monitoring --> resolved: five minutes of stability
resolved --> [*]

Opening

On each run, for every failing monitor not covered by an in-progress maintenance window:

1. Trailing consecutive failures are counted from persisted history, once for the whole batch.
2. If no active automatic incident already covers the monitor, and the count has reached notifications.failureThreshold, an incident is created.

The generated incident is:

{  title: "API is unavailable",  severity: "major",  affectedMonitors: ["API"],  status: "investigating",  auto: true,  // message: the monitor's own failure message}

An incident covers a monitor when its affectedMonitors contains the name — or is empty, which means global.

Recovering

When a monitor that has an open automatic incident succeeds:

• the incident moves to monitoring with "<monitor> has recovered. Monitoring stability.", and
• once it has been monitoring for five minutes, it is resolved with "<monitor> has remained operational for five minutes. Resolved.".

The five-minute timer is the timestamp of the monitoring timeline entry, not a process timer, so it survives restarts, redeploys and serverless invocations that share nothing.

If the monitor fails again while the incident is monitoring, the incident returns to investigating with "<monitor> is failing again. Continuing investigation." — a single flapping check can never resolve an outage early.

Events

These are dispatched together with the monitor events from the same run, so a channel sees monitor.down and incident.created in one batch.

Tuning

failureThreshold is shared with notifications on purpose: the moment a monitor is considered down is one decision, not two that can drift apart.

Interaction with manual incidents

• Manual incidents (auto: false) are never auto-updated or auto-resolved.
• An automatic incident can be edited by hand through PATCH /admin/incidents/:id; it keeps auto: true and remains eligible for automatic resolution.
• To stop reconciliation on one incident, resolve it — reconciliation only looks at investigating, identified and monitoring.