ADR-0027: Fault correlation is a debounced, domain-scoped pure recompute of world state

Status

Accepted

Date

2026-07-03

Context

A single feeder cut raises hundreds of alarms within seconds. The correlation engine must group them into one ticket (or the right number of tickets) under the kernel’s decreed consistency model — idempotent replay plus at-least-once workers, no cross-port transactions (ADR-0013). The naive design — every alarm triggers an incremental find-or-create against a ticket — is a race-condition factory under at-least-once execution: interleaved runs split one fault into several tickets unless locking machinery is introduced, which ADR-0013 declines to have.

Decision

Correlation is a pure recompute of a correlation domain, debounced.

Each RaiseAlarm/ClearAlarm assert enqueues a correlation job whose idempotency key is the correlation domain — the OLT subtree (a PON tree is a natural fault-isolation boundary), found by upstream traversal from the affected entity — plus a debounce bucket. An alarm storm collapses to one job per domain per window.
The job ignores the triggering alarm entirely. It reads all active alarms in the domain from current world state, partitions them by root-most implicated ancestor (traversal over the network.pon_downstream family, direction-inverted), and emits one ticket per partition — merging into open tickets rather than duplicating, attaching fault.grouped_into and fault.symptom_of links, and writing the Prediction statement and PROVISIONAL fault.suspected_root_cause link per ADR-0025. A partition that has emptied (alarms cleared) auto-resolves its ticket via a machine Act per ADR-0026.
Because the job is a pure function of world state, running it twice — or concurrently, or after a crash — converges to the same tickets. ADR-0013’s “consistency is repaired, not prevented” made into an algorithm.

Determinism is load-bearing for quality: the synthetic alarm injector knows ground truth (which strand it cut), so tests can score the engine — same alarm state in, same tickets and root causes out. The debounce constant is configuration (~2s demo, ~30s NOC-realistic); delayed execution piggybacks the existing job retry-scheduling columns rather than a new Schedule verb.

Alternatives Considered

Per-alarm incremental correlation: rejected — lowest per-alarm latency, but find-or-create under concurrent at-least-once runs needs locking or leases the consistency model doesn’t offer, and failure mode is split tickets.
Periodic sweep: rejected — simplest code, but correlation latency equals the sweep period, and it converts an event-driven pipeline into a polling engine.

Consequences

Two simultaneous faults in one PON tree fall out naturally: the partition step yields two root-most ancestors, hence two tickets.
Correlation latency is bounded below by the debounce constant — a conscious trade against storm-collapse. The constant is per-deployment configuration, not code.
Recompute cost scales with domain size. PON trees are small; revisit trigger: correlation domains that stop being tree-scale (metro rings, mesh cores), where partition-by-ancestor also needs the multi-root (DAG) treatment the diamond traversal contract already anticipates.
New alarms during an open incident merge into the open ticket for their partition; ticket churn is bounded by partition changes, not alarm volume.