]> Phono Technologies Inc.

Kitchener Ontario Canada mateenali66@gmail.com

Operations and Management Network Management Operations anomaly detection evaluation benchmarking telemetry AIOps The Network Management Operations (NMOP) working group has adopted documents describing an architecture, an operational lifecycle, and a semantics for network anomaly detection. Those documents direct implementers to minimize false positives and false negatives, but do not define how the accuracy of an anomaly detection implementation is to be measured, compared, or tracked over time. This document describes an evaluation methodology for machine-learning-based anomaly detection systems operating on network and infrastructure telemetry: the metrics to report and their known failure modes, a benchmarking procedure based on controlled fault injection and replay, and the properties a benchmark dataset needs in order to support reproducible, comparable evaluation. The methodology is informational and complements the adopted NMOP anomaly-detection documents.

Introduction The NMOP working group has adopted three documents that together describe how network anomaly detection is to be architected, operated, and annotated: an architecture for Service Disruption Detection , an operational lifecycle for iterative refinement of detection systems , and a semantics for annotating detected anomalies . All three documents treat detection accuracy as a central objective. The architecture requires that "the operator can trust that a targeted Service Disruption will be detected (no false negatives), while not spamming the operator with Alarms that do not reflect an actual issue within the network (false positives)" (Section 3.5 of ). The lifecycle states that "Providers of solutions for network anomaly detection should aim at increasing accuracy, by minimizing False Positives and False Negatives" (Section 1.3 of ), and its refinement stage concludes by "'replaying' the network anomaly and checking if there is any measurable improvement" (Section 5.3.2). None of the three documents defines how that accuracy or that improvement is to be measured. No metric, formula, or comparison procedure is specified. The consequence is visible in the architecture document's own deployment report, which counts 20 monitored service disruptions against 52 false positives (Section 4.1 of ) but has no framework in which to express whether that ratio is acceptable, improving, or comparable to an alternative detector. This document supplies the missing companion. It describes an evaluation methodology for anomaly detection systems operating on network and infrastructure telemetry: which metrics to report and the known failure modes of each (), a benchmarking procedure based on controlled fault injection and replay (), the properties a benchmark dataset needs for reproducible comparison (), and the minimum contents of an evaluation report so that results from different systems can be compared at all (). The methodology is grounded in a published peer-reviewed evaluation of eight machine-learning detection methods on unified telemetry , summarized as a worked example in .

Relationship to the NMOP Anomaly Detection Documents This document is designed to be used inside the lifecycle defined by , not alongside it: The lifecycle's Validation stage decides, per detected Symptom, whether a real problem was signaled. Accumulated validation outcomes are exactly the labels this document's metrics consume (). The lifecycle's Refinement stage requires a check for "measurable improvement" after replaying an anomaly. turns that check into a defined procedure with defined measures. The architecture's Replaying capability (Section 3.8 of ), which preserves the data that led to an Alarm so that the detection algorithms can be rerun on the same input, is the natural execution environment for the replay-based evaluation in . One definitional difference must be reconciled explicitly. defines a False Positive as "a detected anomaly which has been identified during the postmortem to be not anomalous", an operational, postmortem judgment. Dataset-based evaluation instead defines false positives against ground-truth labels fixed before evaluation. Both definitions are useful and they converge as validated postmortem outcomes accumulate into a labeled corpus; defines the ground-truth variants used in this document, and describes how operational validation records become benchmark labels.

Scope This document specifies methodology, not protocol. It defines no new data models, YANG modules, or wire formats. It applies to anomaly detection systems that consume telemetry signals (metrics, logs, traces, or their combination) regardless of the detection technique, although the metric guidance in is motivated by the failure modes observed when evaluating machine-learning-based detectors.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This document uses the following terms as defined elsewhere and does not redefine them: System, State, Problem, Symptom, and Alarm from ; Outlier Detection, Service Disruption Detection (SDD), and Service Disruption Detection System (SDDS) from ; and False Positive, False Negative, and Confidence Score in their operational sense from (see for the reconciliation with the ground-truth sense used here). The following evaluation-specific terms are defined for this document:

Ground Truth:

The authoritative record of which intervals or events in a telemetry corpus are anomalous, fixed before an evaluation begins. Ground truth may originate from a controlled injection schedule () or from accumulated, validated postmortem outcomes.

Label:

The per-interval or per-event value assigned by the ground truth (anomalous or non-anomalous), against which detector output is scored.

Evaluated Set:

The exact collection of intervals or events over which metrics are computed. Metrics computed over different evaluated sets are not comparable ().

Prevalence:

The fraction of the evaluated set that is labeled anomalous.

Detection Window:

The fixed time interval into which telemetry is aggregated before a detection decision is made for that interval.

Detection Latency:

The elapsed time between the onset of an anomalous condition per the ground truth and the first detection decision that flags it.

Evaluation Metrics Detection quality cannot be summarized in a single number. This section defines a minimum metric set and, for each threshold-dependent metric, the conditions under which it misleads. The guidance here is grounded in observed failure modes from a controlled evaluation of eight detection methods ; the worked example in shows each failure mode with real values.

Threshold-Independent Metrics An anomaly detector typically produces a continuous score per detection window, converted to a binary decision by a threshold. Metrics computed over the score distribution, independent of any threshold, are the most robust basis for comparing detectors: AUC-ROC (area under the receiver operating characteristic curve) SHOULD be the metric of record for ranking and acceptance decisions. It is threshold-independent and insensitive to prevalence, so values remain comparable across evaluated sets with different anomaly rates. An AUC-ROC near 0.5 indicates a detector performing no better than chance, regardless of any other metric reported for it. PR-AUC (area under the precision-recall curve) SHOULD be reported alongside AUC-ROC. It focuses on the anomalous class and is the more informative of the two when prevalence is low, the common operational condition.

Threshold-Dependent Metrics and the Prevalence Problem Precision, recall, and their harmonic mean F1 describe a single operating point and remain the terms in which operators reason ("how many alarms are real", "how many disruptions are missed"). They MUST NOT, however, be used in isolation for acceptance or comparison, for two reasons. F1 has a nontrivial floor. A degenerate detector that flags every window achieves recall 1 and F1 = 2p/(1+p), where p is the prevalence of the evaluated set. At a prevalence of 0.4 this floor is approximately 0.57; a reported F1 of 0.6 at that prevalence is therefore nearly indistinguishable from a detector that has learned nothing. The same degenerate detector falls to F1 of roughly 0.10 at 5% prevalence, which is why F1 values obtained at different prevalences are not comparable at all. An evaluation MUST report the prevalence of its evaluated set and the F1 of the trivial flag-everything detector on that set as a baseline, and SHOULD verify AUC-ROC exceeds 0.5 before any F1 value is credited. Threshold selection can additionally mask model quality in either direction. If the threshold is tuned on labeled validation data, reported F1 reflects an optimistic operating point unavailable in production, where thresholds are typically set from the score distribution of normal traffic alone (for example, a high percentile of normal-only scores). An evaluation MUST state how thresholds were selected, and SHOULD report the sensitivity of its threshold-dependent metrics to that choice.

Episode-Level Detection and Latency Window-level metrics measure classification quality; operators additionally need to know whether whole incidents are caught and how quickly. An episode is one contiguous anomalous condition per the ground truth (for a benchmark, one fault-injection run). An episode counts as detected if at least one window within its labeled span is flagged. Episode-level detection rate is a recall-oriented measure and is trivially satisfied by the flag-everything detector, so it MUST be reported together with window-level precision and AUC-ROC, never alone. Where onset timestamps are available, detection latency () SHOULD be reported as a distribution, not only a mean.

Window Sensitivity The detection window is an experimental parameter with direct effect on results: aggregation over a longer window smooths transient faults, a shorter one raises noise. Conclusions that hold only at one window size are fragile; in the evaluation underlying this document, one detector performed well at a 30-second window and no better than chance at 10 and 60 seconds (). An evaluation MUST state its primary window size and its rationale, and SHOULD re-run its headline comparison at at least one neighboring window size, reporting whether the qualitative conclusions are stable.

Benchmarking Procedure This section describes a repeatable procedure for producing the ground truth and the evaluated set that the metrics in require. The procedure is based on controlled fault injection into an instrumented environment; it is the offline complement to the operational Validation stage of .

Environment and Instrumentation The environment SHOULD be a representative multi-component system emitting the telemetry signal types the detector under test consumes, unified through a common collection pipeline so that metrics, logs, and traces share a timeline. Two instrumentation patterns exist in practice, and a benchmark gains external validity by covering both: components instrumented natively at development time, and components instrumented after the fact through operator- or agent-based retrofit. A baseline period of normal operation MUST be captured before any fault is injected; it supplies training data for detectors that learn from normal behavior only, and the score distribution from which production-style thresholds are derived ().

Fault Injection Faults MUST be injected under an explicit, recorded schedule; the schedule is the ground truth. A useful minimum taxonomy, drawn from , covers: added network latency at graded severities, injected request errors, CPU saturation, memory saturation, and compound cascades that combine a fault on one component with resource stress on a dependent component. Injection targets SHOULD be chosen by architectural position (entry point, mid-chain dependency, backend) rather than convenience, and each scenario SHOULD be repeated multiple times so that dispersion across repetitions can be reported. Each run SHOULD include a defined active-fault phase followed by a cooldown phase long enough for the system to return to nominal behavior.

Ground-Truth Labeling Labels derive from the injection schedule: windows within an active-fault phase are anomalous. Cooldown windows require an explicit decision. The system is recovering but no fault is active; labeling them anomalous inflates prevalence with ambiguous intervals, while silently including them as normal penalizes detectors that correctly flag residual disturbance. The evaluation MUST state how cooldown windows were treated, and SHOULD report its headline results with cooldown windows excluded from the evaluated set, with the resulting prevalence disclosed. When evaluation data is partitioned for training, validation, and testing, the partition MUST be by episode (whole runs), never by random sampling of windows; windows from the same run are temporally correlated, and random partitions leak that correlation into inflated results. Operational deployments accumulate an equivalent ground truth over time: each postmortem-validated outcome under the lifecycle of is a label. describes how such records become benchmark datasets.

Replay-Based Evaluation Section 3.8 of calls for an SDDS to preserve the data that led to an Alarm so that the detection algorithms can be rerun on the same input. That capability turns the lifecycle's "measurable improvement" check into a defined measurement: hold the evaluated set fixed, rerun the modified detector on the identical preserved input, and compare the metrics of before and after the change. A comparison in which the evaluated set itself changed between runs is not a measurement of the detector and MUST NOT be reported as an improvement.

Benchmark Dataset Requirements A benchmark dataset supports reproducible, comparable evaluation only if it contains, at minimum: the telemetry itself, covering every signal type the benchmark claims, on a common timeline; the ground truth: the full injection schedule (or validated postmortem record), including fault type, target, severity, and the active and cooldown interval boundaries for every episode; the baseline period of normal operation, clearly delimited; enough topology and deployment metadata to interpret the faults (component inventory, dependency relationships, instrumentation method per component); documentation of the collection pipeline, since preprocessing choices (aggregation windows, feature construction) are part of what is being reproduced. Datasets intended for comparison across publications SHOULD be deposited with a persistent identifier and explicit versioning. Where a repository distinguishes a concept identifier (resolving to the latest version) from version identifiers, an evaluation MUST cite exactly one version and state which, since results are not reproducible against "latest". An example of a dataset meeting these requirements is . Results obtained on a single dataset, however carefully constructed, are results about that dataset's fault distribution. In the evaluation underlying this document, the detector that ranked first on the study's primary benchmark performed at or below chance on an independently collected third-party benchmark, while the failure modes described in reproduced on both (). An evaluation making claims beyond a single environment SHOULD therefore include at least one dataset collected by a different team with a different pipeline, and SHOULD present ranking claims as specific to the evaluated fault distribution.

Reporting Requirements An evaluation report that permits comparison MUST state: the definition of the evaluated set, its prevalence, and the treatment of cooldown/recovery windows (); the detection window size and, where checked, window sensitivity (); the threshold selection procedure, including whether labeled anomalies were available to it; AUC-ROC and PR-AUC, plus precision, recall, and F1 at the stated operating point, each with dispersion across folds or repetitions rather than a single point value; the flag-everything F1 baseline for the evaluated set's prevalence; episode-level detection rate together with the window-level metrics it qualifies; the exact dataset version used, by persistent identifier. A report SHOULD additionally state the known limits of its claims: the fault distribution evaluated, the observation horizon (whether multi-day effects such as seasonality or drift are covered at all), and whether the rankings reported are claimed to transfer beyond the evaluated environment. Rankings SHOULD be presented as relative performance under the stated fault distribution, not as universal detector quality.

Security Considerations Benchmark datasets derived from operational environments can reveal network topology, service names, traffic patterns, capacity limits, and failure behavior of the originating operator. Datasets intended for publication SHOULD be anonymized with respect to addresses, hostnames, service identifiers, and tenant identifiers, and SHOULD be reviewed for indirect disclosure through correlated fields before release. Replay-based evaluation environments reproduce failure conditions by design and MUST be isolated from production systems.

IANA Considerations This document has no IANA actions.

In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Swisscom Swisscom INSA-Lyon Deutsche Telekom This document describes the motivation and architecture of a Network Anomaly Detection Framework and the relationship to other documents describing network Symptom semantics and network incident lifecycle. The described architecture for detecting IP network service interruption is designed to be generic applicable and extensible. Different applications are described and examples are referenced with open-source running code. Huawei Swisscom Swisscom Deutsche Telekom This document defines a structured, iterative lifecycle for network anomaly detection systems to enable "human-in-the-loop" refinements. Key contributions include defining three lifecycle stages, a state machine for anomaly annotations, and YANG data models for standardized labeling and exchange. Swisscom Swisscom Deutsche Telekom Huawei The document proposes a unified symptoms vocabulary for network anomaly metadata to improve data sharing and analysis among human network operators, network analytics implementers, and AI systems to improve accuracy of Service Disruption Detection. This document sets out some terms that are fundamental to a common understanding of network fault and problem management within the IETF. The purpose of this document is to bring clarity to discussions and other work related to network fault and problem management -- in particular, to YANG data models and management protocols that report, make visible, or manage network faults and problems.

Worked Example: An Application of This Methodology This informative appendix summarizes the published evaluation as an end-to-end application of and , including the observed failure modes that motivate the normative guidance. Environment and campaign. Two Kubernetes-hosted microservice testbeds were instrumented and unified through an OpenTelemetry Collector exporting metrics, logs, and traces to a common store: one natively instrumented 14-service system and one 7-service system retrofitted through operator-based auto-instrumentation, deliberately covering both instrumentation patterns of . A 24-hour normal-operation baseline was captured, then 40 fault scenarios (graded network latency, injected request errors, CPU and memory saturation, and compound cascades, targeted by architectural position) were each run 10 times, with a 5-minute active-fault phase and 5-minute cooldown per run, producing roughly 51 GB of telemetry over a 33.5-hour campaign. Labels derived from the injection schedule; the primary evaluated set excluded cooldown windows, yielding prevalences of roughly 0.27 to 0.41 depending on signal, all disclosed. Partitioning was by repetition, meaning independent experimental replicates separated in time, across five folds. The primary detection window was 60 seconds, with the full pipeline re-run at 10 and 30 seconds as a sensitivity check. Eight semi-supervised detection methods spanning four families were evaluated, with a supervised baseline for reference. The F1-inflation failure mode. At the metrics signal's prevalence, the flag-everything F1 floor was 0.582. Three of the eight models, all autoencoder variants, converged to exactly that value (F1 0.582 +/- 0.001) while their AUC-ROC lay between 0.17 and 0.22, far below chance: seemingly respectable F1 produced by degenerate detectors. Because the inversion yields high F1 at near-random AUC-ROC, the study adopted AUC-ROC as the metric of record and F1 as an operating-point summary read against its prevalence baseline, the rule generalized in . Window sensitivity. Qualitative conclusions were stable across the three window sizes, but one detector (a one-class deep method) performed well only at the 30-second window (AUC-ROC 0.916) and near chance at 10 and 60 seconds (0.509 and 0.553) despite F1 near 0.74 at both failing windows: a single-window evaluation at either extreme would have credited a detector that was not discriminating at all. Prevalence extrapolation. Re-scoring at a synthetic 5% prevalence, closer to production conditions, collapsed the flag-everything F1 floor to about 0.10 and reduced the best detector's F1 from 0.906 to 0.670, quantifying how strongly experimental prevalence overstates practical utility, the reason requires prevalence disclosure. External validity. The strongest detector on the primary benchmark (a density-estimation deep model, F1 0.906 +/- 0.060 and AUC-ROC 0.960 +/- 0.043 on metrics) performed at or below chance (AUC-ROC 0.33 to 0.52) on an independently collected third-party benchmark, where a classical isolation-based method was strongest (AUC-ROC up to 0.95). The failure modes, by contrast, transferred: on the external benchmark the reconstruction-based methods again posted deceptively non-trivial F1 (0.65 to 0.71) at AUC-ROC of only 0.50 to 0.60. The study accordingly claims no universal ranking, and this document's requirement for an independent second dataset follows directly from that observation. The processed feature sets, per-fold results, and full reproduction code are available at .

Acknowledgments