]> Phono Technologies Inc.

Kitchener Ontario Canada mateenali66@gmail.com

Operations and Management Network Management Operations anomaly detection evaluation benchmarking telemetry AIOps The Network Management Operations (NMOP) working group has adopted documents describing an architecture, an operational lifecycle, and a semantics for network anomaly detection. Those documents direct implementers to minimize false positives and false negatives, but do not define how the accuracy of an anomaly detection implementation is to be measured, compared, or tracked over time. This document describes an evaluation methodology for machine-learning-based anomaly detection systems operating on network and infrastructure telemetry: the metrics to report and their known failure modes, a benchmarking procedure based on controlled fault injection and replay, and the properties a benchmark dataset needs in order to support reproducible, comparable evaluation. The methodology is informational and complements the adopted NMOP anomaly-detection documents.

Introduction The NMOP working group has adopted three documents that together describe how network anomaly detection is to be architected, operated, and annotated: an architecture for Service Disruption Detection , an operational lifecycle for iterative refinement of detection systems , and a semantics for annotating detected anomalies . All three documents treat detection accuracy as a central objective. The architecture requires that "the operator can trust that a targeted Service Disruption will be detected (no false negatives), while not spamming the operator with Alarms that do not reflect an actual issue within the network (false positives)" (Section 3.5 of ). The lifecycle states that "Providers of solutions for network anomaly detection should aim at increasing accuracy, by minimizing False Positives and False Negatives" (Section 1.3 of ), and its refinement stage concludes by "'replaying' the network anomaly and checking if there is any measurable improvement" (Section 5.3.2). None of the three documents defines how that accuracy or that improvement is to be measured. No metric, formula, or comparison procedure is specified. The consequence is visible in the architecture document's own deployment report, which counts 20 monitored service disruptions against 52 false positives (Section 4.1 of ) but has no framework in which to express whether that ratio is acceptable, improving, or comparable to an alternative det