



























Abstract:Translating high-level network intents into correct multivendor configurations remains a central challenge in network automation, as syntactically valid outputs may still violate the intended operational state. Despite recent advances in Large Language Models (LLMs), the field still lacks reproducible semantic benchmarks for rigorous cross-vendor evaluation. This paper presents a reproducible DSM-to-CLI semantic benchmark covering five cloud LLMs, three vendors, five representative use cases, and ten repeated runs per experimental cell under fixed judges and an explicit failure taxonomy. Our results show that semantic quality and operational reliability are orthogonal, vendor effects dominate use-case effects, and repeated-run dispersion strongly predicts vote instability, with Huawei VRP exposing failure modes hidden by aggregate metrics. These findings demonstrate that multivendor, repeated-execution semantic benchmarks are essential for scientifically rigorous comparison of LLM-based network configuration systems.
From: Diego Kreutz [view email]
[v1]
Sun, 5 Apr 2026 14:03:21 UTC (405 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。