Paper deep dive
Free-Text Evaluation of LLMs for 5G Domain Knowledge and Fault Analysis using LLM-as-Judge
Rishiraj Sengupta, Sotiris Chatzimiltis, Mohammad Shojafar, Xiatian Zhu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 99%
Last extracted: 8/24/2026, 5:54:16 AM
Summary
This paper evaluates the capability of lightweight, edge-deployable Large Language Models (LLMs) to perform free-text 5G domain knowledge and fault analysis. Using three benchmarks (TeleQNA ORAN FT, 5G-Faults FT, TeleInter FT), the study assesses three student models (Claude-Haiku-4.5, GPT-5.4-Mini, Gemini-3.1-Flash-Lite) via an LLM-as-Judge framework with three independent frontier judges. Results indicate that while models achieve high accuracy (>90%) on fault diagnosis, they struggle with zero-shot recall of 3GPP/O-RAN specifications (<60%). Gemini-3.1-Flash-Lite is identified as the most suitable for production due to its balance of accuracy, low cost, and latency.
Entities (12)
Relation Signals (8)
Claude-Haiku-4.5 → evaluatedon → TeleQNA ORAN FT
confidence 100% · evaluate three lightweight LLMs... on free-text 5G domain knowledge and fault-analysis tasks across three benchmarks, TeleQNA ORAN FT
GPT-5.4-mini → evaluatedon → 5G-Faults FT
confidence 100% · evaluate three lightweight LLMs... on free-text 5G domain knowledge and fault-analysis tasks across three benchmarks... 5G-Faults FT
Gemini 3.1 Flash Lite → evaluatedon → TeleInter FT
confidence 100% · evaluate three lightweight LLMs... on free-text 5G domain knowledge and fault-analysis tasks across three benchmarks... TeleInter FT
LLM-as-Judge → usedby → GPT-5.5
confidence 100% · Three independent frontier judges score outputs... GPT-5.5... LLM-as-Judge methodology
LLM-as-Judge → usedby → Gemini 3.1 Pro
confidence 100% · Three independent frontier judges score outputs... Gemini-3.1-Pro... LLM-as-Judge methodology
LLM-as-Judge → usedby → Gemini 3.5 Flash
confidence 100% · Three independent frontier judges score outputs... Gemini-3.5-Flash... LLM-as-Judge methodology
Gemini 3.1 Flash Lite → hasbesttradeoff → efficiency
confidence 95% · Gemini-3.1-Flash-Lite offers the best efficiency trade-off, combining competitive accuracy with the lowest inference cost and latency
→ →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Real-world fault analysis in 5G and emerging 6G networks demands domain expertise to analyze free-text diagnostics, including root-cause explanations and recommended actions. LLMs have emerged as a promising approach to automating this, yet whether lightweight, edge-deployable models are capable of performing in-depth free-text diagnostics remains an open question. While existing benchmarks rely on restrictive MCQs with fixed answer keys, this paper evaluates 5G domain understanding and fault analysis in a free-text generation format. Transitioning to this paradigm requires evaluating lightweight, edge-deployable AI models on open-ended diagnostic reasoning, alongside a dependable framework to validate these text outputs at scale. To address this we evaluate three lightweight LLMs, Claude-Haiku-4.5, GPT-5.4-Mini, and Gemini-3.1-Flash-Lite, on free-text 5G domain knowledge and fault-analysis tasks across three benchmarks, TeleQNA ORAN FT, 5G-Faults FT, and TeleInter FT. Three independent frontier judges score outputs, and pairwise inter-judge agreement is measured as an empirical test of the LLM-as-Judge methodology. All three models reach at least 90% accuracy on fault diagnosis, while zero-shot recall of 3GPP and O-RAN specifications remains the critical gap, with all models scoring below 60%. Mean inter-judge agreement is at least 0.90 across all runs, indicating that multi-judge LLM scoring produces consistent, reproducible grades for open-ended telecom responses. Operationally, Gemini-3.1-Flash-Lite offers the best efficiency trade-off, combining competitive accuracy with the lowest inference cost and latency, making it the most suitable candidate for production telecom deployments.
Tags
Links
- Source: https://arxiv.org/abs/2608.21021v1
- Canonical: https://arxiv.org/abs/2608.21021v1
Trouble viewing inline? Open PDF directly →
Full Text
31,203 characters extracted from source content.
Expand or collapse full text
Free-Text Evaluation of LLMs for 5G Domain Knowledge and Fault Analysis using LLM-as-Judge Rishiraj Sengupta 1,2,∗ , Sotiris Chatzimiltis 1 , Mohammad Shojafar 1 , Xiatian Zhu 2,3 1 5G/6G Innovation Centre, Institute for Communication Systems, University of Surrey, Guildford, Surrey, UK 2 Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, Guildford, Surrey, UK 3 Surrey Institute for People-Centered Artificial Intelligence Email: rs02294, sc02449, m.shojafar, xiatian.zhu@surrey.ac.uk Abstract—Real-world fault analysis in 5G and emerging 6G networks demands domain expertise to analyze free-text diagnos- tics, including root-cause explanations and recommended actions. LLMs have emerged as a promising approach to automating this, yet whether lightweight, edge-deployable models are capable of performing in-depth free-text diagnostics remains an open ques- tion. While existing benchmarks rely on restrictive MCQs with fixed answer keys, this paper evaluates 5G domain understanding and fault analysis in a free-text generation format. Transitioning to this paradigm requires evaluating lightweight, edge-deployable AI models on open-ended diagnostic reasoning, alongside a dependable framework to validate these text outputs at scale. To address this we evaluate three lightweight LLMs, Claude- Haiku-4.5, GPT-5.4-Mini, and Gemini-3.1-Flash-Lite, on free- text 5G domain knowledge and fault-analysis tasks across three benchmarks, TeleQNA ORANFT, 5G-FaultsFT, and TeleIn- terFT. Three independent frontier judges score outputs, and pairwise inter-judge agreement is measured as an empirical test of the LLM-as-Judge methodology. All three models reach at least 90% accuracy on fault diagnosis, while zero-shot recall of 3GPP and O-RAN specifications remains the critical gap, with all models scoring below 60%. Mean inter-judge agreement is at least 0.90 across all runs, indicating that multi-judge LLM scoring produces consistent, reproducible grades for open-ended telecom responses. Operationally, Gemini-3.1-Flash-Lite offers the best efficiency trade-off, combining competitive accuracy with the lowest inference cost and latency, making it the most suitable candidate for production telecom deployments. Index Terms—free-text evaluation, LLM-as-Judge, inter-judge agreement, LLMs, 5G fault analysis, Open RAN I. INTRODUCTION Fault management in 5G and emerging 6G networks is be- coming increasingly challenging as network complexity grows. Failures spanning handover procedures, session establishment and core network signalling are difficult to isolate manually, and traditional rule-based approaches do not scale to the diver- sity of faults in modern deployments. Large language models (LLMs) have recently emerged as a promising approach to au- tomate root cause analysis (RCA), given their ability to reason over natural-language fault descriptions and interpret protocol specifications. ETSI ISG ENI identifies LLMs and generative AI as key technologies for advancing network operations to- ward higher autonomy, including cross-layer fault analysis and operations, administration and maintenance (OAM) decision- making [1]. *now at Google A key theme across these initiatives is the need for small, efficient models. For operators, LLM adoption must be eval- uated not only on technical performance but on energy con- sumption, cost, and latency, which pose significant barriers to deployment at scale [2]. Operators cannot realistically run frontier-scale models across distributed network elements due to inference cost, latency, and data-privacy constraints. ETSI describes a phased approach in which lightweight models first handle Q&A and knowledge tasks before progressing to fault diagnosis and fuller autonomous operation [1]. This raises the first research question: Can lightweight, cost-efficient LLMs handle 5G specification knowledge and fault analysis tasks? Answering it has direct implications for standardization, where specification knowledge is the foundation on which any progression toward higher network autonomy depends. Evaluating these models, however, presents a challenge of its own. Unlike multiple-choice benchmarks such as Tele- QNA [3], real fault analysis requires free-form responses that include explanations, diagnostic steps, and recommended actions, which cannot be scored against a fixed answer key. Yet automated evaluation at scale demands a practical alternative to human annotation. A growing body of work addresses this through the LLM-as-Judge paradigm, where a frontier model scores free-text outputs on behalf of human evalua- tors [4]. While promising, this approach remains unvalidated for telecom, and the broader literature raises concerns about systematic judge biases, including positional, length, and self- enhancement bias, as well as self-consistency [5]. When multiple frontier judges are deployed, a further question arises as to whether they converge on the same verdicts. This raises the second research question: Do frontier AI judges agree with each other when evaluating telecom free-text outputs? This paper addresses both questions through a con- trolled evaluation of three lightweight, edge-deployable LLMs, Claude-Haiku-4.5, GPT-5.4-Mini, and Gemini-3.1-Flash-Lite on free-text 5G specification and fault-analysis tasks drawn from the TeleQNA [3] and ORAN-Bench-13K [6] datasets. Rather than relying on a single evaluator, we score each response with three independent frontier judges, GPT-5.5, Gemini-3.1-Pro, and Gemini-3.5-Flash, and measure their agreement as an empirical test of the LLM-as-Judge method itself. Alongside correctness, we report inference cost, latency, and token usage as practical indicators of edge deployment arXiv:2608.21021v1 [cs.CL] 21 Aug 2026 viability. The contributions of this paper are as follows: • A free-text evaluation of lightweight LLMs on tele- com tasks, using a dataset rewritten into open-ended form across three benchmarks (TeleQNAORANFT, 5G-FaultsFT, and TeleInterFT) that requires models to generate explanations, diagnostic steps, and recom- mended actions rather than pick from fixed options. • An operational study of cost-efficient, edge-deployable models on these tasks, reporting inference cost, latency, and token usage alongside correctness as signs of deploy- ment viability. • A multi-judge evaluation framework, one example of automated free-text scoring, using three independent frontier judges and inter-judge agreement analysis to test how reliable LLM-as-Judge is for telecom. I. RELATED WORK The first systematic evaluation of LLM telecom knowledge, TeleQNA [3], introduced 10,000 multiple-choice questions (MCQs) from 3GPP standards, publications, and lexicons, showing that LLMs can rival active professionals on knowl- edge tasks. ORAN-Bench-13K [6] extended this to Open RAN with 13,952 MCQs from 116 O-RAN specifications, finding current models not yet proficient. Tele-LLMs [7] then showed that domain-adapted smaller models can outperform general- purpose frontier models, while flagging MCQ selection bias as a methodological limitation. Most recently, 6G-Bench [8] contributed 10,000 MCQs across 30 decision-making tasks from 6G standardization. All of these efforts share a reliance on multiple-choice format. No free-text evaluation framework exists for the domain. Beyond benchmarking, a growing body of work targets operational network tasks. Singh [9] established the motiva- tion for AI-assisted troubleshooting, citing cost and energy efficiency as primary drivers. TeleLogs [10] introduced a curated dataset of annotated 5G RCA problems from realistic drive-test scenarios, showing that general LLMs struggle with fault analysis without domain adaptation. The AI-on-RAN framework [11] established the feasibility of LLM inference at the RAN level for anomaly detection, providing operational context for edge deployment. None of these, however, evaluate free-text diagnostic outputs or quantify the operational metrics that determine edge viability. Our evaluation methodology builds on the LLM-as-Judge paradigm of Zheng et al. [4], who showed frontier LLMs can serve as reliable proxies for human judgment on open- ended responses. Gu et al. [12] survey the paradigm, noting that LLMs offer finer-grained assessment than rule-based metrics where semantic alignment matters, as in free-text telecom answers, where many valid phrasings express the same concept. Jung et al. [13] introduce a trust-or-escalate framework with provable human-agreement guarantees, moti- vating our use of inter-judge agreement rate as a reliability signal, and the DeepMind taxonomy of rankers, judges, and assistants [14] grounds our choice of binary correctness judges TEST DATASETS TeleQNA_ORAN 3GPP / ORAN Standards 100 samples 5G-Faults 5G Fault Diagnosis 100 samples TeleInter 5G Interview / Network 100 samples System Prompt + Questions STUDENT MODELS Claude Haiku 4.5 Anthropic via OpenRouter GPT-5.4-Mini OpenAI via OpenRouter Gemini 3.1 Flash Lite Google Deep Mind via OpenRouter LLM AS JUDGE GPT-5.5 OpenAI LLM Judge Gemini 3.1 Pro Preview Google Deep Mind LLM Judge Gemini 3.5 Flash Google Deep Mind SLM Judge Ref. Answer. + Judge Criteria FINAL EVALUATION Correlation & Comparison AccuracyJudge AgreementToken Usage Latency Cost Prompt Output Scores Fig. 1: Proposed Evaluation Framework. Test datasets are passed to student models via the solver pipeline, responses are scored by three independent LLM judges, and final metrics covering accuracy, judge agreement, token usage, latency and cost are computed. over preference rankers. Crucially, none applies LLM-as-Judge to telecom, and no prior study has measured inter-judge agreement on telecom free-text outputs. I. METHODOLOGY This section describes the experimental setup used to eval- uate lightweight LLMs on telecom free-text tasks. We present the proposed evaluation framework and its components, the student and judge models, the prompting strategy, the bench- mark datasets, and the metrics collected. Fig. 1 illustrates the end-to-end pipeline. A. Proposed Evaluation Framework The evaluation pipeline is built on Inspect AI [15], an open-source LLM evaluation framework developed by the UK AI Security Institute. The pipeline extends the LLM evaluation curriculum proposed by Arena Education [16], adapting the solver and scorer components to support expert telecom prompting and multi-judge binary grading. Inspect AI provides deterministic task definitions, reproducible sampling, structured logging in a compressed .eval format, and a multi-scorer architecture that enables parallel judge invoca- tions within a single evaluation run. The pipeline is organized into two stages: a solver stage and a scorer stage. The solver stage consists of three se- quential components. First, prompt_template() injects the expert telecom system prompt and structures the question. Second, chain_of_thought() enforces the two-part Stu- dent Explanation and Student Answer output template. Third, generate() issues the API call to the student model. The scorer stage invokes all three judge models in parallel, each independently awarding a binary grade of Correct (C) or Incorrect (I) across the 100-sample run. All student and judge calls are routed through OpenRouter, a unified gateway over Anthropic, OpenAI, and Google, TABLE I: Student Models ModelOpenRouter IDProvider Claude-Haiku-4.5anthropic/claude-haiku-4.5Anthropic GPT-5.4-Miniopenai/gpt-5.4-miniOpenAI Gemini-3.1-Flash-Litegoogle/gemini-3.1-flash-liteGoogle ensuring consistent rate limiting, token accounting, and cost attribution across providers. Nine evaluation experiments were conducted in total, one per (student model, dataset) com- bination. For each run, 100 samples are loaded from the corresponding Hugging Face dataset, responses are generated sequentially through the OpenRouter API with default sam- pling parameters, and the complete run, including per-sample inputs, outputs, scores and token usage, is persisted as a compressed .eval log file. Accuracy, inter-judge agreement, token usage, latency and cost statistics are then aggregated from the logs. B. Student Models Three lightweight, commercially available LLMs serve as student models in this evaluation. Table I lists each model along with its provider and OpenRouter identifier. The mod- els were selected to represent cost-optimized, low-latency offerings from three major AI providers, namely Anthropic, OpenAI, and Google, making them practical candidates for edge deployment in telecommunications environments. Each model generates a free-text response directly from the prompt without additional inference-time computation. C. Prompting Strategy All student models receive a structured prompt consisting of an expert telecom system prompt followed by the question. To facilitate downstream judge evaluation, each prompt enforces a two-part output template that separates the model’s reasoning trace from its final answer. The required response format is as follows: You are an expert Telecommunication Engineer. Provide short, accurate, professional and standard-compliant answers. Your response should be in the following format: Student Explanation: explanation in 2 lines Student Answer: final answer in 1 line This structure enables the judge models to assess factual correctness independently of the reasoning trace, ensuring that scoring is based on the final answer while retaining the explanation for qualitative inspection. Student models operate in a zero-shot setting, without few-shot examples, RAG, or tool use. D. LLM-as-Judge Evaluation Since task responses are open-ended and generative, string- matching and overlap-based metrics such as ROUGE are unsuitable, as there is no fixed ground truth against which to measure similarity. We adopt the LLM-as-Judge paradigm, in which three independent frontier judge models assess each stu- dent response against the reference answer and award a binary TABLE I: Judge Models ModelOpenRouter IDProvider GPT-5.5openai/gpt-5.5OpenAI Gemini-3.1-Pro-Previewgoogle/gemini-3.1-pro-previewGoogle Gemini-3.5-Flashgoogle/gemini-3.5-flashGoogle grade. Table I lists the judge models used in this evaluation. Using three independent judges mitigates single-judge bias and enables cross-provider agreement analysis. Each judge receives the original question, the student explanation, the student answer, and the reference correct answer. Each judge is instructed to act as a Telecommunications Domain Expert with 3GPP, 5G, LTE, and O-RAN expertise, evaluating factual correctness primarily from domain knowledge and us- ing the reference answer only as a semantic guide. Each judge outputs a binary grade, either Correct (C) or Incorrect (I), alongside a one-sentence justification. The structured output format is enforced via regex parsing, where grade C maps to a score of 1.0 and grade I to 0.0. Per-judge accuracy is then computed as the mean score across all 100 samples. E. Datasets Three telecom-specific benchmarks are used for the per- formance evaluation, each comprising 100 questions sampled from curated datasets hosted on the Hugging Face Hub. All multiple-choice options are removed from the original datasets to enforce free-text generation, and the reference answer is passed to the judges as a semantic guide rather than an exact- match target. TeleQNA ORANFT [3], [6] draws from 3GPP 5G and O- RAN specifications, probing factual knowledge of standards, protocols, interfaces and architectural components. Each ques- tion requires a concise, technically precise natural language answer, making it the most demanding benchmark in terms of specification recall. 5G-FaultsFT [17] presents realistic 5G network fault sce- narios derived from operational logs. Models are required to identify fault causes, affected components and recommended remediation actions, evaluating applied diagnostic reasoning in a network operations context. TeleInter FT tests the ability to interpret and explain 5G network concepts, configurations and technical details at the depth expected from experienced network engineers, covering architecture, physical layer, layer-2, layer-3 and advanced features. This benchmark is distinctive in that it requires not only factual recall but also assessment of applied technical concepts, as expected from a domain expert in an original equipment manufacturer (OEM) hiring context. All datasets are publicly available on the Hugging Face Hub under the rishieee namespace. Table I summarizes the three benchmarks. F. Metrics The following metrics are collected and reported for each (student model, dataset) pair across all nine evaluation runs. TABLE I: Evaluation Datasets NameDomainTaskn TeleQNAORANFT3GPP / O-RAN specsKnowledge Q&A100 5G-FaultsFT5G fault diagnosisFault analysis100 TeleInterFT5G / OEM conceptsTechnical Q&A100 Accuracy. Let s i,j ∈ 0, 1 denote the score assigned to sample i by judge j, where: s i,j = ( 1 if grade is Correct 0 otherwise (1) Per-judge accuracy and mean accuracy across all judges are defined as: Acc j = 1 N N X i=1 s i,j (2) Acc = 1 J J X j=1 Acc j (3) where N = 100 is the number of samples per run and J = 3 is the number of judge models. Inter-Judge Agreement. Inspired by the pairwise evalua- tion methodology of Zheng et al. [4], let c i,a,b ∈ 0, 1 denote whether judges a and b agree on sample i, where: c i,a,b = ( 1 if s i,a = s i,b 0 otherwise (4) Pairwise agreement between judges a and b is then: P (a = b) = 1 N N X i=1 c i,a,b (5) Three pairwise values are computed per run, covering GPT-5.5 vs. Gemini-3.1-Pro-Preview, GPT-5.5 vs. Gemini-3.5-Flash, and Gemini-3.1-Pro-Preview vs. Gemini-3.5-Flash. Token Usage. Total reasoning and completion tokens are computed as: T k = N X i=1 t i,k , k ∈ reasoning, completion(6) where t i,k is the token count of type k for sample i. Generation Latency. Average response latency per sample is defined as: Lat = 1 N N X i=1 ∆t i (7) where ∆t i is the API call duration for sample i in seconds. API Cost. Total inference cost per (student model, dataset) pair is reported in USD and extracted from OpenRouter activity logs. Cost attribution is matched by model identifier and API key corresponding to each benchmark task. IV. RESULTS AND DISCUSSION This section presents and discusses the evaluation results across the three benchmarks for the three student models, covering accuracy, inter-judge agreement, API cost, latency and token usage. TABLE IV: Mean Accuracy per Student Model (%) ModelTeleQNAORANFT5G-FaultsFTTeleInterFTOverall Claude-Haiku-4.547.090.076.071.0 GPT-5.4-Mini56.398.391.782.1 Gemini-3.1-Flash-Lite49.095.390.078.1 A. Accuracy Fig. 2 reports per-judge accuracy for each (student model, dataset) combination, while Table IV summarizes the overall mean accuracy per model. GPT-5.4-Mini achieves the highest overall mean accuracy at 82.1%, followed by Gemini-3.1- Flash-Lite at 78.1% and Claude-Haiku-4.5 at 71.0%. TeleQNA ORANFT is the most challenging benchmark, with all three models scoring below 60%, indicating that precise zero-shot recall of 3GPP and O-RAN standards re- mains difficult and suggesting that retrieval augmentation or domain fine-tuning is needed before deployment. Judge agreement stays high despite the low scores, confirming these figures reflect genuine model limitations rather than evalu- ator disagreement. 5G-Faults FT yields the highest scores overall, with GPT-5.4-Mini reaching near-perfect accuracy at 98.3%, suggesting 5G fault scenarios are more accessible to lightweight models than open-ended specification knowledge. TeleInter FT is intermediate, with GPT-5.4-Mini and Gemini- 3.1-Flash-Lite maintaining strong performance while Claude- Haiku-4.5 drops to 76.0%, a 15 percentage-point gap behind the leader. B. Inter-Judge Agreement Table V reports the pairwise inter-judge agreement. The agreement rate is the probability that two distinct judges assign the same binary grade to a randomly selected ques- tion. Agreement is consistently high throughout, with the two Gemini judges reaching 0.95–0.99 in every case, re- flecting shared provider training and evaluation philosophy. TeleQNA ORANFT shows the most balanced cross-provider agreement (0.930–0.970), validating that the low accuracy figures on this benchmark reflect genuine model limitation rather than evaluator disagreement. The most informative divergence occurs on 5G-FaultsFT with Claude-Haiku-4.5, where both GPT-5.5 pairwise values drop to 0.860 while P(Gemini-3.1-Pro = Gemini-3.5-Flash) = 0.980. This pattern reveals that GPT-5.5 systematically disagrees with both Gemini judges on a subset of questions, grading Claude-Haiku-4.5 responses as Incorrect while both Gemini models accept them as Correct. This explains why GPT-5.5 scores Claude-Haiku-4.5 at 82% while both Gemini judges score it at approximately 94%, as visible in Fig. 2, and points to a provider-level grading style difference on fault diagnosis responses. This finding illustrates the importance of multi-judge evaluation: a single judge would have obscured this systematic divergence entirely. C. API Cost Fig. 3 reports the total API inference cost per student model across all three benchmarks. Gemini-3.1-Flash-Lite is consis- tently the most cost-efficient model, remaining below $0.10 Claude Haiku 4.5 GPT-5.4-Mini Gemini 3.1 Flash Lite (a) TeleQNA_ORAN_FT 0 20 40 60 80 100 120 Accuracy (%) 47% 49% 45% 56%56% 57% 49% 47% 51% Claude Haiku 4.5 GPT-5.4-Mini Gemini 3.1 Flash Lite (b) 5G-Faults_FT 0 20 40 60 80 100 120 Accuracy (%) 82% 94%94% 98%98% 99% 92% 97%97% Claude Haiku 4.5 GPT-5.4-Mini Gemini 3.1 Flash Lite (c) TeleInter_FT 0 20 40 60 80 100 120 Accuracy (%) 73% 76% 79% 90% 93% 92% 85% 94% 91% GPT-5.5Gemini-3.1-ProGemini-3.5-Flash Fig. 2: Accuracy by student model and judge across the three benchmarks. Each group of three bars represents one student model, coloured by judge model. TABLE V: Pairwise Inter-Judge Agreement Rates Model GPT/Pro GPT/Flash Pro/Flash Mean A.R. TeleQNAORANFT Haiku0.9600.9400.9600.953 Mini0.9400.9300.9700.947 Flash0.9600.9400.9600.953 5G-FaultsFT Haiku0.8600.8600.9800.900 Mini0.9800.9700.9900.980 Flash0.9300.9500.9800.953 TeleInterFT Haiku0.9300.9200.9500.933 Mini0.9500.9600.9900.967 Flash0.8900.9200.9500.920 Note: GPT = GPT-5.5, G-Pro = Gemini-3.1-Pro-Preview, G-Flash = Gemini- 3.5-Flash, Haiku = Claude-Haiku-4.5, Mini = GPT-5.4-Mini, Flash = Gemini- 3.1-Flash-Lite, Mean A.R. = Mean Agreement Rate. per 100-sample run across all datasets. Claude-Haiku-4.5 is the most expensive, exceeding $0.42 in every case, while GPT- 5.4-Mini sits between the two. Considering that GPT-5.4-Mini achieves the highest overall accuracy at 82.1% while Gemini- 3.1-Flash-Lite achieves 78.1% at approximately 4.5x cheaper than Claude-Haiku-4.5 and 2.5x cheaper than GPT-5.4-Mini, Gemini-3.1-Flash-Lite offers the most compelling accuracy- to-cost trade-off for edge deployment in telecommunications environments. D. Operational Metrics Table VI reports token usage across all nine runs. Claude- Haiku-4.5 consistently generates the highest total token vol- umes, with reasoning tokens accounting for the majority of its output across all benchmarks, reflecting an extended internal reasoning process regardless of task difficulty. GPT-5.4-Mini shows a different profile, with reasoning tokens dominating only on TeleQNA ORANFT, suggesting deeper reasoning is engaged primarily for specification-heavy queries. Gemini- TeleQNA_ORAN_FT5G-Faults_FTTeleInter_FT 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Total Cost (USD) $0.43 $0.44 $0.42 $0.34 $0.16 $0.18 $0.09 $0.09 $0.09 Claude Haiku 4.5GPT-5.4-MiniGemini 3.1 Flash Lite Fig. 3: Total API inference cost per student model across the three benchmarks. 3.1-Flash-Lite produces the lowest completion token volumes across all datasets, indicating a concise generation behavior that contributes directly to its latency and cost advantage. Fig. 4 reports mean generation latency per sample. Gemini- 3.1-Flash-Lite achieves the lowest latency in every case (3.4s– 4.3s), while Claude-Haiku-4.5 peaks at 12.3s on TeleInter FT, consistent with its heavy reasoning token usage. GPT-5.4-Mini records its highest latency on TeleQNAORANFT at 9.1s, but improves considerably on the remaining benchmarks. Taken together with Fig. 3, Gemini-3.1-Flash-Lite demonstrates the most favorable operational profile, combining competitive accuracy with the lowest inference cost, latency, and token output across all benchmarks. V. CONCLUSIONS This paper presented a free-text evaluation of lightweight LLMs on telecom tasks, a more challenging and practically relevant setting than multiple-choice benchmarks. Across three telecom-specific benchmarks, all three models reached at least 90% accuracy on 5G fault diagnosis, with two of three also reaching 90% on network interpretation, showing that lightweight models are already capable on structured free- TABLE VI: Token Usage per Student Model DatasetStudent Model Completion ReasoningTotal TeleQNAORANFT Haiku25,70457,03382,737 Mini8,23665,00273,238 Flash8,21551,45259,667 5G-FaultsFT Haiku23,91960,62584,544 Mini13,62320,11633,739 Flash11,21248,52359,735 TeleInterFT Haiku27,87654,47082,346 Mini11,99126,78838,779 Flash9,84748,85558,702 Note: Haiku = Claude-Haiku-4.5, Mini = GPT-5.4-Mini, Flash = Gemini-3.1- Flash-Lite. TeleQNA_ORAN_FT5G-Faults_FTTeleInter_FT 0.0 2.5 5.0 7.5 10.0 12.5 15.0 Mean Latency (s/sample) 7.4s 7.9s 12.3s 9.1s 4.9s 4.8s 4.3s 3.4s3.4s Claude Haiku 4.5GPT-5.4-MiniGemini 3.1 Flash Lite Fig. 4: Mean generation latency per sample (seconds) across the three benchmarks. text tasks. Zero-shot recall of 3GPP and O-RAN specifications remains the critical gap, with all models below 60%, pointing to the need for retrieval-augmented or domain-adapted ap- proaches. The three-judge panel maintained inter-judge agree- ment of 0.90–0.98 across all nine runs, indicating stable, reproducible grading of open-ended telecom responses. We note this establishes consistency among judges rather than ground-truth correctness, motivating human-expert validation. Among the models evaluated, Gemini-3.1-Flash-Lite offers the best efficiency trade-off, combining competitive accuracy with the lowest cost and latency, making it the strongest candidate for production telecom deployment. Future work will focus on three directions. First, incor- porating human expert judgment alongside the automated judge panel to assess how well LLMs align with domain specialist evaluation in a telecom context, providing external validation of the LLM-as-Judge methodology. Second, ad- dressing the specification knowledge gap identified on Tele- QNA ORANFT through retrieval-augmented generation and domain fine-tuning, and assessing whether the resulting mod- els meet the standards-recall threshold needed for autonomous OAM operation. Third, expanding from this baseline capability assessment to a more advanced study of Agentic AI frame- works, where multiple specialized models leverage tools and search capabilities to collaborate on complex network tasks. ACKNOWLEDGMENT This work is supported by the CHEDDAR (Communica- tions Hub for Empowering Distributed Cloud Computing Ap- plications and Research) program, funded by the UK EPSRC under grant numbers EP/Y037421/1 and EP/X040518/1, and in part by the Open Challenge Fund Call through Datacom Industry Association AISBL (the specific call being sponsored by Huawei Technologies). REFERENCES [1] Y. Zeng et al., “AI Technologies in Experiential Networked Intelligence to Increase Autonomous Operation,” European Telecommunications Standards Institute (ETSI), White Paper 64, Nov. 2024. [2] S. Adnan et al., “Large-Scale AI in Telecom: Charting the Roadmap for Innovation, Scalability, and Enhanced Digital Experiences,” 2025. [Online]. Available: https://arxiv.org/abs/2503.04184 [3] A. Maatouk, F. Ayed, N. Piovesan, A. D. Domenico, M. Debbah, and Z.-Q. Luo, “TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledge,” 2023. [Online]. Available: https://arxiv.org/abs/2310.15051 [4] L. Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” in Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, p. 46 595–46 623. [5] S. Tan, S. Zhuang, K. Montgomery, W. Y. Tang, A. Cuadron, C. Wang, R. A. Popa, and I. Stoica, “JudgeBench: A Benchmark for Evaluating LLM-based Judges,” 2025. [Online]. Available: https: //arxiv.org/abs/2410.12784 [6] P. Gajjar and V. K. Shah, “ORAN-Bench-13K: An Open Source Benchmark for Assessing LLMs in Open Radio Access Networks,” in 2025 IEEE 22nd Consumer Communications & Networking Conference (CCNC), 2025, p. 1–4. [7] A. Maatouk, K. C. Ampudia, R. Ying, and L. Tassiulas, “Tele-LLMs: A Series of Specialized Large Language Models for Telecommunications,” IEEE Access, vol. 14, p. 86 424–86 441, 2026. [8] M. A. Ferrag, A. Lakas, and M. Debbah, “6G-Bench: An Open Bench- mark for Semantic Communication and Network-Level Reasoning With Foundation Models in AI-Native 6G Networks,” IEEE Open Journal of the Communications Society, vol. 7, p. 3305–3330, 2026. [9] P. Singh, “Leveraging AI for advanced troubleshooting in telecommuni- cations: Enhancing network reliability, customer satisfaction, and social equity,” Journal of Science & Technology, vol. 2, no. 2, p. 99–138, 2021. [10] M. Sana, N. Piovesan, A. D. Domenico, Y. Kang, H. Zhang, M. Debbah, and F. Ayed, “Reasoning Language Models for Root Cause Analysis in 5G Wireless Networks,” 2025. [Online]. Available: https://arxiv.org/abs/2507.21974 [11] S. Chatzimiltis, M. Shojafar, M. B. Mashhadi, and R. Tafazolli, “AI-on- RAN for Cyber Defense: An XAI-LLM Framework for Interpretable Anomaly Detection,” IEEE Transactions on Network Science and Engi- neering, vol. 13, p. 3301–3319, 2026. [12] J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y. Shen, S. Ma, H. Liu et al., “A survey on LLM-as-a-judge,” The Innovation, vol. 7, no. 6, 2026. [13] J. Jung, F. Brahman, and Y. Choi, “Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement,” in International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, Eds., vol. 2025, 2025, p. 3101–3125. [14] K. Balog, D. Metzler, and Z. Qin, “Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluation,” in Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, ser. SIGIR ’25.New York, NY, USA: Association for Computing Machinery, 2025, p. 3865–3875. [Online]. Available: https://doi.org/10.1145/3726302.3730348 [15] UK AI Security Institute, “Inspect AI: Framework for Large Language Model Evaluations.” [Online]. Available: https://github.com/ UKGovernmentBEIS/inspect ai [16] Arena Education, “LLM Evaluation Curriculum,” https://w.arena. education/curriculum, 2024. [17] L.Lai,“5G-Faults-Full-v2,”https://huggingface.co/datasets/ greenwich157/5G-Faults-Full-v2, 2024, accessed: 2026.