Paper deep dive
Measuring LLM Trust Allocation Across Conflicting Software Artifacts
Noshin Ulfat, Ahsanul Ameen Sabit, Soneya Binta Hossain
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%
Last extracted: 4/10/2026, 2:10:00 AM
Summary
The paper introduces TRACE, a framework for evaluating how LLMs allocate trust across conflicting software artifacts (Javadoc, method signatures, implementations, and test prefixes). By testing seven LLMs on 456 Java method bundles with controlled perturbations, the authors find that while models are sensitive to input quality, they exhibit an 'implementation-only' blind spot and asymmetric sensitivity, where documentation bugs are detected more readily than code-level drift. The study highlights the need for explicit artifact-level trust reasoning in software engineering workflows.
Entities (4)
Relation Signals (2)
TRACE → evaluates → LLM
confidence 95% · We present TRACE... a framework that elicits structured artifact-level trust traces over Javadoc, method signatures, implementations, and test prefixes
LLM → exhibits → implementation-only blind spot
confidence 90% · yet show a systematic blind spot when only the implementation drifts while the documentation remains plausible
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:LLM-based software engineering assistants fail not only by producing incorrect outputs, but also by allocating trust to the wrong artifact when code, documentation, and tests disagree. Existing evaluations focus mainly on downstream outcomes and therefore cannot reveal whether a model recognized degraded evidence, identified the unreliable source, or calibrated its trust across artifacts. We present TRACE (Trust Reasoning over Artifacts for Calibrated Evaluation), a framework that elicits structured artifact-level trust traces over Javadoc, method signatures, implementations, and test prefixes under blind perturbations. Using 22,339 valid traces from seven models on 456 curated Java method bundles, we evaluate per-artifact quality assessment, inconsistency detection, affected artifact attribution, and source prioritization. Across all models, quality penalties are largely localized to the perturbed artifact and increase with severity, but sensitivity is asymmetric across artifact types: documentation bugs induce a substantially larger heavy-to-subtle gap than implementation faults (0.152-0.253 vs. 0.049-0.123). Models detect explicit documentation bugs well (67-94%) and Javadoc and implementation contradictions at 50-91%, yet show a systematic blind spot when only the implementation drifts while the documentation remains plausible, with detection dropping by 7-42 percentage points. Confidence is poorly calibrated for six of seven models. These findings suggest that current LLMs are better at auditing natural-language specifications than at detecting subtle code-level drift, motivating explicit artifact-level trust reasoning before correctness-critical downstream use.
Tags
Links
- Source: https://arxiv.org/abs/2604.03447v1
- Canonical: https://arxiv.org/abs/2604.03447v1
Trouble viewing inline? Open PDF directly →
Full Text
62,040 characters extracted from source content.
Expand or collapse full text
Measuring LLM Trust Allocation Across Conflicting Software Artifacts Noshin Ulfat noshin.ulfat@utdallas.edu University of Texas at Dallas Texas, USA Ahsanul Ameen Sabit ahsanulameen.sabit@iqvia.com IQVIA Inc. Dhaka, Bangladesh Soneya Binta Hossain sbhossain@utdallas.edu University of Texas at Dallas Texas, USA Abstract LLM-based software engineering assistants fail not only by pro- ducing incorrect outputs, but also by allocating trust to the wrong artifact when code, documentation, and tests disagree. Existing evaluations focus mainly on downstream outcomes and therefore cannot reveal whether a model recognized degraded evidence, iden- tified the unreliable source, or calibrated its trust across artifacts. We present TRACE (Trust Reasoning over Artifacts for Calibrated Evaluation), a framework that elicits structured artifact-level trust traces over Javadoc, method signatures, implementations, and test prefixes under blind perturbations. Using 22,339 valid traces from seven models on 456 curated Java method bundles, we evaluate per-artifact quality assessment, inconsistency detection, affected- artifact attribution, and source prioritization. Across all models, quality penalties are largely localized to the perturbed artifact and increase with severity, but sensitivity is asymmetric across artifact types: documentation bugs induce a substantially larger heavy-to- subtle gap than implementation faults (0.152–0.253 vs. 0.049–0.123). Models detect explicit documentation bugs well (67–94%) and Javadoc and implementation contradictions at 50–91%, yet show a system- atic blind spot when only the implementation drifts while the doc- umentation remains plausible, with detection dropping by 7–42 percentage points. Confidence is poorly calibrated for six of seven models. These findings suggest that current LLMs are better at auditing natural-language specifications than at detecting subtle code-level drift, motivating explicit artifact-level trust reasoning before correctness-critical downstream use. 1 Introduction Large language models (LLMs) are increasingly used in software engineering (SE) tasks such as test generation, patching, summa- rization, and code review [7,11]. In these settings, however, models rarely operate over a single clean specification. Instead, they must reason over bundles of heterogeneous artifacts—including source code, method signatures, documentation, and test scaffolding—that may be incomplete, stale, or mutually inconsistent. In practice, many failures stem not from generating implausible text, but from misallocating trust across artifacts. When evidence conflicts, an LLM may silently rely on the wrong source while still producing an output that appears reasonable, making such failures difficult to detect from outputs alone. Testing provides a particularly sharp setting for studying this prob- lem. To generate or assess a unit-test assertion, a model must recon- cile multiple sources of behavioral evidence: the method signature constrains types and exceptions, the Javadoc captures documented intent, the method under test (MUT) encodes the current implemen- tation, and the test prefix (setup and invocation context) defines the intended execution scenario. Prior work has shown that documen- tation quality materially affects downstream oracle generation [10], and that learned or LLM-based oracle generators can suffer false positives and generalization gaps when they over-trust noisy con- text [8,9]. We study testing not to optimize oracle generation, but because it provides a concrete, high-stakes, and controlled setting in which input quality and cross-artifact contradictions can be defined and perturbed precisely. These observations suggest that current LLM-based SE workflows lack an explicit mechanism for artifact-level trust reasoning over input quality and cross-artifact consistency. Despite these advances, most prior work evaluates models primar- ily through downstream task outcomes—for example, whether a generated assertion is correct—rather than through the reasoning process that led to that output. This leaves a critical gap. A model may produce a plausible output while relying on the wrong ar- tifact, and when documentation and implementation disagree, it may implicitly privilege one source without surfacing the conflict. As a result, downstream accuracy alone cannot reveal whether the model recognized that its inputs were unreliable, identified the faulty artifact, or calibrated its trust appropriately. Existing ap- proaches such as [6,8,10] substantially improve our understanding of oracle generation and the role of documentation, but they do not directly evaluate whether LLMs can explicitly assess, compare, and prioritize contradictory software artifacts before making down- stream decisions. To study these hidden failures, evaluation must make the model’s trust allocation over artifacts explicit rather than infer it only from the final output. Throughout this paper, we use artifact-level trust in an operational sense: it denotes the model’s explicit allocation of reliability across the artifacts in an input bundle. We operationalize this allocation through three observable outputs: (1) per-artifact quality assess- ments, (2) inconsistency judgments with affected-artifact attribu- tion, and (3) source prioritization over competing artifacts. This lets us evaluate trust reasoning directly, rather than infer it indirectly from downstream task success. We address this gap with TRACE (Trust Reasoning over Artifacts for Calibrated Evaluation), a framework that treats artifact-level trust reasoning as a first-class output (Figure 1). TRACE addresses this gap in three ways. First, it elicits structured reasoning traces that make trust judgments explicit: each trace records per-artifact arXiv:2604.03447v1 [cs.SE] 3 Apr 2026 , ,Noshin Ulfat, Ahsanul Ameen Sabit, and Soneya Binta Hossain 456 Unique Samples Rule-based Fileting i. - description i. - @return i. - (description + @return) i. MUT + bug i. javadoc + bug i. MUT, javadoc conflict Deterministic LLM-based Shared system prompt i. artifact symmetry i. scoring i. conflict analysis + ranking User prompt 4 code artifacts 22,339 Traces i. artifact quality i. priority list i. confidence score iv . IC, IR & PCA vi. assumptions & limitations RQ1 i. Score sensitivity to artifact perturbation i. Judgement change with perturbation type and severity RQ2 i. Detect MUT-Javadoc inconsistency i. Deprioritize faulty artifact RQ3 i. LLM family difference on same prompt i. Role of instruction following & cross artifact reasoning Measure SE artifact assessment capabilities of LLMs 1. Benchmark construction 2. Blind trace elicitation 3. Evaluation 7 model ensambles Anthropic OpenAI DeepSeek xAI 456*7 = 3192 Samples 220K Samples 6 Controlled Perturbation Figure 1: Overview of TRACE Pipeline. and overall quality assessments, pairwise conflict analysis, explicit inconsistency and anomaly judgments, and a reliability ranking over sources. Second, TRACE enforces artifact symmetry: it does not assume that the MUT is correct because it is executable, or that the Javadoc is correct because it is documentation. Instead, each artifact must earn reliability through cross-validation with the others. Third, TRACE evaluates these traces on a provenance- annotated, perturbation-enriched benchmark derived from 25 real- world Java systems, in which documentation faults, implementation faults, and Javadoc–MUT contradictions are systematically varied by type and severity. This lets us measure not only whether a model reacts to degraded inputs, but also whether it correctly attributes the fault and whether that attribution aligns with perturbation provenance. Guided by this goal, we study the following research questions: •RQ1: Input-quality sensitivity. To what extent do LLM-generated reasoning traces reflect degraded input quality when SE artifacts are perturbed? In particular, do quality judgments change pro- portionally with perturbation type and severity? Core finding: Yes. Across all seven models, score penalties are largely localized to the perturbed artifact and increase with severity. Documen- tation perturbations are judged much more sharply than code perturbations: the heavy-to-subtle gap is 0.152–0.253 for Javadoc bugs versus 0.049–0.123 for MUT bugs. •RQ2: Javadoc–MUT inconsistency detection. Can LLMs ex- plicitly detect semantic inconsistencies between Javadoc and MUT implementations, and how robust is this detection when faults originate in documentation versus code? Core finding: Par- tially. Models detect explicit documentation errors well (68–95%) and Javadoc–MUT contradictions at 50–91%, but they show a strong implementation-only blind spot: when only the MUT drifts while the documentation remains plausible, detection drops by roughly 21–43 percentage points. Confidence is poorly calibrated for six of seven models. • RQ3: Semantic code understanding as a driver of trace fi- delity. To what extent can inter-model variation in trace fidelity be explained by differences in semantic code understanding? Core finding: Largely. Models with stable contradiction detec- tion under subtle perturbations—especially Sonnet, DS-V3.2, and Haiku—also produce more accurate bug descriptions, while GPT- 4o degrades sharply by 44 p and Opus mainly fails in fault localization. These differences explain much of the code-level asymmetry observed in RQ1 and RQ2. In summary this work makes the following contributions: •First, we formalize artifact-level trust reasoning for heteroge- neous software-engineering artifacts and instantiate it in TRACE, a model-agnostic pipeline for eliciting structured reasoning traces (§3.1). •Second, we construct a provenance-annotated perturbation bench- mark of 456 curated Java method bundles with six aligned degra- dation and contradiction variants, together with ground-truth metadata about bug type, severity, and mutation provenance (§3.3). •Third, we provide a large-scale cross-model study of trace fi- delity over contradictory software artifacts, showing severity- proportional but artifact-asymmetric sensitivity, a systematic implementation-only blind spot, and interpretable sources of inter-model variation (§4). 2 Related Work The closest prior work is limited. Doc2OracLL [10] studies how Javadoc affects LLM-based test-oracle generation, but its target remains downstream oracle quality rather than explicit trust al- location across conflicting artifacts. METAMON is also close in spirit: it detects inconsistencies between program documentation and observed behavior using generated tests and LLM reasoning, but it remains a pairwise documentation–behavior inconsistency detector [12]. DocPrism [16] also detects code–documentation in- consistencies across multiple languages using LLMs, but targets false-positive reduction via its LCEF methodology rather than ex- plicit trust allocation. Dietrich et al.’s work on LLM-based multi- artifact consistency verification for programming exercise quality assurance is the nearest multi-artifact analogue we found, but it targets educational QA and currently instantiates only structural and semantic inconsistency categories over educational artifacts [5]. Measuring LLM Trust Allocation Across Conflicting Software Artifacts, , In contrast, TRACE formalizes artifact-symmetric input-quality rea- soning traces over Javadoc, signature, implementation, and test pre- fix under a blind perturbation protocol, and evaluates per-artifact quality assessment, conflict analysis, outlier attribution, and source prioritization rather than only a final downstream decision. To the best of our knowledge, this explicit trace-level trust reason- ing over contradictory testing artifacts has not been studied before. 3 Approach At a high level, our approach consists of three steps: formalizing the reasoning-trace task, constructing a controlled benchmark of clean and perturbed artifact bundles, and using TRACE to elicit and validate structured traces from multiple LLMs. The remainder of this section presents these steps in turn. 3.1 Problem Formulation Input bundle. LetIbe the input space where each sample푠 ∈Iis a tuple 푠=⟨MUT,휎,퐷,휏 prefix ⟩, whereMUTis the method under test,휎is the method signature, 퐷is the Javadoc, and휏 prefix is the test setup prefix. Most samples (419 of 456) additionally include a reference assertion푂 ref , either a value-checking assertion or an expected-exception block. We use푂 ref only as an annotation for perturbation construction and optional sanity checks; it is not the prediction target of this work. Reasoning trace. LetRbe the space of structured input-quality traces. We define artifact-level trust as the model’s explicit allo- cation of reliability across the artifacts in an input bundle, oper- ationalized through per-artifact assessments, inconsistency judg- ments with affected-artifact attribution, and source prioritization. Each trace 푟 ∈ R is 푟=⟨퐴,푃,퐶,휓⟩, where 퐴 contains per-artifact quality assessments 퐴=⟨푎 javadoc ,푎 signature ,푎 mut ,푎 testprefix ,푎 overall ⟩, 푃 is a total order overJavadoc, Signature, MUT, TestPrefix with confidence scores,퐶reports inconsistencies and the most defensible behavioral hypothesis. It encodes three complementary conflict signals: pairwise conflict analysis (퐶 PCA ), which labels every artifact pair as consistent, contradictory, or incomplete, firing when the Javadoc–MUT pair is labeled contradictory; identified conflicts (퐶 IC ), firing when both Javadoc and MUT appear among the involved artifacts; and the inconsistency report (퐶 IR ), firing when an inconsistency is flagged and both Javadoc and MUT appear among the affected artifacts. Two aggregate signals are derived: Union (any signal fires) and Majority (≥2 signals agree).휓stores assumptions, limitations, and uncertainty. Goal. Starting from a curated base dataset퐷 0 and a perturbation setV, we query an LLM ensembleM and construct 푅=(푠,푣,푟,푐,푚), where푣 ∈Videntifies the dataset variant,푟is the generated trace, 푐 ∈ [0,1]is the model’s overall confidence, and푚 ∈ Mis the generating model. Our objective is to collect high-precision traces through structured prompting and conservative validation. 3.2 Model Selection We evaluate seven models served through Microsoft Foundry/Azure: Claude Opus 4.6 [2], Claude Sonnet 4.6 [3], Claude Haiku 4.5 [1], GPT-5.2 Chat [14], GPT-4o [13], DeepSeek-V3.2-Speciale [4], and Grok 4 Fast Reasoning [15]. The suite balances provider diversity and capability tier: the Claude models give a within-family scaling slice, while the others broaden cross-provider coverage under a common deployment environment. We therefore use it as a diverse but availability-constrained benchmark, not an exhaustive sam- ple. All models were run with the same harness and prompts at temperature 0.0. We excluded DeepSeek-R1-0528 and Qwen3-32B because service constraints prevented reliable completion under our fixed prompt budget; these exclusions reflect deployment limits, not model-quality judgments. 3.3 Benchmark Construction 3.3.1 Base-sample curation. We build on OE25, a large real-world benchmark curated from 25 diverse Java systems consisting of 56 modules and 223,557 inputs samples [9], containing methods under test (MUTs), Javadoc, and test artifacts. From this corpus, we re- tain 456 high-quality Java method bundles through a conservative automated filtering pipeline. We retain only non-trivial methods with 8–60 non-comment executable lines, explicit return types for non-void methods, and substantive logic. A method is considered non-trivial if it contains at least two distinct control-flow elements (e.g., conditionals, loops, or exception handling) or performs sub- stantial multi-step computation, operationalized as at least four assignments and four method calls. We exclude simple null-check- plus- delegate patterns, builder chains, and single-line wrappers. We also remove constructors, entry points, trivial getters/setter- s/wrappers, extraction artifacts, and method bodies containing in- line comments. We require English Javadocs with a substantive prose description longer than 15 characters that explains what the method does — usage warnings alone (e.g., “BEWARE: don’t call this directly” ) do not qualify. If the method is non-void, the Javadoc must include an @returntag; if it has parameters, at least one@paramtag is required. Approximately 16% of retained samples are void methods and are therefore exempt from the@returnrequirement. We exclude in- herited (@inheritDoc) and tag-only documentation, require a non-empty test prefix (≥20 characters), and remove duplicate (MUT, docstring) pairs. This pruning yields a compact high-quality subset of the original benchmark, so that score changes under perturbation reflect injected degradation rather than noise in the base data. 3.3.2 Perturbation variants. For each curated base sample we con- struct six perturbation variants, yielding seven dataset versions including the clean base. The variants are summarized in Table 1. The first three variants are deterministic, rule-based transforma- tions and are therefore exactly reproducible. The remaining three are generated using structured mutation prompts with three sever- ity tiers: heavy (H) introduces an explicit and readily observable , ,Noshin Ulfat, Ahsanul Ameen Sabit, and Soneya Binta Hossain Table 1: TRACE perturbation variants. VariantOperationSeverity - doc 푑푒푠푐. Remove Javadoc prose description; retain tags— - doc 푟푒푡푢푟푛 Remove @return clause only— - doc 푑푒푠푐.+푟푒푡푢푟푛 Remove prose description and @return— + bug 푑표푐 Perturb Javadoc against pristine MUT (wrong behavior/return/params, missing info) H/N/S + bug 푀푈푇 Perturb MUT against pristine Javadoc (logic, null- check, boundary, API-misuse bugs) H/N/S + contradict 푑표푐+푀푈푇 Mutate MUT and Javadoc to contradict each otherH/N/S contradiction, normal (N) introduces a plausible defect that mani- fests under specific inputs or boundary conditions, and subtle (S) introduces a minimal corner-case deviation that requires careful cross-artifact reasoning to detect. For all generated variants, the method signature is held fixed, the mutated artifact must remain natural and realistic, and no comments, markers, or formatting cues may reveal the injected fault. We manually verified every perturbed sample to ensure that each injected change is correctly realized, and aligned with its intended severity tier. Each sample also re- tains detailed metadata, including the modified artifact(s), fault category, a concise change description, the difficulty tier, and a one- line ground-truth summary, ensuring transparency and reusability for future evaluation. For each base sample, we construct six perturbation variants, yield- ing seven aligned inputs per item and 3,192 evaluation instances in total. Because these variants share the same underlying task, perturbation analyses treat them as paired observations rather than independent samples. The full evaluation matrix therefore com- prises 7× 3,192= 22,344 model–input trials. 3.4 TRACE: Reasoning-Trace Construction 3.4.1 Pipeline. Figure 1 summarizes TRACE. For each (model, dataset-variant, sample) tuple, the pipeline loads the artifact bundle, instantiates a fixed user prompt, queries the model with a shared system prompt, parses the returned JSON, validates required fields, and appends the trace to disk incrementally. The pipeline is parallel and resumable: interrupted runs continue without reprocessing completed samples, and model-specific concurrency, retry, and backoff parameters are tuned per endpoint. In the canonical run, this recovery logic yielded 22,339 valid traces out of 22,344 API calls. 3.4.2 System prompt design. All models receive the same fixed system prompt. Rather than reproduce the full prompt verbatim, we highlight the four invariants that matter for interpreting TRACE. •First, the response must be a single valid JSON object with no surrounding text. • Second, the prompt enforces artifact symmetry: no artifact is ground truth a priori, and models are explicitly told not to default to trusting the MUT, the Javadoc, or the test prefix. •Third, the analysis is structured into six required components: input-quality assessment (퐴), pairwise conflict analysis (퐶 PCA ), identified conflicts (퐶 IC ), inconsistency report (퐶 IR ), anomaly assessment, and source prioritization (푃 ). •Fourth, scoring is standardized: each artifact receives a score in[0,1]plus a LOW/MEDIUM/HIGH label, and source prioritiza- tion is a total order overJavadoc, Signature, MUT, TestPrefix. These constraints make the outputs directly comparable and auto- matically measurable across models. 3.4.3 Blind perturbation protocol. The user prompt presents only the four artifacts as labeled sections. It never reveals whether a sample is clean or perturbed, which artifact was modified, or the perturbation severity. This blind setup ensures that score drops, inconsistency flags, and prioritization changes reflect the model’s own sensitivity to artifact quality rather than compliance with hints. 3.4.4 Trace fields used for evaluation. Three trace fields are central to evaluation. First, the input-quality assessment assigns each arti- fact an individual score in[0,1]reflecting its own completeness, clarity, and internal consistency, independent of whether other arti- facts have problems. It also includes a holistic overall bundle score that captures whether the artifact set as a whole supports confident reasoning, accounting for unresolvable conflicts and missing specifi- cation context that per-artifact scores alone cannot express. Second, the consistency field captures cross-artifact conflict signals at three levels: pairwise verdicts across all six artifact pairs, an explicit enu- meration of identified conflicts, and a consolidated inconsistency report whose natural-language description serves as the unit of semantic alignment comparison against dataset ground-truth anno- tations. Third, the source-prioritization field provides a total order over the four artifact sources, together with per-rank confidence scores, and measures whether models correctly deprioritize faulty artifacts. Together, these fields make the model’s quality judgments, contradiction detection, holistic assessment, and source-level trust decisions directly and automatically measurable. 3.4.5 Stored outputs. Each retained record stores the input bundle, the optional reference assertion, perturbation provenance, the gen- erated trace fields (Assessment,Prioritization,Consistency, Implications, and metadata), the model identifier, and the over- all confidence score. This representation supports direct analysis, human inspection, and future supervision or fine-tuning. 4 Experimental Study 4.1 RQ1: Input-quality sensitivity If LLMs are to serve as input-quality reasoners in SE pipelines, their traces must do more than flag that an artifact is flawed; they must indicate where degradation occurs and how much evidence quality has deteriorated. RQ1 therefore asks whether trace-level quality judgments respond proportionally to controlled perturbations rather than collapsing to binary fault detection, across both removal-type perturbations (missing@return, description removal, and both) and injection-type perturbations (semantic bugs in the Javadoc, MUT, or both, each at three severity tiers). 4.1.1 Experimental Setup. For each trace, we extract the model’s quality assessments for Javadoc, Signature, MUT, Test Prefix, and Measuring LLM Trust Allocation Across Conflicting Software Artifacts, , BR1R2R3BdBmC 0.0 0.2 0.4 0.6 0.8 1.0 Mean Score Javadoc BR1R2R3BdBmC Signature BR1R2R3BdBmC MUT BR1R2R3BdBmC Test Prefix BR1R2R3BdBmC Overall DS-V3.2 GPT-4o GPT-5.2 Grok-4 Haiku Opus Sonnet C = +(doc vs MUT contradiction) R3 = (desc + @return)Bd = +(buggy doc)Bm = +(buggy MUT) B = Base (clean)R1 = (desc)R2 = (@return) Figure 2: Mean input-quality scores by dataset variant and model across five artifact dimensions (Javadoc, Signature, MUT, Test Prefix, Overall), revealing baseline calibration differences and the largest perturbation-induced drops per artifact. R1R2R3BdBmC GPT-4o Grok-4 GPT-5.2 Opus Sonnet DS-V3.2 Haiku -0.12-0.09-0.30-0.04-0.01-0.02 -0.24-0.15-0.46-0.07-0.02-0.01 -0.16-0.10-0.34-0.03-0.010.02 -0.28-0.14-0.44-0.17-0.02-0.07 -0.23-0.15-0.39-0.20-0.03-0.08 -0.19-0.16-0.360.00-0.020.05 -0.19-0.13-0.37-0.18-0.03-0.09 Javadoc R1R2R3BdBmC 0.000.00-0.010.010.010.01 0.100.100.080.120.110.11 0.040.050.010.050.070.06 -0.020.02-0.060.080.030.05 0.010.03-0.030.040.040.04 0.090.100.070.110.100.10 0.010.02-0.000.050.030.04 Signature R1R2R3BdBmC -0.000.00-0.01-0.03-0.02-0.04 -0.01-0.00-0.02-0.02-0.07-0.05 -0.00-0.000.00-0.04-0.08-0.08 -0.01-0.01-0.02-0.03-0.14-0.10 -0.01-0.01-0.01-0.02-0.10-0.07 -0.01-0.01-0.01-0.02-0.09-0.05 -0.02-0.01-0.02-0.02-0.11-0.09 MUT R1R2R3BdBmC -0.00-0.01-0.02-0.01-0.00-0.01 -0.01-0.00-0.01-0.000.000.01 -0.01-0.01-0.02-0.01-0.02-0.01 -0.00-0.01-0.000.00-0.00-0.00 -0.01-0.01-0.02-0.00-0.000.00 -0.01-0.02-0.010.01-0.020.01 -0.01-0.01-0.01-0.00-0.01-0.01 Test Prefix R1R2R3BdBmC -0.05-0.04-0.12-0.08-0.02-0.07 -0.06-0.03-0.14-0.14-0.04-0.10 -0.05-0.03-0.11-0.14-0.06-0.12 -0.09-0.05-0.15-0.11-0.05-0.08 -0.08-0.06-0.15-0.15-0.07-0.12 -0.06-0.05-0.13-0.16-0.09-0.14 -0.08-0.05-0.14-0.16-0.08-0.14 Overall 0.5 0.4 0.3 0.2 0.1 0.0 0.1 0.2 score R1 = (desc)R2 = (@return)R3 = (desc + @return)Bd = +(buggy doc)Bm = +(buggy MUT)C = +(doc vs MUT contradiction) Figure 3: Score changes from base to perturbed datasets. Delta from base (Δ= perturbed− base) by metric, perturbation, and model. Larger negative deltas are concentrated in the perturbed artifact, indicating localized sensitivity with limited spillover. Heavy NormalSubtle 0.0 0.2 0.4 0.6 0.8 1.0 Mean Overall Score MUT Bug Injected Heavy NormalSubtle Doc Bug Injected Heavy NormalSubtle MUT+Doc Contradiction DS-V3.2 GPT-4o GPT-5.2 Grok-4 Haiku Opus Sonnet Figure 4: Severity breakdown for documentation bugs, MUT bugs, and MUT+Doc contradictions (grouped bars: mean overall quality score per model and severity tier; error bars:±1 std). All models preserve severity monotonicity, with larger gaps for documentation than code perturbations. Overall input quality. We then compute three summary views. First, we report mean scores by model and dataset variant to characterize baseline calibration and gross perturbation effects. Second, because raw scores are not directly comparable across models, we use delta from base,Δ= mean(perturbed)− mean(base), as the main cross- model sensitivity measure, where more negative values indicate stronger penalization. Third, for the three severity-graded perturba- tion families (documentation bug, MUT bug, and MUT+Doc contra- diction), we break results down by severity to test whether scores decrease monotonically from subtle to heavy perturbations. 4.1.2 Results. Baseline calibration differs substantially across mod- els. Figure 2 shows that even on the clean base dataset, models use markedly different scoring scales.GPT-4oandGrok 4 Fast Reasoning assign the highest mean overall score (0.713), whereas , ,Noshin Ulfat, Ahsanul Ameen Sabit, and Soneya Binta Hossain Sonnet 4.6is lowest (0.555). This 0.16-point spread on identical clean inputs indicates that absolute score levels are model-specific. Accordingly, we use within-model deltas from base, rather than raw scores, as the primary sensitivity measure in the remainder of RQ1. Perturbation effects are concentrated on the altered artifact, with limited spillover. Figure 3 shows that score changes are largely artifact-specific rather than globally pessimistic. When both the Javadoc description and@returnare removed, Javadoc scores drop by−0.300 to−0.463, whereas MUT scores move by less than 0.020 and overall scores decline by only−0.109 to−0.155. Models con- centrate penalties on the degraded source, with limited spillover into holistic judgment. Documentation perturbations are more salient than implementation perturbations. Across the model suite, perturbations to the Javadoc induce larger overall penalties than comparable perturbations to the MUT. Injecting Javadoc bugs reduces overall scores by−0.082 to−0.163, while removing a single Javadoc field causes only milder degradation (−0.029 to−0.095). In contrast, removing both the de- scription and@returnyields a much larger drop (−0.109 to−0.155), comparable to injecting incorrect Javadoc content. The important distinction is therefore not only missing vs. present, but also partially incomplete vs. materially misleading. In the current setting, models are more responsive to natural-language specification defects than to fine-grained implementation defects. Scores track severity monotonically, but much more sharply for docu- mentation. Figure 4 shows that every model assigns lower overall scores as perturbation severity increases: heavy<normal<subtle in all three graded perturbation families. For documentation bug injection, the heavy-to-subtle gap ranges from 0.152 (GPT-4o) to 0.253 (Sonnet 4.6); for MUT bug injection, the corresponding gap is much smaller (0.049–0.123). The same ordinal trend is visible for MUT+Doc contradictions. RQ1 therefore goes beyond binary fault detection: the traces preserve graded severity information, but they do so much more reliably for documentation defects than for code defects. Model-specific magnitude varies, but the asymmetry is consistent. The strength of penalization varies across models.Sonnet 4.6and Opus 4.6are typically the most sensitive, whileGPT-4oexhibits the smallest overall deltas. The remaining models lie between these extremes. However, the qualitative pattern does not change: all seven models react more strongly to Javadoc degradation than to implementation degradation, and all preserve the same severity ordering. We therefore treat inter-model differences here as differ- ences in calibration and sensitivity magnitude; their likely causes are investigated in RQ3. Practical implication. Taken together, these results suggest that current LLM traces are more reliable as auditors of documentation quality than as detectors of subtle code-level faults. For SE pipelines, Javadoc should therefore be treated as a high-leverage signal for prompt construction, input triage, and trust calibration. At the same time, this should not be read as a license to trust documentation alone: code-only drift is under-signaled relative to documentation faults, so explicit code-aware or cross-artifact consistency checks remain necessary. RQ1 Findings LLM reasoning traces are sensitive to input degradation, but unevenly across artifact types. Across all seven models, penal- ties concentrate on the perturbed artifact and scores broadly follow the expected severity ordering, indicating more than bi- nary fault flagging. Yet documentation degradations produce larger drops and clearer heavy–normal–subtle separation than implementation degradations. Current models are thus better at auditing natural-language specifications than at detecting fine-grained code faults, so documentation is a strong sig- nal for prompting and quality gating, but not a substitute for explicit cross-artifact consistency checks. 4.2 RQ2: Javadoc–MUT consistency detection A useful contradiction detector must do more than lower quality scores. It must surface a Javadoc–MUT mismatch explicitly, localize the faulty artifact, and describe the mismatch precisely enough to act on. RQ2 therefore asks whether LLMs can perform explicit, attributed, and faithful inconsistency detection, and whether that ability holds when faults originate in documentation, in code, or only in code while the Javadoc remains plausible. 4.2.1 Experimental Setup. We evaluate four quantities. First, ex- plicit detection rate: the fraction of traces that flag a Javadoc–MUT inconsistency under Doc Bug Injected, MUT Bug Injected, and MUT+Doc Contradiction, compared against the clean-sample false-positive floor. Our primary metric is IR-strict; PCA, IC, Union, and Ma- jority are supplementary views. Second, attribution correctness: whether the trace identifies the injected artifact as the outlier. Third, description fidelity: measured using semantic cosine sim- ilarity between the model’s generated descriptions from multi- ple signals and the ground-truth fault annotation, computed via BAAI/bge-base-en-v1.5sentence embeddings. Fourth, confidence calibration: the separation between confidence on correct and incor- rect detections. Fifth, we evaluate source prioritization discrimina- tion by computing Kendall’s휏 푏 between a binary provenance label vector (1 for the known faulty artifact, 0 for clean artifacts) and the model’s rank vector from source prioritization logic, where positive 휏 푏 indicates the model assigns lower reliability to the faulty artifact. This metric is computed across 456 samples per model on each of the three perturbed datasets (DocBug, MUTBug, and Contradic- tion), stratified by perturbation type, contradiction strategy, and severity. 4.2.2 Results. Models detect explicit faults reliably, but noise floor matters. Figure 5 reports IR-strict detection rates across all perturba- tion types and models. Three complementary detection signals are considered, evaluated both individually and through two composite metrics (Union and Majority) [see sec 3.1]. On clean base samples, false-positive rates range from 6% (GPT- 4o) to 39% (Sonnet), establishing markedly different per-model noise floors. This spread means that raw detection rates are not Measuring LLM Trust Allocation Across Conflicting Software Artifacts, , DS-V3.2 GPT-4o GPT-5.2 Grok-4 Haiku Opus Sonnet 31%24%26%12%92%59%85% 6%9%6%2%67%20%50% 11%9%6%4%87%39%78% 23%21%18%12%89%51%79% 34%44%40%41%88%61%83% 15%15%14%10%70%58%78% 39%40%41%33%94%70%91% (1) IR strict [PRIMARY] DS-V3.2 GPT-4o GPT-5.2 Grok-4 Haiku Opus Sonnet 26%22%24%11%92%58%84% 1%1%1%0%59%11%42% 6%7%5%4%87%38%78% 15%15%15%10%88%48%78% 10%7%7%4%82%39%72% 11%10%10%7%90%50%84% 14%11%12%7%92%48%83% (2) PCA DS-V3.2 GPT-4o GPT-5.2 Grok-4 Haiku Opus Sonnet 34%30%31%19%93%63%87% 6%9%6%2%67%20%50% 20%20%20%11%89%51%82% 28%32%27%27%89%56%81% 50%75%64%70%89%71%87% 77%87%81%90%97%89%98% 71%83%77%82%97%86%96% (3) IC DS-V3.2 GPT-4o GPT-5.2 Grok-4 Haiku Opus Sonnet 35%31%31%19%93%63%87% 6%9%6%2%67%20%50% 21%20%20%11%89%51%82% 29%33%28%27%90%57%82% 52%77%65%73%90%73%88% 79%88%82%90%98%91%98% 73%84%79%84%98%88%97% (4) Union (any) BR1R2R3BdBmC DS-V3.2 GPT-4o GPT-5.2 Grok-4 Haiku Opus Sonnet 31%25%25%12%92%59%86% 6%9%6%2%67%20%50% 10%9%5%4%87%39%78% 22%20%18%12%89%51%79% 33%43%39%37%88%61%82% 17%17%17%11%91%59%86% 38%39%40%31%95%70%91% (5) Majority (>=2/3) 0% 20% 40% 60% 80% 100% Rate Bd = +(buggy doc)Bm = +(buggy MUT)C = +(doc vs MUT contradiction) B = Base (clean)R1 = (desc)R2 = (@return)R3 = (desc + @return) Figure 5: Inconsistency Detection Rates of Javadoc-MUT us- ing 5-Metric comparisons - panels by perturbation type, with bars per model and three severity tiers per bar group; the false-positive baseline is shown as a dashed line. Net detec- tion gain above baseline is the key quantity. directly comparable: a model with a high false-positive rate can appear to detect more contradictions simply by flagging more ag- gressively. Net detection gain above baseline is therefore the more informative comparison. For Doc Bug Injected, net gains range from +54.8 to+76.5 p across models; GPT-5.2 achieves the highest gain (+76.5 p) despite its moderate detection rate (87.5%), owing to its low false-positive baseline (11%), while Haiku and Opus yield the smallest gains (≈+54.8 p). Under the most demanding condition— MUT+Doc Contradiction—detection rates reach 91.2% (Sonnet) and fall to 49.6% (GPT-4o). Detection generally declines with pertur- bation subtlety across all models, a pattern examined further in Section 4.3. The multi-signal analysis shows that more permissive aggregation (Union) recovers detections missed by IR-strict, particularly for models with localization gaps (e.g., Opus: 78.1% IR-strict vs. 98.5% Union on MUT+Doc Contradiction), but at the cost of substantially higher false-positive rates on clean samples (15.1% to 78.9% for Opus). Majority vote tracks IR-strict closely and offers little addi- tional sensitivity gain. Detection correlates with description faithfulness. Figure 6 confirms that explicit detection is not merely flag-and-forget: detected traces align more closely with ground-truth fault descriptions than missed ones (mean combined cosine: 0.833 vs. 0.730, gap=+0.104), and this gap holds universally across all seven models. In other words, models that recognize the inconsistency are also more likely to describe it accurately and with the correct fault emphasis. Among the models, Opus, Sonnet, and GPT-5.2 obtain the highest combined cosine similarity, indicating stronger explanation fidelity, whereas GPT-4o performs worst across every signal. This suggests that GPT- 4o not only fails to detect more contradictions, but also produces the least faithful descriptions when it does detect them. Importantly, this pattern indicates that inconsistency detection and explanatory quality are coupled capabilities rather than independent behaviors. Confidence is not a reliable default gate. As shown in Figure 7, only DeepSeek-V3.2-Specialeshows meaningful calibration (confi- dence gap: 0.73). The other six models are weakly calibrated or effectively uncalibrated, so confidence should not be used for auto- mated gating without per-model recalibration. Detection collapses when only the implementation diverges. The most consequential finding of RQ2 is a sharp strategy-level asymmetry re- vealed by Figure 8. When both artifacts are mutated (BOTH) or only the Javadoc is perturbed (Docstring_Only), detection rates remain close to Doc Bug Injected performance (median+3.9 and+4.3 p respectively across models). But when only the MUT diverges and the Javadoc remains correct (MUT_Only), detection drops 21–43 p below BOTH across all models (mean gap:−27.7 p vs. bug injected doc). This asymmetry is not a calibration artifact: it holds across all seven models and all five detection signals. It reveals that cur- rent models reason primarily from documentation-side signals — when the Javadoc is intact and only the code is wrong, the model tends not to raise an alarm. Detection reliability depends heavily on which side carries the fault. The DocBug net gains (+54.8–76.5 p) consistently dominate MUTBug gains (+14.0–42.5 p), confirming that LLM reasoning traces are systematically better at surfacing specification-side faults than implementation-side faults. Source prioritization reveals weak fault discrimination overall. Only Sonnet (휏 푏 =0.33) and Opus (휏 푏 =0.30) show meaningful posi- tive concordance on documentation faults, while no model reliably discriminates faulty implementations (휏 푏 ≤0.10). Discrimination follows severity monotonically, ranging from 0.08 to 0.63 at heavy documentation perturbations but dropping to near zero or neg- ative at subtle severity. This pattern mirrors the documentation- over-code asymmetry and severity sensitivity observed in explicit detection rates. Practical implication. These findings have direct consequences for how LLM-based consistency checking should be deployed in prac- tice. For documentation auditing — detecting stale, incorrect, or missing Javadoc against a trusted implementation — current models are sufficiently reliable for automated triage, with net detection gains of+54.8 to+76.5 p above the false-positive floor. This makes them viable as a first-pass filter in code review pipelines or CI/CD workflows, flagging Javadoc drift for human attention without re- quiring full manual inspection. For implementation auditing — de- tecting code drift against correct documentation — the MUT_Only blind spot means that silent failures are the norm rather than the exception, and human review or complementary static analysis , ,Noshin Ulfat, Ahsanul Ameen Sabit, and Soneya Binta Hossain 0.00.20.40.60.81.0 Combined cosine similarity to ground truth Missed (0) Detected (1) Detection outcome (0=missed, 1=detect) Detection by signal IR strict: 2477/3189 (78%) PCA: 2373/3189 (74%) IC: 2646/3189 (83%) Combined: 2661/3189 (83%) Model DS-V3.2 GPT-4o GPT-5.2 Grok-4 Haiku Opus Sonnet Detected mean: 0.83 (n=2477) Missed mean: 0.73 (n=712) DS-V3.2 GPT-4o GPT-5.2 Grok-4 Haiku Opus Sonnet Model 0.0 0.2 0.4 0.6 0.8 1.0 Cosine Similarity with Ground Truth Signal PCA IC IR Combined (a)(b) Figure 6: Multi-signal cosine similarity to ground-truth fault descriptions (MUT+Doc Contradiction;푁= 3,189; embedding: BAAI/bge-base-en-v1.5). Figure (a): cosine similarity vs. IR-strict detection outcome (detected mean:0.83,푛= 2,477; missed mean: 0.73, 푛= 712). Figure (b): per-sample cosine distribution by signal (PCA, IC, IR, Combined) across all models. DS-V3.2GPT-4oGPT-5.2Grok-4HaikuOpusSonnet 0.0 0.2 0.4 0.6 0.8 1.0 Mean Confidence 0.93 0.20 ∆=0.73 0.88 0.78 ∆=0.09 0.93 0.84 ∆=0.09 0.95 0.95 ∆=0.00 0.95 0.93 ∆=0.02 0.94 0.85 ∆=0.08 0.91 0.83 ∆=0.09 Detected Not detected DS-V3.2GPT-4oGPT-5.2Grok-4HaikuOpusSonnet 0.0 0.2 0.4 0.6 0.8 Confidence Gap (Detected Not Detected) 0.73 n=67 0.09 n=230 0.09 n=100 0.00 n=97 0.02 n=78 0.08 n=100 0.09 n=40 (a) (b) Figure 7: Confidence calibration by model on MUT+Doc Contradiction (IR-strict). Figure (a): mean confidence for detected vs. missed traces. Figure (b): calibration gap (푛= missed samples shown below bars); only DS-V3.2 achieves meaningful separation (gap= 0.73); all other models are effectively uncalibrated. Measuring LLM Trust Allocation Across Conflicting Software Artifacts, , BBdCdBmCmCbCo DS-V3.2 GPT-4o GPT-5.2 Grok-4 Haiku Opus Sonnet 31%92%97%59%62%97%85% 6%67%61%20%25%62%50% 11%87%91%39%50%93%78% 23%89%94%51%51%91%79% 34%88%91%61%66%91%83% 15%70%81%58%62%91%78% 39%94%99%70%77%98%91% Net Gain vs Base (p) Doc: +61.5 MUT: +28.2 Doc: +61.0 MUT: +14.0 Doc: +76.5 MUT: +28.5 Doc: +66.0 MUT: +28.1 Doc: +54.8 MUT: +27.4 Doc: +54.8 MUT: +42.5 Doc: +55.0 MUT: +30.7 0% 20% 40% 60% 80% 100% Cb = Contra: both (n=152)Co = Contra: overall (n=456) Cd = Contra: doc (n=152)Cm = Contra: MUT (n=152) B = Base (n=456)Bd = buggy doc (n=456)Bm = buggy MUT (n=456) Figure 8: Doc-vs-code detection asymmetry under IR-strict evaluation. Doc-side faults are detected more reliably than code-side faults across all seven models, with net gains of+54.8–+76.5p for Doc Bug Injected vs.+14.0–+42.5p for MUT Bug Injected. remains necessary. In real-world codebases, where method changes frequently outpace documentation updates, this blind spot is not an edge case but the dominant failure mode; LLM-based consis- tency checkers deployed without awareness of this limitation will systematically miss the most common class of inconsistency. Confi- dence scores should not be used as a proxy for detection reliability without per-model recalibration. When description accuracy is the priority — for instance, in fault localization or patch generation workflows where a precise characterization of the inconsistency matters more than recall — Opus is the preferred model; when raw detection rate matters most, Sonnet leads. For pipelines requiring both high detection and faithful description, a two-model ensemble of Sonnet (primary detector) and Opus (secondary explainer) offers the best practical trade-off. RQ2 Answer LLMs detect explicit documentation errors with high net ac- curacy (+54.8–+76.5 p above false-positive baseline) and describe detected faults faithfully when using structured pair- wise analysis. However, they exhibit a systematic blind spot for implementation drift: when only the MUT changes and documentation remains correct, detection drops 21–43 p be- low the both-changed baseline across all models. Confidence scores are poorly calibrated for six of seven models and cannot be relied upon without recalibration. Together, these findings cast current LLMs as reliable documentation auditors but un- reliable code-drift detectors. 4.3 RQ3: Semantic code understanding as a driver of trace fidelity RQ1 and RQ2 establish what models do but leave open why some produce more faithful reasoning traces than others under identical prompts. Two asymmetries call for explanation: models discrimi- nate documentation-level severity far more sharply than code-level severity (RQ1), and detection drops substantially when only the MUT diverges while the Javadoc remains correct — the MUT_Only blind spot (RQ2). Both patterns suggest that many models rely more on natural-language cues than on semantic interpretation of imple- mentation behavior. RQ3 tests this directly: can these asymmetries, and the broader inter-model variation in trace fidelity, be explained by differences in semantic code understanding? SonnetDS-V3.2HaikuGPT-5.2Grok-4OpusGPT-4o 40% 60% 80% 100% Detection Rate (%) 96% 92% 91% 90% 88% 77% 76% 91% 83% 83% 76% 81% 81% 41% 86% 80% 75% 68%68% 76% 32% Severity Heavy Normal Subtle Figure 9: Severity-stratified MUT–Javadoc contradiction de- tection rates (IR-strict;n= 152per severity tier). Six of seven models satisfy heavy>normal≥subtle monotonic- ity; Sonnet is the most robust (96%→91%→86%) and GPT- 4o shows the steepest drop (76%→41%→32%). Opus deviates (77%→81%→76%) due to a fault-localization error rather than detection failure. 4.3.1 Experimental Setup. To isolate semantic code understanding from surface-level pattern matching, we evaluate MUT–Javadoc contradiction detection across the three severity tiers defined in , ,Noshin Ulfat, Ahsanul Ameen Sabit, and Soneya Binta Hossain PCAICIRComb.PCAICIRComb.PCAICIRComb. Opus Sonnet GPT-5.2 Grok-4 Haiku DS-V3.2 GPT-4o 0.830.820.800.830.840.800.820.840.840.800.810.83 0.820.810.800.820.830.790.820.840.830.790.810.83 0.810.760.780.820.830.680.790.830.820.620.770.83 0.800.720.780.810.800.690.790.820.800.590.770.81 0.810.750.780.810.810.690.790.810.810.660.780.81 0.810.760.770.810.810.720.730.820.820.690.690.82 0.750.580.680.750.740.330.520.750.730.260.520.73 Heavy (n=1064) Normal (n=1063)Subtle (n=1062) 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Mean Cosine Similarity Figure 10: Semantic fidelity of inconsistency descriptions by perturbation severity (cosine similarity to ground truth; embedding: BAAI/bge-base-en-v1.5; dataset: MUT+Doc Contradiction). IC and IR scores are zero when the model does not flag a Javadoc– MUT conflict (empty signal text). Section 3.3.2 on the full MUT+Doc Contradiction dataset. As per- turbations move from heavy to subtle, lexical and structural cues progressively disappear — models relying on such cues should degrade sharply, while models reasoning about implementation behavior should remain stable. We assess two complementary dimensions: explicit contradiction detection (whether the trace flags a Javadoc–MUT inconsistency in IR-strict) and description fidelity (semantic cosine similarity be- tween the model’s explanation and the annotated fault summary). We summarize degradation via the heavy-to-subtle drop in both detection rate (Δ= heavy− subtle) and cosine similarity: large drops indicate surface-cue reliance; small drops indicate stable cross-artifact semantic reasoning. 4.3.2 Results. Severity stratification exposes clear differences in code understanding. Figure 9 reveals three tiers of behavior. Sonnet is the most stable (96%→91%→86%,Δ=10 p), followed by DS- V3.2 (92%→83%→80%,Δ=12 p) and Haiku (91%→83%→75%, Δ=16 p). These models remain comparatively robust as surface cues disappear, consistent with deeper cross-artifact behavioral reasoning. GPT-5.2 and Grok-4 form an intermediate group, both reaching 68% at subtle severity (Δ=22 p and 20 p, respec- tively), suggesting partial reliance on surface cues. GPT-4o shows the sharpest collapse, falling from 76% at heavy to 32% at subtle (Δ=44 p), consistent with a surface-driven strategy that fails once explicit divergence weakens. Opus is a distinct case: its heavy- severity score is unusually low (77%), while normal and subtle performance remains relatively flat (88%, 76%). This pattern is bet- ter explained by a localization error than by failure to recognize the contradiction itself. Opus correctly identifies the semantic incon- sistency but consistently flags the Javadoc as the affected artifact excluding MUT, causing IR strict evaluation. Description fidelity follows the same pattern. Figure 10 shows that weakening detection is mirrored by weakening description fidelity. The IC signal is the most diagnostic: GPT-4o drops from 0.58 at heavy to 0.33 at normal and 0.26 at subtle, indicating that as sur- face cues disappear it not only misses more contradictions, but also describes the detected ones less faithfully. On the other hand, Opus and Sonnet both of them remain stable across severity tiers on both IC (0.82→0.80→0.80 and 0.81→0.79→0.79) and Combined (0.83→0.84→0.83 and 0.82→0.84→0.83), suggesting that their ex- planations remain anchored to the underlying fault rather than collapsing into generic inconsistency language. The PCA signal is flatter across models, implying that pairwise structural compar- ison is easier to preserve under subtle perturbations than richer semantic characterization. Semantic code understanding connects the RQ1 and RQ2 asymme- tries. Both asymmetries share a common cause: when implemen- tation faults leave weak lexical cues, models lacking deep code comprehension have little to detect — manifesting as weak severity discrimination in RQ1 and theMUT_Onlyblind spot in RQ2. The correspondence is clearest at the extremes: Sonnet and Haiku re- main strong on both axes; GPT-4o and GPT-5.2 deteriorate on both. DS-V3.2 is more mixed — stable across severity tiers but retaining a sizableMUT_Onlygap — indicating that semantic code understand- ing explains much, but not all, of the observed variation. Practical implication. These results provide concrete guidance for model selection in cross-artifact SE workflows. Sonnet, DS-V3.2, and Haiku are the strongest choices when subtle implementation- level contradictions must be surfaced without strong lexical cues. GPT-4o is better suited to documentation-facing checks than to subtle code-drift detection. Opus occupies a different niche: it is often strong at describing a contradiction once it is surfaced, but its localization errors make it less suitable as a sole detector. In Measuring LLM Trust Allocation Across Conflicting Software Artifacts, , practice, it is best treated as a second-stage explainer rather than the primary contradiction screen. RQ3 Findings Inter-model variation in trace fidelity is largely explained by differences in semantic code understanding. Sonnet, DS-V3.2, and Haiku remain comparatively stable as contradiction cues become subtle and also preserve stronger description fidelity, indicating deeper cross-artifact reasoning. GPT-4o collapses sharply once surface divergence weakens, consistent with surface-level pattern matching. Opus forms a distinct case: it often understands the contradiction semantically, but mis- localizes the faulty artifact. Together, these patterns explain much of the weaker code-level severity discrimination in RQ1 and the implementation-only blind spot in RQ2. 5 Discussion Why high-precision filtering matters. TRACE favors fewer trust- worthy traces over many noisy ones. Discarding low-confidence traces reduces coverage, but it improves supervision quality and lowers the risk of training on fabricated rationales. Because retained traces preserve provenance and validation metadata, the resulting selection effects remain measurable rather than hidden. Toward dynamic prompting. A practical use of TRACE is as a pre- flight stage before downstream SE tasks. Artifacts judged low qual- ity or internally inconsistent can be downweighted or omitted, reducing spurious overreliance. Conversely, high-quality Javadoc can be emphasized for intent-oriented tasks such as test generation, patching, and summarization. We view this as a promising deploy- ment direction rather than a fully validated prompting policy. 5.1 What LLMs Can Do 1. Encode graded degradation. Across all seven models, quality scores preserve the expected severity ordering rather than be- having like binary fault flags. This is one of the clearest positive findings of the study. 2.Localize degradation to the affected artifact. Score penalties are concentrated on the perturbed artifact instead of being spread uniformly across the bundle, indicating artifact-level attribution rather than undifferentiated suspicion. 3.Audit documentation-side faults reliably. Current models surface explicit Javadoc faults and many Javadoc–MUT contra- dictions at useful rates, making them viable for documentation- centered triage. 4.Expose meaningful model differences. Inter-model variation is interpretable rather than noisy: Sonnet, DS-V3.2, and Haiku remain comparatively stable under subtle contradictions, GPT- 4o collapses as surface cues weaken, and Opus often explains contradictions well even when localization is weaker. 5.2 What LLMs Cannot Do 1.Support raw cross-model comparison without calibration. Different models operate with markedly different false-positive floors and confidence behavior, so raw scores should not be compared without normalization or per-model calibration. 2.Discriminate subtle code faults as well as documentation faults. Severity separation is consistently weaker for implemen- tation perturbations than for Javadoc perturbations, showing that fine-grained code defects remain under-signaled. 3.Reliably detect implementation-only drift. The strongest limitation is theMUT_Onlyblind spot: when the Javadoc remains plausible and only the implementation diverges, detection drops sharply across all models. 4. Use confidence as a safe default gate. For six of seven mod- els, confidence does not cleanly separate correct from incorrect detections, so confidence alone should not control automated decisions. 5. Localize every detected contradiction correctly. Some mod- els recognize that a mismatch exists but still assign fault to the wrong artifact, which limits their usefulness as stand-alone au- ditors. Practical Implications. In deployment terms, current models are bet- ter used as documentation auditors than as autonomous code-drift detectors. TRACE is most useful as a triage stage before correctness- critical downstream tasks. High-quality Javadoc is a strong sig- nal for prompt construction and intent-oriented generation, but it should not be treated as authoritative when code-only drift is possi- ble. For raw contradiction detection, Sonnet is the strongest default choice. For subtler implementation-level contradictions, Sonnet, DS-V3.2, and Haiku are the most reliable options. Opus is best used as a second-stage explainer once a contradiction has already been surfaced, rather than as the sole detector. DeepSeek-V3.2-Speciale is the most useful second opinion when confidence-aware triage is needed, because it is the only model with meaningful calibration. Novelty. TRACE adds three things. First, it treats artifact-level trust reasoning as a first-class prediction target rather than evaluating models only through downstream task accuracy. Second, it intro- duces a controlled perturbation benchmark that aligns clean, de- graded, and contradictory artifact bundles, enabling direct evalua- tion of per-artifact assessment, inconsistency detection, attribution, and source prioritization. Third, it shows that benchmark differ- ences can be linked to interpretable behavioral limits—most notably differences in semantic code understanding—yielding guidance that final-output accuracy alone cannot provide. 6 Threats to Validity Internal Validity. Our perturbations are LLM-generated, so samples labeled as subtle may still expose unintended cues. We mitigate this by retaining only compilable, natural-looking variants and strat- ifying analyses using provenance-baseddetection_difficulty. Ground-truth one-line summaries are somewhat subjective; we therefore emphasize distributional trends over individual annota- tions. , ,Noshin Ulfat, Ahsanul Ameen Sabit, and Soneya Binta Hossain External Validity. Our benchmark includes 456 curated Java method bundles from 25 projects, which may limit generalization to other languages, domains, or artifact types. Additionally, our controlled perturbations are cleaner than real-world inconsistencies, which are often partial, distributed, and semantically entangled. Construct Validity. TRACE quality scores reflect perceived infor- mativeness and internal consistency, not ground-truth correctness. Kendall’s휏orderings derive from perturbation design rather than human rankings, and cosine similarity with sentence-transformer embeddings is only a proxy for description fidelity, not a substitute for human evaluation. 7 Conclusion We presented TRACE, a framework for eliciting and evaluating structured artifact-level trust traces over heterogeneous SE arti- facts. Across 22,339 valid traces from seven models on 456 curated Java method bundles, quality penalties are largely localized to the perturbed artifact and increase with severity, but much more sharply for documentation than for code. Models reliably surface documentation-side faults yet show a systematic blind spot when only the implementation drifts while the Javadoc remains plausible; confidence is also poorly calibrated for six of seven models. These findings suggest that current LLMs are better at auditing natural- language specifications than at detecting subtle code-level drift, and that much of the observed inter-model variation is explained by dif- ferences in semantic code understanding: stronger models remain stable as surface cues disappear, whereas weaker models collapse under subtle implementation-level contradictions. We view TRACE as a foundation for artifact-aware evaluation, dataset construction, and quality-aware SE assistants that can decide what to trust before making correctness-critical downstream decisions. References [1] Anthropic. 2025. Claude Haiku 4.5 System Card. Technical Report. Anthropic. https://w.anthropic.com/claude-haiku-4-5-system-card [2] Anthropic. 2026. Claude Opus 4.6 System Card. Technical Report. Anthropic. https: //w-cdn.anthropic.com/0d865075ad3132672e0ab40b05a53f14cf5288.pdf [3] Anthropic. 2026. Claude Sonnet 4.6 System Card. Technical Report. Anthropic. https://anthropic.com/claude-sonnet-4-6-system-card [4]DeepSeek. 2025.DeepSeek-V3.2 Release.DeepSeek API Documentation. https://api-docs.deepseek.com/news/news251201 Introduces DeepSeek-V3.2 and DeepSeek-V3.2-Speciale. [5] Felix TJ Dietrich, Yuchen Zhou, Tobias Wasner, Stephan Krusche, and Mari- bel Acosta. 2025. LLM-Based Multi-Artifact Consistency Verification for Pro- gramming Exercise Quality Assurance. In Proceedings of the 25th Koli Calling International Conference on Computing Education Research. 1–11. [6]Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K. Lahiri. 2022. TOGA: a neural method for test oracle generation. In Proceedings of the 44th International Conference on Software Engineering (Pittsburgh, Pennsylvania) (ICSE ’22). Association for Computing Machinery, New York, NY, USA, 2130–2141. doi:10.1145/3510003.3510141 [7] Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. 2023. Large Language Models for Software Engineer- ing: Survey and Open Problems. In Proceedings - 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering, ICSE-FoSE 2023. 31–53. doi:10.1109/ICSE-FoSE59343.2023.00008 [8] Soneya Binta Hossain and Matthew B. Dwyer. 2025. TOGLL: Correct and Strong Test Oracle Generation with LLMS. In 2025 IEEE/ACM 47th International Con- ference on Software Engineering (ICSE). 1475–1487. doi:10.1109/ICSE55347.2025. 00098 [9] Soneya Binta Hossain, Antonio Filieri, Matthew B. Dwyer, Sebastian Elbaum, and Willem Visser. 2023. Neural-Based Test Oracle Generation: A Large-Scale Evalua- tion and Lessons Learned. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (San Francisco, CA, USA) (ESEC/FSE 2023). Association for Computing Machinery, New York, NY, USA, 120–132. doi:10.1145/3611643.3616265 [10]Soneya Binta Hossain, Raygan Taylor, and Matthew Dwyer. 2025. Doc2OracLL: Investigating the Impact of Documentation on LLM-Based Test Oracle Generation. Proc. ACM Softw. Eng. 2, FSE, Article FSE084 (June 2025), 22 pages. doi:10.1145/ 3729354 [11]Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. ACM Transactions on Software Engineering and Methodology 33, 8, Article 220 (2024). doi:10.1145/ 3695988 [12]Hyeonseok Lee, Gabin An, and Shin Yoo. 2025. Metamon: Finding inconsistencies between program documentation and behavior using metamorphic LLM queries. In 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code). IEEE, 120–127. [13]OpenAI. 2024. GPT-4o System Card. Technical Report. OpenAI. https://cdn. openai.com/gpt-4o-system-card.pdf [14] OpenAI. 2025. Update to GPT-5 System Card: GPT-5.2. Technical Report. OpenAI. https://openai.com/index/gpt-5-system-card-update-gpt-5-2/ Covers GPT-5.2 family; experiments used GPT-5.2 Chat endpoint. [15]xAI. 2025. Grok 4 Fast Model Card. Technical Report. xAI.https://data.x. ai/2025-09-19-grok-4-fast-model-card.pdf Covers Grok 4 Fast reasoning and non-reasoning modes. [16] Xinye Xu, Zainab Wahab, Reid Holmes, and Caroline Lemieux. 2025. DocPrism: Local Categorization and External Filtering to Identify Relevant Code- Documentation Inconsistencies. arXiv preprint arXiv:2511.00215 (2025).