Paper deep dive
DeBERTa-Sentinel: Toward Transparent and Trustworthy Detection of AI-Generated Text
Muhammad Yousaf Rehman, Muhammad Islam
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The rapid spread of large language models (LLMs) across the web raises concerns about misinformation, academic integrity, automated content manipulation, and risks to vulnerable online communities. Existing transformer-based detectors, such as GPT-Sentinel, show promise but struggle to generalize to diverse model outputs and paraphrasing attacks, limiting their role in building trustworthy web ecosystems. This work introduces DeBERTa-Sentinel, a responsible AI-generated text detection framework leveraging DeBERTa-v3's disentangled attention to capture subtle structural irregularities in synthetic content. A central design principle is transparency: unlike black-box commercial detectors, DeBERTa-Sentinel exposes token-level explanations of its decisions, enabling affected stakeholders journalists, educators, and platform trust and safety teams to audit, challenge, and contextualize detection outcomes. Using the GLC-AIText dataset of 28,057 human and LLM-generated samples (GPT, LLaMA, and Claude) with a 60-20-20 split, DeBERTa-Sentinel achieves 98.21\% validation accuracy and surpasses the RoBERTa-Sentinel baseline from NeurIPS 2025, achieving 97.53\% test accuracy, 95.89\% precision, 99.33\% recall, and 99.53\% ROC-AUC, and maintaining a 0.665\% false negative rate. The model's interpretability reveals linguistic markers such as academic phrasing and formal transitions associated with synthetic text, directly supporting stakeholder needs for verifiable, auditable content-authenticity decisions. By advancing responsible detection methods that reduce bias and enhance explainability, DeBERTa-Sentinel promotes trustworthy, ethical, and human-centric AI systems. Code and data are available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.01046v1
- Canonical: https://arxiv.org/abs/2608.01046v1
Trouble viewing inline? Open PDF directly →
Full Text
50,468 characters extracted from source content.
Expand or collapse full text
Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (C BY 4.0). CLEAR-AI 2026: Workshop on Collaborative Methods and Tools for Engineering and Evaluating Transparency in AI, co-located with IJCAI-ECAI 2026, August 16, 2026, Bremen, Germany [1] [orcid=0000-0002-4845-4808] [1] [1] [1]Corresponding author. [1]These authors contributed equally to this work. DeBERTa-Sentinel: Toward Transparent and Trustworthy Detection of AI-Generated Text Muhammad Yousaf Rehman Muhammad Islam SPECS, University of Hertfordshire, Hatfield, UK College of Science and Engineering, James Cook University, Cairns, 4878, QLD, Australia Centre for AI and Data Science Innovation, James Cook University, Cairns, QLD 4878, Australia (2026) Abstract The rapid spread of large language models (LLMs) across the web raises concerns about misinformation, academic integrity, automated content manipulation, and risks to vulnerable online communities. Existing transformer-based detectors, such as GPT-Sentinel, show promise but struggle to generalize to diverse model outputs and paraphrasing attacks, limiting their role in building trustworthy web ecosystems. This work introduces DeBERTa-Sentinel, a responsible AI-generated text detection framework leveraging DeBERTa-v3’s disentangled attention to capture subtle structural irregularities in synthetic content. A central design principle is transparency: unlike black-box commercial detectors, DeBERTa-Sentinel exposes token-level explanations of its decisions, enabling affected stakeholders journalists, educators, and platform trust and safety teams to audit, challenge, and contextualize detection outcomes. Using the GLC-AIText dataset of 28,057 human and LLM-generated samples (GPT, LLaMA, and Claude) with a 60-20-20 split, DeBERTa-Sentinel achieves 98.21% validation accuracy and surpasses the RoBERTa-Sentinel baseline from NeurIPS 2025, achieving 97.53% test accuracy, 95.89% precision, 99.33% recall, and 99.53% ROC-AUC, and maintaining a 0.665% false negative rate. The model’s interpretability reveals linguistic markers such as academic phrasing and formal transitions associated with synthetic text, directly supporting stakeholder needs for verifiable, auditable content-authenticity decisions. By advancing responsible detection methods that reduce bias and enhance explainability, DeBERTa-Sentinel promotes trustworthy, ethical, and human-centric AI systems. Code and data are available at https://github.com/Galileo-Galili/HUMAN-VS-AI-TEXT-DETECTION. keywords: AI text detection attention models language models AI classification 1 Introduction As large language models (LLMs) like GPT [1], LLaMA [2], and Claude [3] grow increasingly capable, distinguishing AI-generated text from human writing has become a critical challenge across journalism, education, and legal domains. While early approaches relied on statistical methods like logistic regression and SVMs, the emergence of advanced LLMs necessitated the use of deep transformer-based detectors [4]. Transformer-based classifiers have likewise become the dominant approach for related web-content integrity tasks such as misinformation and fake news detection, including in low-resource language settings [5]. One such method, GPT-Sentinel [9], used a frozen RoBERTa encoder with a classification head, achieving strong results on GPT-generated text. However, recent studies have revealed key limitations in such detectors—particularly their lack of generalization to outputs from diverse models and paraphrased inputs [13, 15]. These limitations stem partly from architectural constraints inherent to RoBERTa: its attention mechanism processes content and positional information jointly within a single representation space. This entanglement means that when the model computes attention scores between tokens, it cannot separately evaluate “what is being said” (semantic content) versus “where it appears” (structural position). Consequently, RoBERTa may struggle to detect subtle positional regularities—such as the consistent placement of hedging phrases or the formulaic positioning of topic sentences—that are characteristic markers of synthetic text. Beyond raw detection accuracy, transparency has become an increasingly important consideration in AI-generated text detection, with explainable methods providing insight into the linguistic and stylistic features that influence detector predictions [18]. In practical deployments, transparent and interpretable outputs are valuable for stakeholders such as educators investigating potential academic misconduct, journalists assessing the authenticity of written content, and platform moderators enforcing content policies, as they provide supporting evidence that can be reviewed alongside human judgment rather than relying solely on opaque model predictions. This need is further reinforced by evidence that existing detectors can exhibit systematic biases against non-native English writers, highlighting the importance of decisions that can be inspected and challenged when necessary [21]. In this study, we improve upon the GPT-Sentinel framework by introducing DeBERTa-Sentinel, a classifier that replaces the RoBERTa backbone with DeBERTa-v3. DeBERTa addresses these limitations through disentangled attention and enhanced relative positional encoding [12], which we describe fully in Section 4. We also diversify the training data by incorporating outputs from multiple LLMs (GPT-3.5, LLaMA, Claude), improving robustness across generation styles. Crucially, DeBERTa-Sentinel is designed from the outset for transparency: it provides token-level attribution that allows end-users to understand and audit which textual features drive each detection decision. Our contributions are fourfold: 1. We present DeBERTa-Sentinel, a novel detection framework that incorporates disentangled attention to more effectively capture subtle structural irregularities in synthetic text, while providing built-in token-level explainability for transparent, auditable predictions. 2. We construct the GLC-AIText dataset, consisting of 28,057 paraphrased samples produced by diverse large language models (GPT-3.5, LLaMA, and Claude), with the aim of enabling broader generalization and improving model robustness. 3. Our empirical evaluation demonstrates that DeBERTa-Sentinel achieves consistent improvements over RoBERTa-Sentinel and traditional machine learning baselines, including a held-out generator experiment where the model achieves 98.46% accuracy on Claude-generated text despite being trained exclusively on GPT-3.5 and LLaMA outputs, demonstrating strong cross-generator generalization. 4. An explainability analysis shows the proposed model provides interpretable token-level predictions, identifying discriminative linguistic patterns such as transitional phrases and academic terminology commonly induced in synthetic content—directly enabling the transparency requirements of journalists, educators, and civic institutions. The rest of this paper is structured as follows: Section 2 surveys related work; Section 3 details our dataset; Section 4 outlines the model architecture and training; Section 5 presents evaluation results; Section 6 discusses transparency and responsible deployment; Section 7 provides further discussion; and Section 8 concludes the article. 2 Related Work The detection of AI-generated text has emerged as a critical area of research, particularly with the rapid proliferation of large language models (LLMs) capable of producing highly fluent and coherent text [11][4]. One of the earliest prominent contributions in this domain was GLTR [6], which employed statistical analysis of token-level probabilities and rank-based detection methods to visualize and identify machine-generated content. GLTR used probability distribution analysis and entropy calculations to detect synthetic text, but suffered from limited adaptability to evolving generation techniques and poor performance against newer language models. Subsequent works, such as Ippolito et al. [7], utilized perplexity-based detection methods and human evaluation studies, showing that automatic detectors perform best when generated text closely resembles human-authored text. However, these approaches were limited by their reliance on surface-level linguistic features and poor generalization across different domains. Similarly, Solaiman et al. [8] employed content-based filtering and release strategy frameworks but was constrained by the lack of robust detection mechanisms for sophisticated language models. A more recent advancement is GPT-Sentinel [9], which introduced a transformer-based classification framework using frozen RoBERTa encoders with a multi-layer perceptron (MLP) classification head. GPT-Sentinel demonstrated strong performance on binary classification between human and AI-generated text, particularly from GPT-3.5, but was limited by RoBERTa’s joint processing of content and positional information, which can obscure the structural patterns that distinguish synthetic text, and poor generalization to newer generative models. To overcome these architectural limitations, the present study draws on DeBERTa [12], which incorporates disentangled attention mechanisms and enhanced relative positional encoding. Through mathematical decomposition of the attention mechanism, DeBERTa independently models content-to-content, content-to-position, and position-to-content interactions, enabling more precise identification of structural artifacts in synthetic text. While existing implementations have demonstrated DeBERTa’s effectiveness across various NLP tasks, limited work has explored task-specific optimization of its disentangled architecture for AI text detection. Further comparative evaluations by Trivedi and Sivanesan [13] demonstrated ensemble methods with DeBERTa-based detectors trained on model-diverse data, achieving improved robustness. Complementary work has emphasized interpretability in detection: Shah et al. [18] combine stylistic features with explainable AI techniques to detect and unmask AI-generated text, reinforcing that interpretability is now a first-class design concern alongside raw accuracy. Recent research has explored zero-shot detection methods, with Mitchell et al. [14] introducing DetectGPT, which uses probability curvature analysis for machine-generated text detection without requiring training data. However, Krishna et al. [15] demonstrated that paraphrasing techniques can evade many detection systems, highlighting vulnerabilities in current approaches and the need for more robust detection frameworks. The challenge of reliable detection has been further emphasized by studies examining the limitations of commercial detection tools [19, 20], which demonstrated that existing detectors show bias against non-native English writers [21] and struggle with adversarial modifications to AI-generated content. Studies of commercial AI-text detection tools have shown that their performance can be unreliable in educational settings and that some detectors exhibit systematic bias against non-native English writers, raising concerns about their use in high-stakes academic assessment[19, 21]. In summary, while prior works established foundational detection approaches, they often fail to capture deeper representational differences between human and synthetic text, especially as LLMs become more advanced. By introducing DeBERTa’s disentangled attention into the classification pipeline, curating a more diverse synthetic corpus, and embedding token-level explainability as a core design feature, our study addresses these architectural, methodological, and transparency limitations. 3 Dataset This study builds upon the OpenGPTText dataset introduced by [9], which paired human-written samples from OpenWebText with paraphrased output from OpenAI’s gpt-3.5-turbo model. To improve generalization and robustness in AI-generated text detection, we extend the dataset by generating paraphrases using multiple LLMs while keeping the original human-written samples unchanged. 3.1 GLC-AIText Overview The GLC-AIText dataset consists of paraphrased textual samples that were generated using a combination of gpt-3.5-turbo, Meta’s LLaMA, and Claude language models. The original human-written content was taken from the cleaned OpenWebText corpus, specifically the OpenWebText-Final subset as used in the base paper [9], and serves as the shared source for paraphrasing across all models. Each paraphrased sample corresponds to a human-written text from the OpenWebText corpus and shares the same unique identifier (UID) to preserve alignment. The GLC-AIText dataset contains approximately 28,057 paraphrased samples, covering around 1% of the full OpenWebText corpus in specific subsets. The same subset IDs were used to collect both human-written and paraphrased samples. The number of samples in each subset is listed in Table 1. Table 1: Dataset breakdown by subset and model Subset GPT Generated LLaMA Generated Claude Generated Urlsf_00 1386 1300 1212 Urlsf_01 1308 1300 801 Urlsf_02 874 1300 922 Urlsf_03 1299 1300 1295 Urlsf_04 1156 1300 946 Urlsf_05 1103 1300 1164 Urlsf_06 1031 1300 723 Urlsf_09 1217 1300 1220 Total 9374 10400 8283 3.2 Data Source The human-written samples used in this study were obtained from the OpenWebText corpus, a publicly available dataset comprising web content sourced from URLs shared on Reddit with a minimum of three upvotes [16]. This corpus serves as a reconstitution of the original WebText dataset described by [17]. Since the OpenWebText corpus was compiled in 2019, the textual content it contains was not algorithmically generated, making it suitable as ground truth human-written text for AI detection tasks. Specifically, we utilized the cleaned OpenWebText-Final subset provided by the GPT-Sentinel dataset [9], which underwent preprocessing to remove stylistic disparities and normalize formatting inconsistencies. This cleaned version ensures that our models focus on semantic and linguistic features rather than superficial formatting artifacts that could lead to overfitting. 3.3 Data Collection Method We constructed the AI-generated portion by paraphrasing the cleaned human-written samples from the OpenWebText-Final corpus [9], covering subsets urlsf_00 to urlsf_06 and urlsf_09. Each subset was divided into three parts: one retained gpt-3.5-turbo paraphrases from the base dataset, while the other two were paraphrased using Meta’s LLaMA and Claude models. All models were prompted with the same instruction used in the original OpenGPTText collection: “Rephrase the following paragraph by paragraph.” Following the methodology of [9], samples longer than 2,000 words were filtered out due to model input limitations, and content blocked by safety filters was excluded. Outputs were additionally filtered for fluency and coherence, and near-duplicate generations were removed. Both the original human-written OpenWebText samples and the AI-generated paraphrases from all three models (GPT-3.5, LLaMA, and Claude) were used during training and testing phases to ensure comprehensive exposure to diverse writing styles and generation patterns. 3.4 Choice of Generator Models Rather than more recent frontier models such as GPT-4o or Claude 3.5, we deliberately selected GPT-3.5, LLaMA, and Claude for three reasons. First, they span architecturally distinct LLM families, a proprietary, RLHF-tuned chat model from OpenAI [23, 24], Meta’s open-weight foundation model [2], and Anthropic’s Constitutional AI model [25], so training on their outputs exposes the detector to diverse generation styles and reduces dependence on single-generator artifacts. Second, robust detectors should generalize across generators and domains: the MAGE benchmark shows that existing detectors degrade sharply on unseen generators [26], and our use of paraphrased text, which strongly evades detection [15], further stresses this setting. Third, GPT-3.5 enables a fair comparison with GPT-Sentinel, while holding out Claude (Section 5.10) tests whether the detector learns transferable rather than generator-specific representations. 3.5 Dataset Cleaning The human-written samples from OpenWebText-Final had already undergone cleaning in the base paper [9], including removal of excessive newline characters and mapping of Unicode characters to ASCII equivalents. This preprocessing was originally implemented to eliminate potential confounding factors such as ChatGPT’s tendency to use Unicode quotation marks (U+201D) instead of ASCII quotation marks (U+0022) and different newline formatting patterns. For the AI-generated paraphrases from LLaMA and Claude, we retained their raw output form with minimal post-processing, applying only basic filtering to remove incoherent or incomplete generations. Unlike the base paper’s approach, we preserved model-specific stylistic and lexical features to maintain the authentic characteristics of each language model’s output, which supports downstream tasks that benefit from retaining stylistic features for detection [18]. 4 Methodology We trained our model using the OpenWebText-Final dataset containing human-written samples and corresponding AI-generated paraphrases. Combined with the corresponding 29,142 human-written samples from OpenWebText-Final, the complete dataset comprises 58,537 total samples (28,057 AI-generated + 29,142 human-written after balancing). The dataset was partitioned using a rigorous three-way split with a fixed random seed (42): 60% for training (35,121 samples), 20% for validation (11,708 samples), and 20% for testing (11,708 samples). This configuration enables principled model selection based on validation performance while maintaining an independent test set for final evaluation. Input sequences were truncated or padded to a maximum length of 256 tokens for computational efficiency, which differs from the standard 512-token setting but was empirically validated to be sufficient for our classification task. Padding was handled with <PAD> tokens as needed, ensuring consistent sequence lengths across batches. 4.1 DeBERTa-Sentinel Model Architecture Our proposed DeBERTa-Sentinel model leverages the DeBERTa-v3-small transformer architecture, specifically chosen for its enhanced disentangled attention mechanisms that provide superior classification accuracy for text discrimination tasks. The key architectural advantage over traditional transformer models lies in DeBERTa’s separation of content and positional information during self-attention computation, enabling more precise capture of structural irregularities characteristic of AI-generated content, including unnatural sentence transitions, over-regularized phrasing patterns, and systematic vocabulary biases. The model architecture follows an end-to-end fine-tuning paradigm, as illustrated in Figure 1. Input sequences undergo tokenization using the AutoTokenizer with consistent padding/truncation to 256 tokens. The tokenized sequence is processed through 12 layers of disentangled attention in the DeBERTa encoder, where the special [CLS] token serves as a global sequence representation. This contextualized [CLS] embedding is subsequently fed into an internal classification head comprising a fully connected feedforward network that outputs binary class probabilities: P(Human)P(Human) and P(AI)P(AI). Figure 1: The diagram illustrates the DeBERTa-Sentinel architecture. The input sequence (e.g., “This is AI Generated…”) is embedded and processed through 12 layers of disentangled attention in Step 3 (our proposed approach). The final [CLS] token representation is used for classification via the internal feedforward layer. The DeBERTa-Sentinel architecture is fully end-to-end fine-tuned with gradients backpropagating through all layers of the encoder The DeBERTa-Sentinel architecture employs full end-to-end fine-tuning, allowing gradient backpropagation through all encoder layers. This approach enables the model to adapt its learned representations specifically for the AI text detection task, contrasting with frozen encoder approaches that limit representational adaptation. The fine-tuning process optimizes both the pre-trained DeBERTa weights and the classification head simultaneously, resulting in task-specific feature learning. 4.1.1 Disentangled Attention Mechanism The core innovation of our approach leverages DeBERTa’s disentangled self-attention mechanism, which explicitly decomposes attention computation into three mathematically distinct components (Figure 2): Figure 2: DeBERTa’s disentangled self-attention architecture as applied in DeBERTa-Sentinel. Standard attention (top) combines content and position into a single query/key/value operation. DeBERTa decomposes this into three independent components: (1) Content-to-Content (C2C), capturing semantic co-occurrence patterns; (2) Content-to-Position (C2P), modelling how a token’s semantic role varies with structural location; and (3) Position-to-Content (P2C), capturing how position influences meaning in context. These three scores are summed to produce the final attention output. For AI text detection, C2C identifies unnatural vocabulary patterns, C2P detects systematic structural biases (e.g., formulaic sentence-opening transitions), and P2C reveals templating behaviours characteristic of LLM outputs. Aijc2c A_ij^c2c =softmax(c(i)c(j)Td)c(j), =softmax\! ( Q_c^(i)K_c^(j)^T d )V_c^(j), (1) Aijc2p A_ij^c2p =softmax(c(i)p(δ(i,j))Td)p(δ(i,j)), =softmax\! ( Q_c^(i)K_p^(δ(i,j))^T d )V_p^(δ(i,j)), (2) Aijp2c A_ij^p2c =softmax(p(δ(j,i))c(j)Td)c(j), =softmax\! ( Q_p^(δ(j,i))K_c^(j)^T d )V_c^(j), (3) where c,c,cQ_c,K_c,V_c represent content-based query, key, and value matrices, p,p,pQ_p,K_p,V_p denote position-based projections, and δ(i,j)=clip(j−i,−k,k)δ(i,j)=clip(j-i,-k,k) defines relative positional distance. This tripartite decomposition enables sophisticated detection of AI writing patterns: content-to-content attention captures unnatural vocabulary co-occurrences and semantic inconsistencies, content-to-position attention identifies systematic positional biases in AI-generated text structure, and position-to-content attention detects templating behaviors and formulaic language patterns. These mechanisms collectively provide superior discrimination capability compared to standard attention mechanisms that conflate semantic and positional representations, making DeBERTa particularly effective for AI-generated content detection tasks. 5 Evaluation 5.1 Training Configuration and Optimization The model training process employed the AdamW optimizer with a learning rate of 2×10−52× 10^-5, batch size of 8, and cross-entropy loss function over 5 epochs. Training was conducted without weight decay regularization, relying instead on the inherent regularization provided by the pre-trained DeBERTa representations. The training process showed clear learning progression, with training accuracy improving from 96.36% in epoch 1 to 99.58% by epoch 5. However, validation accuracy peaked at 98.21% in epoch 2, with subsequent epochs showing signs of overfitting as training accuracy continued to increase while validation performance declined. The best model was selected based on validation set performance (epoch 2), demonstrating effective knowledge transfer from the pre-trained model while maintaining generalization capability. The detailed training configuration for DeBERTa can be found in Table 3. 5.2 Evaluation Metrics We evaluated both models using standard binary classification metrics, including F1 score, false positive rate (FPR), false negative rate (FNR), and area under the ROC curve (AUC). We also report detection accuracy, precision, recall, and model confidence. AI-generated text is treated as the positive class, and human-written text as the negative class. These metrics provide a holistic view of the model’s discrimination ability, threshold robustness, and reliability. 5.3 Model Selection and Hyperparameter Tuning We employed a rigorous 60/20/20 train/validation/test split (35,121 training samples, 11,708 validation samples, 11,708 test samples) using a fixed random seed (42) to ensure reproducibility. This three-way split enabled principled hyperparameter selection and model checkpoint selection based on validation performance, addressing a key methodological consideration in machine learning experiments. Our training configuration closely followed the successful approach demonstrated in the GPT-Sentinel work [9], adapting their parameters for DeBERTa-v3-small: a learning rate of 2×10−52× 10^-5, batch size of 8, and AdamW optimizer. We trained for 5 epochs, monitoring both training and validation metrics to detect overfitting. The model achieved peak validation accuracy of 98.21% at epoch 2, with training accuracy continuing to increase to 99.58% by epoch 5, indicating overfitting in later epochs. The best model (epoch 2) was selected based on validation performance and used for all subsequent testing. These values align with standard configurations reported in the DeBERTa literature for binary text classification tasks [12]. The validation-based model selection strategy ensures that reported test performance reflects true generalization capability rather than overfitting to the test set. Training curves showed the expected pattern of continued training improvement alongside validation plateau, validating our checkpoint selection methodology. 5.4 Confidence Score Distribution DeBERTa-Sentinel demonstrates tighter, more decisive confidence distributions, with a higher peak around 0.9–1.0 for correctly predicted samples. This not only implies improved model certainty but also enhances practical usability in threshold-sensitive applications (e.g., education, media verification). 5.5 Model Performance Analysis 5.5.1 ROC and AUC Performance Our DeBERTa-Sentinel model demonstrates exceptional discrimination capability with near-perfect ROC-AUC performance. Figure 3 presents comprehensive ROC curve analysis showing our model’s superior ability to distinguish between AI-generated and human-written text. Table 2: Hyperparameter configuration for DeBERTa-Sentinel model training Hyperparameter DeBERTa Epochs 5 Batch Size 8 Learning Rate 2×10−52× 10^-5 Weight Decay 0 (default) Optimizer AdamW Loss Function Cross Entropy Scheduler (not used explicitly) Dataset GLC-AIText + OpenWebText-Final Max Token Length 256 tokens Tokenizer Microsoft/deberta-v3-small Architecture DebertaV2ForSequenceClassification Fine-Tuning Full (encoder + classifier) Figure 3: ROC curve for DeBERTa-Sentinel showing exceptional discrimination capability with 99.53% AUC. The near-perfect curve hugging the top-left corner demonstrates superior ability to distinguish between AI and human text compared to random classification. The model achieves outstanding performance metrics: • ROC-AUC Score: 99.53% — indicating near-perfect discrimination ability • Average Precision Score: 99.11% — demonstrating excellent precision-recall balance • Optimal Threshold: 0.984 (Youden’s J statistic) with 98.58% TPR and 2.28% FPR • Perfect Class Separation: Probability distributions show minimal overlap between classes, with AI predictions concentrated near 1.0 and human predictions near 0.0 5.5.2 Baseline Comparison We conducted performance analysis using ROC curves and comparative evaluation against baseline models to assess DeBERTa-Sentinel’s classification effectiveness. The analysis includes standard binary classification metrics across multiple baseline approaches. Figure 4: ROC-AUC comparison across baseline models. DeBERTa-v3-small achieves 99.53% AUC (0.995), substantially outperforming traditional baselines including TF-IDF + LogReg (93.6% AUC) and random/majority class baselines (50.0% AUC). The results demonstrate the effectiveness of fine-tuned transformer models for AI text detection. Figure 5: F1-score comparison across baseline models showing DeBERTa-v3-small performance (0.976) relative to traditional ML approaches. The comparison illustrates a 12.6% improvement over TF-IDF + LogReg. Note: Majority class baseline achieves zero F1-score because it always predicts the same class, resulting in zero precision for the minority class in a balanced dataset. The evaluation results show DeBERTa-Sentinel’s performance across standard binary classification metrics. As demonstrated in Figure 5, the ROC analysis yields 99.53% AUC, while Figure 5 shows the performance comparison against baseline approaches. The detailed metrics presented in Table 4 report 97.53% accuracy with an F1-score of 0.976. Notably, the zero-shot DeBERTa-v3 baseline — the same architecture without fine-tuning — achieves only 50% accuracy by predicting every sample as AI, confirming that fine-tuning on the GLC-AIText dataset, rather than DeBERTa’s pre-trained representations alone, is responsible for the observed performance gains. 5.6 Ablation Study To isolate the contribution of each component, we evaluate two controlled configurations against our full model (Table 3). First, removing fine-tuning entirely collapses performance to 50% accuracy, confirming that DeBERTa’s pre-trained representations alone are insufficient for reliable detection. Second, the held-out generator experiment in Section 5.10 demonstrates that training on diverse multi-LLM data (GPT-3.5 + LLaMA) produces a model that generalizes to unseen generators (Claude) with 98.46% accuracy, implicitly validating the contribution of data diversity in GLC-AIText. Table 3: Ablation study results on GLC-AIText test set isolating the contribution of fine-tuning and multi-LLM data diversity. Configuration Accuracy (%) F1 DeBERTa-v3 (zero-shot, no fine-tuning) 50.00 0.667 DeBERTa-Sentinel (fine-tuned, GLC-AIText) 97.53 0.976 These findings confirm that both fine-tuning and training data diversity are essential components of DeBERTa-Sentinel’s performance, with the disentangled attention mechanism providing the architectural foundation that enables effective learning from diverse synthetic text patterns. Table 4: Performance metrics comparison across models on GLC-AIText test set (11,708 samples). Model Accuracy (%) F1-Score Precision (%) Recall (%) Random Baseline 49.06 0.491 49.15 49.05 TF-IDF + LogReg 86.33 0.867 84.64 88.82 DeBERTa-v3 (zero-shot) 50.00 0.667 50.00 100.00 DeBERTa-Sentinel (fine-tuned) 97.53 0.976 95.89 99.33 5.7 Comparison with Commercial Detection Tools To contextualize our results, we compare DeBERTa-Sentinel against commercial detection tools evaluated in prior literature. While we did not directly test these systems on our dataset, extensive research has documented their performance characteristics. Table 5: Performance comparison with commercial detectors. Commercial detector results from prior literature [9, 19] on similar datasets. The RoBERTa-Sentinel method [10] was evaluated on the GLC-AIText test set for this comparison. Model F1 Score Dataset ZeroGPT [9] 0.43 OpenGPTText-Final OpenAI Classifier [9] 0.32 OpenGPTText-Final GPTZero [19] 0.40–0.75* Various ChatGPT text RoBERTa-Sentinel (NeurIPS 2025) [10] 0.953 GLC-AIText DeBERTa-Sentinel (Ours) 0.976 GLC-AIText *Range reported across different evaluation conditions. In contrast, DeBERTa-Sentinel achieves 0.976 F1-score, substantially outperforming commercial systems. Beyond quantitative advantages, and crucially for stakeholder transparency, our approach provides: (1) token-level explainability enabling interpretation and human auditing of detection decisions; (2) model customization for domain-specific applications; (3) methodological transparency supporting reproducible research; and (4) adaptability to emerging LLMs. These features are essential for academic integrity monitoring and legal applications requiring auditable detection systems [21, 22]. Proprietary commercial detectors are also opaque, providing no insight into why a given text is flagged—a serious limitation in high-stakes settings where affected individuals need to understand and potentially challenge automated decisions. While direct comparison is complicated by dataset differences, our results align with broader findings that fine-tuned transformer models significantly outperform proprietary black-box detection systems [14, 15]. 5.8 Explainability Analysis To better understand the decision-making process of our models, we applied attention-based explainability techniques using the transformers-interpret pipeline on both the DeBERTa-Sentinel and RoBERTa-Sentinel models. This analysis reveals which textual features most strongly influence classification decisions, providing valuable insights into the models’ detection mechanisms. We treat AI-generated text as the positive class and human-written text as the negative class, consistent with our binary classification framework. The explainability analysis highlights token-level contributions, where words pushing predictions toward the AI class are identified with positive influence scores, while those favoring human classification receive negative scores. Figure 7 presents the analysis of feature importance aggregated across multiple text samples, revealing the linguistic patterns that DeBERTa-Sentinel has learned to distinguish AI from human text. The model demonstrates sophisticated pattern recognition, identifying formal vocabulary (“system”, “background”), structured language elements (“situation”, “actions”), contextual markers (“whether”, “White”, “Lemay”), and procedural language (“fixed”, “small”, “cases”) as key indicators of AI-generated content. Notably, high-importance terms reflect the model’s attention to formal discourse patterns and structured narrative elements characteristic of LLM outputs. Table 6 presents representative examples of token-level influence patterns observed in our analysis, while Figure 7 provides a concrete visualization of how these patterns appear in actual text samples. Figure 6: Top 20 influential words averaged across samples, illustrating DeBERTa-Sentinel’s learned linguistic patterns. High-importance terms such as “system”, “background”, and “situation” reflect formal vocabulary and contextual structures associated with AI-generated content. Figure 7: Token-level explainability visualization showing words contributing to AI-text classification. Highlighted regions demonstrate how DeBERTa-Sentinel captures formal transitional phrases and structured language patterns. Table 6: Token-level explainability analysis examples. Influence scores range from -1.0 to +1.0, where positive values (+) indicate features that bias the model toward AI classification, and negative values (-) indicate features that bias toward human classification. Scores closer to ± 1.0 represent a stronger influence on the final prediction. Token Influence Score Classification Bias Human-written Sample researchers +0.02 AI have -0.06 Human observed -0.11 Human anomaly +0.04 AI in -0.01 Human patterns -0.08 Human AI-generated Sample In -0.01 Human conclusion +0.12 AI this -0.03 Human demonstrates +0.14 AI the -0.02 Human model’s +0.09 AI Table 7: Performance comparison summary using GLC-AIText and OpenWebText-Final datasets. DeBERTa-Sentinel results from best validation checkpoint (epoch 2, val_acc=98.21%). Metric RoBERTa-Sentinel DeBERTa-Sentinel Improvement Detection Accuracy 95.3% 97.53% +2.23p Precision (AI class) 94.5% 95.89% +1.39p Recall (AI class) 96.3% 99.33% +3.03p F1-Score 95.3% 97.58% +2.28p False Negative Rate 3.7% 0.67% -3.03p False Positive Rate 5.7% 4.28% -1.42p ROC-AUC – 99.53% – The comprehensive explainability analysis reveals several key insights into DeBERTa-Sentinel’s decision-making process: • Formal Language Detection: As shown in Figure 7, the model prioritizes formal vocabulary and structured language patterns, with words like “background” and “system” showing high importance scores • Contextual Understanding: Figure 7 demonstrates that the model considers context rather than isolated words, highlighting transitional phrases like “Although” in formal academic discourse • Stylistic Pattern Recognition: The model successfully identifies characteristic LLM patterns such as formal conclusions (“demonstrates”, “conclusion”) and academic terminology that distinguish AI from human writing • Balanced Analysis: The model uses both positive and negative feature contributions, avoiding over-reliance on single indicators and ensuring robust classification decisions These findings align with known characteristics of LLM outputs, which often exhibit more formal, structured language patterns compared to natural human writing [18]. The explainability analysis validates that DeBERTa-Sentinel has learned meaningful linguistic distinctions rather than superficial artifacts, contributing to its robust performance and generalization capabilities. 5.9 Summary of Gains The superior performance of DeBERTa-Sentinel validates our hypothesis that incorporating positional disentanglement significantly enhances AI-text detection, especially in adversarial or distribution-shifted scenarios. The substantial reduction in false negative rate (from 3.7% to 0.67%) is of particular importance for transparency: fewer missed AI-generated texts mean that human reviewers can place greater confidence in “human” verdicts returned by the system, reducing the risk of over-burdening review workflows with false flags. 5.10 Generalization to Unseen Generators To assess whether DeBERTa-Sentinel generalizes beyond its training distribution, we conducted a held-out generator evaluation. The model was trained and validated exclusively on GPT-3.5 and LLaMA generated samples, with Claude-generated text entirely withheld from training. The test set consisted of 11,658 samples — Claude AI text paired with human-written text from OpenWebText-Final — providing a strict unseen-generator evaluation scenario that simulates real-world deployment against novel LLMs. Table 8: Held-out generator evaluation: DeBERTa-Sentinel trained on GPT-3.5 + LLaMA only, tested on Claude-generated AI samples paired with human text (11,658 samples). The model was never exposed to Claude output during training. Metric DeBERTa-Sentinel Test Accuracy 98.46% Precision (AI) 97.02% Recall (AI) 100.00% F1-Score 0.985 False Negative Rate 0.000% False Positive Rate 3.071% Test Samples 11,658 As shown in Table 8, DeBERTa-Sentinel achieves 98.46% accuracy and a perfect recall of 100% on Claude-generated text, with a false negative rate of 0%. This means the model correctly identified every AI-generated sample despite never encountering Claude output during training. The 3.07% false positive rate indicates that a small proportion of human-written text is misclassified as AI, consistent with the model’s tendency to flag formal writing structures. These results suggest that the disentangled attention mechanism captures generalizable structural features of synthetic text rather than generator-specific artifacts, supporting the robustness of DeBERTa-Sentinel under real-world distribution shift. 6 Transparency and Responsible Deployment Beyond detection accuracy, transparency is an important consideration for the practical deployment of AI-generated text detectors, particularly in settings where model predictions may influence consequential human decisions. This consideration is further reinforced by evidence that existing detectors can exhibit systematic bias against non-native English writers [21]. In this context, DeBERTa-Sentinel is designed to complement its predictions with token-level attribution scores that provide insight into the textual evidence contributing to each classification. 6.1 Stakeholder Needs and Transparency Requirements Educators and Academic Institutions. AI-generated text detectors are increasingly considered for educational settings, where automated predictions may influence investigations of academic integrity. Rather than providing only a binary prediction, DeBERTa-Sentinel highlights the tokens and linguistic patterns that contribute most strongly to its decisions (Figure 7). These explanations provide additional evidence that educators may use alongside their own judgment when reviewing flagged submissions. Journalists and Fact-Checkers. For users verifying the authenticity of written content, token-level attribution provides insight into which portions of a document most strongly influence the detector’s prediction. These explanations are intended to support human assessment rather than replace editorial judgment. Platform Trust-and-Safety Teams. For large-scale content moderation, both detection performance and interpretability are desirable. The proposed model achieves an 81.9% reduction in false negatives while also exposing the textual evidence underlying each prediction, enabling practitioners to better inspect model behaviour on their own content. Researchers. The proposed approach emphasizes reproducibility by providing mathematical formulation (Equations 1–3), implementation details, and experimental evaluation that can be independently verified and extended in future work. 6.2 Limitations and Responsible Use DeBERTa-Sentinel was trained using paraphrased web text and therefore inherits limitations associated with this training distribution. In particular, highly formal human-written prose may occasionally receive elevated AI likelihood scores. Token-level attribution allows users to inspect the evidence underlying these predictions, although it should not be interpreted as proof that a document was AI-generated. Consistent with prior work highlighting the limitations of existing AI-generated text detectors in educational settings [19], we do not recommend using detector outputs as the sole basis for consequential decisions such as academic misconduct findings. Instead, DeBERTa-Sentinel is intended as a decision-support tool that provides both a prediction and interpretable evidence to assist human reviewers during the evaluation process. 7 Discussion Our findings show that DeBERTa-Sentinel offers meaningful advancements over RoBERTa-Sentinel in terms of robustness, precision, and interpretability. These enhancements stem from the DeBERTa model’s ability to separately encode content and position representations, making it more adept at capturing the subtle irregularities often present in AI-generated text. These empirical results reinforce DeBERTa as a promising direction for future research in generative AI forensics. The explainability analysis reveals interesting patterns in how the model distinguishes between human and AI-generated text. Formal transitional phrases and academic terminology consistently bias the model toward AI classification, while casual language patterns favor human classification. This suggests that current LLMs tend to produce more formal, structured output compared to natural human writing—a finding consistent with prior work using stylistic features for interpretable detection [18]. 8 Conclusion and Future Work In this work, we introduced DeBERTa-Sentinel, an enhanced AI-generated text detection system that leverages disentangled attention mechanisms to improve upon existing approaches while placing transparency and stakeholder accountability at the center of its design. Through comprehensive evaluation across multiple datasets and model architectures, we demonstrated consistent improvements in detection accuracy, precision, and recall. The model’s enhanced generalization capabilities, particularly on distribution-shifted data, highlight the importance of architectural innovations in addressing the evolving challenge of AI-generated content detection. Our key findings include: • DeBERTa-Sentinel achieves 97.53% test accuracy (98.21% validation accuracy at optimal epoch), outperforming RoBERTa-Sentinel’s 95.3% baseline by 2.23 percentage points • The model demonstrates superior robustness with 81.9% reduction in false negative rate (0.67% vs 3.7%), critical for high-sensitivity AI detection applications • A held-out generator experiment confirms cross-generator generalization: trained on GPT-3.5 and LLaMA only, the model achieves 98.46% accuracy and 100% recall on unseen Claude-generated text, with a false negative rate of 0% • Exceptional discrimination capability with 99.53% ROC-AUC and 99.11% average precision • Token-level explainability reveals discriminative patterns in AI-generated text, with formal vocabulary and structured language serving as key indicators—directly supporting the transparency needs of journalists, educators, and platform operators • Rigorous validation-based model selection prevents overfitting, as evidenced by the gap between epoch 5 training accuracy (99.58%) and selected epoch 2 validation accuracy (98.21%) Future work should explore the application of these techniques to multilingual settings and investigate robustness against more sophisticated adversarial attacks on AI-generated text [15]. Extending the dataset to include newer frontier models (e.g., GPT-4o, Claude 3.5) would test whether the structural signatures learned from older model families generalize to increasingly human-like outputs. Additionally, user studies with journalists and educators would quantify the practical utility of token-level explanations for real-world decision-making, and domain-specific detection models (e.g., for academic writing or social media) could further refine the framework for targeted applications. References [1] OpenAI, “GPT-3.5,” 2023. [Online]. Available: https://openai.com [2] H. Touvron et al., “LLaMA: Open and Efficient Foundation Language Models,” arXiv:2302.13971, 2023. [3] Anthropic, “Claude: A Large Language Model,” 2023. Available: https://w.anthropic.com [4] J. Wu, S. Yang, R. Zhan, Y. Yuan, L. S. Chao, and D. F. Wong, “A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions,” Computational Linguistics, p. 1–65, 2025. [5] M. Islam, “Unified Large Language Models for Misinformation Detection in Low-Resource Linguistic Settings,” arXiv:2506.01587, 2025. [6] S. Gehrmann, H. Strobelt, and A. Rush, “GLTR: Statistical Detection and Visualization of Generated Text,” in Proc. ACL System Demonstrations, 2019, p. 111–116. [7] D. Ippolito, D. Duckworth, C. Callison-Burch, and D. Eck, “Automatic Detection of Generated Text is Easiest when Humans are Fooled,” in Proc. ACL, 2020, p. 1808–1822. [8] I. Solaiman et al., “Release Strategies and the Social Impacts of Language Models,” arXiv:1908.09203, 2019. [9] Y. Chen, H. Kang, V. Zhai, L. Li, R. Singh, and B. Raj, “GPT-Sentinel: Distinguishing Human and ChatGPT Generated Content,” arXiv:2305.07969, 2023. [10] P. D. Joshi, S. Pocker, R. A. Dandekar, R. Dandekar, and S. Panat, “HULLMI: Human vs LLM Identification with Explainability,” in Advances in Neural Information Processing Systems (NeurIPS), 2025. Preprint: arXiv:2409.04808. [11] M. Islam, T. Huang, E. Ahn, and U. Naseem, “Multimodal generative AI for human motion understanding and generation: A survey and way forward,”Information Fusion, 2026. [12] P. He, X. Liu, J. Gao, and W. Chen, “DeBERTa: Decoding-Enhanced BERT with Disentangled Attention,” in ICLR, 2021. [13] A. Trivedi and S. Sivanesan, “Sarang at DEFACTIFY 4.0: Detecting AI-Generated Text Using Noised Data and an Ensemble of DeBERTa Models,” in De-Factify 4.0 Workshop at AAAI, 2025. [14] E. Mitchell, Y. Lee, A. Khazatsky, C. D. Manning, and C. Finn, “DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature,” in ICML, 2023, p. 24950–24962. [15] K. Krishna, Y. Song, M. Karpinska, J. Wieting, and M. Iyyer, “Paraphrasing Evades Detectors of AI-Generated Text, but Retrieval is an Effective Defense,” in NeurIPS, vol. 36, 2023. [16] A. Gokaslan and V. Cohen, “OpenWebText Corpus,” 2019. [Online]. Available: https://skylion007.github.io/OpenWebTextCorpus/ [17] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language Models are Unsupervised Multitask Learners,” OpenAI Technical Report, 2019. [18] A. Shah et al., “Detecting and Unmasking AI-Generated Texts through Explainable Artificial Intelligence using Stylistic Features,” IJACSA, vol. 14, no. 10, p. 968–976, 2023. [19] L. Weber-Wulff et al., “Testing of Detection Tools for AI-Generated Text,” Int. J. Educational Integrity, vol. 19, no. 1, p. 26, 2023. [20] V. S. Sadasivan, A. Kumar, S. Balasubramanian, W. Wang, and S. Feizi, “Can AI-Generated Text be Reliably Detected?” arXiv:2303.11156, 2023. [21] W. Liang, M. Yuksekgonul, Y. Mao, E. Wu, and J. Zou, “GPT Detectors are Biased Against Non-Native English Writers,” Patterns, vol. 4, no. 7, p. 100779, 2023. [22] X. Yang, Y. Cheng, W. Pang, and Y. Chen, “A Survey on Detection of LLMs-Generated Content,” arXiv:2310.15654, 2023. [23] T. B. Brown et al., “Language Models are Few-Shot Learners,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, p. 1877–1901, 2020. [24] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., “Training Language Models to Follow Instructions with Human Feedback,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022. Preprint: arXiv:2203.02155. [25] Y. Bai et al., “Constitutional AI: Harmlessness from AI Feedback,” arXiv:2212.08073, 2022. [26] Y. Li, Q. Li, L. Cui, W. Bi, Z. Wang, L. Wang, L. Yang, S. Shi, and Y. Zhang, “MAGE: Machine-Generated Text Detection in the Wild,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), p. 36–53, 2024.