Paper deep dive
Interpretable Predictability-Based AI Text Detection: A Replication Study
Adam Skurla, Dominik Macko, Jakub Simko
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/22/2026, 5:17:26 AM
Summary
This paper presents a replication and extension study of the AuTexTification 2023 system for machine-generated text detection and authorship attribution. The authors identify reproducibility challenges stemming from data splits, model availability, and implementation ambiguities. They propose a unified multilingual configuration using newer generative models (Qwen, mGPT) and mDeBERTa-v3-base, demonstrating that additional stylometric features and multilingual models improve performance and interpretability via SHAP analysis.
Entities (5)
Relation Signals (3)
mDeBERTa-v3-base → usedfor → Contextual representations
confidence 100% · For contextual representations, we used mDeBERTa-v3-base
SHAP → analyzes → Stylometric features
confidence 95% · We also extend the set of stylometric features and evaluate their contribution to both performance and interpretability through feature importance analysis.
AuTexTification 2023 → evaluatedby → Przybyla et al. (2023) system
confidence 90% · One such approach of Przybyla et al. (2023) was introduced in the AuTexTification 2023 shared task
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This paper replicates and extends the system used in the AuTexTification 2023 shared task for authorship attribution of machine-generated texts. First, we tried to reproduce the original results. Exact replication was not possible because of differences in data splits, model availability, and implementation details. Next, we tested newer multilingual language models and added 26 document-level stylometric features. We also applied SHAP analysis to examine which features influence the model's decisions. We replaced the original GPT-2 models with newer generative models such as Qwen and mGPT for computing probabilistic features. For contextual representations, we used mDeBERTa-v3-base and applied the same configuration to both English and Spanish. This allowed us to use one shared configuration for Subtask 1 and Subtask 2. Our experiments show that the additional stylometric features improve performance in both tasks and both languages. The multilingual configuration achieves the results that are comparable to or better than language-specific models. The study also shows that clear documentation is important for reliable replication and fair comparison of systems.
Tags
Links
- Source: https://arxiv.org/abs/2603.15034v1
- Canonical: https://arxiv.org/abs/2603.15034v1
Trouble viewing inline? Open PDF directly →
Full Text
66,097 characters extracted from source content.
Expand or collapse full text
Interpretable Predictability-Based AI Text Detection: A Replication Study Adam Skurla 1,2 and Dominik Macko 2 and Jakub Simko 2 1 Faculty of Information Technology, Brno University of Technology, Brno, Czechia 2 Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia adam.skurla, dominik.macko, jakub.simko@kinit.sk Abstract This paper replicates and extends the system used in the AuTexTification 2023 shared task for authorship attribution of machine-generated texts. First, we tried to reproduce the origi- nal results. Exact replication was not possi- ble because of differences in data splits, model availability, and implementation details. Next, we tested newer multilingual language mod- els and added 26 document-level stylometric features. We also applied SHAP analysis to examine which features influence the model’s decisions. We replaced the original GPT-2 models with newer generative models such as Qwen and mGPT for computing probabilis- tic features. For contextual representations, we used mDeBERTa-v3-base and applied the same configuration to both English and Span- ish. This allowed us to use one shared con- figuration for Subtask 1 and Subtask 2. Our experiments show that the additional stylomet- ric features improve performance in both tasks and both languages. The multilingual configu- ration achieves the results that are comparable to or better than language-specific models. The study also shows that clear documentation is important for reliable replication and fair com- parison of systems. 1 Introduction The rapid development of large language models (LLMs), such as ChatGPT (OpenAI, 2023) (and its variants GPT-3.5, GPT-4, and GPT-5), PaLM (Chowdhery et al., 2023), Claude (Bai et al., 2022), and LLaMA (Grattafiori et al., 2024), has signifi- cantly impacted the field of text generation. These models are capable of producing fluent, stylisti- cally convincing texts across multiple languages and domains at remarkable speed. Their outputs are often indistinguishable from human-written texts at first glance. As AI-generated texts become more prevalent in sectors like education and journalism, it raises important concerns, emphasizing the need for reliable automated tools to detect and attribute machine-generated texts. As the quality of generated texts improves, re- search has evolved from simple binary detection (human vs. AI) to the more complex task of model attribution. The goal is no longer just to determine whether a text was generated, but also to identify the specific model responsible. This task is more challenging because modern language models of- ten generate outputs that are strikingly similar, with differences typically manifesting in subtle stylistic, lexical, or probabilistic features. At the same time, many recent approaches rely on fine-tuned neural models that behave largely as black boxes, which motivates the use of more interpretable methods for attribution (Luo and Specia, 2024). In response to these challenges, several ap- proaches have been proposed. One such approach of Przybyla et al. (2023) was introduced in the Au- TexTification 2023 shared task (Sarvazyan et al., 2023), where a hybrid system combined predictabil- ity features, derived from generative language mod- els, with features from a fine-tuned language model (FLM) and additional linguistic features such as word frequency and grammatical correctness. This approach achieved competitive results in both bi- nary detection and model attribution. The main goal of this paper is to replicate and extend the original hybrid approach from Au- TexTification 2023. We aim to faithfully reproduce the published experiments, including the configu- ration, feature set and training setup. Additionally, we investigate the impact of more recent multi- lingual language models on system performance and introduce an extended set of document-level stylometric features. We address the following research questions: •RQ1: To what extent can the results of the original AuTexTification 2023 system (Przy- byla et al., 2023) be reproduced? 1 arXiv:2603.15034v1 [cs.CL] 16 Mar 2026 •RQ2: How does the choice of the base lan- guage model influence the performance of embedding- and predictability-based features? •RQ3: To what extent do the newly intro- duced stylometric features improve classifi- cation performance and the interpretability of the model’s decisions? Unlike the original work of Przybyla et al. (2023), our focus is not solely on achieving the highest possible performance. We also analyze the factors influencing the stability and transparency of the results. Furthermore, we propose a unified configuration based on recent multilingual mod- els, which enables a shared configuration across languages without the need for language-specific adjustments. We also extend the set of stylomet- ric features and evaluate their contribution to both performance and interpretability through feature importance analysis. Our main contributions are: • A systematic replication of the AuTexTifica- tion 2023 system, identifying several factors that affect reproducibility of the reported re- sults. •A unified multilingual configuration based on recent language models that achieves com- parable or higher average performance with- out language-specific components. • An extension of the stylometric feature set together with an interpretability analysis using SHAP, providing insights into which stylistic signals contribute to model decisions. To ensure reproducibility, the implementation code is publicly available. 1 2 Related Work Early research in detection of machine-generated texts predominantly focused on binary detection, i.e., distinguishing between human- and machine- authored texts.One of the pioneering works (Zellers et al., 2019) presented Grover, a model designed not only to generate but also to detect text. Another milestone was the GLTR system (Gehrmann et al., 2019), which leveraged token probability visualizations to expose the higher pre- dictability of machine-generated texts. The results 1 https://anonymous.4open.science/r/ autextification-replication-E09F/ suggest that generated texts are easier to detect be- cause they are more predictable than human texts. Building on these foundations, subsequent work has explored probabilistic methods.Detect- GPT (Mitchell et al., 2023) introduced an ap- proach based on analyzing the curvature of a lan- guage model’s probability surface, assuming that machine-generated texts tend to lie in regions of low curvature. Fast-DetectGPT (Bao et al., 2023) proposed a more efficient and computationally less demanding variant. These methods demonstrated that probabilistic features can be useful for binary detection tasks, although they remain constrained by their reliance on reference models and language- specific properties. With the growing diversity of LLMs, the fo- cus has shifted from binary detection (Spiegel and Macko, 2024; Hans et al., 2024; Su et al., 2023) to multi-way attribution (Mikros et al., 2023; Ale- cakir et al., 2024; Wang et al., 2024b; Soto et al., 2024). The task is no longer only to detect machine- generated text, but also to identify which model created it. This is harder because modern models produce very similar and fluent texts. The differ- ences are often small and appear in style or token probabilities. Several approaches have been proposed to ad- dress this challenge. One approach is the use of stylometry (Przybyla et al., 2023; Alecakir et al., 2024; Schaaff et al., 2023), where lexical and syn- tactic features of the text are analyzed. These meth- ods are interpretable, but they often require high computational resources and do not work equally well across languages. Other approaches employ embedding-based (Wang et al., 2024a; Kuznetsov et al., 2024; Kadhim et al., 2025) and contrastive methods (Guo et al., 2024; Zhang et al., 2024), leveraging representations from transformer mod- els combined with contrastive learning to amplify distinctions between models. Recent research on multilingual AI-text detec- tion has moved beyond English-centric approaches, increasingly using cross-lingual transformer ar- chitectures such as mBERT (Pires et al., 2019), XLM-RoBERTa (Conneau et al., 2019), and mGPT (Shliazhko et al., 2024). Multilingual evaluations (Macko et al., 2023, 2025; Cava et al., 2025) indi- cate that fine-tuned multilingual transformer mod- els outperform statistical and monolingual methods, achieving stronger cross-lingual generalization and highlighting the benefits of shared semantic repre- sentations for multilingual detection. 2 Tasks of machine-generated text detection and at- tribution increasingly appear in the form of shared tasks, such as SemEval (Wang et al., 2024c), AuTexTification (Chiruzzo et al., 2024), or Ru- ATD (Shamardina et al., 2022). Within AuTexTifi- cation 2023, the system by Przybyla et al. (2023) in- troduced an architecture that combined transformer- based embeddings, probabilistic features derived from language models, and traditional linguistic statistics. This approach improved performance, but it had some limitations. It used a simple feature combination, relied on older models, and covered only a few languages. In our work, we therefore use this architecture as a baseline and propose an extended approach that directly addresses these shortcomings. 3 Methodology Our study addresses three research questions (see Section 1) focusing on the replication of the orig- inal system, the impact of base language model selection, and the extension of the stylometric fea- ture set. To provide the answers, we first reproduce the original experimental setup using the publicly available code 2 (the original and extended system configurations are overviewed in Table 1). We then keep the configuration unchanged and replace the language models used to compute probabilistic fea- tures. Finally, we extend the stylometric feature set with additional indicators and evaluate their im- pact on model performance and interpretability. A schematic overview of the architecture and feature extraction pipeline is provided in Figure 1. Dataset. We conduct our experiments on the AuTexTification 2023 dataset 3 (Sarvazyan et al., 2023), following the original study. The shared task was organized in two languages, English and Spanish, and consisted of two subtasks. The first subtask is a binary classification task (Human vs. AI), where the goal is to determine whether a text was written by a human or generated by a language model. The second subtask focuses on model at- tribution: all texts are machine-generated, and the objective is to identify which model produced each text. During the competition, only limited infor- mation about the language models was provided. Participants were informed that six models were used, ranging from 2B to 175B parameters, and labeled A–F. Now we know that the generators are 2 https://github.com/piotrmp/autext 3 https://huggingface.co/datasets/symanto/ autextification2023 ConfigurationDescription FLMFine-tuned pre-trained encoder with [CLS] classification head. PredBiLSTM over token-level probabilistic features. HybridFLM and Pred with concatenated out- puts. Hybrid+Hybrid extended with word frequency and grammar correctness features. LingRFRandom Forest over document-level lin- guistic features. LingRF + PredOutLingRF concatenated with Pred output probabilities. UltrahybridHybrid+ output probabilities + linguistic features into RF/XGB/MLP. Hybrid_flatStatistically pooled token-level features + linguistic features into RF/XGB/MLP (no BiLSTM/FLM). Table 1: Overview of system configurations (variants). The original configurations of Przybyla et al. (2023) are provided in the upper part, the lower part contains the proposed additional configurations. BLOOM-1B1, BLOOM-3B, BLOOM-7B1, Bab- bage, Curie, and text-davinci-003 (we have not utilized this information in the experiments). The dataset covers five genres: Legal, News, Reviews, Tweets, and Wiki. The training split includes Legal, Tweets, and Wiki, while the test split consists of News and Reviews. This setup evaluates a chal- lenging cross-domain generalization, as the test domains are not seen during training. The dataset is balanced across classes and domains. In Subtask 1, the dataset includes about 16k samples for training and 10k samples for testing per each class in each language. In Subtask 2, there are about 3.6k sam- ples for training and 900 samples for testing per each class (generator) in each language. We follow the official train–test split for the shared task. (RQ1) Replication setup. In RQ1, we focus on replicating the original system, primarily based on the information provided in the paper. We im- plemented the system configuration, feature set, and training procedure according to the available methodological description. A more detailed analy- sis of the original study is provided in Appendix A. The authors also released the implementation of their best-performing system, which we used to verify implementation details and ensure consis- tency of the individual components. The objec- tive was to preserve the implementation described in the paper. During replication, we retained the originally reported hyperparameters and evaluation setup, including the official train–test split. 3 (RQ2) Base model variants. In RQ2, we keep the overall system configuration and evaluation setup unchanged and vary only the base models used in two components of the pipeline: (i) the language models used to compute probabilistic (predictability-based) features and (i) the encoder component used to extract contextual representa- tions in the FLM module. The original study relied on GPT-2 models for probabilistic features and RoBERTa-base (bertin-RoBERTa-base-spanish for Spanish) in the FLM component. In the first iter- ation of our experiments, we evaluated different model families and parameter scales to assess their suitability for both components of the system. In the subsequent phase, we focused on construct- ing a unified configuration applicable across lan- guages and subtasks, selecting three multilingual model groups for final evaluation (Table 2). For the FLM component, in addition to the original RoBERTa-base encoder (Liu et al., 2019), we eval- uated XLM-RoBERTa-base (Conneau et al., 2019) and mDeBERTa-v3-base (He et al., 2021). The ob- jective was to determine whether the same encoder and probabilistic models could be applied to both languages without language-specific adjustments. The configuration, feature set, and training protocol remain fixed across all variants. (RQ3) Extension of stylometric features. In RQ3, we build on the original set of stylometric features used in the prior study. Since the authors do not provide a complete list of extracted features but only their groups (Appendix A), we extend this set with 26 additional features divided into seven categories. The full list is provided in Appendix B. All features are computed at the document level, and frequency-based measures are normalized by the number of tokens or sentences to reduce the effect of text length. We compare the extended feature set to the original and use SHAP values to analyze feature importance. Training and implementation details. We fol- GroupModel sets XGLMxglm-564M;xglm-1.7B;Qwen2.5-1.5B; BLOOM-1B1 mGPTmGPT;LLaMA-3.2-3B;Qwen2.5-1.5B; BLOOM-1B7 Large Qwen2.5-3B;LLaMA-3.2-3B;xglm-2.9B; BLOOM-1B7 Table 2: Final probabilistic model groups evaluated in RQ2. low the official train–test split. The original train- ing set was further partitioned into training and validation subsets, where 20% of the training data were allocated for validation. The test set was kept unchanged. All splits were generated using a fixed random seed (10). The neural component is trained for up to 20 epochs with early stopping, follow- ing the setup of the original study. The optimizer, learning rate, and other training hyperparameters are kept identical to the original implementation. The Random Forest classifier uses the same pa- rameter settings as in the original work. All ex- periments are conducted on an NVIDIA A40 GPU (cumulatively consumed about 400 GPU hours). For comparability, we use the same random seed (10) as the original study. We report performance using Macro F1 as the primary evaluation metric (the official metric of the shared task). 4 RQ1: Reproduction of the Original System When attempting to reproduce the system proposed in (Przybyla et al., 2023), we obtained results that differed from those reported in the original publi- cation. Based on these differences, we conducted a detailed analysis of the official repository refer- enced in the paper. Although the repository does not include implementations of all components (e.g., the linguistic features and the LingRF model are missing), its inspection allowed us to identify several factors that affect the reproducibility of the system. These factors can be grouped into three categories: (i) differences between the methodolog- ical description in the paper and the available im- plementation, (i) external dependencies and model availability, and (i) ambiguities in the implemen- tation of linguistic features. Differences between the paper and the imple- mentation. The first group concerns differences be- tween the experimental setup described in the paper and the publicly available implementation. While the paper reports a random 80/20 train–validation split, the available code uses a predefined topic- based split (fold-0 from the LDA split). This dif- ference may result in a validation set with differ- ent properties and can therefore influence model selection. The paper also describes an early stop- ping strategy based on selecting the first epoch that reachesf max − 0.01. In the analyzed version of the code, however, this mechanism is not explicitly implemented; the model is trained for a fixed 20 4 epochs and predictions are stored for each epoch. It is therefore not fully clear how the final model was selected in the original experiments. We also ac- knowledge that the publicly available implementa- tion may not exactly correspond to the final version used for the shared-task submission. Model availability for Spanish. An impor- tant limitation of our reproduction concerns the availability of the language models used for Span- ish. For extracting probabilistic features, the au- thors relied onPlanTL-GOB-ES/gpt2-base-bne andPlanTL-GOB-ES/gpt2-large-bne.At the time of our experiments (11/2025), these mod- els were no longer available in their origi- nal form, which prevented their direct use. In our baseline reproduction, we therefore re- placed them with the closest available alter- natives, namelyDeepESP/gpt2-spanishand datificate/gpt2-small-spanish. Since these are different models with different training data and vocabularies, identical results cannot be ex- pected, which may partly explain the differences observed for Spanish. Ambiguities in feature extraction. The third group of factors relates to uncertainties in feature extraction. For grammatical features, the imple- mentation compares tokens within a context win- dow of±5positions, which is not specified in the paper and slightly alters the feature definition. Word-frequency features also depend on the tok- enization and its alignment with the external fre- quency resource. For document-level aggregated linguistic features, the authors did not provide an implementation or an exact list of features (only their types are described), so we relied on the in- formation available in the paper. The resulting feature representation depends on the version of the spaCy library and the aggregation of linguis- tic categories. The modelsen_core_web_smand es_core_news_smmay produce annotations that differ from those at the time of publication, af- fecting the number of extracted features. We also observed small output differences between ver- sions 4.X and 5.X of the Hugging Face Transform- ers library, likely due to internal implementation changes. Table 3 compares the number of features re- ported in the original paper with the number ob- tained in our reproduction for the LingRF and Lin- gRF+PredOut models. The number of features in the English configurations is either identical or very close to the values reported in the original paper. Subtask Lang VariantOur Orig S1 en LingRF163165 LingRF+PredOut 165165 es LingRF311299 LingRF+PredOut 313299 S2 en LingRF163163 LingRF+PredOut 169163 es LingRF307288 LingRF+PredOut 313288 Table 3: Comparison of the number of linguistic features between our reproduction (Our) and the original paper (Orig) for the LingRF and LingRF+PredOut models. Larger differences appear for Spanish, where our implementation produces a higher number of lin- guistic features. This may be related to the richer morphological structure of Spanish, as well as to differences in the specific spaCy version and the aggregation of linguistic categories. Despite the differences discussed above, we were able to reproduce the core experimental pipeline. In addition, we successfully ran the repos- itory officially referenced by the authors, though we had to introduce several minor adjustments, in- cluding replacing unavailable Spanish language models and adding early stopping according to the criterionf max − 0.01. Table 4 presents three sets of results: the scores reported in the original publi- cation (Paper), the results obtained by running the publicly available code (GitHub), and the results of our reproduction (Ours). We observe strong performance of the model based solely on predictability-based features (Pred), particularly in the binary detection setting (S1), where it achieves the highest score among all vari- ants. In this setting, adding additional features to the Pred model slightly decreases performance compared to the standalone variant, suggesting that predictability-based features alone already capture most of the relevant signal. An exact numerical reproduction of the origi- nally reported results was not achieved. Across settings where comparison with the paper was pos- sible, the differences between the reported and our results range from 0.008 to 0.070 F1 points, with the largest gap observed for S1 EN (Hybrid). These deviations indicate that even relatively small dif- ferences in data splits, early stopping procedures, model availability, or feature extraction details can lead to measurable changes in performance. Our reproduced results are therefore used as the base- 5 Subtask Lang Method Paper GitHub Ours S1en FLMn/a0.584 0.564 Predn/a0.850 0.887 Hybrid0.7400.603 0.810 Hybrid+ 0.8100.699 0.796 S1es FLMn/a0.550 0.551 Predn/a0.740 0.764 Hybrid0.6800.659 0.717 Hybrid+ 0.7100.655 0.743 S2en FLMn/a0.562 0.573 Predn/a0.441 0.478 Hybrid0.5760.558 0.584 Hybrid+ 0.5700.544 0.586 S2es FLMn/a0.559 0.583 Predn/a0.340 0.421 Hybrid0.6200.534 0.592 Hybrid+ 0.6100.553 0.567 Table 4: Comparison of results reported in the orig- inal paper (Paper), obtained by running the official GitHub code (GitHub), and produced by our reproduc- tion (Ours). line reference in RQ2 and RQ3. Our findings underline the importance of making the final experimental code fully available and ensuring consistency between the published methodologi- cal description and the released implementation in order to support reliable reproducibility. 5RQ2: Impact of Base Language Models In this section, we analyze how the choice of base language models affects system performance. The system configuration, training procedure, and fea- ture set remain unchanged; we only vary the gen- erative models used to compute probabilistic fea- tures and the encoder in the FLM component. Our goal is to verify whether it is possible to create a unified multilingual configuration that can be used for both languages and both subtasks without language-specific modifications, and at the same time compare the performance of different model groups. We report results for the two strongest con- figurations from the original work, namely Hybrid and Hybrid+, as defined in Table 1. Impact of the encoder in the FLM component. The goal was to determine whether newer multilin- gual encoders provide measurable improvements over the original configuration. As mentioned in Section 3, we have focused on multilingual models for comparison with the original RoBERTa-base encoder. Table 5 summarizes the results across EncoderConfiguration S1-en S1-es S2-en S2-esAvg XLM-R Hybrid0.793 0.532 0.568 0.588 0.620 Hybrid+0.828 0.612 0.479 0.594 0.628 mDeBERTa-v3 Hybrid0.695 0.593 0.603 0.613 0.626 Hybrid+0.722 0.724 0.611 0.627 0.671 Baseline Hybrid0.810 0.717 0.584 0.592 0.676 Hybrid+0.796 0.743 0.586 0.567 0.673 Table 5: Comparison of FLM encoders in Hybrid and Hybrid+ configurations. Avg = mean across S1/S2 and EN/ES. all languages and subtasks, together with the av- erage score for each encoder. We can see that the language-specific baseline (the replicated system from RQ1) achieves the highest average score in the Hybrid configuration and a very similar result in Hybrid+. The mDeBERTa-v3 model in the Hy- brid+ configuration is only slightly below the base- line on average, and it represents a single shared multilingual encoder for both languages. XLM-R achieves similar averages in Hybrid and Hybrid+, but overall it performs worse than both the baseline and mDeBERTa-v3. Since the goal of RQ2 is to propose a unified configuration without language- specific modifications, we selected mDeBERTa-v3 as the shared encoder for both languages in the following experiments. Impact of generative models on probabilis- tic features. In the next step, we analyzed how the choice of generative models used to compute predictability-based features affects the overall sys- tem performance. As in the previous experiment, we evaluated only the Hybrid and Hybrid+ config- urations. The difference was that we now used the selected multilingual encoder mDeBERTa-v3. The experiment was conducted in several iterations; in this section, we report the final setting in which we compare three multilingual model groups (see Section 3), denoted as XGLM, mGPT, and Large. The goal was to determine which group provides the highest and most stable performance across both languages and both subtasks. The results are shown in Table 6. The Large group achieves the highest average performance in both configurations (Hybrid and Hybrid+), and at the same time it ob- tains the best or near-best results in most individual configurations. The mGPT group shows competi- tive results, especially in the Hybrid configuration, but its performance in Hybrid+ is less consistent. The XGLM group achieves stable but overall lower scores. 6 GroupConfiguration S1-en S1-es S2-en S2-esAvg XGLM Hybrid0.713 0.663 0.595 0.621 0.648 Hybrid+0.675 0.754 0.611 0.626 0.667 mGPT Hybrid0.747 0.795 0.607 0.637 0.697 Hybrid+0.667 0.702 0.597 0.633 0.650 Large Hybrid0.796 0.798 0.625 0.650 0.717 Hybrid+0.765 0.704 0.595 0.635 0.675 Baseline Hybrid0.810 0.717 0.584 0.592 0.676 Hybrid+0.796 0.743 0.586 0.567 0.673 Table 6: Probabilistic model groups. Avg = mean across S1/S2 and EN/ES. Compared to the original language-specific baseline, the selected multilingual configuration (mDeBERTa-v3 + Large group) achieves higher av- erage performance in the Hybrid configuration and comparable or slightly better results in Hybrid+. We also evaluated its robustness across multiple random seeds and compared its time complexity with the baseline configuration (see Appendix C.2). At the same time, it provides a single solution that can be used for both languages and both subtasks without relying on language-specific models. We also explore additional architectural configurations derived from Hybrid+, including a two-stage vari- ant (Ultrahybrid) and a simplified architecture with- out sequential components (Hybrid_flat). The re- sults are reported in Appendix C.1. Overall, the re- sults suggest that a carefully chosen multilingual configuration can match or slightly outperform the language-specific baseline while offering a unified approach across languages and subtasks. 6 RQ3: Effect of Additional Stylometric Features The original study states as one of its main lim- itations that the used features do not sufficiently capture deeper stylistic and structural characteris- tics of the text. In this section, we analyze the contribution of newly introduced document-level stylometric features added to the original set of lin- guistic features. The goal is to examine whether extending the text representation improves classifi- cation performance and whether the new features provide additional information beyond the original setup. We compare two configurations. The first configuration uses the original feature set from the replicated study. The second configuration extends this set with 26 additional document-level stylo- metric features described in Appendix B. In both cases, the model configuration and training setup SubtaskLangBaselineExtended∆ S1en0.4090.467+0.058 S1es0.5000.608+0.108 S2en0.4080.478+0.071 S2es0.4380.500+0.062 Table 7: Comparison of the original and extended fea- ture sets for LingRF (macro F1 on the test set). remain unchanged in order to isolate the effect of the extended feature set. LingRF. Table 7 presents the results for the Lin- gRF model, which relies only on document-level linguistic features and does not use probabilistic outputs from the neural component. Extending the feature set improves performance across all lan- guage and subtask combinations. The extended feature set consistently improves performance across languages and subtasks. The largest relative improvement is observed for Span- ish in Subtask 1 (+21.6%), followed by English in Subtask 2 (+17.2%). Improvements of +14.2% are observed for both English in Subtask 1 and Spanish in Subtask 2. These results indicate that the added stylometric features capture informa- tion not present in the original configuration. The strongest effect appears for Spanish, possibly re- flecting language-specific stylistic patterns better captured by the document-level features. We further analyzed feature importance using SHAP (SHapley Additive exPlanations) (Lundberg and Lee, 2017) for the LingRF model with the ex- tended feature set. Several newly added document- level stylometric features (Table 9) appear among the most important features based on mean abso- lute SHAP values (see Figure 2 and 3). Higher scores are mainly associated with features related to lexical diversity, sentence structure, and func- tional markers. This suggests that the model relies on these characteristics and that their contribution is comparable to the original linguistic statistics. Selected SHAP visualizations are provided in Ap- pendix D. LingRF + PredOut. We then analyzed the sec- ond model, LingRF + PredOut, which combines document-level stylometric features with proba- bilistic features extracted using generative mod- els. Table 8 reports three configurations: (i) Base- line, which corresponds to the original probabilistic models and the original feature set, (i) Old-prob, where the original probabilistic models are used together with the extended feature set including the 7 SubtaskLangBaselineOld-probNew-prob S1en0.8780.8750.912 S1es0.7620.7890.881 S2en0.5050.5360.599 S2es0.4620.5120.562 Table 8: Comparison of feature configurations for Lin- gRF+PredOut (macro F1 on the test set). 26 new features, and (i) New-prob, which uses the probabilistic models selected in RQ2 together with the extended feature set. Table 8 shows that combining the extended fea- ture set with the new probabilistic models consis- tently yields the best performance across all set- tings. Using the extended features with the original probabilistic models (Old-prob) provides moderate gains, especially in Subtask 2. The largest improve- ments are obtained with the New-prob configura- tion, reaching +21.6% for Spanish and +18.6% for English in Subtask 2, while smaller gains are ob- served in Subtask 1 (+15.6% for Spanish and +3.9% for English). SHAP analysis further shows that, alongside probabilistic outputs (PRED_PROB_*), several of the newly introduced stylometric fea- tures appear among the most important predictors, dominating the purely stylometric model (LingRF) and complementing the probabilistic signal in the combined model. These results answer RQ3 by showing that ex- tending the stylometric feature set has a measur- able impact. Performance improves both in the standalone linguistic model and when combined with probabilistic features. SHAP analysis (see Figure 4 and 5) further indicates that several newly added features play an important role in the model’s decisions. The extended feature set therefore in- creases performance and improves interpretabil- ity by enabling clearer identification of stylis- tic features that influence the model’s predic- tions. The comparison with the state-of-the-art detector mdok (Macko, 2025) in Appendix C.3 also shows that our approach can outperform mdok in the binary subtask while remaining more inter- pretable. Future work may explore combining both approaches. 7 Discussion and Conclusion In this paper, we replicated and extended a hybrid system from the AuTexTification 2023 shared task. We focused on three points: (i) reproduction of the original results, (i) the effect of different base language models, (i) and the impact of additional stylistic features. The experiments showed practi- cal issues with reproducibility and confirmed that feature design has a strong effect on performance. Reproducibility of experiments. Even though we followed the description in the paper and used the available repository, we did not obtain exactly the same results. The differences were caused by several factors, such as different data splits, miss- ing language models, early stopping settings, and changes in external tools. This shows that small technical details can influence the final scores. To make future work more reliable, it is impor- tant to release the exact code version that was used for the final results. The training procedure, model selection rules, and preprocessing steps should be clearly described. Without this information, it is difficult to compare systems or verify improve- ments. Choice of models and feature design. The ex- periments also showed that the selection of base language models matters. When we replaced older models with newer multilingual ones, we were able to build one shared configuration for both lan- guages and both subtasks. This simplified the setup and still achieved strong results. It suggests that multilingual models can be a practical alternative to language-specific solutions. Adding new document-level stylistic features improved performance in all tested settings. The SHAP analysis confirmed that several of these fea- tures were actively used by the model. In the purely linguistic model, they were among the most im- portant predictors. In the combined model, they complemented the probabilistic features instead of replacing them. This indicates that stylistic in- formation still provides useful signals, even when strong neural models are used. Stylometric analysis therefore remains relevant for machine-generated text attribution, particularly as a complementary signal alongside probabilistic and neural features. One open question is whether a smaller set of universal features could work across tasks and languages. Such features could help iden- tify stylistic patterns shared across generators and those specific to particular models. Overall, our study shows that results in this task depend not only on model architecture but also on implementation details and feature design. Careful experimentation and transparent reporting are therefore essential for fair comparison and meaningful progress in this area. 8 Limitations Reproducibility and replacement of unavailable models. In RQ1, we focused on reproducing the original system. However, some models used in the baseline were no longer available and had to be re- placed with suitable alternatives. Other researchers might choose different replacement models or con- figurations, which could lead to different results. Limited selection of base language models. RQ2 focused on the selection of multilingual en- coders and generative model groups. We tested only the selected subset of models, which does not cover the full range of available architectures or the most recent models. Other models or configu- rations could lead to different outcomes. Limited set of linguistic features. In RQ3, we extended the original set of stylometric features, which the original paper described as its main limi- tation. However, our feature set is not exhaustive. Although, it provides a deeper analysis than the original set, there are still additional lexical, syn- tactic, or discourse-level features that could further improve performance. We did not perform a sys- tematic evaluation of all possible feature combina- tions. Limited data. We have focused on replication of a system submitted to the AuTexTification 2023 shared task. Therefore, we have limited the training and evaluation data to the official data of the shared task. It is inherently limited by domains, generators and languages included, and generalization to other data has not been tested. Comparison with shared task systems. Our comparison with the AuTexTification 2023 shared task systems should be interpreted with caution. Unlike the shared task participants, who had to submit a fixed system under strict evaluation con- ditions, our experiments in RQ2 and RQ3 allowed iterative experimentation and component tuning. In particular, we used the test set labels to evaluate dif- ferent configurations and select the best-performing variant. Such a procedure would not be possible in the shared task evaluation setup, where the test labels are not available during system development. Ethical Considerations This work focuses on the analysis of text gener- ation and detection models and does not involve personal or sensitive data. We did not use datasets containing identifiable information about individu- als, and data processing complies with applicable legislation. Generative models were not used to write this paper or to design the methodology. Acknowledgments This work was partially supported by the EU NextGenerationEU through the Recovery and Re- silience Plan for Slovakia under the project No. 09I01-03-V04-00059, partially by CEDMO 2.0, a project funded by the European Union under the GA No. 101158609, and partially by LorAI – Low Resource Artificial Intelligence, a project funded by Horizon Europe under GA No.101136646. References Huseyin Alecakir, Puja Chakraborty, Pontus Hennings- son, Matthijs Van Hofslot, and Alon Scheuer. 2024. Groningen team a at semeval-2024 task 8: Hu- man/machine authorship attribution using a combi- nation of probabilistic and linguistic features. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 1926– 1932. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, and 1 others. 2022. Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073. Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. 2023. Fast-detectgpt: Effi- cient zero-shot detection of machine-generated text via conditional probability curvature. arXiv preprint arXiv:2310.05130. Lucio La Cava, Dominik Macko, Róbert Móro, Ivan Srba, and Andrea Tagarelli. 2025. Authorship at- tribution in multilingual machine-generated texts. Preprint, arXiv:2508.01656. Luis Chiruzzo, Salud María Jiménez-Zafra, and Fran- cisco Rangel. 2024.Overview of iberlef 2024: natural language processing challenges for spanish and other iberian languages.In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2024), co-located with the 40th Conference of the Spanish Society for Natural Language Processing (SEPLN 2024), CEUR-WS. org. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, and 1 others. 2023. Palm: Scaling language modeling with pathways. Journal of Ma- chine Learning Research, 24(240):1–113. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco 9 Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. Sebastian Gehrmann, Hendrik Strobelt, and Alexan- der M Rush. 2019.Gltr: Statistical detection and visualization of generated text. arXiv preprint arXiv:1906.04043. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Xun Guo, Yongxin He, Shan Zhang, Ting Zhang, Wan- quan Feng, Haibin Huang, and Chongyang Ma. 2024. Detective: Detecting ai-generated text via multi-level contrastive learning. Advances in Neural Information Processing Systems, 37:88320–88347. Abhimanyu Hans,Avi Schwarzschild,Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2024. Spotting llms with binoculars: Zero-shot detection of machine-generated text.Preprint, arXiv:2401.12070. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre- training with gradient-disentangled embedding shar- ing. arXiv preprint arXiv:2111.09543. Ahmed K Kadhim, Lei Jiao, Rishad Shafik, and Ole- Christoffer Granmo. 2025. Adversarial attacks on ai- generated text detection models: A token probability- based approach using embeddings. arXiv preprint arXiv:2501.18998. Kristian Kuznetsov,Eduard Tulchinskii,Laida Kushnareva, German Magai, Serguei Barannikov, Sergey Nikolenko, and Irina Piontkovskaya. 2024. Robust ai-generated text detection by restricted embeddings. arXiv preprint arXiv:2410.08113. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:1907.11692. Scott M Lundberg and Su-In Lee. 2017. A unified ap- proach to interpreting model predictions. Advances in neural information processing systems, 30. Haoyan Luo and Lucia Specia. 2024. From understand- ing to utilization: A survey on explainability for large language models. arXiv preprint arXiv:2401.12874. Dominik Macko. 2025. mdok of kinit: robustly fine- tuned llm for binary and multiclass ai-generated text detection. arXiv preprint arXiv:2506.01702. Dominik Macko, Jakub Kopál, Robert Moro, and Ivan Srba. 2025. MultiSocial: Multilingual benchmark of machine-generated text detection of social-media texts. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 727–752, Vienna, Aus- tria. Association for Computational Linguistics. Dominik Macko, Robert Moro, Adaku Uchendu, Ja- son Lucas, Michiharu Yamashita, Matúš Pikuliak, Ivan Srba, Thai Le, Dongwon Lee, Jakub Simko, and Maria Bielikova. 2023. MULTITuDE: Large-scale multilingual machine-generated text detection bench- mark. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9960–9987, Singapore. Association for Com- putational Linguistics. George K Mikros, Athanasios Koursaris, Dimitrios Bil- ianos, and George Markopoulos. 2023. Ai-writing detection using an ensemble of transformers and sty- lometric features. In IberLEF@ SEPLN. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023. De- tectgpt: Zero-shot machine-generated text detection using probability curvature. In International con- ference on machine learning, pages 24950–24962. PMLR. OpenAI. 2023.Gpt-4 technical report.Preprint, arXiv:2303.08774. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert?arXiv preprint arXiv:1906.01502. Piotr Przybyla, Nicolau Duran-Silva, and Santiago Egea Gómez. 2023.I’ve seen things you machines wouldn’t believe: Measuring content predictability to identify automatically-generated text. In IberLEF@ SEPLN. Areg Mikael Sarvazyan, José Ángel González, Marc Franco-Salvador, Francisco Rangel, Berta Chulvi, and Paolo Rosso. 2023. Overview of AuTexTifica- tion at IberLEF 2023: Detection and attribution of machine-generated text in multiple domains. Proce- samiento del Lenguaje Natural, 71:275–288. Kristina Schaaff, Tim Schlippe, and Lorenz Mindner. 2023. Classification of human- and ai-generated texts for english, french, german, and spanish. Preprint, arXiv:2312.04882. Tatiana Shamardina, Vladislav Mikhailov, Daniil Cher- nianskii, Alena Fenogenova, Marat Saidov, Anas- tasiya Valeeva, Tatiana Shavrina, Ivan Smurov, Elena Tutubalina, and Ekaterina Artemova. 2022. Findings of the the RuATD shared task 2022 on artificial text detection in Russian. In Computational Linguistics and Intellectual Technologies. RSUH. Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Anastasia Kozlova, Vladislav Mikhailov, and Tatiana 10 Shavrina. 2024. mgpt: Few-shot learners go multilin- gual. Transactions of the Association for Computa- tional Linguistics, 12:58–79. Rafael Rivera Soto, Kailin Koch, Aleem Khan, Barry Chen, Marcus Bishop, and Nicholas Andrews. 2024.Few-shot detection of machine-generated text using style representations.arXiv preprint arXiv:2401.06712. Michal Spiegel and Dominik Macko. 2024. IMGTB: A framework for machine-generated text detection benchmarking. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 3: System Demonstrations), pages 172–179, Bangkok, Thailand. Association for Com- putational Linguistics. Jinyan Su, Terry Yue Zhuo, Di Wang, and Preslav Nakov. 2023. Detectllm: Leveraging log rank information for zero-shot detection of machine-generated text. arXiv preprint arXiv:2306.05540. Cheng Wang, Xinyang Lu, See-Kiong Ng, and Bryan Kian Hsiang Low. 2024a. Trace: Transformer-based attribution using contrastive embeddings in llms. arXiv preprint arXiv:2407.04981. Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Osama Mo- hanned Afzal, Tarek Mahmoud, Giovanni Puc- cetti, Thomas Arnold, and 1 others. 2024b. M4gt-bench: Evaluation benchmark for black-box machine-generated text detection. arXiv preprint arXiv:2402.11175. Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Osama Mo- hammed Afzal, Tarek Mahmoud, Giovanni Puccetti, and Thomas Arnold. 2024c. SemEval-2024 task 8: Multidomain, multimodel and multilingual machine- generated text detection.In Proceedings of the 18th International Workshop on Semantic Evalua- tion (SemEval-2024), pages 2057–2079, Mexico City, Mexico. Association for Computational Linguistics. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. Advances in neural information processing systems, 32. Shiyuan Zhang, Yubin Ge, and Xiaofeng Liu. 2024. Nuanced multi-class detection of machine-generated scientific text. In Proceedings of the 38th Pacific Asia Conference on Language, Information and Computa- tion, pages 119–130. A Original Study To Be Replicated The original study by Przybyla et al. (2023) fo- cuses on detecting automatically generated texts using probabilistic and textual features. The au- thors hypothesized that automatically generated texts differ in their token probability distributions from human-written texts, and that these differ- ences can be used for classification. Their sys- tem was submitted to the AuTexTification 2023 shared task, which comprised two subtasks. Sub- task 1 addressed binary classification, distinguish- ing between human-written and machine-generated texts in English and Spanish. Subtask 2 extended this setup to a multi-way classification problem, where the objective was to attribute each machine- generated text to one of several specific generative models. A.1 System Overview The authors proposed a hybrid system combin- ing token-level predictability features derived from generative language models with word-level lin- guistic features and contextual representations from a fine-tuned transformer model. The central as- sumption is that machine-generated texts tend to be less surprising and more predictable for language models, while human-written texts exhibit higher entropy and variability. The overall architecture and feature extraction pipeline are illustrated in Figure 1. The system is composed of the following feature groups (see Figure 1): •Predictability-based features: for each token position, the log-probability of the observed token, the log-probability of the most likely token, and the entropy of the token probabil- ity distribution, computed using generative language models (GPT-2 variants). •Word-level features: logarithmic word fre- quency derived from the Google Books Ngrams corpus and grammatical correctness indicators obtained using LanguageTool. •Text-level linguistic features: aggregated statistics of part-of-speech tags, morphologi- cal categories, dependency relations, named entities, and rare-word indicators. •Contextual representations: pooled [CLS] representations extracted from fine-tuned RoBERTa models. A.2 Methodology As mentioned above, the authors participated in the AuTextTification 2023 shared task and used the dataset provided within the competition. A more 11 Figure 1: Overview of the architecture proposed by Przybyla et al. (2023). The upper part shows the evaluated model configurations, while the lower part illustrates the feature extraction pipeline used to derive token-level probabilistic features and document-level linguistic features. detailed description of the dataset is presented in Section 3 of our work. Text processing. Each text was processed using GPT-2 language models of different sizes to com- pute token-level predictability features. For each token position (up to length 128), the following quantities were extracted: (i) the log-probability of the observed token, (i) the log-probability of the most likely token, and (i) the entropy of the to- ken probability distribution. Instead of aggregating these values into a single perplexity score, the full sequence of token-level features was retained. In addition, two types of word-level features were introduced: •Word frequency – obtained from the Google Books Ngrams v3 dataset for English and Spanish, aligned with the language model to- kenization; •Grammatical correctness – obtained using LanguageTool, where tokens were marked ac- cording to whether they were preserved or modified by the grammar checker. To enrich the text representation, the authors fine-tuned RoBERTa (RoBERTa-base for English and RoBERTa-base-BNE for Spanish) and used the pooled [CLS] vector as a contextual embedding. They also extracted linguistic information at the document level, including part-of-speech tags, de- pendency relations, morphological categories, and named entities. These annotations were converted into aggregated statistics. Model configuration. The main neural compo- nent is a bidirectional LSTM (BiLSTM) operating on sequences of token-level predictability features obtained from several GPT-2 models (DistilGPT-2, GPT-2, GPT-2 Medium, and GPT-2 Large). The hidden size is 64 in each direction, yielding a 128-dimensional representation after concatenation. This vector can be combined with the RoBERTa embedding and passed to a classification layer. In addition, a Random Forest classifier is trained on the aggregated linguistic features. Model variants. Based on their overall config- uration, the authors defined the following variants for evaluation: •Pred: a BiLSTM network using predictability features, •FLM: only the fine-tuned language model, used as a baseline, •Pred + FLM: a linear classifier combining representations from Pred and FLM, referred to as Hybrid, •Pred + FLM + Add: as above, but including additional token-level features, referred to as Hybrid+, •LingRF: a Random Forest trained on linguis- tic features, 12 •LingRF + PredOut: a Random Forest using linguistic features and probabilities returned by the BiLSTM network operating on the pre- dictability features, referred to as Linguisti- cRF. Implementation. The neural models were im- plemented in PyTorch, and pretrained language models were loaded via HuggingFace Transform- ers. The Adam optimizer was used with a learning rate of10 −3 for the BiLSTM and2× 10 −5 for fine-tuning RoBERTa. In the combined setting, RoBERTa was frozen for the first five epochs and then trained jointly with the rest of the network using the smaller learning rate. Topic-based experiments were trained for 10 epochs. For the final submission, the data were split into 80% training and 20% development sets and trained for up to 20 epochs. Model selection was based on development macro F1-score, with early stopping at the first epoch reaching at least (f max − 0.01). Random Forest classifiers were implemented in Scikit-learn.After tuning, 200 trees with maximum depth 60 were used for all settings. Linguistic features were annotated using spaCy (en_core_web_sm,es_core_news_sm) and ag- gregated into document-level statistics. Word- frequency features were derived from the Google Books Ngrams corpus. The official implementation is available athttps://github.com/piotrmp/ autext. B Additional Stylometric Features The additional stylometric features introduced as part of RQ3 are summarized in Table 9. These features extend the original setup by incorporat- ing measures of lexical diversity, sentence struc- ture, repetition patterns, word-level statistics, func- tional and stylistic markers, readability metrics, and punctuation usage, thereby providing a more fine-grained characterization of writing style. All features are extracted at the document level using lightweight rule-based methods. Texts are tokenized with regex-based word splitting and sen- tence segmentation based on terminal punctuation. Lexical diversity, repetition, and word-level statis- tics are computed from token frequency distribu- tions, while stylistic marker ratios are obtained by matching tokens against curated word lists. Read- ability metrics are calculated using thetextstat library. C Additional results In this section, we present additional experiments that were not included in the main analysis due to space limitations. These experiments are not directly tied to the main research questions, but illustrate possible extensions of the original config- uration. We evaluate two additional configurations de- rived from the baseline, namely Ultrahybrid and Hybrid_flat. Their architectures and feature pro- cessing are described in the following subsection. We also provide a more detailed comparison be- tween the original baseline and the proposed mul- tilingual configuration, focusing on model robust- ness across different random seeds and the compu- tational cost of training and inference. C.1 New Configurations As part of the original extension, we also explored additional model configurations. This section de- scribes their structure and the way input features are processed. The Ultrahybrid configuration is implemented as a two-stage model. In the first stage, the Hybrid+ model is trained end-to-end. Sequential token-level features (probabilistic channels, token frequency, and grammar flag) are processed by the BiLSTM component, while the input text is processed by the encoder component. After training, the softmax class probabilities are extracted for each document. In the second stage, these probabilities are concate- nated with document-level linguistic features. The resulting feature matrix is used to train traditional machine-learning classifiers: Random Forest, XG- Boost, and a multilayer perceptron. The results of the Ultrahybrid configuration are reported in Table 10. From the table, we can see that in the baseline setting, UltraHybrid does not improve the results for Subtask 1 in either lan- guage. The class probabilities produced by Hy- brid+ already represent a very strong signal, and the second-stage classifiers cannot significantly en- hance it. The additional document-level features therefore do not provide useful complementary in- formation. For the more challenging Subtask 2, however, UltraHybrid consistently outperforms the original Hybrid+. In the multilingual setting, the benefit of UltraHybrid is more visible. In all four combinations (Subtask 1/2 × EN/ES), it achieves higher scores than Hybrid+. This suggests that in this setting, the first-stage probabilities are not as 13 CategoryFeatureDescription Lexical Diversity ttrType–token ratio (unique tokens / total tokens). root_ttrTTR normalized by the square root of token count. log_ttrLogarithmic variant of TTR. hapax_ratio Ratio of words occurring once in the document. dis_legomena_ratioRatio of words occurring exactly twice. rare_word_burstinessBurstiness (clustering) of rare words across the text. Sentence Structure avg_sentence_lengthAverage tokens per sentence. sentence_length_stdStandard deviation of sentence length. sentence_length_cvCoefficient of variation of sentence length. sentence_countNumber of sentences in the document. Repetition Patterns bigram_repetition Ratio of excess repeated bigrams to total bi- grams. trigram_repetitionRatio of excess repeated trigrams to total tri- grams. Word-Level Statistics avg_word_lengthAverage token length in characters. word_length_stdStandard deviation of token length. word_countTotal number of tokens in the document. Functional & Stylistic Markers function_word_ratioRatio of function words to all tokens. transition_word_ratio Ratio of discourse transition words. hedge_word_ratioRatio of hedging expressions (e.g., maybe, pos- sibly). first_person_ratioRatio of first-person pronouns. formal_word_ratioRatio of formal register vocabulary. Readability Metrics flesch_reading_easeFlesch Reading Ease score. flesch_kincaid_gradeFlesch–Kincaid grade level. Punctuation Usage punctuation_ratioRatio of punctuation characters to total charac- ters. comma_ratioAverage number of commas per sentence. exclamation_ratioAverage number of exclamation marks per sen- tence. question_ratioAverage number of question marks per sentence. Table 9: Additional stylometric features introduced in this study. Hybrid+UltraHybrid SubtaskLangRFXGBMLP Baseline prob models S1en0.7960.7960.7950.779 S1es0.7430.7200.7260.713 S2en0.5860.5990.6010.591 S2es0.5670.5910.5900.598 Multilingual prob models S1en0.7650.7710.7770.780 S1es0.7040.7310.7380.717 S2en0.5950.6360.6350.637 S2es0.6350.6470.6520.645 Table 10: Test F1-scores for UltraHybrid (baseline vs. multilingual). dominant, and the second-stage classifier can make better use of its additional features. The Hybrid_flat configuration uses the same token-level features as Hybrid+, but without the sequential component and without the FLM com- ponent. The same probabilistic channels are used as in previous experiments, together with the loga- rithmic token frequency and a binary grammatical correctness flag. Instead of processing the full sequence with a BiLSTM, each real token-level channel is aggre- gated to the document level using four statistics: mean, maximum, minimum, and standard devia- tion. Letx (k) i denote the value of thek-th channel for thei-th valid token in a document, and letNbe the number of valid tokens given by the mask. For each channelk, the following aggregated features are computed: μ k = 1 N N X i=1 x (k) i , max k = max 1≤i≤N x (k) i , min k = min 1≤i≤N x (k) i , 14 Baseline prob models Modelen-S1es-S1en-S2es-S2 RF0.8250.7810.5020.502 XGB0.8160.7890.5360.532 MLP0.8130.7490.5470.521 Multilingual prob models ModelS1-enS1-esS2-enS2-es RF0.8430.8750.5330.549 XGB0.8470.8770.5660.585 MLP0.8080.7820.5950.568 Table 11: Test Macro F1-scores across subtasks and languages for Hybrid_flat. σ k = v u u t 1 N N X i=1 x (k) i − μ k 2 . If the number of real token-level channels is denoted byK, the dimensionality of the aggregated part is 4× K. In the English baseline setting (12 probabilistic channels, 1 frequency, and 1 grammar channel), we haveK = 14, which results in4× 14 = 56 aggregated features. These aggregated features are concatenated with standard linguistic document-level features, form- ing a fixed-length vector representing the whole document. This vector is used as input to the final classifier. As in the Ultrahybrid configuration, we experimented with Random Forest, XGBoost, and a multilayer perceptron. The results are shown in the table 11. As shown in the table, we compared the pro- posed configuration in two settings: baseline (prob- abilistic models used in RQ1 to replicate the origi- nal study) and multilingual (probabilistic models proposed as an improved configuration in RQ2). The results show that the configuration using multi- lingual probabilistic models achieves better results than the baseline, especially for tasks focused on Spanish. Regarding the individual models, XG- Boost achieves the best results in three out of four cases, while MLP performs best in one case. The proposed configuration achieves high results over- all and, particularly for Subtask 1, clearly outper- forms the original solution that we replicated (see Table 4). Lang ModelTest F1Time (min) Baseline prob models S1 enHybrid0.788± 0.03476.1 enHybrid+ 0.793± 0.07777.4 esHybrid0.721± 0.00655.1 esHybrid+ 0.728± 0.01255.4 S2 enHybrid0.580± 0.01147.0 enHybrid+ 0.588± 0.00247.5 esHybrid0.589± 0.00337.5 esHybrid+ 0.573± 0.01137.8 Multilingual prob models S1 enHybrid0.828± 0.023122.1 enHybrid+ 0.790± 0.031123.5 esHybrid0.747± 0.044119.5 esHybrid+ 0.769± 0.047120.2 S2 enHybrid0.632± 0.00975.3 enHybrid+ 0.620± 0.01776.1 esHybrid0.639± 0.00875.7 esHybrid+ 0.637± 0.00276.2 Table 12: Robustness across three runs (Test F1: mean ± std). Runtime is reported for a single run (seed=10). C.2 Robustness and Time Complexity In this section, we analyze the stability of the pro- posed multilingual configuration (mDeBERTa-v3 + Large group) from RQ2. The goal is to verify whether the observed improvements are not caused by random initialization and whether the results are consistent across different seeds. We repeated the experiments with three random seeds (10, 11, 12) and report the mean performance and standard deviation for both the baseline and the multilingual configuration. In addition to ro- bustness, we also analyze time complexity by com- paring the total training and inference time of the baseline and the proposed multilingual solution. The results are shown in Table 12. From Table 12, the results are stable across the three seeds. The standard deviation is low in most cases, which indicates that the performance is not strongly affected by random initialization. Higher variability appears only in some configurations in Subtask 1, but this is not a systematic effect. Training times are significantly lower in the base- line setting. On average, the multilingual configu- ration is about 80% more time-consuming in both subtasks. This is related to the use of a larger model group and more complex probabilistic models. In terms of the trade-off between performance and time, the multilingual configuration provides 15 ModelS1-en S1-es S2-en S2-es mdok0.763 0.679 0.626 0.670 LingRF+PredOut 0.912 0.881 0.599 0.562 Table 13: Comparison with the state-of-the-art detector mdok (macro F1 on the test set). the largest benefit in Subtask 2, where Test Macro F1 improves in both languages. In Subtask 1, the situation is less clear. Hybrid improves for EN, while Hybrid+ achieves comparable or slightly lower performance. Overall, the proposed mul- tilingual configuration is stable and improves per- formance mainly in the more challenging second subtask, but at the cost of higher time complexity. C.3 Comparison with State-of-the-Art Detector We further compared the LingRF+PredOut con- figuration with mdok (based on Qwen3-4B-Base) (Macko, 2025), a state-of-the-art detector. In our experiments (see Section 6), we extend this config- uration with newly introduced probabilistic models and additional stylometric features. As shown in Table 13, our approach substantially outperforms mdok in Subtask 1, while mdok achieves higher performance in the more challenging Subtask 2. Al- though the neural detector performs better for gen- erator attribution, our stylometry-based approach provides interpretable signals, allowing us to ana- lyze which stylistic features influence the model’s decisions. D SHAP Interpretability To better understand the contribution of individual stylometric features, we analyze the models using SHAP values. Figures 2–5 present the SHAP sum- maries for selected classes in both subtasks and lan- guages. The results highlight the most influential linguistic and stylistic features driving the predic- tions of the LingRF Style model and its extended variant with PredOut features. Overall, we observe that the features introduced in our work frequently rank among the most influential, often exhibiting the highest impact on the model’s predictions. 16 Figure 2: SHAP Summary for Subtask 1 (English) - LingRF Style, Class 1 Figure 3: SHAP Summary for Subtask 2 (Spanish) - LingRF Style, Class 5 17 Figure 4: SHAP Summary for Subtask 1 (English) - LingRF Style + PredOut, Class 1 Figure 5: SHAP Summary for Subtask 2 (Spanish) - LingRF Style + PredOut, Class 5 18