Paper deep dive
Protocol-Sensitive Evaluation of Log Anomaly Detection: Component Costs and Target-Access Sensitivity on HDFS and BGL
Hang Xiao, Janet Sung, Zhaoyi Li, Gangzhen Qian, Chuhong Xu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Protocol choices can change the conclusions drawn from log anomaly detection benchmarks even when detector settings are fixed. We present a joint empirical study of split construction, representation visibility, and component costs using six fixed count, sequence, and semantic configurations on Hadoop Distributed File System (HDFS) and Blue Gene/L (BGL) logs. Random splits place several configurations near the average-precision ceiling, whereas group-disjoint HDFS and chronological BGL evaluation produce lower scores and different observed orderings. At a fixed BGL cutoff, parser choice spans 0.124 in semantic XGBoost mean average precision while preserving its lead over count XGBoost; the earliest rolling period reverses that ordering. A two-factor cross-system ablation contrasts source-only representations with offline transductive access to unlabeled target templates through the representation corpus and inverse document frequency: HDFS-to-BGL mean average precision moves from 0.191 with source-only access to 0.325 with union-corpus, target-IDF access, and the intermediate conditions reveal direction-dependent interactions in average precision and retrieval at fixed review budgets. Component-level profiling separates parsing and representation costs from classifier training, prediction, and storage. Together, these findings connect detector comparisons to the test population, preprocessing state, visible information, and measured pipeline stages, and identify the protocol fields needed alongside a score to support interpretable comparisons of log anomaly detection accuracy and resource use.
Tags
Links
- Source: https://arxiv.org/abs/2610.05807v1
- Canonical: https://arxiv.org/abs/2610.05807v1
Trouble viewing inline? Open PDF directly â
Full Text
43,905 characters extracted from source content.
Expand or collapse full text
Protocol-Sensitive Evaluation of Log Anomaly Detection: Component Costs and Target-Access Sensitivity on HDFS and BGL PubID: pubid: © 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. Hang Xiao Affiliation: Fortinet, Inc. Sunnyvale, USA xhang@fortinet.com Janet Sung Affiliation: Google LLC Mountain View, USA janetsung@google.com Zhaoyi Li Affiliation: Independent Researcher Mountain View, USA lzy9776@gmail.com Affiliation: [ ] Gangzhen Qian Affiliation: Google LLC Mountain View, USA irisqian@google.com Chuhong Xu Affiliation: Sony Corporate of America San Jose, USA chuhong.xu@sony.com Abstract Protocol choices can change the conclusions drawn from log anomaly detection benchmarks even when detector settings are fixed. We present a joint empirical study of split construction, representation visibility, and component costs using six fixed count, sequence, and semantic configurations on Hadoop Distributed File System (HDFS) and Blue Gene/L (BGL) logs. Random splits place several configurations near the average-precision ceiling, whereas group-disjoint HDFS and chronological BGL evaluation produce lower scores and different observed orderings. At a fixed BGL cutoff, parser choice spans 0.124 in semantic XGBoost mean average precision while preserving its lead over count XGBoost; the earliest rolling period reverses that ordering. A two-factor cross-system ablation contrasts source-only representations with offline transductive access to unlabeled target templates through the representation corpus and inverse document frequency: HDFS-to-BGL mean average precision moves from 0.191 with source-only access to 0.325 with union-corpus, target-IDF access, and the intermediate conditions reveal direction-dependent interactions in average precision and retrieval at fixed review budgets. Component-level profiling separates parsing and representation costs from classifier training, prediction, and storage. Together, these findings connect detector comparisons to the test population, preprocessing state, visible information, and measured pipeline stages, and identify the protocol fields needed alongside a score to support interpretable comparisons of log anomaly detection accuracy and resource use. Index Terms: log anomaly detection, dependability, empirical evaluation, evaluation protocol, target access, computational cost â aftertitle: I Introduction During incident response, an operator usually needs a short list of sessions or windows to inspect, not another stream of raw messages. A detector therefore has two practical jobs: rank suspicious activity near the top and keep the review queue manageable. Benchmark results influence both, so the protocol used to obtain a score matters. Early log-mining systems turned console messages into event features; the HDFS study by Xu and colleagues introduced the dataset used here [1]. In our HDFS data, most sessions repeat one of a few event-count patterns, so a random split places familiar behavior on both sides of the boundary. He and colleagues organized log anomaly detection as a pipeline of parsing, feature extraction, and detection [2]. Such models are evaluated on the same small group of public logs, yet their scores are hard to compare because the settings differ. DeepLog shifted attention to event order [3] and LogAnomaly added semantic template vectors [4]. We examine three protocol choices that shape benchmark comparisons: split construction, the information available during representation building (target access), and cost accounting. Six configurations with fixed hyperparameters provide a common basis for examining these choices. SedlĂĄÄek and colleagues compared shuffled and sequential evaluation and preprocessing choices within a unified pipeline [5]. Building on this protocol perspective, our study combines a target-catalog and inverse document frequency (IDF) visibility ablation, tie-aware review-budget metrics, and component-level cost accounting. All six configurations run under one implementation, using a stratified random split, an HDFS holdout keeping sessions with identical event-count vectors together, and a chronological BGL split with a parser snapshot at the cutoff. Further runs vary the BGL parser and cutoff, and two Word2vec corpora cross two IDF sources in both directions. The study makes three contributions: 1. It shows how random, group-disjoint, and chronological evaluation change detector comparisons under a common implementation, and characterizes parser and temporal-cutoff sensitivity on BGL. 2. It separates parsing, representation construction, classifier training, and prediction costs, and reports serialized classifier sizes to make the resource requirements of the measured stages explicit. 3. It provides a 2Ă22Ă 2 target-catalog and IDF visibility ablation with tie-aware precision and recall at fixed review budgets, stating for each cell whether the representation saw unlabeled target artifacts. Every score below is reported with its split, visible catalog, and measured cost component, so each observation stays tied to the protocol that produced it. I Background and related work I-A Log parsing Log parsers replace identifiers, addresses, counters, and other variable fields with event templates, fixing the vocabulary seen downstream. Drain organizes candidate templates in a fixed-depth tree and provides the basis for our compact BGL parser [6]; we use depth 4, similarity threshold 0.4, and maximum fan-out 100. HDFS retains its released preprocessing while BGL is parsed from raw messages. Khan and colleagues showed that the parser propagates into detection accuracy and that parser accuracy alone does not predict downstream behavior [7]; our BGL sensitivity runs vary this choice for the same reason. I-B Detection models Classical detectors commonly operate on event counts. Sequence models treat templates as tokens and learn their order, while semantic models retain words from the template text; the boundaries are loose, and a sequential classifier may consume semantic event vectors. Ali and colleagues evaluated traditional and deep techniques in terms of detection accuracy, time performance, and sensitivity to hyperparameter settings [8]. We use fixed count, sequence, and semantic configurations to compare how each representation responds to split and preprocessing choices. I-C Evaluation protocols and benchmark critique A parallel line of work questions the benchmarks themselves. Landauer and colleagues surveyed the datasets, preprocessing, models, and evaluation procedures used across log-anomaly studies and found broad variation [9], then reviewed the common sequence-based data sets in detail, documenting labeling conventions and structural properties that shape the evaluation tasks these corpora support [10]. Our study fixes the detectors and names the protocol dimensions that differ. Le and Zhang showed that training selection, grouping, class distribution, and noise materially change reported results [11], and the study of SedlĂĄÄek and colleagues is closest to ours [5]. I-D Cross-system anomaly detection Cross-system detection adds a vocabulary shift to the usual distribution shift. LogTransfer trains a model across two systems [12], and LogDLR learns a domain-invariant latent representation for unsupervised cross-system detection [13]. Our experiment examines representation construction: the classifier family, hyperparameters, and 100-dimensional representation size are fixed while the Word2vec corpus and IDF source vary. This design measures how target-catalog visibility changes the cross-system feature space. I Evaluated configurations The six configurations span count, sequence, and semantic representations under fixed settings, enabling comparisons across protocol conditions. Each HDFS block session or BGL window is encoded in three ways. The count representation has one coordinate per template and records occurrences, dropping event order but keeping the frequency profile; logistic regression (LR) is the linear baseline, and random forest (RF) and XGBoost the two tree ensembles. The sequence representation preserves the ordered template IDs, paired with a single-layer LSTM using a 32-dimensional embedding, 64 hidden units, and class-balanced loss; its vocabulary produces 28,161 parameters for HDFS, 115,105 for the BGL random split, and 76,033 for the frozen-past BGL run. The semantic representation, called Template2Vec here, is shared across systems. We tokenize the templates and fit a 100-dimensional skip-gram Word2vec model [14]; word coordinates retain the same dimensionality in HDFS and BGL. For a template t, we compute: t=âwâtIDFâĄ(w)âÏâ(w)âwâtIDFâĄ(w),e_t= _wâ tIDF(w)Ï(w) _wâ tIDF(w), when at least one token is in the vocabulary; otherwise the template receives a zero vector. The session vector is the count-weighted mean of its template vectors, and LR and XGBoost are trained on that representation. For the union-corpus cells, Word2vec is fitted on the unlabeled union of source and target template catalogs. The source-corpus, source-IDF cell uses source templates alone and is strictly inductive with respect to target-side access. All runs use a process-independent SHA-256 word hash, one Word2vec worker, and no external pretrained model, and five predefined seeds vary fitting and representation initialization. IV Study design IV-A Operational setting and data access The study uses an offline benchmark assembled from logs collected by a trusted monitoring pipeline. Training has access to source logs, templates, and labels. Labels for scored data remain held out. Inductive runs fit representation models using training-side artifacts; offline transductive runs additionally use unlabeled artifacts from the data to be scored. Random and grouped semantic runs share a corpus over both unlabeled catalogs; in the cross-system ablation, the source-corpus, source-IDF cell is inductive, and cells using a union corpus or target IDF are offline transductive. The resulting scores rank benchmark sessions or windows for review. Section VI describes the security assumptions and the operational meaning of the released labels. TABLE I: Dataset characteristics. HDFS and BGL differ in scale, vocabulary, prevalence, repetition, and session structure. Characteristic HDFS BGL Unit block session 100-line window Sessions/windows 575,061 47,479 Anomalies 16,838 (2.93%) 4,802 (10.11%) Event vocabulary 29 2,746 Mean length 19.4 100 Distinct count vectors 589 6,280 Repeated-vector share 99.96% 88.86% IV-B Datasets Table I summarizes the two datasets. HDFS messages are grouped by block identifier and inherit the released block-session label. BGL contains line-level alerts from a Blue Gene/L supercomputer [15]. We keep the chronological order, form non-overlapping 100-line windows, and label a window anomalous when it contains at least one alert. HDFS and BGL are historical benchmarks with documented recurrence patterns, event vocabularies, and labeling units [10]. Their contrasting structures provide two settings for examining protocol-dependent behavior. Extending the same accounting to further labeled Loghub corpora, such as Thunderbird [16], would broaden the empirical coverage. Repetition is pronounced in both. HDFS contains 589 count vectors across 575,061 sessions; 99.96% of sessions share a count vector with another, and 98.09% share a complete template-ID sequence. BGL has 6,280 count vectors, with corresponding shares of 88.86% and 72.96%. Under the seed-0 random split, 99.95% of HDFS test sessions and 88.39% of BGL test windows match a training count vector. The group and chronological protocols reduce that recurrence. IV-C Research questions and protocols RQ1: How do the observed results and component costs change across split protocols? We begin with a stratified 70/30 random split for each dataset. HDFS is also evaluated with repeated group-disjoint holdouts: sessions sharing a count vector form one group, assigned intact to fitting, validation, or test. BGL supplies timestamps, so its second protocol trains on the first 70% of windows and tests on the remaining 30%; HDFS uses group-disjoint evaluation because its released sessions lack a reliable global order. The protocols jointly vary the test population and trainingâtest recurrence; BGL chronological evaluation also changes the calendar interval. TABLE I: Information available before final test scoring, and the resulting access regime. Condition Labels visible Template catalog Access regime Within random/group own training full catalog transductive* BGL chronological past training cutoff snapshot inductive Cross: source corpus, source IDF source training source only inductive Cross: union corpus or target IDF source training source + target transductive * Scored labels are hidden; transductive access uses unlabeled scored-data artifacts (Section IV). TABLE I: Fixed hyperparameters used across protocols and seeds. Model Configuration LR inverse regularization; balanced weights; 1,000 iterations; standardized input RF 200 trees; balanced-subsample class weights XGBoost 300 trees; depth 6; rate 0.1; histogram method; class-ratio positive weight Word2vec skip-gram; 100 dimensions; window 5; minimum count 1; 30 epochs LSTM 32-d embedding; 64 hidden; Adam 10â310^-3; batch 256; at most five epochs The chronological BGL run saves the parser state at the cutoff, where 1,524 templates are known and the Word2vec and IDF models are fitted. The parser then advances window by window: count and sequence inputs map new templates to unk, while semantic inputs use words from the frozen vocabulary and return zero when none is known. Of the 2,746 templates eventually produced, 1,222 first appear in the test period, and 55.5% of anomalous test windows contain at least one. This follows the concern raised by LogRobust: logging code and observed event patterns change over time [17]. RQ2: How stable is the BGL chronological result across parsers and cutoffs? At the 70/30 cutoff, the compact parser is run with similarity thresholds 0.3, 0.4, and 0.5, and Drain3 v0.9.11 supplies a fourth configuration at depth 4 and threshold 0.4; fan-out, masking, and windowing remain fixed. We then use the default parser for rolling-origin tests at 50%, 60%, 70%, and 80%, scoring the next 10% of windows each time. The fixed-cutoff panel varies parser configuration while retaining the other settings. Rolling-origin evaluation examines temporal scenarios in which training-prefix size, calendar period, event mix, vocabulary state, and test interval change together. RQ3: How does unlabeled target-catalog visibility change cross-system feature coverage? Two Word2vec corpora (source or union) are crossed with two IDF sources (source or target); the union corpus and target IDF use target templates without labels, and Table VIII reports XGBoost in both directions. Domain-adaptation methods such as LogTAD learn an explicit alignment across systems [18]; here the four cells keep the classifier family, hyperparameters, labels, and source training set fixed while varying catalog and IDF visibility. Each cell constructs its own features and fits a separate classifier. Table I records what is available and labels each condition inductive or offline transductive; a union-corpus token absent from source IDF receives the logged fallback weight 1.0. IV-D Training, selection, and repetition Each training portion is divided 80/20 into fitting and validation subsets. HDFS groups remain disjoint, BGL validation follows training in time, and validation F1F_1 selects the decision threshold. RF follows the random-forest construction of Breiman [19]; we use the scikit-learn implementation [20] with 200 trees and its balanced_subsample class weighting. Hyperparameters are unchanged across protocols (Table I). XGBoost uses 300 depth-6 trees, learning rate 0.1, the histogram method, and a class-ratio positive weight, following the scalable boosted-tree system of Chen and Guestrin [21]. The LSTM trains for at most five epochs and keeps the epoch with the highest validation AP. HDFS sequences are truncated at 50 events, of which only 0.01% are longer, and BGL keeps all 100 per window. Five predefined seeds control the applicable split, group assignment, and stochastic fitting; the chronological period remains fixed across seeds. HDFS groups are assigned by a seeded stratified procedure targeting 56/14/30 percent for fit, validation, and test. Unequal group sizes produce realized test shares of 13%â31% and prevalence of 2.51%â6.99%, and groups containing both classes receive an any-anomaly label during stratified assignment within each seed. In Table IV, each HDFS group assignment therefore defines a test set with its own size and anomaly prevalence; the reported means and standard deviations summarize variation across group assignments and model fitting. IV-E Metrics and statistical reporting We report anomaly-class precision, recall, and F1F_1 at the validation-selected threshold. Average precision (AP) is computed with the average_precision_score function of sklearn.metrics [20], as a weighted step sum of precision over recall increments. Test prevalence is the random-ranking reference. Table VIII reports tie-aware precision and recall at 1% and 5% review budgets; a boundary cutting through a score tie receives the expected fractional allocation under uniform selection within that tied group, and budgets count benchmark sessions or windows. Tables report archived means and standard deviations across five seeds. HDFS group-holdout dispersion combines variation in group assignments, test composition, and model fitting; chronological BGL dispersion reflects fitting within a fixed test period. The archive does not specify whether the standard-deviation denominator is n or nâ1n-1. The analysis reports mean differences and across-run dispersion; inferential tests and confidence intervals are outside the reported analysis. IV-F Cost accounting Timing is recorded separately for parsing, representation fitting, feature transformation, classifier training, and classifier-only inference, and each boundary is reported on its own. Classifier-only throughput is the median of three passes over the test split, fit time is summarized across the five seeded runs, and serialized classifier bytes define artifact size: pickle for the classical configurations, a saved state dictionary for the LSTM. Word2vec fitting and semantic transformation are timed in the same runs; BGL parsing is timed once from raw logs, and HDFS begins from released preprocessing. Classifier-only throughput measures prediction on prepared features; parsing, feature construction, model loading, I/O, and analyst delay are outside that metric, and parsing and representation costs are reported separately. Archived timings were produced on two threads of an Intel Xeon 2.10 GHz virtual CPU with two cores, 8 GB RAM, Linux, and OpenBLAS. The resource measurements comprise component runtimes and serialized classifier sizes; peak memory and complete deployment-bundle size were not recorded in the archive. The Apple runs behind the parser and rolling-origin panels contribute accuracy results; their detailed timing configuration is unavailable. V Results TABLE IV: HDFS results across five seeds (mean ± SD); random/group access is transductive. Random split Group-disjoint Configuration AP F1F_1 AP F1F_1 Count/LR .991±.004 .989±.006 .867±.103 .934±.046 Count/RF 1.000±.000 .998±.000 .743±.227 .808±.259 Count/XGB .999±.001 .998±.000 .692±.175 .582±.337 Sequence/LSTM .999±.001 .993±.002 .627±.210 .502±.324 Semantic/LR .982±.003 .981±.003 .725±.174 .792±.275 Semantic/XGB 1.000±.000 .998±.000 .624±.235 .318±.241 TABLE V: BGL results across five seeds (mean ± SD); random access is transductive, chronological access is inductive. Random split Chronological Configuration AP F1F_1 AP F1F_1 Count/LR .987±.003 .976±.004 .277±.000 .423±.000 Count/RF .998±.001 .984±.002 .388±.005 .400±.012 Count/XGB .998±.001 .989±.004 .448±.000 .396±.000 Sequence/LSTM .946±.012 .916±.017 .236±.151 .282±.091 Semantic/LR .916±.006 .870±.007 .205±.095 .267±.043 Semantic/XGB .992±.000 .955±.006 .676±.086 .492±.165 Fig. 1: Mean AP (± standard deviation over five seeds) of the six configurations under random splitting and each datasetâs alternative protocol, from Tables IV and V. Group-disjoint HDFS and chronological BGL yield lower mean scores and different leading configurations. V-A RQ1: Split construction changes the observed ordering Tables IV and V address RQ1; Fig. 1 plots their mean AP. Random splitting produces near-ceiling AP clusters. On HDFS four configurations report mean AP within 0.001 of each other, with count/LR at 0.991 and semantic/LR at 0.982 below them; on BGL three configurations lie within 0.006, with count/LR at 0.987, the LSTM at 0.946, and semantic/LR at 0.916 below them. Count/XGB records AP 0.999±0.0010.999± 0.001 on HDFS and 0.998±0.0010.998± 0.001 on BGL. Nearly every HDFS test session and most BGL test windows already match a training count vector, so the table largely measures interpolation over recurrence. Separating the HDFS count-vector groups changes the picture: mean AP falls for all six configurations and varies far more across assignments, and count/LR records the highest mean AP at 0.867±0.1030.867± 0.103, followed by RF at 0.743±0.2270.743± 0.227 and semantic/LR at 0.725±0.1740.725± 0.174, with the LSTM and semantic/XGB averaging 0.627 and 0.624. The standard deviations summarize variation across the five group assignments and model fitting. The protocol change alters which configuration has the highest reported mean AP. RF and semantic/XGB have the same reported mean AP in the HDFS random table, whereas their mean AP differs by 0.119 after grouping, and count/LR moves from below the four near-ceiling configurations to the highest grouped mean. Its mean F1F_1 falls from 0.989 to 0.934 while its mean AP falls from 0.991 to 0.867: the validation-selected operating point and the full ranking respond differently to the protocol change. The chronological BGL holdout produces a third ordering, with the period fixed so seeds vary fitting only. Semantic/XGB records the highest mean AP at 0.676±0.0860.676± 0.086, count/XGB reaches 0.448, and the LSTM reaches 0.236±0.1510.236± 0.151. The semantic representation is fitted on the 1,524 templates available at the cutoff; of the 1,222 appearing later, 418 contain no word from the frozen vocabulary when first observed and receive a zero vector. Refitting the representation on the full catalogâan offline transductive variant that sees post-cutoff templatesâchanges semantic/XGB AP from 0.676±0.0860.676± 0.086 to 0.796±0.0530.796± 0.053. Full-catalog access changes zero-vector coverage together with the rest of the representation, so this contrast measures their joint effect. AP and F1F_1 identify the same highest-scoring BGL configuration; semantic/XGB has an F1F_1 standard deviation of 0.165. Count/XGB produces identical refits within this fixed period. The compact LSTM records greater run-to-run variation and lower AP than the count models. LogBERT provides a transformer-based, self-supervised sequence alternative [22]. TABLE VI: BGL chronological sensitivity over five seeds. Parser: 70/30 split; rolling: next 10% after each cutoff. Test anomaly Count/XGB LSTM Semantic/XGB Panel Setting rate AP AP AP Parser Compact s=.3s=.3 10.1% .460±.000 â .741±.054 Parser Compact s=.4s=.4 10.1% .448±.000 â .676±.086 Parser Compact s=.5s=.5 10.1% .448±.000 â .631±.109 Parser Drain3 s=.4s=.4 10.1% .432±.000 â .617±.177 Rolling 50â 60% 1.3% .902±.000 .475±.147 .838±.047 Rolling 60â 70% 10.8% .376±.000 .201±.048 .663±.077 Rolling 70â 80% 13.2% .588±.000 .373±.248 .760±.103 Rolling 80â 90% 5.9% .332±.000 .273±.130 .776±.079 V-B RQ2: Parser and temporal sensitivity Table VI reports the RQ2 experiments. At the fixed 70/30 split semantic/XGB records a higher mean AP than count/XGB in all four parser configurations; its mean AP spans 0.617â0.741 and the Drain3 run has a standard deviation of 0.177. With the cutoff and remaining settings fixed, parser choice spans 0.124 in semantic/XGB mean AP while preserving the ordering of the two configurations. The rolling origins vary more. Semantic/XGB records the higher mean AP in the periods beginning at 60%, 70%, and 80%, while count/XGB is higher at 50%, reaching 0.902 AP against 0.838. Test prevalence ranges from 1.3% to 13.2%, and every later training prefix contains the earlier one. Prefix length, calendar period, anomaly rate, and carried-over vocabulary state move together across these rows, so row-to-row differences reflect their joint variation. LogSD also evaluates BGL chronologically, though its self-supervised model and split differ from ours [23]. The two panels provide complementary views: the fixed-cutoff panel isolates parser sensitivity, while the rolling-origin panel characterizes performance as training history, calendar period, prevalence, and vocabulary state evolve together. SemParser showed that richer parsing semantics can change downstream analytics [24] and Khan and colleagues report the same sensitivity for deep detectors [7]. Even syntax-oriented thresholds move BGL AP comparably here. All parser runs use the same raw BGL release in recorded order, with provenance through Loghub [16], and the 50â60% rolling period is the single interval in which the order changes. TABLE VII: Component-level cost over five seeds. Throughput covers classifier-only inference; size covers the serialized classifier. System Configuration Fit (s) Infer. (k/s) Size (MB) HDFS Count/LR 2.6± 1.1 6771± 2151 .002 HDFS Count/RF 17.8± 2.1 312± 91 2.234 HDFS Count/XGB 3.8± 1.8 660± 94 .423 HDFS Sequence/LSTM 309.3± 13.2 37± 4 .116 HDFS Semantic/LR 31.4± 21.3 1257± 113 .004 HDFS Semantic/XGB 9.4±.8 696± 189 .416 BGL Count/LR 16.5± 4.4 70± 31 .069 BGL Count/RF 38.9± 18.6 29± 3 10.869 BGL Count/XGB 10.2± 3.2 222± 74 .372 BGL Sequence/LSTM 37.8± 3.9 15± 2 .386 BGL Semantic/LR 6.1± 1.7 1812± 754 .004 BGL Semantic/XGB 1.6±.3 505± 28 .684 Shared Word2vec fit .4±.0 â â Semantic transform: HDFS/BGL 1.0/1.4 â â Frozen BGL semantic fit 3.0±.5 â â BGL parsing, one run 54 â â V-C Where time and storage are spent Table VII separates classifier cost from representation work. On HDFS, count/LR is the smallest serialized classifier and processes 6.77 million sessions per second on prepared features on the archived hardware; count/XGB occupies 0.423 MB, and the LSTM state is smaller at 0.116 MB although its fit takes 309.3 seconds against 3.8 for XGBoost. Semantic classifiers achieve the highest classifier-only throughput on BGL; Word2vec fitting and semantic transformation are reported as separate pipeline stages. Storage and runtime emphasize different components: semantic LR uses a four-kilobyte classifier together with embedding and IDF resources, the compact LSTM has substantial recurrent-computation cost, and raw BGL parsing adds 54 seconds. Yu and colleagues similarly found lightweight classical methods competitive when preprocessing and training costs are included [25]. Component-level accounting makes these resource trade-offs explicit. TABLE VIII: Cross-system XGBoost target-catalog visibility over five seeds. Union corpus and target IDF are offline transductive; budgets use tie-aware allocation. Word2vec IDF AP P@1% R@1% P@5% R@5% HDFSâ Source Source .191±.107 .242±.160 .024±.016 .220±.174 .109±.086 Source Target .187±.084 .259±.163 .026±.016 .253±.163 .125±.081 Union Source .297±.136 .443±.266 .044±.026 .293±.021 .145±.010 Union Target .325±.159 .446±.312 .044±.031 .318±.122 .157±.060 BGLâ Source Source .089±.066 .151±.228 .051±.078 .075±.057 .128±.097 Source Target .132±.100 .248±.297 .085±.102 .095±.067 .162±.114 Union Source .039±.017 .032±.018 .011±.006 .025±.007 .043±.011 Union Target .142±.137 .201±.233 .068±.080 .129±.121 .221±.207 V-D RQ3: Target-catalog visibility and feature coverage Count vectors and event-ID sequences have system-specific dimensions, so cross-system use needs a template mapping, retraining, or an adaptation model. Template2Vec has a common dimension and applies directly, exposing the choices in Table VIII. That table examines cross-system feature coverage and detection scores under different template-corpus and IDF visibility conditions in both directions. Between the strictly inductive source-only cell and the fully transductive union-corpus, target-IDF cell, XGBoost mean AP changes from 0.191 to 0.325 for HDFS-to-BGL and from 0.089 to 0.142 for BGL-to-HDFS. The intermediate cells show direction-dependent interactions between corpus coverage and IDF weighting: supplying only target IDF or only the union corpus is associated with a higher mean AP in one direction and a lower one in the other, and the union-corpus, source-IDF cell for BGL-to-HDFS falls to 0.039, below its own source-only value. At the union-corpus, target-IDF cell, a 1% review budget retrieves 4.4% of BGL anomalies and 6.8% of HDFS anomalies on average; a 5% budget corresponds to recall of 15.7% and 22.1%, with precision of 31.8% and 12.9%. These results describe the review queues for each target population under its stated access condition; Table VIII reports the corresponding dispersion. VI Discussion and threats to validity VI-A Reading protocol-sensitive results and their dispersion The ordering of six fixed configurations changes with the protocol, consistent with [5]. HDFS group holdout evaluates unseen count profiles, whereas chronological BGL evaluates a later trace segment with a frozen representation. These distinct test conditions matter when comparing benchmark scores [9]. Changes from random splitting are therefore interpreted within each datasetâs corresponding protocol. HDFS dispersion combines variation in test composition and model fitting, while chronological BGL dispersion reflects refitting within one fixed period (Section IV). We report differences in mean scores together with their across-run dispersion and protocol conditions. Four reporting fields make these comparisons interpretable: the share of test patterns already seen in training for random splits; the parser snapshot and at least one neighboring period for chronological splits; the representation corpus and IDF source for cross-system semantic results; and budget recall with run-to-run dispersion. VI-B Preprocessing, parser state, and label semantics The parser panel measures downstream detection sensitivity, with a 0.124 span in semantic/XGB mean AP and an unchanged leading configuration. Template-level ground-truth annotations are unavailable for this BGL evaluation. HDFS labels variable-length block sessions, whereas BGL labels fixed 100-line windows anomalous after one alert. Review-budget precision is therefore measured per block session and per window. New count/sequence templates map to unk, while Template2Vec uses known frozen-vocabulary words or zero. The 418 later templates lacking usable words identify the coverage boundary of the frozen semantic representation. VI-C Temporal estimates as compound scenarios The primary BGL experiment evaluates refitting at a 70/30 cutoff, and four rolling origins extend the analysis across nested training prefixes and subsequent test periods. The resulting variation is consistent with the sensitivity to training selection, class distribution, and noise reported by Le and Zhang [11]. The earliest-period reversal shows why evaluation across periods complements repeated fitting within one period. A factorial extension incorporating methods such as LogETA [26] could separate training-prefix and calendar-period effects. VI-D Cross-system results as a coverage diagnostic The two directions expose the interaction between corpus structure and representation access. In the HDFS-to-BGL source-only condition, 91.7% of target templates and 67.3% of target sessions map to zero. Union-corpus training adds coordinates for those words, and target IDF changes their weighting. The intermediate conditions show direction-dependent interactions between these choices. Related work sits at other points on this axis: MetaLog adapts with a small labeled target set [27], and LogDLR learns domain-invariant representations through reconstruction and domain-adversarial training [13]. Template2Vec keeps the classifier family, hyperparameters, source labels, and representation dimension fixed while varying catalog visibility and IDF source; each condition generates its own feature vectors and fits a separate classifier. LogGT uses a heterogeneous graph before domain-adapted transfer [28]. HDFS contributes 29 templates from regular block sessions, whereas BGL contributes 2,746 templates from a system-wide stream, providing contrasting coverage conditions. Additional systems would extend this analysis across other logging styles and failure modes. VI-E Model scope and cost boundaries The six fixed configurations cover count, sequence, and semantic representations, enabling the same protocol changes to be examined across distinct feature families. Recent comparative studies provide complementary coverage of detector families [8] and benchmark pipelines [5]. The LSTM uses one layer with 64 hidden units and at most five epochs. Evaluating RT-Log [29] and LogLLM [30] under matched split indices, access conditions, and review budgets would extend the model coverage. Parser and rolling-origin sensitivity results were computed in a separate Apple environment; cost measurements in Table VII use the documented archived Xeon environment. VI-F Security scope and threat model The evaluation measures failure- and alert-oriented anomaly screening using released HDFS failure labels and BGL alert flags. CAShift provides a complementary setting with attack scenarios under normality shift [31]. The current threat model assumes trusted collection, timestamps, parsing, and labels. Because vocabulary and detector inputs derive from the scored stream, an adversary influencing an unlabeled target catalog could alter vocabulary and IDF before scoring without changing a label. An adversarial extension would examine poisoning, injection, tampering, evasion, label corruption, and parser compromise through controlled perturbations, catalog-provenance checks, and an explicit attacker model. VI-G Documentation and reproducibility status The paper documents the protocols and the values they produce: split construction, parser thresholds and snapshot points, temporal cutoffs, high-level tokenization and vocabulary handling, the fixed hyperparameters, visible catalog, and cost boundaries of Tables I, I, and VII, and the review-budget convention. The HDFS and BGL corpora are public and distributed with their preprocessing provenance through Loghub [16]. Release of a versioned experimental replication package containing split indices, per-seed predictions, and table-regeneration scripts is planned as future work. VII Conclusion Protocol choice changes the comparison among the six fixed configurations: count/LR records the highest grouped mean AP on HDFS, semantic XGBoost the highest chronological mean AP on BGL, and several configurations form a near-ceiling cluster under random splitting. At a fixed BGL cutoff, parser choice spans 0.124 in semantic XGBoost mean AP while preserving its lead over count XGBoost; the earliest rolling period reverses that ordering. The cross-system ablation shows direction-dependent interactions between catalog coverage and IDF weighting. Offline transductive access to unlabeled target templates is associated with changes in reported AP and early review queues, making pre-scoring information access an explicit experimental variable. Together, these findings identify split construction, parser state, catalog visibility, review budget, and cost boundary as essential context for interpreting results on HDFS and BGL. Reporting these properties alongside scores supports protocol-aware comparison of detector accuracy and component costs. Acknowledgment Codex(OpenAI) was used for language editing, experiment running, and assistance in script development. The authors reviewed the manuscripts, verified the reported results and citations, and take responsibility for the final text. References References [1] W. Xu, L. Huang, A. Fox, D. Patterson, and M. I. Jordan, âDetecting large-scale system problems by mining console logs,â in Proc. ACM SIGOPS 22nd Symp. Operating Systems Principles, 2009, p. 117â132, doi: 10.1145/1629575.1629587. [2] S. He, J. Zhu, P. He, and M. R. Lyu, âExperience report: System log analysis for anomaly detection,â in Proc. IEEE 27th Int. Symp. Software Reliability Engineering, 2016, p. 207â218, doi: 10.1109/ISSRE.2016.21. [3] M. Du, F. Li, G. Zheng, and V. Srikumar, âDeepLog: Anomaly detection and diagnosis from system logs through deep learning,â in Proc. ACM SIGSAC Conf. Computer and Communications Security, 2017, p. 1285â1298, doi: 10.1145/3133956.3134015. [4] W. Meng, Y. Liu, Y. Zhu, S. Zhang, D. Pei et al., âLogAnomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs,â in Proc. 28th Int. Joint Conf. Artificial Intelligence, 2019, p. 4739â4745, doi: 10.24963/ijcai.2019/658. [5] O. SedlĂĄÄek, M. ĆœĂĄdnĂk, and V. BartoĆĄ, âAnomaly detection in log data: A comparative study,â in Proc. Int. Conf. Network and Service Management (CNSM), 2025, p. 1â7, doi: 10.23919/CNSM67658.2025.11297503. [6] P. He, J. Zhu, Z. Zheng, and M. R. Lyu, âDrain: An online log parsing approach with fixed depth tree,â in Proc. IEEE Int. Conf. Web Services, 2017, p. 33â40, doi: 10.1109/ICWS.2017.13. [7] Z. A. Khan, D. Shin, D. Bianculli, and L. C. Briand, âImpact of log parsing on deep learning-based anomaly detection,â Empirical Software Engineering, vol. 29, no. 6, Art. no. 139, 2024, doi: 10.1007/s10664-024-10533-w. [8] S. Ali, C. Boufaied, D. Bianculli, P. Branco, and L. Briand, âA comprehensive study of machine learning techniques for log-based anomaly detection,â Empirical Software Engineering, vol. 30, no. 5, Art. no. 129, 2025, doi: 10.1007/s10664-025-10669-3. [9] M. Landauer, S. Onder, F. Skopik, and M. Wurzenberger, âDeep learning for anomaly detection in log data: A survey,â Machine Learning with Applications, vol. 12, Art. no. 100470, 2023, doi: 10.1016/j.mlwa.2023.100470. [10] M. Landauer, F. Skopik, and M. Wurzenberger, âA critical review of common log data sets used for evaluation of sequence-based anomaly detection techniques,â Proc. ACM Softw. Eng., vol. 1, no. FSE, Art. no. 61, p. 1354â1375, 2024, doi: 10.1145/3660768. [11] V.-H. Le and H. Zhang, âLog-based anomaly detection with deep learning: How far are we?â in Proc. 44th Int. Conf. Software Engineering, 2022, p. 1356â1367, doi: 10.1145/3510003.3510155. [12] R. Chen, S. Zhang, D. Li, Y. Zhang, F. Guo, W. Meng, D. Pei, Y. Zhang, X. Chen, and Y. Liu, âLogTransfer: Cross-system log anomaly detection for software systems with transfer learning,â in Proc. IEEE ISSRE, 2020, p. 37â47, doi: 10.1109/ISSRE5003.2020.00013. [13] J. Zhou et al., âLogDLR: Unsupervised cross-system log anomaly detection through domain-invariant latent representation,â IEEE Trans. Dependable Secure Comput., vol. 22, no. 4, p. 4456â4471, 2025, doi: 10.1109/TDSC.2025.3548050. [14] T. Mikolov, K. Chen, G. Corrado, and J. Dean, âEfficient estimation of word representations in vector space,â arXiv:1301.3781, 2013, doi: 10.48550/arXiv.1301.3781. [15] A. Oliner and J. Stearley, âWhat supercomputers say: A study of five system logs,â in Proc. 37th Annu. IEEE/IFIP Int. Conf. Dependable Systems and Networks, 2007, p. 575â584, doi: 10.1109/DSN.2007.103. [16] J. Zhu, S. He, P. He, J. Liu, and M. R. Lyu, âLoghub: A large collection of system log datasets for AI-driven log analytics,â in Proc. IEEE ISSRE, 2023, p. 355â366, doi: 10.1109/ISSRE59848.2023.00071. [17] X. Zhang, Y. Xu, Q. Lin, B. Qiao, H. Zhang et al., âRobust log-based anomaly detection on unstable log data,â in Proc. ACM ESEC/FSE, 2019, p. 807â817, doi: 10.1145/3338906.3338931. [18] X. Han and S. Yuan, âUnsupervised cross-system log anomaly detection via domain adaptation,â in Proc. 30th ACM Int. Conf. Information and Knowledge Management, 2021, p. 3068â3072, doi: 10.1145/3459637.3482209. [19] L. Breiman, âRandom forests,â Mach. Learn., vol. 45, no. 1, p. 5â32, 2001, doi: 10.1023/A:1010933404324. [20] F. Pedregosa et al., âScikit-learn: Machine learning in Python,â Journal of Machine Learning Research, vol. 12, p. 2825â2830, 2011. [21] T. Chen and C. Guestrin, âXGBoost: A scalable tree boosting system,â in Proc. ACM KDD, 2016, p. 785â794, doi: 10.1145/2939672.2939785. [22] H. Guo, S. Yuan, and X. Wu, âLogBERT: Log anomaly detection via BERT,â in Proc. Int. Joint Conf. Neural Networks, 2021, p. 1â8, doi: 10.1109/IJCNN52387.2021.9534113. [23] Y. Xie, H. Zhang, and M. A. Babar, âLogSD: Detecting anomalies from system logs through self-supervised learning and frequency-based masking,â Proc. ACM Software Engineering, vol. 1, no. FSE, Art. no. 93, p. 2098â2120, 2024, doi: 10.1145/3660800. [24] Y. Huo, Y. Su, C. Lee, and M. R. Lyu, âSemParser: A semantic parser for log analytics,â in Proc. ICSE, 2023, p. 881â893, doi: 10.1109/ICSE48619.2023.00082. [25] B. Yu, J. Yao, Q. Fu, Z. Zhong, H. Xie, Y. Wu, Y. Ma, and P. He, âDeep learning or classical machine learning? An empirical study on log-based anomaly detection,â in Proc. IEEE/ACM 46th Int. Conf. Software Engineering, 2024, Art. no. 35, p. 1â13, doi: 10.1145/3597503.3623308. [26] K. Gong, S. Luo, L. Pan, L. Zhang, Y. Zhang, and H. Yu, âLogETA: Time-aware cross-system log-based anomaly detection with inter-class boundary optimization,â Future Generation Computer Systems, vol. 157, p. 16â28, 2024, doi: 10.1016/j.future.2024.03.028. [27] C. Zhang, T. Jia, G. Shen, P. Zhu, and Y. Li, âMetaLog: Generalizable cross-system anomaly detection from logs with meta-learning,â in Proc. IEEE/ACM 46th Int. Conf. Software Engineering, 2024, Art. no. 154, p. 1â12, doi: 10.1145/3597503.3639205. [28] P. Wang, X. Zhang, Z. Cao, W. Xu, and W. Li, âLogGT: Cross-system log anomaly detection via heterogeneous graph feature and transfer learning,â Expert Systems with Applications, vol. 251, Art. no. 124082, 2024, doi: 10.1016/j.eswa.2024.124082. [29] P. Jia, S. Cai, B. C. Ooi, P. Wang, and Y. Xiong, âRobust and transferable log-based anomaly detection,â Proc. ACM Manag. Data, vol. 1, no. 1, Art. 64, p. 1â26, 2023, doi: 10.1145/3588918. [30] W. Guan, J. Cao, S. Qian, J. Gao, and C. Ouyang, âLogLLM: Log-based anomaly detection using large language models,â arXiv:2411.08561, 2024, doi: 10.48550/arXiv.2411.08561. [31] J. Yu et al., âCAShift: Benchmarking log-based cloud attack detection under normality shift,â Proc. ACM Softw. Eng., vol. 2, no. FSE, Art. no. FSE076, p. 1687â1709, 2025, doi: 10.1145/3729346.