Paper deep dive
LEXIC: Lightweight Eye-tracking eXtension via Injected Complexity
Sumin Lee, Kyeonghun Kim, Subeen Lee, Jiwon Yang, Tien Nguyen, Ken Ying-Kai Liao, Nam-Joon Kim
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/10/2026, 4:17:31 AM
Summary
This paper introduces LEXIC, a lightweight framework for enhancing gaze-only eye-tracking models to predict reading comprehension. By injecting precomputed word-level difficulty signals (GPT-2 surprisal, word frequency, word length) into the AhnCNN baseline via direct concatenation (LEXIC-Concat) or a residual mechanism (LEXIC-Res), the authors achieve statistically significant AUROC improvements on the OneStop dataset from the EyeBench benchmark. The study highlights that direct concatenation generalizes better to unseen readers, while the residual mechanism suffers from a reader-transfer bottleneck.
Entities (11)
Relation Signals (10)
EyeBench â evaluatesdataset â OneStop
confidence 96% · EyeBench v1.0 [14] also evaluates text-aware models... We use the OneStop reading comprehension task from EyeBench v1.0
EyeBench â measurestask â Reading Comprehension
confidence 95% · On the recent EyeBench benchmark, predicting reading comprehension from eye movements exposes a stark gap
LEXIC-Concat â usesbackbone â AhnCNN
confidence 95% · Building on the EyeBench AhnCNN baseline, LEXIC-Base, we propose two mechanisms... direct concatenation, LEXIC-Concat
LEXIC-Res â usesbackbone â AhnCNN
confidence 95% · Building on the EyeBench AhnCNN baseline, LEXIC-Base, we propose two mechanisms... residual mechanism, LEXIC-Res
LEXIC-Res â injectsfeature â GPT-2 Surprisal
confidence 94% · inject three precomputed word-level difficulty signals, GPT-2 surprisal, word frequency, and word length, into the per-fixation input: ... a residual mechanism, LEXIC-Res
LEXIC-Concat â injectsfeature â GPT-2 Surprisal
confidence 94% · inject three precomputed word-level difficulty signals, GPT-2 surprisal, word frequency, and word length, into the per-fixation input: direct concatenation, LEXIC-Concat
LEXIC-Concat â injectsfeature â Word Length
confidence 93% · inject three precomputed word-level difficulty signals, GPT-2 surprisal, word frequency, and word length
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:On the recent EyeBench benchmark, predicting reading comprehension from eye movements exposes a stark gap: text-aware models using pretrained language models reach 56--63% AUROC, while gaze-only models operate at chance. We ask how far a gaze-only model can be pushed by lightweight, language-model-free conditioning. Building on the EyeBench AhnCNN baseline, LEXIC-Base, we propose two mechanisms to inject three precomputed word-level difficulty signals, GPT-2 surprisal, word frequency, and word length, into the per-fixation input: direct concatenation, LEXIC-Concat, and a residual mechanism, LEXIC-Res, where a small head predicts typical-reader gaze response and the encoder is conditioned on the deviation. On the OneStop reading comprehension task, with K=5 seed-ensemble training across ten folds, both mechanisms produce statistically consistent AUROC gains on Unseen Text, +1.8 to +2.2 percentage points, Wilcoxon p <= 0.065. LEXIC-Concat additionally lifts Unseen Reader by +2.9 percentage points, p = 0.010. We trace an architectural boundary in LEXIC-Res on Unseen Reader, +1.8 percentage points, p = 0.19, to the prediction head being calibrated to training readers, transferring imperfectly to out-of-distribution readers.
Tags
Links
- Source: https://arxiv.org/abs/2607.08152v1
- Canonical: https://arxiv.org/abs/2607.08152v1
Trouble viewing inline? Open PDF directly â
Full Text
20,786 characters extracted from source content.
Expand or collapse full text
LEXIC: Lightweight Eye-tracking eXtension via Injected Complexity Sumin Lee Kyeonghun Kim Subeen Lee Jiwon Yang Seoul National University OUTTA Seoul National University Seoul National University cirtuare@snu.ac.kr kyeonghun.kim@outta.ai 5584sb@snu.ac.kr jwyang29@snu.ac.kr Tien Nguyen Ken Ying-Kai Liao Nam-Joon Kimâ Seoul National University NVIDIA Seoul National University tien_noob@snu.ac.kr kenyingkail@nvidia.com knj01@snu.ac.kr Abstract On the recent EyeBench benchmark, predicting reading comprehension from eye movements exposes a stark gap: text-aware models using pretrained language models reach 5656â63%63\% AUROC, while gaze-only models operate at chance. We ask how far a gaze-only model can be pushed by lightweight, language-model-free conditioning. Building on the EyeBench AhnCNN baseline (LEXIC-Base), we propose two mechanisms to inject three precomputed word-level difficulty signals (GPT-2 surprisal, word frequency, and word length) into the per-fixation input: direct concatenation (LEXIC-Concat) and a residual mechanism (LEXIC-Res) where a small head predicts typical-reader gaze response and the encoder is conditioned on the deviation. On the OneStop reading comprehension task, with K=5K=5 seed-ensemble training across ten folds, both mechanisms produce statistically consistent AUROC gains on Unseen Text (+1.8+1.8 to +2.2+2.2p, Wilcoxon pâ€0.065p†0.065); LEXIC-Concat additionally lifts Unseen Reader by +2.9+2.9p (p=0.010p=0.010). We trace an architectural boundary in LEXIC-Res on Unseen Reader (+1.8+1.8p, p=0.19p=0.19) to the prediction head being calibrated to training readers, transferring imperfectly to out-of-distribution readers. â footnotetext: â Corresponding author I Introduction Eye movements collected during reading reflect cognitive processes ranging from lexical access to discourse comprehension[5]. Predicting reader-level cognitive states or readerâtext interactions from these movements has both scientific value for refining cognitive theories and practical value for adaptive interfaces. The EyeBench benchmark [14] standardizes these prediction tasks over six datasets, evaluating models across out-of-distribution generalization regimes on corpora such as OneStop [2] and MECO [15]. On the readerâtext tasks of EyeBenchâspecifically binary comprehension classificationâpredicting cognitive processing has become an active interdisciplinary target [8, 7]. The leaderboard exposes a sharp dichotomy: models that ingest text content via large pretrained language models (PLMs) reach AUROC in the 5656â63%63\% range [19, 13], whereas pure gaze-only models operate at or near chance (49.649.6â50.0%50.0\%) [1, 14]. Models that augment gaze with linguistic features without a language model, like BEyeLSTM [12], sit between the two regimes (52.5%52.5\%). Comprehension depends largely on the content being read; thus, gaze-only models lack text-side signal. We ask: can a gaze-only model be pushed above chance by injecting lightweight text signals, rather than the full text? We use three precomputed word-level difficulty signals: GPT-2 [10] surprisal, English word frequency [17], and word length. All three are computed offline once per dataset and merged onto fixation rows; no language-model forward pass is invoked at inference. The architectural changes to the gaze-only AhnCNN backbone are minimal: three additional input channels (concatenation), or a small auxiliary head predicting typical-reader gaze, whose residual against the observed gaze augments the input (residual injection). Our contributions are: âą We propose two lightweight mechanisms for injecting precomputed word-level difficulty features into the gaze-only AhnCNN baseline of EyeBench, requiring no language-model inference at deployment. âą Across ten cross-validation folds and three regimes, with a K=5K=5 seed-ensemble, both mechanisms yield statistically significant AUROC improvements (Wilcoxon pâ€0.065p†0.065) of 1.8 to 2.9 percentage points on Unseen Text and (for concatenation) Unseen Reader, reproducing the baseline within 0.30.3p. âą We characterize an architectural boundary condition: the residual mechanismâs typical-reader prediction head, calibrated to training readers, attenuates its gain in the Unseen Reader regime â clarifying when the simpler concatenation mechanism is preferable. Figure 1: Proposed LEXIC variants: (a) LEXIC-Base consumes a 4-channel fixation tensor. (b) LEXIC-Concat appends precomputed difficulty signals (surprisal, frequency, word length), yielding a 7-channel input. (c) LEXIC-Res uses a NormativeGazeHead (MLP 3â32â23â 32â 2) to predict typical-reader gaze g g; the residual rt=gtobsâstopgradâ(g^t)r_t=g_t^obs-stopgrad( g_t) is concatenated to base channels. All variants share the downstream backbone. I Related Work Predictive Modeling on Eye Movements: Eye movements have long been studied as indicators of lexical access, attention allocation, and comprehension during reading [11]. Recent computational work has moved from handcrafted gaze features to neural models over fixation sequences and scanpaths [1]. Ahn et al. [1] introduced architectures adopted by EyeBench [14] as the AhnCNN and AhnRNN baselines; these remain strong gaze-only baselines but operate near chance on OneStop reading comprehension. Reich et al. [12] introduced BEyeLSTM, ingesting fixation-level and word-level linguistic features, sitting between the gaze-only and text-aware regimes. EyeBench v1.0 [14] also evaluates text-aware models integrating PLMs [19, 13], which dominate the leaderboard at high inference costs. Lightweight Linguistic Features: Word-level surprisal, frequency, and length are standard psycholinguistic predictors of fixation duration. Surprisal traces back to Haleâs expectation-based account [4] and its logarithmic relationship with reading time [16]; frequency and length have independent and well-documented effects on fixation durations and skipping probabilities [6]. We use these signals because they strongly correlate with fixation behavior while remaining computationally lightweight: all features are precomputed offline and reused at inference without any runtime language-model execution. Position of This Work: Rather than maximizing leaderboard performance, we study how much performance gain can be obtained under a strict lightweight-inference constraint. Our constraint is no language-model forward pass at inference; we examine the lift achievable from a chance-level baseline under that constraint. I Method I-A Backbone: LEXIC-Base We use the EyeBench AhnCNN baseline [1, 14] as our backbone. LEXIC-Base consumes a per-fixation tensor of four channelsâduration, pupil size, and position (x, y)âthrough three 1D convolutional layers (kernel size 3, ReLU). A max-pooling layer, dropout, and flattening lead to a three-Linear-layer fully connected classifier head, producing logits. We preserve the architecture and optimization protocol of the published baseline [14]. I-B Difficulty Features For each word we use three difficulty signals precomputed by the EyeBench pipeline: âą gpt2_surprisal: surprisal under GPT-2 (base) [10]. âą wordfreq_frequency: English frequency from wordfreq [17]. âą word_length: character length of the word. These signals are defined at the word level and are never regenerated at inference. I-C Concatenation Mechanism (LEXIC-Concat) The simplest injection appends the three difficulty signals as additional channels, expanding the input width from 4 to 7 via the inherited fixation_dim mechanism. We apply a nan_to_num pass in the forward step, replacing any NaN entries with zero to prevent NaNs on unaligned punctuation. I-D Residual Mechanism (LEXIC-Res) This mechanism interprets gaze as the deviation from what a typical reader would produce given local difficulty. A NormativeGazeHead (MLP 3â32â23â 32â 2) maps difficulty tââ3d_t ^3 to typical-reader prediction g^tââ2 g_t ^2 (duration and saccade amplitude). The residual is: rt=gtobsâstopgradâ(g^t),r_t\;=\;g_t^obs\;-\;stopgrad ( g_t ), (1) where gtobsââ2g_t^obs ^2 is the observed value. The CNN input is the concatenation of the 4 base channels and 2 residual channels (6 total). The NormativeGazeHead is shaped through an auxiliary MSE loss: âaux=1|0|ââtâ0âg^tâgtobsâ22.L_aux= 1|T_0| _t _0 g_t-g_t^obs _2^2. (2) This loss is restricted to Incorrect-answer trials (0T_0). The total loss is: â=âcls+λââaux,λ=0.3.L\;=\;L_cls\;+\;λ\,L_aux, λ=0.3. (3) λ=0.3λ=0.3 was inherited from the original residual head specification and held fixed across all experiments. I-E Inference Cost Both mechanisms perform a single backbone forward pass. LEXIC-Res adds a forward pass through a tiny 3â32â23â 32â 2 MLP. Because the difficulty signals are text-side features computed once per stimulus and shared across readers, no language model is invoked at inference. IV Experimental Setup Dataset and Task: We use the OneStop reading comprehension task from EyeBench v1.0 [14], based on the OneStop Eye Movements corpus [2], focusing on binary classification of whether a reader correctly answered a comprehension question ([Incorrect, Correct]). As standardized by EyeBench [14], OneStop comprises 180 participants reading passages totaling 19,428 words (1.1M fixations). The task contains 9,718 trial instances (54 passages per participant). We use the Ordinary Reading subset. Cross-Validation Protocol: EyeBench defines ten cross-validation folds evaluated under three out-of-distribution regimes: Unseen Text (unseen texts, seen readers), Unseen Reader (seen texts, unseen readers), and Unseen Both. We evaluate all ten folds. For each fold, we train a K=5K=5 independent seed ensemble, averaging per-trial probabilities. Run-level selection is based on validation cross-entropy loss. Metrics and Threshold Policy: AUROC is our primary threshold-free metric, directly comparable to the published AhnCNN leaderboard (49.7%49.7\%). Balanced accuracy at 0.5 (balacc0.5) enables comparison with the leaderboardâs balanced accuracy (50.0%50.0\%). Balanced accuracy at a tuned threshold (balaccÏ) serves as a secondary metric. Per fold, Ï is selected to maximize balanced accuracy on validation trials pooled across the three regimes, and applied unchanged to each regimeâs test trials. Pooling stabilizes threshold selection given limited per-regime validation size. We report paired deltas, sign consistency, and Wilcoxon signed-rank test p-values [18, 3]. V Results Baseline Reproduction: LEXIC-Base closely reproduces the published AhnCNN baseline on EyeBench [14]: AUROC 0.4890.489â0.5120.512 across regimes (leaderboard: 0.4970.497) and balacc0.5 of 0.500 (leaderboard: 0.5000.500), validating our pipeline. Main Results: Table I shows absolute performance. Both mechanisms lift AUROC above the chance baseline: to 0.510.51â0.560.56. Under threshold tuning, balaccÏ increases to 0.510.51â0.540.54, reaching the same performance tier as BEyeLSTM (published 52.5%52.5\% [14, 12]), which consumes significantly richer global and linguistic features. TABLE I: Test AUROC and balanced accuracy at tuned threshold (balaccÏ), reported as mean ± standard deviation across ten folds (K=5K=5 seed ensemble). Best means are bolded. Model Unseen Text Unseen Reader Unseen Both AUROC LEXIC-Base 0.495±0.0190.495± 0.019 0.489±0.0280.489± 0.028 0.512±0.0540.512± 0.054 LEXIC-Concat 0.513±0.0250.513± 0.025 0.518±0.0490.518± 0.049 0.516±0.0800.516± 0.080 LEXIC-Res 0.517±0.0270.517± 0.027 0.507±0.0260.507± 0.026 0.557±0.0750.557± 0.075 Balanced accuracy at tuned threshold (balaccÏ) LEXIC-Base 0.500±0.0120.500± 0.012 0.484±0.0170.484± 0.017 0.476±0.0390.476± 0.039 LEXIC-Concat 0.511±0.0230.511± 0.023 0.521±0.0320.521± 0.032 0.532±0.0380.532± 0.038 LEXIC-Res 0.513±0.0220.513± 0.022 0.507±0.0200.507± 0.020 0.536±0.0570.536± 0.057 Paired Deltas and Sign Consistency: Table I details the paired within-fold gains. LEXIC-Concat shows significant AUROC gains on Unseen Text (+1.82+1.82p, 9/10, p=0.065p=0.065) and Unseen Reader (+2.92+2.92p, 8/10, p=0.010p=0.010). LEXIC-Res achieves a significant AUROC gain on Unseen Text (+2.18+2.18p, 9/10, p=0.010p=0.010), but a weaker trend on Unseen Reader (+1.84+1.84p, 7/10, p=0.19p=0.19). Under threshold tuning, balaccÏ deltas are uniformly larger and statistically significant at pâ€0.05p†0.05 in five of six cells. TABLE I: Paired delta versus baseline within fold. The sign indicates positive-delta folds out of ten. Bold values indicate statistically significant positive improvements (Î>0 >0 and pâ€0.10p†0.10). Mechanism Regime Î (p) sign p AUROC LEXIC-Concat Unseen Text +1.82+1.82 9/109/10 0.0650.065 Unseen Reader +2.92+2.92 8/108/10 0.0100.010 Unseen Both +0.33+0.33 5/105/10 1.0001.000 LEXIC-Res Unseen Text +2.18+2.18 9/109/10 0.0100.010 Unseen Reader +1.84+1.84 7/107/10 0.1930.193 Unseen Both +4.44+4.44 6/106/10 0.2750.275 Balanced accuracy at tuned threshold (balaccÏ) LEXIC-Concat Unseen Text +1.15+1.15 9/109/10 0.0840.084 Unseen Reader +3.65+3.65 9/109/10 0.0200.020 Unseen Both +5.51+5.51 9/109/10 0.0200.020 LEXIC-Res Unseen Text +1.27+1.27 9/109/10 0.0840.084 Unseen Reader +2.28+2.28 9/109/10 0.0200.020 Unseen Both +5.96+5.96 7/107/10 0.0490.049 Boundary Condition: LEXIC-Res on Unseen Reader: LEXIC-Res exhibits a clear Unseen Text-versus-Unseen Reader asymmetry (AUROC gain of +1.84+1.84p vs. +2.18+2.18p). The gain on Unseen Reader is roughly half the LEXIC-Concat gain (+2.92+2.92p). We trace this transfer bottleneck to the typical-reader prediction head in LEXIC-Res, which is calibrated to a population-averaged gaze response on training readers. For unseen readers, the residual rt=gtobsâg^tr_t=g_t^obs- g_t mixes typical-reader gaze with an out-of-distribution readerâs observed gaze, collapsing toward noise. The direct input injection in LEXIC-Concat bypasses this bottleneck. VI Discussion Closing Part of the Gap: Without invoking a language model at inference, our lightweight LEXIC variants reach performance tiers comparable to BEyeLSTM (0.510.51â0.560.56 AUROC). This demonstrates how far gaze-only models can be pushed without heavy PLM architectures. Ranking versus Calibration: While balacc0.5 stays near chance, balaccÏ improvements are significant. Lightweight injection shifts the ranking of trials by predicted probability without affecting default-threshold calibration. Deployments with a fixed threshold require calibration (e.g., Platt scaling [9]). Practical Recommendation: LEXIC-Concat is simpler and statistically more robust across generalization regimes, making it the recommended default when reader generalization is expected. LEXIC-Res is most effective when the reader population overlaps with training. Limitations: We only cover the OneStop reading comprehension task and evaluate on a single backbone (LEXIC-Base). Extending our mechanisms to other gaze backbones (AhnRNN, BEyeLSTM) and alternative EyeBench reading comprehension tasks (SB-SAT, PoTeC) remain future priorities. VII Conclusion We presented two lightweight mechanisms for injecting precomputed word-level difficulty signals into a gaze-only CNN baseline. Without PLM inference at deployment, both variants produce statistically consistent gains on the OneStop reading comprehension task. The direct concatenation mechanism (LEXIC-Concat) is robust across regimes, while the residual mechanism (LEXIC-Res) is constrained by a reader-transfer bottleneck. This quantifies the value of lightweight features in gaze-based cognitive modeling. References [1] S. Ahn, C. Kelton, A. Balasubramanian, and G. Zelinsky (2020) Towards predicting reading comprehension from gaze behavior. In ACM Symposium on Eye Tracking Research and Applications, p. 1â5. External Links: Document Cited by: §I, §I, §I-A. [2] Y. Berzak, J. Malmaud, O. Shubi, Y. Meiri, E. Lion, and R. Levy (2025) OneStop: a 360-participant english eye tracking dataset with different reading regimes. Scientific Data 12 (1). External Links: Link Cited by: §I, §IV. [3] J. DemĆĄar (2006) Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7, p. 1â30. Cited by: §IV. [4] J. Hale (2001) A probabilistic earley parser as a psycholinguistic model. In Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL), p. 1â8. Cited by: §I. [5] M. A. Just and P. A. Carpenter (1980) A theory of reading: from eye fixations to comprehension. Psychological Review 87 (4), p. 329â354. External Links: Document Cited by: §I. [6] R. Kliegl, E. Grabner, M. Rolfs, and R. Engbert (2004-01) Length, frequency, and predictability effects of words on eye movements in reading. European Journal of Cognitive Psychology 16, p. 262â284. External Links: Document Cited by: §I. [7] S. Makowski, L. A. JĂ€ger, A. Abdelwahab, N. Landwehr, and T. Scheffer (2018) A discriminative model for identifying readers and assessing text comprehension from eye movements. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2018, Proceedings, Part I, p. 209â225. External Links: Document, Link Cited by: §I. [8] C. MĂ©ziĂšre, E. D. Relchle, T. von der Malsburg, and G. McArthur (2023) Using eye-tracking measures to predict reading comprehension. Reading Research Quarterly 58 (3), p. 425â449. Cited by: §I. [9] J. Platt (1999) Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers, Vol. 10, p. 61â74. Cited by: §VI. [10] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever (2019) Language models are unsupervised multitask learners. Technical report OpenAI. External Links: Link Cited by: §I, 1st item. [11] K. Rayner (1998) Eye movements in reading and information processing: 20 years of research. Psychological Bulletin 124 (3), p. 372â422. External Links: Document Cited by: §I. [12] D. R. Reich, P. Prasse, C. Tschirner, P. Haller, F. Goldhammer, and L. A. JĂ€ger. (2022) Inferring native and non-native human reading comprehension and subjective text difficulty from scanpaths in reading. In 2022 Symposium on Eye Tracking Research and Applications (ETRA), p. 1â8. External Links: Document Cited by: §I, §I, §V. [13] O. Shubi, Y. Meiri, C. A. Hadar, and Y. Berzak (2024) Fine-grained prediction of reading comprehension from eye movements. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 3372â3391. External Links: Link Cited by: §I, §I. [14] Shubi,Omer, D. R. Reich, K. Gruteke Klein, Y. Angel, P. Prasse, L. A. JĂ€ger, and Y. Berzak (2025) EyeBench: predictive modeling from eye movements in reading. In The Thirty-Ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, External Links: Link Cited by: §I, §I, §I, §I-A, §IV, §V, §V. [15] N. Siegelman, S. Schroeder, C. AcartĂŒrk, H. Ahn, S. Alexeeva, S. Amenta, R. Bertram, R. Bonandrini, M. Brysbaert, D. Chernova, S. M. Da Fonseca, N. Dirix, W. Duyck, A. s. Fella, R. Frost, C. A. Gattei, A. Kalaitzi, N. Kwon, K. LĂ”o, M. Marelli, T. C. Papadopoulos, A. Protopapas, S. Savo, D. E. Shalom, N. Slioussar, R. Stein, L. Sui, A. Taboh, V. TĂžnnesen, K. A. Usal, and V. Kuperman (2022) Expanding horizons of cross-linguistic research on reading: the multilingual eye-movement corpus (meco). Behavior Research Methods 54, p. 2843â2863. External Links: Link Cited by: §I. [16] N. J. Smith and R. Levy (2013) The effect of word predictability on reading time is logarithmic. Cognition 128 (3), p. 302â319. External Links: Document Cited by: §I. [17] R. Speer, J. Chin, A. Lin, S. Jewett, and L. Nathan (2018) LuminosoInsight/wordfreq: v2.2. Zenodo. External Links: Document, Link Cited by: §I, 2nd item. [18] F. Wilcoxon (1945) Individual comparisons by ranking methods. Biometrics Bulletin 1 (6), p. 80â83. External Links: Document Cited by: §IV. [19] D. Yang and N. Hollenstein (2023) PLM-AS: pre-trained language models augmented with scanpaths for sentiment classification. In Proceedings of the Northern Lights Deep Learning Workshop, External Links: Link Cited by: §I, §I.