Paper deep dive
Event Interaction in Low-Rank Bottlenecks for Temporal Relation Extraction
Wei Sun, Tingyu Qu, Jesse Davis, Marie-Francine Moens
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/12/2026, 3:39:16 AM
Summary
The paper proposes Convolutional Bottleneck Interaction (CBI), a parameter-efficient fine-tuning (PEFT) architecture for temporal relation extraction. CBI addresses the limitation of low-rank bottlenecks in Adapters and LoRA by applying lightweight depthwise convolution to enhance event representations before using element-wise multiplication to capture explicit event-event interactions. This approach yields significant performance gains across multiple datasets and backbone models compared to standard concatenation or fusion methods.
Entities (13)
Relation Signals (10)
Convolutional Bottleneck Interaction → improvesperformanceon → Temporal Relation Extraction
confidence 95% · CBI provides consistent and substantial gains, up to +31.7 micro F1 ... showing that explicit interaction inside low-rank spaces is crucial for temporal relation extraction.
Convolutional Bottleneck Interaction → uses → element-wise multiplication
confidence 95% · and then uses element-wise multiplication to capture effective event-event interactions inside the bottleneck.
Convolutional Bottleneck Interaction → uses → depthwise convolution
confidence 95% · CBI ... first applies lightweight depthwise convolution to enhance event representations
Convolutional Bottleneck Interaction → isappliedin → LoRA
confidence 90% · Across five datasets and seven backbone models in the Adapter and LoRA settings, CBI provides consistent and substantial gains
Convolutional Bottleneck Interaction → isappliedin → Adapter
confidence 90% · Across five datasets and seven backbone models in the Adapter and LoRA settings, CBI provides consistent and substantial gains
Convolutional Bottleneck Interaction → isevaluatedon → TDD-Auto
confidence 90% · We evaluate our method on five temporal relation extraction datasets: ... and TDD-Auto
Convolutional Bottleneck Interaction → isevaluatedon → TimeBank-Dense
confidence 90% · We evaluate our method on five temporal relation extraction datasets: TimeBank-Dense
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Temporal relation extraction determines whether an event occurs before, after, or simultaneously with another event, and therefore relies on accurately modeling how the two events interact. Mainstream systems achieve this by concatenating event spans or using shallow fusion, which works well when all model parameters are trainable. However, in parameter-efficient fine-tuning, low-rank bottlenecks restrict information flow and prevent these interaction signals from passing through, leading to clear performance drops. To address this limitation, we propose a theoretically grounded architecture, Convolutional Bottleneck Interaction (CBI), which first applies lightweight depthwise convolution to enhance event representations and then uses element-wise multiplication to capture effective event-event interactions inside the bottleneck. Across five datasets and seven backbone models in the Adapter and LoRA settings, CBI provides consistent and substantial gains, up to +31.7 micro F1, while adding minimal computational cost, showing that explicit interaction inside low-rank spaces is crucial for temporal relation extraction. The code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.06731v1
- Canonical: https://arxiv.org/abs/2609.06731v1
Trouble viewing inline? Open PDF directly →
Full Text
92,019 characters extracted from source content.
Expand or collapse full text
Event Interaction in Low-Rank Bottlenecks for Temporal Relation Extraction Wei Sun Tingyu Qu Jesse Davis Marie-Francine Moens Affiliation: Department of Computer Science, KU Leuven Affiliation: Celestijnenlaan 200A 3001 Heverlee, Belgium Affiliation: sun.wei, tingyu.qu, jesse.davis, sien.moens@kuleuven.be Abstract Temporal relation extraction determines whether an event occurs before, after, or simultaneously with another event. Therefore, it relies on accurately modeling how the two events interact. Mainstream systems achieve this by concatenating event spans or using shallow fusion, which works well when all model parameters are trainable. However, in parameter-efficient fine-tuning, low-rank bottlenecks restrict information flow and prevent these interaction signals from passing through, leading to clear performance drops. To address this limitation, we propose a theoretically grounded architecture, Convolutional Bottleneck Interaction (CBI), which first applies lightweight depthwise convolution to enhance event representations and then uses element-wise multiplication to capture effective event-event interactions inside the bottleneck. Across five datasets and seven backbone models in the Adapter and LoRA settings, CBI provides consistent and substantial gains, up to +31.7 micro F1, while adding minimal computational cost, showing that explicit interaction inside low-rank spaces is crucial for temporal relation extraction. The code is available at https://github.com/VRCMF/CIF.git. 1 Introduction (a) (b) (c) (d) (e) Figure 1: Visualization of event interaction maps of the temporal relation extraction task for (a) Full Fine-tuning, Text Concatenation with (b) Adapter fine-tuning, (c) LoRA fine-tuning, Embedding Fusion with (d) Adapter fine-tuning, and (e) LoRA fine-tuning . marks tokens emphasized by different methods. Attention in Text Concatenation and Embedding Fusion is diffused over irrelevant tokens, limiting temporal cue propagation through the bottleneck, while full fine-tuning captures these cues effectively but at a high computational cost. Temporal relations indicate whether an event occurs before, after, or simultaneously with another event. These relations are important for narrative understanding (Yao and Huang, 2018; Ballesteros et al., 2020), story generation (Ye et al., 2022; Wang et al., 2025), and answering temporal questions (Qin et al., 2021; Roccabruna et al., 2024). Inaccurate temporal predictions can propagate through these systems and lead to incoherent or misleading task outputs. The goal of temporal relation extraction is to automatically identify temporal event relations from natural language text (Ning et al., 2018b; Leeuwenberg and Moens, 2019). Although several architectures have been explored, many temporal relation extraction systems use BERT- and RoBERTa-based encoders (Zhou et al., 2022; Man et al., 2022; Huang et al., 2023). A current approach is to fine-tune large language models to perform temporal relational extraction. Because of the computational cost, it would be desirable to employ parameter-efficient fine-tuning (PEFT) methods such as adapters (Pfeiffer et al., 2021) or LoRA (Hu et al., 2022). While PEFT reduces training cost, it also constrains learning by limiting updates to a small set of low-rank parameters. As a result, task-specific information must be encoded within a restricted subspace. Temporal relation extraction, which depends on fine-grained interactions between events, is particularly sensitive to this constraint. In this work, we investigate this limitation and propose methods to mitigate it. Unfortunately, PEFT suffers from a key limitation in this context: all task-specific information must pass through low-rank bottlenecks. This requirement is at odds with how most existing approaches model event interaction: they concatenate the two event spans and feed the combined representation into a classifier (Ballesteros et al., 2020; Fang and Glass, ; Wen and Ji, 2021; Zhou et al., 2022; Man et al., 2022; Huang et al., 2023; Roccabruna et al., 2024), which we refer to as Text Concatenation. They also augment the concatenated representation by performing shallow fusion via an element-wise product at the input-level (Chaturvedi et al., 2025), called Embedding Fusion below. These operations work well when the entire model is trainable. In contrast, in a PEFT setting only a small set of low-rank parameters is trainable. Although cross-event interactions are computed by the frozen backbone, the model can adjust them only through low-rank bottlenecks, which limits its ability to learn task-specific interaction patterns. To show this limitation, consider an example from the TDD-Man dataset (Naik et al., 2019): Sentence 1: … bedeviled by a number of [event1]disputes[/event1] - Black Sea naval bases. Sentence 2: … has stalled ratification, [event2]trying[/event2] to tie it to an agreement. The gold relation is INCLUDES because “bedeviled” and “has stalled” both signal an ongoing background situation, “disputes” denotes a temporally extended state, and “trying” introduces a more local action unfolding within that state, so the time span of “disputes” contains the time of “trying”. Figure 1 shows that in both Text Concatenation and Embedding Fusion methods, attention is dispersed over irrelevant tokens instead of concentrating on tokens representing the temporal cues. This indicates that crucial evidence fails to propagate effectively through the bottlenecked representation space. In contrast, full fine-tuning can successfully capture these important cues. However, it requires markedly more computational resources and longer training time, as shown in Table 3. Based on this observation, we propose Convolutional Bottleneck Interaction (CBI) as a lightweight and theoretically grounded architecture that can be integrated into existing PEFT schemes. CBI has two key components. First, it directly models event–event interactions within the bottleneck by fusing the hidden states of each event. We test three operations: multiplication, addition, and subtraction. Multiplication performs the best because it captures channel-wise co-activation and approximates diagonal bilinear interactions. Because each event is encoded independently, contextual cues such as tense or discourse markers are often misaligned, and the element-wise multiplication pairs informative features with uninformative ones, amplifying noise that low-rank modules cannot correct. Consequently, our second component involves performing a lightweight depthwise convolution before the interaction step. This convolution enlarges the receptive field of each token, thus capturing nearby temporal cues, smoothing noise, and reducing the sensitivity to small positional shifts of tokens. We evaluate our method on five temporal relation extraction datasets: TimeBank-Dense (Cassidy et al., 2014), MATRES (Ning et al., 2018b), TCR (Ning et al., 2018a), TDD-Man (Naik et al., 2019), and TDD-Auto (Naik et al., 2019), using seven backbone models: BERT-large (Devlin et al., 2019), RoBERTa-large (Liu et al., 2019), Gemma2-2B (Team et al., 2024), Phi3.5-3.8B (Abdin et al., 2024), Mistral-7B (Jiang et al., 2023), Qwen2.5-7B (Bai et al., 2023), and Llama 3.1-8B (Touvron et al., 2023). Under both adapter and LoRA configurations, CBI achieves consistent improvements, reaching gains of up to +31.7 micro-F1 while adding negligible computational and memory cost. These results show that explicitly modeling event interaction inside low-rank bottlenecks is both effective and necessary, and that convolution-enhanced multiplicative interaction provides a simple and effective solution for temporal relation extraction in a PEFT setting. Attention in Text Concatenation and Embedding Fusion is diffused over irrelevant tokens, limiting temporal cue propagation through the bottleneck, while full fine-tuning captures these cues effectively but at a high computational cost. We make the following contributions: • We show why PEFT underperforms in temporal relation extraction: low-rank bottlenecks miss key cross-event interactions, and common text concatenation or shallow fusion fails to pass temporal cues through a bottlenecked representation space. • We propose an event interaction PEFT framework and find multiplication to be the most effective interaction. We analyze its weaknesses: token misalignment and noise amplification, and then introduce a lightweight depthwise convolution to stabilize and enhance multiplicative interaction by enlarging the receptive field and smoothing local noise. • We propose a theoretically grounded architecture called Convolutional Bottleneck Interaction (CBI), which applies convolution followed by element-wise multiplication inside the bottleneck space. Across five datasets and seven backbone models in adapter and LoRA settings, CBI produces large and consistent improvements, up to +31.7 micro-F1, with minimal computation. 2 Related Work 2.1 Temporal Relation Extraction Pretrained language models improved temporal relation extraction by providing rich contextual event representations. The MATRES dataset (Ning et al., 2018c) makes this requirement explicit because its densely annotated event pairs can only be predicted correctly when models capture contextual and co-occurrence cues. Contextual encoders such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), and RoBERTa (Liu et al., 2019) naturally support joint modeling of two segments. Thus, the task was widely cast as sentence-pair classification: mark two events, feed their combined context into the encoder, and classify their temporal relation. This formulation produced a mainstream architecture in which concatenated event contexts are encoded jointly so that self-attention can model event interactions. Many extensions follow this paradigm (Ballesteros et al., 2020; Fang and Glass, ; Wen and Ji, 2021; Zhou et al., 2022; Man et al., 2022; Huang et al., 2023), while including multitask learning, joint prediction of order and relation types, and graph-based modeling for document or cross-document consistency. Chaturvedi et al. (2025) further strengthen pairwise fusion signals by introducing an element-wise product of paired event embeddings at the input level. All these methods assume full-model fine-tuning, where self-attention layers freely adjust parameters to encode event–event interaction. Under parameter-efficient fine-tuning, most parameters are frozen, and information must pass through low-rank bottlenecks. These bottlenecks limit the ability of the model to adjust representations for encoding event interaction, making concatenation or other shallow fusion strategies insufficient for transmitting detailed temporal cues. 2.2 Parameter-Efficient Fine-Tuning Large language models (LLMs) have become widely used across NLP because of their strong generalization ability. In temporal relation extraction, LLMs have mainly been explored through zero- and few-shot prompting rather than fine-tuning. Chan et al. (2024) showed that ChatGPT struggles with temporal ordering, and Yuan et al. (2023) proposed prompt decomposition for zero-shot extraction but still observed substantial gaps compared to supervised models. Recent work has explored parameter-efficient fine-tuning (PEFT) methods such as LoRA (Hu et al., 2022). Roccabruna et al. (2024) demonstrated that LoRA yields noticeable gains over instruction-tuned LLMs, suggesting that parameter-efficient updates offer a promising direction. While prior work such as Roccabruna et al. (2024) demonstrates the effectiveness of LoRA for temporal relation extraction, it focuses on a single PEFT method in limited experimental settings. PEFT methods such as Adapters (Pfeiffer et al., 2021; Lu et al., 2024) and LoRA have been shown to be effective in many NLP tasks by significantly reducing the number of trainable parameters (Ben Zaken et al., 2022; Chen et al., 2024; Wang et al., 2022). Yet, their applicability to temporal relation extraction remains unclear, especially because common concatenation-based architectures were originally designed for fully trainable models rather than for settings where most parameters are frozen. To address this gap, we fine-tune LLMs using Adapters and LoRA across five temporal relation extraction datasets, providing the first comprehensive evaluation of PEFT methods for this task. 2.3 Modeling Cross-Event Interactions Temporal relation extraction systems differ mainly in how they model interactions between two events. Most adopt implicit interaction: they concatenate the two event spans and rely on a transformer to infer cross-event dependencies through self-attention. This requires a sufficient model capacity. However, in a PEFT setting low-rank adapters constrain the representational space and limit how much interaction-related information can pass through, making implicit modeling unreliable. To overcome this bottleneck limitation, we draw inspiration from vision–language PEFT models. Qu et al. (2024) introduced feature routing functions that enhance cross-modal alignment by directing information through low-rank bottlenecks, a strategy later adopted by Guo et al. (2025) and Borchert et al. (2025). The latter applied this approach to low-resource cross-lingual transfer, consistently outperforming naïve concatenation. Following this idea, we design three explicit event interaction functions within PEFT bottlenecks: element-wise multiplication, addition, and subtraction of event representations. Multiplication performs best because it captures channel-wise co-activation patterns essential for temporal relation extraction. However, it is sensitive to representation misalignment and noise. To mitigate this, we introduce a depthwise convolution operation, which performs token-level alignment and interaction within the PEFT bottlenecks. To our knowledge, our paper proposes the first temporal relation extraction method to implement explicit event interaction in PEFT settings instead of relying on implicit event interactions. 3 Method While LoRA and Adapters efficiently introduce low-rank updates, they primarily operate on each input independently and lack explicit mechanisms to model structured interactions between multiple inputs (e.g., event pairs). To address this limitation, we propose Convolutional Bottleneck Interaction, which augments the PEFT bottleneck with explicit cross-event interaction modeling while preserving parameter efficiency. 3.1 Parameter-Efficient Adaptation via LoRA and Adapters Before introducing our CBI adapter, we briefly review two common PEFT strategies shown in Figure 2(a) and Figure 2(b). LoRA: LoRA adds trainable low-rank matrices to key and value projections while keeping pretrained weights frozen. As shown in Figure 2(a), each projection (e.g., QW_Q, VW_V) receives a rank-r update: Δ=downup, =W_downW_up, (1) where down∈ℝd×rW_down ^d× r is the down-projection and up∈ℝr×dW_up ^r× d is the up-projection, with r≪dr d. d is the dimension of hidden representations. Only these two matrices are trained. Adapters: Adapters insert a small bottleneck module inside each transformer block (Figure 2(b)). Given hidden state H: Adapter()=+up(down),Adapter(H)=H+W_up σ(W_downH), (2) where σ(⋅)σ(·) is the activation function, and downW_down, upW_up form the learnable low-rank bottleneck. 3.2 Overview of CBI (a) LoRA in a transformer block. (b) Adapter in a transformer block. (c) Convolutional Bottleneck Interaction. Figure 2: Illustration of (a) LoRA, (b) Adapter, and (c) The Convolutional Bottleneck Interaction adapter, which respectively replaces the LoRA or adapter block when integrated in the transformer block of (a) and (b). The parameters in the blue boxes are learned during the fine-tuning. The Convolutional Bottleneck Interaction (CBI) Adapter (Figure 2(c)) is inserted inside each low-rank bottleneck. Given two events, CBI enhances bottleneck features and explicitly models cross-event exchange through: (a) Dual-Tower Local Enrichment: down-project each event into a rank-r bottleneck and apply depthwise convolution to aggregate local temporal context; (b) Explicit Cross-Event Interaction: interact the enriched bottleneck features via element-wise multiplication to capture cross-event interaction. A complete theoretical analysis of CBI is provided in Appendix 7. 3.3 Notation and Input Representations Let 1,2∈ℝL×dH_1,H_2 ^L× d be contextualized embeddings of two textual segments containing events e1e_1 and e2e_2, produced by a frozen backbone fθf_θ. Both sequences are padded to length L. CBI operates only on these two events within the PEFT bottleneck. 3.4 Dual-Tower Local Enrichment We first map each event to the bottleneck space: 1=1down,2=2downE_1=H_1W_down, _2=H_2W_down (3) where i∈ℝL×rE_i ^L× r contains r latent channels (a channel is a dimension in the bottleneck representation) per token. down∈ℝd×r,r≪dW_down ^d× r,\ r d. In a rank-r bottleneck, each channel is narrow and may miss local context. We then apply a depthwise convolution independently to each channel: ~i=DWConvi(i),i∈1,2, E_i=DWConv_i(E_i), i∈\1,2\, (4) where DWConviDWConv_i uses a 1D convolution kernel =[θ−k/2,…,θk/2]∈ℝk θ=[ _-k/2,…, _k/2] ^k along the token axis. This operation can be equivalently expressed as a matrix multiplication, where ∈ℝL×LT ^L× L is the Toeplitz matrix induced by the kernel θ. For each event: ~i=i∈ℝL×r,equivalently~i(l)=∑jθji(l+j). E_i=TE_i ^L× r, \\ e_i(l)= _j _j\,e_i(l+j). (5) Depthwise convolution expands the local receptive field and smooths noise, yielding enriched bottleneck features ~i∈ℝL×r E_i ^L× r. 3.5 Explicit Cross-Event Interaction After enrichment, we form cross-event interaction features by element-wise multiplication: =~1⊙~2,∈ℝL×r.F= E_1 E_2, ^L× r. (6) For each token position l and channel c, [l,c]=~1[l,c]~2[l,c].F[l,c]= E_1[l,c] E_2[l,c]. (7) The product [l,c]F[l,c] is a second-order event interaction. Element-wise products over tokens in a fixed channel recover the diagonal bilinear term (i.e., the result of a bilinear interaction that only involves element-wise interactions between corresponding dimensions): ∑l=1L[l,c]=(~1⊤~2)c,c. _l=1^LF[l,c]= ( E_1 E_2 )_c,c. (8) Therefore, F is an explicit token-wise representation of diagonal bilinear cross-event interactions, which are hard to learn implicitly under a rank-r PEFT update. Depthwise convolution with multiplication further allows near-aligned tokens to interact even if their exact positions differ. This benefit of the convolution is particularly important because element-wise multiplication implicitly relies on token-level alignment: even noise coming from small positional deviations can perturb the product and lead to unstable second-order interactions (theoretical proof is in Appendix 7.3). By introducing a depthwise convolution prior to interaction, CBI mitigates this misalignment through local aggregation, which acts as a soft alignment and denoising operator. This preserves the desired diagonal bilinear signal while suppressing noise, resulting in more robust cross-event interaction features; see Appendix 7.4 for the theoretical proof. 3.6 Residual Injection We convert the interaction field back to hidden size d using the PEFT up-projection and inject it through a residual path: Δ=up,1′=1+Δ, =FW_up, _1=H_1+ , (9) where up∈ℝr×dW_up ^r× d is the PEFT up-projection. 1′H _1 is then passed to the next transformer blocks. Backbone PEFT Methods MATRES TB-Dense TCR TDD-Auto TDD-Man AVG / FFT (upper bound) 82.15± 0.34 63.27± 0.41 81.12± 0.38 86.71± 0.33 80.82± 0.45 78.81± 0.36 RoBERTa-large Adapter Text Concatenation 58.31± 0.91 41.69± 1.07 59.72± 0.66 32.85± 1.12 31.59± 0.53 44.83± 0.78 Embedding Fusion 58.31± 0.74 41.69± 0.88 59.72± 1.13 32.85± 0.97 57.14± 0.49 49.94± 1.05 CBI 74.35± 0.85 68.18± 1.07 73.92± 1.61 86.85± 1.32 79.67± 1.18 76.59± 1.22 LoRA Text Concatenation 67.58± 0.62 55.17± 0.97 67.44± 0.74 71.85± 0.91 64.57± 0.83 65.32± 0.58 Embedding Fusion 59.52± 1.01 46.08± 0.79 62.35± 1.18 64.95± 0.55 68.24± 0.47 60.23± 0.88 CBI 70.16± 1.38 62.70± 0.74 67.75± 1.20 89.35± 0.83 76.24± 1.27 73.24± 1.00 / FFT (upper bound) 81.76± 0.32 60.78± 0.41 79.35± 0.36 81.77± 0.33 80.53± 0.39 76.84± 0.35 BERT-large Adapter Text Concatenation 58.31± 0.72 41.69± 1.31 59.72± 0.91 32.85± 0.59 31.59± 0.94 44.83± 0.86 Embedding Fusion 58.31± 0.88 41.69± 0.97 59.72± 1.13 32.85± 0.52 58.37± 0.43 50.19± 1.02 CBI 71.69± 0.97 67.55± 1.08 72.53± 1.51 90.60± 1.44 79.02± 1.19 76.28± 1.21 LoRA Text Concatenation 66.69± 0.59 54.86± 0.98 65.90± 1.04 71.65± 1.22 65.55± 0.47 64.93± 0.66 Embedding Fusion 58.31± 1.12 45.30± 0.76 61.11± 0.83 63.75± 1.09 60.98± 0.61 57.89± 0.95 CBI 71.94± 1.05 65.36± 0.89 68.21± 0.62 88.25± 1.19 76.82± 0.53 74.12± 0.76 / FFT (upper bound) 78.35± 0.39 73.36± 0.61 73.63± 0.67 89.74± 0.45 76.23± 0.57 78.26± 0.53 / ZST (lower bound) 53.56± 1.88 25.78± 2.32 53.62± 2.01 24.79± 2.41 26.79± 2.14 36.91± 1.93 Gemma2-2B Adapter Text Concatenation 59.19± 0.63 44.51± 1.32 59.10± 1.04 52.05± 0.77 59.27± 0.81 54.82± 0.92 Embedding Fusion 53.55± 1.18 39.18± 0.74 55.71± 0.86 31.65± 0.49 25.80± 1.29 41.18± 1.03 CBI 62.58± 1.02 59.09± 1.14 64.20± 1.48 77.20± 1.55 68.24± 1.27 66.26± 1.29 LoRA Text Concatenation 60.40± 0.74 49.69± 0.85 61.42± 0.57 63.70± 1.23 59.76± 0.94 58.99± 0.69 Embedding Fusion 52.90± 1.09 38.56± 0.52 57.87± 0.98 26.95± 0.41 24.90± 1.21 40.24± 0.82 CBI 66.53± 0.78 59.40± 0.58 64.51± 0.84 83.35± 0.81 69.47± 0.45 68.65± 0.40 / FFT (upper bound) 81.53± 0.48 71.84± 0.62 75.83± 0.57 89.09± 0.39 75.83± 0.51 78.82± 0.46 / ZST (lower bound) 47.88± 1.73 23.18± 2.11 45.87± 1.84 25.36± 2.25 26.95± 1.96 33.85± 1.67 Phi-3.5-3.8B Adapter Text Concatenation 59.35± 0.95 48.43± 1.27 60.96± 0.83 58.85± 0.71 60.41± 0.62 57.60± 0.86 Embedding Fusion 56.29± 0.77 40.13± 0.88 55.86± 1.09 31.20± 0.49 26.86± 1.21 42.07± 1.02 CBI 61.21± 0.92 59.09± 1.17 63.89± 1.42 71.60± 1.34 67.59± 1.26 64.68± 1.21 LoRA Text Concatenation 61.13± 0.78 54.70± 0.94 61.73± 1.11 62.20± 0.65 63.43± 0.57 60.64± 0.76 Embedding Fusion 50.81± 0.93 27.90± 1.19 48.61± 0.72 27.40± 0.41 26.12± 1.24 36.17± 0.88 CBI 70.00± 1.14 59.25± 0.91 70.22± 1.11 84.25± 1.06 72.90± 0.85 71.32± 0.60 / FFT (upper bound) 72.89± 0.68 62.43± 0.79 72.97± 0.64 90.87± 0.88 65.89± 0.71 73.01± 0.83 / ZST (lower bound) 40.02± 2.31 32.54± 2.47 39.27± 2.12 30.69± 2.76 19.73± 2.58 32.45± 2.34 Mistral-7B Adapter Text Concatenation 58.31± 1.04 41.69± 0.87 59.72± 1.16 32.85± 0.59 31.59± 0.74 44.83± 0.93 Embedding Fusion 56.61± 0.79 37.46± 0.91 56.79± 0.98 30.15± 0.46 29.31± 1.08 42.06± 1.03 CBI 64.60± 0.95 60.97± 1.19 64.20± 1.44 77.90± 1.58 70.53± 1.24 67.64± 1.23 LoRA Text Concatenation 57.58± 0.84 47.96± 1.11 61.11± 1.23 61.75± 0.63 62.94± 0.89 58.27± 0.73 Embedding Fusion 56.61± 0.77 37.46± 1.18 56.79± 0.85 30.15± 1.04 29.31± 0.91 42.06± 1.12 CBI 68.31± 0.77 58.78± 1.09 70.52± 0.45 86.30± 0.47 75.43± 0.79 71.87± 0.55 / FFT (upper bound) 74.93± 0.39 81.47± 0.31 77.38± 0.44 90.39± 0.36 80.79± 0.28 80.99± 0.39 / ZST (lower bound) 43.27± 1.58 33.89± 1.91 41.03± 1.47 28.83± 2.03 21.47± 1.69 33.70± 1.61 Qwen2.5-7B Adapter Text Concatenation 61.69± 0.92 47.65± 1.11 62.50± 0.58 51.35± 1.31 61.14± 0.44 56.87± 0.73 Embedding Fusion 58.31± 0.66 41.69± 1.27 59.72± 0.49 32.85± 0.95 31.59± 0.37 44.83± 1.12 CBI 61.94± 1.08 59.72± 1.21 63.58± 1.43 65.65± 1.54 63.76± 1.12 62.93± 1.27 LoRA Text Concatenation 58.95± 0.73 55.80± 1.22 63.43± 0.47 65.25± 1.08 64.08± 0.41 61.50± 0.95 Embedding Fusion 50.73± 1.19 36.05± 0.62 51.54± 0.88 27.30± 1.37 29.14± 0.52 38.95± 0.69 CBI 62.74± 1.20 59.72± 0.59 65.12± 0.64 76.70± 0.81 64.00± 0.68 65.66± 0.83 / FFT (upper bound) 69.02± 0.42 72.79± 0.55 73.36± 0.63 91.71± 0.48 68.78± 0.51 75.13± 0.60 / ZST (lower bound) 28.47± 1.72 27.81± 1.95 27.83± 2.10 30.65± 1.84 17.37± 2.25 26.43± 1.67 Llama-3.1-8B Adapter Text Concatenation 61.69± 0.77 51.88± 1.36 63.43± 0.59 62.30± 0.42 60.98± 1.18 60.06± 0.88 Embedding Fusion 57.02± 1.08 38.40± 0.64 59.57± 0.91 31.30± 0.55 27.18± 1.21 42.69± 0.73 CBI 64.84± 0.88 55.80± 1.14 63.43± 1.41 89.20± 1.52 72.98± 1.09 69.25± 1.21 LoRA Text Concatenation 65.40± 1.03 58.93± 0.71 65.43± 1.19 71.90± 0.68 63.02± 1.34 64.94± 0.82 Embedding Fusion 54.52± 0.69 36.99± 1.24 57.10± 0.81 30.55± 1.41 28.90± 0.63 41.61± 0.97 CBI 71.05± 0.36 58.78± 0.98 69.44± 1.33 89.35± 0.56 77.88± 0.62 73.30± 1.08 Table 1: Performance comparison of Convolutional Bottleneck Interaction (CBI) with two baselines (concatenation of event embeddings and element-wise product of event embeddings) under Adapter and LoRA PEFT settings evaluated on five temporal relation extraction datasets (micro-F1%). Each score shows the mean± of three runs. Full fine-tuning (FFT) and zero-shot tuning (ZST) are reported as reference upper and lower bounds for PEFT methods, respectively. The best score in each setting (with the same backbone and PEFT method) is shown in bold blue, the second-best is shown in gray underlined. 4 Experiments We conduct extensive empirical evaluations across diverse datasets and model families to demonstrate the utility of our framework. Moreover, we perform several experiments to investigate the effects of various design choices. 4.1 Benchmarks and Setup We evaluate the proposed Convolutional Bottleneck Interaction (CBI) on five widely used benchmarks for temporal relation extraction: These benchmarks differ in the granularity of temporal reasoning they require, some focus on intra-sentential event pairs, while others emphasize inter-sentential (discourse-level) relations that require long-range context understanding. Detailed statistics of the datasets are shown in Appendix 8. Intra-sentential benchmarks: TimeBank-Dense (TB-Dense) (Cassidy et al., 2014): An extension of the TimeBank corpus where all event pairs in each document (including across sentences) are annotated with temporal links. Although it contains both intra- and inter-sentential pairs, the majority are sentence-internal, making it primarily a local reasoning benchmark. MATRES (Ning et al., 2018b): A dataset that refines TimeBank-Dense by filtering out uncertain annotations, focusing on reliably labeled event pairs in news articles. It uses three labels (BEFORE, AFTER, SIMULTANEOUS). Most event pairs occur within the same sentence, emphasizing local temporal cues. TCR (Temporal and Causal Reasoning) (Ning et al., 2018a): A dataset of sentences annotated for both temporal and causal relations between events. We use only the temporal relations (BEFORE, AFTER, SIMULTANEOUS). This benchmark tests fine-grained intra-sentential reasoning due to the dense annotation of single-sentence event pairs. Inter-sentential (discourse-level) benchmarks: TDDiscourse-Man (TDD-Man) (Naik et al., 2019): A discourse-level temporal event ordering dataset. It contains event pairs that are often far apart in the text (multiple sentences away), emphasizing long-range temporal reasoning across discourse units. The label set follows an interval relation scheme (BEFORE, AFTER, INCLUDES, IS_INCLUDED, SIMULTANEOUS). TDDiscourse-Auto (TDD-Auto) (Naik et al., 2019): An automatically constructed large-scale dataset inferred from news articles using heuristics and external time annotations. It shares the same label set as TDD-Man but is much larger (tens of thousands of pairs) and noisier. Both TDD variants mainly involve inter-sentential temporal relations. 5 Implementation Details We experiment with seven representative backbones that span both encoder-only and decoder-style large language models: BERT-large, RoBERTa-large, and five instruction-tuned LLMs, including Gemma2-2B, Phi-3.5-3.8B, Mistral-7B, Qwen2.5-7B, and Llama-3.1-8B. Our method is evaluated in two widely used PEFT paradigms, Adapter and LoRA, to ensure generality across architectures. Each event pair is represented by two contextual segments marked as [event1]…[/event1] and [event2]…[/event2], which are jointly fed into the model to predict their temporal relation. For classification, we use the hidden state of the last non-padding token. Hidden states are projected to logits via a linear layer, and the logits at the last valid position are used for training and inference. In all experiments, we fine-tune each model on each dataset independently. Encoder-based models are trained with a batch size of 12 for up to 10 epochs, while LLMs, which are more computationally demanding, use a batch size of 8 under the same epoch limit. In both cases, we employ the AdamW optimizer with a learning rate of 1×10−41× 10^-4 and a dropout rate of 0.1. The best checkpoint is selected according to validation micro-F1. Class imbalance is handled by reporting micro-F1 and, where appropriate, oversampling minority classes during training. The kernel size k is set to 5. The hidden-state dimensionality follows the default configuration of each backbone model. Following Qu et al. (2024), the rank of bottleneck r is fixed at 128 for both the adapter and LoRA settings. All experiments are conducted on NVIDIA A100 GPU under a unified training protocol to ensure fair comparison. For full fine-tuning, we use a learning rate of 2×10−52× 10^-5 with a batch size of 2 by following Pareja et al. (2024); Kim et al. (2025). Training is conducted on two NVIDIA A100 GPUs using Fully Sharded Data Parallel (FSDP) with the FULL_SHARD strategy. To reduce memory consumption, we enable bfloat16 (bf16) precision during training. For zero-shot tuning, we use the following prompt: "You are an expert in temporal information extraction. Determine the temporal relation between the two events based on their contexts. Possible relations: label set. Reply with ONLY one label from: input text. No explanation.". The input text is: "Event 1 context: text1. Event 2 context: text2. Temporal relation:" and "label set" is a list all labels. 5.1 Comparison with state-of-the-art Methods The experiments are designed to assess whether CBI generalizes under varying backbone capacities and PEFT configurations. We compare CBI with two strong baselines: Text Concatenation (Zhou et al., 2022; Roccabruna et al., 2024), a non-interactive approach that concatenates the two event texts into a single input sequence, encodes them into a joint representation, and then feeds this representation into a low-rank bottleneck, and Embedding Fusion (Chaturvedi et al., 2025), which performs event fusion at the input level by taking the element-wise product of the two event embeddings and using the fused embedding as the model input.11 1 In the original design, the two event embeddings and their element-wise product are concatenated along the hidden dimension. In the PEFT setting, however, the backbone hidden size is fixed, making this three-way concatenation infeasible; following Chaturvedi et al. (2025), we therefore retain the element-wise product, which is the core fusion component. In contrast to Embedding Fusion, which completes event fusion before the bottleneck and propagates a single fused embedding, CBI places the interaction inside the low-rank bottleneck: the two events remain separate at the input, and the limited bottleneck capacity is explicitly devoted to modeling cross-event interactions, thereby strengthening event interaction precisely in a space where such dependencies are otherwise hard to learn. The experimental results for five datasets and seven backbones are summarized in Table 1. Across all configurations, CBI consistently achieves the highest micro-F1, demonstrating robust temporal relation extraction in PEFT settings. In the Adapter setting, CBI improves RoBERTa-large from 44.83% (Text Concatenation) and 49.94% (Embedding Fusion) to 76.59% (+31.76 percentage points, abbreviated as p / +26.65p), and BERT-large from 44.83% / 50.19% to 76.28% (+31.45p / +26.09p). A similar trend holds for LLMs, where CBI achieves gains of +11.44p on Gemma2-2B, +7.08p on Phi-3.5-3.8B, +22.81p on Mistral-7B, +6.06p on Qwen2.5-7B, and +9.19p on Llama-3.1-8B compared to Text Concatenation. Notably, Embedding Fusion provides limited or even negative gains on larger models, indicating that multiplication interaction of the input embeddings of the respective events is insufficient to capture complex temporal dependencies. Under the LoRA configuration, CBI maintains clear superiority, improving over Text Concatenation by an average of +6.36p, with the largest margin (+13.6p) observed on Mistral-7B. It is clear from Table 2 that, within the same backbone, all PEFT methods achieve higher micro-F1 than zero-shot tuning.22 2 Few-shots (5-shots) learning performs slightly worse than zero-shot and exhibits higher variance, so we use zero-shot tuning performance as the lower bound for the PEFT method. For example, instruction-tuned models such as Mistral-7B and Llama 3.1 8B are around 25 to 35 micro-F1 score in zero-shot, while PEFT results are mostly around 60 to 70 micro-F1 score. This shows that PEFT is necessary and zero-shot defines a clear lower bound. However, PEFT is still worse than full fine-tuning, which defines the upper bound.33 3 We applied full fine-tuning to both Text Concatenation and Embedding Fusion and found that Text Concatenation largely outperformed Embedding Fusion. On Llama-3.1-8B, the best PEFT result is 73.30 micro-F1 score with LoRA and CBI. This is still below FFT performance (75.13 micro-F1 score), which indicates a slightly reduced representational capacity of the PEFT models. CBI consistently achieves the best results in the PEFT setting. Under Adapter, CBI improves average micro-F1 score from 60.06 (Text Concatenation) to 69.25. Under LoRA, it improves from 64.94 to 73.30 micro-F1 score. Similar gains are observed across datasets. This shows that CBI recovers more of the lost capacity. 5.2 Effectiveness of CBI The core principle in our framework is to model the interaction in the feature (i.e., latent) space between two event representations using an element-wise mathematical operation. Therefore, we compare the effect of three different possible element-wise operations: Multiplication, Addition and Subtraction. Multiplication captures token co-activation but can amplify noise; Addition smooths the interaction but may blur important distinctions; Subtraction highlights contrasts but lacks cues for temporal order.44 4 We take the absolute difference to avoid a bias caused by the textual position of events. Note that none of these proposed variants perform the convolution operation. Consequently, Multiplication represents an ablated version of CBI. Table 2 shows the results of all three variants, as well as CBI for the adapters and LoRA settings in all backbone architectures and datasets. Although all four interaction functions are our designs, they reflect different ways events may interact through the low-rank bottlenecks. Empirically, CBI consistently outperforms these simpler operations. For example, on RoBERTa-large (Adapter), Multiplication reaches 73.35% F1 and Subtraction 62.87% F1, while CBI achieves 76.59% F1. A similar pattern holds for Qwen-2.5-7B (LoRA), where CBI obtains 65.66% F1 compared to 59.64% for Multiplication and 61.53% for Subtraction, but with much better stability and transferability between datasets. Overall, CBI preserves the efficiency of simple interaction functions while adding structure awareness through local depthwise convolutions and improving local coherence by filtering and aligning temporally relevant cues before cross-event interaction. This leads to more consistent and interpretable representations across architectures. (a) (b) Figure 3: Effect of training data proportion on model performance. Both CBI and Multiplication are methods proposed in this paper, while Text Concatenation serves as the baseline. Backbone PEFT Routing Matres TB-dense TCR TDD-auto TDD-man AVG Roberta-large Adapter Multiplication 71.61± 0.78 60.82± 1.34 68.83± 1.22 89.15± 1.49 76.33± 0.91 73.35± 1.18 Adapter Addition 58.79± 1.55 41.69± 0.92 59.72± 1.36 56.35± 1.21 34.04± 1.44 50.12± 1.47 Adapter Subtraction 58.31± 1.09 41.69± 1.23 59.72± 1.28 85.40± 1.63 69.22± 0.98 62.87± 1.25 Adapter CBI 74.35± 0.85 68.18± 1.07 73.92± 1.61 86.85± 1.32 79.67± 1.18 76.59± 1.22 LoRA Multiplication 68.79± 1.31 60.34± 0.52 70.06± 0.95 87.40± 0.83 71.02± 0.59 71.52± 0.90 LoRA Addition 66.45± 0.38 61.60± 0.83 65.43± 1.33 78.65± 0.68 73.71± 1.08 69.17± 0.99 LoRA Subtraction 70.56± 1.22 52.82± 1.10 68.21± 0.63 80.00± 0.72 68.00± 0.60 67.92± 0.52 LoRA CBI 70.16± 1.38 62.70± 0.74 67.75± 1.20 89.35± 0.83 76.24± 1.27 73.24± 1.00 Bert-large Adapter Multiplication 74.35± 0.83 59.56± 1.31 68.21± 1.25 87.25± 1.48 75.84± 1.02 73.04± 1.17 Adapter Addition 58.31± 1.26 46.87± 1.12 59.72± 1.09 73.60± 1.34 60.00± 1.72 59.70± 1.33 Adapter Subtraction 58.31± 1.05 53.76± 1.42 59.72± 1.14 83.05± 1.69 72.49± 1.21 65.47± 1.36 Adapter CBI 71.69± 0.97 67.55± 1.08 72.53± 1.51 90.60± 1.44 79.02± 1.19 76.28± 1.21 LoRA Multiplication 67.34± 1.25 61.13± 0.99 69.14± 0.62 87.95± 1.13 74.53± 1.11 72.02± 0.40 LoRA Addition 67.42± 0.60 57.52± 1.19 65.59± 0.98 80.25± 1.14 71.76± 0.46 68.51± 0.65 LoRA Subtraction 70.56± 1.04 52.98± 0.57 66.82± 1.09 83.85± 1.09 71.35± 0.64 69.11± 1.36 LoRA CBI 71.94± 1.05 65.36± 0.89 68.21± 0.62 88.25± 1.19 76.82± 0.53 74.12± 0.76 Gemma2 2B Adapter Multiplication 58.87± 1.34 53.61± 1.17 63.12± 1.22 63.60± 1.31 62.29± 1.09 60.30± 1.37 Adapter Addition 59.52± 1.21 57.99± 1.47 63.89± 1.33 60.90± 1.28 65.31± 1.52 61.52± 1.44 Adapter Subtraction 61.13± 0.89 58.31± 1.63 63.58± 1.17 59.55± 1.26 65.39± 1.31 61.59± 1.35 Adapter CBI 62.58± 1.02 59.09± 1.14 64.20± 1.48 77.20± 1.55 68.24± 1.27 66.26± 1.29 LoRA Multiplication 63.55± 0.60 55.80± 0.69 64.35± 0.93 67.95± 0.70 65.55± 0.58 63.44± 1.37 LoRA Addition 61.53± 1.30 56.43± 1.10 63.58± 1.09 59.50± 0.90 62.37± 1.34 60.68± 0.85 LoRA Subtraction 61.45± 0.64 57.68± 1.14 64.81± 0.97 60.15± 0.55 61.71± 0.98 61.16± 1.32 LoRA CBI 66.53± 0.78 59.40± 0.58 64.51± 0.84 83.35± 0.81 69.47± 0.45 68.65± 0.40 Phi-3.5 3.8B Adapter Multiplication 63.31± 1.19 58.31± 1.22 63.58± 1.07 61.55± 1.45 61.96± 1.12 61.74± 1.33 Adapter Addition 62.82± 1.07 58.46± 1.28 64.33± 1.21 61.50± 1.56 63.67± 1.09 62.25± 1.27 Adapter Subtraction 62.74± 1.44 58.62± 1.03 64.81± 1.35 60.45± 1.72 63.35± 1.18 61.99± 1.39 Adapter CBI 61.21± 0.92 59.09± 1.17 63.89± 1.42 71.60± 1.34 67.59± 1.26 64.68± 1.21 LoRA Multiplication 62.34± 1.43 54.86± 0.99 64.04± 0.88 64.65± 1.13 64.08± 0.36 61.99± 1.13 LoRA Addition 60.24± 0.63 57.21± 0.84 64.81± 0.85 60.30± 1.06 60.98± 0.89 60.71± 0.64 LoRA Subtraction 62.66± 0.58 60.50± 0.96 65.59± 0.76 59.70± 1.18 60.57± 0.77 61.80± 0.61 LoRA CBI 70.00± 1.14 59.25± 0.91 70.22± 1.11 84.25± 1.06 72.90± 0.85 71.32± 0.60 Mistral-7B Adapter Multiplication 63.87± 1.22 54.23± 1.38 61.88± 1.29 62.30± 1.51 61.63± 1.11 60.78± 1.28 Adapter Addition 58.31± 1.18 41.69± 1.27 59.72± 1.33 32.85± 1.47 46.12± 1.36 47.74± 1.42 Adapter Subtraction 58.31± 1.06 41.69± 1.41 59.72± 1.21 37.20± 1.68 31.59± 1.09 45.70± 1.37 Adapter CBI 64.60± 0.95 60.97± 1.19 64.20± 1.44 77.90± 1.58 70.53± 1.24 67.64± 1.23 LoRA Multiplication 68.06± 0.45 60.19± 1.07 67.75± 1.22 82.25± 0.95 72.24± 1.19 70.10± 0.90 LoRA Addition 62.74± 0.88 56.43± 1.17 64.97± 0.45 59.60± 0.61 60.49± 0.64 60.85± 1.06 LoRA Subtraction 63.95± 0.55 56.90± 0.92 64.35± 0.40 60.20± 0.99 59.18± 0.77 60.92± 0.85 LoRA CBI 68.31± 0.77 58.78± 1.09 70.52± 0.45 86.30± 0.47 75.43± 0.79 71.87± 0.55 Qwen2.5 7B Adapter Multiplication 60.08± 1.11 53.29± 1.33 61.88± 1.24 60.40± 1.42 60.73± 1.09 59.28± 1.25 Adapter Addition 63.47± 0.97 57.52± 1.26 64.20± 1.35 60.25± 1.31 62.45± 1.41 61.58± 1.23 Adapter Subtraction 63.31± 1.15 58.15± 1.18 64.18± 1.17 60.20± 1.37 62.20± 1.28 61.61± 1.29 Adapter CBI 61.94± 1.08 59.72± 1.21 63.58± 1.43 65.65± 1.54 63.76± 1.12 62.93± 1.27 LoRA Multiplication 60.16± 1.22 56.58± 0.58 61.42± 0.60 60.30± 0.53 59.76± 0.48 59.64± 0.67 LoRA Addition 62.26± 0.88 57.05± 0.62 63.12± 0.82 56.90± 0.82 62.94± 0.52 60.45± 0.79 LoRA Subtraction 63.87± 0.99 58.46± 0.65 62.81± 0.90 60.05± 1.34 62.45± 0.92 61.53± 0.73 LoRA CBI 62.74± 1.20 59.72± 0.59 65.12± 0.64 76.70± 0.81 64.00± 0.68 65.66± 0.83 Llama 3.1 8B Adapter Multiplication 62.50± 0.91 56.11± 1.28 61.27± 1.32 70.40± 1.44 60.73± 1.07 62.20± 1.19 Adapter Addition 62.90± 1.34 57.21± 1.23 65.12± 1.25 60.80± 1.57 62.69± 1.16 61.74± 1.26 Adapter Subtraction 63.71± 1.07 56.27± 1.36 64.97± 1.18 60.90± 1.61 62.29± 1.24 61.63± 1.22 Adapter CBI 64.84± 0.88 55.80± 1.14 63.43± 1.41 88.20± 1.52 72.89± 1.09 69.25± 1.21 LoRA Multiplication 71.45± 0.74 63.64± 0.60 70.83± 0.52 88.15± 0.73 74.94± 0.83 73.80± 0.53 LoRA Addition 62.82± 0.62 50.78± 1.13 61.88± 0.41 60.25± 0.43 60.16± 0.94 59.18± 1.13 LoRA Subtraction 63.63± 0.95 56.74± 0.72 66.05± 0.45 59.25± 0.95 63.35± 0.95 61.80± 0.57 LoRA CBI 71.05± 0.36 58.78± 0.98 69.44± 1.33 89.35± 0.56 77.88± 0.62 73.30± 1.08 Table 2: Temporal relation extraction under two PEFT settings (Adapter / LoRA), comparing different interaction methods that help routing features through the low-rank bottlenecks and their backbone LLMs. The best score in each setting (with the same backbone and PEFT method) is shown in bold blue, the second-best is shown in gray underlined. 5.2.1 Data Efficiency and Robustness of CBI To understand how different interaction methods behave when training data is limited, we train each method using varying portions of the dataset (20%–100%). This allows us to observe how well each mechanism maintains its performance as the amount of available data decreases and how sharply its effectiveness drops under data-scarce conditions. Figure 3 shows the micro-F1 scores of Text Concatenation, Multiplication, and CBI under both the Adapter and LoRA configurations.55 5 We omit Embedding Fusion due to its consistently inferior performance. In the Adapter setting, Text Concatenation remains highly sensitive to data scarcity, increasing from 42.12% to 60.98% as data grows. Multiplication is more stable (57.14% to 60.73%). CBI achieves the strongest performance, starting at 62.60% with 20% of the data and reaching 72.89% with full supervision, showing substantially better generalization under limited annotation. The LoRA setting exhibits broadly similar patterns, although the differences between methods become smaller in scenarios with extremely limited annotation. With complete data, CBI again achieves the best micro-F1 (77.88%). At moderate data levels (e.g., 60%), CBI reaches 67.56%, outperforming Multiplication (65.35%) and Text Concatenation (60.78%). Under extremely limited annotation conditions, performance differences narrow, likely because LoRA introduces more trainable parameters than Adapter-based tuning, partially mitigating underfitting when supervision is sparse. Across both tuning strategies, Text Concatenation shows the steepest degradation as data decreases, while CBI degrades much more gracefully. This validates our hypothesis that limited temporal-relation annotations reduce the ability of the model to align event semantics, and that feature-level interaction is crucial in low-resource settings. Overall, CBI is both data-efficient and robust: it maintains strong performance in the PEFT setting when supervision is scarce. Its convolutional expansion and gated nonlinearity strengthen local event interactions while suppressing noise, resulting in resilient and interpretable temporal representations. 5.2.2 Noise Filtering and Cue Localization in CBI We revisit the example introduced in Section 1 to analyze why CBI outperforms the baselines at the bottleneck representation level. We visualize the event representations of each model along with their interpretable interaction maps in Figure 4. (a) Multiplication interaction (Figure 4(a)): The interaction map assigns high importance to nearly all tokens in sentence 2, but fails to isolate the key cues. This observation aligns with our earlier analysis: while multiplication captures co-activation patterns, it also amplifies noise under limited supervision, making it difficult to highlight decisive evidence. (b) Additive interaction (Figure 4(b)): The additive variant highlights parts of sentence 2, including “disputes” and “has stalled”, yet still fails to precisely identify the most decisive tokens. This supports our claim that additive interaction tends to smooth distinctions, distributing attention over broader regions rather than producing sharp, focused signals. (c) Subtraction interaction (Figure 4(c)): This variant fails entirely, placing most of its attention on tokens in sentence 1. Without an explicit directional comparison between the two events, the model struggles to determine which event representation should be emphasized. (d, e) CBI interaction: Before applying the two separate depthwise convolutions on the respective event representations (Figure 4(d)), the model identifies “bedeviled” and “disputes” in sentence 1 as important, while assigning high importance to most tokens in sentence 2, similar to multiplication but still is differently focused. After applying the convolutions (Figure 4(e)), irrelevant information is effectively filtered out. Attention to “bedeviled”, “has stalled”, “disputes”, and “trying” is strongly amplified, while attention to other tokens is significantly suppressed. These results show that depthwise convolution of CBI effectively filters noise and strengthens relevant cues, enabling more accurate temporal relation classification. The theoretical analysis is found in Appendix 7. (a) (b) (c) (d) (e) Figure 4: Event interaction map for each event interaction method. Computation Metrics PEFT Multiplication Addition CBI Subtraction Text Concatenation Embedding Fusion Full Fine-tuning Memory Cost LoRA 29269.37 29269.37 29271.05 29269.37 29271.40 29269.37 148792.17 Adapter 29030.91 29030.91 29031.75 29030.91 29030.91 29030.91 Time LoRA 223.84 216.71 221.12 217.16 375.81 214.07 3939.75 Adapter 216.42 216.31 221.8 216.55 376.06 213.58 Table 3: Computational analysis of LoRA and Adapter on Llama-3.1-8B, reporting GPU memory usage (MB) and runtime in terms of seconds (s) across six temporal relation extraction models. 5.3 Sensitivity Analysis Kernel Size MATRES TB-Dense TCR TDD-Auto TDD-Man AVG Adapter 7 67.90 56.11 63.89 88.80 71.43 69.63 5 64.84 55.80 63.43 89.20 72.98 69.25 3 66.13 58.31 65.58 86.35 70.45 69.36 1 62.34 54.86 62.96 78.75 61.31 64.04 LoRA 7 69.76 57.05 67.44 88.80 77.63 72.14 5 71.05 58.78 69.44 89.35 77.88 73.30 3 69.92 59.72 68.52 87.55 76.57 72.46 1 69.52 59.40 66.36 84.65 72.16 70.42 Table 4: Influence of the convolutional kernel size for CBI under Adapter and LoRA settings. Kernel size 5 provides the most stable and competitive performance across all datasets. Best results are in bold blue, second-best in gray underlined. To assess the role of convolutional refinement within CBI, we conduct a sensitivity analysis on the TDD-man dataset with Llama 3.1 8B. We analyze how performance varies with different convolutional kernel sizes across five temporal relation extraction benchmarks. We vary the convolutional kernel size from 1 to 7 to assess its influence on temporal relation modeling. As shown in Table 4, extremely small kernels (e.g., size 1) significantly underperform in both Adapter and LoRA settings, indicating that minimal receptive fields provide insufficient temporal context. Kernel size 5 performs reliably across all five datasets, whereas kernel size 7 exhibits greater variance in average micro-F1. This stability is essential for cross-dataset generalization, and therefore we adopt kernel size 5 as the default configuration of CBI. 5.4 Computational Analysis We compare several interaction-based designs in PEFT settings to assess their efficiency and effectiveness (Table 3). Multiplication and CBI substantially reduce training time by more than 40% compared to Text Concatenation while keeping GPU memory usage nearly unchanged. Text Concatenation processes concatenated event pairs through a low-rank bottleneck without explicit interactions, which leads to much longer training (over 370 seconds per epoch) and weaker performance. Other lightweight variants (Embedding Fusion, Subtraction, and Addition) achieve similar computational cost but capture fewer inter-event dependencies, resulting in slightly inferior accuracy. Building on Multiplication, CBI adds only two depthwise convolutional layers with small kernels, increasing parameters by less than 0.1% and introducing negligible overhead. As shown in Table 3, CBI matches the efficiency of Multiplication while consistently achieving the best performance, demonstrating that explicitly modeling event interactions within the low-rank bottleneck is both computationally efficient and crucial for effective temporal reasoning. 6 Conclusion In this work, we analyzed why PEFT performs poorly on temporal relation extraction despite the strong general ability of large language models. We found that low-rank bottlenecks block cross-event interactions, preventing key cues such as discourse markers from being properly encoded. To overcome this limitation, we proposed Convolutional Bottleneck Interaction (CBI), a simple architecture that injects event–event interaction directly within the bottleneck. CBI first applies a lightweight depthwise convolution to enlarge the receptive field of each event and to reduce local noise, and then performs element-wise multiplication to model channel-wise co-activation between events. This design addresses the two main weaknesses of multiplicative interaction, token misalignment and noise amplification, while keeping the computational cost minimal. Experiments on five datasets and seven LLM backbones with adapters and LoRA show that CBI delivers large and consistent gains, up to +31.7 micro-F1, with negligible overhead. These results demonstrate that explicitly modeling event interaction within the low-rank space is both effective and necessary for temporal relation extraction in PEFT. More broadly, our findings suggest that relational tasks benefit from interaction-aware bottleneck designs, offering a promising direction for future parameter-efficient architectures. Beyond temporal relation extraction, the idea of modeling interactions within low-rank bottlenecks may benefit a wider range of relational reasoning tasks. Many NLP problems require capturing dependencies between events, entities, or discourse units, where effective interaction modeling is essential. Future work could explore extending interaction-aware bottleneck designs to tasks such as causal relation extraction, discourse relation classification, and document-level reasoning. Acknowledgements This research were funded by the AIDAVA (EU HORIZON-HLTH-2021-TOOL-06-03), the CHIST-ERA iTRUST project (CHIST-ERA Call 2021, FWO grant G0L0822N) and Flanders Artificial Intelligence Research program (FAIR). The computations described in this research were performed using the The Flemish Supercomputer Center Tier-1 HPC service (https://w.vscentrum.be/) and the high-performance computing resources provided by CINECA (https://w.cineca.it/), Italy. Finally, we would like to thank Minye Wu, as well as our action editor and anonymous reviewers, for their time, effort, and valuable feedback. References Abdin et al. (2024) M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kauffmann, J. R. Lee, Y. T. Lee, Y. Li, W. Liu, C. C. T. Mendes, A. Nguyen, E. Price, G. de Rosa, O. Saarikivi, A. Salim, S. Shah, X. Wang, R. Ward, Y. Wu, D. Yu, C. Zhang, and Y. Zhang Phi-4 technical report. External Links: 2412.08905, Link Cited by: §1. Bai et al. (2023) J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, S. Wang, W. Wang, S. Wu, B. Xu, J. Xu, A. Yang, H. Yang, J. Yang, S. Yang, Y. Yao, B. Yu, H. Yuan, Z. Yuan, J. Zhang, X. Zhang, Y. Zhang, Z. Zhang, C. Zhou, J. Zhou, X. Zhou, and T. Zhu Qwen technical report. External Links: 2309.16609, Link Cited by: §1. Ballesteros et al. (2020) M. Ballesteros, R. Anubhai, S. Wang, N. Pourdamghani, Y. Vyas, J. Ma, P. Bhatia, K. McKeown, and Y. Al-Onaizan Severing the edge between before and after: neural architectures for temporal ordering of events. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), B. Webber, T. Cohn, Y. He, and Y. Liu (Eds.), Online, p. 5412–5417. External Links: Link, Document Cited by: §1, §1, §2.1. Ben Zaken et al. (2022) E. Ben Zaken, Y. Goldberg, and S. Ravfogel BitFit: simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), S. Muresan, P. Nakov, and A. Villavicencio (Eds.), Dublin, Ireland, p. 1–9. External Links: Link, Document Cited by: §2.2. Borchert et al. (2025) P. Borchert, I. Vulić, M. Moens, and J. De Weerdt Language fusion for parameter-efficient cross-lingual transfer. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 25848–25868. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §2.3. Cassidy et al. (2014) T. Cassidy, B. McDowell, N. Chambers, and S. Bethard An annotation framework for dense event ordering. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), K. Toutanova and H. Wu (Eds.), Baltimore, Maryland, p. 501–506. External Links: Link, Document Cited by: §1, §4.1. Chan et al. (2024) C. Chan, C. Jiayang, W. Wang, Y. Jiang, T. Fang, X. Liu, and Y. Song Exploring the potential of ChatGPT on sentence level relations: a focus on temporal, causal, and discourse relations. In Findings of the Association for Computational Linguistics: EACL 2024, Y. Graham and M. Purver (Eds.), St. Julian’s, Malta, p. 684–721. External Links: Link, Document Cited by: §2.2. Chaturvedi et al. (2025) R. Chaturvedi, P. Baghershahi, S. Medya, and B. Di Eugenio Temporal relation extraction in clinical texts: a span-based graph transformer approach. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 25765–25788. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §1, §2.1, §5.1, footnote 1. Chen et al. (2024) Y. Chen, S. Qian, H. Tang, X. Lai, Z. Liu, S. Han, and J. Jia Longlora: efficient fine-tuning of long-context large language models. 2024, p. 8220–8238. Cited by: §2.2. Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova BERT: pre-training of deep bidirectional transformers for language understanding. External Links: 1810.04805, Link Cited by: §1, §2.1. [11] W. Fang and J. Glass Cross-document temporal relation extraction with temporal anchoring events. Cited by: §1, §2.1. Guo et al. (2025) Z. Guo, X. Cheng, Y. Wu, and T. Jin A wander through the multimodal landscape: efficient transfer learning via low-rank sequence multimodal adapter. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 16996–17004. Cited by: §2.3. Hu et al. (2022) E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. External Links: Link Cited by: §1, §2.2. Huang et al. (2023) Q. Huang, Y. Hu, S. Zhu, Y. Feng, C. Liu, and D. Zhao More than classification: a unified framework for event temporal relation extraction. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, p. 9631–9646. External Links: Link, Document Cited by: §1, §1, §2.1. Jiang et al. (2023) A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed Mistral 7b. External Links: 2310.06825, Link Cited by: §1. Kim et al. (2025) M. Kim, J. M. Kwak, L. Alssum, B. Ghanem, P. Torr, D. Krueger, F. Barez, and A. Bibi Rethinking safety in LLM fine-tuning: an optimization perspective. External Links: Link Cited by: §5. Leeuwenberg and Moens (2019) A. Leeuwenberg and M. Moens A survey on temporal reasoning for temporal information extraction from text. Journal of Artificial Intelligence Research 66, p. 341–380. Cited by: §1. Liu et al. (2019) Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Cited by: §1, §2.1. Lu et al. (2024) H. Lu, Y. Huo, G. Yang, Z. Lu, W. Zhan, M. Tomizuka, and M. Ding UniAdapter: unified parameter-efficient transfer learning for cross-modal modeling. External Links: Link Cited by: §2.2. Man et al. (2022) H. Man, N. T. Ngo, L. N. Van, and T. H. Nguyen Selecting optimal context sentences for event-event relation extraction. In Proceedings of the AAAI conference on artificial intelligence, Vol. 36, p. 11058–11066. Cited by: §1, §1, §2.1. Naik et al. (2019) A. Naik, L. Breitfeller, and C. Rose TDDiscourse: a dataset for discourse-level temporal ordering of events. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, S. Nakamura, M. Gašić, I. Zukerman, G. Skantze, M. Nakano, A. Papangelis, S. Ultes, and K. Yoshino (Eds.), Stockholm, Sweden, p. 239–249. External Links: Link, Document Cited by: §1, §1, §4.1, §4.1, §9.3. Ning et al. (2018a) Q. Ning, Z. Feng, H. Wu, and D. Roth Joint reasoning for temporal and causal relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), I. Gurevych and Y. Miyao (Eds.), Melbourne, Australia, p. 2278–2288. External Links: Link, Document Cited by: §1, §4.1. Ning et al. (2018b) Q. Ning, H. Wu, H. Peng, and D. Roth Improving temporal relation extraction with a globally acquired statistical resource. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), M. Walker, H. Ji, and A. Stent (Eds.), New Orleans, Louisiana, p. 841–851. External Links: Link, Document Cited by: §1, §1, §4.1. Ning et al. (2018c) Q. Ning, H. Wu, and D. Roth A multi-axis annotation scheme for event temporal relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), I. Gurevych and Y. Miyao (Eds.), Melbourne, Australia, p. 1318–1328. External Links: Link, Document Cited by: §2.1. Pareja et al. (2024) A. Pareja, N. S. Nayak, H. Wang, K. Killamsetty, S. Sudalairaj, W. Zhao, S. Han, A. Bhandwaldar, G. Xu, K. Xu, L. Han, L. Inglis, and A. Srivastava Unveiling the secret recipe: a guide for supervised fine-tuning small llms. External Links: 2412.13337, Link Cited by: §5. Peters et al. (2018) M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), M. Walker, H. Ji, and A. Stent (Eds.), New Orleans, Louisiana, p. 2227–2237. External Links: Link, Document Cited by: §2.1. Pfeiffer et al. (2021) J. Pfeiffer, A. Kamath, A. Rücklé, K. Cho, and I. Gurevych AdapterFusion: non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, P. Merlo, J. Tiedemann, and R. Tsarfaty (Eds.), Online, p. 487–503. External Links: Link, Document Cited by: §1, §2.2. Qin et al. (2021) L. Qin, A. Gupta, S. Upadhyay, L. He, Y. Choi, and M. Faruqui TIMEDIAL: temporal commonsense reasoning in dialog. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), C. Zong, F. Xia, W. Li, and R. Navigli (Eds.), Online, p. 7066–7076. External Links: Link, Document Cited by: §1. Qu et al. (2024) T. Qu, T. Tuytelaars, and M. Moens Introducing routing functions to vision-language parameter-efficient fine-tuning with low-rank bottlenecks. In European Conference on Computer Vision, p. 291–308. Cited by: §2.3, §5. Roccabruna et al. (2024) G. Roccabruna, M. Rizzoli, and G. Riccardi Will LLMs replace the encoder-only models in temporal relation classification?. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 20402–20415. External Links: Link, Document Cited by: §1, §1, §2.2, §2.2, §5.1. Team et al. (2024) G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, J. Ferret, P. Liu, P. Tafti, A. Friesen, M. Casbon, S. Ramos, R. Kumar, C. L. Lan, S. Jerome, A. Tsitsulin, N. Vieillard, P. Stanczyk, S. Girgin, N. Momchev, M. Hoffman, S. Thakoor, J. Grill, B. Neyshabur, O. Bachem, A. Walton, A. Severyn, A. Parrish, A. Ahmad, A. Hutchison, A. Abdagic, A. Carl, A. Shen, A. Brock, A. Coenen, A. Laforge, A. Paterson, B. Bastian, B. Piot, B. Wu, B. Royal, C. Chen, C. Kumar, C. Perry, C. Welty, C. A. Choquette-Choo, D. Sinopalnikov, D. Weinberger, D. Vijaykumar, D. Rogozińska, D. Herbison, E. Bandy, E. Wang, E. Noland, E. Moreira, E. Senter, E. Eltyshev, F. Visin, G. Rasskin, G. Wei, G. Cameron, G. Martins, H. Hashemi, H. Klimczak-Plucińska, H. Batra, H. Dhand, I. Nardini, J. Mein, J. Zhou, J. Svensson, J. Stanway, J. Chan, J. P. Zhou, J. Carrasqueira, J. Iljazi, J. Becker, J. Fernandez, J. van Amersfoort, J. Gordon, J. Lipschultz, J. Newlan, J. Ji, K. Mohamed, K. Badola, K. Black, K. Millican, K. McDonell, K. Nguyen, K. Sodhia, K. Greene, L. L. Sjoesund, L. Usui, L. Sifre, L. Heuermann, L. Lago, L. McNealus, L. B. Soares, L. Kilpatrick, L. Dixon, L. Martins, M. Reid, M. Singh, M. Iverson, M. Görner, M. Velloso, M. Wirth, M. Davidow, M. Miller, M. Rahtz, M. Watson, M. Risdal, M. Kazemi, M. Moynihan, M. Zhang, M. Kahng, M. Park, M. Rahman, M. Khatwani, N. Dao, N. Bardoliwalla, N. Devanathan, N. Dumai, N. Chauhan, O. Wahltinez, P. Botarda, P. Barnes, P. Barham, P. Michel, P. Jin, P. Georgiev, P. Culliton, P. Kuppala, R. Comanescu, R. Merhej, R. Jana, R. A. Rokni, R. Agarwal, R. Mullins, S. Saadat, S. M. Carthy, S. Cogan, S. Perrin, S. M. R. Arnold, S. Krause, S. Dai, S. Garg, S. Sheth, S. Ronstrom, S. Chan, T. Jordan, T. Yu, T. Eccles, T. Hennigan, T. Kocisky, T. Doshi, V. Jain, V. Yadav, V. Meshram, V. Dharmadhikari, W. Barkley, W. Wei, W. Ye, W. Han, W. Kwon, X. Xu, Z. Shen, Z. Gong, Z. Wei, V. Cotruta, P. Kirk, A. Rao, M. Giang, L. Peran, T. Warkentin, E. Collins, J. Barral, Z. Ghahramani, R. Hadsell, D. Sculley, J. Banks, A. Dragan, S. Petrov, O. Vinyals, J. Dean, D. Hassabis, K. Kavukcuoglu, C. Farabet, E. Buchatskaya, S. Borgeaud, N. Fiedel, A. Joulin, K. Kenealy, R. Dadashi, and A. Andreev Gemma 2: improving open language models at a practical size. External Links: 2408.00118, Link Cited by: §1. Touvron et al. (2023) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom Llama 2: open foundation and fine-tuned chat models. External Links: 2307.09288, Link Cited by: §1. Wang et al. (2025) Q. Wang, J. Hu, Z. Li, Y. Wang, D. Li, Y. Hu, and M. Tan Generating long-form story using dynamic hierarchical outlining with memory-enhancement. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), L. Chiruzzo, A. Ritter, and L. Wang (Eds.), Albuquerque, New Mexico, p. 1352–1391. External Links: Link, Document, ISBN 979-8-89176-189-6 Cited by: §1. Wang et al. (2022) Y. Wang, S. Agarwal, S. Mukherjee, X. Liu, J. Gao, A. H. Awadallah, and J. Gao AdaMix: mixture-of-adaptations for parameter-efficient model tuning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Y. Goldberg, Z. Kozareva, and Y. Zhang (Eds.), Abu Dhabi, United Arab Emirates, p. 5744–5760. External Links: Link, Document Cited by: §2.2. Wen and Ji (2021) H. Wen and H. Ji Utilizing relative event time to enhance event-event temporal relation extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, M. Moens, X. Huang, L. Specia, and S. W. Yih (Eds.), Online and Punta Cana, Dominican Republic, p. 10431–10437. External Links: Link, Document Cited by: §1, §2.1. Yao and Huang (2018) W. Yao and R. Huang Temporal event knowledge acquisition via identifying narratives. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), I. Gurevych and Y. Miyao (Eds.), Melbourne, Australia, p. 537–547. External Links: Link, Document Cited by: §1. Ye et al. (2022) A. Ye, C. Cui, T. Shi, and M. O. Riedl Neural story planning. Cited by: §1. Yuan et al. (2023) C. Yuan, Q. Xie, and S. Ananiadou Zero-shot temporal relation extraction with ChatGPT. In Proceedings of the 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, D. Demner-fushman, S. Ananiadou, and K. Cohen (Eds.), Toronto, Canada, p. 92–102. External Links: Link, Document Cited by: §2.2. Zhao et al. (2025) J. Zhao, W. Ning, Y. Fei, Y. Feng, and L. Li GDLLM: a global distance-aware modeling approach based on large language models for event temporal relation extraction. p. 8080–8091. External Links: Link, Document, ISBN 979-8-89176-335-7 Cited by: §7.6. Zhou et al. (2022) J. Zhou, S. Dong, H. Tu, X. Wang, and Y. Dou RSGT: relational structure guided temporal relation extraction. In Proceedings of the 29th International Conference on Computational Linguistics, N. Calzolari, C. Huang, H. Kim, J. Pustejovsky, L. Wanner, K. Choi, P. Ryu, H. Chen, L. Donatelli, H. Ji, S. Kurohashi, P. Paggio, N. Xue, S. Kim, Y. Hahm, Z. He, T. K. Lee, E. Santus, F. Bond, and S. Na (Eds.), Gyeongju, Republic of Korea, p. 2001–2010. External Links: Link Cited by: §1, §1, §2.1, §5.1. 7 Theoretical Analysis 7.1 Motivation PEFT methods such as LoRA and Adapter restrict learnable updates to a low-rank subspace of pretrained models. Although efficient, this ranking bottleneck limits the ability of the model to learn high-rank cross-event interactions required in temporal relation extraction. In particular, when two event mentions are represented by latent sequences, a full bilinear interaction between them is generally high-rank, whereas PEFT updates cannot reliably parameterize such transformations. To alleviate this deficiency, we propose Convolutional Bottleneck Interaction (CBI), which introduces two inductive biases on the feature map: (a) Depthwise convolution for local temporal alignment and noise suppression; (b) Element-wise multiplication for explicit second-order event interaction. We formalize why these operations restore relational expressivity without increasing the PEFT rank. 7.2 Preliminaries and Notation Let 1,2∈ℝL×dH_1,H_2 ^L× d be the contextual token representations of two event spans produced by the backbone transformer, where L is the span length and d the hidden size. PEFT introduces a low-rank bottleneck via a projection matrix down∈ℝd×rW_down ^d× r, r≪dr d, yielding i=idown∈ℝL×r,i∈1,2.E_i=H_iW_down ^L× r, i∈\1,2\. (10) We denote by i(l)∈ℝre_i(l) ^r the l-th token vector of iE_i: i(l)=i[l,:].e_i(l)=E_i[l,:]. (11) Ideal bilinear interaction: A general second-order cross-event interaction can be written as a bilinear form s⋆(1,2)=⟨,1⊤2⟩=∑c1=1r∑c2=1rc1,c2⟨1[:,c1],2[:,c2]⟩,s (E_1,E_2)= ,\,E_1 E_2 \\ = _c_1=1^r _c_2=1^rW_c_1,c_2\, _1[:,c_1],E_2[:,c_2] , (12) where ∈ℝr×rW ^r× r is a task-specific bilinear weight matrix and ⟨⋅,⋅⟩ ·,· denotes the standard inner product. When r is small due to PEFT, it is difficult to directly learn full W (or equivalently complete cross-channel mixing) is difficult. CBI operations: CBI first applies a depthwise convolution along the token axis. Let ∈ℝL×LT ^L× L be the Toeplitz matrix associated with a 1D depthwise kernel of width k. For each event: ~i=i∈ℝL×r,equivalently~i(l)=∑jθji(l+j). E_i=TE_i ^L× r, \\ e_i(l)= _j _j\,e_i(l+j). (13) Then CBI constructs the interaction feature field =~1⊙~2∈ℝL×r,[l,c]=~1[l,c]~2[l,c].F= E_1 E_2 ^L× r,\\ [l,c]= E_1[l,c] E_2[l,c]. (14) Finally, any downstream output (e.g., pooling or attention-based aggregation) maps F to a scalar score: sCBI(1,2)=⟨,vec()⟩,s_CBI(E_1,E_2)= ,\,vec(F) , (15) where vec()∈ℝLrvec(F) ^Lr stacks all tokens and channels into a vector, and ∈ℝLrp ^Lr is a vector of learned linear weights. We now show that F provides an explicit bilinear interaction compatible with such outputs. 7.3 Element-wise Multiplication as Explicit Diagonal Bilinear Interaction We first characterize the bilinear structure induced by F. Lemma 7.1 (Diagonal bilinear identity). For each channel c∈1,…,rc∈\1,…,r\, ∑l=1L[l,c]=(~1⊤~2)c,c. _l=1^LF[l,c]= ( E_1 E_2 )_c,c. (16) Proof. Let channel c be arbitrary. By definition of matrix multiplication, (~1⊤~2)c,c=∑l=1L~1[l,c]~2[l,c]. ( E_1 E_2 )_c,c= _l=1^L E_1[l,c] E_2[l,c]. (17) By the definition of F in (14), ~1[l,c]~2[l,c]=[l,c] E_1[l,c] E_2[l,c]=F[l,c]. This yields (16). Consequence. Lemma 7.1 shows that F is an explicit token-wise representation of diagonal bilinear interactions. Thus, any linear output that aggregates tokens within each channel (a special case of (15)) can directly realize a diagonal bilinear score sdiag(1,2)=∑c=1rc(~1⊤~2)c,c=∑c=1rc∑l=1L[l,c],s_diag(E_1,E_2)= _c=1^rw_c\, ( E_1 E_2 )_c,c\\ = _c=1^rw_c _l=1^LF[l,c], (18) for some ∈ℝrw ^r. Lemma 7.2 (Linear recoverability). For any ∈ℝrw ^r, there exists ∈ℝLrp_w ^Lr such that sdiag(1,2)=⟨,vec()⟩,s_diag(E_1,E_2)= _w,\,vec(F) , (19) where vecvec denotes column-wise vectorization. Proof. Construct p_w by assigning weight wcw_c to all L entries corresponding to channel c in vec()vec(F), and zero elsewhere. Then ⟨,vec()⟩=∑c=1rwc∑l=1L[l,c]=sdiag(1,2), _w,vec(F) = _c=1^rw_c _l=1^LF[l,c]=s_diag(E_1,E_2), using (18). Interpretation. Element-wise multiplication converts two first-order representations into an explicit second-order interaction field. Any later linear aggregation can pick up these diagonal bilinear signals, whereas a PEFT-constrained backbone would otherwise need to implicitly provide them through low-rank updates. 7.4 Depthwise Convolution as Soft Alignment and Denoising We next show that depthwise convolution preserves diagonal interactions up to a controlled error and improves their signal-to-noise ratio. Signal-to-noise decomposition. Write each low-rank event sequence as i=i+i,E_i=S_i+ _i, (20) where iS_i is the latent semantic signal relevant to temporal relations, and i _i represents noise. Lemma 7.3 (Stability of diagonal interaction). Assume ‖⊤−‖2≤ϵ\|T T-I\|_2≤ε. Then for any 1,2∈ℝL×rS_1,S_2 ^L× r, ‖diag(1⊤⊤2)−diag(1⊤2)‖2≤ϵ‖1‖F‖2‖F. \|diag(S_1 T TS_2)-diag(S_1 S_2) \|_2\\ ≤ε\,\|S_1\|_F\,\|S_2\|_F. (21) Proof. diag(1⊤⊤2)−diag(1⊤2) (S_1 T TS_2)-diag(S_1 S_2) =diag(1⊤(⊤−)2). =diag (S_1 (T T-I)S_2 ). Taking norms and using ‖diag()‖2≤‖F\|diag(A)\|_2≤\|A\|_F, ‖⋯‖2 \|·s\|_2 ≤‖1⊤(⊤−)2‖F ≤\|S_1 (T T-I)S_2\|_F ≤‖⊤−‖2‖1‖F‖2‖F ≤\|T T-I\|_2\,\|S_1\|_F\,\|S_2\|_F ≤ϵ‖1‖F‖2‖F. ≤ε\,\|S_1\|_F\,\|S_2\|_F. Meaning. Lemma 7.3 formalizes soft alignment: a low-pass depthwise kernel makes ⊤T T close to I on smooth signals, so diagonal bilinear interactions are approximately preserved. Lemma 7.4 (Noise contraction and SNR improvement). Assume ‖i‖22≤α‖i‖22\|T _i\|_2^2≤α\| _i\|_2^2 for some 0<α<10<α<1. Then the expected diagonal noise interaction energy satisfies ‖diag((1)⊤(2))‖22≤α2‖diag(1⊤2)‖22.E \|diag((T _1) (T _2)) \|_2^2≤\\ α^2\,E \|diag( _1 _2) \|_2^2. (22) Consequently, the signal-to-noise ratio of diagonal interactions after convolution is improved by at least 1/α21/α^2. Proof. Under the assumption, the energy of each noise sequence is reduced by a factor α after applying T. Since diagonal interaction is a sum of products of corresponding noise entries, its second-moment energy scales multiplicatively, giving a factor α2α^2. 7.5 Effective Second-order Expressivity under PEFT Constraints PEFT constrains parameter updates to rank-r forms, e.g., Δ=downup, =W_downW_up, (23) where down∈ℝd×rW_down ^d× r is the down-projection and up∈ℝr×dW_up ^r× d is the up-projection, with r≪dr d. This limits the ability of the backbone to create high-rank bilinear interactions between two events through parameter adaptation alone. CBI instead produces second-order interactions in the feature space: CBI(1,2)=(1down)⊙(2down)=. _CBI(H_1,H_2)=\\ (TH_1W_down) (TH_2W_down)=F. (24) By Lemma 7.1, F explicitly encodes diagonal bilinear interactions, and by Lemma 7.2, any linear output can exploit them directly. Lemmas 7.3–7.4 further show that depthwise convolution stabilizes these interactions under token misalignment and improves their signal-to-noise ratio. Therefore, CBI restores second-order relational expressivity while keeping the PEFT rank unchanged and adding negligible parameters. 7.6 Generalizing CBI to Graph Neural Network (GNN)–Based Temporal Relation Extraction Models Methods MATRES TB-Dense TCR TDD-Auto TDD-Man AVG Adapter Text Concatenation 60.25 51.25 61.49 65.77 64.41 60.63 Embedding Fusion 57.70 43.13 59.76 48.31 37.20 49.22 CBI 66.61 56.23 65.15 90.02 75.99 70.80 LoRA Text Concatenation 66.24 59.41 63.04 72.02 64.50 65.04 Embedding Fusion 56.20 41.97 59.76 36.74 36.73 46.28 CBI 72.37 58.93 68.36 90.09 78.79 73.71 Table 5: Performance comparison of Convolutional Bottleneck Interaction (CBI) with two baselines under Adapter and LoRA settings augmented with inference over a global document-level event graph applied on five temporal relation extraction datasets (micro-F1%) using the Llama 3.1 8B backbone. Best results are in bold blue, second-best in gray underlined. We also investigate whether our interaction methods generalize to GNN-based temporal relation extraction, focusing on GDLLM (Zhao et al., 2025), a recent state-of-the-art model. GDLLM first uses an LLM to encode all event mentions in a document. It then builds a global event graph, where nodes are events, and edges reflect their discourse distance. A distance-aware GNN propagates information across this graph, helping the model reason about long-range temporal dependencies. To test our approach in this setting, we replace the GDLLM event-pair encoder with three interaction strategies: Text Concatenation, Embedding Fusion, and our CBI module in both Adapter and LoRA configurations. The results in Table 5 show three clear trends. First, Text Concatenation performs reasonably well, but the improvements remain limited. Even with a strong GNN on top, simply merging two event representations does not provide enough direct interaction between events. Second, Embedding Fusion performs the worst in all settings. This confirms that shallow element-wise operations are easily affected by the low-rank constraints of PEFT modules, causing important temporal cues to be lost before the GNN can use them. Third, CBI achieves the best results in all datasets and all PEFT configurations. These improvements suggest that CBI provides cleaner and more informative pair representations, which the GNN can then integrate more effectively during global message passing. Overall, even after adding a GNN module, our method remains the strongest interaction mechanism. This shows that better event interaction inside the PEFT bottleneck is crucial, and that CBI offers an effective and lightweight solution in both transformer-only and GNN-augmented settings. 8 Dataset statistics We conduct experiments on five widely used datasets for temporal relation extraction, whose statistics are provided in Table 6. Dataset Train Dev Test MATRES 3,715 1,240 1,238 TB-Dense 1,910 638 636 TCR 1,939 648 646 TDD-Auto 22,981 7,660 7,661 TDD-Man 3,673 1,225 1,224 Table 6: Statistics of the five temporal relation extraction datasets in terms of number of temporal relation examples. 9 Experimental Analysis 9.1 Results of Varying Bottleneck Capacity Table 7 shows that increasing the bottleneck rank r brings only small improvements. For example, when r increases from 128 to 512, Text Concatenation improves by about 3 points in both Adapter and LoRA settings. Embedding Fusion shows similarly limited gains. However, even when r is larger, these methods still perform much worse than CBI. In contrast, CBI achieves clearly better results without increasing the rank. It indicates that performance improvement of CBI does not come from a larger parameter capacity, but from a more effective event interaction mechanism. Methods Adapter (rank r) LoRA (rank r) r=128 r=256 r=512 r=128 r=256 r=512 Text Concatenation 60.06 61.87 63.36 64.94 65.92 66.32 Embedding Fusion 42.69 43.37 44.14 41.61 42.78 43.94 CBI 69.25 70.31 71.34 73.30 73.94 74.71 Table 7: Performance comparison of CBI results with results baseline methods obtained with different bottleneck ranks r under Adapter and LoRA settings (the averaged micro-F1 score across all five benchmarks). Best results are in bold blue, second-best in gray underlined. 9.2 Error Analysis (a) (b) (c) (d) (e) (f) (g) Figure 5: Visualization of event interaction maps for evaluating the temporal relation extraction task: (a) Full Fine-tuning, (b) Text Concatenation with Adapter fine-tuning, (c) LoRA fine-tuning, Embedding Fusion with (d) Adapter fine-tuning, and (e) LoRA fine-tuning, CBI with (f) Adapter fine-tuning, and (j) LoRA fine-tuning. marks tokens emphasized by different methods. CBI offers a computationally efficient alternative to full fine-tuning while effectively capturing distributed temporal cues, outperforming other parameter-efficient methods in preserving critical signals. We perform a unified error analysis across both LoRA and adapter settings, each consisting of 500 samples, to examine whether model behavior is consistent across parameter-efficient tuning methods. Our analysis focuses on the types and structure of errors based on manual analysis. In both settings, two baselines (Text Concatenation and Embedding Fusion) show a consistent failure pattern: they often reduce containment relations to a simple order relation. In the LoRA setting, for gold IS_INCLUDED, Embedding Fusion and Text Concatenation misclassify 53.7% and 43.5% of cases as BEFORE/AFTER, respectively, while CBI reduces this to 28.7%. Similar patterns are observed in the adapter setting. This shows that the baselines struggle to represent nested temporal structure, while CBI better preserves it. Another common error is confusion between INCLUDES and IS_INCLUDED. In the LoRA setting, direction confusion accounts for 24.1% and 27.8% of containment errors for Embedding Fusion and Text Concatenation, respectively, compared to only 9.3% for CBI. In the adapter setting, Text Concatenation further shows a strong bias toward predicting INCLUDES across diverse cases. These results indicate that the baselines often ignore directionality, while CBI captures it more reliably. However, CBI introduces a different error pattern. For gold order relations (BEFORE/AFTER), CBI predicts containment in 17.2% of cases, compared to 13.1% for Embedding Fusion. Similar trends appear in the adapter setting. This suggests that stronger interaction modeling may sometimes lead to interpreting semantic relatedness as temporal containment. Finally, all models struggle with the simultaneity relation. In the LoRA setting, CBI correctly predicts 20.0% (3/15) of SIMULTANEOUS cases, while the baselines achieve 0%. Overall, the error patterns are consistent across LoRA and adapter. The baselines tend to simplify structure (reducing containment relations to order relations or ignoring their direction). 9.3 Case Study To understand why we need event interaction in a PEFT setting, we add one more example from the TDD-Man dataset (Naik et al., 2019): Sentence 1: … luded he had probably been [event1]killed[/event1] soon after being captured. Sentence 2: art in mid-1996 after the [event2]defection[/event2] of one of its top leader. The gold relation is BEFORE and this is supported by the following cues. “Been killed” indicates a completed past event, suggesting temporal precedence. “Soon after” and “after” provide explicit ordering signals within the narrative. “Defection”, anchored by “in mid-1996”, serves as a later temporal reference point. Together, these cues indicate that “killed” happened before “defection”. Model analysis shows different sensitivities to these cues. Full fine-tuning captures been “killed”, “after”, and “defection”. Text Concatenation fails in both adapter and LoRA settings. Embedding Fusion captures “defection” with adapters, and “been killed”, “after”, and “defection” with LoRA. CBI captures “soon after” and “after” with adapters, and “been killed”, “soon after”, and “defection” with LoRA, suggesting better preservation of the distributed temporal signals. While full fine-tuning is effective at capturing these critical cues, it comes with significantly higher computational costs and longer training times. In comparison, CBI provides a lightweight alternative while still being able to capture the temporal cues. 9.4 Wider Context of Examples The full context for the example shown in Figure 1 and Figure 4 is: But while the two Slavic neighbors see themselves as natural partners, their relations since the breakup of the Soviet Union have been bedeviled by a number of disputes _ Black Sea naval bases, border problems and Ukraine’s natural gas debts. “We have covered the entire list of questions and discussed how we will be tackling them,” Yeltsin was quoted as saying by the ITAR-Tass news agency. “I must say there are no unsettled problems any more. We have solved them all.” But his sweeping statement contained no details, and gave no indication of how the [event1]disputes[/event1] could be resolved. Their solution would require a compromise between the two nations’ parliaments. A major dispute concerns a broad political treaty calling for border demarcation, which the two presidents signed last May In effect, the treaty amounts to Russian recognition of Ukraine’s sovereignty and borders, and the Ukrainian parliament has already ratified it. However, Russia has stalled ratification, [event2]trying[/event2] totiet it to an agreement that would permit the Russian navy to use a naval base on Ukraine’s Crimean peninsula for at least 20 more years. The full context for the example shown in Figure 5 is : January this year, British police officers who had been searching for Howes concluded he had probably been [event1]killed[/event1] soon after being captured. The Foreign Office said it had informed the family of Howes, 37 years old when he was kidnapped, that he probably died within weeks or months of his capture on March 26, 1996. Obviously, it is deeply discouraging for the family after 22 months, but there is no proof of life. But there is no evidence in either direction _ that there is proof of life or death,” said a Foreign Office spokesman, speaking with customary anonymity. We will continue to do everything we can to establish what has happened.” Thai military officials who monitor Cambodian affairs said privately Thursday that Britain, through its embassies in Thailand and Cambodia, has been pushing hard to resolve the Howes case as the second anniversary of his abduction nears. Nhek Bunchhay, who had been closely involved in the search for Howes before having to flee the Cambodian capital after a coup d’etat last year, appeared confident he would find Howes’ remains. He said he received information from Khmer Rouge guerrillas on where the body had been buried, and recently ordered 10 of his men from a force of 500 near Khmer Rouge headquarters in Anlong Veng to conduct the search. If and when the remains are found, he said, they would be turned over to the British Embassy, apparently meaning they would be sent across the border into Thailand and onward to Bangkok. Nhek Bunchhay said he now believed Howes had been killed within a week of his capture by a Khmer Rouge faction loyal to Pol Pot, then the guerrilla group’s leader. Pol Pot is considered responsible for the radical policies that led to the deaths of as many as 1.7 million Cambodians when the communist group held power in the late 1970s. At the time Howes was captured, the Khmer Rouge were a more or less united guerrilla force with more than 10,000 men under arms. But the group began to fall apart in mid-1996 after the [event2]defection[/event2] of one of its top leaders, Ieng Sary.