Paper deep dive
FSE: Continual Learning for Named Entity Recognition by Fast-Slow Experts
Yunan Zhang, Yang Fan, Heng Li, Xiangping Wu, Qingcai Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/2/2026, 12:57:42 PM
Summary
The paper introduces FSE (Fast-Slow Experts), a novel model for Continual Learning for Named Entity Recognition (CLNER). FSE utilizes a shared 'fast expert' to learn token-level links for filtering unlikely spans and task-specific 'slow experts' for span classification. This architecture aims to mitigate catastrophic forgetting and improve plasticity by reducing the learning burden. The method also incorporates a length-decay negative sampling strategy to handle span imbalance. Experiments on OntoNotes and FewNERD datasets demonstrate state-of-the-art performance.
Entities (8)
Relation Signals (9)
FSE → usescomponent → Fast-Slow Experts
confidence 95% · This paper proposes FSE, a Fast-Slow Experts enhanced span-based NER model
Fast-Slow Experts → consistsof → Fast Expert
confidence 92% · The shared fast expert learns token-level links... while the task-specific slow expert performs span classification
Fast-Slow Experts → consistsof → Slow Expert
confidence 92% · The shared fast expert learns token-level links... while the task-specific slow expert performs span classification
FSE → appliestechnique → Length-Decay Negative Sampling
confidence 90% · A length-decay negative sampling strategy to mitigate span imbalance is also introduced.
FSE → evaluatedon → FewNERD
confidence 90% · Extensive experiments on OntoNotes and FewNERD synthestic datasets demonstrate that FSE achieves state-of-the-art performance
FSE → evaluatedon → OntoNotes
confidence 90% · Extensive experiments on OntoNotes and FewNERD synthestic datasets demonstrate that FSE achieves state-of-the-art performance
FSE → solvesproblem → Catastrophic Forgetting
confidence 90% · However, existing methods suffer from catastrophic forgetting... This paper proposes FSE
Fast Expert → learns → Token-Level Links
confidence 88% · The shared fast expert learns token-level links to efficiently filter out unlikely spans
Slow Expert → performs → Span Classification
confidence 88% · the task-specific slow expert performs span classification only on the remaining candidates
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Continual Learning for Named Entity Recognition (CLNER) enable models to incrementally learn new entity types without forgetting previously acquired ones. However, existing methods suffer from catastrophic forgetting and insufficient exploitation of shared information across tasks. This paper proposes FSE, a Fast-Slow Experts enhanced span-based NER model for CLNER. The shared fast expert learns token-level links to efficiently filter out unlikely spans, while the task-specific slow expert performs span classification only on the remaining candidates. It stabilizes learning by promoting knowledge sharing across tasks and maintains plasticity by reducing learning burden at each task. A length-decay negative sampling strategy to mitigate span imbalance is also introduced. Extensive experiments on OntoNotes and FewNERD synthestic datasets demonstrate that FSE achieves state-of-the-art performance in CLNER scenarios, with effectiveness of each component, empirical evidence of faster convergence and expected functionality of both experts.
Tags
Links
- Source: https://arxiv.org/abs/2607.22075v1
- Canonical: https://arxiv.org/abs/2607.22075v1
Trouble viewing inline? Open PDF directly →
Full Text
44,818 characters extracted from source content.
Expand or collapse full text
FSE: Continual Learning for Named Entity Recognition by Fast-Slow Experts Yunan Zhang zhangyunan@stu.hit.edu.cn Yang Fan yfan@stu.hit.edu.cn Heng Li 20b951012@stu.hit.edu.cn Xiangping Wu wuxiangping@hit.edu.cn Qingcai Chen qingcai.chen@hit.edu.cn Abstract Continual Learning for Named Entity Recognition (CLNER) enable models to incrementally learn new entity types without forgetting previously acquired ones. However, existing methods suffer from catastrophic forgetting and insufficient exploitation of shared information across tasks. This paper proposes FSE, a Fast-Slow Experts enhanced span-based NER model for CLNER. The shared fast expert learns token-level links to efficiently filter out unlikely spans, while the task-specific slow expert performs span classification only on the remaining candidates. It stabilizes learning by promoting knowledge sharing across tasks and maintains plasticity by reducing learning burden at each task. A length-decay negative sampling strategy to mitigate span imbalance is also introduced. Extensive experiments on OntoNotes and FewNERD synthestic datasets demonstrate that FSE achieves state-of-the-art performance in CLNER scenarios, with effectiveness of each component, empirical evidence of faster convergence and expected functionality of both experts. keywords: Continual Learning, Named Entity Recognition, Fast-Slow Experts †journal: Pattern Recognition Letters [label1]organization=School of Computer Science and Technology, Harbin Institute of Technology, city=Shenzhen, state=Guangdong, country=China 1 Introduction Continual learning is pivotal for advancing evolvable artificial general intelligence, enabling models to acquire new knowledge from streaming data [1]. In this field, continual learning for named entity recognition (CLNER) is key to developing real-time or personal application that requires dynamic adaptation to changing scenarios. NER aims to extract entities from unstructured text, CLNER further enables the incremental learning of new entity types over time without forgetting previously learned ones [2]. For example, after learning to detect medical terminology by training on medical text, the model can still accurately identify previously learned entity types like persons and locations, without requiring retraining from scratch. CLNER always encounters catastrophic forgetting [3, 4], where models trained on new tasks tends to rapidly disrupt previously learned ones. To mitigate this, standard NER models often incorporate anti-forgetting techniques such as knowledge distillation or model expansion. Meanwhile, these efforts also witness the paradigm shift of NER from sequence tagging to span-based methods [5]. Here we focus on span-based methods, which enumerates and classify all possible text spans, providing richer representations and often yielding superior performance. Notably, recent large language models (LLMs) show promise in CLNER owing to their strong generalization capabilities. However, empirical studies [6] reveal that LLMs still underperform their smaller, NER-specific Bert-based counterparts. Moreover, The prohibitive computational demands and challenges in dynamic updates of large parameters also hinder its practical deployment. Figure 1: Compared to learning massive spans in traditional methods, the fast expert in FSE efficiently learns a few links to filter out most spans, which reduces the learning burden on the slow expert that ultimately detects entities. Hence, span-based CLNER methods remain preferable, but we argue that there still exist unsolved challenges from two aspects: 1) First, task interference may lead to conflicting parameter updates during continual learning. Thus, models capable of explicitly modeling task-shared information can facilitate stability in this process. However, existing span-based methods, which only share the Bert base encoder, fail to explore additional high-level information specific to NER that could be shared between different tasks. For example, when a span text ”[PER] at” (a person entity but followed by a preposition) has already been learned as a non-entity span in previous tasks, this pattern is also unlikely to be the entity in subsequent tasks. Such knowledge could be shared. 2) Second, model’s plasticity gradually diminishes as learning progresses [7], making the acquisition of new entity types increasingly difficult. Thus, reducing the learning burden for each task would benefit continual learning. For example, given a currently learned non-entity span ”[PER] at”, model should readily identify the extended span ”[PER] at the” as a non-entity as well. Here we propose that a properly designed ”links” between adjacent tokens can address both considerations. Specifically, we introduce a Fast-Slow Experts enhanced span-based model (FSE), where the fast expert (shared across tasks) models these links, while the slow experts (task-specific) performs standard span classification at each task. As shown in Fig 1, given a sentence with n tokens, traditional span-based model would laboriously enumerate all 0.5(n2−n)0.5(n^2-n) spans to detect entities for each task. Instead, our fast expert first learns the relatively fewer links (n−1n\!-\!1) between adjacent tokens. We define that links within an entity span are all strong, while non-entity spans must contain at least one weak link. Notably, this link mechanism is merely a designed inductive bias imposed on the model, the specific link strengths are still automatically learned by the model itself during training. Upon this design, we expect that once the fast expert learn a weak links (e.g, between ”[PER] at”, massive spans cover this link can be filtered out rapidly. Then the remaining fewer spans are focused by the slow expert. This substantially alleviated the span learning burden of slow expert, and the sharable mechanism of fast experts also benefit the newcome slow experts in subsequent tasks. In implementation, dual experts are jointly optimized to avoid error propagation. First, the fast expert applies softmin pooling to aggregate the learned link scores within a certain span into this span’s ”fast score”, ensuring the span-level fast score tends to be low if any weak link is detected. Next, The fast score is fused with the slow expert’s output score for each span, ensuring that an entity is recognized only when the fast expert confidently identifies strong links and the slow expert confirms it as a valid entity. Spans without meeting both conditions would be rejected. During continual learning, we use knowledge distillation to distill the fused outputs of dual experts in previous tasks for transferring prior knowledge. Moreover, benefiting from fast expert’s efficiency in filtering out spans, fewer negative spans are needed for explicit learning. This makes negative sampling, a technique aiming to mitigate the positive-negative span imbalance inherent in span-based models, particularly suitable. We propose a length-decay negative sampling strategy that adjusts the sampling probability based on span length, applying it to both current task learning and old task distillation. Extensive experiments on synthesized datasets from OntoNotes and FewNERD show the effectiveness of FSE, achieving SoTA performance among various baseline including LLMs methods. The contributions of this paper include: • We identify the potential of adjacent token links to promote the stability and plasticity in span-based CNLER model. • We introduce jointly optimized fast-slow experts to enhance standard span-based CLNER model, where the fast expert utilizes links to guide the span modeling of slow expert. This reduces task-specific learning difficulty while enabling effective inter-task knowledge sharing. • Extensive experiments including LLM-based competitor demonstrate superior performance of our FSE in CLNER scenarios. 2 Related Work Named Entity Recognition constitutes a cornerstone of information extraction in natural language processing (NLP), aiming to recognize predefined entity types from unstructured text. NER is traditionally formalized as sequence tagging by adopting CRF after neural representations [8, 9]. In contrast, span-based methods enumerate all possible spans within the text and directly classify them without tagging scheme (BIO) conversion and accommodate nested entities [10, 11]. Besides, Machine Reading Comprehension (MRC)-based methods use pointer networks to locate entity positions in the input text [12, 13]. Beyond these extraction-based paradigm, generation-based approaches such as Seq2Seq-based directly generate entity mentions from the given input [14, 15]. Recent LLM-based NER methods are also generative, exhibiting remarkable generalization capabilities in handling both seen and unseen entity types. However, smaller models remain valuable due to their efficiency and domain-specific adaptability. Continual Learning NER tackles the evolving scenarios where new entity types emerge sequentially. This progressive shift in data distribution violates the i.i.d. assumption, leading to catastrophic forgetting of previously learned entity types. CLNER aim to balance the retention of past learning with the acquisition of new information, also known as the stability-plasticity dilemma. Monaikul [2] first framed CLNER as class-incremental continual learning and introduce sequence tagging-based AddNER and ExtendNER. The training data for each sequential task are annotated only with the the entity types currently being learned, and the model is evaluated on data that requires predicting all accumulated learned entity types. AddNER uses an individual head for each new task and ExtendNER uses a single unified head shared across all tasks, both incorporating knowledge distillation. Then, L&R [16] and ExtendNER+DLD [17] both build upon ExtendNER to further improve its performance. SpanKL [11] first uses span-based modeling and binary classification in CLNER to alleviate conflicts in non-entity types across different tasks. SKD-NER [18] further uses reinforcement learning upon it. However, all these methods share information solely through the underlying BERT parameters and neglect the potential to leverage other types of information shared across tasks. In contrast, in CV task such as continual semantic segmentation, architecture decomposition methods have explored decomposing the model into task-specific and task-sharing components, where the task-sharing part supports reconciling old and new knowledge simultaneously, with the task-specific part adapting to new tasks [19]. For example, Representation Compensation Networks [20] uses structural re-parameterization to decouple CNN modules into two parallel branches for shared and task-specific information. LAG [21] disentangles semantic features under channel-wise and spatial-level to simultaneously reconcile knowledge inheritance and new-task learning, where semantic-invariant knowledge is modeled as abstract prototypes shared across tasks. To the best of our knowledge, we are the first to explore shareable information specific to NER under CL setting, especially span-based methods, to improve overall performance across tasks. Figure 2: Illustration of FSE: The task-shared fast expert for links modeling outputs fast score. The task-specific slow expert for span modeling outputs span score. Both scores are finally fused. Previous tasks and current tasks are optimized by KL loss and BCE loss, respectively. Negative sampling apply in both optimizations. 3 Method In this section, we first formulate CLNER, and then introduce the base architecture of FSE, including slow expert, fast expert and their collaboration mechanism. Finally, we describe the training techniques employed. 3.1 CLNER Formulation CLNER is formally defined as: An NER model is sequentially trained on a stream of tasks 1,2,…,LT_1,T_2,…,T_L. Each task lT_l introduces a new set of entity types ℰl=el1,el2,…E_l=\e_l^1,e_l^2,…\. Dataset for lT_l only annotates its corresponding entity types ℰlE_l. Typically, during training on task lT_l, the model cannot access data from previous tasks 1,…,l−1T_1,…,T_l-1. The learning process begins by training the initial model ℳ1M_1 on 1T_1 to recognize entity types in ℰ1E_1. For each subsequent step l>1l>1, the model ℳlM_l is incrementally trained on the data of lT_l based on ℳl−1M_l-1, aiming to recognize all previously learned entity types ⋃i=1lℰi _i=1^lE_i. 3.2 Model Architecture Fig. 2 shows the overall architecture of FSE, including contextual encoder for token learning, task-shared fast expert for link learning and task-specific slow expert for span learning. Negative sampling applies in both KL and BCE optimizations. 3.2.1 Contextual Encoder Given input sentence X with n tokens [x1,x2,…,xn][x_1,x_2,...,x_n], we define span sijs_ij as continuous tokens from xix_i to xjx_j, where 1≤i≤j≤n1\!≤\!i≤\!j≤\!n. The contextual encoder capture tokens dependencies and generates contextualized representations for each token. We employ the widely-used Bert-base PLMs as our contextual encoder, formulated as: =(X)H= Encoder(X) (1) where =[1,2,…,n]∈ℝn×dhH=[h_1,h_2,…,h_n]\!∈\!R^n× d^h represents the contextualized representations vector for each token, dhd^h is hidden size. 3.2.2 Slow Expert The slow expert models the representation of each span to facilitate span classification. We follow SpanKL to use a simple scaled dot-product interaction between the ”start” and ”end” feature space of token. This means the slow expert is boundary-aware and responsible for detecting span boundaries, enabling the fast expert to focus on the information within the spans. Specifically, for each entity types, we use two distinct single-layer feedforward network (FFN) to yield the start and end representations of tokens, respectively. Then span representation hsijh^s_ij is computed as: sij ^s_ij =[h1sij,h2sij,…hKsij]∈ℝK =[h^s_ij_1,h^s_ij_2,… h^s_ij_K] ^K (2) hksij h^s_ij_k =kstart(i)⊺⋅kend(j)×(do)−0.5 = FFN^start_k(h_i) · FFN^end_k(h_j)×(d^o)^-0.5 where k denote the k-th entity type among the total K types currently being learned. All the start FFN^start and end FFN^end relative to each entity type have the same output dimension dod^o. There are totally 2×K2× K distinct FFNs. This is similar to multi-head attention mechanism by treating start, end and entity type as Query, Key and Head and enables seamless addition of FFNs initialized for the newcome tasks. 3.2.3 Fast Expert Let αi∈ℝ _i denote the link score, which quantifies the link strength between adjacent tokens xix_i and xi+1x_i+1 in the sequence X, structured as: x1,α1¯,x2,α2¯,x3,…,xn−1,αn−1¯,xnx_1, _1,x_2, _2,x_3,...,x_n-1, _n-1,x_n, where higher score indicates stronger link. There are total n−1n\!-\!1 link scores within sentence of length n and each link score is dynamically computed by the fast expert as: ai=linkstart(i)⊺⋅linkend(i+1)×(dl)−0.5 a_i= FFN^start_link(h_i) · FFN^end_link(h_i+1)×(d^l)^-0.5 (3) where linkstart FFN^start_link and linkend FFN^end_link are two distinct FFNs modeling the link’s start and end feature space respectively. dld^l is output dimension. This process is similar to span modeling in Equ. 2, allowing it to be treated as an ”additional entity types” modeling step, which facilitates parallel computation in practice. Notably, linkstart FFN^start_link and linkend FFN^end_link of fast expert are shared across tasks. 3.2.4 Span-Level Fast Score To propagate link information to each span, we need to broadcast the obtained 1D token-level link scores into 2D span-level scores (termed as fast scores f). As shown in Fig. 3, we aggregate all link scores within a certain span and pooling them into a single relative fast score. Given our design objective that a span should be rejected if it contains even a single weak link (i,e,. the weakest link acts as an upper bound), we introduce softmin-pooling, a stable and smooth version of min-pooling, to mitigate potential gradient sparsity and instability issues. The softmin is implemented via an inverted logsumexp operator as: fsij f^s_ij =Pooling(ai,…,aj−1)=−τ⋅log∑a∈exp(−a/τ) =Pooling(\a_i,…,a_j-1\)=-τ· _a (-a/τ) (4) where =ai,…,aj−1A\!=\!\a_i,…,a_j-1\ is the set of link scores within span sijs_ij, temperature τ controls the sharpness of the pooling and is set to 0.10.1. Notably, we also address the challenge of batch-level paralleled implementation for softmin-pooling by heuristically using the logcumsumexp operator in PyTorch. Figure 3: Using softmin-pooling to propagate link scores into span-level fast scores from a matrix perspective. 3.2.5 Dual Experts Fusion Given both span-level scores derived from the slow expert: sij∈ℝKh^s_ij\!∈\!R^K (containing K scalar scores regarding each entity type) and from the fast expert: fsijf^s_ij, we fuse them into final score for classification. We proactively apply the sigmoid activation to both scores into 0∼10 1. As expected, recognition of entity requires consensus from both experts, where a low score from either expert leads to rejection. This fusion requirement can be satisfied by Harmonic Mean. Fused score ~sij h^s_ij is computed as: hksij≔σ(hksij),fsij≔σ(fsij) h_k^s_ij σ(h_k^s_ij), f^s_ij σ(f^s_ij) (5) ~sij=Fuse(sij,fsij)=[fuse(h1sij,fsij),…,fuse(hKsij,fsij)] h^s_ij=\!Fuse(h^s_ij,f^s_ij)=\![fuse(h_1^s_ij,f^s_ij),…,fuse(h_K^s_ij,f^s_ij)] (6) fuse(x,y)=2xy/(x+y) (x,y)=2xy/(x+y) (7) where σ(⋅)σ(·) is sigmoid function normalizing both scores, then Fuse() function implements a broadcastable Harmonic Mean operation, ensuring that the link information contributes to all entity types. Notably, spans of length 1 lack relative link score so their fused scores directly adopt the slow expert scores. We employ Binary Cross-Entropy loss function for coherent optimization [11], which independently determines whether a span belongs to a specific entity type. Notably, Eq. 5 ensures that the final fused score ~sij h^s_ij is probabilities, we denote here as p^ijk p_ij^k. Then the loss is computed as: ℒbce=−∑i,j,k[pijklogp^ijk+(1−pijk)log(1−p^ijk)] _bce=- _i,j,k [p_ij^k p_ij^k+(1-p_ij^k) (1- p_ij^k) ] (8) where pijkp_ij^k is the one-hot label for span sijs_ij being k-th entity type. 3.2.6 Knowledge Retention We use knowledge distillation (KD) to transfer knowledge from both experts learned in previous tasks by distilling their fused scores. Specifically, before training on a new task, we perform a one-shot prediction on the new task’s data to generate pseudo labels (i.e., distilled labels) of all previously learned entity types. Than the distilled labels are used to optimize the predictions for old tasks in current step via a Bernoulli KL-divergence loss: ℒkd=∑i,j,k[p¯ijklog(p¯ijk/p^ijk)+(1−p¯ijk)log((1−p¯ijk)/(1−p^ijk))] _kd= _i,j,k [ p_ij^k ( p_ij^k/ p_ij^k)+(1- p_ij^k) ((1- p_ij^k)/(1- p_ij^k)) ] (9) where p¯ijk p_ij^k is the distilled label generated by span sijs_ij for the k-th entity type from old tasks. Here, p^ijk p_ij^k is fused score similar to that in Equ. 8 but specially for the old entity types. 3.2.7 Length-Decay Negative Sampling In span-based models, positive spans (entities) are significantly outnumbered by negative spans. This severe imbalance leads to biased optimization during standard training. Negative sampling aims to address the imbalance by randomly removing a portion of negative spans during learning, but it may result in insufficient learning or the omission of hard negative spans. Opportunely, fast expert can provide shortcut information to reject negative spans, enabling lossless integration of negative sampling. It also mitigates the risk of overly optimizing weak links. Given the empirical observation that longer spans are less likely to be entities, we design a length-decay negative sampling strategy for our loss computation. As shown in Fig. 4, negative sampling probabilities PnegP_neg corresponding to span length ℓ is design as: Pneg(ℓ)=η⋅(0.95⋅σ(−0.8⋅ℓ+11)+0.05)P_neg( )=η· (0.95·σ(-0.8· +11)+0.05 ) (10) where the base sampling ratio η is set to 1 in experiments, 0.95 and 0.05 are used to scale the domain of the probability curve to the range [0.05, 1], ensuring that even overly long spans have a non-zero probability of being sampled. 0.8 controls the decay slope and 11 controls the shift of sigmoid, i.e., the length threshold at which the sampling probability begins to decline. These values and their approximate ranges were tested in preliminary experiments. The current values performed best, but other approximate settings also yielded comparable results owing to the robustness derived from the probabilistic nature. We apply negative sampling when computing both ℒbceL_bce and ℒkdL_kd losses, obtaining ℒbcensL_bce^ns and ℒkdnsL_kd^ns, respectively. Noted that sampling only apply to negative spans. The spans of entities and the spans within entities are forcibly kept. In KD, we treat span’s distilled labels with values below 0.5 as negative spans. Finally, the training loss is the weighted sum as: ℒ=α⋅ℒbcens+β⋅ℒkdnsL=α·L_bce^ns+β·L_kd^ns (11) where the weight is set to α=β=1α\!=\!β\!=\!1 in experiments. Figure 4: The Curve of designed negative sampling probability against the length of the negative span. 4 Experiments In this section, we first introduce the datasets and metrics, followed by the training details and the selected baselines. We then present the overall performance comparisons and conduct an ablation study. Finally, we analyze the learning efficiency and provide case visualizations. 4.1 Datasets In CLNER, evaluating models using datasets synthesized from traditional NER benchmarks become common practice [2, 11], where the original dataset is partitioned into distinct segments, each serving as an individual continual learning task. Following [11], we randomly split training data and retain annotations only for entity types designated for the current learning task. The full test data is preserved for evaluation, but only entity types learned up to the current task are visible for testing. This setup ensures the inclusion of unannotated entities from previously learned or future types, as well as purely negative sentences that contain no entities, thereby better aligning with realistic CLNER scenario. We use two source dataset: OntoNotes5.0-EN [22], annotated with 18 entity types, is converted by selecting 6 types to ensure adequate training samples per task and each task learn single entity type: Organization(ORG), Person(PER), Geo Political Entity(GPE), Date(DATE), Cardinal(CARD), Nationalities/Religious/Political Groups(NORP). FewNERD [23], hierarchically annotated with 8 coarse-grained and 66 fine-grained entity types, is converted by assigning coarse-grained types for each task, i.e., each task learns multiple (6∼ 12) fine-grained types. Coarse-grained types are Location(LOC), Person(PER), Organization(ORG), Other(OTH), Product(PROD), Building(BUID), Art(ART), Event(EVET). 4.2 Metrics For each task, model is incrementally trained on its training set and evaluated on its test set using the best checkpoint based on its development set. To factor out task order impact, we follow previous works [2, 11] to use 6 permutations for OntoNotes and 4 permutations for FewNERD. Results are averaged across all permutations and the standard deviations are typically reported at the final step, when all permutations have cumulatively learned the same complete set of entities. We report Macro-F1 across all entity types learned up to each incremental step. Macro-F1 of the final step is compared between models. Notably, FewNERD contains multiple types per task, we compute Micro-F1 for this fine-grained types within tasks due to severe imbalance, while still maintaining Macro-F1 for the final coarse-grained types. For all span-based models, we only keep the entity with the highest predicted score and discard the overlapping others when predicting overlapped spans. 4.3 Implementation Details For contextual encoder, we use PLM bert-base-cased [24] from HuggingFace followed by a BiLSTM with an output hidden size of 400 and subsequently apply 0.2 dropout rate. (i.e., dh=400d^h\!=\!400). We set do=dl=50d^o\!=\!d^l\!=\!50 on FFNs to enable simultaneous modeling of fast and slow expert as treating link computation as an additional head in span modeling. We fine-tune all parameters using AdamW optimizer [25] by setting learning rate 1e−51e^-5 for PLM bert and 1e−31e^-3 for the remaining, with schedule of warmup at first 200 steps followed by a cosine decay. Sentences are tokenized by PLM and truncated to a maximum length of 512 tokens. We aggregate tokens within each word by using mean-pooling of BERT output to represent the final word-level output. We maintained a consistent batch size of 32 and identical hyperparameters across all datasets, except for training 10 epochs on OntoNotes and 5 epochs on FewNERD. 4.4 Baselines We compare FSE with the following representative baselines: SeqFT sequentially fine-tunes the model on new data without any anti-forgetting strategy based on the conventional BERT sequence tagging, providing the lower-bound. AddNER and ExtendNER both incorporate KD under the sequence labeling framework, by using single and multiple heads for learning new tasks, respectively. L&R improves ExtendNER by reviewing and generating synthetic samples to ensure the presence of old entity types. ExtendNER+DLD equips ExtendNER with decomposed logits distillation (DLD) to strengthen discriminative ability in distillation. SpanKL adopts span-based paradigm with binary classification and Bernoulli KD improving the coherence of CLNER. SKD-NER builds upon SpanKL with using reinforcement learning to optimize KD process. For LLM, we select GPT-5 (---- gpt- 5- 2025- 08- 05) [26] and Llama3.1-8B [27] for comparison. GTP-5 is directly prompted at each step to recognize the entity types intended to learn so far, while Llama is finetuned using LoRA on the same synthesized CL datasets. We also evaluate our method under a non-CL standard training regime, which uses all available training data up to the current step while retaining all annotations of entity types learned so far, as the upper-bound performance. Table 1: Macro-F1 scores (%) of different methods at each incremental step on OntoNotes under continual learning. Method Incremental Steps Step1 Step2 Step3 Step4 Step5 Step6Step6 SeqFT [24] 82.46 56.73 39.38 29.5 28.07 23.64±4.223.64_± 4.2 AddNER [2] 82.52 83.90 84.66 85.02 85.48 85.03±.1885.03_±.18 ExtendNER [2] 82.79 83.54 84.48 84.67 85.12 84.96±.1584.96_±.15 L&R [16] 84.02 82.44 82.02 81.88 83.22 84.42±.2084.42_±.20 ExtendNER+DLD [17] 83.97 85.50 85.94 86.45 86.58 86.85±.1186.85_±.11 SpanKL [11] 85.60 87.92 88.22 88.76 89.02 88.98±.1088.98_±.10 SKD-NER [18] 85.49 87.98 88.30 88.67 89.11 88.92±.1388.92_±.13 LLMs GPT-5 [26] 61.21 69.53 73.88 69.76 65.31 67.2367.23 Llama3.1 (Finetuned) 78.30 65.15 36.62 25.17 9.10 14.74±1.114.74_± 1.1 Llama3.1 (Finetuned)m 78.30 76.67 84.93 81.73 75.25 74.58±.8974.58_±.89 FSE (Ours) 85.70 88.93 88.81 88.98 89.02 89.23±.0989.23_±.09 standard non-CL (upper-bound) 85.70 89.23 89.27 89.69 89.75 90.07±.1390.07_±.13 Table 2: Macro-F1 scores (%) of different methods at each incremental step on FewNERD under continual learning. Method Incremental Steps Step1 Step2 Step3 Step4 Step5 Step6 Step7 Step8Step8 SeqFT [24] 63.87 37.00 21.61 14.69 12.34 11.12 10.79 7.21±2.87.21_± 2.8 AddNER [2] 64.01 61.32 60.54 59.43 58.74 59.32 60.41 59.32±.1359.32_±.13 ExtendNER [2] 64.06 59.02 57.05 55.72 55.46 55.96 56.85 56.16±.0856.16_±.08 L&R [16] 64.06 59.78 58.07 55.89 55.45 56.10 57.21 57.02±.1157.02_±.11 ExtendNER+DLD [17] 64.06 60.12 57.83 56.22 56.63 57.17 58.02 58.82±.1058.82_±.10 SpanKL [11] 67.81 64.16 63.62 62.31 61.67 62.17 63.24 62.15±.0962.15_±.09 SKD-NER [18] 67.68 64.55 63.90 62.78 61.88 62.84 63.20 62.42±.1162.42_±.11 LLMs GPT-5 [26] 59.04 58.03 53.89 53.10 51.95 49.72 49.48 47.9247.92 Llama3.1 (Finetuned) 73.58 39.08 34.25 32.86 17.40 17.13 15.04 13.03±.8413.03_±.84 Llama3.1 (Finetuned)m 73.58 51.14 46.20 39.62 38.61 39.81 30.06 35.57±.7735.57_±.77 FSE (Ours) 66.61 62.43 62.39 62.64 62.27 63.78 64.20 63.69±.0863.69_±.08 standard non-CL (upper-bound) 66.61 63.53 63.77 64.56 64.89 66.65 67.11 66.83±.1166.83_±.11 Table 3: Performance of final step on two datasets evaluated by removing different components of our FSE. Method Dataset Training Throughput (steps/second) OntoNotes FewNERD FSE (Ours) 89.23±.09_±.09 63.69±.08_±.08 ≈11.70≈ 11.70 w/o shared Fast Expert 89.02±.08_±.08 63.43±.06_±.06 ≈11.19≈ 11.19 (i.e., w/o shared information) w/o Fast Expert 88.77±.14_±.14 62.95±.11_±.11 ≈12.56≈ 12.56 w/o Negative Sampling 89.00±.12_±.12 63.46±.09_±.09 ≈9.36≈ 9.36 replace Fast Expert with hard pruning (pruning span with length ¿ 10) 86.67±.14_±.14 57.46±.12_±.12 ≈12.84≈ 12.84 Figure 5: The loss curves for the initial 100 steps of each task on both datasets show the convergence speed. The red line is FSE while blue line is FSE without fast expert. A faster decrease in loss indicates a lighter learning burden and more efficient continual learning for each task. 4.5 Overall Performance We report the task-wise performance after sequentially learning each tasks on two synthesized datasets of different models. The final step’s metrics best represent the overall performance. OntoNotes’s results are shown in Tab. 1, where FSE achieves the highest score at the final step6, demonstrating the strongest recognition capability for all accumulated entity types. FewNERD’s results are shown in Tab. 2, where FSE consistently maintains SoTA performance even in this challenging scenario typically learning more than 6 entity types per incremental step. The improvement of FSE is more pronounced here, indicating the potential of shared fast expert to learn sizable entity types. LLM performance on both datasets reveals that neither GPT-5 nor Llama3.1 achieve optimal results, primarily due to inherent limitations of prompt-based method: As the number of predefined entity types filled in the instruction increases, it become challenging to accurately follow instruction. Also, LLMs may not fully comprehend the predefined entity type names, and the uncontrollable LLM-generated outputs occasionally lead to parsing failure, both causing incorrect extractions. This is evidenced by the poorer results on FewNERD which contains total 66 entity types. Notably, original result of finetuned Llama encounters train-test inconsistency since the instruction are filled with single task entity types in training but involve multiple tasks in prediction. We thereby perform multiple predictions (results with m), each focusing on a single task entities, and then aggregate them. This finally improve the performance but still remain suboptimal. Figure 6: Values probed from the span matrix view of a sampled test sentence on FewNERD, with coarse-grained types (ORG, PER), using FSE final step model. (a) Loose span scores from span modeling by slow expert, showing reduced learning burden. (b) Span-level fast scores derived from link scores by fast expert, are shared up to the last step. (Hollow boxes denote spans of length 1.) (c) Fused scores between both experts, where fast expert can refine the ’arbitrary’ results from the slow expert to accurately recognize entities. 4.6 Ablation Study We conduct ablation study to verify the effectiveness of each design in FSE. The ablated variants includes: w/o shared FE removes the sharing mechanism of the fast expert across tasks, meaning each task independently initializes a new fast expert, which can exclusively evaluate the effect of using shared information. w/o FE removes the entire fast expert module, retaining only the slow expert for span modeling. w/o NS removes the negative sampling strategy. We report the final step result as the overall performance. As shown in Tab. 3, removing different components of FSE results in a decline in overall performance. Non-shared fast expert leads to a performance drop, which indicates that the sharing mechanism of the fast expert is useful as it allows each task to leverage the previously learned fast expert, utilizing the shared information across tasks. Removing the entire fast expert module results in a more significant performance degradation, demonstrating the advantage of fast expert in reducing the learning burden to comprehensively enhance model performance. Results without negative sampling strategy also confirm its contribution to FSE. The training throughput on H800 when using different components are also reported, showing that fast expert does not significantly slow down the hardware training speed due to its lightweight architecture and the parallel implementation of fast score propagation. We also evaluate a hard pruning variant that replaces the neural-learning based link and fast score of the fast expert with a heuristically set length threshold to prune overly long spans. The threshold is set to 10 to consider avoiding unnecessary loss of long entities appearing in both corpora. The results show that this rigid strategy reduces performance, as it lacks the flexibility in handling long entities and the plasticity in continual learning. 4.7 In-Depth Analysis Learning Efficiency. FSE is expected to reduce the learning burden for each entity type by rapidly soft filtering out the majority of candidate spans using neural-based fast scores. Here, we plot the loss curves during the initial 100 training steps of each task to verify whether FSE improves the convergence rate, where a faster convergence rate implies easier learning. Fig. 5 shows the loss curves for each training tasks on OntoNotes and FewNERD, respectively. Compared to removing the fast expert, FSE accelerates learning per task, as reflected in faster loss convergence. It becomes more pronounced after the first task, indicating that the learned fast expert is successfully shared to promotes the newcome tasks. Notably, the counterpart (i.e., w/o fast expert, having non-weak performance in the ablation study) essentially represents existing competitive span-based method (e.g., SpanKL, with the same span modeling architecture). The comparison results thereby reveal a defect in current strong span-based approaches. Reducing the learning difficulty of each task enable models to better handle the incoming tasks, which explains the improvement during continual learning. Moreover, the learning efficiency can help save training time by allowing fewer training epochs in practice. Visualization. To intuitively verify the functionality of both experts, we probe the values in the span matrix for visualization. The span matrix is defined as the scores of each span arranged in a matrix perspective. Here, we consider three matrices underlying FSE: (a) the original span score matrix derived from slow experts, (b) the fast score matrix derived from the fast expert, and (c) the resulting fused score between them. Noted that we aggregate the span scores of all entity types by max-pooling for simultaneous display. Fig. 6 visualizes a sample randomly selected from test set of FewNERD. Evidently, the presence of considerable number of high scores in (a) indicate that the slow expert can afford some mistake in span modeling, as these raw scores will ultimately be correctly deactivated by the fast score of fast expert. This robust span modeling allows the slow expert to focus only on the spans remained in link-enabled area (the high scores area in (b)), thereby enhancing its learning capacity. 5 Conclusion This paper propose a fast-slow experts enhanced span-based model FSE for CLNER. The fast expert mitigates task interference by modeling shared link information between adjacent tokens to filter improbable spans across tasks, improving stability. The slow expert efficiently focuses on the remaining spans to reduce learning burden, improving plasticity. The tailored length-decay negative sampling strategy applied to both experts handles the span imbalance issue. Comprehensive experiments on synthetic datasets derived from real-world corpora OntoNotes and FewNERD confirm that FSE achieves superior performance compared to baseline including LLM-based competitor. The ablation studies validating the contribution of each component. The loss curve verifies the accelerated convergence per task, and the visualized results validate the expected functionality of both experts. Future work will explore more sophisticated link modeling mechanisms and extend this framework to broader information extraction tasks under continual learning. References Wang et al. [2024] L. Wang, X. Zhang, H. Su, J. Zhu, A comprehensive survey of continual learning: Theory, method and application, IEEE transactions on pattern analysis and machine intelligence 46 (2024) 5362–5383. Monaikul et al. [2021] N. Monaikul, G. Castellucci, S. Filice, O. Rokhlenko, Continual learning for named entity recognition, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021, p. 13570–13577. Benkő [2024] B. Benkő, Example forgetting and rehearsal in continual learning, Pattern Recognit. Lett. 179 (2024) 65–72. Zhou and Song [2024] D. Zhou, Y. Song, Pnsp: Overcoming catastrophic forgetting using primary null space projection in continual learning, Pattern Recognit. Lett. 179 (2024) 137–143. Seow et al. [2025] W. L. Seow, I. Chaturvedi, A. Hogarth, R. Mao, E. Cambria, A review of named entity recognition: from learning methods to modelling paradigms and tasks, Artif. Intell. Rev. 58 (2025) 315. Liu et al. [2025] H. Liu, X. Xin, W. Peng, J. Song, J. Sun, Concept-driven knowledge distillation and pseudo label generation for continual named entity recognition, Expert Systems with Applications 270 (2025) 126546. Dohare et al. [2024] S. Dohare, J. F. Hernandez-Garcia, Q. Lan, P. Rahman, A. R. Mahmood, R. S. Sutton, Loss of plasticity in deep continual learning, Nature 632 (2024) 768 – 774. Lample et al. [2016] G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, C. Dyer, Neural architectures for named entity recognition, in: Proceedings of NAACL-HLT, 2016, p. 260–270. Straková et al. [2019] J. Straková, M. Straka, J. Hajič, Neural architectures for nested ner through linearization, arXiv preprint arXiv:1908.06926 (2019). Yu et al. [2020] J. Yu, B. Bohnet, M. Poesio, Named entity recognition as dependency parsing, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. Zhang and Chen [2023] Y. Zhang, Q. Chen, A neural span-based continual named entity recognition model, in: Proceedings of the AAAI conference on artificial intelligence, volume 37, 2023, p. 13993–14001. Li et al. [2020] X. Li, J. Feng, Y. Meng, Q. Han, F. Wu, J. Li, A unified MRC framework for named entity recognition, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. Fei and Xu [2023] Y. Fei, X. Xu, Gfmrc: A machine reading comprehension model for named entity recognition, Pattern Recognition Letters 172 (2023) 97–105. Yan et al. [2021] H. Yan, T. Gui, J. Dai, Q. Guo, Z. Zhang, X. Qiu, A unified generative framework for various ner subtasks, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, 2021. Lu et al. [2022] Y. Lu, Q. Liu, D. Dai, X. Xiao, H. Lin, X. Han, L. Sun, H. Wu, Unified structure generation for universal information extraction, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022, p. 5755–5772. Xia et al. [2022] Y. Xia, Q. Wang, Y. Lyu, Y. Zhu, W. Wu, S. Li, D. Dai, Learn and review: Enhancing continual named entity recognition via reviewing synthetic samples, in: Findings of the association for computational linguistics: ACL 2022, 2022, p. 2291–2300. Zhang et al. [2023] D. Zhang, Y. Yu, F. Chen, X. Chen, Decomposing logits distillation for incremental named entity recognition, in: Proceedings of the 46th international ACM SIGIR conference on research and development in information retrieval, 2023, p. 1919–1923. Chen and He [2023] Y. Chen, L. He, Skd-ner: Continual named entity recognition via span-based knowledge distillation with reinforcement learning, in: Proceedings of the 2023 conference on empirical methods in natural language processing, 2023, p. 6689–6700. Yuan and Zhao [2024] B. Yuan, D. Zhao, A survey on continual semantic segmentation: Theory, challenge, method and application, IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (2024) 10891–10910. Zhang et al. [2022] C.-B. Zhang, J.-W. Xiao, X. Liu, Y.-C. Chen, M.-M. Cheng, Representation compensation networks for continual semantic segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, p. 7053–7064. Yuan et al. [2024] B. Yuan, D. Zhao, Z. Shi, Learning at a glance: Towards interpretable data-limited continual semantic segmentation via semantic-invariance modelling, IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (2024) 7909–7923. Weischedel et al. [2013] R. Weischedel, M. Palmer, M. Marcus, E. Hovy, S. Pradhan, L. Ramshaw, N. Xue, A. Taylor, J. Kaufman, M. Franchini, M. El-Bachouti, R. Belvin, A. Houston, Ontonotes Release 5.0, 2013. Ding et al. [2021] N. Ding, G. Xu, Y. Chen, X. Wang, X. Han, P. Xie, H.-T. Zheng, Z. Liu, Few-nerd: A few-shot named entity recognition dataset, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, 2021, p. 3198–3213. Devlin et al. [2019] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, in: North American Chapter of the Association for Computational Linguistics, 2019, p. 4171–4186. Loshchilov and Hutter [2017] I. Loshchilov, F. Hutter, Decoupled weight decay regularization, in: International Conference on Learning Representations, 2017. OpenAI [2025] OpenAI, Gpt-5, 2025. URL: https://openai.com/zh-Hans-CN/gpt-5/. Grattafiori et al. [2024] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al., The llama 3 herd of models, arXiv preprint arXiv:2407.21783 (2024).