Paper deep dive
Target Concept Tuning Improves Extreme Weather Forecasting
Shijie Ren, Xinyue Gu, Ziheng Peng, Haifan Zhang, Peisong Niu, Bo Wu, Xiting Wang, Liang Sun, Jirong Wen
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/23/2026, 12:02:19 PM
Summary
TaCT (Targeted Concept Tuning) is an interpretable, concept-gated fine-tuning framework designed to improve deep learning-based meteorological forecasting for extreme events like typhoons. By using Sparse Autoencoders (SAEs) and counterfactual reasoning, TaCT identifies failure-related internal concepts and applies selective parameter updates only when these concepts are activated, effectively mitigating the trade-off between extreme-event accuracy and general forecasting performance.
Entities (5)
Relation Signals (3)
TaCT → appliedto → Typhoon Forecasting
confidence 98% · We demonstrate the effectiveness of this approach on typhoon forecasting
TaCT → uses → Sparse Autoencoders
confidence 95% · TaCT leverages Sparse Autoencoders (SAEs) to promote functional disentanglement
TaCT → improves → Baguan
confidence 90% · The foundation weather forecasting model used in this work is Baguan... TaCT improves typhoon forecast performance
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Deep learning models for meteorological forecasting often fail in rare but high-impact events such as typhoons, where relevant data is scarce. Existing fine-tuning methods typically face a trade-off between overlooking these extreme events and overfitting them at the expense of overall performance. We propose TaCT, an interpretable concept-gated fine-tuning framework that solves the aforementioned issue by selective model improvement: models are adapted specifically for failure cases while preserving performance in common scenarios. To this end, TaCT automatically discovers failure-related internal concepts using Sparse Autoencoders and counterfactual analysis, and updates parameters only when the corresponding concepts are activated, rather than applying uniform adaptation. Experiments show consistent improvements in typhoon forecasting across different regions without degrading other meteorological variables. The identified concepts correspond to physically meaningful circulation patterns, revealing model biases and supporting trustworthy adaptation in scientific forecasting tasks. The code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2603.19325v1
- Canonical: https://arxiv.org/abs/2603.19325v1
Trouble viewing inline? Open PDF directly →
Full Text
61,418 characters extracted from source content.
Expand or collapse full text
Target Concept Tuning Improves Extreme Weather Forecasting Shijie Ren ∗ Renmin University of China Beijing, China shj_ren@ruc.edu.cn Xinyue Gu ∗ Alibaba Group Hangzhou, China guxinyue.gxy@alibaba-inc.com Ziheng Peng ∗ Renmin University of China Beijing, China ziheng.peng@ruc.edu.cn Haifan Zhang Alibaba Group Hangzhou, China zhanghaifan.zhf@alibaba-inc.com Peisong Niu Alibaba Group Hangzhou, China niupeisong.nps@alibaba-inc.com Bo Wu Institute of Atmospheric Physics, Chinese Academy of Sciences Beijing, China wubo@mail.iap.ac.cn Xiting Wang † Renmin University of China Beijing, China xitingwang@ruc.edu.cn Liang Sun † Alibaba Group Hangzhou, China liang.sun@alibaba-inc.com Jirong Wen Renmin University of China Beijing, China jrwen@ruc.edu.cn Abstract Deep learning models for meteorological forecasting often fail in rare but high-impact events such as typhoons, where relevant data is scarce. Existing fine-tuning methods typically face a trade-off between overlooking these extreme events and overfitting them at the expense of overall performance. We propose TaCT, an in- terpretable concept-gated fine-tuning framework that solves the aforementioned issue by selective model improvement: models are adapted specifically for failure cases while preserving performance in common scenarios. To this end, TaCT automatically discovers failure-related internal concepts using Sparse Autoencoders and counterfactual analysis, and updates parameters only when the cor- responding concepts are activated, rather than applying uniform adaptation. Experiments show consistent improvements in typhoon forecasting across different regions without degrading other meteo- rological variables. The identified concepts correspond to physically meaningful circulation patterns, revealing model biases and sup- porting trustworthy adaptation in scientific forecasting tasks. The code is available at https://anonymous.4open.science/r/Concept- Gated-Fine-tune-62AC. CCS Concepts • Computing methodologies→Knowledge representation and reasoning. ∗ Both authors contributed equally to this research. † Corresponding author Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Conference acronym ’X, Woodstock, NY © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-X-X/2018/06 https://doi.org/X.X Keywords Concept-Gated Fine-Tuning, AI Weather Forecasting, Interpretabil- ity, Tropical Cyclone Forecasting, Sparse Autoencoders ACM Reference Format: Shijie Ren, Xinyue Gu, Ziheng Peng, Haifan Zhang, Peisong Niu, Bo Wu, Xiting Wang, Liang Sun, and Jirong Wen. 2018. Target Concept Tuning Improves Extreme Weather Forecasting. In Proceedings of Make sure to enter the correct conference title from your rights confirmation email (Con- ference acronym ’X). ACM, New York, NY, USA, 12 pages. https://doi.org/ X.X 1 Introduction Weather forecasting is critical to socio-economic stability. Nearly $3 trillion in annual private industry revenues, one-third of the sector’s total, is exposed to weather and climate risk [8]. In energy management, weather forecasts are vital for predicting load demand and renewable power supply, thereby preventing costly blackouts and curtailment [15]. Recently, AI-based forecasting models [2,21] have emerged as powerful alternatives to traditional Numerical Weather Prediction (NWP) systems, achieving high accuracy on standard variables such as temperature and wind with much faster inference [32,35]. However, their performance drops during extreme events such as typhoons (also known as Tropical Cyclones, TC), heatwaves and cold surges. Although rare, these events can have severe conse- quences, including loss of life, grid collapse, and infrastructure damage. These high-stakes scenarios require reliable forecasts for early warning and emergency response [18], and reliable deploy- ment requires not only accuracy but also trustworthy models that can diagnose and interpret model behavior when failures occur. Achieving such reliability, however, is challenging for extreme weather forecasting due to the severe data imbalance that far ex- ceeds typical imbalanced learning settings. For example, the proba- bility of a typhoon forming within any 5 ◦ × 5 ◦ region over 24 hours is below 0.039% [34]. Current learning paradigms fail under extreme imbalance because they are inherently a trade-off: both full-model arXiv:2603.19325v1 [cs.LG] 17 Mar 2026 Conference acronym ’X, June 03–05, 2018, Woodstock, NYReh et al. fine-tuning and parameter-efficient methods [17,24,25,41] mit- igate data imbalance by trading off performance across different distributions, rather than correcting model errors in both common and rare scenarios. This trade-off becomes particularly severe in extreme imbalance settings, where hyperparameter tuning becomes highly sensitive [39] and resampling or reweighting becomes in- effective [36]. Furthermore, the lack of interpretability in these methods limits control over when updates take effect, hinders un- derstanding of why failures occur, and undermines trust—both essential in high-stakes deployments. We aim to avoid the trade-off between accuracy on rare extreme and common events, while simultaneously providing interpretabil- ity during fine-tuning. Our approach is inspired by cognitive neu- roscience, which shows that individuals with more modular brain organization exhibit greater neural plasticity when learning new tasks [10], suggesting localized task-specific updates minimize inter- ference with previously acquired functions. In contrast, neurons in deep learning models often encode superposed features — multiple concepts mixed within a single activation [9]— and it is difficult to update specific knowledge without overwriting existing represen- tations [14]. Motivated by this observation, we propose to fine-tune models within a disentangled representation space, where activa- tions correspond to mono-semantic and interpretable concepts. By targeting only the concepts responsible for failures, our method en- ables precise, interpretable fine-tuning for extreme weather events without compromising the general ability of the models. Specifically, we introduce TaCT (Targeted Concept Tuning), an interpretable, concept-guided fine-tuning framework for failure- aware model adaptation. TaCT leverages Sparse Autoencoders (SAEs) [40] to promote functional disentanglement, decomposing superposed representations into quasi-modular, mono-semantic units [40]. These units naturally correspond to coherent meteoro- logical structures, such as typhoon vortices or high-pressure ridges. Using a small set of extreme-event cases (438 out of 1460 samples), TaCT automatically identifies concepts consistently activated dur- ing model failures via continuous counterfactual reasoning. Fine- tuning is then performed in a concept-gated manner, updating newly added model parameters only when failure-associated con- cepts are active, keeping other predictions unaffected. This mech- anism yields concept-level plasticity: similar to that observed in modular human brains, the model learns precisely where and when it fails, without interfering with other functional modules and there- fore preserving its general predictive ability. We demonstrate the effectiveness of this approach on typhoon forecasting across multiple cyclone basins (Northern Atlantic, West- ern Pacific and Eastern Pacific). The resulting models achieve promi- nent improvements across key typhoon characteristics including sea-level pressure (9.3% MAE reduction in 72h forecast) and near- surface winds (4.8% MAE reduction in 72h forecast), while main- taining the forecast accuracy on other variables (error change: ours -2 vs LoRA +4 in z850 and 0 vs +0.1 in T850). Beyond performance gains, the concepts identified by TaCT correspond to physically meaningful atmospheric structures, such as mid-latitude transient waves, which meteorologists recognize as key drivers of typhoon behavior. TaCT not only improves typhoon forecast performance but also reveals concepts that have been improved during fine- tuning, fostering trust and facilitating integration into operational weather prediction workflows. Our contributions are as follows: • We propose TaCT, an interpretable fine-tuning framework that leverages brain-inspired modular learning to disentangle super- posed representations into physically grounded concepts, and uses them to guide targeted adaptation. It serves as a generic add-on for intermediate layers and is also generalizable for other deep learning domains. • We introduce a counterfactual concept localization method, which integrates SAE with continuous counterfactual reasoning to au- tomatically identify concepts to be improved in extreme weather conditions. This module requires only a small amount of extreme weather data points and can operate fully automatically without manual intervention. •We design a concept-gated fine-tuning algorithm that conditions parameter updates on the activation of failure-related concepts. This module precisely corrects extreme-event failures while pre- serving performance in common weather regimes, avoiding the trade-off between rare-event performance and overall accuracy. 2 Problem Formulation We consider the standard AI weather forecasting task [4]: given the current state of the weather (e.g., temperature, humidity, wind speed), predict its future state at a specified lead time. The input - global weather data - is represented as a 3D tensor discretized over a latitude-longitude grid. To reduce memory consumption, the input is typically partitioned into patches, each represented as a single unit for model input. Formally, let푋 푡 ∈ R 퐻×푊×퐶 denote the weather conditions at time푡, where퐶is the number of weather variables, and퐻and푊denote the number of patches along latitude and longitude, respectively. Our goal is to predict푋 푡+Δ푡 ∈ R 퐻×푊×퐶 at lead time푡+Δ푡using an AI weather modelFwith parametersΘ: ˆ 푋 푡+Δ푡 =F(푋 푡 ;Θ)(1) For long-range weather forecasting, predictions are made au- toregressively. Specifically, to predict weather conditions at time 푡+푎Δ푡, we iteratively apply the model with lead timeΔ푡for푎times, feeding each prediction back as input for the next step: ˆ 푋 푡+푎Δ푡 =F (푎) (푋 푡 ;Θ)=F(·F(F | z 푎 times (푋 푡 ;Θ);Θ)· ;Θ)(2) whereF (푎) denotes 푎 iterative applications ofF . 3 Methods This section introduces the Targeted Concept Tuning (TaCT) frame- work, which enables interpretable and precise fine-tuning for ex- treme weather events without compromising the general abilities. Analogous to modular human brains [10], TaCT fine-tunes in a dis- entangled representation space, where each activation corresponds to a mono-semantic, quasi-modular concept, enabling targeted adap- tation without overwriting unrelated knowledge. Specifically, our framework consists of two major modules as shown in Figure 1: (1) counterfactual concept localization, Target Concept Tuning Improves Extreme Weather ForecastingConference acronym ’X, June 03–05, 2018, Woodstock, NY Input Hidden Representations ℎ Module1: Counterfactual Concept Localization ∆푧 Unsupervised Concept Decomposition W !"# b !"# ∆ℎ 휕퐿 $%&'()* 휕∆푧 Continuous Counterfactual Reasoning . . . . . . . . . . . . ... Relevant Concepts Module2: Concept-Gated Fine-tuning W "+# b "+# ℎ . . . Threshold 훽 Concept-level Gating Only relevant concepts ActivateFine-tuning Output Adapter FC ReLU FC LoRA 푊 !" 푊 #$%& ℳ Aggregated Optimize ∆푧: ERA5 Dataset (e.g. msl, z, u, v, q, h) SmallSetof Cases ℳ (e.g. Typhoon) Train Frozen Pre-trainedModel Improved Weather Forecasting Model Better Performance on Extreme Events Preserve General Forecasting Ability Module1: Counterfactual Concept Localization Unsupervised Concept Decomposition Continuous Counterfactual Reasoning Module2: Concept-Gated Fine-tuning ActivateFine-tuning Concept-level Gating Figure 1: The overall framework of TaCT, comprising two main modules: (1) counterfactual concept localization, which decomposes input hidden representations and identifies key concepts through continuous counterfactual reasoning; (2) concept- gated fine-tuning, which selectively fine-tunes activated concepts without affecting others. which disentangles the model’s internal representations into mono- semantic concepts without any supervision signals about which concepts exist for the given model, and then identifies key con- cepts relevant to specific weather scenarios (e.g., extreme events or regional biases) via counterfactual analysis; and (2) concept- gated fine-tuning, which performs selective updates to model parameters based on the localized target concepts. Next, we will introduce the two modules and discuss the theo- retical relation between TaCT and existing fine-tuning methods. 3.1 Counterfactual Concept Localization This module disentangles the model’s hidden representations into interpretable concepts and identify those most responsible for pre- diction failures under the target scenario (i.e., extreme weather sce- narios). Given a small set of extreme weather samples(푋 푡 ,푋 푡+Δ푡 ), we obtain a concept set퐶that guides the subsequent fine-tuning through two stages: unsupervised concept decomposition and con- tinuous counterfactual reasoning. 3.1.1 Unsupervised Concept Decomposition. Given hidden embed- ding퐻 푙 = F 1:푙 (푋 푡 ) ∈ R 퐻×푊×푑 from layer푙, the goal of unsuper- vised concept decomposition is to decompose퐻 푙 = F 1:푙 (푋 푡 )into sparse concepts푍 ∈ R 퐻×푊×푛 (푛> 푑) without additional supervi- sion signal about which concepts exist in the weather forecasting models. We achieve this goal by employing Sparse Autoencoders (SAEs) [3,11], which constructs a disentangled concept space푍by learning sparse representations with an auto-encoder while accu- rately reconstructing the original input. Each concept푧 (푖,푗) in푍 then corresponds to a mono-semantic feature relevant to meteoro- logical structures (e.g., typhoon vortices or pressure ridges). Mathematically, given a sample푋 (푖,푗) at geographic coordinate (푖, 푗), the encoder projects the푙-th layer hidden representation ℎ (푖,푗) 푙 ∈ R 푑 into a high-dimensional sparse concepts space푧 푖,푗 ∈ R 푛 (푛> 푑 ), and the decoder reconstructs the original embedding: 푧 (푖,푗) = Enc(ℎ (푖,푗) 푙 )= Topk(ReLU(푊 푒푛푐 (ℎ (푖,푗) 푙 −푏 푑푒푐 )+푏 푒푛푐 ,푘)), (3) ˆ ℎ (푖,푗) 푙 = Dec(푧 (푖,푗) )=푊 푑푒푐 푧 (푖,푗) +푏 푑푒푐 ,(4) where푊 푒푛푐 ∈ R 푛×푑 ,푊 푑푒푐 ∈ R 푑×푛 ,푏 푒푛푐 ∈ R 푛 ,푏 푑푒푐 ∈ R 푑 are parame- ters to be learned, and TopK(·)retains only the푘largest activations and setting other elements to 0, ensuring sparse activation. The encoder-decoder is then learned to reconstruct the hid- den representations accurately, with an auxiliary loss function 퐴푢푥(·)[11] to reactivate dead concepts whose activations푧 (푖,푗) are consistently low for almost all samples: L 푆퐴퐸 =| ˆ ℎ (푖,푗) 푙 −ℎ (푖,푗) 푙 | 2 2 + 휆퐴푢푥(푧 (푖,푗) ).(5) Appendix A introduces 퐴푢푥(·) and SAE training in detail. 3.1.2 Continuous Counterfactual Reasoning. Given푛discovered concepts, where푛> 푑is usually large, we then discover a sub- set퐶of concepts that is most responsible for prediction failures under extreme weather scenarios. This requires quantifying each concept’s contribution to the forecasting loss. Our idea is based on counterfactual reasoning [1,30,33], which estimates concept importance by measuring how minimal interventions on individual concepts affect losses in a small set of extreme weather samples: min 푧 ′ dist(푧,푧 ′ )s.t. 퐹 푙+1:퐿 (Dec(푧 ′ ))=푦 푡푎푟푔푒푡 .(6) Here, we omit the position index(푖, 푗)of푧for conciseness, and the minimal intervention magnitude|푧−푧 ′ |represents the importance of the corresponding concepts in decreasing loss. Unlike prior counterfactual methods shown in the last equation, which are designed for discrete classification, weather forecasting is a continuous regression problem. We therefore change the hard Conference acronym ’X, June 03–05, 2018, Woodstock, NYReh et al. constraint퐹 푙+1:퐿 (Dec(푧 ′ ))=푦 푡푎푟푔푒푡 into a soft reconstruction objec- tive and optimize perturbations directly in concept space, enabling continuous counterfactual reasoning: min Δ 푧 L=|퐹 푙+1:퐿 (Dec(푧 ′ ))−푦 푡푎푟푔푒푡 | 2 2 + 휆|푧 ′ −푧|,(7) 푦 푡푎푟푔푒푡 = 푋 (푖,푗) 푡+Δ 푡 , 푧 ′ = 푧+ I(푧> 0)Δ푧,(8) Δ푧= 푧 ′ −푧.(9) Here, the퐿 1 penalty encourages sparse perturbations,I(·)is an indicator function whereI(true)=1 andI(false)=0.푧 ′ = 푧 + I(푧> 0)Δ푧ensures that a counterfactual intervention will be conducted only for relevant samples. Intuitively, each entry in Δ푧= 푧 ′ −푧measures how much a specific concept must change to reduce prediction error: larger magnitudes indicate concepts whose adjustment most improves the forecast. Finally, the top-k concepts with the highest average magnitudes across all푚extreme weather samples are then selected as the target concepts for subsequent modification. This selection process is formally defined in Eq. (10): C= Topk 푖푑 ( 1 푚 푚 ∑︁ 푡=1 |Δ푧 푡 |,푘).(10) where the resulting setCcontains the key concepts to be fine-tuned for the target scenario. These concepts exhibit the most significant influence on the model’s performance for the target task, andTopk 푖푑 is used to find the indices of the 푘 largest elements. 3.2 Concept-Gated Fine-tuning To enhance specific model capabilities by modifying the concepts identified through counterfactual reasoning—while simultaneously preserving its general performance—we introduce a concept-based gating mechanism. Specifically, this mechanism leverages the acti- vation states of concepts to precisely control whether the current data point should be used to optimize a particular concept. Such a structured fine-tuning strategy ensures that biases are corrected with precision, thereby minimizing any degradation of the model’s performance on other data. 3.2.1 Concept-Gated Fine-tuning. Previous work has demonstrated that certain knowledge can be modified or forgotten by manipulat- ing specific neurons [5]. Inspired by this, we introduce a concept activation-gated mechanism to precisely modify performance un- der specific concepts without affecting the model’s performance in other scenarios, such as extreme low pressure in typhoon scenarios. Specifically, for each concept푐 푖 in the concept setCidentified in Section 3.1.2, we define an threshold훽 푖 . This threshold ensures that the fine-tuned parameters are only activated when the correspond- ing concept is triggered. Formally, for the푙-th layer targeted for intervention, let the intermediate representation be h 푙 . The concept activation vector obtained through SAE encoding is denoted as z= Enc(h 푙 ). We modify the model’s forward pass by introducing a gated residual term. This controlled modification term,Δh 푙 , is computed as shown in Eq.(11), whereI(·)is the indicator func- tion shown in Eq.(12), which determines whether the activation of the푖-th concept exceeds the threshold훽 푖 . Here,푓 ′ represents an existing fine-tuning method, such as LoRA or Adapter. Δh 푙 = I ( ∨ 푖∈퐶 (푧 푖 > 훽 푖 ) ) · 푓 ′ (h 푙 ).(11) I(∨ 푖∈퐶 (푧 푖 > 훽 푖 ))= ( 1,if Ture 0,otherwise .(12) When the input does not involve the target concepts, the term Δh 푙 vanishes as the indicator function evaluates to zero. Finally, we inject the generated gated residual into the original model to obtain the modified final prediction, formalized as Eq.(13), where 퐹 푙+1:퐿 represents the subsequent processing of the model from the intervention layer to the output layer, ˆ 푋 ′ 푡+Δ푡 represents the model output after intervention. ˆ 푋 ′ 푡+Δ푡 = 퐹 푙+1:퐿 (h 푙 +Δh 푙 ).(13) 3.2.2 Loss Function. The loss function follows previous work [4], employing a latitude-weighted loss, formulated as Eq.(14), where 퐻, and푊denote the number of grid points in the latitude and longitude directions, respectively. ˆ 푋 ′ and푋are the predicted and ground-truth values for a specific variable and location (latitude and longitude) at time step푡 +Δ푡.푤 푖 represents the weight at latitude 푖, which decreases as the latitude increases. The loss is averaged across all grid points and variables. L= 1 퐻푊 퐻 ∑︁ 푖=1 푊 ∑︁ 푗=1 푤 푖 ·| ˆ 푋 ′ 푡+Δ푡 − 푋 푡+Δ푡 |.(14) 3.3 Method Comparison From the perspective of concept intervention, different model edit- ing methods essentially adopt different strategies for intervening on concepts. Our method proposes a more general framework for concept-gated intervention. “Black-box” methods, such as PEFT, employ a global and indis- criminate intervention strategy. They do not differentiate based on concept importance, applying adjustments uniformly across all internal states. This can be viewed as a special case with the most lenient intervention condition: the gating unit is always open, intervening on all concepts퐶 all , as formalized in Eq. (15). I LoRA = I(∨푖 ∈ 퐶 all (푧 푖 > 0))= 1 .(15) Representation engineering is another approach where interven- tion concepts are pre-defined. The intervention target퐶 pre , relies entirely on human prior knowledge and manual specification. An intervention is executed only when a pre-defined concept is acti- vated, as formalized in Eq. (16). I RepE (∨푖 ∈ 퐶 pre (푧 푖 > 훽 pre 푖 ))= ( 1,if True 0,otherwise .(16) Unlike PEFT’s indiscriminate updates and ReFT’s reliance on pre-defined concepts. Our methods learns to automatically discover when and on which key concepts to intervene, thereby enabling automated form of concept intervention. Target Concept Tuning Improves Extreme Weather ForecastingConference acronym ’X, June 03–05, 2018, Woodstock, NY 4 Experiments 4.1 Experimental Settings Datasets. We utilize the ERA5 dataset [13] for model training and test as both inputs and ground truths. Produced by the European Centre for Medium-Range Weather Forecasts (ECMWF), ERA5 is a comprehensive global atmospheric reanalysis dataset providing detailed climate and weather information from 1940 to the present. It encompasses a wide array of variables, including temperature, humidity, precipitation, and mean sea level pressure. ERA5 offers a high spatial resolution of 0.25° latitude-longitude and includes 37 vertical pressure levels. We adopt the International Best Track Archive for Climate Stewardship (IBTrACS) [20] and CMA Best Track Dataset [26,38] as our primary data source. IBTrACS is a global typhoon dataset maintained by the U.S. National Oceanic and Atmospheric Administration (NOAA) and is widely considered the gold standard for research in this field. The CMA Best Track Dataset is a typhoon best track dataset released and maintained by the Shanghai Typhoon Institute (STI) under the China Meteorological Administration (CMA). It is one of the most authoritative historical records of typhoon over the Western North Pacific and the South China Sea. Foundation Model. The foundation weather forecasting model used in this work is Baguan [29], a large-scale pre-trained weather model developed by Alibaba. Baguan is a deep learning model designed for global medium-range weather forecasting, trained end-to-end on the ERA5 dataset with robust spatiotemporal mod- eling capabilities. It operates at a spatial resolution of 0.25°× 0.25° and covers major meteorological factors such as geopotential, wind speed, temperature, humidity, and precipitation, with a 6-hour time step. Benefiting from extensive pre-training on large-scale weather data and the implicit learning of atmospheric physics, Baguan has demonstrated predictive accuracy comparable to traditional Nu- merical Weather Prediction (NWP) systems. Baseline and Metric. To provide a comprehensive evaluation, we compare with four baselines: the Baguan-Weather forecast- ing model, two parameter-efficient fine-tuning (PEFT) methods: LoRA and Adapter and a representation-based fine-tuning method: LoREFT. The implementation details and hardware configuration can be found in the Appendix C. Our evaluation metrics are the min- imum mean sea-level pressure (MSL) and the maximum 10-meter wind speed (V10) of tropical cyclones. Implementation Details. To evaluate the model’s performance, typhoon data from 2022 is used exclusively as an external test set to ensure complete isolation from the training process and prevent potential data leakage. The training phase utilizes unlabeled ERA5 data spanning from 1979 to 2021. Due to limited computational resources, we restricted the optimization process by uniformly fixing the number of training steps at 6,000. This approach ensures model convergence while maximizing overall training efficiency. 4.2 Overall Performance Results. We evaluated the 72-hour forecast accuracy of typhoon intensity, including MSL and V10, across multiple regions. The results for the Western Pacific are illustrated in Figure 2, while the remaining results are presented in Appendix F. TaCT demonstrated comparable performance for both typhoon minimum pressure and Figure 2: 72-Hours Performance. WP denotes the Western Pacific. “Typhoon pressure” refers to the minimum sea-level pressure in Pa, and “Typhoon wind speed” represents the maximum wind speed in m/s. wind speed forecasts. The performance of LoREFT is highly similar to that of the Base Model. This is because LoREFT only modifies a subset of tokens, which may be located far from the typhoon center. As AI weather models typically focus on data within a local neighborhood [29], the impact of these distant modifications is minimal. In contrast, Lora and Adapter achieved sub-optimal results due to a lack of fine-grained conceptual guidance. To compare with models fine-tuned on tropical cyclone data, we evaluate our method against several fine-tuning approaches trained on the same tropical cyclone dataset. The experimental results are presented in Appendix G. Figure 3: Sensitivity Analysis. (a, b) and (c,d) present the 6- hour forecast results of minimum sea level pressure (MSL) and maximum wind speed of typhoons across different re- gions under varying parameter sizes. (e) presents the 6-hour MSL forecast results under different model activation thresh- old settings. (f) presents the MSL forecast results under dif- ferent concept number settings. Conference acronym ’X, June 03–05, 2018, Woodstock, NYReh et al. Table 1: Ablation study: MAE of MSL under different methods. M1 denotes counterfactual concept localization, and “ran- dom” and “statistic” refer to two different concept localiza- tion strategies. Regionw/o M1 Random Statistics Ours West Pacific80.2182.3078.0976.88 East Pacific67.3467.9764.8063.53 North Atlantic126.50131.48121.36119.21 Avg91.3593.9288.0886.54 Δ (%)5.277.851.75- 4.3 Ablation Study and Sensitivity Analysis Ablation Study. We conduct an ablation study to evaluate the contribution of each component in our framework. To assess the effect of the concept-gated module, we remove the concept-gated unit, which degrades the method to a simple Adapter. To evalu- ate the continuous counterfactual reasoning module, we consider two ablated variants: one with randomly sampled concepts and another using a statistical method for concept identification. The specific implementation details are provided in Appendix B. The results in Table 1 demonstrate that the interpretability module plays the primary role in enhancing model performance. Replacing our components with these alternatives leads to a consistent drop in performance, thereby validating the effectiveness of our proposed method. Threshold of Concepts. To evaluate the impact of the number of gated concepts on model performance. We choose three acti- vation thresholds that filter out 90%, 99%, and 99.9% of the data, respectively, as shown in Figure 3 (e). The results clearly indicate that the number of concepts is correlated with model effectiveness. Within a certain range, increasing the number of concepts leads to a lower MAE, as the model tends to learn better representations when more concepts are adjusted. However, once the number of concepts reaches a certain threshold, model performance begins to decline. This is because too many concepts can introduce more noise, degrading the quality of the learned concepts. Number of Concepts. We evaluate the effect of the number of gated concepts on model performance, as shown in Figure 3 (f ). The results clearly indicate that the number of concepts is correlated with model effectiveness. Within a certain range, increasing the number of concepts leads to a lower MAE, as the model tends to learn better representations with more concepts being adjusted. However, once the number of concepts reaches a certain threshold, model performance begins to decline. This is because a large number of concepts can introduce more noise, leading to a degradation in the quality of the learned concepts. Randomized Scenarios. To minimize randomness as much as possible, we evaluated our method with three different random seeds and under varying fine-tuning parameter budgets, as shown in Figure 3 (a-d) and Table 2. The results demonstrate that our method consistently outperforms the baseline across different settings. Table 2: MAE of MSL under different random seeds. RegionSeed=1979 Seed=2025 Seed=2026 West Pacific76.8877.9275.84 East Pacific63.5364.0661.97 North Atlantic119.21121.55116.56 4.4 General Capabilities To verify the impact of fine-tuning on the model’s general capa- bilities, we conducted a comprehensive evaluation of the model’s performance on global weather forecasting tasks. We validate the model’s 72-hour forecast performance across the entire globe. The performance changes relative to the base model are illustrated in Figure 4, where values closer to zero indicate better preservation of general capabilities. The experimental results demonstrate that our method not only achieves significant performance improvements in typhoon-specific scenarios but also maintains the strongest general forecasting capabilities among all compared fine-tuning approaches. Notably, our approach exhibits minimal degradation across all evalu- ated variables, with performance changes consistently smaller than those of other methods, confirming the effectiveness of our coun- terfactual concept localization strategy in balancing task-specific optimization with the retention of foundational model knowledge. 4.5 Case Study To interpret the physical meaning of these concepts, we adopt an automated approach based on Multimodal Large Language Mod- els (MLLMs). In brief, the MLLM-based method feeds a concept’s activation locations together with its corresponding input–output information into an MLLM, leveraging its strong pre-trained ca- pabilities to summarize the concept’s function. More details are provided in Appendix D. 4.5.1 Concepts-general Analysis. To gain a deeper understanding of the physical concepts learned by the model, we first analyze the list of concepts presented in Figure 5. This table enumerates a series of concepts, including their names and detailed descriptions as generated by a MLLM. The Improved Acc column indicates the proportion in Mean Absolute Error (MAE) after fine-tuning when a concept is activated; a higher value signifies a more substantial improvement in model performance. The Activate Ratio column quantifies the number of typhoon cases in which the concept was activated. From the data, several key observations can be made. The first three concepts, namely #11736, #10970, and #14233, which were identified by our counterfactual concept localization method, are activated in a large number of typhoon cases. They also significantly improve the model’s predictive accuracy. This demonstrates that the model successfully captures critical physical patterns closely associated with typhoon events. In contrast, the remaining three concepts, namely #14889, #3873, and #7476, were not identified by our counterfactual concept lo- calization method. Concept #14889 is rarely activated in typhoon cases (only once), suggesting its correlation with the typhoon phe- nomenon is weak. It is also noteworthy that concepts #3873 and Target Concept Tuning Improves Extreme Weather ForecastingConference acronym ’X, June 03–05, 2018, Woodstock, NY Figure 4: The impact of different fine-tuning methods on general capabilities. Where 0 represents the performance of the base model, and the values indicate the change of different methods relative to the original model, with lower values being better. #7476, although not directly identified by counterfactual concept lo- calization, are also relevant to typhoon phenomena. Consequently, they are activated in many cases and contribute positively to the model’s performance enhancement. These findings validate the effectiveness of our proposed method in automatically identifying and utilizing physical concepts related to specific extreme weather events like typhoons, thereby laying the groundwork for the subsequent case study analysis. Concept #11736 Description Improved Acc% #10970 #14233 TransientWaves Large-scale planetary waves in the atmosphere and oceans. prominentfeatures:z500, v500 Polar Vortex Edge Filamentation Fine-scale vorticity filamentation at the polar vortex edge caused by wave–mean flow interaction.prominentfeatures:z50 Tropical Cyclone Barotropic Core Barotropically aligned low-to mid-level circulation typical of tropical cyclones. prominentfeatures:msl, z850, z500, z1000 9.314 6.459 11.448 #14889 Double-jet Waveguide Pattern Two zonally-separated westerly jets acting as dual waveguides for wave activity. prominentfeatures:z200, u200 #3873 #7476 The Sun's electromagnetic energy, which is the primary driver of Earth's weather and climate. prominentfeatures:avg sdswrf Ocean–Land Teleconnection Pattern Large-scale ocean–land pressure contrast with ENSO-like teleconnection structure. prominentfeatures:msl, z1000, z500 0.003 6.964 8.750 Activated Ratio % 52.85 62.28 50.44 0.22 40.13 56.36 SolarRadiation FinetuneConceptNon-FinetuneConcept Figure 5: Six concepts learned by SAE. 4.5.2 Concept-specific Cases. We further illustrate a critical con- cept 11736 activations whose semantic meanings are interpreted by an MLLM. Transient Waves. Figure 6 and Figure 7 shows the activation map of concept 11736 over the 500 hPa meridional wind field (푣 500 , 푧 500 ). Black-shaded regions indicate areas of strong activation. Based on LLM interpretation of the most activated samples, this concept corresponds to atmospheric transient waves: A series of troughs and ridges on quasi-horizontal surfaces in the major belt of upper tropospheric westerlies. which are closely coupled with the mid- latitude upper-level jet stream. These oscillations are not stationary; rather, they continuously evolve and migrate under the influence of the prevailing westerlies. According to the wave characteristic analysis, the extracted transient wave train exhibits distinct zonal periodicity, with a zonal wave number stable between 9 and 10 8. These waves propagate in a meandering fashion within the mid- latitude jet stream, constituting the complex and variable synoptic background for mid-to-high latitude regions. Figure 6: V-component of wind on 500hPa, where red in- dicates positive values, blue indicates negative values, and black mask indicates concept activation locations. Figure 7: Geopotential on 500hPa, where red indicates posi- tive values, blue indicates negative values, and black mask indicates concept activation locations. Identification of such wave features is essential for accurate fore- cast for intensity of the tropical cyclone. These transient waves function as a critical determinant in tropical cyclone predictions by perturbing the mid-latitude steering currents. Moreover, given that the trajectory and intensity of tropical cyclones are profoundly sensitive to the surrounding atmospheric state, the dynamical in- teraction between mid-latitude wave systems and tropical cyclones Conference acronym ’X, June 03–05, 2018, Woodstock, NYReh et al. 20°E40°E60°E80°E100°E 2018-09-21 00 2018-09-21 06 2018-09-21 12 2018-09-21 18 2018-09-22 00 2018-09-22 06 2018-09-22 12 2018-09-22 18 2018-09-23 00 2018-09-23 06 2018-09-23 12 ERA5 V500 20°E40°E60°E80°E100°E ERA5 Z500 20°E40°E60°E80°E100°E Mask (Concept) 50 40 30 20 10 0 10 20 30 Wind spped (m/s) 47000 48000 49000 50000 51000 52000 53000 54000 Geopotential Height (gpm) 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00 Concept Value Figure 8: Co-movement of wind speed maxima, troughs, and the conceptual height center. governs their lifetime. Accurately recording the phase speed and amplitude of these transient waves in forecasting models is neces- sary to reduce errors in long-term intensity predictions. 5 Related Work Efficient Fine-Tuning. As the scale of pre-trained models con- tinues to expand, full fine-tuning imposes prohibitive memory and computational costs. This has motivated the development of Parameter-Efficient Fine-Tuning (PEFT) [7], which aims to adapt models using a minimal set of trainable parameters while maintain- ing competitive performance. Early representative methods, such as Adapters [16,31], insert small feed-forward modules between Trans- former layers while freezing the original weights. Although they introduce only a few million parameters, they require structural modifications to the architecture. Alternatively, Prefix-Tuning [23] and Prompt Tuning [22] guide model behavior by prepending learn- able continuous vectors to input embeddings or hidden states with- out altering the backbone network. Recently, Low-Rank Adaptation (LoRA) and its variants [17,24] have gained widespread adoption due to their simplicity and efficiency. LoRA approximates weight updates through low-rank matrix decomposition: it injects trainable low-rank matrices into attention layers while keeping the original weights frozen. In contrast to these methods, our approach can automatically select concepts for fine-tuning, enabling targeted model adjustments without compromising general capabilities. Model Interpretability and Intervention. As the scale and diver- sity of data increase, modifying model behavior through retraining becomes intractable. Consequently, a large of works has shifted its focus to model intervention [12,27,28], which refers to the ability to understand, analyze, and modify the internal knowledge and behavior of a model after it has been trained. Model Interven- tion contains model editing [5,6] and machine unlearning [37]. Model editing achieve precise modifications of model behavior by intervening in the model’s internal representations or parameters. Machine unlearning aims to remove the influence of specific data from a trained model without the prohibitive cost of retraining from scratch. In contrast, model editing focuses on directly modifying or injecting new knowledge. In the realm of large language mod- els, knowledge editing techniques like ROME [27] have achieved remarkable success, as they can locate and alter the weights as- sociated with specific facts (e.g., "The Eiffel Tower is in Paris"). Furthermore, some studies have attempted to edit more abstract "concepts," for instance, by using linear transformations to suppress or enhance a model’s expression of a certain concept [19]. 6 Limitations and Ethical Considerations Limitations. Compared to direct fine-tuning, our method requires pre-training an SAE, which introduces a certain training overhead. However, this is a one-time cost, and we argue it is acceptable given the SAE’s demonstrated potential to enhance both performance and model interpretability. Additionally, since our method is concept- based, its fine-tuning effectiveness is correlated with concept quality. Fortunately, a large of existing work is dedicated to improving the interpretability of SAEs. Therefore, our method stands to achieve further performance gains by incorporating more advanced SAE techniques in the future. Ethical Considerations. Our study uses meteorological reanal- ysis/products (e.g., ERA5) and public cyclone tracks. All data are used in aggregated form for forecasting validation and do not in- clude personally identifiable information. No human participants are involved, and no interventions are performed. All data were obtained and used in accordance with the corresponding data-use policies and institutional regulations; thus, IRB review and informed consent are not applicable. 7 Conclusion The TaCT framework proposed in this study introduces a new paradigm for the application of AI in meteorological science. Its core value can be summarized as follows: Key Contribution to the Domain: The primary contribution of this work is empowering AI models with a “surgical” correction capability. This enables the model to specifically rectify critical failures in predicting high-impact events, such as tropical cyclones, rather than undergoing a broad, general optimization. This directly addresses a core obstacle to the operational trust of AI models: their unreliability in handling high-risk, low-frequency events, thereby paving the way for their deeper integration into disaster mitigation decisions. Specific AI/ML Challenges Tackled: Through our “concept- centric” fine-tuning approach, we effectively address the training challenge of data scarcity for extreme weather. Concurrently, by linking model corrections to physical concepts, we significantly enhance model interpretability, directly confronting the “black-box” problem. General Challenges with Using AI/ML: Our “concept-gating” mechanism facilitates precise interventions on specific errors, thereby mitigating the common issue of "catastrophic forgetting" that occurs during specialized optimization. This provides a tangible technical solution for balancing the "specialist" and "generalist" capabilities of AI models. GenAI Disclosure We used large language models (LLMs) only to assist with and polish the writing. Specifically, LLMs were used to: (1) check grammar and spelling; and (2) help improve the phrasing in professional academic English. LLMs were not used to generate research ideas, methods, experiments, figures, or results. Target Concept Tuning Improves Extreme Weather ForecastingConference acronym ’X, June 03–05, 2018, Woodstock, NY References [1]Abubakar Abid, Mert Yuksekgonul, and James Zou. 2022. Meaningfully debugging model mistakes using conceptual counterfactual explanations. In International Conference on Machine Learning. PMLR, 66–88. [2]Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. 2023. Accurate medium-range global weather forecasting with 3D neural networks. Nature 619, 7970 (2023), 533–538. [3]Bart Bussmann, Patrick Leask, and Neel Nanda. [n. d.]. BatchTopK Sparse Au- toencoders. In NeurIPS 2024 Workshop on Scientific Methods for Understanding Deep Learning. [4]Lei Chen, Xiaohui Zhong, Feng Zhang, Yuan Cheng, Yinghui Xu, Yuan Qi, and Hao Li. 2023. FuXi: a cascade machine learning forecasting system for 15-day global weather forecast. npj climate and atmospheric science 6, 1 (2023), 190. [5]Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 8493–8502. [6]N De Cao, W Aziz, and I Titov. 2021. Editing Factual Knowledge in Language Models. In EMNLP 2021-2021 Conference on Empirical Methods in Natural Language Processing, Proceedings. 6491–6506. [7]Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al.2023. Parameter- efficient fine-tuning of large-scale pre-trained language models. Nature machine intelligence 5, 3 (2023), 220–235. [8] John A Dutton. 2002. Opportunities and priorities in a new era for weather and climate services. Bulletin of the American Meteorological Society 83, 9 (2002), 1303–1311. [9]Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al.2022. Toy models of superposition. https://transformer- circuits.pub/2022/toy_model/index.html. (2022). [10] Courtney L Gallen and Mark D’Esposito. 2019. Brain modularity: a biomarker of intervention-related plasticity. Trends in cognitive sciences 23, 4 (2019), 293–304. [11]Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. [n. d.]. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Representations. [12]Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Watten- berg, Jacob Andreas, Yonatan Belinkov, and David Bau. [n. d.]. Linearity of Relation Decoding in Transformer Language Models. In The Twelfth International Conference on Learning Representations. [13]H Hersbach, B Bell, P Berrisford, S Hirahara, A Horányi, J Muñoz-Sabater, J Nicolas, C Peubey, R Radu, and D Schepers. 2020. The ERA5 global reanalysis, quarterly journal of the royal meteorological society. (2020). [14] Eleanor Holton. 2025. Parallels between human and artificial minds when new learning erases old knowledge. [15]Tao Hong, Pierre Pinson, Yi Wang, Rafał Weron, Dazhi Yang, and Hamidreza Zareipour. 2020. Energy forecasting: A review and outlook. IEEE Open Access Journal of Power and Energy 7 (2020), 376–388. [16] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In International conference on machine learning. PMLR, 2790–2799. [17] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.2022. Lora: Low-rank adaptation of large language models. ICLR 1, 2 (2022), 3. [18]Richard W Katz and Allan H Murphy. 1997. Economic value of weather and climate forecasts. Cambridge University Press. [19]Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al.2018. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning. PMLR, 2668–2677. [20] Kenneth R Knapp, Michael C Kruk, David H Levinson, Howard J Diamond, and Charles J Neumann. 2010. The international best track archive for climate stewardship (IBTrACS) unifying tropical cyclone data. Bulletin of the American Meteorological Society 91, 3 (2010), 363–376. [21]Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, et al.2023. Learning skillful medium-range global weather forecasting. Science 382, 6677 (2023), 1416–1421. [22]Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The Power of Scale for Parameter-Efficient Prompt Tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 3045–3059. [23]Xiang Lisa Li and Percy Liang. 2021. Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Proceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 4582–4597. [24]Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning. [25]Wenhao Liu, Xiaohua Wang, Muling Wu, Tianlong Li, Changze Lv, Zixuan Ling, Zhu JianHao, Cenyuan Zhang, Xiaoqing Zheng, and Xuan-Jing Huang. 2024. Aligning large language models with human preferences through representation engineering. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 10619–10638. [26] Xiaoqin Lu, Hui Yu, Ming Ying, Bingke Zhao, Shuai Zhang, Limin Lin, Lina Bai, and Rijin Wan. 2021. Western North Pacific tropical cyclone database created by the China Meteorological Administration. Advances in Atmospheric Sciences 38, 4 (2021), 690–699. [27] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in neural information processing systems 35 (2022), 17359–17372. [28]Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. [n. d.]. Mass-Editing Memory in a Transformer. In The Eleventh International Conference on Learning Representations. [29]Peisong Niu, Ziqing Ma, Tian Zhou, Weiqi Chen, Lefei Shen, Rong Jin, and Liang Sun. 2025. Utilizing strategic pre-training to reduce overfitting: Baguan-a pre-trained weather forecasting model. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 2186–2197. [30]Yushu Pan and Elias Bareinboim. 2024. Counterfactual Image Editing. In Interna- tional Conference on Machine Learning. PMLR, 39087–39101. [31]Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 7654–7673. [32]Stephan Rasp, Stephan Hoyer, Alexander Merose, Ian Langmore, Peter Battaglia, Tyler Russell, Alvaro Sanchez-Gonzalez, Vivian Yang, Rob Carver, Shreya Agrawal, et al.2024. WeatherBench 2: A benchmark for the next generation of data-driven global weather models. Journal of Advances in Modeling Earth Systems 16, 6 (2024), e2023MS004019. [33]Alexis Ross, Ana Marasović, and Matthew E Peters. 2021. Explaining NLP mod- els via minimal contrastive editing (MiCE). In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. 3840–3852. [34] Andrea B Schumacher, Mark DeMaria, and John A Knaff. 2009. Objective estima- tion of the 24-h probability of tropical cyclone formation. Weather and Forecasting 24, 2 (2009), 456–471. [35]Jimeng Shi, Azam Shirali, Bowen Jin, Sizhe Zhou, Wei Hu, Rahuul Rangaraj, Shaowen Wang, Jiawei Han, Zhaonan Wang, Upmanu Lall, et al.2025. Deep learning and foundation models for weather prediction: A survey. arXiv preprint arXiv:2501.06907 (2025). [36] Yiming Teng, Zaharah Bukhsh, Yingqian Zhang, and Hang Yin. 2025. Self- supervise ensemble for extreme imbalance data streams with concept drift. Neu- rocomputing (2025), 132154. [37]Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2024. Large language model unlearning. Advances in Neural Information Processing Systems 37 (2024), 105425–105475. [38] Ming Ying, Wei Zhang, Hui Yu, Xiaoqin Lu, Jingxian Feng, Yongxiang Fan, Yongti Zhu, and Dequan Chen. 2014. An overview of the China Meteorological Adminis- tration tropical cyclone database. Journal of Atmospheric and Oceanic Technology 31, 2 (2014), 287–301. [39]Fan Zhang, Melissa Petersen, Leigh Johnson, James Hall, and Sid E O’Bryant. 2022. Hyperparameter tuning with high performance computing machine learning for imbalanced Alzheimer’s disease data. Applied Sciences 12, 13 (2022), 6670. [40] Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Chaojun Xiao, Xiaozhi Wang, Xu Han, Zhiyuan Liu, Ruobing Xie, Maosong Sun, and Jie Zhou. 2023. Emergent Modularity in Pre-trained Transformers. In Findings of the Association for Com- putational Linguistics: ACL 2023, Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 4066–4083. doi:10.18653/v1/2023.findings-acl.250 [41]Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. 2023. Representation Engineering: A Top-Down Approach to AI Transparency. CoRR (2023). Conference acronym ’X, June 03–05, 2018, Woodstock, NYReh et al. A Implementation Details of Sparse Autoencoders For the SAE, we follow the setting of Gao et al. [11] and employ a Top-k function for sparsity. Additionally, we normalize the hidden embeddings before feeding them into SAE. Specifically, given an input climate field푋 푡 ∈ R 퐻×푊×퐶 , let the hidden embeddings at the 푙-th layer be퐻 ∈ R 퐻×푊×퐷 . We first reshape these embeddings into a 2D matrix, which we denote as 퐻 ′ , of shape(퐻 ×푊)×푑 : 퐻 ′ ∈ R (퐻×푊)×푑 .(17) We then compute the mean휇 퐻 ∈ R 푑 and standard deviation휎 퐻 ∈ R 푑 across the second dimension (the flattened spatial dimensions). Finally, these statistics are used to standardize the matrix퐻 ′ . The standardized matrix, ˆ 퐻 ′ , is computed as follows: ˆ 퐻 ′ = 퐻 ′ − 휇 퐻 휎 퐻 +휖 ,(18) where 휖 is a small constant added for numerical stability. To alleviate the issue of “dead concepts”, we introduce an aux- iliary loss,퐴푢푥(·), during training. Specifically, for concepts that remain inactive for extended periods, we sample the top-k residuals from the SAE’s reconstruction corresponding to these concepts, as formalized in Eq. 19 . 퐴푢푥=||(퐻 ′ − ˆ 퐻 ′ )− 퐷푒푐(푇표푝푘(푧 푑푒푎푑 ))|| 2 .(19) B Statistical Methods for Concept Discovery As a component of our ablation study (see Section 4.3), we bench- marked our proposed method against a traditional statistical ap- proach for identifying model concepts associated with typhoon events. The objective of this baseline method is to isolate concepts from SAE that exhibit the most significant difference in activation between typhoon and non-typhoon conditions. The methodology is implemented through the following steps: B.1 Data Collection and Patch Labeling We first leverage the IBTrACS (International Best Track Archive for Climate Stewardship) dataset to source the historical best- track positions (latitude and longitude) of typhoons at 6-hour in- tervals. The model’s input data, derived from the ERA5 dataset, is segmented into a grid of patches. Let the set of all input patches be denoted byP. We define a mapping function,푀, which associates a given typhoon’s geo- coordinate with its corresponding patch inP. Using this, we parti- tion all patches into two distinct subsets: • P typhoon : The set of patches containing a typhoon center. P typhoon =푝 ∈ P | 푀(lat, lon)= 푝,(푙푎푡,푙표푛) ∈ 퐼퐵푇푟퐴퐶푆 • P non-typhoon : The set of patches not containing a typhoon center. P non-typhoon =P typhoon B.2 Concept Activation Analysis For each concept,푐 푖 , within the SAE’s concept library, we compute its mean activation value across all patches in theP typhoon set. Let 푎(푐 푖 ,푝)be the activation of concept푐 푖 for a given patch푝. The mean activation is formulated as: ̄ 퐴(푐 푖 ,P typhoon )= 1 |P typhoon | ∑︁ 푝∈P typhoon 푎(푐 푖 ,푝)(20) Similarly, we calculate the mean activation for the same concept across theP non-typhoon set: ̄ 퐴(푐 푖 ,P non-typhoon )= 1 |P non-typhoon | ∑︁ 푝∈P non-typhoon 푎(푐 푖 ,푝)(21) B.3 Difference Calculation and Ranking To quantify the relevance of each concept to the presence of a typhoon, we calculate the delta (Δ) between the two mean activation values: Δ(푐 푖 )= ̄ 퐴(푐 푖 ,P typhoon )− ̄ 퐴(푐 푖 ,P non-typhoon )(22) All SAE concepts are then ranked in descending order based on their resultingΔ value. B.4 Relevant Concept Selection Concepts with the highest positiveΔvalues are considered to be the most statistically correlated with typhoon phenomena. In our ablation study, these top-ranked concepts were selected to represent the outcome of the statistical discovery method for comparative analysis. C Implementation Details For our SAE, we targeted the 18th layer of the base model. The SAE was configured with an intermediate dimension of 15,360, a ReLU activation function, and a top-k value of 320. The SAE was trained for 12 hours on 6*A800 GPUs using the AdamW optimizer with a learning rate of 5e-5. For LoREFT, we apply interventions to the first 128 token posi- tions. For counterfactual concept localization, we randomly sampled 438 typhoon data with active tropical cyclone events from the year 2021 as the data for concept localization analysis. To ensure that the analysis accurately reflects the internal er- rors of the base model rather than the SAE’s own reconstruction errors, the training objective is to keep the output of the model’s subsequent layers as consistent as possible, whether the input is the original hidden embedding or the one reconstructed by the SAE, as formalizedF 푙 :퐿 (ℎ 푙 ) ≈F 푙 :퐿 ( ˆ ℎ 푙 ). D Interpretation Methods To automate the process and reduce manual effort, we design an MLLM-based automated interpretation framework. The prompt provided to the MLLM is detailed in Appendix E. Our input con- sists of the following components: input-field data, output-field data, the climatological mean, the importance of each variable for the current concept, latitude–longitude information, and land–sea mask information. Specifically, we compute the concept’s input- field data by averaging all variables over a local region centered at each activated location for the same concept. Similarly, we apply the same procedure to obtain the concept’s output-field data. To capture the differences between the concept-related fields and the Target Concept Tuning Improves Extreme Weather ForecastingConference acronym ’X, June 03–05, 2018, Woodstock, NY Figure 9: 72-Hours Results. EP denotes the Eastern Pacific and NA denotes the North Atlantic. “Typhoon pressure” refers to the minimum sea-level pressure in Pa, and “Typhoon wind speed” represents the maximum wind speed in m/s. Table 3: Typhoon Data Fine-tuning Results across Different Ocean Basins. RegionBase Model LoRA Adapter LiREFT Ours North Atlantic132.09124.18123.28131.99 119.21 Western Pacific82.0978.3978.3582.01 76.88 Eastern Pacific67.7365.1364.5867.67 63.53 Avg93.9789.2388.7493.89 86.54 Δ (%)7.913.002.487.83- mean state, we also provide the climatological mean as a reference baseline. We derive the importance of each variable to the concept by computing the gradients of the concept with respect to each variable. For latitude–longitude information, we use binning and take the activation frequency within each bin as the spatial distri- bution feature. For land–sea information, we categorize locations into three types: land, ocean, and coastline. E Interpretation Prompt The detailed prompt template employed for SAE concept interpre- tation is presented in Figure 10. F 72-Hours Results Figure 9 shows the 72-hour forecast results for other regions, where EP denotes the Eastern Pacific and NA denotes the North Atlantic. G Fine-tuning with Same Data To compare with models fine-tuned on typhoon data, we evaluate our method against several fine-tuning approaches trained on the same typhoon dataset. Our method still achieves competitive per- formance. In contrast, other methods may suffer from overfitting due to the scarcity of tropical cyclone data, where excessive fine- tuning on such limited samples can degrade generalization ability, the experimental results are shown in Table 3. Conference acronym ’X, June 03–05, 2018, Woodstock, NYReh et al. You are given data from a large-scale weather prediction model. Model logic: -Input: Assimilated initial Earth field -Output: Residual change = (future climate − initial climate) -Predicted future field = Input + Output -Positive value → increase in future state -Negative value → decrease in future state The provided data describes a concept extracted by a Sparse Autoencoder (SAE). A concept = a meteorological feature detected from the input that causes specific changes in the output. Example: “El Niño” detected in inputs → outputs show warmer ocean temperatures in expected regions. Your task: Using the provided concept data (input features, output residuals, activation distribution, and land–sea location probabilities), identify three possible meteorological phenomena the concept could represent, and rank them in descending order of confidence. Data Format Definitions 1. Input Section: ... 2. Output Section: ... 3. Activation Position Distribution: ... 4. Land–Sea Distribution: ... Current Concept Data Below: %s Required Output Format: [ "Reasoning": "<Explanation>", "Phenomenon": "<Name>", "Confidence": "<1-4>" , ...] Confidence scoring: -4: All trends strongly match one phenomenon -3: Most trends match, minor inconsistencies -2: Some trends match, weaker evidence -1: Very little matches, unsure Checklist for Reasoning (Model must check each item): 1. Variable Influence: Identify which input/output variables ranked highest and their importance. 2. Mean vs Global Mean: Determine if concept regions are above/below average values. 3. Gradients:Interpret whether increasing/decreasing the variable activates or suppresses the concept. 4. Geographic Activation: Match lat/lonactivation zones to known climate phenomenon regions. 5. Land/Sea Context: Confirm if the event is oceanic, continental, or coastal. 6. Output Residual Pattern: Check how the future change pattern matches expected phenomenon impacts. Figure 10: Prompt for interpreting concepts in multimodal models