Paper deep dive
Gradient-Guided Decoupled Adaptation for Geospatial Vision-Language Models
Dongdong Wang, Deepak Balakrishnan, Ravi Srinivasan, Shenhao Wang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Existing geospatial vision-language models (Geo-VLMs) typically optimize diverse geospatial tasks through a unified multi-task adaptation paradigm without explicitly accounting for the heterogeneous optimization characteristics. Our empirical observations reveal heterogeneous gradient characteristics across tasks, including vision-language differences, intra-branch gradient relationships, and task interference, which hinder effective multi-task optimization. Motivated by these observations, we propose Gradient-Guided Decoupled Adaptation (G2DA), a gradient-aware optimization framework for multi-task Geo-VLM learning. G2DA first partitions tasks into vision- and language-centric groups through gradient-guided cross-modal decoupling. It then constructs modality-specific curricula based on task gradient similarity and employs bidirectional rehearsal to mitigate the recency effects introduced by sequential optimization. We evaluate G2DA on three Geo-VLM benchmarks using six InternVL3 and Qwen3.5-VL variants, along with GeoChat and GeoLLaVA. Across all 24 benchmark-model combinations, G2DA consistently outperforms representative baselines, improving over the strongest competitor by 3.08, 4.30, and 2.81 percentage points on UrBench-MCQ, XLRS-Bench-Lite, and VRS-Bench-VQA, respectively. These results demonstrate the effectiveness of gradient-guided task organization for Geo-VLM adaptation.
Tags
Links
- Source: https://arxiv.org/abs/2609.32737v1
- Canonical: https://arxiv.org/abs/2609.32737v1
Trouble viewing inline? Open PDF directly →
Full Text
44,397 characters extracted from source content.
Expand or collapse full text
Gradient-Guided Decoupled Adaptation for Geospatial Vision-Language Models Dongdong Wang Deepak Balakrishnan Ravi Srinivasan Shenhao Wang Abstract Existing geospatial vision-language models (Geo-VLMs) typically optimize diverse geospatial tasks through a unified multi-task adaptation paradigm without explicitly accounting for the heterogeneous optimization characteristics. Our empirical observations reveal heterogeneous gradient characteristics across tasks, including vision-language differences, intra-branch gradient relationships, and task interference, which hinder effective multi-task optimization. Motivated by these observations, we propose Gradient-Guided Decoupled Adaptation (G2DA), a gradient-aware optimization framework for multi-task Geo-VLM learning. G2DA first partitions tasks into vision- and language-centric groups through gradient-guided cross-modal decoupling. It then constructs modality-specific curricula based on task gradient similarity and employs bidirectional rehearsal to mitigate the recency effects introduced by sequential optimization. We evaluate G2DA on three Geo-VLM benchmarks using six InternVL3 and Qwen3.5-VL variants, along with GeoChat and GeoLLaVA. Across all 24 benchmark–model combinations, G2DA consistently outperforms representative baselines, improving over the strongest competitor by 3.08, 4.30, and 2.81 percentage points on UrBench-MCQ, XLRS-Bench-Lite, and VRS-Bench-VQA, respectively. These results demonstrate the effectiveness of gradient-guided task organization for Geo-VLM adaptation. 1University of Florida Gainesville, FL, USA dongdongwang, deepakbd, sravi, shenhaowang@ufl.edu Introduction Recent vision-language models (VLMs) have achieved strong performance across a wide range of multimodal reasoning tasks. Building on these advances, geospatial vision-language models (Geo-VLMs) have emerged as a powerful framework for geospatial intelligence by jointly modeling multi-view geospatial imagery and natural language (Mai et al. 2024). To support diverse reasoning capabilities within a single model, including visual perception, object grounding, spatial reasoning, retrieval, and quantitative analysis, existing approaches predominantly adopt unified multi-task learning, where tasks with substantially different objectives are jointly optimized using shared model parameters (Kuckreja et al. 2024; Feng et al. 2025; Wang et al. 2026). While unified multi-task learning facilitates knowledge sharing across tasks, it jointly optimizes heterogeneous tasks using shared model parameters, implicitly assuming compatible optimization objectives. However, geospatial tasks exhibit substantially different optimization characteristics due to their varying reliance on visual perception and language reasoning (Zhou et al. 2025). Consequently, they generate heterogeneous optimization signals, and jointly optimizing them within a shared parameter space often introduces conflicting gradients that hinder effective multi-task adaptation (Jeong and Yoon 2025). Existing multi-task optimization methods mitigate gradient conflicts through projection (Yu et al. 2020), reweighting (Chen et al. 2018), or balancing (Liu et al. 2023a), yet still optimize all tasks within a unified parameter space. This assumption is particularly challenging for Geo-VLMs, where tasks exhibit heterogeneous optimization demands across vision and language encoders due to their varying reliance on visual perception and language reasoning. Consequently, optimization strategies designed for a unified parameter space may fail to fully exploit these cross-modal characteristics. Our empirical gradient analysis reveals substantial gradient conflicts across the task set, distinct optimization preferences between the vision and language components, and persistent intra-encoder interference within each component. Motivated by these observations, we propose G2DA, a gradient-guided optimization framework for Geo-VLM adaptation. Unlike prior methods that optimize all tasks within a unified parameter space, G2DA organizes multi-task optimization according to heterogeneous cross-modal gradient characteristics through gradient-guided decoupling, modality-specific curriculum learning, and bidirectional rehearsal. Our contributions are summarized as follows: • We present a gradient-centric analysis of multi-task optimization in Geo-VLMs. Our empirical study reveals hierarchical optimization heterogeneity, including cross-task gradient conflicts, distinct optimization preferences between vision and language components, and persistent intra-component task interference. • Given the empirical analyses, we design G2DA, a unified gradient-guided optimization framework that integrates cross-modal optimization decoupling, modality-specific curriculum learning, and cross-task rehearsal to address optimization heterogeneity in Geo-VLM learning. • Extensive experiments on multiple Geo-VLM benchmarks and diverse model families demonstrate that G2DA consistently outperforms representative baselines, establishing gradient-guided optimization as an effective paradigm for adapting Geo-VLMs. Related Work Geospatial Vision-Language Models Recent advances in VLMs (Bai et al. 2023; Chen et al. 2024) have enabled the adaptation of general-purpose VLMs to geospatial domains through instruction tuning (Liu et al. 2023b), leading to models such as GeoChat (Kuckreja et al. 2024) and GeoLLaVA (Wang et al. 2026). At the same time, geospatial benchmarks including UrBench (Zhou et al. 2025), XLRS-Bench (Wang et al. 2025), and VRS-Bench (Li et al. 2024) have accelerated Geo-VLM development by supporting unified training and evaluation across diverse reasoning tasks. Despite their architectural differences, existing Geo-VLMs are typically adapted through unified multi-task learning, jointly optimizing heterogeneous tasks under a shared objective without considering their optimization compatibility (Kuckreja et al. 2024; Feng et al. 2025; Wang et al. 2026). As task diversity continues to increase, this unified optimization strategy can introduce gradient interference, limiting the efficiency and effectiveness of multi-task adaptation. Gradient Conflict in Multi-task Learning Multi-task learning improves generalization by jointly optimizing multiple tasks within a shared model. Existing optimization methods primarily mitigate task interference caused by conflicting gradients. PCGrad (Yu et al. 2020) alleviates conflicts through gradient projection, while CAGrad (Liu et al. 2021) computes conflict-aware gradient updates via adaptive gradient combination. GradNorm (Chen et al. 2018) and FAMO (Liu et al. 2023a) balance task optimization through adaptive gradient weighting. More recently, Selective Group Update (STGU) (Jeong and Yoon 2025) shifts the focus from gradient modification to task scheduling, demonstrating improved optimization for heterogeneous multi-task learning. However, these methods, including STGU, still optimize heterogeneous tasks within a unified parameter space, overlooking the distinct optimization characteristics of the vision and language components in Geo-VLMs. Consequently, they are not designed to exploit task-specific cross-modal optimization preferences that naturally arise in Geo-VLM adaptation. Curriculum and Rehearsal Learning Curriculum learning improves optimization by organizing training samples or tasks into a structured learning sequence rather than random presentation (Bengio et al. 2009), and has been extended to multi-task learning through task-level scheduling strategies that improve optimization efficiency (Wang et al. 2021). However, sequential curricula may introduce optimization imbalance across tasks as training progresses and earlier tasks are no longer revisited. Indeed, one-pass curriculum scheduling has been linked to continual learning, where stage-wise optimization may lead to catastrophic forgetting (Wang et al. 2021). Rehearsal-based learning mitigates this issue by revisiting previously learned samples or tasks during optimization, and has proven particularly effective for heterogeneous tasks with low similarity (Deng et al. 2025). Nevertheless, existing rehearsal methods are primarily designed for continual learning, while their application to mitigating optimization imbalance in sequential curriculum learning for multi-task Geo-VLM adaptation remains unexplored. Method Problem Formulation Let =T1,T2,…,TNT=\T_1,T_2,…,T_N\ denote a set of Geo-VLM tasks. Each task TiT_i is associated with a dataset i=(,,)D_i=\(x,q,y)\, where x is an image, q is a language query, and y is the supervision. A unified Geo-VLM f(⋅,Θ)f(·; ) is optimized over all tasks with the objective ℒ(Θ)=∑i=1Nλiℒi(Θ),L( )= _i=1^N _iL_i( ), (1) where ℒiL_i is the task-specific loss and λi _i balances its contribution to the overall objective. Unlike conventional multi-task learning, which primarily aims to improve the converged solution, we study a fixed-budget optimization setting. Let B denote the total number of parameter updates available during training. Under the same initialization Θ0 _0 and optimization budget B, the final optimization outcome depends on how heterogeneous task data are presented throughout training. We represent the entire training schedule as Π=D(1),D(2),…,D(B), = \D^(1),D^(2),…,D^(B) \, (2) where each mini-batch D(t)∈⋃i=1NiD^(t)∈ _i=1^ND_i is sampled from one of the task datasets. Different schedules Π induce different optimization trajectories and therefore different model parameters after exactly B updates, denoted by ΘBΠ _B . Our objective is to design an effective training schedule that improves the joint optimization objective under the same optimization budget: ℒ(ΘBΠ)=∑i=1Nλiℒi(ΘBΠ),s.t.|Π|=B.L( _B )= _i=1^N _iL_i( _B ), .t. | |=B. (3) This formulation raises a fundamental question: How should heterogeneous Geo-VLM tasks be organized under a fixed optimization budget? To answer this question, we analyze task gradients to characterize optimization heterogeneity and use these observations to guide the design of G2DA. Gradient Characterization Figure 1: Task-gradient analysis on UrBench-MCQ. (a) Pairwise task-gradient cosine similarity matrix. Positive (red) and negative (blue) values indicate aligned and conflicting optimization directions. (b) Normalized proportions of the total gradient norm contributed by the vision (green) and language (pink) encoders for each task. To better understand the optimization challenges under a fixed update budget, we characterize heterogeneous geospatial vision-language tasks through gradient analysis. Specifically, for each task, we compute the gradient with respect to the trainable parameters and use the gradient cosine similarity as a proxy to quantify pairwise optimization alignment. Sij=i⊤j‖i‖2‖j‖2,S_ij= g_i g_j\|g_i\|_2\|g_j\|_2, (4) where ig_i and jg_j denote the task gradients of tasks i and j, respectively. To characterize each task’s cross-modal optimization preference, we further compute its vision gradient proportion as another proxy, ri=‖iv‖2‖iv‖2+‖il‖2,r_i= \|g_i^v\|_2\|g_i^v\|_2+\|g_i^l\|_2, (5) where ivg_i^v and ilg_i^l represent the gradients on the vision and language encoders, respectively. With this proxy, a larger rir_i indicates stronger optimization demand on the vision encoder, while a smaller value indicates greater reliance on the language encoder. Together, the gradient cosine similarity and gradient proportion characterize inter-task optimization compatibility and cross-modal optimization preference, respectively. We first use UrBench (Zhou et al. 2025), where all questions are reformulated into multiple-choice format (UrBench-MCQ), as a pilot benchmark to empirically analyze the gradient behavior of heterogeneous Geo-VLM tasks using Qwen3.5-VL-2B. The resulting analyses reveal three observations that motivate the design of G2DA. Corresponding results on additional benchmarks are included in the supplementary material. Observation 1: Inter-task gradient conflict. Different Geo-VLM tasks often induce substantially different optimization directions. As shown by the pairwise gradient cosine similarity matrices, some task pairs exhibit highly aligned gradients, whereas others are nearly orthogonal or negatively correlated. A negative cosine similarity indicates that decreasing the loss of one task may increase the loss of another under the same parameter update. Therefore, uniformly mixing heterogeneous tasks in conventional multi-task training introduces optimization interference and reduces optimization efficiency. Observation 2: Cross-modal optimization imbalance. Different tasks exhibit substantially different vision–language gradient proportions: some are predominantly optimized through visual representations, whereas others rely primarily on language reasoning, with many requiring balanced optimization across both modalities. This heterogeneous optimization demand suggests that applying identical parameter updates across both encoders is inefficient, motivating modality-aware optimization. Observation 3: Intra-encoder gradient heterogeneity. Even after separating tasks according to their cross-modal optimization characteristics, substantial gradient heterogeneity remains within each encoder. Pairwise gradient cosine similarities reveal that tasks optimized on the same encoder still exhibit diverse optimization directions, indicating that encoder decoupling alone cannot fully eliminate optimization interference. This observation motivates gradient-aware curriculum scheduling within each encoder. Figure 2: Task-level gradient cosine similarity after cross-modal decoupling and curriculum ordering on UrBench-MCQ. Optimization Figure 3: Overview of the proposed G2DA framework, consisting of gradient-guided cross-modal decoupling, gradient-aware curriculum scheduling, and bidirectional rehearsal. Motivated by the observations above, we propose G2DA, a unified optimization framework for fixed-budget Geo-VLM adaptation. Unlike conventional multi-task learning, Geo-VLM tasks exhibit heterogeneous optimization preferences across vision and language encoders due to their varying reliance on visual perception and language reasoning. Therefore, rather than organizing tasks solely according to pairwise gradient similarity, G2DA first decouples tasks based on their cross-modal optimization preferences and then models task compatibility within each modality through gradient-aware curriculum scheduling, followed by bidirectional rehearsal for knowledge preservation. The overall framework is illustrated in Figure 3. Gradient-guided cross-modal decoupling. Motivated by Observation 2, we use the vision gradient proportion rir_i in Eq. 5 as a proxy to characterize each task’s relative preference for the vision and language encoders. Based on the estimated gradient proportion rir_i, tasks are divided into two groups: vision-centric and language-centric tasks. Grouping tasks with similar cross-modal preferences reduces interference caused by heterogeneous encoder demands and allows the optimization budget to be allocated more effectively. As shown in Figure 2, cross-modal decoupling increases local gradient cosine similarity, indicating improved optimization compatibility within each branch. However, considerable intra-branch gradient conflicts remain, motivating the branch-specific curriculum learning introduced in the next stage. Gradient-aware curriculum scheduling. Motivated by Observation 3, we compute gradient cosine similarities within each modality-centric optimization group to characterize task compatibility. Specifically, for task tit_i, the curriculum score is defined as ϕie=1|e|−1∑j∈ej≠i(¯ie)⊤¯je‖¯ie‖2‖¯je‖2, _i^e= 1|T^e|-1 _ subarraycj ^e\\ j≠ i subarray ( g_i^\,e) g_j^\,e\| g_i^\,e\|_2\| g_j^\,e\|_2, (6) where eT^e denotes one of the vision- or language-centric task groups. Tasks with higher average compatibility are optimized earlier, yielding a curriculum that progresses from broadly aligned to relatively conflicting tasks. By constructing the curriculum within each modality-centric group, the proposed strategy replaces random shuffling with gradient-aware sequential training, improving optimization compatibility between consecutive tasks. However, this sequential optimization also introduces recency effects and catastrophic forgetting (Kemker et al. 2018), motivating the bidirectional rehearsal strategy described next. Bidirectional rehearsal for knowledge preservation. Once the curriculum is established, tasks are optimized sequentially according to the curriculum order. Such sequential optimization inevitably introduces a recency effect, where knowledge acquired in earlier stages is gradually overwritten by later stages. To alleviate this issue, we optimize each task group in both the ascending curriculum order and its reverse, allowing every curriculum stage to be optimized once in descending compatibility order and once in the reverse order. Meanwhile, for each curriculum direction, the effective training set of stage k is defined as ~k=k∪ℳk, D_k=D_k _k, (7) where kD_k denotes the current curriculum stage and ℳk∼Uniform(kc),|ℳk|=|k|,M_k (D_k^c), |M_k|=|D_k|, (8) is a rehearsal set uniformly sampled from the complementary curriculum stages kcD_k^c such that the total number of rehearsal samples equals the size of the current stage. During optimization, mini-batches are uniformly sampled from the effective training set ~k D_k. By traversing the curriculum in both ascending and descending similarity orders while jointly optimizing the current-stage and rehearsal samples, the proposed bidirectional rehearsal preserves knowledge across the entire curriculum, mitigates optimization drift, and improves balanced performance across heterogeneous tasks. G2DA Framework Algorithm 1 summarizes the optimization procedure of G2DA. Given the task-level average gradients ¯tv,¯tlt=1T\ g_t^v, g_t^l\_t=1^T, G2DA first computes the gradient proportion rtr_t for each task and partitions the task set into vT^v and lT^l. For each modality e∈v,le∈\v,l\, pairwise gradient cosine similarities SijeS_ij^e are computed to obtain the curriculum scores ϕie _i^e, from which the curriculum order Πe ^e is constructed. Finally, for each curriculum order and its reverse, rehearsal samples ℳkM_k are uniformly drawn from the remaining datasets to form the augmented training set ~k D_k, which is used to optimize the corresponding parameters Θe ^e. Algorithm 1 G2DA Optimization 0: Task datasets tt=1T\D_t\_t=1^T, parameters Θ=Θv,Θl =\ ^v, ^l\ 0: Optimized parameters Θ 1: Compute task-level average gradients ¯tv,¯tlt=1T\ g_t^v, g_t^l\_t=1^T 2: Stage 1: Cross-modal decoupling 3: for t=1,…,Tt=1,…,T do 4: rt←‖¯tv‖2‖¯tv‖2+‖¯tl‖2r_t← \| g_t^v\|_2\| g_t^v\|_2+\| g_t^l\|_2 5: end for 6: v←t:rt>0.5T^v←\t:r_t>0.5\, l←t:rt≤0.5T^l←\t:r_t≤ 0.5\ 7: Stage 2: Gradient-aware curriculum 8: for e∈v,le∈\v,l\ do 9: Sije←cos(¯ie,¯je)S_ij^e← ( g_i^e, g_j^e), i,j∈ei,j ^e 10: ϕie←1|e|−1∑j≠iSije _i^e← 1|T^e|-1 _j≠ iS_ij^e 11: Πe←Sort↓(ϕiei∈e) ^e _ (\ _i^e\_i ^e) 12: end for 13: Stage 3: Bidirectional rehearsal 14: for e∈v,le∈\v,l\ do 15: for π∈Πe,Reverse(Πe)π∈\ ^e,Reverse( ^e)\ do 16: for k∈πD_k∈π do 17: kc←⋃i∈π,i≠kiD_k^c← _D_i∈π,\,i≠ kD_i 18: ℳk∼Uniform(kc)M_k (D_k^c), |ℳk|=|k||M_k|=|D_k| 19: ~k←k∪ℳk D_k _k _k 20: Optimize Θe ^e on ~k D_k 21: end for 22: end for 23: end for 24: return Θ Experiment Experimental Setup Benchmarks and evaluation. We evaluate G2DA on three Geo-VLM benchmarks: UrBench-MCQ, XLRS-Bench-Lite, and the VQA subset of VRS-Bench (VRS-Bench-VQA). All VQA tasks are reformulated as multiple-choice questions with images resized to 1024×10241024× 1024. Performance is measured by accuracy (%) averaged across all tasks. All experiments are repeated with three random seeds, and the mean and standard deviation are reported. Additional benchmark details are provided in the supplementary material. Models and baselines. We evaluate G2DA on two general-purpose VLM families, InternVL3 and Qwen3.5-VL, across multiple model scales, together with two representative Geo-VLMs, GeoChat and GeoLLaVA. We compare against representative multi-task optimization methods, including Joint training, GradNorm (Chen et al. 2018), PCGrad (Yu et al. 2020), CAGrad (Liu et al. 2021), FAMO (Liu et al. 2023a), and STGU (Jeong and Yoon 2025), covering gradient balancing, conflict mitigation, adaptive multi-objective optimization, and task scheduling. All methods are evaluated under identical optimization budgets. Implementation details. Experiments are conducted on four NVIDIA B200 GPUs (180 GB each) using bfloat16 precision and the AdamW optimizer. The batch size is 1 with gradient accumulation over 8 steps, and the generation temperature is fixed at 1. Baselines are trained for 8 epochs, while G2DA is trained for 2 epochs. Since each epoch of G2DA performs both forward and reverse curricula with bidirectional rehearsal, the total number of optimization steps is comparable across methods. The learning rate and weight decay are 5×10−65× 10^-6 and 1×10−61× 10^-6, respectively. LoRA adapters use rank r=8r=8, scaling factor α=16α=16, and dropout 0.10.1. Results Table 1 summarizes the overall performance. G2DA achieves the best results in all 24 benchmark-model combinations, indicating consistent performance across diverse architectures and model scales. Compared with the strongest competing baseline in each setting, G2DA still improves performance. On UrBench-MCQ, G2DA improves performance by 3.08 p on average, with gains ranging from 1.0 to 7.0 p. Existing gradient manipulation methods, such as PCGrad and CAGrad, often perform comparably to or even worse than joint optimization, while STGU is generally the strongest baseline. On the more challenging XLRS-Bench-Lite, the advantage of G2DA becomes more pronounced. G2DA improves upon the strongest baseline by an average of 4.30 p, with improvements reaching 8.0 p on Qwen3.5-VL-27B and 7.0 p on InternVL3-14B. On VRS-Bench-VQA, G2DA achieves an average improvement of 2.81 p over the strongest baseline, with gains of up to 5.0 p. Table 1 also evaluates G2DA on two representative Geo-VLMs, GeoChat and GeoLLaVA. Compared with the strongest competing baseline, G2DA improves performance by 2-5 p, demonstrating that the proposed optimization framework is not limited to general-purpose VLMs but also transfers effectively to domain-specific Geo-VLMs. Table 1: Performance (%) over three independent runs. Best results are shown in bold, and second-best are underlined. Benchmark Model Optimization Method Joint PCGrad GradNorm STGU CAGrad FAMO G2DA UrBench-MCQ GeoChat-7B 51.1±0.251.1_± 0.2 48.5±0.248.5_± 0.2 49.9±0.249.9_± 0.2 51.2±0.351.2_± 0.3 48.3±0.248.3_± 0.2 50.2±0.350.2_± 0.3 54.1±0.254.1_± 0.2 GeoLLaVA-8B 51.7±0.251.7_± 0.2 48.5±0.248.5_± 0.2 50.4±0.350.4_± 0.3 50.3±0.250.3_± 0.2 48.6±0.248.6_± 0.2 50.3±0.250.3_± 0.2 54.8±0.254.8_± 0.2 InternVL3-2B 44.2±0.244.2_± 0.2 39.2±0.139.2_± 0.1 43.4±0.343.4_± 0.3 45.8±0.245.8_± 0.2 39.9±0.239.9_± 0.2 43.2±0.343.2_± 0.3 48.8±0.348.8_± 0.3 InternVL3-8B 51.9±0.151.9_± 0.1 48.1±0.448.1_± 0.4 51.9±0.451.9_± 0.4 54.9±0.354.9_± 0.3 48.1±0.248.1_± 0.2 51.8±0.251.8_± 0.2 57.8±0.157.8_± 0.1 InternVL3-14B 61.1±0.261.1_± 0.2 53.8±0.253.8_± 0.2 61.1±0.261.1_± 0.2 63.8±0.263.8_± 0.2 54.1±0.154.1_± 0.1 61.4±0.261.4_± 0.2 66.8±0.266.8_± 0.2 Qwen3.5-VL-2B 44.8±0.144.8_± 0.1 40.9±0.340.9_± 0.3 45.0±0.345.0_± 0.3 46.1±0.246.1_± 0.2 41.1±0.341.1_± 0.3 46.1±0.246.1_± 0.2 49.0±0.349.0_± 0.3 Qwen3.5-VL-8B 55.0±0.355.0_± 0.3 52.0±0.352.0_± 0.3 56.0±0.356.0_± 0.3 56.0±0.456.0_± 0.4 52.0±0.352.0_± 0.3 55.0±0.155.0_± 0.1 59.0±0.359.0_± 0.3 Qwen3.5-VL-27B 64.8±0.364.8_± 0.3 64.1±0.364.1_± 0.3 60.2±0.360.2_± 0.3 65.1±0.365.1_± 0.3 60.9±0.360.9_± 0.3 64.1±0.264.1_± 0.2 68.7±0.268.7_± 0.2 XLRS-Bench-Lite GeoChat-7B 38.4±0.238.4_± 0.2 35.3±0.235.3_± 0.2 37.4±0.237.4_± 0.2 39.0±0.239.0_± 0.2 35.3±0.235.3_± 0.2 38.5±0.238.5_± 0.2 41.0±0.241.0_± 0.2 GeoLLaVA-8B 43.8±0.243.8_± 0.2 39.9±0.239.9_± 0.2 42.4±0.242.4_± 0.2 43.0±0.243.0_± 0.2 39.9±0.239.9_± 0.2 42.7±0.242.7_± 0.2 47.8±0.247.8_± 0.2 InternVL3-2B 34.8±0.234.8_± 0.2 31.1±0.131.1_± 0.1 34.1±0.234.1_± 0.2 35.8±0.235.8_± 0.2 31.1±0.231.1_± 0.2 32.8±0.232.8_± 0.2 38.1±0.238.1_± 0.2 InternVL3-8B 38.1±0.338.1_± 0.3 34.5±0.434.5_± 0.4 36.1±0.336.1_± 0.3 38.6±0.438.6_± 0.4 34.3±0.334.3_± 0.3 35.9±0.235.9_± 0.2 42.2±0.442.2_± 0.4 InternVL3-14B 35.8±0.235.8_± 0.2 32.8±0.132.8_± 0.1 34.8±0.234.8_± 0.2 38.1±0.238.1_± 0.2 34.1±0.234.1_± 0.2 34.8±0.234.8_± 0.2 45.1±0.245.1_± 0.2 Qwen3.5-VL-2B 33.2±0.233.2_± 0.2 31.5±0.231.5_± 0.2 34.1±0.234.1_± 0.2 35.8±0.135.8_± 0.1 30.9±0.230.9_± 0.2 33.8±0.133.8_± 0.1 37.0±0.237.0_± 0.2 Qwen3.5-VL-8B 32.9±0.132.9_± 0.1 30.9±0.330.9_± 0.3 33.8±0.233.8_± 0.2 36.1±0.236.1_± 0.2 32.1±0.232.1_± 0.2 33.1±0.133.1_± 0.1 42.1±0.242.1_± 0.2 Qwen3.5-VL-27B 34.1±0.234.1_± 0.2 31.9±0.231.9_± 0.2 33.8±0.233.8_± 0.2 35.8±0.335.8_± 0.3 30.8±0.130.8_± 0.1 33.8±0.333.8_± 0.3 44.1±0.244.1_± 0.2 VRS-Bench-VQA GeoChat-7B 65.3±0.265.3_± 0.2 61.3±0.261.3_± 0.2 63.5±0.263.5_± 0.2 65.9±0.265.9_± 0.2 61.3±0.261.3_± 0.2 64.1±0.264.1_± 0.2 69.9±0.269.9_± 0.2 GeoLLaVA-8B 70.8±0.270.8_± 0.2 65.1±0.265.1_± 0.2 67.8±0.267.8_± 0.2 70.7±0.270.7_± 0.2 65.1±0.265.1_± 0.2 66.8±0.266.8_± 0.2 74.9±0.274.9_± 0.2 InternVL3-2B 65.4±0.365.4_± 0.3 60.9±0.460.9_± 0.4 65.4±0.365.4_± 0.3 65.1±0.465.1_± 0.4 61.0±0.361.0_± 0.3 63.0±0.463.0_± 0.4 68.5±0.368.5_± 0.3 InternVL3-8B 68.3±0.468.3_± 0.4 65.4±0.365.4_± 0.3 67.4±0.167.4_± 0.1 70.2±0.170.2_± 0.1 65.8±0.365.8_± 0.3 63.1±0.463.1_± 0.4 72.8±0.272.8_± 0.2 InternVL3-14B 70.2±0.370.2_± 0.3 66.1±0.466.1_± 0.4 65.8±0.365.8_± 0.3 70.9±0.370.9_± 0.3 65.1±0.365.1_± 0.3 65.2±0.365.2_± 0.3 75.2±0.475.2_± 0.4 Qwen3.5-VL-2B 64.8±0.164.8_± 0.1 60.2±0.260.2_± 0.2 64.7±0.164.7_± 0.1 64.7±0.264.7_± 0.2 61.9±0.261.9_± 0.2 61.1±0.261.1_± 0.2 66.1±0.366.1_± 0.3 Qwen3.5-VL-8B 67.8±0.267.8_± 0.2 64.8±0.164.8_± 0.1 67.8±0.267.8_± 0.2 69.9±0.269.9_± 0.2 62.8±0.262.8_± 0.2 67.1±0.267.1_± 0.2 73.0±0.173.0_± 0.1 Qwen3.5-VL-27B 69.9±0.269.9_± 0.2 65.8±0.265.8_± 0.2 67.1±0.267.1_± 0.2 70.9±0.170.9_± 0.1 65.8±0.265.8_± 0.2 65.8±0.165.8_± 0.1 74.1±0.274.1_± 0.2 Analyses Table 2: Per-task accuracy (%\%) on UrBench-MCQ with tasks grouped into vision and language branches. Model Vision-oriented (V) Tasks Language-oriented (L) Tasks Average Cam. Traffic Road Ori. Match City Count Img. Attr. Ground Role Comp. Scene VPR V L All Qwen3.5-VL-27B (Joint) 41.2 68.1 66.7 30.1 59.8 70.2 78.1 45.3 49.9 70.4 71.1 85.7 76.1 85.2 53.2 70.2 65.1 Qwen3.5-VL-27B (G2DA) 45.0 72.0 72.5 32.5 62.6 70.0 83.8 45.0 52.5 72.5 75.0 90.0 85.0 88.9 56.9 73.6 68.7 GeoLLaVA-8B (Joint) 38.2 73.1 50.1 24.9 44.3 54.8 49.1 40.2 46.2 62.4 63.3 54.5 53.2 70.2 46.1 54.9 51.7 GeoLLaVA-8B (G2DA) 40.0 75.0 56.7 25.0 48.2 55.0 54.1 41.0 49.0 65.0 67.5 58.7 62.5 74.0 49.0 58.5 54.8 Existing multi-task optimization methods show limited transferability to Geo-VLMs. From Table 1, none of the representative multi-task optimization methods consistently outperform joint training with random task shuffling. PCGrad and CAGrad incur the largest performance degradation, while GradNorm and FAMO exhibit comparatively smaller drops, although neither surpasses the joint-training baseline. This suggests that optimization strategies developed for conventional multi-task learning do not readily transfer to Geo-VLMs. One possible explanation is that the substantially higher task heterogeneity limits the effectiveness of generic gradient-based optimization. In particular, explicitly modifying gradient directions, as in PCGrad and CAGrad, may become unreliable when tasks differ significantly, while gradient reweighting methods such as GradNorm and FAMO are less disruptive but remain insufficient. In contrast, STGU achieves more competitive performance by incorporating task scheduling rather than directly modifying gradient directions, suggesting that organizing heterogeneous tasks may be a more effective alternative to explicit gradient manipulation. G2DA further extends this optimization paradigm by first performing gradient-guided cross-modal task partitioning and then constructing modality-specific curricula within each branch. This hierarchical organization reduces both cross-modal interference and intra-branch task conflicts, consistently outperforming STGU across all benchmarks and model architectures. Together, these results suggest that modality-aware optimization better leverages task heterogeneity for Geo-VLM adaptation. Compatibility across model architectures. G2DA consistently improves performance across both general-purpose and remote sensing-specific vision-language models, demonstrating good architectural compatibility. The gains are particularly pronounced on InternVL3 and GeoLLaVA, both of which adopt relatively decoupled vision-language architectures that better align with the proposed cross-modal optimization strategy. In contrast, improvements on Qwen3.5-VL are generally smaller but remain consistent across all model scales, indicating that G2DA does not depend on a specific backbone design. Furthermore, the consistent improvements observed on GeoChat and GeoLLaVA suggest that the proposed optimization framework generalizes beyond general-purpose VLMs to models specifically designed for remote sensing applications. Benchmark-dependent behavior. The performance gains of G2DA vary across benchmarks, likely reflecting differences in benchmark characteristics. The largest improvements are observed on XLRS-Bench-Lite, which contains higher-resolution imagery and more fine-grained reasoning tasks. By comparison, VRS-Bench shows more moderate gains, possibly because its lower-resolution imagery and recognition-oriented tasks present a comparatively simpler optimization problem. UrBench also exhibits substantial improvements, likely due to the optimization heterogeneity arising from diverse street-view, cross-view, and remote-sensing tasks. These observations suggest that G2DA is more beneficial when visual representation learning or task heterogeneity becomes more challenging. Task-aware analysis. We further examine per-task performance on UrBench-MCQ in Table 2 by grouping tasks according to the vision- and language-oriented partition. Compared with joint training, G2DA consistently improves nearly all tasks, with only marginal performance drops on Qwen3.5-VL-27B for City Retrieval (−0.2-0.2) and Image Retrieval (−0.3-0.3), possibly because these retrieval tasks are particularly challenging under limited training data and rely more heavily on the pretrained language model’s semantic representations (Zhou et al. 2025). The improvements are well balanced across both modalities, with Qwen3.5-VL-27B achieving gains of +3.7 and +3.4 percentage points on vision- and language-oriented tasks, respectively, and GeoLLaVA-8B showing comparable improvements of +2.9 and +3.6 percentage points. This balanced improvement suggests that G2DA effectively mitigates cross-modal optimization interference without introducing modality bias. Furthermore, the largest gains are consistently observed on reasoning-intensive tasks, including Scene Recognition, Road Understanding, and Counting, which exhibit the strongest gradient conflicts before decoupling (Fig. 1) and remain among the most conflicting tasks even after decoupling (Fig. 2), indicating that G2DA is particularly beneficial for tasks with persistent optimization conflicts. Gradient interaction after training. We further evaluate gradient conflict alleviation after training. We measure the conflict rate as the proportion of task pairs with negative cosine similarity, Rconflict=2N(N−1)∑i<j(Sij<0),R_conflict= 2N(N-1) _i<jI(S_ij<0), (9) where (⋅)I(·) is the indicator function. A lower conflict rate indicates fewer conflicting optimization directions across tasks. The results are listed in Table 3. Across all benchmarks, G2DA consistently increases the average pairwise gradient cosine similarity while reducing the proportion of conflicting task pairs, although the degree of improvement varies. Compared with joint training, gradient conflicts are reduced by up to 6.0 percentage points, while gradient alignment increases by 5.1-13.1%\%. The largest improvements are observed on UrBench-MCQ, likely due to its diverse cross-view and multi-view tasks with greater optimization heterogeneity. In contrast, VRS-Bench-VQA shows lower gradient conflicts and smaller gains, likely because its predominantly recognition-oriented tasks and lower-resolution imagery yield a more compatible optimization landscape, leaving less room for improvement. Table 3: Gradient interaction statistics after training. Higher cosine similarity and lower conflict rate (%\%) indicate less gradient interference. UrBench-MCQ XLRS-Bench-Lite VRS-Bench-VQA Method Cos.↑ Conf.↓ Cos.↑ Conf.↓ Cos.↑ Conf.↓ Joint 0.0191 47.3 0.0221 43.0 0.0236 41.2 G2DA 0.0216 41.3 0.0240 41.1 0.0248 40.5 Ablation Study We conduct an ablation study to evaluate the contribution of the three key components in G2DA: gradient-guided cross-modal Decoupling (D), gradient-aware Curriculum (C), and bidirectional Rehearsal (R). Table 4 summarizes the results on two benchmarks using two representative model architectures. Gradient-guided task decoupling consistently provides the largest individual improvement over joint multi-task training, with gains of +4.1/+1.8 p on InternVL3-8B and +2.6/+1.9 p on GeoLLaVA-8B for UrBench/XLRS, respectively. This demonstrates that separating tasks according to their dominant optimization modality effectively alleviates cross-modal optimization interference. Building upon decoupling, gradient-aware curriculum scheduling further improves performance on the more challenging XLRS benchmark (+0.8 and +0.5 p for InternVL3-8B and GeoLLaVA-8B, respectively), indicating that ordering tasks according to gradient compatibility is particularly beneficial under stronger optimization heterogeneity. On UrBench, however, curriculum scheduling yields a slight performance drop compared with decoupling alone (56.0→ 54.9 and 53.3→ 52.1), suggesting that the benefit of gradient-aware ordering may be partially offset by the recency effects associated with sequential training. Finally, incorporating bidirectional rehearsal consistently recovers and further improves performance, achieving the best results across all benchmark–model combinations. These results demonstrate that the three components are complementary: decoupling mitigates cross-modal interference, curriculum exploits intra-modal gradient compatibility, and bidirectional rehearsal alleviates the recency effects introduced by sequential optimization. Table 4: Component ablation of G2DA (accuracy, %\%). Method Components InternVL3-8B GeoLLaVA-8B D C R UrBench XLRS UrBench XLRS Joint 51.9 38.1 50.7 43.8 + Decoupling ✓ 56.0 39.9 53.3 45.7 + Curriculum ✓ ✓ 54.9 40.7 52.1 46.2 G2DA ✓ ✓ ✓ 57.8 42.2 54.8 47.8 Conclusion This paper investigated the optimization challenges of multi-task Geo-VLM learning from a gradient-centric perspective. Our empirical analysis revealed that heterogeneous geospatial tasks exhibit substantial optimization conflicts, including inter-task gradient interference, distinct cross-modal optimization preferences, and persistent intra-encoder gradient heterogeneity. To address these challenges, we proposed G2DA, which integrates gradient-guided cross-modal decoupling, branch-specific curriculum scheduling, and bidirectional rehearsal into a unified optimization framework. Extensive experiments on three Geo-VLM benchmarks using six general-purpose VLMs and two representative Geo-VLMs demonstrate that G2DA consistently outperforms representative multi-task optimization baselines across all 24 benchmark–model settings under identical optimization budgets. These results indicate that explicitly organizing optimization according to task-level gradient characteristics effectively alleviates optimization interference and provides an effective optimization paradigm for multi-task Geo-VLM learning. References Bai et al. (2023) J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609. Cited by: Geospatial Vision-Language Models. Bengio et al. (2009) Y. Bengio, J. Louradour, R. Collobert, and J. Weston Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, p. 41–48. Cited by: Curriculum and Rehearsal Learning. Chen et al. (2018) Z. Chen, V. Badrinarayanan, C. Lee, and A. Rabinovich Gradnorm: gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, p. 794–803. Cited by: Introduction, Gradient Conflict in Multi-task Learning, Models and baselines.. Chen et al. (2024) Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. Intern vl: scaling up vision foundation models and aligning for generic visual-linguistic tasks. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 24185–24198. Cited by: Geospatial Vision-Language Models. Deng et al. (2025) J. Deng, Q. Wu, P. Ju, S. Lin, Y. Liang, and N. Shroff Unlocking the power of rehearsal in continual learning: a theoretical perspective. In International Conference on Machine Learning, p. 13277–13326. Cited by: Curriculum and Rehearsal Learning. Feng et al. (2025) J. Feng, S. Wang, T. Liu, Y. Xi, and Y. Li UrbanLLaVA: a multi-modal large language model for urban intelligence. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 6209–6219. Cited by: Introduction, Geospatial Vision-Language Models. Jeong and Yoon (2025) W. Jeong and K. J. Yoon SELECTIVE task group updates for multi-task optimization. In 13th International Conference on Learning Representations, ICLR 2025, p. 31681–31714. Cited by: Introduction, Gradient Conflict in Multi-task Learning, Models and baselines.. Kemker et al. (2018) R. Kemker, M. McClure, A. Abitino, T. Hayes, and C. Kanan Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32. Cited by: Optimization. Kuckreja et al. (2024) K. Kuckreja, M. S. Danish, M. Naseer, A. Das, S. Khan, and F. S. Khan Geochat: grounded large vision-language model for remote sensing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 27831–27840. Cited by: Introduction, Geospatial Vision-Language Models. Li et al. (2024) X. Li, J. Ding, and M. Elhoseiny Vrsbench: a versatile vision-language benchmark dataset for remote sensing image understanding. Advances in Neural Information Processing Systems 37, p. 3229–3242. Cited by: Geospatial Vision-Language Models. Liu et al. (2023a) B. Liu, Y. Feng, P. Stone, and Q. Liu Famo: fast adaptive multitask optimization. Advances in Neural Information Processing Systems 36, p. 57226–57243. Cited by: Introduction, Gradient Conflict in Multi-task Learning, Models and baselines.. Liu et al. (2021) B. Liu, X. Liu, X. Jin, P. Stone, and Q. Liu Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems 34, p. 18878–18890. Cited by: Gradient Conflict in Multi-task Learning, Models and baselines.. Liu et al. (2023b) H. Liu, C. Li, Q. Wu, and Y. J. Lee Visual instruction tuning. Advances in neural information processing systems 36, p. 34892–34916. Cited by: Geospatial Vision-Language Models. Mai et al. (2024) G. Mai, W. Huang, J. Sun, S. Song, D. Mishra, N. Liu, S. Gao, T. Liu, G. Cong, Y. Hu, et al. On the opportunities and challenges of foundation models for geoai (vision paper). ACM Transactions on Spatial Algorithms and Systems 10 (2), p. 1–46. Cited by: Introduction. Wang et al. (2026) F. Wang, M. Chen, Y. Li, D. Wang, H. Wang, Z. Guo, Z. Wang, S. Boqi, L. Lan, Y. Wang, et al. Geollava-8k: scaling remote-sensing multimodal large language models to 8k resolution. Advances in Neural Information Processing Systems 38, p. 159185–159218. Cited by: Introduction, Geospatial Vision-Language Models. Wang et al. (2025) F. Wang, H. Wang, Z. Guo, D. Wang, Y. Wang, M. Chen, Q. Ma, L. Lan, W. Yang, J. Zhang, et al. Xlrs-bench: could your multimodal llms understand extremely large ultra-high-resolution remote sensing imagery?. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 14325–14336. Cited by: Geospatial Vision-Language Models. Wang et al. (2021) X. Wang, Y. Chen, and W. Zhu A survey on curriculum learning. IEEE transactions on pattern analysis and machine intelligence 44 (9), p. 4555–4576. Cited by: Curriculum and Rehearsal Learning. Yu et al. (2020) T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn Gradient surgery for multi-task learning. Advances in neural information processing systems 33, p. 5824–5836. Cited by: Introduction, Gradient Conflict in Multi-task Learning, Models and baselines.. Zhou et al. (2025) B. Zhou, H. Yang, D. Chen, J. Ye, T. Bai, J. Yu, S. Zhang, D. Lin, C. He, and W. Li Urbench: a comprehensive benchmark for evaluating large multimodal models in multi-view urban scenarios. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 10707–10715. Cited by: Introduction, Geospatial Vision-Language Models, Gradient Characterization, Task-aware analysis..