Paper deep dive
CLIVO: Differentiable Cluster-Based Instruction Mixing with Learned Validation Objectives
Yuhan Wang, Rui Hu, Batian Di, Tan Jiang, Dongsheng Ma, Leichao Dong, Yuanzi Li, Yalin Feng
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Target-aware multimodal instruction selection requires useful examples and an appropriate balance of supervision. Sample-level rankings can overrepresent similar content without optimizing this balance; group-level methods face coarse units, predefined set-level validation signals, and costly or inaccurate mixture search. We propose CLIVO, a target-aware framework connecting fine-grained semantic allocation, automatically learned validation objectives, and differentiable mixture search. Specifically, semantic clusters provide independently adjustable shares at the hundred-cluster scale. Correlations between per-example merged-proxy losses and real probe performance identify target-relevant validation examples; learned nonnegative weights combine their losses into an objective. A single backward pass through the merged proxy supplies gradients for all shares, avoiding an additional mixture-response regression model. Selecting 100K instructions from an 8M pool, CLIVO attains 97.2% of the full-data four-target average score with only 1.25% of the data. It improves on the strongest evaluated selection baseline by 1.16 points using random within-cluster sampling; combining learned quotas with sample ranking increases this advantage to 1.55 points. Analyses show that finer units benefit most targets, short-trained experts preserve mixture responses, sample weighting retains useful validation information, and direct search consistently reduces the learned objective. CLIVO also supports interpretable allocation, modular selection, and reuse across targets and granularities.
Tags
Links
- Source: https://arxiv.org/abs/2610.16876v1
- Canonical: https://arxiv.org/abs/2610.16876v1
Trouble viewing inline? Open PDF directly →
Full Text
85,828 characters extracted from source content.
Expand or collapse full text
CLIVO: DIFFERENTIABLE CLUSTER-BASED INSTRUCTION MIXING WITH LEARNED VALIDATION OBJECTIVES Yuhan Wang 1 Rui Hu 1 Batian Di 2 Tan Jiang 3 Dongsheng Ma 1 Leichao Dong 4 Yuanzi Li 2 Yalin Feng 5 1 Peking University 2 Renmin University of China 3 Zhejiang University 4 Xi’an Jiaotong University 5 Nanyang Technological University ABSTRACT Target-aware multimodal instruction selection requires useful examples and an appropriate balance of supervision. Sample-level rankings can overrepresent similar content without optimizing this balance; group-level methods face coarse units, predefined set-level validation signals, and costly or inaccurate mixture search. We propose CLIVO, a target-aware framework connecting fine-grained semantic allocation, automatically learned validation objectives, and differentiable mixture search. Specifically, semantic clusters provide independently adjustable shares at the hundred-cluster scale. Correlations between per-example merged-proxy losses and real probe performance identify target-relevant validation examples; learned nonnegative weights combine their losses into an objective. A single backward pass through the merged proxy supplies gradients for all shares, avoiding an additional mixture-response regression model. Selecting 100K instructions from an 8M pool, CLIVO attains 97.2% of the full-data four-target average score with only 1.25% of the data. It improves on the strongest evaluated selection baseline by 1.16 points using random within-cluster sampling; combining learned quotas with sample ranking increases this advantage to 1.55 points. Analyses show that finer units benefit most targets, short-trained experts preserve mixture responses, sample weighting retains useful validation information, and direct search consistently reduces the learned objective. CLIVO also supports interpretable allocation, modular selection, and reuse across targets and granularities. 1INTRODUCTION Vision-language models (VLMs) have demonstrated strong capabilities in visual understanding, document parsing, multimodal reasoning, and interactive tasks (Comanici et al., 2025; Wang et al., 2025; Bai et al., 2025a; Hong et al., 2025; Huang et al., 2026; Team et al., 2026). Their development combines multimodal representation pretraining with instruction-following post-training (Yin et al., 2025; Team et al., 2025b;a; Radhakrishna et al., 2025; Lin et al., 2024). Multimodal instruction tuning develops instruction following through instruction–response examples about visual inputs (Liu et al., 2023a; Dai et al., 2023; Du et al., 2025). The training data therefore shape both the supervision a model receives and the tasks it practices. Effective tuning depends on supervision quality (Luo et al., 2025) as well as the composition of the training set (Zhang et al., 2025). Large instruction pools offer broad coverage of tasks and response formats (Xu et al., 2024), yet this breadth does not ensure that every example is useful: incorrect responses and hallucinated content can degrade model performance (Gou et al., 2025). This motivates data selection, which concentrates training on useful supervision. Prior work shows that carefully selected subsets can match full-data performance and even surpass it on individual benchmarks (Liu et al., 2025). Compact subsets also support repeated adaptation to different applications. A document assistant, for example, needs a different balance of text reading, visual recognition, and reasoning from a general visual assistant. Among useful examples, relative abundance determines how much training each type of supervision receives. Selection must therefore address both which examples to retain and how to balance their contribution to the intended capability. At a fixed training-data budget, we ask: 1 arXiv:2610.16876v1 [cs.AI] 12 Oct 2026 How should instruction data be mixed to best support a target capability? Existing data selection methods operate at the sample or group level. Sample-level methods rank examples using signals such as gradient influence in ICONS (Wu et al., 2024), necessity in MLLM- Selector (Ma et al., 2025), and importance and diversity in CoIDO (Yan et al., 2026). However, selecting the highest-scoring examples can overrepresent similar content, reducing diversity and degrading downstream performance (He et al., 2025). Diversity-aware scores help, but often leave content shares implicit. Group-level methods explicitly allocate shares across sources, domains, or semantic groups, requiring choices of allocation unit, evaluation signal, and mixture learning rule. Unit: fine-grained allocation remains difficult to search. Predefined source shares tie heteroge- neous instructions together. Semantic clusters offer finer control, but COINCIDE’s density and trans- ferability heuristics (Lee et al., 2024) do not optimize measured target performance. CLIMB (Diao et al., 2025) uses downstream feedback to search 21 groups in its main setting, with sampling costs limiting substantially finer search. Signal: validation relies on predefined sets and coarse loss averages. Costly candidate training motivates proxy validation losses. CLIMB uses predefined task sets, while CAMEL (Li et al., 2026a) maps validation losses to benchmark scores. Neither automatically selects target-relevant examples, and their set-level averages can obscure informative differences between individual responses. Learning: search is costly and susceptible to response-prediction error. DoReMi (Xie et al., 2023a) couples reweighting to proxy training, limiting reuse across targets and groupings. Reg- Mix (Liu et al., 2024b), DeMix (Li et al., 2026c), and MergeMix (Wang et al., 2026) instead fit response predictors from evaluated mixtures. High-dimensional candidate labeling is costly, and selecting predicted optima can amplify errors on unseen mixtures (Smith & Winkler, 2006). We propose CLIVO (CLuster-based Instruction Mixing with Learned Validation Objectives) to jointly address these challenges. Specifically, fine-grained semantic units provide adjustable shares at the hundred-cluster scale. A learned validation signal selects examples using proxy-loss correlations with real probe scores and fits nonnegative weights to aggregate their losses. Expert merging supplies proxies without retraining for each mixture (Berasi et al., 2026). Finally, gradient- based mixture learning differentiates this objective through fixed experts to update all shares jointly, without an additional mixture-response predictor. Using 100K instructions (1.25% of the pool) and random within-cluster sampling, CLIVO attains 97.2% of the full-data four-target average score. It improves on the strongest evaluated data selection baseline at this budget by 1.16 points; within-quota ranking increases the advantage to 1.55 points. Our analyses show that finer units benefit most targets, moderate-budget experts and sample weighting provide reliable feedback while preserving individual responses, and direct gradient search lowers every target’s learned objective. Importantly, CLIVO also supports interpretable, flexible, and reusable data curation. Semantic visualizations expose content choices; modular encoders, objectives, and selectors accommodate different applications. With the base and partition fixed, experts, probes, and cached losses support new targets. A single backward pass supplies gradients for all shares even at finer granularities, avoiding an additional higher-dimensional regression fit. Our contributions are: •A unified target-aware framework. We connect fine-grained semantic allocation, auto- matic validation-example selection and weighting, and differentiable mixture search through reusable expert proxies. • Evidence for fine-grained allocation, reliable signals, and direct search. We demonstrate benefits from finer units on most targets, reliable feedback from moderate-budget experts and individual validation responses, and objective improvement through direct search for every target. •Interpretable, flexible, and reusable curation. We make allocations interpretable through visualization, demonstrate flexible combinations with sample selectors, and reuse experts and feedback across targets and granularities. 2 backprop repeat Start mixture Multimodal instruction pool TXT · Multimodal embedding & clustering · K-means · 1.3 Region expert training θ₀ base model θ₁θ₂θ₃θ k · Per-region training data Proxy instantiation · Mixture weights w₁ w₂ w₃ · w K Merged proxy 123 Unit Partition data and train region experts Signal Learn a target-specific validation objective from real training feedback Learning Optimize the mixture and train the final model Held-out validation examples 퐀 · each clusters sample equal examples Probe mixtures probe mixtures · Merged-proxy losses p r o b e s Real probe scores y (d) 1 y (d) 2 y (d) 3 y (d) 128 · Screening for global and upper-range agreement Example vⱼ Spearman GlobalTop-half · · · Nonnegative ridge on the selected examples Selected examples 퐀 퐀 · Learned weights Learned target-specific validation objective J d (w) = ∑ j ∈ S d α d,j ℓ(θ(w); v j ) . Alignment (from real outcomes) Merged proxy for evaluation (fixed) Differentiable search (through fixed proxy) Initialization from feasible historical probes Feasible candidates · Periodic evaluation and selection Best mixture Sample data and train final model Sample training set Draw instructions from each region according to 퐀 ∗ Train from original checkpoint 퐀 0 Supervised training Target-adapted model ∆ 퐀 ∆ 퐀 =퐀 퐀 −퐀 퐀 ∆ 퐀 · ∆ 퐀 ∆ 퐀 ∆ 퐀 =퐀 퐀 −퐀 퐀 ∆ 퐀 =퐀 퐀 −퐀 퐀 ∆ 퐀 =퐀 퐀 −퐀 퐀 × 퐀 (퐀) Score v₁ 0.82 0.780.80 v₂ 0.630.670.65 v₃ 0.370.290.33 v₁₀,₀ 0.120.140.13 · select examples by scores: 퐀 퐀 퐀 퐀,퐀 퐀 퐀,퐀 퐀 퐀,퐀 ...퐀 퐀,퐀 Fit nonnegative example weights Lowest validation objective examples Full-support gradients and projected Adam Mixture weights Merged proxy Validation objective Full-support gradients Adam update Updated mixture weights Figure 1: CLIVO overview: learning what to train on and how to evaluate the mixture. (1) Unit: a fixed multimodal encoder andK-means organize the instruction pool into semantic clusters with adjustable training shares. (2) Signal: probe models are trained on sampled mixtures and evaluated on downstream benchmarks. Cluster experts are merged using the same shares to construct proxies. Correlations between per-example proxy losses and real scores identify target-relevant validation examples; learned nonnegative weights combine their losses into an objective. (3) Learning: with experts and validation weights frozen, gradients update the training shares through the merged proxy under budget and capacity constraints. Periodic evaluations select the lowest-loss mixture. Its quotas specify how many examples to sample from each cluster for training from the original base. 2METHOD Framework overview.CLIVO learns target-specific training shares through three decisions: how to group the data, how to evaluate a mixture, and how to update its shares (Figure 1 and Algorithm 1). We define the task (Section 2.1), then present semantic units (Section 2.2), validation learning (Section 2.3), and gradient search (Section 2.4). 2.1PROBLEM SETUP Given an instruction poolD, an initial VLMθ 0 , a target capabilityd, and a budgetN, we seek a size- Nsubset maximizing the target score after instruction tuning. For a partitionD k K k=1 , we optimize admissible training sharesw = (w 1 ,...,w K ) ∈ W, withw k ≥ 0and P k w k = 1 . These shares determine sample countsn k (w) ≈ Nw k totalingN. We fix the within-group ruleπ U to uniform sampling without replacement, yieldingS(w,π U ) = S k π U (D k ,n k (w)). Letθ S = Train(θ 0 ,S) denote the model trained with a fixed recipe and R d (θ) its target score. Our objective is w ⋆ d ∈ arg max w∈W F d (w;π U ), F d (w;π U ) =E S [R d (θ S )].(1) The expectation accounts for score variation across random subsets drawn with the same shares. 2.2UNIT: SEMANTIC ALLOCATION Predefined groups rely on source labels or hand-crafted task rules. Semantic clustering instead provides unsupervised allocation units whose granularity is controlled by the number of clusters. We therefore encode instructions with a fixed external multimodal encoder E, independent of the VLM being tuned, and applyK-means (McQueen, 1967) to obtainD k K k=1 . Distinct content within a source can receive different shares, while related content across sources can be allocated together. For nested partitions, independent fine-cluster shares retain every coarse allocation and admit additional compositions, expanding the optimization space. The best attainable target score is therefore no lower and can improve when the additional freedom admits a better mixture (Section A.1). This flexibility also creates a higher-dimensional search problem, making candidate-by-candidate evaluation costly. 3 We address this with a learned validation objective on reusable proxies (Section 2.3) and gradients that update all shares jointly (Section 2.4). 2.3SIGNAL: LEARNING THE VALIDATION OBJECTIVE Algorithm 1 CLIVO: target-specific instruction mixing Require: D,θ 0 ,E,R d ,N ; K,m,T,w i Y i=1 Ensure: Shares b w d ; subsetS ⋆ d 1. Unit: semantic allocation EncodeD with fixed external encoder E. Apply K-means; obtain clustersD k . 2. Signal: learned validation objective Train probes atw i ; evaluate y (d) . Train cluster expertsθ k from θ 0 . V ← BalancedSample(D k ) X ij ←−ℓ( e θ(w i );v j ) S d ← Top m ((ρ all,j + ρ upper half,j )/2) Fit Eq. (3); rescale and normalize toα d . 3. Learning: mixture search Freezeθ k ,α d . w 0 ← arg min w i ∈W J d (w i ); b w d ← w 0 for t = 1,...,T do g t ←∇ w J d (w t−1 ) by Eq. (5) w t ← Π W (Adam(w t−1 , g t )) Update best b w d at checkpoints. end for Convert b w d to sample counts n totaling N . S ⋆ d ← S k π U (D k ,n k ) Real-training feedback for validation selection. We trainYprobe models fromθ 0 using mixturesw i . Uniform within-cluster sampling fixes the selec- tion rule so that the probes test changes in composition (Section A.2). Their scoresy (d) i supervise validation selec- tion and weighting. Proxy losses from merged experts. Training a model for every candidate mixture is costly.Following prior work (Li et al., 2026c; Wang et al., 2026), we approximate its losses by merging cluster expertsθ k trained from θ 0 . The candidate’s shares determine how these fixed experts are combined. With∆ k = θ k − θ 0 , constructing a proxy requires no further training: e θ(w) = θ 0 + K X k=1 w k ∆ k = K X k=1 w k θ k .(2) We form a diverse validation candidate poolV =v j H j=1 by uniformly sampling equal numbers of examples from every cluster. For theYreal-probe mixtures, we construct proxies and compute each candidate example’s negative loss,X ij =−ℓ( e θ(w i );v j ), whereℓis the mean cross-entropy over its answer tokens. We use these losses and real scores to identify target-relevant validation examples. Downstream-aligned weighted signals. Search favors low-loss mixtures, so the signal should distinguish strong candidates while ranking the full candidate set. For each example, we average two Spearman correlations between negative loss and target score: one over all fitting probes and one over the half with the highest real scores. Since examples greatly outnumber probes, we retain the topmexamples asS d to limit the number of fitted coefficients (Fan & Lv, 2008). To rank mixtures, we fit centered target scores with nonnegative ridge regression (Hoerl & Kennard, 1970): b β d = arg min β≥0 e X I,S d β− (y (d) I − ̄y d 1) 2 2 + λ∥β∥ 2 2 .(3) HereIindexes all fitting probes, e X contains their standardized negative losses, ̄y d is their mean target score, andλregularizes the coefficients. Nonnegative coefficients make lower example losses contribute to higher predicted scores. Rescaling the coefficients to raw-loss units and normalizing them gives weights α d,j ≥ 0 summing to one and the objective J d (w) =L d ( e θ(w)) = X j∈S d α d,j ℓ( e θ(w);v j ).(4) Individual weights preserve differences in how examples’ losses respond to a mixture, which cluster averaging would remove before fitting. LowerJ d corresponds to a higher predicted score, so the conversion preserves the fitted ranking. The validation weightsα d determine how mixtures are evaluated; the training shareswdetermine their composition. Detailed settings and the weight conversion are given in Section A.3. 4 2.4LEARNING: GRADIENT-BASED MIXTURE SEARCH The differentiable objective above enables direct, efficient gradient search over the mixture shares. We freeze the expertsθ k and validation weightsα d , and initializew 0 with the lowest-loss previously evaluated feasible mixture to accelerate convergence. At each step, all positive-weight validation examples contribute to the merged proxy’s loss. Applying the chain rule (Baydin et al., 2018) gives ∂J d ∂w k = D ∇ θ L d (θ)| θ= e θ(w) , ∆ k E .(5) Taking inner products of the accumulated parameter gradient with allKexpert differences gives the gradients of every share. Adam updates the shares jointly while the experts remain fixed, so no separate backward pass is needed for each coordinate. Periodic evaluations of the full objective select the lowest-loss iterate b w d among the evaluated checkpoints, including the start. Finally, we convert the shares into integer counts totalingN, sample uniformly within clusters, and train from θ 0 onS( b w d ,π U ). Section B.3 gives search and sampling details. 3EXPERIMENTS In this section, we describe the experimental setup (Section 3.1), compare fixed-budget selection quality (Section 3.2), and conduct ablation studies of CLIVO’s components (Section 3.3). 3.1EXPERIMENTAL SETUP Model and evaluation. We build a 7B VLM with a Qwen2.5-VL-style architecture (Bai et al., 2025b), following the multimodal training approach of LLaVA-1.5 (Liu et al., 2024a). Alignment and mid-training produce the common baseθ 0 for all experts, probes, and final models; construction details are in Section B.1. Evaluation covers General visual understanding (G: MMBench-EN, MMStar, GQA), Parsing (P: OCRBench, ChartQA), and Knowledge and text-centric reasoning (K: MMLU, MMLU-Pro). Each domain’s benchmark mean defines a target; Overall (O) averages all seven benchmarks. We learn one mixture per target, and Average summarizes their four scores. Evaluation protocols are in Section B.3. Data pool. We collect approximately 40M instructions from public datasets and their derivatives, covering diverse visual content, tasks, and response formats. To keep selection costs manageable, experiments use an 8M candidate pool. All selection methods draw 100K instructions from this pool; Full uses all 8M. Section B.2 provides data statistics. Baselines. For sample-level selection, we choose PPL (Jelinek et al., 1977), CLIP-Score (Hessel et al., 2021), M-Selector (Ma et al., 2025), PRISM (Bi et al., 2025), ICONS (Wu et al., 2024), DSIR (Xie et al., 2023b), and DataTailor (Yu et al., 2025). For group-level selection, we choose COINCIDE (Lee et al., 2024) and SemDeDup (Abbas et al., 2021). We also evaluate two controlled regression variants following mixture-response search (Liu et al., 2024b; Li et al., 2026c): Acc + Reg. predicts target accuracy from semantic-cluster shares, whereas Source + Reg. searches source-defined shares using a proxy-loss predictor. Random samples 100K examples uniformly; Full uses all 8M. Because ICONS and DSIR require validation or reference data, both receive CLIVO’s selected validation examples for a fair comparison with common reference content. CLIVO samples randomly within clusters by default. To test compatibility with sample ranking, CLIVO + ICONS applies the strong ICONS selector within unchanged quotas. Random, Acc + Reg., Source + Reg., and CLIVO report three-run means; others use one run (Sections B.4 and B.5). Implementation details.We embed instructions with Qwen3-VL-Embedding-8B (Li et al., 2026b), useK = 100clusters, and train experts with the nominal step-10 budget. Validation learning uses Y = 128real probes andH = 10,000candidates sampled equally across clusters, retainingm = 64 examples per target and fitting their weights withλ = 10. We run 120 gradient updates, retain the best periodically evaluated mixture, and train its 100K-example subset for 100 optimizer steps. Budget definitions, optimizer settings, and evaluation protocols are in Section B. 5 3.2TARGET-SPECIFIC SELECTION QUALITY Table 1: Selection quality at a 100K budget. O/G/P/K: four targets; Average: their mean. Random, Acc + Reg., Source + Reg., and CLIVO average three runs. Bold/underline: best/second-best at 100K. MethodOGPKAverage Full (8M)71.0169.2380.8763.8171.23 Random66.6163.5777.7360.0566.99 PPL64.2660.7875.6558.0964.70 CLIP-Score66.1662.2379.4258.8066.65 M-Selector66.4664.0576.9659.5966.77 PRISM67.2765.7376.9759.8667.46 ICONS67.3264.1279.3059.8867.65 DSIR67.3167.56 72.6260.4266.98 DataTailor65.7261.5077.5860.2066.25 SemDeDup64.0057.5377.8759.8464.81 COINCIDE66.2062.6578.1459.6066.65 Acc + Reg.67.5365.4477.8061.4068.04 Source + Reg.67.0062.9179.1259.8067.21 CLIVO68.1367.2879.6461.7669.20 CLIVO + ICONS68.0568.2779.9662.0969.59 CLIVO achieves competitive fixed- budget selection. With random within- cluster sampling, CLIVO reaches a four-target mean of 69.20, exceeding the strongest baseline, Acc + Reg., by 1.16 points and the strongest sample- level selector, ICONS, by 1.55 points (Table 1). It also surpasses Source + Reg. by 1.99 points. CLIVO leads the baselines on Overall, Parsing, and Knowledge with scores of 68.13, 79.64, and 61.76, showing the value of target- specific semantic allocation without ranking individual training examples. Baselines favor different capabilities: CLIP-Score, which measures image– text alignment, achieves the highest baseline Parsing score (79.42); DSIR, which matches a reference distribution, achieves the highest baseline General score (67.56). ICONS and DSIR also obtain competitive results using CLIVO’s selected validation examples, showing that this reference data can support existing selectors. Benchmark-level results are in Section B.6. Allocation and ranking are complementary. Keeping CLIVO’s quotas fixed and replacing random sampling with within-cluster ICONS improves most targets, raising the four-target mean from 69.20 to 69.59 and General by 0.99 points. The combination also exceeds global ICONS on all four targets, with gains of 4.15 points on General and 1.94 on the four-target mean. Thus, the same ranking method is more effective when it fills target-specific quotas: allocation determines how much of each content type to include, and ranking chooses its representatives. Global sample ranking alone leaves these composition choices implicit (Section C.2). 3.3ABLATION STUDIES We examine three questions: whether finer allocation improves selection, whether proxy losses provide reliable target feedback, and whether gradient search uses that feedback to improve mixtures. The value of finer allocation depends on the target. We isolate allocation freedom by tying the same 100 expert coefficients into 20 or 50 parent shares, or optimizing all 100 independently. The proxy bank, validation objective, starting composition, and training budgets stay fixed. Tying is algebraically equivalent to premerging child experts at fixed ratios (Section B.7). 2050100 66.8 67.0 67.2 Target score 67.20 66.83 67.02 General 2050100 79.8 80.0 80.2 79.81 80.12 80.26 Parsing 2050100 61.4 61.6 61.8 61.42 61.57 61.83 Knowledge 2050100 67.7 67.9 68.1 67.99 67.85 68.02 Overall Adjustable allocation shares Figure 2: Allocation freedom and downstream performance. Each panel varies adjustable shares on the same 100-expert bank. Scores use 100K examples, random within-cluster sampling, and one training seed. All vertical axes span 0.6 points. Search records: Table 10. In Figure 2, 100 adjustable shares give the highest scores on most targets. Parsing and Knowledge improve by 0.45 and 0.41 points over 20 shares, whereas General performs best at 20. Finer allocation is useful when a target benefits from separating content within coarse groups; the preferred granularity therefore depends on the target. 6 0.901.20 1.25 1.30 1.35 Mean proxy loss Step 1 ρ=0.485 0.901.20 1.00 1.10 Step 5 ρ=0.647 0.901.20 0.96 1.04 Step 10 ρ=0.730 0.901.20 0.90 0.96 1.02 Full schedule ρ=0.644 Mean real-model loss Figure 3: Expert budget and proxy–real loss agreement. Each point compares mean proxy and real-model losses for the same mixture on a shared 10,000-example panel. Panels show different expert budgets; ρ is Spearman correlation across 128 mixtures. Budget details: Section B.8. Moderate-budget experts capture real-model loss variation. To test proxy reliability, we compare expert checkpoints indexed byb∈1, 5, 10and a full-training reference on identical mixtures and examples (Figure 3). Proxy–real loss correlation increases from 0.485 to 0.730 across the short budgets, with checkpoint 10 giving the strongest agreement. Full training lowers absolute loss but does not improve this agreement. Checkpoint 10 also improves held-out downstream-score prediction over checkpoint 1 on every target (Table 11). These results support its use as the shared proxy bank; budget definitions and protocols are in Section B.8. Table 2: Validation aggregation. Held-out Spearman over 20 folds, with 64 matched exam- ples. C/S: cluster/sample; E/W: equal/weighted. Maximum correlations are bold. TargetC–EC–WS–ES–W General0.6210.6840.6190.714 Parsing0.3970.4300.3790.461 Knowledge0.5860.5940.5890.596 Overall0.4690.5690.4700.591 Individual responses preserve useful validation in- formation. We compare cluster-level and sample-level aggregation, each with equal or learned weights, on the same 64 examples per fitting fold (Table 2). Cluster- level rules average losses within each represented clus- ter before weighting, while sample-level rules retain each example’s response to mixture changes. Learned sample weights give the highest held-out Spearman on all four targets, improving over equal sample averaging by up to 0.121. Their advantage over weighted cluster means supports retaining individual responses: fitting can emphasize informative examples whose signals would otherwise be diluted by averaging. The fit concentrates weight on a subset of examples (Section A.3), emphasizing their contribution to target evaluation. Table 3: From feedback to final training.K = 100. CV-ρ: held-out Spearman;t ⋆ : best step;D J : loss decrease (%);B: best probe; ̄ F: three-run mean;∆F = ̄ F −B. Protocols: Section B.9. TargetCV-ρ t ⋆ D J B ̄ F∆F Gen.0.7141003.72466.60067.28+0.68 Pars.0.4611200.29379.31079.64+0.33 Know.0.596400.05661.46561.76+0.30 Ovrl.0.5911002.04067.91968.13+0.21 Learned signals guide mixture search. Held-out correlations show that the learned objectives predict downstream rankings for all four tar- gets (Table 3). Gradient search lowers every fixed objective, reducing loss by 0.056–3.724% from the evaluated starts. These signals therefore support direct optimization of the mixture shares. All four final-training means also exceed the best historical probe scores, by 0.21–0.68 points. Section B.9 provides search trajectories and training details. 4INTERPRETABLE, FLEXIBLE, AND REUSABLE DATA CURATION In this section, we show how CLIVO supports interpretable, flexible, and reusable data curation. Semantic visualizations make target-specific selection decisions inspectable (Section 4.1); modular components accommodate different applications, while shared experts and feedback support reuse across targets and allocation granularities (Section 4.2). 4.1DATA VISUALIZATION AND MANAGEMENT Targets favor different, concentrated mixtures. We use GPT-based task annotations to group clusters into coarse capability categories for interpretation (Section C.1). Figure 4 shows distinct 7 a Coarsecapabilityallocation b Within-capability allocation Target (objective) Effective clusters Dominant coarse capability enrichment Dominant cluster allocation (top clusters) Overall 7-benchmark average 4.36 Grounding & Detection 6.19×uniform c60 69.4% c26 6.3% General General visual grounding 3.02 Grounding 6.12×uniform VisualQA 0.58×uniform c60 73.4% c23 6.3% Parsing Structured visual parsing 10.13 Charts & Data 1.91×uniform c27 30.8% c35 21.7% Knowledge Knowledge / text- centric reasoning 3.92 Knowledge & Science 5.94×uniform c36 59.4% c97 13.1% Higher-weight semantic regionLower-weight semantic region General / Grounding & Detection c60 73.41% Short visual QA Q Thebiketotheright ofthecathaswhich color? ABlack c38 0.00% Coordinate grounding Q Pleaseprovidethe boundingbox coordinateofthetop ofamulti-colored surfboard. A[50, 390, 510, 540] Parsing / Charts & Data c27 30.77% Direct value reading QWhatwasSpain's medianagein 1950? A27.5 c14 0.32% Long-form chart response QWhatwasthe unemploymentrate inMassachusettsin 2020? A<SUMMARY> ... <CAPTION> ... <REASONING> ... <CONCLUSION> 8.9 Knowledge / Knowledge & Science c36 59.42% Text-only STEM rules QWhat is the general rule for rounding whenadding or subtracting numbers? A(A)Roundthefinalanswertonot havemoredecimalplacesthanthe leastprecisenumberusedinthe calculation. c20 0.00% Diagram-based mechanics T M Mg g/4 QAcordisusedtolower verticallyablockofmassMa distancedataconstant downwardaccelerationofg/4. A Answer: -3Mgd/4 Figure 4: Target-specific composition, from distributions to examples. (a) Allocation concentration and dominant semantic regions. (b) Paired clusters within the same coarse capability receive different training shares. Percentages give each cluster’s share of the complete training budget. Uniform allocates 1% to each of the 100 clusters;×uniform compares a category’s total learned share with its share under this equal-cluster allocation. Definitions and cases: Section C.1. preferences: General and Overall emphasize c60, illustrated by short visual questions; Parsing favors c27, illustrated by chart reading; and Knowledge favors c36, illustrated by text-based STEM supervision. Allocation is concentrated but target-dependent: the entropy-based effective cluster count is 10.13 for Parsing and 3.02–4.36 for the other targets, out of 100 clusters. Labels and examples help interpret these differences (Table 15). Within-category contrasts highlight the value of fine-grained allocation. Clusters within the same coarse capability receive markedly different shares (Figure 4(b)). Within Grounding & Detec- tion, General assigns 73.41% to c60, illustrated by short visual questions, and zero to c38, illustrated by bounding-box coordinates. Within Charts & Data, Parsing assigns 30.77% to c27, illustrated by direct value reading, and 0.32% to c14, illustrated by long responses. Within Knowledge & Science, Knowledge assigns 59.42% to c36, illustrated by text-only STEM tasks, and zero to c20, illustrated by mechanics diagrams. These contrasts reveal target preferences within broad categories, helping explain why independently adjustable fine-grained shares can be useful (Figure 2). 4.2FLEXIBLE AND REUSABLE DATA CURATION Modular components enable flexible selection. CLIVO separates semantic organization, target evaluation, and within-cluster selection, so each component can be configured for the application. Its quotas also integrate with state-of-the-art sample-level selectors: allocation determines how much of each content type to include, and ranking chooses the examples within those quotas. The CLIVO + ICONS results demonstrate this combination (Section 3.2), allowing existing ranking methods to benefit from target-specific composition. One-time construction supports new targets. After constructing the embeddings, semantic partition, expert bank, real probe models, and candidate-loss cache, CLIVO can reuse these assets for new targets on the same base and partition. Evaluating the saved probes on a new benchmark provides the target scores; validation-example selection and weighting then operate on cached losses, followed by gradient search under the new objective. Thus, target adaptation requires neither retraining experts or probes nor recomputing the cached per-example losses (Section C.3). Shared gradients support finer allocation.As allocation becomes finer on the same expert bank, a single backward pass supplies gradients to update all mixture weights jointly (Eq.(5)). CLIVO thus avoids fitting an additional regression model over the higher-dimensional mixture space. In the K = 20/50/100experiments, costs to reach the best recorded losses show no consistent monotonic increase across targets as allocation becomes finer (Table 10). With the same expert bank and 8 validation objectives, the four gradient searches use 26.6% fewer peak-BF16-normalized GPU-hours than labeling 4,096 mixtures for regression, a budget at which held-out proxy-loss prediction shows diminishing gains. This reuse of model gradients supports fine-grained search without an additional mixture-response fitting stage (Sections C.3 and C.4). 5RELATED WORK Sample-level methods. Sample-level methods score individual examples and select a subset according to those scores. Their signals fall into four broad categories. Quality and alignment scores assess instruction quality and complexity in DEITA (Liu et al., 2023b), human preferences in Align2LLaVA (Huang et al., 2025), and image–text compatibility in CLIP-Score (Hessel et al., 2021). Model-response signals capture necessity in MLLM-Selector and selector confidence in Once-For-All (Dong et al., 2026). Influence-based methods estimate training effects: TIVE combines instance influence with task difficulty, ICONS seeks cross-task consensus, and BIDS (Dai et al., 2025) balances task-dependent influence scales in text instruction tuning. Distribution and diversity signals address coverage: DSIR (Xie et al., 2023b) resamples toward a reference distribution, PRISM (Bi et al., 2025) uses visual-feature geometry, DataTailor (Yu et al., 2025) combines informativeness, uniqueness, and representativeness, and PreSel (Safaei et al., 2025) clusters images before generating instructions. CoIDO jointly models importance and diversity. These categories can overlap with allocation; TIVE and DataTailor also determine task quotas. CLIVO learns target-specific semantic quotas that can be filled using existing sample-level selectors. Group-level methods.Group-level methods partition a pool and determine how much training data to draw from each group. They can be classified by how shares are chosen. Heuristic allocation con- verts group statistics into quotas, as in COINCIDE’s use of concept–skill density and transferability. Online adaptive weighting updates shares during training; DoReMi uses group distributionally robust optimization on reference-relative losses. Response-model-based search fits mixture outcomes, using flexible regressors in RegMix and CLIMB or parametric laws in Data Mixing Laws (Ye et al., 2024), BiMix (Ge et al., 2024), and CAMEL. The latter additionally links validation losses to benchmark scores. Expert reuse changes how these response models obtain feedback: MDE (Belenki et al., 2025) constructs features from combined expert predictions, while DeMix and MergeMix evaluate parameter-merged experts to train LightGBM predictors (Ke et al., 2017). Direct proxy evaluation compares merged candidates without a fitted mixture-response function (Berasi et al., 2026). Related work on adaptive merging optimizes coefficients using entropy minimization to improve the merged model itself (Yang et al., 2024). CLIVO learns a sample-loss objective and differentiates it through fixed experts to optimize training shares. 6CONCLUSION We presented CLIVO, a target-aware instruction-selection framework connecting fine-grained se- mantic allocation, learned validation objectives, and differentiable mixture search. Learning from individual validation losses turns real-training feedback into an objective for optimizing data compo- sition. Experiments show that learned quotas improve fixed-budget selection and complement sample ranking. This distinction between allocating content and ranking examples provides a practical way to adapt a shared instruction pool to different capability needs. The learned objective links these allocation decisions to target performance, while semantic visualization reveals which content is emphasized. Modular components accommodate different encoders and selectors, and shared experts support reuse across targets and granularities. Together, these properties make the selected subset both adaptable and open to inspection. Future work will explore adaptive probe selection to reduce real-training feedback, transfer of learned validation objectives across base models and training budgets, and joint optimization of multiple capability targets. Hierarchical semantic units could also adapt allocation granularity to the task as instruction pools grow, extending the framework to evolving data collections. We hope this work encourages joint study of example quality and data composition for transparent, target-aware curation. 9 AI USE STATEMENT AI tools were used to polish the manuscript’s language and improve readability. GPT also assisted semantic annotation for data visualization; the annotation procedure and human verification are described in Section C.1. The authors independently developed the core research ideas and methodol- ogy, designed and conducted the experiments, and formulated the scientific conclusions. The authors take full responsibility for this work. ETHICS STATEMENT This work uses publicly available datasets and derived instruction data in accordance with their applicable licenses and terms of use. We document the data sources and processing scope in Section B.2. Our experiments do not involve recruiting human participants or collecting new personal information. Data processing and reporting follow applicable privacy requirements, and qualitative examples are presented without personally identifying information. CLIVO is intended to support transparent, target-aware data selection for research. REPRODUCIBILITY STATEMENT Section 2 specifies validation-example screening, nonnegative ridge fitting, weight conversion, expert merging, and mixture search. Section A gives the method derivations and validation objective; Section B documents data, training, baselines, and detailed results. Annotation procedures, example provenance, and reuse accounting appear in Section C. REFERENCES Amro Abbas, Kushal Tirumala, D ́ aniel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication, 2023. URL https://arxiv. org/abs/2303.09540, 2021. Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Didi Zhu, et al. Llava-onevision-1.5: Fully open framework for democratized multimodal training. arXiv preprint arXiv:2509.23661, 2025. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025a. Shuai Bai, Ke qin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. ArXiv, abs/2502.13923, 2025b. URLhttps://api.semanticscholar.org/CorpusID: 276449796. Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. Journal of machine learning research, 18 (153):1–43, 2018. Lior Belenki, Alekh Agarwal, Tianze Shi, and Kristina Toutanova. Optimizing pre-training data mixtures with mixtures of data expert models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 32570–32587, 2025. Davide Berasi, Matteo Farina, Massimiliano Mancini, and Elisa Ricci. Linear model merging unlocks simple and scalable multimodal data mixture optimization. arXiv preprint arXiv:2602.04937, 2026. Jinhe Bi, Yifan Wang, Danqi Yan, Wenke Huang, Zengjie Jin, Xiaowen Ma, Sikuan Yan, Artur Hecker, Mang Ye, Xun Xiao, et al. Prism: Self-pruning intrinsic selection method for training-free multimodal data selection. arXiv preprint arXiv:2502.12119, 2025. 10 Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. Qirun Dai, Dylan Zhang, Jiaqi W Ma, and Hao Peng. Improving influence-based instruction tuning data selection for balanced learning of diverse capabilities. In EMNLP (Findings), p. 7079–7102, 2025. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems, 36:49250–49267, 2023. Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, et al. Climb: Clustering-based iterative data mixture bootstrapping for language model pre-training. arXiv e-prints, p. arXiv–2504, 2025. Mingkang Dong, Hongyi Cai, Xiwen Lei, Jie Li, Tao Zhang, and Muxin Pu. Once-for-all: A train-once and select-anytime framework for multimodal instruction tuning. arXiv preprint arXiv:2605.26761, 2026. Yifan Du, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, Jinpeng Wang, Chuyuan Wang, Mingchen Cai, Ruihua Song, and Ji-Rong Wen. What makes for good visual instructions? synthesizing complex visual reasoning instructions for visual instruction tuning. In Proceedings of the 31st International Conference on Computational Linguistics, p. 8197–8214, 2025. Jianqing Fan and Jinchi Lv. Sure independence screening for ultrahigh dimensional feature space. Journal of the Royal Statistical Society Series B: Statistical Methodology, 70(5):849–911, 2008. Ce Ge, Zhijian Ma, Daoyuan Chen, Yaliang Li, and Bolin Ding. Bimix: A bivariate data mixing law for language model pretraining. arXiv preprint arXiv:2405.14908, 2024. Yunhao Gou, Hansi Yang, Zhili Liu, Kai Chen, Yihan Zeng, Lanqing Hong, Zhenguo Li, Qun Liu, Bo Han, James Kwok, et al. Corrupted but not broken: Understanding and mitigating the negative impacts of corrupted data in visual instruction tuning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 25945–25971, 2025. Hongyi He, Xiao Liu, Zhenghao Lin, Mingni Tang, Yi Cheng, Jintao Wang, Wenjie Li, Peng Cheng, and Yeyun Gong. Learning from the best, differently: A diversity-driven rethinking on data selection. arXiv preprint arXiv:2510.18909, 2025. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. In Proceedings of the 2021 conference on empirical methods in natural language processing, p. 7514–7528, 2021. Arthur E Hoerl and Robert W Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55–67, 1970. Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al. Glm-4.5 v and glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv preprint arXiv:2507.01006, 2025. Ailin Huang, Chengyuan Yao, Chunrui Han, Fanqi Wan, Hangyu Guo, Haoran Lv, Hongyu Zhou, Jia Wang, Jian Zhou, Jianjian Sun, et al. Step3-vl-10b technical report. arXiv preprint arXiv:2601.09668, 2026. Hongzhe Huang, Jiang Liu, Zhewen Yu, Li Cai, Dian Jiao, Wenqiao Zhang, Siliang Tang, Juncheng Li, Hao Jiang, Haoyuan Li, et al. Align2llava: Cascaded human and large language model preference alignment for multi-modal instruction curation. In Findings of the Association for Computational Linguistics: ACL 2025, p. 8759–8781, 2025. 11 Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. Perplexity—a measure of the difficulty of speech recognition tasks. The journal of the Acoustical Society of America, 62(S1): S63–S63, 1977. Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017. Jaewoo Lee, Boyang Li, and Sung Ju Hwang. Concept-skill transferability-based data selection for large vision-language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 5060–5080, 2024. Jingwei Li, Xinran Gu, and Jingzhao Zhang. Capacity-aware mixture law enables efficient llm data optimization. arXiv preprint arXiv:2603.08022, 2026a. Mingxin Li, Yanzhao Zhang, Dingkun Long, Keqin Chen, Sibo Song, Shuai Bai, Zhibo Yang, Pengjun Xie, An Yang, Dayiheng Liu, et al. Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking. arXiv preprint arXiv:2601.04720, 2026b. Shengrui Li, Fei Zhao, Kaiyan Zhao, Jieying Ye, Haifeng Liu, Fangcheng Shi, Zheyong Xie, Yao Hu, and Shaosheng Cao. Decouple searching from training: Scaling data mixing via model merging for large language model pre-training. arXiv preprint arXiv:2602.00747, 2026c. Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 26679–26689. IEEE, 2024. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023a. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 26286–26296. IEEE, 2024a. Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training. arXiv preprint arXiv:2407.01492, 2024b. Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:2312.15685, 2023b. Zikang Liu, Kun Zhou, Wayne Xin Zhao, Dawei Gao, Yaliang Li, and Ji-Rong Wen. Less is more: High-value data selection for visual instruction tuning. In Proceedings of the 33rd ACM International Conference on Multimedia, p. 3712–3721, 2025. Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Yongbin Li, Minzheng Wang, Pengpeng Zeng, et al. Mmevol: Empowering multimodal large language models with evol-instruct. In Findings of the Association for Computational Linguistics: ACL 2025, p. 19655–19682, 2025. Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Jiayi Ji, Jie Lou, Debing Zhang, and Rongrong Ji. Mllm- selector: Necessity and diversity-driven high-value data selection for enhanced visual instruction tuning. arXiv preprint arXiv:2503.20502, 2025. James B McQueen. Some methods of classification and analysis of multivariate observations. In Proc. of 5th Berkeley Symposium on Math. Stat. and Prob., p. 281–297, 1967. Shruthan Radhakrishna, Aman Tiwari, Aanjaneya Shukla, Masoud Hashemi, Rishabh Maheshwary, Shiva Krishna Reddy Malay, Jash Mehta, Pulkit Pattnaik, Saloni Mittal, Khalil Slimi, et al. Apriel- 1.5-15b-thinker. arXiv preprint arXiv:2510.01141, 2025. 12 Bardia Safaei, Faizan Siddiqui, Jiacong Xu, Vishal M Patel, and Shao-Yuan Lo. Filter images first, generate instructions later: Pre-instruction data selection for visual instruction tuning. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 14247–14256. IEEE, 2025. James E Smith and Robert L Winkler. The optimizer’s curse: Skepticism and postdecision surprise in decision analysis. Management Science, 52(3):311–322, 2006. Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report. arXiv preprint arXiv:2504.07491, 2025a. Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026. Meituan LongCat Team, Bairui Wang, Bin Xiao, Bo Zhang, Bolin Rong, Borun Chen, Chang Wan, Chao Zhang, Chen Huang, Chen Chen, et al. Longcat-flash-omni technical report. arXiv preprint arXiv:2511.00279, 2025b. Jiapeng Wang, Changxin Tian, Kunlong Chen, Ziqi Liu, Jiaxin Mao, Wayne Xin Zhao, Zhiqiang Zhang, and Jun Zhou. Mergemix: Optimizing mid-training data mixtures via learnable model merging. arXiv preprint arXiv:2601.17858, 2026. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. Xindi Wu, Mengzhou Xia, Rulin Shao, Zhiwei Deng, Pang Wei Koh, and Olga Russakovsky. Icons: Influence consensus for vision-language data selection. arXiv preprint arXiv:2501.00654, 2024. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36:69798–69818, 2023a. Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 34201–34227, 2023b. Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, and Lifu Huang. Vision-flan: Scaling human-labeled tasks in visual instruction tuning. In Findings of the Association for Computational Linguistics: ACL 2024, p. 15271–15342, 2024. Yichen Yan, Ming Zhong, Qi Zhu, Xiaoling Gu, Jinpeng Chen, and Huan Li. Coido: Efficient data selection for visual instruction tuning via coupled importance-diversity optimization. Advances in Neural Information Processing Systems, 38:167045–167073, 2026. Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In International Conference on Learning Representations, volume 2024, p. 22743–22763, 2024. Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403.16952, 2024. Weijie Yin, Yongjie Ye, Fangxun Shu, Yue Liao, Zijian Kang, Hongyuan Dong, Haiyang Yu, Dingkang Yang, Jiacong Wang, Han Wang, et al. Sail-vl2 technical report. arXiv preprint arXiv:2509.14033, 2025. Qifan Yu, Zhebei Shen, Zhongqi Yue, Yang Wu, Bosheng Qin, Wenqiao Zhang, Yunfei Li, Juncheng Li, Siliang Tang, and Yueting Zhuang. Mastering collaborative multi-modal data selection: A focus on informativeness, uniqueness, and representativeness. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 155–165, 2025. 13 Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, et al. Mm1. 5: Methods, analysis & insights from multimodal llm fine-tuning. In International Conference on Learning Representations, volume 2025, p. 64669–64714, 2025. 14 APPENDIX CONTENTS A Method Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 A.1 Finer Allocation Can Improve the Attainable Optimum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 A.2 Mixture Interpretation of Within-Cluster Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 A.3 Learning the Validation Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 A.4 Direct Search Avoids an Additional Response-Fitting Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 B Experimental Details and Additional Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 B.1 Model Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 B.2 Data Pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 B.3 Training and Evaluation Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 B.4 Baseline Selection Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 B.5 Repeated Runs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 B.6 Benchmark-Level Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 B.7 Allocation Freedom on a Fixed Expert Bank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 B.8 Expert Training Budget and Proxy Agreement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 B.9 Gradient Search Progress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 C Data Interpretation, Flexibility, and Reuse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 C.1 Data Annotation and Composition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 C.2 Why Allocation and Sample Ranking Complement Each Other . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 C.3 Reusing Feedback Across Targets and Granularities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 C.4 Incremental Search Budget . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 AMETHOD DETAILS A.1FINER ALLOCATION CAN IMPROVE THE ATTAINABLE OPTIMUM A coarse allocation fixes the proportions of fine clusters within each parent. Write this restriction asw = Bz, where each column ofBcontains the nonnegative child proportions of one parent and sums to one. The fine familyW and its coarse restrictionZ =z : Bz∈W satisfy sup z∈Z F d (Bz) ≤ sup w∈W F d (w).(6) Every coarse mixture is also a fine mixture, so optimizing over the larger family cannot lower the best attainable score. The improvement can be strict when a better composition requires changing the fixed within-parent proportions. This is the benefit of fine-grained allocation: it preserves coarse choices and allows additional ones. A.2MIXTURE INTERPRETATION OF WITHIN-CLUSTER SAMPLING With uniform within-cluster sampling fixed, composition is the selection decision being varied. For cluster sizeM k , drawn k examples uniformly without replacement and letr k = n k /Nbe its realized share. For any fixed per-example quantity f , E " 1 N X x∈S f (x) # = X k r k 1 M k X x∈D k f (x) ! .(7) Indeed, each example in clusterkis included with probabilityn k /M k ; summing its expected contribution gives the identity. Thus the subset represents the specified mixture in expectation, without introducing a preference for particular examples within a cluster. Under the fixed training recipe, a real probe supplies a noisy observation of that mixture’s expected downstream score. Repeated sampling reduces variation in the mean under the same within-cluster rule. A.3LEARNING THE VALIDATION OBJECTIVE The candidate panel contains 100 examples from each of 100 clusters. We evaluate proxies at the same 128 mixtures as the real probes. Each loss averages cross-entropy over that example’s answer tokens, so response length is normalized before examples are aggregated. 15 Screening and fitting. Let X ij =−ℓ( e θ(w i );v j ). On fitting indicesI, the screening score is q d,j = 1 2 ρ Sp (X I,j ,y (d) I ) + 1 2 ρ Sp (X I + d ,j ,y (d) I + d ),(8) whereI + d is the higher-scoring half of the fitting probes. We retain the topm = 64examples and standardize their negative losses using fitting-probe meansμ j and standard deviationss j . We then fit their coefficients by Eq.(3), withλ = 10. The fit uses centered numerical target scores; Spearman correlation measures the resulting ordering. Returning to loss units.Withc j = b β j /s j ,C d = P j c j > 0, andα d,j = c j /C d , the fitted score is by i = ̄y d + X j b β j X ij − μ j s j = b d − C d J d (w i ), b d = ̄y d − X j c j μ j .(9) Consequently, minimizing the normalized weighted loss preserves the fitted score ranking. The final objective is fitted on all 128 probes and frozen for search. Preserving individual responses. A weighted average of cluster-mean losses assigns the same effective weight to every selected example within a cluster. Individual weights include this special case but also distinguish examples with different responses to the training mixture. They therefore preserve information that a cluster mean removes before fitting. Matched evaluation. The four aggregation rules in Table 2 share the same 64 screened examples in each of five repeated four-fold splits. Screening, standardization, and weight fitting use only the 96 fitting probes; the remaining 32 evaluate ranking. Cluster-level rules average losses within each represented cluster; sample-level rules retain individual losses. Equal weighting uses a uniform average; learned weighting uses the same nonnegative ridge fit. The sample-weighted fits retain, on average, 15.3/12.2/13.6/16.1 positive weights for General/Parsing/Knowledge/Overall, respectively. All splits reuse the same 128 development probes. A.4DIRECT SEARCH AVOIDS AN ADDITIONAL RESPONSE-FITTING ERROR Suppose a response regressorhapproximates the fixed validation objectiveJ d on a candidate family C, with sup C |h− J d |≤ ξ. If its selected mixture b w h is η-optimal for h, then J d ( b w h )− min C J d ≤ η + 2ξ.(10) To see this, letw J minimizeJ d . ThenJ d ( b w h )≤ h( b w h ) +ξ ≤ h(w J ) +η +ξ ≤ J d (w J ) +η + 2ξ. An η-optimal solution to J d has a gap of at most η, avoiding the additional 2ξ term from regression. For the downstream interpretation, assume|F d (w)− (a d − c d J d (w))| ≤ ε d onC, withc d > 0. Directη-optimal search then givesmax C F d −F d ( b w)≤ 2ε d +c d η, while the response-fit route gives 2ε d + c d (η + 2ξ). Direct search uses the same signal without the extra regression term. BEXPERIMENTAL DETAILS AND ADDITIONAL RESULTS B.1MODEL CONSTRUCTION We retain the pretrained Qwen language backbone and randomly initialize the visual head and alignment MLP. Alignment jointly trains the visual head and MLP while keeping the language backbone fixed, using LLaVA-Pretrain (558K). We then perform multimodal mid-training on LLaVA- OneVision-1.5 Mid-Training-85M (An et al., 2025) to obtain the common base θ 0 . B.2DATA POOL Some public datasets combine multiple constituent sources. We manually separate these into smaller source branches to preserve their identities, then remove duplicate and noisy records. The resulting instruction collection contains approximately 40M processed records across 30 branches. Table 4 lists 16 the upstream releases and processed counts. Upstream sizes refer to the indicated release or branch; processed rows follow our training format, which can combine multiple turns into one conversation. Derived branches can therefore differ in counting unit from their parent releases. Sampling 20% from each semantic region gives the shared 8,007,947-example candidate pool. Every selection method draws its 100K subset from this pool; Full uses the complete candidate pool. Table 4: Instruction-pool sources. Public release sizes and processed source-branch rows before 20% sampling. Derived branches list parent-release sizes. Dataset or branchLinkUpstream size Processed rows ECD-10K ImagesHF ∼321.5K321,544 Honey-Data-1MHF1,072,1111,070,304 Llama-Nemotron-VLM v1, VQA-9HF46,74546,745 MGrounding-630KHF ∼630K623,979 M-IFInstruct-23KHF22,55422,555 M-IFInstruct-23K, second branchHF22,55422,551 SPIQA, training splitHF262,524262,524 CMM-Math, image-bearing trainHF6,2725,004 DocReason25KHF25,87725,741 MP-DocReason51KHF51,75449,959 Enigmata, generated snapshotHF217,5414,758 IQBenchCode500500 LLaVA-OneVision Stage-2, videoHF ∼350K videos ∼1,090,000 LLaVA-OneVision Stage-2, imageHF ∼2.7M2,633,205 LLaVA-OneVision Stage-2, textHF ∼450K522,701 LLaVA-OneVision-1.5 InstructHF ∼21.2M21,592,501 LLaVAR-2 Hybrid-InstructHF ∼42K images43,660 LongAlign-10KHF9,8889,861 LongWriter-6KHF6,0005,926 Magpie-Align, filteredHF500K479,743 MMDU-45KHF45,20828,979 M-MathInstruct-3MHF2,871,9882,871,988 MTVQA, training splitHF6,678 images6,678 SmolTalk2, SFTHF3,383,2423,382,846 MMPR-v1.2HF815,293814,687 Tulu-3 Persona-IFHF29,98029,980 VisualWebInstruct, conversationHF1,004,0701,004,068 VisualWebInstruct, text derivativeHF1,004,070558,467 COYO-derived instructionsHF746,972,26999,965 WildChat branchHF3,199,8602,398,331 B.3TRAINING AND EVALUATION SETTINGS We report compute on a common 64-H100-equivalent basis. Model training uses AdamW, BF16, and zero weight decay, with the schedules in Table 5. Experts use a constant learning rate; other stages use cosine decay with 1% warmup. Final tuning uses a global batch of 128 packed sequences for 100 updates. Training/loader seed pairs are 42/52, 43/53, and 44/54; single-run comparisons use 42/52. Table 5: Training schedules. Expert checkpoint indices are defined in Section B.8. StageUpdates / checkpointPeak learning rateToken budget Alignment4,50010 −3 16,384 Mid-training35,999 of 40,00010 −5 8,192 Cluster experts1 / 5 / 1010 −5 16,384 Real probes5010 −5 16,384 Final instruction tuning10010 −5 16,384 Selection settings.Search updates only the FP32 mixture shares using Adam at learning rate10 −3 for 120 steps, with zero weight decay. Each update uses all positive-weight validation examples; full BF16 evaluations every 20 steps, including step 0, select the best mixture. Integer quotas total 100K by rounding shares down and distributing the remainder to the largest fractional parts, followed by uniform within-cluster sampling. 17 Target scores. For target benchmark setB d ,y (d) i = |B d | −1 P b∈B d s i,b . Scores use percentage- point units, including OCRBench rescaled to that range. Overall averages all seven benchmarks; Average in the main table averages the four target columns. Table 6: Benchmark-to-target mapping. Domain targets average their constituent benchmark scores equally. Overall averages all seven scores, not the three domain means. BenchmarkDomain targetDomain weightOverall weight MMBench-ENGeneral1/31/7 MMStarGeneral1/31/7 GQAGeneral1/31/7 OCRBenchParsing1/21/7 ChartQAParsing1/21/7 MMLUKnowledge1/21/7 MMLU-ProKnowledge1/21/7 B.4BASELINE SELECTION METHODS All selection methods construct 100K-example subsets from the same candidate pool and use the same final-training recipe. Random samples uniformly without replacement; Full uses all 8M. Sample-level methods.PPL (Jelinek et al., 1977) favors examples with lower response perplexity, using model likelihood as the selection signal. CLIP-Score (Hessel et al., 2021) uses image–text embedding similarity to favor visually aligned instructions. MLLM-Selector (Ma et al., 2025), labeled M-Selector in our tables, combines necessity scores from a seed instruction-tuned model with diversity-aware sampling. PRISM (Bi et al., 2025) derives training-free selection scores from intrinsic visual features and their redundancy. ICONS (Wu et al., 2024) estimates gradient influence on reference examples and aggregates task-level evidence by consensus. DSIR (Xie et al., 2023b) estimates source-to-reference importance ratios in a lightweight feature space and resamples toward the reference distribution. DataTailor (Yu et al., 2025) combines informativeness, within-cluster uniqueness, and cross-cluster representativeness. Because ICONS and DSIR require validation or reference data, both receive CLIVO’s selected validation examples to keep target-specific reference content consistent across comparisons. Group-based methods.SemDeDup (Abbas et al., 2021) identifies semantically similar examples in embedding clusters and removes redundant representatives. COINCIDE (Lee et al., 2024) organizes examples by concept–skill representations and selects using cluster density and transferability. Our two regression controls follow mixture-response modeling (Liu et al., 2024b; Li et al., 2026c). Acc + Reg. fits a predictor from the 128 semantic-mixture vectors to real target scores, searches for high predicted accuracy, and fills the selected cluster quotas uniformly. Source + Reg. instead optimizes shares over the 30 source branches, fitting a predictor of target-aligned merged-proxy loss and sampling uniformly within each source. Within-cluster variant.CLIVO uses uniform sampling by default. CLIVO + ICONS preserves its learned quotas and selects the highest-scoring ICONS examples within each cluster. Global ICONS instead selects across the entire pool. Both use the same target-specific reference examples. B.5REPEATED RUNS Random, Acc + Reg., Source + Reg., and CLIVO report three-run means; the other methods report one run. CLIVO’s repeats keep the selected mixture fixed and vary the final sampling/training seed. Table 7 gives the mean, sample standard deviation, and sample variance, using then− 1 denominator. Table 8 reports the available baseline statistics: Random, Acc + Reg., and Source + Reg. use population standard deviation over three data-sampling seeds (42, 43, and 44). 18 Table 7: CLIVO final-training repeats. Three runs at fixed searched shares with uniform within-cluster sampling. Variance uses the n− 1 denominator. TargetMeanStandard deviationSample variance General67.280.530.28 Parsing79.640.590.34 Knowledge61.760.110.01 Overall68.130.130.02 Table 8: Baseline repeats. Three-seed mean± population standard deviation. MethodRunsOverallGeneralParsingKnowledge Random366.61± 0.2363.57± 0.2877.73± 0.0960.05± 0.44 Source + Reg.367.00± 0.1762.91± 0.3979.12± 0.2659.80± 0.16 Acc + Reg.367.53± 0.0565.44± 0.3877.80± 0.0361.40± 0.26 B.6BENCHMARK-LEVEL RESULTS Table 9 reports CLIVO’s three-run benchmark means and the single-run CLIVO + ICONS results. Each target has its own selected subset and trained model, so its benchmark profile is reported separately. Domain scores and the four-target Average are calculated before rounding. Table 9: Benchmark scores by target (%). CLIVO: three runs with random sampling; CLIVO + ICONS: one run, same quotas. TargetBenchmarkCLIVOCLIVO + ICONS General MMBench-EN84.8785.22 MMStar55.9858.27 GQA60.9961.32 Parsing OCRBench81.2381.00 ChartQA78.0478.92 Knowledge MMLU72.2372.25 MMLU-Pro51.2951.93 Overall MMBench-EN84.0384.43 MMStar53.3853.00 GQA59.8760.68 OCRBench80.9379.90 ChartQA77.0177.28 MMLU70.7470.71 MMLU-Pro50.9350.38 B.7ALLOCATION FREEDOM ON A FIXED EXPERT BANK We keep the same 100 experts, initial mixture, validation objective, 120-update search, and final- training recipe. A nested Ward hierarchy of cluster centroids groups the experts into 20 or 50 parents; using 100 shares leaves every expert’s coefficient independently adjustable. For parentg, fix child ratiosa k|g = w 0 k / P h∈G g w 0 h . A zero-mass parent uses ratios proportional to child cluster sizes. The optimized parent shareszexpand asw = Bz, whereB kg = a k|g within parentgand zero elsewhere. Exact premerging identity. Define∆ g = P k∈G g a k|g ∆ k . Then e θ(Bz) = θ 0 + X g z g ∆ g .(11) Thus tying fine coefficients is exactly equivalent to merging fixed coarse expertsθ g = θ 0 +∆ g . The parent gradients are B ⊤ ∇ w J d . This isolates allocation freedom with fixed experts. 19 Relation to briefly training a coarse expert. LetL k be the mean training loss of childkand L g = P k a k|g L k . For one gradient-descent step from the commonθ 0 , with common learning rateη, X k a k|g θ 0 − η∇L k (θ 0 ) = θ 0 − η∇L g (θ 0 ). Thus, merging after one gradient-descent step exactly matches an update on the coarse mixture. For smooth losses and a fixed numberTof small gradient-descent steps,θ (T ) k = θ 0 −Tη∇L k (θ 0 )+O(η 2 ). Averaging this expansion gives the coarse model’s first-order update, with a difference of orderO(η 2 ). The local approximation motivates premerging. Our experiment uses the exact identity in Eq. (11). Table 10: Allocation freedom.J ⋆ : minimum proxy loss; t ⋆ : best step;C ⋆ : 64-H100-equivalent hours;F: single-seed score. Bold: best F . Accounting: Section C.4. TargetKJ ⋆ t ⋆ C ⋆ F General 200.326391200.66267.197 500.3280180 ≥ 0.34366.833 1000.321451000.80867.017 Parsing 200.8595200.0079.810 500.859651000.83380.120 1000.858661200.99380.260 Knowledge 200.6873140 ≥ 0.36961.415 500.687431201.12861.570 1000.68760400.40261.825 Overall 200.37324400.36067.994 500.370971000.86567.846 1000.369041000.81868.021 Table 10 gives the full results underlying Figure 2. Each cell uses the same single-seed training protocol. Parsing and Knowledge improve with finer shares; General peaks at 20; Overall is not monotone. B.8EXPERT TRAINING BUDGET AND PROXY AGREEMENT We train one expert per cluster from the common base on 64 H100 GPUs at a constant learning rate of10 −5 , with four packed sequences per GPU and a 16,384-token budget per sequence. We compare checkpoints indexedb = 1, 5, 10, holding this recipe fixed. The step-10 source checkpoints record 11 completed updates and a median of 16,591 processed examples per expert. A fully trained bank provides the reference. All banks use the same 128 mixtures and 10,000 examples. Table 11: Expert budget and two kinds of agreement. Fidelity compares proxy with real-model loss over 128 mixtures. G/P/K/O are held-out correlations with real benchmark scores after budget-specific validation- cluster selection. Bold: highest in each correlation column. BankMean lossFidelity ρG ρP ρK ρO ρ Step 11.2830.4850.3830.4350.5230.367 Step 51.0070.6470.4090.4620.5240.387 Step 100.9720.7300.5180.4480.5270.460 Full schedule0.9400.6440.5770.3820.4400.457 Fidelity is the Spearman correlation between proxy and real-model mean losses over the 128 mixtures. For downstream prediction, five repeated four-fold splits use 96 fitting and 32 held-out probes. Within each fitting fold, clusters are ranked by the correlation between negative mean loss and target score; three-fold inner validation chooses the number retained from1, 2, 4, 8, 16, 32. The final columns report mean held-out correlations after budget-specific selection. The strongest proxy–real agreement and useful score prediction across targets support using step 10 as the shared expert bank. 20 B.9GRADIENT SEARCH PROGRESS The four searches reduce their frozen validation objectives (Figure 5 and Table 12). Full-panel evaluations every 20 updates select the best checkpoint, at steps 100, 120, 40, and 100 for General, Parsing, Knowledge, and Overall. General enters a near-best plateau at step 80, whereas losses for the other targets continue to fluctuate. Periodic evaluation keeps the best observed mixture throughout the 120-update search. 060120 0.324 0.330 0.336 Weighted proxy loss Best step: 100 General 060120 0.860 0.864 Best step: 120 Parsing 060120 0.6875 0.6880 0.6885 Best step: 40 Knowledge 060120 0.368 0.372 0.376 Best step: 100 Overall Gradient-search step Figure 5: Search progress on fixed validation objectives. Stars mark the lowest recorded losses. General’s shaded region retains at least 95% of its observed loss decrease with two subsequent checks. Lines connect full-panel evaluations at 20-update intervals. Table 12: Search starts and checkpoints. Full BF16 evaluation selectst ⋆ ; remerged step-zero loss J 0 defines the decrease D J . TargetStart recipe t ⋆ J 0 J ⋆ General1011000.3338850.321450 Parsing1191200.8611850.858662 Knowledge27400.6879880.687602 Overall1011000.3767300.369045 The decrease in Table 3 isD J = 100(J 0 − J ⋆ )/J 0 , using the same run’s evaluated start. Best-of- probe references are 66.600, 79.310, 61.465, and 67.9186 for General, Parsing, Knowledge, and Overall; the final score column uses three-run means. Table 5 lists the probe and final-model budgets. CDATA INTERPRETATION, FLEXIBILITY, AND REUSE C.1DATA ANNOTATION AND COMPOSITION We sample a 10,000-example panel across the 100 clusters for annotation with gpt-5.2-2025-12-11; the archive contains 9,555 valid outputs. The sample prompt la- bels task, content, format, and difficulty. The cluster prompt summarizes label frequencies and representative annotations. The response limits are 500 and 800 tokens, respectively; images use a maximum side length of 512 pixels and videos supply up to ten frames. An author-conducted check of sampled annotations yielded a human–GPT agreement rate of 0.92. These semantic labels support data visualization and interpretation. Table 13 gives both prompts. 21 Table 13: GPT annotation prompts. Sample labels describe task, content, format, and difficulty; cluster summaries aggregate these annotations. Sample annotation. The interaction’s text and visual inputs follow this prompt. You are an expert Data Annotation Specialist. Your task is to analyze User- Assistant interaction samples and classify them into 4 dimensions using STRICT standardized labels. The data comes from diverse multi-modal sources including VQA, Document Reasoning, Mathematical Logic, Web Agents, and Grounding tasks. You must output a SINGLE valid JSON object. ### 1. Task Dimension (Hierarchical) Classify the user’s primary intent. - "task_category": Choose one of ["Perception", "Reasoning", "Generation", " Knowledge", "Chat"] - "task_type": Choose the specific sub-type based on the category: - If Perception: ["OCR", "Object Detection", "Image Captioning", "Visual Grounding", "Video Tracking"] - If Reasoning: ["Math Solving", "Chart/Data Analysis", "Document Understanding", "Logic/Puzzle", "Web/UI Navigation", "Scientific Analysis "] - If Generation: ["Creative Writing", "Translation", "Summarization", " Code Implementation", "Long-form Content"] - If Knowledge: ["Fact Lookup", "Concept Explanation", "Trivia", "How-to Guide"] - If Chat: ["Roleplay", "General Chat", "Multi-turn Dialogue"] ### 2. Content Dimension (Visual & Semantic split) Analyze the domain and visual inputs. - "visual_content": Choose one: ["No Image", "Natural Scene", "Human/Face", " Document/Text", "Chart/Graph/Plot", "UI/Webpage", "Art/Illustration", " Abstract/Pattern"] - "text_domain": Choose one: ["General/Daily Life", "STEM", "Humanities/Art", "Medical/Health", "Legal/Finance", "Code/Programming", "Web/Internet"] ### 3. Format Dimension (Input & Output Structure) Analyze the structural form. - "input_composition": Choose one: ["Text-Only", "Single-Image", "Multi-Image ", "Video", "Interleaved-Text-Image"] - "response_style": Choose one: ["Multiple Choice", "Yes/No", "Short Answer", "Long Explanation/CoT", "Code Block", "Structured Format (JSON/Table)", " Bounding Box/Coordinates"] ### 4. Difficulty Dimension (Cognitive Load) - "difficulty_level": Choose one: ["Low", "Medium", "High"] ** CRITICAL: ** Output raw JSON only. No markdown. No explanations. Continued on the next page. 22 Table 13: GPT annotation prompts (continued). Cluster summary. Inputs follow underCluster Aggregated Statistics(label histograms) and Sampled Annotations (up to 100 records). You are an expert Data Analyst and Dataset Curator. I will provide you with the aggregated annotation statistics and a sample of individual interactions from a specific data cluster. These individual samples were previously annotated across 4 strict dimensions: Task, Content, Format, and Difficulty. Your objective is to analyze this aggregated data and the samples to construct a comprehensive, multi-dimensional profile of this specific cluster. Please output a valid JSON object strictly matching the following schema. Use the predefined label vocabularies from the sample annotations when identifying dominant categories. "task_dimension_summary": "dominant_task_category": "The most frequent task category (e.g., Perception, Reasoning, etc.)", "top_task_types": ["List the top 1-3 specific task_types present in this cluster"], "task_diversity_note": "A 1-sentence analysis of whether this cluster is highly focused on one task or mixes several." , "content_dimension_summary": "primary_visual_content": ["List the top 1-2 most frequent visual_content types"], "dominant_text_domain": "The most frequent text_domain", "content_fusion_analysis": "A 1-sentence description of how the visual and textual domains interact in this cluster." , "format_dimension_summary": "dominant_input_composition": "The most frequent input_composition (e. g., Interleaved-Text-Image)", "typical_response_style": ["List the top 1-2 expected response_styles "] , "difficulty_dimension_summary": "overall_difficulty_level": "Low / Medium / High / Mixed (based on majority distribution)", "cognitive_load_description": "A 1-sentence explanation of why the data requires this level of cognitive load." , "comprehensive_cluster_profile": "short_definition": "A precise, 1-sentence title or definition for this cluster (e.g., ’Complex Scientific Document Reasoning and Formatting ’).", "detailed_description": "A comprehensive 3-4 sentence paragraph synthesizing all 4 dimensions. Describe the typical user intent, the nature of the multi-modal inputs, the expected output structure, and the overall technical or cognitive demands of the data within this cluster." ** CRITICAL INSTRUCTIONS: ** 1. Output raw JSON ONLY. 2. Do NOT wrap the JSON in markdown code blocks (e.g., no ‘json ... ‘). 3. Do NOT include any introductory or concluding text. 4. Ensure all dimensional analyses strictly reflect the provided sample data and statistics. 23 From fine tasks to coarse capabilities. We map the first task in each cluster summary’s toptasktypesto a coarse capability using Table 14. These annotations make the learned allocation interpretable; embeddings define clusters and the target objective sets their shares. Table 14: Coarse capability labels for visualization. Each cluster’s primary annotated task determines its coarse label. The original task descriptions are retained for finer interpretation. Coarse capabilityPrimary task types Visual QA & CaptioningImage Captioning Grounding & DetectionObject Detection; Visual Grounding; Video Tracking OCR & DocumentsOCR; Document Understanding Charts & DataChart/Data Analysis Math & LogicMath Solving; Logic/Puzzle Knowledge & ScienceScientific Analysis; Concept Explanation; Fact Lookup Generation & OtherRemaining task types, including code and creative generation Composition statistics and cases. For normalized shares, the effective cluster count isK eff = exp(− P k:w k >0 w k logw k ) : 3.024, 10.134, 3.923, and 4.362 for General, Parsing, Knowledge, and Overall. Uniform allocation gives each cluster 1%; a categoryAtherefore has uniform share|A|/100 and enrichment( P k∈A w k )/(|A|/100) . The within-capability cases pair c60/c38 for General, c27/c14 for Parsing, and c36/c20 for Knowledge. Percentages are shares of the total budget. Table 15 compares the complete mixtures. Total variation is 1 2 P k |p k − q k | ; Jensen–Shannon divergence is 1 2 KL 2 (p∥m) + 1 2 KL 2 (q∥m), where m = (p + q)/2. Table 15: Target-dependent mixtures. TV and JSD compare final cluster shares. Target pairTVJSD (bits) General / Parsing0.97830.9481 General / Knowledge0.99990.9995 General / Overall0.26100.2126 Parsing / Knowledge0.98950.9789 Parsing / Overall0.88230.8338 Knowledge / Overall0.97650.9235 C.2WHY ALLOCATION AND SAMPLE RANKING COMPLEMENT EACH OTHER Any size-NsubsetSinduces cluster countsn k = |S ∩D k |and sharesr k = n k /N. Its empirical sampling distribution factors as p S (x) = r k q S (x| k), q S (x| k) = 1x∈S n k , x∈D k , n k > 0.(12) Group-level allocation explicitly chooses the marginal sharesr k ; within-group selection determines which examples realize them. At the finest partition, every group contains one example, and a share of 1/N or zero specifies selection. Group allocation thus recovers sample-level selection. For intermediate partitions, composition and ranking offer complementary control. Global top-N ranking maximizes the sum of sample scores and lets cluster shares follow from that ranking. Fixing learned quotas instead maximizes the same scores subject to|S∩D k | = n k , solved by taking the top n k examples in each cluster. Thus, quotas control target-relevant coverage while ranking chooses representatives within that coverage. CLIVO + ICONS implements this factorization, combining learned composition with an existing ranker (Table 1). C.3REUSING FEEDBACK ACROSS TARGETS AND GRANULARITIES LetC shared denote the construction cost of embeddings, clusters, experts, real probes, and the per-example proxy-loss cache. The incremental cost of selecting for a new target d is ∆C target (d) = C score (Y,d) + C fit (Y,H,m d ) + C grad (G,T,m d ).(13) 24 The terms are benchmark evaluation of theYsaved probes, example screening and weight fitting on cached losses, andTsearch updates withGadjustable shares andm d active validation examples. Expert/probe training and cache construction are reused. Final model training is a separate cost common to subset-selection methods. For a new granularity at a fixed target, the incremental cost is∆C gran (G) = C tie (G) + C grad (G,T,m d ), whereC tie constructs the parent-to-child mapping. On the fixedK 0 = 100 expert bank, writeD = [∆ 1 ,..., ∆ K 0 ]andw = B G z. A single accumulated backward pass supplies∇ z J d = B ⊤ G D ⊤ ∇ θ J d . For our implementation, the search cost decomposes as C grad (G,T,m d ) = T C merge (K 0 ) + C FB (m d ) + C directions (K 0 ) + C update (G) + C check . (14) The dominant merge, forward/backward, and expert-direction computations are shared across the 20/50/100-share settings; only the coefficient mapping and update change. Finer allocation therefore adds no model backward pass per extra share and no higher-dimensional response fit. In comparison, a regression search that constructsQnew candidate labels incursQC label + C regfit (Q,G)before proposal and recheck costs. Section C.4 reports the measured search workloads. C.4INCREMENTAL SEARCH BUDGET We report compute as 64-H100-equivalent hours: H100-equivalent GPU-hours divided by 64, using the same peak-BF16 normalization across runs. This is a common compute-budget unit. The measured workload includes iterative merging and model forward/backward computation; shared construction and final model training are excluded. Regression reference. LightGBM predictors use nested budgets of 256, 512, 1,024, 2,048, and 4,096 mixtures labeled with proxy losses. Five-fold cross-validation Spearman reaches 0.935/0.909/0.898/0.885 at 4,096 labels for General/Parsing/Knowledge/Overall. The final dou- bling improves these values by 0.024/0.013/0.055/0.036, smaller than the preceding doubling for every target. We use these 4,096 shared labels as the high-budget reference. Table 16: Incremental search compute. Gradient costs cover all 120 updates. Regression cost cov- ers constructing the shared 4,096 proxy-loss labels. Units: 64-H100-equivalent hours. WorkloadCompute budget Gradient: General0.960 Gradient: Parsing0.993 Gradient: Knowledge1.123 Gradient: Overall0.971 Gradient: four-target total4.046 Regression: 4,096 shared labels5.509 Table 17: Search endpoints. Single-seed 100K scores. GD: gradient search; regression: 4,096 labeled mixtures. TargetGD 20GD 50GD 100Regression General67.2066.8367.0267.09 Parsing79.8180.1280.2680.10 Knowledge61.4261.5761.8362.17 Overall67.9967.8568.0268.22 The four gradient searches use 26.6% less normalized compute than regression label- ing. In Table 10,C ⋆ measures cost to the first lowest-loss checkpoint. Upper endpoints summarize reconstructed tim- ing bounds;≥denotes incomplete timing. Parsing with 20 shares retains its start, but the complete search costs 0.312 hours in the same normalized units. Regression endpoint protocol. Regression ranks 10 million proposals by predicted log loss. Its 4,096 labels cover 115 feasible probes, four gradient endpoints, 994 perturbations of gradient solutions, and 2,983 Dirichlet samples. Gradient endpoints contribute labels but are excluded from final selection. Table 17 reports single-seed scores for the selected candidates. 25