Paper deep dive
Paired Multimodal Scaling Laws
Marcus Ma, Shrikanth Narayanan
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Existing multimodal scaling laws fit multimodality terms empirically after testing and never vary how much data is multimodally paired at fixed data budgets. We investigate how, under the same total data per modality, changing the number of paired data affects loss curves in multimodal classification tasks. We train models in three different environments and run experiment sweeps varying data sizes and pairing budget. Pairing ratios have a dramatic impact on loss and this impact is directly tied to how much information synergy the task contains. Only paired data is able to reduce synergistic loss, while unpaired data can reduce redundant or unimodal information up until unimodal floors. Unlike traditional scaling laws where loss drops immediately in power law decay, synergy acquisition is gated, requiring a critical threshold of paired data before synergistic loss falls at all. We introduce a new family of multimodal scaling laws where total data-attributable loss is the sum of four individual power laws corresponding to the four different information channels of redundancy, a unique channel per modality, and synergy, and show how this law is both more theoretically sound and empirically valid across our experiments. This law predicts multimodal loss in our experiments more accurately than existing laws, with 3.2% error on fit tests versus 10.4% error for the best pairing extension of published laws.
Tags
Links
- Source: https://arxiv.org/abs/2609.36263v1
- Canonical: https://arxiv.org/abs/2609.36263v1
Trouble viewing inline? Open PDF directly →
Full Text
66,150 characters extracted from source content.
Expand or collapse full text
Paired Multimodal Scaling Laws Marcus Ma Affiliation: University of Southern California Email: mjma@usc.edu Shrikanth Narayanan Affiliation: University of Southern California Abstract Existing multimodal scaling laws fit multimodality terms empirically after testing and never vary how much data is multimodally paired at fixed data budgets. We investigate how, under the same total data per modality, changing the number of paired data affects loss curves in multimodal classification tasks. We train models in three different environments and run experiment sweeps varying data sizes and pairing budget. Pairing ratios have a dramatic impact on loss and this impact is directly tied to how much information synergy the task contains. Only paired data is able to reduce synergistic loss, while unpaired data can reduce redundant or unimodal information up until unimodal floors. Unlike traditional scaling laws where loss drops immediately in power law decay, synergy acquisition is gated, requiring a critical threshold of paired data before synergistic loss falls at all. We introduce a new family of multimodal scaling laws where total data-attributable loss is the sum of four individual power laws corresponding to the four different information channels of redundancy, a unique channel per modality, and synergy, and show how this law is both more theoretically sound and empirically valid across our experiments. This law predicts multimodal loss in our experiments more accurately than existing laws (3.2% error on fit tests versus 10.4% error for pairing extensions of published laws) and can be used to determine optimal pairing fractions under constrained budgets. 1 Introduction Scaling laws predict how neural networks perform as a function of resources spent on it. This is traditionally represented in terms of model parameters N and training examples D (Hestness et al., 2017; Kaplan et al., 2020). Hoffmann et al. (2022) formulate this equation as L(N,D)=E⏟irreducible error+ANα⏟approximation error+BDβ⏟estimation error, L(N,D)= E_irreducible error+ AN^α_approximation error+ BD^β_estimation error, (1) where the irreducible error represents the Bayes-optimal theoretical error floor, the approximation error is loss due to finite model size, and the estimation error is loss due to finite data sample. Multimodal extensions of scaling laws keep the same shape as Hoffmann et al. (2022) but divide D into multiple subcomponents, i.e. by modeling loss as L(N,D1,D2)L(N,D_1,D_2). Aghajanyan et al. (2023) decompose each modality into its own unimodal scaling law, then adjust empirically with multimodal interaction constants. Other laws represent loss in terms of D but with additional terms that capture the per-domain fraction of the total data (Ye et al., 2025; Tao et al., 2025). These laws represent multimodality by modeling how much of each modality is present. However, a multimodal corpus contains a second quantity that is not modeled in existing scaling laws: multimodal pairing, or the presence of both modalities for a single data point. For an amount of paired data npn_p, we represent the pairing fraction p as p=2npD1+D2∈[0,1].p= 2n_pD_1+D_2∈[0,1]. p=1p=1 only when every example is a pair and adding unpaired examples of either modality lowers p. Existing works have never varied p while holding D1D_1 and D2D_2 fixed, and we hypothesize that their failure to model p explicitly could partially explain why each work arrives at a different scaling law equation fitting their empirical results. When adding extra unpaired data has been tested, it is always in the text modality, and it always has come with the confound of adding extra total data, which means the effect of pairing alone has never been isolated. We run experiments that vary p across three settings at fixed data budget and find that p has a direct impact on downstream loss that existing laws fail to capture. We formulate new multimodal scaling laws that incorporate this variable based on Partial Information Decomposition (Bertschinger et al., 2014; Liang et al., 2023) and find our laws to be more theoretically grounded and empirically accurate. In particular, paired data is the only data stream that enables learning of synergistic information, or emergent information that arises only when both modalities are present. We also find unpaired data helps reduce non-synergistic information loss, but only with sufficient paired data alongside. Synergy is also gated: unlike traditional scaling law decay, synergistic loss stays at chance until a certain pair count threshold. Using these findings, we introduce a new four-channel multimodal scaling law, where data loss is the sum of four independent power law curves: L(N,D1,D2,np)=E+ANα+LR+LU1+LU2+LS L(N,D_1,D_2,n_p)=E+ AN^α+L_R+L_U_1+L_U_2+L_S (2) where each L term is fed data only by relevant modalities: redundant R by either modality, unique U1U_1 and U2U_2 by modalities 1 and 2, respectively, and synergistic S only by paired data npn_p. This family of scaling laws fits our empirical results significantly better than existing multimodal scaling laws and are the only laws capable of modeling the synergistic phenomena we observe. Their form is also more grounded in multimodal interaction theory. 2 Related Work Our work builds on multimodal scaling laws (§2.1), data-mixture laws (§2.2), and partial information decomposition (§2.3), a subset of information theory that motivates our experiments and laws. 2.1 Multimodal scaling laws In unimodal settings (Eq. 1), while loss is a variable of N and D, A and B set the intercept of loss and the exponents α and β determine the rate of loss scaling. Exponents are set mainly by the task and the data distribution while architecture mainly moves the starting points A and B (Hestness et al., 2017; Kaplan et al., 2020). Aghajanyan et al. (2023) fits separate scaling curves per modality and bimodal runs to identify a synergy constant Ci,jC_i,j at odds with competition terms in N and D. Because synergy is modeled as a constant, a “competition barrier” exists at an N-D curve where synergy outweighs competition. Shukor et al. (2025b) train 457 image-text models across a variety of architectures and find little variation at scale but that early-fusion models perform better at lower parameter counts. Like Aghajanyan et al. (2023), this alludes to a “synergy threshold” at scale. Bill (Qingyun) Sun et al. (2024) propose weighting multimodal mixtures by tokenization efficiency and Tao et al. (2025) fit a compute-dependent modality mixture with a learned interaction matrix. Several of these works mix a text-only pool into paired image-text data, which lowers p but also adds data, so the role of p is never isolated. 2.2 Data-mixture scaling laws Data mixture laws ask what mixture a corpus, typically text, each domain (web text, books, papers) should be assigned, either fit before training or updated online. Ye et al. (2025) present the law as minimization of individual domain losses and Liu et al. (2025) regress loss on a mixture of proxy models targeting 2% compute of a target run. Que et al. (2024) investigate the ratio of domains r as a scaling term and Shukor et al. (2025a) find mixture ratios that are fixed regardless of scale perform best. Other approaches operate online; Albalak et al. (2023) presents training as a bandit problem, and Jiang et al. (2025) calculate per-domain power laws during the run. Li et al. (2025) estimate loss bounds per-task via gradient norms. However, Chen et al. (2025) identify no method above consistently beats uniform sampling in general. They diagnose that all of the functional forms fail to model interaction terms between mixtures, so no equations generalize. 2.3 Partial information decomposition Partial Information Decomposition, introduced by Williams and Beer (2010), decomposes the joint mutual information shared by two feature variables and a target variables into I(X1,X2,Y)=R+U1+U2+SI(X_1,X_2;Y)=R+U_1+U_2+S, where R is redundant information shared between both modalities on Y, U1U_1 and U2U_2 are unique information per modality, and S is joint information present only with both modalities. Liang et al. (2023) formulate this problem for multimodal training loss. Given saturated unimodal training losses L1L_1 and L2L_2, multimodal loss L1,2L_1,2, and initial loss L, we have: L−L1=U1+R,L−L2=U2+R,L−L1,2=S+U1+U2+R L-L_1=U_1+R, L-L_2=U_2+R, L-L_1,2=S+U_1+U_2+R (3) Using linear combinations of these three values, we are able to calculate R+U1R+U_1, R+U2R+U_2, and S−RS-R, but not R or S in isolation. Bertschinger et al. (2014) and Liang et al. (2023) identify tractable ways to estimate S for joint distributions and multimodal machine learning. We assert that modeling S is the overlooked solution for better multimodal scaling laws. S determines how much co-occurring information is required for a task, which we hypothesize can only be learned during paired interaction. Meanwhile, we hypothesize unpaired data can help reduce U1,U2,U_1,U_2, and R independently but have no impact on S. 3 Experiments We create three environments that vary architectures, data, and multimodal interaction. In the first (Digits), we train image and audio encoders to predict digits in a setup where we can adjust the PID information R,U1,U2,R,U_1,U_2, and S. In the second (VQA), we train classification heads on top of pretrained frozen embeddings for visual question answering using VQA (Antol et al., 2015), a multimodal dataset heavily dominated by the text modality (Goyal et al., 2017). In the third (NLVR2), we finetune a pretrained multimodal transformer on NLVR2 (Suhr et al., 2019), another visual question dataset where mutual information is heavily dominated by S and unimodal models stay at chance. Figure 1: Illustration of each environment along with an example and tested architecture. 3.1 Methodology We apply the same methodology to each environment. We reserve a fixed, unseen test set for every run. Unpaired examples fill in the missing modality with a learnable null vector and a modality-presence bit. During test time, both modalities are always present and loss is cross-entropy. In Appendix C.1, we analyze replacing cross-entropy with accuracy. Grid sweeps and evaluation. We run experimental sweeps across data budgets D, compute budget C, pairing count npn_p, and multiple seeds per combination. For each experiment, we fix a single architecture and parameter count N, as prior scaling law literature has demonstrated that loss due to architecture and parameter counts is independent of loss attributable to data composition (Hestness et al., 2017; Kaplan et al., 2020; Shukor et al., 2025b). For conciseness, we combine the irreducible and approximation errors E and A/NαA/N^α from Eq. 1 into a single term ENE_N. In Appendix C.2, we repeat experiments for Digits and VQA on different architectures and parameter sizes and again find the same qualitative results. Counting paired and unpaired data. We determine total data via D=D1+D2D=D_1+D_2, i.e. the sum of unique examples per modality in the dataset. This means as p→1p→ 1, the number of unique rows decreases. We do not split a pair into two unpaired data: each row is a unique label. As this manner of counting is more favorable to unpaired data, any improvement from p=0p=0 to p=1p=1 is tied directly to pairing. We fix equal compute via training step count, so a model trained at Dp=1D_p=1 has twice as many training epochs as Dp=0D_p=0 but the same gradient steps. In Appendix C.3, we consider other ways of counting paired and unpaired data. 3.2 The Three Environments Figure 1 and Table 1 illustrate each environment and architecture. We describe each experimental setup below with implementation details in Appendix B. Digits. We construct series of four concatenated digits over MNIST images (LeCun et al., 1998) and AudioMNIST spectrograms (Becker et al., 2024) The label is a four-digit number corresponding directly to U1U_1, U2U_2, R, and S for each position: the first and second digits (U1U_1 and U2U_2) are the first and second digit shown in image and audio, the third digit (R) is the image and audio’s third digit value, and the fourth digit (S) are the fourth input digits’ sum mod10 10, so knowing only one modality is uninformative. We add symmetric label noise ϵ=0.1ε=0.1 so U1=U2=R=S=ln10−Hϵ=0.1≈1.758U_1=U_2=R=S= 10-H_ε=0.1≈ 1.758. Loss is averaged over each label digit. The model architecture is two CNN encoders that pool to one value per position then a fused position-aware head. Digits VQA-v2 NLVR2 image audio image question photos sentence Experimental details architecture CNN + 4-way head (476476k) Frozen CLIP + head (5.35.3M) Finetuned LXMERT (210210M) chance loss L0L_0 ln10≈2.303 10≈ 2.303 ln3129≈8.048 3129≈ 8.048 ln2≈0.693 2≈ 0.693 D 88k, 1616k, 3232k, 6464k, 128128k, 256256k 00, 44k … 256256k 44k, 88k, 1616k, 3232k, 6464k C (k(D1+D2)k(D_1+D_2) steps) k∈10,20,40k∈\10,20,40\ k∈ℤ+≤80k _+≤ 80 k∈1,2,3k∈\1,2,3\ runs 408408 (33 seeds) 3,6603,660 (55 seeds) 1,0101,010 (55 seeds) Unimodal data curve fitting L(D)=EN+B⋅D−βL(D)=E_N+B· D^-β at the above N ENE_N 0.5730.573 0.5870.587 4.1694.169 2.6832.683 ≈ln2≈ 2 ≈ln2≈ 2 B 69.769.7 119119 2.172.17 59.859.8 >0>0 >0>0 β 0.6840.684 0.7320.732 0.0950.095 0.4110.411 ≈∞≈∞ ≈∞≈∞ Partial information decomposition (nats) R 1.7581.758 0.5310.531 00 U1U_1 1.7581.758 0.1270.127 00 U2U_2 1.7581.758 1.6141.614 00 S 1.7581.758 0.1440.144 0.5320.532 Table 1: Experimental details, unimodal data curve variables, and PID terms for each environment. Compute C represents number of training steps multiplied by batch size. VQA. We utilize VQA-v2 train, a collection of images and questions with labels in a 3,129 label vocabulary. We freeze CLIP ViT-B/32 (Radford et al., 2021) image and caption embeddings. We then train a fusion head with unpaired images and captions accompanied with a learned null vector. As initially reported by Goyal et al. (2017), the VQA dataset is heavily dominated by the question text alone. NLVR2. NLVR2 is another image-question dataset engineered so multimodality is required. Each example is two photographs plus a sentence that asks a comparative question. All questions are binary with even splits between True/False, so individual unimodal accuracy hovers around 50%. We finetune an existing model, LXMERT Tan and Bansal (2019), which has achieved 74.45% on NLVR2, using the published model code11 1 https://github.com/airsplay/lxmert. Unimodal loss curves and PID. We run large-scale unimodal experiment runs until loss saturation and determine unimodal loss curves for each modality with Hoffmann-style variable estimates in Table 1. We use these curves and task setup to estimate PID variables R, U1U_1, U2U_2, and S. For Digits, this is analytically know to be ≈1.758≈ 1.758 nats. For VQA, the mutual information terms U1U_1, U2U_2, and R are estimated as linear combinations between the soft-label entropy H(y)=4.828H(y)=4.828 nats minus the cross-entropy of saturated unimodal runs and a multimodal run, following Bertschinger et al. (2014). Shuffling questions within majority-answer class raises paired loss from 2.4112.411 to 2.5562.556 which estimates S as 0.1440.144. NLVR2’s unimodal probes always stay at chance, which we use to infer that 100% of reducible loss comes from synergy. We set S=0.532S=0.532 as the lower bound of mutual information based on the published human accuracy of 96.2%96.2\% on the development split (Suhr et al., 2019). 4 How Does Pairing Affect Loss? Paired data, and only paired data, is capable of reducing synergistic loss. Meanwhile, information conveyable in single modalities is learnable from unpaired data, but models require paired data to transfer unpaired learnings to a paired setting. In Figure 2, we keep total data per-modality fixed and equal (D1=D2D_1=D_2) and vary percentage of paired data. Increased pairing decreases loss in all three environments, and we hypothesize the relative percentage of reducible loss depends on task synergy after an initial one-time loss drop following p=0p=0. We identify this initial drop as pairs unlocking unpaired data: unpaired data is only useful in multimodal settings after learning paired interaction. In the Digits environment, for high D, we notice a plateau then second descent. This behavior reflects the model learning R, U1U_1, and U2U_2 labels very quickly and taking a longer time to learn S; a similar flatline then drop arises in NLVR2. Additionally, for NLVR2, some seeds never leave chance, with the likelihood of a “failed-to-start” seed increasing at lower p and lower D. We discuss these phenomena further in §4.3. In VQA, where synergy accounts for 6% of loss, there is a large loss dropoff from p=0%p=0\% to p=5%p=5\% but after a much more modest reduction in loss as p→1p→ 1. At compute higher than k=20k=20 for VQA, optimal p lies interior of p=1p=1, which we note appears to be an artifact of overfitting because total examples halve as p→1p→ 1. We conclude at fixed D, optimal compute decreases with increased p because repeat gradient steps per datum increase. Figure 2: Test cross-entropy loss versus pairing fraction p at fixed per-modality data budgets D1=D2D_1=D_2 and fixed compute (k⋅Dk· D gradient steps; k=20k=20 for Digits and VQA, k=3k=3 for NLVR2). ±1± 1 std shown as shaded bands. For NLVR2, we filter out “failed-to-start” seeds, as discussed in §4.3. 4.1 Controlling for Modality Presence Matching One confound from above is that in our setup, each example in the test set always contains both modalities, so improvement in p could simply be train-test alignment. This explains the significant jump in performance following p=0p=0 for Digits and VQA, because at p=0p=0 there are zero paired examples to learn from. In order to test this, we re-run Digits runs where test examples are also unpaired, i.e. when either modality is nulled out. In Figure 3, we plot loss of each information channel twice, on top when both modalities are present and on the bottom when irrelevant modalities are nulled: for U1U_1, we null audio, for U2U_2, the image, and for R, both individually with the two losses averaged. When controlling for presence matching in this fashion, increasing pairing fraction has no impact on U1U_1 and U2U_2, and slightly increases the loss of R. We hypothesize that as p→1p→ 1, the redundant prediction learns to rely on both channels simultaneously, so zeroing out a single channel actually hurts performance rather than having no effect like for U1U_1 and U2U_2. At low p, showing both modalities increases loss for single-modality channels (U1U_1, U2U_2, R) compared to just showing one modality because the model has not been exposed to enough paired examples in training to learn the other modality is irrelevant. After around 88k pairs, though, the top and bottom readouts converge to the same loss. We conclude, then, when controlling for train-test modality presence matching, paired data only helps reduce synergistic loss, but in general, pairing is required for unpaired training data to be useful during paired deployment. Figure 3: Individual channel losses at D1=D2=128kD_1=D_2=128k depending on which modalities are shown at test time. Top: loss when every test example shows both modalities. Bottom loss when irrelevant modalities are nulled for that given channel. For U1U_1, audio is nulled, for U2U_2, image is nulled, for R, both are nulled then averaged, and for S, no change. 4.2 When Does Extra Unpaired Data Help? Figure 2 holds a total data budget fixed. In Figure 4, we fix paired data and add extra unpaired examples. For Digits, we subdivide loss into the Bayes floor and each individual information channel’s loss. Unpaired data reduces redundant and modality-unique loss but never synergistic loss. Additional unpaired data has the highest impact at lower overall D, with 24k more images or audio clips reducing loss at D=8D=8k from 1.26 to 1.10 nats; however, once the unimodal floors are more saturated at D=32D=32k, adding the same ratio barely affects loss. In both settings, synergistic loss is not reduced. This is consistent with NLVR2 seeing no improvement with additional unpaired data, since NVLR2 is dominated by S. Meanwhile, for VQA, adding unpaired text examples reduces loss more than images, which is consistent with relative PID weights for text and images. How much unpaired data helps also depends on the number of pairs; when np=128n_p=128, 124124k unpaired questions reduce loss by 0.920.92 nats, but at np=4n_p=4k, it reduces loss by 1.201.20 nats. Pairs and unpaired data are therefore complements initially: adding some of one pool raises the value of the other. We use these results to conclude that unpaired data does help reduce loss given sufficient pairs, but only to reduce R, U1U_1, and U2U_2 losses. Figure 4: Test losses across the three environments at fixed npn_p and k epochs, with additional unpaired data added. Digits loss is decomposed into individual information channel losses and the Bayes floor. 4.3 Synergy Acquisition is Gated Both the Digits S channel and the NLVR2 loss curves display atypical behavior at onset: they sit at chance initially then drop after a certain threshold of npn_p. Meanwhile, VQA, whose synergistic loss share is small, and the R, U1U_1, and U2U_2 Digit channels, exhibit no such behavior. Synergy acquisition is gated on two factors: a minimum of paired examples nfloorn_floor and a minimum of paired exposures (including repeats) r. We propose the synergy gate n∗n^* is the number of paired data required before synergistic loss drops, n∗=max(nfloor,r/k)n^*= (n_floor,r/k). For Digits, nfloor≈2900n_floor≈ 2900 and r≈40kr≈ 40k for an interleaved data mixture of paired and unpaired data. We find data presentation order matters. When paired data is entirely frontloaded, synergy is acquired after the paired data but forgotten by the end of training, similar to catastrophic interference (McCloskey and Cohen, 1989). When paired data is entirely backloaded, synergy is acquired, but at a worse rate: synergistic loss drops 0.600.60 nats during interleaved but only 0.150.15 nats during unpaired-then-paired training. In Figure 5, we display several properties of this gate. n∗n^* is influenced by how much of the total loss is composed of synergy. As total loss share of S moves from 25%25\% to 100%100\%, n∗n^* decreases: when S loss constitutes half of total loss, n∗n^* is halved to 20k, and when S loss makes up 75%75\%, n∗n^* decreases to 12k. Synergy gating occurs NLVR2, but unlike Digits, it is not a hard threshold gate but a random variable of n∗n^*. NLVR2 random chance escape is best modeled as log2n∗∼(11.32, 1.362) _2n^* (11.32,\ 1.36^2), which models the actual escape distribution with median n∗=2552n^*=2552 and a χ2χ^2 fit of p=0.81p=0.81. Figure 5: Evidence of a synergy gate. (a) Digits loss of the S channel as a factor of npn_p and how much relative weight the S channel contributes to total loss at D=64kD=64k and k=40k=40. (b) Top: Digits loss of the S channel as a factor of npn_p and training step (batch size=128=128). Bottom: median training step and n∗n^* per data budget when S loss leaves chance. n∗n^* increases with higher data budgets because total repetitions lower. (c) NLVR2’s “failed-to-start” seeds (red X’s) versus seeds that leave chance accuracy (blue dots), plotted against npn_p. 5 Paired Scaling Laws We now attempt to form a generalizable multimodal scaling law with pairing as an input parameter, L(N,D1,D2,np)L(N,D_1,D_2,n_p). We propose several extensions of existing scaling laws by adding p as a variable where at p=1p=1 they simplify to the original equation. We then introduce a new loss family, where data loss is summed as four separate loss curves based on R, U1U_1, U2U_2, and S, and permute four law variations of this family. These new laws consistently outperform pairing-variable extensions of published laws, even at p=1p=1. We provide theoretical backing for this (namely, that existing laws cannot model synergy) and show how these laws can find optimal pairing ratios under a pairing budget where pairs are more expensive than unpaired data. Law k Γ ∂Γ/∂D≠0∂ /∂ D≠ 0 L(N,D1,D2,np)L(N,D_1,D_2,n_p) at p=1p=1 Hoffmann Unimodal 4 ≤0≤ 0 ✓ EN+Bg(D1+D2)E_N+B\,g(D_1+D_2) EN+Bg(2D)E_N+B\,g(2D) Hoffmann on unique D 4 ≤0≤ 0 ✓ EN+Bg(Dtot)E_N+B\,g(D_tot) EN+Bg(D)E_N+B\,g(D) Hoffmann on D1D_1 and D2D_2 7 =0=0 × EN+B1g1(D1)+B2g2(D2)E_N+B_1g_1(D_1)+B_2g_2(D_2) EN+B1g1(D)+B2g2(D)E_N+B_1g_1(D)+B_2g_2(D) Aghajanyan +Cp+Cp 8 =0=0 × EN+B1g1(D1)+B2g2(D2)+CpE_N+B_1g_1(D_1)+B_2g_2(D_2)+Cp EN+B1g1(D)+B2g2(D)+CE_N+B_1g_1(D)+B_2g_2(D)+C Shukor mixture 6 <0<0 ✓ EN+(C1p+C2(1−p))−1+Bg(Dtot)E_N+(C_1p+C_2(1-p))^-1+B\,g(D_tot) EN+C1−1+Bg(D)E_N+C_1^-1+B\,g(D) Ye mixture 7 any ✓ EN+Cexp(α2n2/Dtot+αpnp/Dtot)+Bg(Dtot)E_N+C ( _2n_2/D_tot+ _pn_p/D_tot)+B\,g(D_tot) EN+Ceαp+Bg(D)E_N+C\,e _p+B\,g(D) Four-channel family k / Δk k base law and switch substitutions what it models Base 14 ≤0≤ 0 ✓ EN+∑aBaga(Da)E_N+ _aB_a\,g_a(D_a) one curve per PID channel Shared −6-6 no change ga→g_a→ g one (δ,β)(δ,β) for all channels Gate +0+0 any BSgS(np)→BS/(1+(np/n⋆)γ)B_S\,g_S(n_p)→ B_S/(1+(n_p/n_ )^γ) synergy is gated PID −3-3 no change Ba→s0WaB_a→ s_0W_a amplitudes from measured PID Effective Count +2+2 any Di→Di−(1−λ0)ninλnp+nλD_i→ D_i-(1- _0)\,n_i\, n_λn_p+n_λ pairs unlock unpaired data Table 2: Extensions of existing scaling laws (Top) and the introduced four-channel law and variants (Bottom). Existing laws reduce to their original equations at p=1p=1. Each of the variations of the four-channel law can be applied independently, resulting in 16 four-channel law combinations. k is the number of fitted parameters for the corresponding law, Dtot=D1+D2−npD_tot=D_1+D_2-n_p, ni=Di−npn_i=D_i-n_p, and WaW_a are measured PID weights. Γ=−∂2L/∂D1∂D2 =-∂^2L/∂ D_1∂ D_2 is taken at fixed p; for the four-channel family its sign is that of the base law unless a switch changes it. Γ and ∂Γ/∂D∂ /∂ D are discussed in §5.3. 5.1 Candidate Laws Extending existing laws. We provide six candidates from existing laws (Table 2, Top) and extend their forms to model pairing. The first three are traditional Hoffmann et al. (2022)-style equations with different approaches of counting data: as the total data exposures per modality, the number of unique data rows, and two independent data curves per modality. We then extend an existing multimodal scaling law and two data mixture laws from the literature: Aghajanyan et al. (2023)’s law that adds a constant “synergy” factor, Shukor et al. (2025a)’s law that provides a “usefulness” term to each mixture (paired and unpaired), and Ye et al. (2025)’s exponential of the mixture proportions. The four-channel law. Building on our analysis that R,U1,U2,R,U_1,U_2, and S explains much of the variation between environments, we present an alternative equation family, that total loss can be broken down as the sum of independent power law curves for each information channel: L(N,D1,D2,np)=EN+LR+LU1+LU2+LS L(N,D_1,D_2,n_p)=E_N+L_R+L_U_1+L_U_2+L_S (4) Each individual L term is a data power curve where the total amount of “data” for that term corresponds to the number of examples relevant to its channel. R is fed by D1+κD2D_1+κ D_2, where κ is the learned relative usefulness per modality, and U1U_1, U2U_2 and S are fed by D1D_1, D2D_2, and npn_p. As we fix architecture and model size in our experiments, EN=E+A/NαE_N=E+A/N^α is a constant. We vary four factors across this equation family for a total of sixteen variations: 1. Shared δ and β: whether the data curve intercepts and exponents are shared across the four losses or independent per curve 2. Synergy offset: whether the synergy gating effect is modeled in LSL_S via a soft gate 1/(1+(np/n∗)γ)1/(1+(n_p/n^*)^γ), based on §4.3 3. PID-pinned: whether loss terms’ relative weights are set by the environment’s PID ratios 4. Effective counts: whether unpaired example count is discounted until a critical threshold of paired data, based on the “pairs unlock unpaired data” finding from §4.1 We replace B/DβB/D^β with (L0−EN)(δ/(D+δ))β(L_0-E_N) (δ/(D+δ) )^β, which converge at D>>δD>>δ, for better-behaved curves as D→0D→ 0 and let g(D)=(δ/(D+δ))βg(D)=(δ/(D+δ))^β for conciseness. Full derivations in Appendix A. Digits VQA NLVR2 Overall Law k D↑D p=1p=1 R2R^2 D↑D p=1p=1 R2R^2 D↑D p=1p=1 R2R^2 D↑D p=1p=1 R2R^2 Hoffmann Unimodal 4 13.9% 30.4% 0.00 8.2% 8.1% 0.00 1.8% 6.6% 0.00 8.0% 15.0% 0.00 Hoffmann on unique D 4 14.9% 37.1% -0.12 10.5% 13.1% -1.26 1.7% 6.9% -0.45 9.0% 19.0% -0.61 Hoffmann on D1D_1 and D2D_2 7 11.5% 27.3% 0.00 2.7% 6.7% 0.00 1.8% 6.6% 0.00 5.3% 13.5% 0.00 Aghajanyan +Cp+Cp 8 11.5% 20.4% 0.39 2.3% 5.0% 0.40 1.5% 6.0% 0.22 5.1% 10.4% 0.34 Shukor mixture 6 9.6% 29.1% 0.46 8.2% 9.4% 0.02 1.4% 6.2% -0.09 6.4% 14.9% 0.13 Ye mixture 7 9.9% 31.8% 0.46 5.3% 8.1% 0.06 1.3% 5.7% 0.11 5.5% 15.2% 0.21 Four-channel Shared Gate PID Eff. × × × × 14 9.6% 21.4% 0.63 3.0% 3.4% 0.80 0.8% 4.7% 0.61 4.5% 9.9% 0.68 × × × ✓ 16 9.5% 9.9% 0.75 3.2% 1.8% 0.86 0.8% 4.8% 0.61 4.5% 5.5% 0.74 × ✓ × × 14 9.5% 21.7% -1.27 2.9% 3.0% 0.81 1.1% 2.8% 0.81 4.5% 9.2% 0.12 × ✓ × ✓ 16 9.5% 3.3% 0.95 3.8% 3.5% 0.86 1.1% 3.0% 0.81 4.8% 3.3% 0.87 ✓ × × × 8 9.2% 23.9% 0.66 2.9% 2.1% 0.80 0.8% 4.7% 0.61 4.3% 10.2% 0.69 ✓ × × ✓ 10 9.2% 23.9% 0.74 2.9% 2.4% 0.84 0.8% 4.7% 0.61 4.3% 10.3% 0.73 ✓ ✓ × × 10 9.2% 21.6% -1.56 2.9% 2.8% 0.81 1.1% 2.7% 0.81 4.4% 9.0% 0.02 ✓ ✓ × ✓ 12 9.3% 3.3% 0.95 3.2% 3.5% 0.85 1.1% 2.8% 0.81 4.5% 3.2% 0.87 × × ✓ × 11 8.6% 21.5% 0.63 2.3% 2.7% 0.81 0.8% 4.6% 0.61 3.9% 9.6% 0.68 × × ✓ ✓ 13 9.0% 21.5% 0.74 2.1% 2.2% 0.86 0.8% 4.6% 0.61 4.0% 9.4% 0.74 × ✓ ✓ × 11 9.8% 21.4% -1.10 2.4% 3.1% 0.79 1.1% 2.1% 0.81 4.4% 8.9% 0.17 × ✓ ✓ ✓ 13 9.8% 8.5% 0.95 2.6% 3.5% 0.86 1.1% 2.1% 0.81 4.5% 4.7% 0.87 ✓ × ✓ × 5 10.4% 22.5% 0.49 3.2% 6.2% 0.22 0.8% 4.6% 0.61 4.8% 11.1% 0.44 ✓ × ✓ ✓ 7 10.6% 22.5% 0.54 2.8% 4.7% 0.61 0.8% 4.6% 0.61 4.7% 10.6% 0.59 ✓ ✓ ✓ × 7 9.8% 23.3% 0.63 3.0% 3.2% 0.79 1.1% 2.1% 0.81 4.6% 9.5% 0.74 ✓ ✓ ✓ ✓ 9 9.8% 5.7% 0.95 2.8% 3.7% 0.83 1.1% 2.1% 0.81 4.6% 3.8% 0.86 Table 3: Paired scaling law scores on the three fit tests: percent error for D↑D and p=1p=1 (lower is better), and within-cell R2R^2 (higher is better). 5.2 Evaluation We evaluate each law based on four held-out tests after fitting each equation via least squares to each test’s respective training runs. DmaxD_max represents the size of data for the largest run of each environment. The three tests are: 1. Scale (D↑D ): fit on runs with D∈[0,Dmax/2)D∈[0,D_max/2) and test on runs with D∈[Dmax/2,Dmax]D∈[D_max/2,D_max] 2. Full pairing (p=1p=1): fit on runs where p∈[0,1/2)p∈[0,1/2) and test on the corner p=1p=1 3. Pairing shape (R2R^2): fit on all except one (D1,D2)(D_1,D_2) grid, predict every loss for every p at (D1,D2)(D_1,D_2), subtract the mean, then measure explained variance 5.3 Results Figure 6: Paired scaling laws on four different D grids for the Leave-One-Out test, meaning each curve is fit on other (D1,D2)(D_1,D_2) combinations and tested on this (D1,D2)(D_1,D_2) cell. All cells, laws, and tests can be viewed at this web link. Results of each fit are in Table 3. Extensions of published laws are the weakest fits across all tests and environments, with every variant of the four-channel law beating published laws on D↑D , 14 of 16 on p=1p=1, and 13 of 16 on R2R^2. The two factors that are most helpful for the four-channel law are the synergy gate and effective counts. Shared exponents reduce the number of fitted parameters and also improve overall fit. PID loss weights do not improve test performance, although they do reduce the number of fitted parameters. Theory behind the four-channel law’s improvement. We identify two theoretical reasons why the four-channel law outperforms extensions. Let Γ=−∂2L/∂D1∂D2 =-∂^2L/∂ D_1∂ D_2 at fixed p, which represents how much adding an example of modality 1 changes the value of adding an example of modality 2. Properly modeling synergy requires Γ>0 >0 at some point, since synergistic information can only be acquired when both modalities are present. Additionally, proper modeling requires ∂Γ/∂D∂ /∂ D must be non-zero, i.e. Γ changes with changing data, as different information channels eventually saturate. Every existing law except the Ye mixture violates either or both of these constraints (Table 2), and Ye only models relative share instead of absolute counts as we observed in §4 and §4.3. The four-channel law contains both properties via the S decomposition, where synergy loss is gated before a threshold (allowing Γ>0 >0) and decays (allowing ∂Γ/∂D≠0∂ /∂ D≠ 0). Interactive viewer. We display examples of different law fits plotting CE against p in Figure 6. Four-channel laws are able to successfully model synergy gates, delayed loss, and p-dependent loss. To further illustrate these results, we have included an interactive paired scaling law viewer at this web link. Optimal pairing fractions. These laws can also be used to determine the best pairing fraction under a budget where pairs are expensive. For NLVR2, which is almost all synergy, optimal pairing is always p=1p=1, but for Digits and VQA, it moves well below p=1p=1 when pairs cost two to three times unpaired data (Appendix A.5). 6 Conclusion We investigated how multimodal loss changes as a function of data pairing and found that only pairs can reduce synergistic loss while unpaired data can reduce redundant and modal-unique loss. We introduced a new four-channel multimodal scaling law that outperforms existing scaling laws and also proved that data pairing is an independent variable that has measurable impact on model loss. Scope and limitations. Partial information decomposition is defined with respect to a label Y, so our laws are most applicable to supervised multimodal tasks. We did not investigate self-supervised generative pretraining without labels where pairing itself is the training signal. We leave investigation of the generalizability of our four-channel scaling law in self-supervised training regimes as future work, as well as larger data and parameter scales. We note many established scaling law papers have been tested at scales comparable to our experiments (Hestness et al., 2017; Rosenfeld et al., 2020; Sorscher et al., 2022; Bahri et al., 2024) and that the power-law forms they describe hold at larger scales (Kaplan et al., 2020; Hoffmann et al., 2022; Aghajanyan et al., 2023). AI Use Statement In this work, we performed the following tasks without the help of AI or LLM usage: • Formulating core mathematical claims and theorems • Developing theoretical models and conceptual frameworks • Proposing and refining hypotheses • Discovering relevant research areas and important existing literature • Designing research methodology and experiments • Interpreting results • Identifying experimental parameters • Verifying validity of claims • Designing the paper structure • Crafting the title and keywords • Writing the final manuscript The following tasks were performed with the help of AI and LLM usage, with brief justification: • Providing critical ingredients and writing of proofs: We used AI and LLMs to double-check and verify the proposed proofs as well as simulate different variations of each equation and check for edge cases. • Providing feedback for research methodology and experiments: After outlining the core experiments, we used AI to help brainstorm potential design flaws and ablations. A human was always present to review and approve ablations. • Implementing methods and creating code: We used AI and LLMs to generate the code to execute our experiments. The majority of our code is AI-written, including the pairing viewer, but the authors have reviewed the code manually and ran unit tests, sanity checks, and manually validated that the execution and results are error-free as best to the authors’ ability. • Creating or modifying figures or images: We used AI and LLMs to generate figures and tables, mostly for ease of formatting. All ideas and layouts for figures and tables came from the authors directly. The results and layout were then manually adjusted and refined by the authors. • Cleaning and reformating datasets: We used LLMs to standardize our several datasets in the codebase. • Creating draft sections of the paper: We used AI and LLMs to summarize and create draft sections of several parts of the paper; however, every sentence in the final manuscript is human-written. • Performing related literature sweeps: After the key relevant research areas were identified through manual review, we used AI and LLMs to exhaustively search for all relevant literature. • Grammar and formatting: We used AI and LLMs to perform final grammar and formatting checks in the manuscript. The following tasks are not applicable to this work: • Generating synthetic data sets • Assisting with translation • Supporting qualitative and thematic data analysis • Formulating questions for surveys or interviews • Transcribing recordings We have reviewed all AI-assisted work and take full responsibility for the final content of this work. Ethics Statement We do not believe this work raises any ethical concerns, as this work is solely focused on fundamental machine learning theory research and empirical analysis. This work does not collect new data, deploy a system, or present any potential data privacy concerns. We use existing research datasets in compliance with their usage licenses and do not redistribute external dataset images and text. References Aghajanyan et al. (2023) A. Aghajanyan, L. Yu, A. Conneau, W. Hsu, K. Hambardzumyan, S. Zhang, S. Roller, N. Goyal, O. Levy, and L. Zettlemoyer Scaling laws for generative mixed-modal language models. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 202, p. 265–279. External Links: Link Cited by: §1, §2.1, §5.1, §6. Albalak et al. (2023) A. Albalak, L. Pan, C. Raffel, and W. Y. Wang Efficient online data mixing for language model pre-training. In R0-FoMo: Workshop on Robustness of Few-shot and Zero-shot Learning in Foundation Models at NeurIPS 2023, Note: arXiv:2312.02406 Cited by: §2.2. Antol et al. (2015) S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh VQA: visual question answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), p. 2425–2433. External Links: Document Cited by: §B.2, §3. Bahri et al. (2024) Y. Bahri, E. Dyer, J. Kaplan, J. Lee, and U. Sharma Explaining neural scaling laws. Proceedings of the National Academy of Sciences 121 (27), p. e2311878121. External Links: Document Cited by: §6. Becker et al. (2024) S. Becker, J. Vielhaben, M. Ackermann, K. Müller, S. Lapuschkin, and W. Samek AudioMNIST: exploring explainable artificial intelligence for audio analysis on a simple benchmark. Journal of the Franklin Institute 361 (1), p. 418–428. External Links: ISSN 0016-0032, Document, Link Cited by: §3.2. Bertschinger et al. (2014) N. Bertschinger, J. Rauh, E. Olbrich, J. Jost, and N. Ay Quantifying unique information. Entropy 16 (4), p. 2161–2183. External Links: Document Cited by: §1, §2.3, §3.2. Bill (Qingyun) Sun et al. (2024) Bill (Qingyun) Sun, Gavin (Zhen) Guo, and PIN AI Team Scaling law hypothesis for multimodal model. External Links: 2409.06754, Link Cited by: §2.1. Chen et al. (2025) M. F. Chen, M. Y. Hu, N. Lourie, K. Cho, and C. Ré Aioli: a unified optimization framework for language model data mixing. In International Conference on Learning Representations (ICLR), Note: arXiv:2411.05735 External Links: Link Cited by: §2.2. Goyal et al. (2017) Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh Making the V in VQA matter: elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 6904–6913. External Links: Document Cited by: §B.2, §3.2, §3. Hestness et al. (2017) J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, Md. M. A. Patwary, Y. Yang, and Y. Zhou Deep learning scaling is predictable, empirically. External Links: 1712.00409, Link Cited by: §1, §2.1, §3.1, §6. Hoffmann et al. (2022) J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, O. Vinyals, J. W. Rae, and L. Sifre Training compute-optimal large language models. In Advances in Neural Information Processing Systems, Vol. 35, p. 30016–30030. External Links: Document, Link Cited by: §1, §1, §5.1, §6. Jiang et al. (2025) Y. Jiang, A. Zhou, Z. Feng, S. Malladi, and J. Z. Kolter Adaptive data optimization: dynamic sample selection with scaling laws. In International Conference on Learning Representations (ICLR), Note: arXiv:2410.11820 External Links: Link Cited by: §A.1, §2.2. Kaplan et al. (2020) J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei Scaling laws for neural language models. Note: Preprint; no formal publication found External Links: 2001.08361, Link Cited by: §1, §2.1, §3.1, §6. LeCun et al. (1998) Y. LeCun, C. Cortes, and C. J. C. Burges The MNIST database of handwritten digits. Note: OnlineDataset. The accompanying paper is LeCun, Bottou, Bengio, and Haffner, Proceedings of the IEEE, 86(11):2278–2324, 1998 External Links: Link Cited by: §3.2. Li et al. (2025) Z. Li, Y. Deng, P. Zhong, M. Razaviyayn, and V. Mirrokni PiKE: adaptive data mixing for large-scale multi-task learning under low gradient conflicts. In Advances in Neural Information Processing Systems, Vol. 38, p. 187600–187640. Note: arXiv:2502.06244 External Links: Document Cited by: §2.2. Liang et al. (2023) P. P. Liang, Y. Cheng, X. Fan, C. K. Ling, S. Nie, R. J. Chen, Z. Deng, N. Allen, R. Auerbach, F. Mahmood, R. Salakhutdinov, and L. Morency Quantifying & modeling multimodal interactions: an information decomposition framework. In Advances in Neural Information Processing Systems, Vol. 36, p. 27351–27393. External Links: Document Cited by: §1, §2.3, §2.3. Liu et al. (2025) Q. Liu, X. Zheng, N. Muennighoff, G. Zeng, L. Dou, T. Pang, J. Jiang, and M. Lin RegMix: data mixture as regression for language model pre-training. In International Conference on Learning Representations (ICLR), Note: arXiv:2407.01492 External Links: Link Cited by: §2.2. McCloskey and Cohen (1989) M. McCloskey and N. J. Cohen Catastrophic interference in connectionist networks: the sequential learning problem. In Psychology of Learning and Motivation, Vol. 24, p. 109–165. External Links: Document Cited by: §4.3. Que et al. (2024) H. Que, J. Liu, G. Zhang, C. Zhang, X. Qu, Y. Ma, F. Duan, Z. Bai, J. Wang, Y. Zhang, X. Tan, J. Fu, J. Wang, L. Qu, W. Su, and B. Zheng D-cpt law: domain-specific continual pre-training scaling law for large language models. In Advances in Neural Information Processing Systems, Vol. 37, p. 90318–90354. Note: arXiv:2406.01375 External Links: Document Cited by: §2.2. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 139, p. 8748–8763. External Links: Link Cited by: §B.2, §3.2. Rosenfeld et al. (2020) J. S. Rosenfeld, A. Rosenfeld, Y. Belinkov, and N. Shavit A constructive prediction of the generalization error across scales. In International Conference on Learning Representations, External Links: Link Cited by: §6. Shukor et al. (2025a) M. Shukor, L. Béthune, D. Busbridge, D. Grangier, E. Fini, A. El-Nouby, and P. Ablin Scaling laws for optimal data mixtures. In Advances in Neural Information Processing Systems, Vol. 38, p. 143691–143716. Note: arXiv:2507.09404 External Links: Document Cited by: §2.2, §5.1. Shukor et al. (2025b) M. Shukor, E. Fini, V. G. T. da Costa, M. Cord, J. Susskind, and A. El-Nouby Scaling laws for native multimodal models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 12–23. External Links: Document Cited by: §2.1, §3.1. Sorscher et al. (2022) B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli, and A. S. Morcos Beyond neural scaling laws: beating power law scaling via data pruning. In Advances in Neural Information Processing Systems, Vol. 35, p. 19523–19536. External Links: Document Cited by: §6. Suhr et al. (2019) A. Suhr, S. Zhou, A. Zhang, I. Zhang, H. Bai, and Y. Artzi A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, p. 6418–6428. External Links: Document, Link Cited by: §B.3, §3.2, §3. Tan and Bansal (2019) H. Tan and M. Bansal LXMERT: learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, p. 5100–5111. External Links: Document, Link Cited by: §B.3, §B.3, §3.2. Tao et al. (2025) C. Tao, Z. Li, X. Li, J. Chen, W. Jin, H. Bai, L. Shang, and L. Hou ModalMix: optimizing multimodal data mixtures with compute-dependent regression. Note: OpenReviewSubmitted to ICLR 2026; not in the ICLR 2026 proceedings External Links: Link Cited by: §1, §2.1. Williams and Beer (2010) P. L. Williams and R. D. Beer Nonnegative decomposition of multivariate information. Note: Preprint; no formal publication found External Links: 1004.2515, Link Cited by: §2.3. Ye et al. (2025) J. Ye, P. Liu, T. Sun, J. Zhan, Y. Zhou, and X. Qiu Data mixing laws: optimizing data mixtures by predicting language modeling performance. In International Conference on Learning Representations (ICLR), Note: arXiv:2403.16952 External Links: Link Cited by: §1, §2.2, §5.1. Appendix A Paired scaling-law derivations In this appendix section, we derive δ (A.1) and Γ (A.2), define effective counts (A.3), show two additional law fit results (A.4), and plot optimal pairing ratios under a budget (A.5). A.1 Single-Source Loss Curves with δ The Hoffmann form (Eq. 1) diverges as D→0D→ 0 but classification loss is bounded by the loss of an untrained model, L0L_0 (lnk k for k balanced classes). To make this equation well-behaved at low D, we add a normalization term δ, which acts as a pseudocount of seen examples: L(D)=EN+B(D+δ)β.L(D)=E_N+ B(D+δ)^β. (5) At D=0D=0 this gives L0=EN+B/δβL_0=E_N+B/δ^β, which rewritten yields B=(L0−EN)δβB=(L_0-E_N)\,δ^β and L(D)=EN+(L0−EN)g(D),g(D)=(δD+δ)β.L(D)=E_N+(L_0-E_N)\,g(D), g(D)= ( δD+δ )^β. (6) For D≫δD δ this approaches the original term B/DβB/D^β. Both endpoints L0L_0 and ENE_N can be directly measured via expected loss of a uniform distribution and a single long training run that saturates on data and compute, respectively. This leaves the exponents δ and β to be fit instead of B and β (as B can derived through L0L_0, ENE_N, δ, and β). We can thus estimate the value of a single example at a given loss: dLdD=−βD+δ(L(D)−EN). dLdD=- βD+δ\, (L(D)-E_N ). (7) Jiang et al. (2025) use a similar quantity, without δ, to choose data mixtures during training. A.2 Interaction Sign Γ Let vi=−∂L/∂Div_i=-∂ L/∂ D_i be the value of adding one more example of modality i. This value is always positive as loss curves are monotonically decreasing, and comparing viv_i to vjv_j determines whether modality i or j is more locally helpful. We define an interaction sign Γ as Γ=∂vi∂Dj=−∂2L∂Di∂Dj. = ∂ v_i∂ D_j=- ∂^2L∂ D_i\,∂ D_j. (8) If Γ>0 >0, the modalities are complements: more of j makes an example of i more valuable. If Γ=0 =0, they are independent, and if Γ<0 <0, they are substitutes. A multimodal law that can express synergy should be able to express all three, as there are scenarios when one modality helps, hurts, or is neutral to additional data from the other modality. Proof that existing laws (except Ye) have Γ≤0 ≤ 0 at fixed p. All Hoffmann-style equations take the form L=EN+Wg(Di+κDj)L=E_N+Wg(D_i+κ D_j) with κ≥0κ≥ 0. g is a decreasing convex curve, thus ∂2L∂Di∂Dj=Wκg′(Di+κDj)≥0⟺Γ≤0 ∂^2L∂ D_i\,∂ D_j=Wκ\,g (D_i+κ D_j)≥ 0 ≤ 0 (9) for every parameter setting. This means Hoffmann-derived equations model modalities i and j as substitutes only. Similarly, at fixed p, Aghajanyan’s CpCp and Shukor’s (C1p+C2(1−p))−1(C_1p+C_2(1-p))^-1 are constants, so Aghajanyan’s Γ is 00, and Shukor’s is negative from its pooled g(Dtot)g(D_tot). Ye’s exponent depends on the share (D2−np)/Dtot(D_2-n_p)/D_tot, which at fixed p can change because npn_p is free even at fixed p by allocating additional D1D_1. Proof that the four-channel law does not constrain Γ at fixed p. The four-channel family is L L =EN+LR+LU1+LU2+LS, =E_N+L_R+L_U_1+L_U_2+L_S, (10) DR D_R =D1+κD2,DU1=D1,DU2=D2,DS=np. =D_1+κ D_2, D_U_1=D_1, D_U_2=D_2, D_S=n_p. (11) With power-law channels, La=Bag(Da)L_a=B_ag(D_a), every term is a convex function of one count or of a sum. The base law therefore has Γ≤0 ≤ 0 like the Hoffmann forms. However, adding the synergy gate replaces LSL_S with BS/(1+(np/n⋆)γ)B_S/(1+(n_p/n_ )^γ). For γ>1γ>1 this curve is concave below np=n⋆((γ−1)/(γ+1))1/γn_p=n_ ((γ-1)/(γ+1) )^1/γ, so there LSL_S contributes −(p/2)2LS′>0-(p/2)^2L_S >0 to Γ . Before synergy is learned, more of either modality makes the other more valuable, because together they buy the pairs that open the gate. After the gate, the curve is convex again and the Γ becomes negative. Effective counts change the counts that feed each channel and can in rare cases make Γ positive. A.3 Effective counts A model trained without pairs can model unimodal channels effectively but see increased loss during test-time paired data (§4.1). We model this as unpaired data being less valuable until a certain amount of paired data is available to align paired data with unpaired representations. At zero pairs, an unpaired example is worth λ0≤1 _0≤ 1 of a paired one and effective count is represented as: Dieff=Di−(1−λ0)ninλnp+nλ.D_i^eff=D_i-(1- _0)\,n_i\, n_λn_p+n_λ. (12) ni=Di−npn_i=D_i-n_p is the unpaired count of modality i and nλn_λ is the number of pairs that unlocks half the locked value. More pairs increases the effective counts of unpaired data, and thus pairs and unpaired examples become complements initially. A.4 Additional Law Fit Tests In Table 4, we report two interpolation law fits: the Leave-One-Out test, where an equation curve is fit on all cells except one (D1D_1, D2D_2) grid and then tested on every p of that heldout (D1D_1, D2D_2) cell, and the Full test, where no cells are heldout and one law is fit over every data point. These two tests reveal the same results as the main paper, that the four-channel law outperforms extensions of existing laws. Digits VQA NLVR2 Overall Law k LOO Full LOO Full LOO Full LOO Full Hoffmann Unimodal 4 5.9% 5.6% 3.9% 3.7% 1.0% 1.3% 3.6% 3.5% Hoffmann on unique D 4 5.2% 4.5% 4.0% 3.8% 1.0% 1.2% 3.4% 3.1% Hoffmann on D1D_1 and D2D_2 7 4.2% 3.6% 1.1% 1.0% 1.2% 1.3% 2.2% 2.0% Aghajanyan +Cp+Cp 8 5.7% 4.6% 1.2% 1.0% 0.9% 0.8% 2.6% 2.2% Shukor mixture 6 4.1% 3.9% 4.6% 4.3% 0.8% 1.1% 3.2% 3.1% Ye mixture 7 4.4% 4.1% 2.0% 1.7% 0.9% 1.0% 2.4% 2.3% Four-channel Shared Gate PID Eff. × × × × 14 4.5% 4.2% 0.8% 0.6% 0.5% 0.5% 1.9% 1.8% × × × ✓ 16 2.8% 2.6% 0.7% 0.6% 0.5% 0.5% 1.4% 1.2% × ✓ × × 14 3.2% 4.2% 0.8% 0.7% 0.2% 0.3% 1.4% 1.7% × ✓ × ✓ 16 1.7% 1.4% 0.8% 0.6% 0.2% 0.3% 0.9% 0.8% ✓ × × × 8 5.2% 4.9% 0.8% 0.7% 0.5% 0.5% 2.1% 2.0% ✓ × × ✓ 10 4.1% 3.7% 0.7% 0.6% 0.5% 0.5% 1.8% 1.6% ✓ ✓ × × 10 3.3% 3.9% 0.8% 0.7% 0.2% 0.3% 1.4% 1.6% ✓ ✓ × ✓ 12 1.7% 1.4% 0.7% 0.6% 0.2% 0.3% 0.8% 0.8% × × ✓ × 11 4.6% 4.3% 0.8% 0.6% 0.5% 0.5% 2.0% 1.8% × × ✓ ✓ 13 3.5% 3.1% 0.7% 0.7% 0.5% 0.5% 1.6% 1.4% × ✓ ✓ × 11 3.3% 4.3% 0.8% 0.7% 0.2% 0.3% 1.4% 1.8% × ✓ ✓ ✓ 13 2.7% 1.6% 0.7% 0.6% 0.2% 0.3% 1.2% 0.8% ✓ × ✓ × 5 3.4% 2.9% 1.4% 1.2% 0.5% 0.5% 1.7% 1.5% ✓ × ✓ ✓ 7 4.8% 4.7% 0.9% 0.8% 0.5% 0.5% 2.1% 2.0% ✓ ✓ ✓ × 7 3.5% 2.8% 0.7% 0.7% 0.2% 0.3% 1.5% 1.3% ✓ ✓ ✓ ✓ 9 2.3% 1.9% 0.7% 0.7% 0.2% 0.3% 1.1% 0.9% Table 4: Paired scaling law scores of the two interpolation tests: Leave-One-Out and the Full test. A.5 Choosing a Pairing Fraction Under a Budget We consider the practical problem of the optimal pairing ratio under a resource budget where pairs cost more than unpaired data. Figure 7 plots loss versus p given a pair price ϕ∈1,2,3φ∈\1,2,3\ for the three environments. When ϕ=1φ=1 (i.e. pairs do not cost extra), optimal p=1p=1, but higher ϕφ changes p depending on the environment: for NLVR2, which is dominated by synergy, optimal p is always 1, but for Digits and VQA, this value lies in the interior. Figure 7: Predicted loss versus pairing fraction p at pair prices ϕ∈1,2,3φ∈\1,2,3\. Dots mark optima. Appendix B Additional Experiment Implementation Details In the sections below, we provide additional implementation detail for each of the environments. B.1 Digits Experiment Labels and known constants. Each modality shows four digits, one per label position (§3). The image is four MNIST digits side by side (1×28×1121× 28× 112). The audio is four AudioMNIST log-mel spectrograms joined in time (1×64×4001× 64× 400). Training audio comes from speakers 1–48 and test audio from speakers 49–60. Test images come from the MNIST test split. Symmetric label noise at rate ε=0.1 =0.1 is applied to the label only. Output heads are initialized to zero so that an untrained model predicts the uniform distribution. Model. For each modality, we use the same encoder: three blocks of a 3×33× 3 convolution (32, 64, and 128 channels and max-pooling), then GroupNorm and ReLU. The encoder pools to a single column per digit position and then projects to a 128-d embedding. We train with auxiliary cross-entropy of weight 11 for modality presence to aid with modular sum for the synergy digit. Missing modalities are replaced with a null embedding. The fusion trunk is a two-layer neural network with hidden layer width of 256, which sees both embeddings and the modality presence bits, then feeds into four ten-way heads, one for each label digit. This model has 483k483k trainable parameters. Training and evaluation. We used AdamW with learning rate 3×10−43× 10^-4 and weight decay 10−210^-2, batch size 128 drawn uniformly with replacement, cosine decay over the planned number of steps, and gradient clipping at 22. We hold out one tenth of training for a validation set, used for logging. We trained for k∈10,20,40k∈\10,20,40\ epochs. The test set is 50005000 examples. B.2 VQA-v2 Experiment Data. The train split of VQA-v2 has 443,757443,757 questions on COCO train204 images with 10 annotator answers per question (Antol et al., 2015; Goyal et al., 2017). The vocabulary is the 3,1293,129 answers that occur at least 64 times. The 8,8728,872 questions with no answer in the vocabulary are omitted, leaving 434,885434,885 questions on 82,77282,772 images. The target label is the soft label of annotator votes, and loss is the cross-entropy in nats against the soft label. The test set is 25,00025,000 val2014 questions. Unpaired examples. We define an unpaired image as an image with no text, with the answer label sampled independently from any of the questions that are asociated with that image. An unpaired text example is a normal question example with the image nulled. Model. We used CLIP ViT-B/32 (Radford et al., 2021) to embed images and questions one time into a frozen cache. Using these embeddings, we trained an MLP head (1026→w→w→31291026→ w→ w→ 3129 then GELU) over the two embeddings and two presence bits. In the main experiment, w=1024w=1024 (5.35.3M parameters), and in Appendix C.2 we test w=512w=512 and 256256 (2.392.39M and 1.131.13M). The last layer is zero-initialized. We used AdamW, learning rate 3×10−43× 10^-4, weight decay 0.010.01, batch size 512. One sixteenth of training data was heldout as validation. B.3 NLVR2 Experiment Data. We used the LXMERT release of NLVR2 train, which contains 86,37386,373 examples (Suhr et al., 2019; Tan and Bansal, 2019). Each example is two photographs, one sentence, and a true/false label. The test set is the official development split of 6,9826,982 examples. A photos-only example has both photographs, no sentence, and the label. A sentence-only example has the sentence, no photographs, and the label. Model. We used LXMERT-base (Tan and Bansal, 2019) which has 210210M parameters. Finetuning backpropogated through the entire model. After encoding both photographs, the two outputs were concatenated into a two-layer classifier. We used BertAdam optimizer with learning rate 5×10−55× 10^-5, 10% warmup and linear decay, weight decay 0.010.01, gradient clipping at 5, batch size 32, sentence length 20, label smoothing 0.10.1, and a fresh shuffle each epoch. One sixteenth of training data was heldout as validation. Metric. We used test cross-entropy as our loss after fitting temperature on validation logits, as we encountered improved accuracy but worsened cross-entropy in initial runs due to overconfidence. Appendix C Robustness Checks We test several factors for robustness of our claims in the main paper. We report accuracy instead of cross-entropy (C.1), vary architecture and model size (C.2), and change how we count paired data (C.3). C.1 Accuracy Instead of Cross-Entropy Figure 8 repeats Figure 2 in accuracy instead of cross-entropy, and Figure 9 repeats Figure 3 in per-position accuracy in Digits. Both metrics yield similar findings. • Synergy is gated. In Figure 8, we observe delayed improved test accuracy for Digits and NLVR2, with no such gate for VQA excepting the p=0p=0 jump. This is the same as in cross-entropy. • VQA displays a p=0p=0 jump. In VQA, accuracy rises from 0.370.37 to 0.420.42 with the first 44k pairs, then rises steadily to 0.490.49 at p=1p=1. • Presence-matching has a smaller effect for accuracy. Unlike test cross-entropy, pairing has minimal effect on unimodal and redundant accuracy (Figure 9); at this data budget D1=D2=128D_1=D_2=128k, the U1U_1, U2U_2, and R slots are accurate up to the label noise threshold of 90%90\%. Figure 8: Figure 2 in accuracy instead of test cross-entropy. Figure 9: Figure 3 (Top) in accuracy instead of cross-entropy. Per-position test accuracy in Digits at D1=D2=128D_1=D_2=128k and k=40k=40 with both modalities shown. C.2 Architecture and Model Size Digits. We replace the CNN encoders from Digits with two other encoders of different architectures, both trained from scratch on the same data: an MLP and a transformer. The MLP contains two layers of width 256 for each input digit and the transformer has sixteen patches per digit position with width 128, four layers, and four heads. Figure 10 shows loss curves at D1=D2=32D_1=D_2=32k for the MLP and transformer architecture that are consistent with the CNN encoder. Test CE drops before synergy is learned, and S digit accuracy stays at chance then rises after a threshold. We note that architecture does seem to change where the threshold npn_p is, with S digit accuracy first leaving chance at 1616k pairs for the MLP, at 2424k pairs for the transformer, and at 3232k for the CNN. Figure 10: Digits CE (left) and S digit accuracy (right) versus npn_p with three different encoders at D1=D2=32D_1=D_2=32k and k=40k=40. VQA-v2 encoders. We replace frozen CLIP features with frozen Qwen2.5-VL-3B and Qwen2.5-VL-7B features (2048 and 3584 dimensions) under the same width-1024 head, at D1=D2=128D_1=D_2=128k. The Qwen image vector is the mean-pooled output of the vision encoder. The Qwen text vector is the mean of the question’s input token embeddings with no transformer layers applied. Figure 11 (a) shows the same shape for all three encoders of a steep drop after p=0p=0 then a steady and gradual fall until p=1p=1. VQA-v2 head width. Figures 11 (b) and (c) vary the MLP head width to 512 and 256. Each width observes the same behavior, and at fully paired data at D1=D2=128D_1=D_2=128k, all three heads end within 0.10.1 nats of each other. Model width changes the initial drop and gradual decline of the curve but not the curve shape. Figure 11: VQA-v2 test cross-entropy against p at k=20k=20. (a) Three frozen encoders under the width-1024 head at D1=D2=128D_1=D_2=128k. (b, c) Three head widths on CLIP features at 3232k and 128128k. C.3 Counting paired and unpaired data The main text counts data per modality as D1=n1+npD_1=n_1+n_p and D2=n2+npD_2=n_2+n_p. In this regime, a paired datum counts towards both D1D_1 and D2D_2. There are two other ways to count data: • Counting by label, D1+D2−np=n1+n2+npD_1+D_2-n_p=n_1+n_2+n_p, where a paired datum counts once. • Counting by price, where a pair costs ϕφ times as much as one unpaired example of each modality (Appendix A.5). If measuring by label and using the original definitions of D1D_1 and D2D_2, then a pair is actually cheaper than an unpaired datum with ϕ=12φ= 12. Consequences of counting by label. Because the laws take (D1,D2,np)(D_1,D_2,n_p) as input, the above decision of counting by label does not change any of the law fits, but it does change what “equal in size” means as well as deciding an optimal p under a budget. In Figure 12, we replot Figure 2 with number of labeled examples as the x-axis instead of paired examples. We find in this case, pairing is even more beneficial, as in addition to the synergistic gains demonstrated in this work, paired data also reduces the number of overall labels. This way of counting makes pairs cheaper and biases the optimal pairing ratio under a fixed budget towards more pairs. Figure 12: The runs of Figure 2 replacing the x-axis with number of labelled examples, n1+n2+np=2D−npn_1+n_2+n_p=2D-n_p, instead of pairing fraction. Each colored curve represents one matched budget D1=D2=D_1=D_2=D running from p=0p=0 (right point, 2D2D labeled examples) to p=1p=1 (left point, D labeled examples).