Paper deep dive
Accelerating Video Diffusion via Training-Free Trajectory Routing
Mustafa Munir, Huy Vu, Shreyas Misra, Rohit Jena, Sajad Norouzi, Ali Taghibakhshi, Anis Ahmad, Anjul Patney, Pavlo Molchanov, Nima Tajbakhsh
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/26/2026, 2:54:06 AM
Summary
The paper introduces TRACK (TRajectory-Aware Capacity routing via top-K selection), a training-free method to accelerate video diffusion inference. TRACK uses an offline calibration process to measure the disagreement between large and small model predictions on shared reference latents. Based on these disagreement scores, it creates a switching policy that routes quality-sensitive steps to the large model and less sensitive steps to the small model, achieving significant speedups (1.95x to 2.73x) across various video diffusion pipelines (Wan 2.1, Cosmos 3, TurboDiffusion, FastVideo) without retraining or architectural changes.
Entities (7)
Relation Signals (7)
TRACK → accelerates → Video Diffusion
confidence 95% · TRACK thereby establishes automated, training-free model switching as a practical acceleration paradigm for video diffusion.
TRACK → achievesspeedupon → Wan 2.1
confidence 95% · TRACK yields 1.95x speedup on Wan 2.1
TRACK → achievesspeedupon → Cosmos 3
confidence 95% · TRACK yields 2.04x–2.73x speedup on Cosmos 3
TRACK → achievesspeedupon → FastVideo
confidence 95% · TRACK yields 2.17x speedup on FastVideo
TRACK → achievesspeedupon → TurboDiffusion
confidence 95% · TRACK yields 2.69x speedup on TurboDiffusion
TRACK → evaluatedusing → VBench
confidence 90% · Performance is evaluated on the same 250 prompts for all configurations using VBench’s temporal flicker, motion smoothness, subject consistency, and background consistency
TRACK → uses → Disagreement Score
confidence 90% · Aggregating this signal over a calibration set produces a disagreement score map across diffusion steps, which determines a switching policy
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Video diffusion is computationally expensive, as it requires executing a large model across many denoising steps. Even with step-distillation, inference remains expensive because every distilled step still requires a costly model evaluation. We present TRACK: TRajectory-Aware Capacity routing via top-K selection, a heterogeneous denoising strategy that switches between compatible large and small models at selected steps, reducing the average cost per denoising evaluation. The switching steps are determined using a calibration process. TRACK first rolls out a reference trajectory with the large model. Then at each step, the small model's prediction is also collected and compared against the large model's prediction to obtain a relative disagreement score. Both models receive the same latent, timestep, conditioning, and guidance inputs. Aggregating this signal over a calibration set produces a disagreement score map across diffusion steps, which determines a switching policy for an efficient inference process: quality-sensitive steps keep using the large model, while steps with low disagreement scores are routed to the small model. Inference executes only the selected model at each step, requiring no retraining, architecture or scheduler changes, or online dual-model evaluation. Across Wan 2.1, Cosmos 3, TurboDiffusion, and FastVideo, TRACK yields $1.95\times$, $2.04\times$-$2.73\times$, $2.69\times$, and $2.17\times$ speedups, respectively, with comparable aggregate quality and high diversity retention. TRACK thereby establishes automated, training-free model switching as a practical acceleration paradigm for video diffusion.
Tags
Links
- Source: https://arxiv.org/abs/2609.30096v1
- Canonical: https://arxiv.org/abs/2609.30096v1
Trouble viewing inline? Open PDF directly →
Full Text
61,105 characters extracted from source content.
Expand or collapse full text
2026-9-25 Accelerating Video Diffusion via Training-Free Trajectory Routing Mustafa Munir, Huy Vu, Shreyas Misra, Rohit Jena, Sajad Norouzi, Ali Taghibakhshi * , Anis Ahmad, Anjul Patney, Pavlo Molchanov, Nima Tajbakhsh * Project Lead Abstract: Video diffusion is computationally expensive, as it requires executing a large model across many denoising steps. Even with step-distillation, inference remains expensive because every distilled step still requires a costly model evaluation. We present TRACK: TRajectory-Aware Capacity routing via top-K selection, a heterogeneous denoising strategy that switches between compatible large and small models at selected steps, reducing the average cost per denoising evaluation. The switching steps are determined using a calibration process. TRACK first rolls out a reference trajectory with the large model. Then at each step, the small model’s prediction is also collected and compared against the large model’s prediction to obtain a relative disagreement score. Both models receive the same latent, timestep, conditioning, and guidance inputs. Aggregating this signal over a calibration set produces a disagreement score map across diffusion steps, which determines a switching policy for an efficient inference process: quality-sensitive steps keep using the large model, while steps with low disagreement scores are routed to the small model. Inference executes only the selected model at each step, requiring no retraining, architecture or scheduler changes, or online dual-model evaluation. Across Wan 2.1, Cosmos 3, TurboDiffusion, and FastVideo, TRACK yields 1.95×, 2.04×–2.73×, 2.69×, and 2.17×speedups, respectively, with comparable aggregate quality and high diversity retention. TRACK thereby establishes automated, training-free model switching as a practical acceleration paradigm for video diffusion. Introduction Video diffusion models synthesize high-fidelity visual content with realistic motion, but inference latency remains a primary bottleneck for practical deploy- ment [7,8,11,13,31]. Advanced systems such as Wan 2.1 [37] and Cosmos 3 [1] achieve high physical re- alism and temporal consistency, yet sampling requires repeatedly executing multi-billion-parameter models over high-dimensional spatiotemporal latents. Model families often provide smaller, lightweight checkpoints that run significantly faster, though at the cost of lower generation quality. This raises a natural deploy- ment question: can we selectively combine the speed of the small model with the quality of the large one? Most diffusion acceleration methods reduce the num- ber of denoising evaluations through distillation or specialized few-step objectives [32,34,41], or reduce the cost within each evaluation through caching, streaming, model-sharing, and sparse com- putation [4,17,24,27,42,43]. These directions are essential, but they leave a complementary source of redundancy unexploited. After a sampler and step count have been selected, the remaining evaluations are still assigned to one model at uniform cost. The model switching approach instead reduces the aver- age model cost per remaining step. It can therefore multiply the gains from step reduction rather than compete with them. Allocating heterogeneous model capacity across video diffusion trajectories introduces distinct efficiency and quality challenges. Naive capacity allocation (i.e., model switching) based on rigid or heuristic temporal boundaries or simple monotonic handoffs fails to cap- ture the complex spatiotemporal dynamics of video generation. For example, two models can exhibit similar aggregate prediction errors while disagreeing on spatial detail, camera motion, or frame-to-frame consistency at specific timesteps. Furthermore, un- calibrated capacity reduction risks altering structural commitment and collapsing sample diversity across seeds. Therefore, a practical efficiency framework must identify model-switchability at step-level for a given checkpoint pair. This helps dynamically pro- tect quality-sensitive steps with a large model while offloading switchable steps to the smaller model. We introduce TRACK: TRajectory-Aware Capacity routing via top-K selection, an automated framework that turns the trajectory capacity allocation problem into an offline calibration problem. Given indepen- dently trained large and small checkpoints sharing a latent space and scheduler (e.g., large 14B and small 1.3B checkpoints of the Wan model family), TRACK rolls out an all-large reference trajectory on a cali- © 2026 NVIDIA. All rights reserved. arXiv:2609.30096v1 [cs.CV] 24 Sep 2026 Accelerating Video Diffusion via Training-Free Trajectory Routing Figure 1|Training-Free Trajectory-Aware Capacity routing. Top: Offline calibration rolls out the all-large-model reference trajectory and, at every step, evaluates the large and small checkpoints using the same latent푥 푡 , timestep, conditioning, and guidance inputs. Their guided-prediction disagreement is aggregated across calibration prompts to produce a switching policy that minimizes disagreement scores. Bottom: Online inference applies that policy while preserving the shared latent representation and scheduler update: quality-sensitive steps keep using the large model (green), while low disagreement steps use the small model (purple), and only one denoiser runs at each step. The method requires no retraining, architecture changes, scheduler changes, or online dual-model evaluation. bration set. At each step, both denoisers are evalu- ated on the exact same reference latent푥 푡 , timestep, conditioning, and guidance. We then measure their normalized relative disagreement score to determine steps where the small model accurately approximates the large model. Aggregating these scores produces a disagreement map, which leads to an offline switch- ing policy that routes quality-sensitive steps to the large model and less sensitive, switchable steps to the small model. During inference, exactly one denoiser runs at each step, requiring no retraining, architec- ture or scheduler modifications, or online dual-model evaluations. We evaluate TRACK across four major video diffusion pipelines spanning both many-step and step-distilled models. TRACK achieves 1.95×speedup on Wan 2.1 and 2.04×–2.73×on Cosmos 3 while maintaining vi- sual quality. On few-step distilled pipelines, TRACK yields 2.17×speedup on three-step FastVideo and 2.69×on four-step TurboDiffusion, showing that our model switching policy is effective with aggressive step reduction to maximize inference speed. Be- yond latency reduction gains, our spatial and latent- temporal frequency analyses demonstrate that pre- diction disagreement is phase-dependent, explaining intuitively why intermediate timesteps safely tolerate reduced model capacity. Finally, multi-seed evalua- tions confirm that calibrated, disagreement-guided switching preserves the diversity across generated out- puts. TRACK thus establishes training-free model switching as a practical acceleration paradigm that optimizes inference efficiency while maintaining qual- ity and diversity. We summarize our contributions as follows: •We introduce TRACK, a training-free het- erogeneous denoising strategy that generalizes prior large–small video diffusion switching from a single permanent handoff to disagreement- calibrated, fixed-budget allocation at individual denoising steps, enabling contiguous and non- contiguous schedules without modifying weights, architectures, or schedulers. •We validate TRACK across four video diffusion families spanning many-step and distilled few- step pipelines, obtaining 1.95×–2.73×speedups at comparable visual quality and similar diversity retention as the all-large model baseline. • We provide spatial and latent-temporal frequency analyses that explain the phase-dependent sub- stitutability of large and small denoisers. Related Work Diffusion Models for Visual Generation Denoising diffusion models learn a reverse process that transforms noise into samples through a sequence of denoising updates [13,33]. Improved training and guidance made diffusion competitive for high-fidelity 2 Accelerating Video Diffusion via Training-Free Trajectory Routing conditional image synthesis [7,12], while latent dif- fusion reduced computation by moving the reverse process from pixel space to a learned latent represen- tation [31]. Video diffusion extends this framework with temporal modeling so that appearance, structure, and motion remain coherent across frames [8,11]. Re- cent model families such as Wan 2.1 and Cosmos 3 demonstrate the quality and breadth of modern video generation systems [1,3,37]. However, their high- dimensional spatiotemporal latents and heavy multi- billion parameter denoisers (e.g., Wan 14B, Cosmos 3 - Super 64B) make repeated sampling evaluations a primary deployment efficiency bottleneck. Efficient Diffusion Inference Diffusion acceleration generally acts on either the total number of denoising steps or the cost of each in- dividual step. Step-reduction methods use distillation or specialized training objectives to compress sam- pling trajectories. Faster ODE solvers such as DPM- Solver [20] reduce the number of required steps with- out retraining, while distillation-based methods in- cluding Distribution Matching Distillation, SD-Turbo, SDXS, and LCM push further toward one- or few- step generation [21,32,34,41]. While these methods reduce latency, they require additional training and can introduce model- and budget-dependent quality tradeoffs. Other techniques optimize individual forward passes through spatial redundancy reduction, feature caching, and streaming execution [4,17,18,22,24,45]. FastVideo and TurboDiffusion combine few-step sam- pling with video-specific model and system optimiza- tions [42,43]. TRACK is complementary to both categories: it retains the target sampler and step count while reducing the average model evaluation cost across the denoising process. Its application to three-step FastVideo and four-step TurboDiffusion directly demonstrates this complementarity with step reduction. Adaptive and Multi-Model Diffusion The diffusion denoising process exhibits distinct gener- ation characteristics across timesteps. Spectral analy- ses show that deep networks process spatial frequency bands non-uniformly [15,23,28,30,38], and diffusion trajectories generally progress from coarse structural formation toward finer detail refinement [27]. This timestep dependence has motivated multi-expert and adaptive-computation approaches that allocate spe- cialized denoisers, frequency components, or subnet- works to different portions of the sampling trajec- tory [2,36,39]. Such methods, however, typically re- quire specialized training, architectural modifications, auxiliary control mechanisms, or additional expert parameters. Related ideas arise in model stitching, where representations from networks of different ca- pacities are connected to obtain flexible efficiency– accuracy trade-offs [25,26]. T-Stitch applies model switching to image diffusion by executing a single, one-way handoff—using a small model for the ini- tial steps before switching to a large model for the remainder of the trajectory [27]. Other methods like SRDiffusion [6] accelerate video diffusion on Wan [37] and CogVideoX [40] by switch- ing from a large to a small model based on a runtime latent-change signal computed from the large model alone, enforcing a permanent one-way handoff. This approach has two limitations. First, it prevents the large model from resuming even when disagreement rises again, which our frequency analysis in Figure 4 shows occurs in the final denoising steps. Second, since the switching criterion is based only on the large model’s predictions, it does not take into account the small model’s predictions; therefore, it cannot directly measure how well the small model approximates the large one at a given step. TRACK addresses both limitations: it computes cross-model disagreement on shared reference latents offline, and derives a flexible policy to switch back and forth between models when needed. Other methods like [16] anneal between the score functions of a base model and its reward-fine-tuned counterpart over the denoising trajectory to preserve complementary characteristics of the two models. TRACK differs in both how its routing policy is ob- tained and the setting in which it is applied. Rather than using prescribed temporal schedules or learning adapters, experts, or control modules, TRACK eval- uates compatible, independently trained large and small checkpoints on identical reference latents and uses the disagreement scores from their predictions to derive an offline, per-step switching policy. At inference, only the selected denoiser is evaluated at each step, requiring no retraining, architectural mod- ification, or online dual-model evaluation. While prior work has demonstrated training-free large–small switching for video diffusion via a sin- gle permanent handoff, TRACK generalizes this to disagreement-calibrated, fixed-budget allocation at individual denoising steps, enabling contiguous and non-contiguous schedules and compatibility with step- distilled pipelines. Proposed Methodology TRACK accelerates video diffusion by applying a pre- computed step-dependent model-capacity schedule 3 Accelerating Video Diffusion via Training-Free Trajectory Routing without modifying weights or altering schedulers. Fig- ure 1 illustrates the overall approach. The method op- erates in two stages: an offline calibration phase that measures step-level model disagreement on a shared reference trajectory and an execution phase that eval- uates exactly one denoiser per step according to a calibrated switching policy. The following section 3.1 formalizes the compatible sampling framework, while section 3.2 details reference-trajectory calibration and derives the disagreement-based switching policy. Preliminaries Video diffusion sampling. Let푥 푡 푖 ∈R 퐶×퐹×퐻×푊 denote a noisy video latent at denoising-step index 푖, where퐶,퐹,퐻, and푊denote the channel, frame, height, and width dimensions, respectively. The sampler follows a decreasing sequence of scheduler timesteps 푡 0 > 푡 1 >· > 푡 푁−1 . Given a large model퐿and a small model푆, along with prompt conditioning휏, negative prompt∅, and guidance scale푤, the denoiser푚∈퐿,푆yields the guided prediction (with classifier-free guidance): 푝 푚 (푥 푡 푖 ,푡 푖 ) = 푓 푚 (푥 푡 푖 ,푡 푖 ,∅) + 푤 (︀ 푓 푚 (푥 푡 푖 ,푡 푖 ,휏 )− 푓 푚 (푥 푡 푖 ,푡 푖 ,∅) )︀ , (1) The scheduler maps this prediction to the next latent: 푥 푡 푖+1 =풮(푥 푡 푖 ,푝 푚 (푥 푡 푖 ,푡 푖 ),푡 푖 ,ℎ 푖 ),(2) whereℎ 푖 denotes any scheduler history required by a multistep solver. TRACK preserves the original timestep sequence, prediction target, scheduler, and scheduler-state evolution; it alters only which denoiser 푚∈퐿,푆 supplies 푝 푚 (푥 푡 푖 ,푡 푖 ) in Eq. (2). Compatible model pairs. TRACK operates on pairs of large and small checkpoints that can be seam- lessly interchanged within the same sampling tra- jectory. They must share the latent representation, output shape, conditioning interfaces, prediction tar- get, and scheduler semantics. Although their inter- nal architectures and parameter counts may differ, their guided predictions defined in Eq.(1)must be semantically identical for the scheduler, ensuring ei- ther checkpoint can validly advance the shared latent state in Eq.(2). In practice, this applies to diffusion model families which have multiple model sizes but are based on a similar training and inference process. Offline Per-Step Capacity Scheduling. An offline-derived switching policy휋:0,...,푁 −1→ 퐿,푆assigns a model size to each denoising-step index. Inference then follows 푥 푡 푖+1 =풮 (︀ 푥 푡 푖 ,푝 휋(푖) (푥 푡 푖 ,푡 푖 ),푡 푖 ,ℎ 푖 )︀ .(3) With퐶 푚 denoting the wall-clock latency of a single evaluation of model푚on the target hardware, the total denoising cost of the policy is approximately ∑︀ 푖 퐶 휋(푖) , excluding shared text encoding, decoding, and scheduler overhead. The objective is therefore to minimize this total denoising cost by deriving a policy 휋that selectively assigns the small checkpoint to switchable steps while retaining the large checkpoint at quality-sensitive steps. TRACK derives this휋 offline from a small one-time calibration set, then inference executes directly from the calibrated offline policy with no further online overhead. Disagreement-Based Switching Policy Reference-Trajectory Calibration Let풟=푑 푗 푀 푗=1 be a small calibration set. For each prompt푑 푗 , we initialize the pipeline normally and roll out the complete trajectory using only the large checkpoint. Immediately before each scheduler up- date, we evaluate both small and large checkpoints on the same reference latent푥 (푗) 푡 푘 , timestep푡 푘 , con- ditioning, negative conditioning, and guidance scale. The paired predictions therefore differ only in the de- noiser that produced them. After recording the pair, the large-model prediction advances the trajectory, ensuring that every subsequent probe remains on the all-large reference path. This procedure produces one paired large–small pre- diction at every denoising step and for every calibra- tion prompt. It does not modify either checkpoint or the underlying sampler. Although both denoisers are evaluated during calibration, this cost is incurred only offline and is not part of the online inference. Normalized Prediction Disagreement For calibration prompt푗and denoising step푘, both the small and large model are evaluated on the same all-large reference latent푥 (푗) 푡 푘 , yielding푝 푆 (푥 (푗) 푡 푘 ,푡 푘 ) and 푝 퐿 (푥 (푗) 푡 푘 ,푡 푘 ), respectively. We first compute the nor- malized guided-prediction disagreement score. Here, the score is normalized by the prediction magnitude of the large model: 푟 푗,푘 = ⃦ ⃦ ⃦ 푝 푆 (푥 (푗) 푡 푘 ,푡 푘 )− 푝 퐿 (푥 (푗) 푡 푘 ,푡 푘 ) ⃦ ⃦ ⃦ 2 max (︁ ⃦ ⃦ ⃦ 푝 퐿 (푥 (푗) 푡 푘 ,푡 푘 ) ⃦ ⃦ ⃦ 2 ,휖 )︁ (4) where푡 푘 is the scheduler timestep at denoising-step index푘, and휖 >0 is a small constant for numerical 4 Accelerating Video Diffusion via Training-Free Trajectory Routing ModelConfiguration Switched / Total Steps Speedup Flicker ↑ Motion ↑ Subject ↑ Background ↑ Wan 2.1 [37] All-Large Model (14B)0/501.00×96.7198.3194.6294.79 TRACK30/501.95×96.8698.3994.8194.99 Cosmos 3 [1] All-Large Model (Super)0/351.00×98.9699.4196.7795.39 TRACK (Super/Nano)24/352.04×99.0699.4696.9395.55 TRACK (Super/Edge)24/352.73×98.3398.9897.2495.04 TurboDiffusion [42] All-Large Model (14B)0/41.00×97.2698.6494.1093.32 TRACK3/42.69×96.9498.4294.1093.48 FastVideo [43] All-Large Model (14B)0/31.00×98.1999.1595.8195.91 TRACK2/32.17×97.9999.1296.0796.16 Table 1|Speed–quality tradeoffs across video pipelines. Comparison between All-Large (baseline), and the TRACK method (ours) across full model pipelines (Cosmos 3 [1] and Wan 2.1 [37]) and step-distilled pipelines (TurboDiffusion’s Wan 2.1 [42], FastVideo’s Wan 2.1 [43]). Performance is evaluated on the same 250 prompts for all configurations using VBench’s temporal flicker, motion smoothness, subject consistency, and background consistency [14].Our method shows significant speed-up at comparable aggregate quality. stability that prevents division by zero when the large- model prediction norm is negligible. Then, TRACK computes the mean normalized score across푀cali- bration prompts as the final disagreement score: 푞 푘 = 1 푀 푀 ∑︁ 푗=1 푟 푗,푘 ,(5) Lower푞 푘 indicates closer large–small agreement and therefore a safer candidate for using small-model exe- cution instead of large-model. Constrained Trajectory Divergence Minimiza- tion using Top-K Switching The calibration score provides a non-negative trajec- tory divergence surrogate using the relative difference between the guided predictions of the small and large models. Let푊 푘 =푞 2 푘 be a squared divergence cost associated with choosing the smaller model, and let 훽 푘 denote a relaxed switching variable between the small model푆and large model퐿for denoising iter- ation푘. We associate a benefit of−퐶 0 ∑︀ 푘 훽 푘 with using the small model, where퐶 0 is a user-specified constant representing a cost we want to minimize (e.g. latency on specific hardware). This gives us a box-constrained optimization problem: argmin 훽 0 ,...,훽 푁−1 ∈[0,1] ∑︁ 푘 (푊 푘 − 퐶 0 )훽 푘 (6) where 푁 is the total number of denoising steps. Although Eq. (6) is a continuous optimization prob- lem, its objective is linear and separable in훽 푘 . There- fore, for 푊 푘 ̸= 퐶 0 , an optimal solution is 훽 ⋆ 푘 = ︃ 1, 푊 푘 < 퐶 0 , 0, 푊 푘 > 퐶 0 . (7) When푊 푘 =퐶 0 , any value in [0,1] is optimal; in particular, an integral solution can always be selected. For a fixed switching budget퐾, TRACK routes the퐾 steps with the lowest calibrated disagreement scores 푞 푘 to the small model and retains the large model at the remaining steps. Since푞 푘 ≥0, ranking by푞 푘 is equivalent to ranking by푊 푘 =푞 2 푘 , and choosing 퐶 0 between the퐾-th and (퐾+1)-th smallest푊 푘 makes the threshold solution in Eq. (7) select exactly these퐾steps; specifying퐶 0 is therefore equivalent to specifying퐾. This special case of the continuous relaxation thus admits a discrete optimal solution without a separate mixed-integer formulation, and provides direct control over the compute budget by specifying the number, or equivalently the fraction, of denoising steps assigned to the small model. Inference During inference, generation strictly follows the of- fline pre-computed policy휋via Eq.(3). The pipeline evaluates exactly one checkpoint per step, seamlessly passing the shared latent, conditioning, and sched- uler state between the two models. Because the step allocation is determined entirely offline, inference re- quires no learned routing modules or simultaneous dual-model evaluations. Assuming a single-pass la- tency of퐶 푚 for checkpoint푚, the total denoising cost reduces directly to ∑︀ 푖 퐶 휋(푖) , plus the standard scheduler overhead. Because policy calibration is a one-time offline setup for a given sampler config- 5 Accelerating Video Diffusion via Training-Free Trajectory Routing Figure 2|Speed–quality tradeoffs across video pipelines. Each panel compares the all-large model (baseline) and TRACK (ours) switching policy. All models retain comparable aggregate quality at the selected operating points in most quality dimensions. uration, it introduces zero overhead during actual deployment. Experimental Results Experimental Setup Pipelines and Switching Budgets. We evaluate TRACK on four text-to-video pipelines using their native inference pipelines. We pair the large and small checkpoints of Wan 2.1 (14B/1.3B, 50 steps) [37], Cos- mos 3 (Super/Nano and Super/Edge, 35 steps) [1], FastVideo (14B/1.3B, 3 steps) [43], and TurboDiffu- sion (14B/1.3B, 4 steps) [42]. All pipelines generate 81-frame, 480×832 videos. Through an ablation study (Appendix A), we select the operating-point switching budget퐾as the maximum number of small- model steps that maintains quality comparable to the all-large baseline. Evaluation Protocols. We evaluate speed and quality across 250 text-video prompt pairs from VBench [14], EvalCrafter [19], T2V-CompBench [35], and manually designed prompts. For quality and temporal consistency, we report VBench’s tempo- ral flicker, motion smoothness, subject consistency (DINO ViT-B/16 [5]), and background consistency. Generative diversity is measured via mean pairwise DreamSim distance [9] over ten prompts with eight shared seeds per policy. Figure 3|Human evaluation study. Human eval- uators compare TRACK outputs against the corre- sponding all-large model across Wan 2.1, Cosmos 3, TurboDiffusion, and FastVideo. Each horizontal bar reports the percentage of comparisons rated as TRACK Win, Tie, or Lose against the all-large base- line. Human Evaluation. We conducted a pairwise hu- man evaluation comparing TRACK switching against the all-large baseline across all four pipelines. Evalu- ators were shown side-by-side videos generated from the same prompt and asked to judge which was better on two axes: visual quality and prompt adherence. We collected a minimum of 150 judgments per pipeline (30 prompts × 5 raters each). Quantitative and Qualitative Results Table 1, together with Figure 2, summarizes the se- lected operating point for each pipeline. TRACK accelerates all four families, spanning 50-, 35-, four-, and three-step trajectories, without changing their na- tive schedulers or step counts. On the 50-step Wan 2.1 pipeline, the calibrated switching policy achieves a 1.95×speedup. For Cosmos 3, the policy yields a 2.04×–2.73×speedup depending on whether Nano or Edge model is chosen for the smaller model. TRACK is also effective with step-distilled pipelines, achieving a 2.17×speedup on 3-step FastVideo and a 2.69× speedup on 4-step TurboDiffusion. In terms of quality, Wan 2.1 preserves subject and background consistency relative to the all-large model. Cosmos 3 retains subject and background consistency 6 Accelerating Video Diffusion via Training-Free Trajectory Routing Figure 4|Analysis of Wan 2.1’s disagreement scores across denoising steps. Disagreement scores along the denoising trajectory for Wan 2.1 (14B vs. 1.3B), averaged across 150 prompts at 50 steps. (A) Scalar disagreement score. The green lines denote top-K steps with lowest disagreement score suitable for switching. (B, C) The same score decom- posed over spatial and temporal frequency bands. as well, while TurboDiffusion and FastVideo are sim- ilarly comparable across all metrics, demonstrating that our model switching approach remains effective after few-step distillation. These results show that TRACK reduces the cost of the denoising evaluations on both conventional many-step sampling and aggressive step distillation. Crucially, offline calibration on a small prompt set generalizes seamlessly to the full evaluation bench- mark without manual step tuning. This demonstrates that a disagreement-based switching policy provides a stable, prompt-independent criterion for automated offline model allocation across both many-step and step-distilled video diffusion systems. User Study Figure 3 summarizes the results of the human evalua- tion study. On prompt adherence, evaluators found the two pipelines indistinguishable in the majority of comparisons across all models (67–79% tie rate), with win and lose rates near parity. On visual qual- ity, judgments were more evenly distributed, but the TRACK switching approach matched or edged the full model: TRACK wins are comparable or slightly exceeded full-model wins on TurboDiffusion (37.7% vs. 34.7%), FastVideo (38% vs. 35.3%), Cosmos 3 (32.0% vs. 26.0%), and Wan 2.1 (33.3% vs. 38.0%), with Cosmos 3 showing the largest tie share (42.0%). Across both dimensions, no model family showed a consistent preference for the full model, confirming that the speedups reported in Table 1 come without a perceptible quality penalty. Figure 5 illustrates qualitatively the videos generated by our method and baseline. Further qualitative results can be found in Figure 8 in the Appendix. Analysis of Disagreement across Denoising Steps We visualize the disagreement scores across diffusion steps to understand at which steps the small model can replace the large one. Figure 4A shows the dis- agreement curve for Wan 2.1 over a 50-step denoising trajectory, averaged across 150 prompts. The error is high at the initial and final steps and low in between, with the minimum falling in the latter half of the trajectory. We observe the same pattern for Cosmos 3 (Figure 7 in Appendix). For these models, switching to the small model is therefore safest in the middle steps. To understand what drives this landscape, we de- compose the disagreement scores over spatial and temporal frequency bands (low, mid, high) using an orthonormal Fourier transform. Figures 4B and 4C show that the early-step disagreement is dominated by low-frequency components, whereas the late-step error is dominated by mid and high frequencies. This is consistent with the coarse-to-fine nature of the denoising process: early steps commit to global struc- ture, so any disagreement between the two models appears in the low-frequency band, while late steps refine fine-grained detail, shifting the disagreement to higher frequencies. The disagreement is lowest in the middle of the trajectory - mostly on the latter half, where the coarse structure is already fixed and fine de- tail has not yet been resolved. Hence, these steps are the most promising for switchability without causing significant disagreement between the large and small models. Appendix reports additional disagreement landscapes and frequency analyses for Cosmos 3 [1], TurboDiffusion [42], and FastVideo [43]. Equal-Budget Policy Ablations and Diversity Retention Analysis To isolate the effect of where model switching occurs from the overall amount of compute reduction, we compare TRACK against three equal-budget switch- ing baselines. This ablation helps us understand how our policy compares to other naive or heuristic ap- proaches at a pre-determined number of switching steps. Specifically, each policy replaces exactly퐾 out of the푁denoising steps with the small model: First-퐾replaces the initial퐾steps, Last-퐾replaces the final퐾steps, and Random-퐾randomly selects 7 Accelerating Video Diffusion via Training-Free Trajectory Routing PolicyFlicker ↑Motion ↑Subject ↑Background ↑DreamSim Diversity Retention ↑ All-Large96.3098.0894.2894.95100.0% All-Small96.35 (+0.05%)97.63 (−0.46%)93.83 (−0.48%)94.17 (−0.82%)79.4% (−20.6%) First-퐾95.39 (−0.95%)96.90 (−1.20%)93.77 (−0.54%)94.14 (−0.85%)84.6% (−15.4%) Last-퐾96.04 (−0.27%)97.89 (−0.19%)94.05 (−0.24%)94.51 (−0.46%)97.8% (−2.2%) Random-퐾95.95 (−0.36%)97.55 (−0.54%)93.91 (−0.39%)94.49 (−0.48%)85.7% (−14.3%) TRACK Top-퐾 96.47 (+0.18%) 98.14 (+0.06%) 94.41 (+0.14%) 94.88 (-0.07%)98.3% (-1.7%) Table 2|Equal-budget switching policy comparison. We compare TRACK Top-퐾policy against heuristic policies (First-퐾, Last-퐾, and Random-퐾) on Wan 2.1 with퐾= 30, with each policy replacing exactly퐾denoising steps with the small model. All switching policies therefore use the same number of large- and small-model evaluations. Temporal flicker, motion smoothness, subject consistency, and background consistency are evaluated on a 150-prompt evaluation set. DreamSim diversity [9] is evaluated separately using ten prompts with eight shared seeds per policy. TRACK Top-퐾provides the best performance with the same switching budget. All-Large (Baseline)TRACK (Ours) "A white horse gallops along a beach, mane flowing, waves breaking in the background." "A sea turtle swims over a coral reef, sunlight rays moving through clear water." "A robotic arm assembles a tiny circuit board with precise movements in a clean high-tech lab." Figure 5|Qualitative comparison. Comparing frames from videos generated with TRACK switching policy and the all-large 14B model for Wan 2.1. Our approach preserves the scene composition, subject identity, and motion pattern of the large-model outputs. 퐾 steps. In contrast, TRACK Top-퐾selects the퐾 steps with the lowest calibrated disagreement scores 푞 푘 . Here, First-퐾is aligned with the T-stitch [27] method, in which the first steps use the small model then switch entirely to the large model for the rest. On the other hand, Last-퐾is conceptually similar to SRDiffusion’s [6] approach, in which the first steps use large model while using small model for the rest. One nuanced difference is, Last-퐾fixes the number of switching steps while SRDiffusion dynamically de- termines the number switching steps based on input prompts. We also conduct additional experiments comparing our method and SRDiffusion [6] in the Appendix. All policies use identical prompts and total numbers of large- and small-model evaluations. Besides quality comparison, we also evaluate generative sample diver- sity without conflating it with semantic or temporal drift. We measure mean pairwise DreamSim dis- tance [9] over eight seeds, four frames per video, and ten prompts. Following established diversity proto- cols [10], we adapt five broad prompt categories from their benchmark and supplement them with three motion-heavy prompts and two control prompts. Table 2 summarizes quality and diversity metrics across these switching policies on Wan 2.1 (퐾= 30/50). We find that in both quality and diversity re- tention metrics, our approach outperforms all switch- ing policies compared. We outperform First-K by a particularly large margin in diversity retention, likely because generation diversity primarily stems from the large model’s predictions in the initial steps [10], which First-K has replaced with the small model. Compared to Last-K (and additionally SRDiffusion in the Appendix Section D), our method also out- performs across quality and diversity metrics. This might be explained by our analysis in section 4.4, which shows that the last diffusion steps contain high disagreement between large and small model when finalizing the videos’ high-frequency details - hence, the pipeline should switch back to the large model. This observation once again illustrates our approach’s flexibility when switching models across steps, com- pared to other methods’ one-time handoffs—either small-to-large or large-to-small. Table 2 confirms the effectiveness of the switching policy built on evidence- 8 Accelerating Video Diffusion via Training-Free Trajectory Routing based calibration analysis over heuristic policies. Regarding diversity retention, since in our approach, the switchable steps are mostly in the middle diffusion steps, all initial steps use the large model. This might explain why our method maintains high diversity retention compared to other methods, confirming the findings in [10]. Further representative multi-seed visual generations, as well as additional evaluations following the diversity evaluation protocol from [10], are reported in Section F of the Appendix. Conclusion We introduced TRACK, a training-free approach for converting compatible, independently trained large and small video diffusion checkpoints into a switching denoising system. TRACK calibrates the checkpoints on the same all-large reference latents, uses a normal- ized guided-prediction disagreement score to identify switchable denoising steps, and executes a switching policy with exactly one model evaluation per step. It therefore changes neither model weights nor sched- uler behavior and introduces no online dual-model comparison. Across Wan 2.1, Cosmos 3, FastVideo, and TurboD- iffusion, TRACK provides 1.95×–2.73×speedups at selected comparable-quality operating points. The gains on three- and four-step pipelines show that our method complements step-distillation by further low- ering the average cost of the distilled steps. Spatial and latent-temporal analyses further show that large– small disagreement is structured across the trajectory, providing an empirical and intuitive explanation for model-specific step selection. We further tested our approach against other heuristic switching policies with the same step budget and demonstrated that our method outperforms these baselines in both quality and diversity metrics. Together, these results estab- lish TRACK’s model-switching policy as a practical additional axis for accelerating video diffusion while maintaining quality. References [1]Aditi, Niket Agarwal, Arslan Ali, Jon Allen, Martin Antolini, Adeline Aubame, Alisson Azzolini, Junjie Bai, Maciej Bala, Yogesh Balaji, Josh Bapst, et al. Cosmos 3: Omnimodal world models for physical AI. arXiv preprint arXiv:2606.02800, 2026. [2]Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et al. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324, 2022. [3]Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. In International Conference on Learning Representations, 2024. [4] Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 4599–4603, 2023. [5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 9650–9660, 2021. [6] Shenggan Cheng, Yuanxin Wei, Lansong Diao, Yong Liu, Bujiao Chen, Lianghua Huang, Yu Liu, Wenyuan Yu, Jiangsu Du, Wei Lin, and Yang You. Srdif- fusion: Accelerate video diffusion inference via sketching-rendering cooperation. arXiv preprint arXiv:2505.19151, 2025. [7] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780– 8794, 2021. [8] Patrick Esser, Johnathan Chiu, Parmida Atighe- hchian, Jonathan Granskog, and Anastasis Germani- dis. Structure and content-guided video synthe- sis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7346–7356, 2023. [9]Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. In Advances in Neural Information Processing Systems, 2023. [10]Rohit Gandikota and David Bau. Distilling diversity and control in diffusion models. In 2026 IEEE/CVF Winter Conference on Applications of Computer Vi- sion (WACV), pages 1304–1313. IEEE, 2026. [11]Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662, 2023. [12] Jonathan Ho and Tim Salimans. Classifier-free dif- fusion guidance. arXiv preprint arXiv:2207.12598, 2022. [13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denois- ing diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020. 9 Accelerating Video Diffusion via Training-Free Trajectory Routing [14]Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, Yao- hui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. [15] Aapo Hyvärinen, Jarmo Hurri, and Patrick O. Hoyer. Natural Image Statistics: A Probabilistic Approach to Early Computational Vision. Springer, 2009. [16]Rohit Jena, Ali Taghibakhshi, Sahil Jain, Gerald Shen, Nima Tajbakhsh, and Arash Vahdat. Eluci- dating optimal reward-diversity tradeoffs in text-to- image diffusion models. In 2025 IEEE/CVF Win- ter Conference on Applications of Computer Vision (WACV), pages 232–242. IEEE, 2025. [17]Akio Kodaira, Chenfeng Xu, Toshiki Hazama, Takanori Yoshimoto, Kohei Ohno, Shogo Mitsuhori, Soichi Sugano, Hanying Cho, Zhijian Liu, Masayoshi Tomizuka, et al. Streamdiffusion: A pipeline-level solution for real-time interactive generation. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 12371–12380. IEEE, 2025. [18] Feng Liang, Akio Kodaira, Chenfeng Xu, Masayoshi Tomizuka, Kurt Keutzer, and Diana Marculescu. Looking backward: Streaming video-to-video trans- lation with feature banks. In International Con- ference on Learning Representations, pages 46425– 46445, 2025. [19] Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 22139–22149, 2024. [20]Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chunxiao Li, and Jun Zhu. DPM-Solver: A fast ODE solver for diffusion probabilistic model sam- pling in around 10 steps. In Advances in Neural Information Processing Systems, 2022. [21]Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. In In- ternational Conference on Learning Representations, 2024. [22]Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deep- Cache: Accelerating diffusion models for free. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. [23]Mustafa Munir, Guihong Li, Md Mostafijur Rahman, Alex Zhang, and Radu Marculescu. From data to design: Leveraging frequency statistics for efficient neural network architectures. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion Workshops (CVPRW), pages 3199–3209. IEEE, 2025. [24]Mustafa Munir, Sophia Zalewski, Shiqiu Liu, David Tarjan, Sushmitha Belede, Anjul Patney, and Radu Marculescu. Smoothdiffusion-ve: Real-time gen- erative video editing using adaptive feature cache. In 2026 IEEE/CVF Winter Conference on Applica- tions of Computer Vision (WACV), pages 8468–8478. IEEE, 2026. [25] Zizheng Pan, Jianfei Cai, and Bohan Zhuang. Stitch- able neural networks. In 2023 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 16102–16112. IEEE, 2023. [26]Zizheng Pan, Jing Liu, Haoyu He, Jianfei Cai, and Bohan Zhuang. Stitched vits are flexible vision back- bones. In European Conference on Computer Vision, pages 258–274. Springer, 2024. [27]Zizheng Pan, Bohan Zhuang, De-An Huang, Weili Nie, Zhiding Yu, Chaowei Xiao, Jianfei Cai, and Anima Anandkumar. T-Stitch: Accelerating sam- pling in pre-trained diffusion models with trajectory stitching. In International Conference on Learning Representations, pages 6103–6137, 2025. [28]Namuk Park and Songkuk Kim. How do vision trans- formers work? In International Conference on Learn- ing Representations, 2022. [29]Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In In- ternational Conference on Learning Representations, pages 1862–1874, 2024. [30]Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Proceedings of the 36th In- ternational Conference on Machine Learning, pages 5301–5310. PMLR, 2019. [31]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022. [32]Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach. Fast high-resolution image synthesis with latent adversar- ial diffusion distillation. In SIGGRAPH Asia 2024 Conference Papers, pages 1–11, 2024. [33]Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. 10 Accelerating Video Diffusion via Training-Free Trajectory Routing [34]Yuda Song, Zehao Sun, and Xuanwu Yin. SDXS: Real-time one-step latent diffusion models with image conditions. arXiv preprint arXiv:2403.16627, 2024. [35]Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, and Xihui Liu. T2v-compbench: A comprehensive benchmark for compositional text-to- video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 8406–8416, 2025. [36]Ali Taghibakhshi, Ruisi Cai, Saurav Muralidha- ran, Sharath Turuvekere Sreenivas, Ameya Sunil Mahabaleshwarkar, Marcin Chochowski, Akhiad Bercovich, Ran Zilberstein, Ran El-Yaniv, Yonatan Geifman, Daniel Korzekwa, Yoshi Suhara, Oluwa- tobi Olabiyi, Ashwath Aithal, Nima Tajbakhsh, and Pavlo Molchanov. Star elastic: Many-in-one reason- ing LLMs with efficient budget control. In Forty-third International Conference on Machine Learning, 2026. [37]Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative mod- els. arXiv preprint arXiv:2503.20314, 2025. [38]Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P. Xing. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 8684– 8694, 2020. [39]Xingyi Yang, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Diffusion probabilistic model made slim. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22552–22562. IEEE, 2023. [40]Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. CogVideoX: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. [41]Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribu- tion matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 6613–6623, 2024. [42] Jintao Zhang, Kaiwen Zheng, Kai Jiang, Haoxu Wang, Ion Stoica, Joseph E Gonzalez, Jianfei Chen, and Jun Zhu. Turbodiffusion: Accelerating video diffusion models by 100-200 times. arXiv preprint arXiv:2512.16093, 2025. [43]Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, Ion Stoica, Zhengzhong Liu, and Hao Zhang. Fast video generation with sliding tile attention. In Proceedings of the 42nd International Conference on Machine Learning, 2025. [44]Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipc: A unified predictor-corrector framework for fast sampling of diffusion models. Ad- vances in Neural Information Processing Systems, 36: 49842–49869, 2023. [45]Xuanlei Zhao, Xiaolong Jin, and Yang You. Real- time video generation with pyramid attention broad- cast. In Advances in Neural Information Processing Systems, 2024. 11 Accelerating Video Diffusion via Training-Free Trajectory Routing Accelerating Video Diffusion via Training-Free Trajectory Routing Supplementary Material Ablation on Determining the Switch- ing Budget (퐾) To understand the tradeoff between compute reduc- tion and generation quality, which helps determine the optimal switching budget, we conduct an ablation study varying the switching budget퐾(the number of denoising steps routed to the small model) across all four pipelines. For this analysis, we report the average of temporal flicker, motion smoothness, sub- ject consistency, and background consistency scores, providing a unified metric for visual fidelity. The switching policies are generated using the TRACK Top-퐾selection criterion derived in Section 3, mean- ing steps are routed in order of lowest normalized disagreement score. Figure 6|Effect of trajectory routed budget steps (퐾) on visual quality. We evaluate the aver- age of the temporal flicker, motion smoothness, sub- ject consistency, and background consistency scores from VBench across varying numbers of small-model steps (퐾). From this study, we can determine the maximum퐾used in the TRACK algorithm before the quality starts to degrade. Figure 6 reports the visual consistency across varying budgets. The results demonstrate a clear “plateau of switchability” for video diffusion models. As퐾 increases from 0 (the all-large baseline), visual con- sistency initially remains stable and can even im- prove. This confirms that a substantial portion of the denoising trajectory simply does not require the representational capacity of the largest network. However, as퐾approaches the total number of sam- pling steps푁, the capacity reduction begins to en- croach on quality-sensitive steps, causing a sharp Figure 7|TRACK disagreement score across denoising steps analysis across models. Dis- agreement score error for Cosmos 3 (A–C, 35 steps), TurboDiffusion (D–F, 4 steps), and FastVideo (G–I, 3 steps), averaged across prompts. Columns in order of: scalar disagreement score; spatial-band frequencies amplitude; temporal-band frequencies amplitude. degradation in visual fidelity. For instance, increasing the budget to퐾= 28 on Cosmos 3 or퐾= 40 on Wan 2.1 severely penalizes consistency. Extended Analysis of Disagreement Scores Across Denoising Steps Figures 7A–C show that Cosmos 3 closely follows the behavior of Wan 2.1. The disagreement score is high at the initial and final steps and reaches its minimum in the latter half of the trajectory. The band decomposition reproduces the same frequency handoff: early-step disagreement is concentrated in the low band, while the late-step rise is carried by the mid and high bands. The U-shaped disagreement landscape is therefore not specific to Wan 2.1, and switching is again best placed in the middle steps. TurboDiffusion [42] and FastVideo [43] show a different landscape (Figures 7D–I). The score is largest at the first step and decreases monotonically until the last, with no late-step rise. The band decomposition shows that the disagreement remains low-frequency dominated across all steps — the low band carries the majority of the error energy at every step. i Accelerating Video Diffusion via Training-Free Trajectory Routing Energy Consumption and Efficiency Gains Beyond accelerating inference latency, TRACK’s in- ference efficiency can be translated into substantial reductions in GPU energy consumption. This is a critical metric for the deployment of generative video models at scale, where continuous execution of multi- billion parameter denoisers incurs heavy power and thermal costs. Measurement Protocol. We measure the hard- ware energy consumption of the denoising loop using the NVIDIA Management Library (NVML). For each generated video푖, we read the cumulative GPU en- ergy counter immediately before and after the denois- ing trajectory. The per-video energy consumption in joules is calculated as: 퐸 푖 = NVML end − NVML start 1000 .(8) To ensure a rigorous evaluation, we compute the total percentage of energy saved as the ratio of aggregate means across the entire evaluation set, rather than the unweighted average of individual per-video percent- ages. Let ̄ 퐸 base and ̄ 퐸 TRACK denote the mean per- video energy consumption for the all-large baseline and the TRACK policy, respectively. The aggregate energy savings percentage is defined as: Savings (%) = (︂ 1.0− ̄ 퐸 TRACK ̄ 퐸 base )︂ × 100. (9) All measurements are recorded on a single NVIDIA A100 GPU, isolating the denoising loop and exclud- ing fixed overheads such as text encoding and VAE decoding. Results. Table 3 summarizes the energy reductions across all evaluated pipelines. TRACK cuts the GPU energy footprint by approximately half across the board. For many-step, high-capacity pipelines, the absolute energy savings are significant. On Wan 2.1 (14B to 1.3B), TRACK saves 194.57 kJ per video, translat- ing to 54.05 kWh saved per 1,000 videos generated (a 49.3% reduction). On Cosmos 3 (Super/Edge), TRACK reduces energy consumption by nearly 60%, saving 35.41 kWh per 1,000 videos. Crucially, TRACK remains highly effective even on heavily distilled pipelines where the total step count is already minimized. On the three-step FastVideo pipeline and four-step TurboDiffusion pipeline, TRACK yields 54.4% and 45.6% energy sav- ings, respectively. At scale, this prevents over 1 kWh Pipeline Steps (퐾/푁) Base (kJ) TRACK (kJ) Saved (kJ) Saved (%) 1K Vids (kWh) TurboDiff. [42]2/4 8.314.523.79 45.6% 1.05 FastVideo [43]2/3 9.444.305.14 54.4% 1.43 Cosmos 3 (S/N) [1] 24/35 204.22 97.07 107.15 52.5% 29.77 Cosmos 3 (S/E) [1] 24/35 213.40 85.93 127.46 59.7% 35.41 Wan 2.1 [37]30/50 394.43 199.86 194.57 49.3% 54.05 Table 3|GPU Energy Consumption and Sav- ings. Denoising energy is measured via NVML on a single NVIDIA A100 GPU. Percentage savings are calculated using the ratio of the aggregate means. TRACK reduces the energy footprint of video genera- tion by roughly 45% to 60%, saving up to 54 kWh per 1,000 videos on many-step models and over 1 kWh per 1,000 videos on aggressively step-distilled models. For Cosmos 3, S/N and S/E denote the Super/Nano and Super/Edge checkpoint configurations, respectively. of energy waste per 1,000 videos without requiring any additional training or distillation. In practice, video diffusion pipelines generate millions of videos, and these energy savings scale directly with the number of generated videos. i Accelerating Video Diffusion via Training-Free Trajectory Routing PolicyFlicker ↑Motion ↑Subject ↑Background ↑ DreamSim Diversity ↑ All-Large96.3098.0894.2894.95100.0% SRDiffusion (0.03) [6] 95.84 (−0.48%) 97.57 (−0.52%) 93.76 (−0.55%) 94.35 (−0.63%)87.1% (−12.9%) SRDiffusion (0.01) [6] 95.98 (−0.33%) 97.73 (−0.36%) 93.94 (−0.36%) 94.63 (−0.34%)92.6% (−7.4%) TRACK Top-퐾 (Ours) 96.47 (+0.18%) 98.14 (+0.06%) 94.41 (+0.14%) 94.88 (-0.07%)98.3% (-1.7%) Table 4|TRACK vs. SRDiffusion Comparison. We compare our TRACK Top-퐾policy (퐾= 30) and two threshold variants of SRDiffusion [6] on Wan 2.1. Bold indicates the best performing method. Additional Quantitative Results In this section, we provide a direct, isolated com- parison between our proposed TRACK switching policy and SRDiffusion [6]. SRDiffusion determines its switching point using an online, prompt-adaptive threshold based on the latent rate of change. We compare TRACK against two variants of SRDiffusion (using threshold values of 0.01 and 0.03). As shown in Table 4, TRACK outperforms the SRDif- fusion baselines across both video consistency metrics and sample diversity retention. Because SRDiffusion enforces a permanent, monotonic handoff to the small model, it struggles to preserve the high-frequency tem- poral details in the final denoising steps, leading to loss of DreamSim diversity. By dynamically placing the small model at optimal low-disagreement steps and returning to the large model when necessary, TRACK preserves the large model’s visual fidelity and diversity. Beyond the numerical metrics, visual inspection con- firms that the structural integrity of the generated videos remains intact. As shown in Figure 5, the TRACK switching policy seamlessly preserves the scene composition, subject identity, and motion pat- terns of the all-large baseline outputs compared to contiguous handoff methods. Additional Qualitative Results In Figure 8, we provide additional side-by-side video frame comparisons across all four model families: Wan 2.1, Cosmos 3, TurboDiffusion, and FastVideo. For each model, we show frames generated from the same prompt using the all-large baseline and the TRACK switching policy. Across all families, TRACK preserves scene composition, subject identity, and motion patterns of the large-model outputs, while delivering the speedups reported in Table 1. i Accelerating Video Diffusion via Training-Free Trajectory Routing All-Large (Baseline)TRACK (Ours) Wan "A stylish woman walks down a neon-lit Tokyo street after rain, reflections shimmering on the pavement." "A chef flips vegetables in a wok over a bright flame in a busy restaurant kitchen, steam rising dramatically." "A red sports car drifts around a wet mountain road curve at night, headlights cutting through mist." Cosmos3 "A stylish woman walks down a neon-lit Tokyo street after rain, reflections shimmering on the pavement." "A chef flips vegetables in a wok over a bright flame in a busy restaurant kitchen, steam rising dramatically." "A red sports car drifts around a wet mountain road curve at night, headlights cutting through mist." TurboDiffusion "A stylish woman walks down a neon-lit Tokyo street after rain, reflections shimmering on the pavement." "A chef flips vegetables in a wok over a bright flame in a busy restaurant kitchen, steam rising dramatically." "A red sports car drifts around a wet mountain road curve at night, headlights cutting through mist." FastVideo "A stylish woman walks down a neon-lit Tokyo street after rain, reflections shimmering on the pavement." "A chef flips vegetables in a wok over a bright flame in a busy restaurant kitchen, steam rising dramatically." "A red sports car drifts around a wet mountain road curve at night, headlights cutting through mist." Figure 8|Qualitative comparison across all four base models. Four frames at evenly-spaced timestamps are shown for three prompts generated with each model under two configurations: the full large-model pipeline (baseline) and our TRACK pipeline (Ours). Across all four models: Wan 2.1, Cosmos 3, TurboDiffusion, and FastVideo, TRACK’s switching policy preserves scene composition, subject identity, and motion patterns of the large-model outputs, while reducing inference latency. iv Accelerating Video Diffusion via Training-Free Trajectory Routing Extended Diversity Preservation Analysis When aggressively switching large-model evaluations with a smaller checkpoint, a critical concern is whether the system suffers from mode collapse or loss of sam- ple diversity across different initial noise seeds. As shown qualitatively in Figure 9, TRACK success- fully preserves the varied subjects, compositions, and styles of the all-large baseline across different random seeds, whereas naive heuristic policies like First-퐾 can severely shift or narrow the output modes. Following the diversity evaluation protocol established in recent literature [10], we measure the mean pairwise DreamSim distance across generated outputs. Higher DreamSim distances indicate greater compositional and semantic diversity across seeds. Evaluation Protocol and Baselines. We com- pute the DreamSim distance using five standardized prompt categories utilized in prior diffusion diversity studies [10] (e.g., sunset beach, puppy, futuristic city, person, and Van Gogh art). To isolate the impact of our switching policy, we compare video generation pipelines against the image-based SDXL baseline re- ported in [10]. Because video evaluation averages features across multiple frames (four frames per video across eight seeds), absolute DreamSim values differ across modalities; therefore, the primary metric of interest is the retention percentage relative to the all-large baseline. As reported in Table 5, TRACK preserves>95% of the original large-model diversity across all four video pipelines, successfully mirroring the high retention dynamics observed in image-based models like SDXL [29]. Mean Pairwise DreamSim (↑) PipelineBaseHybridRetention SDXL [29]0.3370.350103.9% FastVideo [43] 0.2710.284104.8% TurboDiff. [42] 0.6070.58195.7% Wan 2.1 [37] 0.5050.50299.4% Cosmos 3 [1] 0.5500.54899.5% Table 5|Sample Diversity Retention. We report mean pairwise DreamSim distance on a standardized 5-category prompt set. FastVideo, TurboDiffusion, Wan 2.1, and Cosmos 3 successfully reproduce the retention dynamics observed in 2D image diffusion, preserving>95% of large-model diversity when the highest capacity model is retained for critical denois- ing steps. v Accelerating Video Diffusion via Training-Free Trajectory Routing Figure 9|Representative multi-seed generations used in the diversity analysis. We show midpoint frames from four seeds per prompt, with each column using the same random seed across rows. Rows compare the all-large reference, a same-budget First-퐾schedule, and TRACK; both switching policies use the same number of small-model evaluations. The top grid shows two prompts for Wan 2.1 with deterministic UniPC sampling [44], while the lower grids show three-step FastVideo with deterministic UniPC sampling and four-step TurboDiffusion with stochastic rCM sampling. TRACK preserves the varied subjects, compositions, and styles of the all-large reference, whereas First-퐾 shifts or narrows its output modes. vi Accelerating Video Diffusion via Training-Free Trajectory Routing Limitations and Future Work TRACK, similar to other diffusion model switching methods, currently still requires storing both the large and small models simultaneously, increasing deploy- ment memory compared to one single-model pipeline. A natural direction for future work is to integrate TRACK with nested model families as introduced in the language domain [36], where the small denoiser is realized as a subnetwork of the large denoiser sharing the same parameters. Under such a design, the full and reduced capacity models are contained within a single checkpoint, eliminating the memory overhead of maintaining two separate models while preserving the flexibility switching benefits demonstrated here. vii