Paper deep dive

VALSE: Vertical Adaptive Layer Skipping for Efficient Inference in Large Language Models

Jia-Dong Zhang

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 121

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 10/13/2026, 3:48:27 AM

Summary

The paper introduces VALSE (Vertical Adaptive Layer Skipping for Efficiency), a method for efficient inference in Large Language Models that performs per-sample, non-contiguous layer skipping based on input difficulty. It establishes a theoretical framework including an Expected FLOPs formula, function-space inclusion theorems, and a structural duality with Mixture-of-Experts (MoE) architectures. VALSE uses a lightweight difficulty estimator to score inputs and per-layer gates to selectively skip redundant layers, aiming to reduce computational cost while maintaining performance.

Entities (13)

VALSE · method · 100%Difficulty Estimator · component · 95%Mixture-of-Experts · architecture · 95%Early Exit · method · 90%Expected FLOPs formula · theorem · 90%function-space superset · theorem · 90%SST-2 · dataset · 90%structural duality · concept · 90%DeepSeek-MoE · model · 85%GPT-4 · model · 85%LayerDrop · method · 85%Llama · model · 85%MoD · method · 85%

Relation Signals (10)

VALSE → isa → Vertical Adaptive Layer Skipping

confidence 100% · we propose VALSE (Vertical Adaptive Layer Skipping for Efficiency)

Mixture-of-Experts → achieves → width-wise sparsity

confidence 95% · MoE achieves width-wise sparse activation

VALSE → achieves → depth-wise sparsity

confidence 95% · vertical layer skipping achieves depth-wise sparsity

VALSE → hascomponent → Difficulty Estimator

confidence 95% · a lightweight difficulty estimator scores each input from the first few layers

VALSE → hascomponent → per-layer gates

confidence 95% · per-layer gates selectively skip redundant layers

VALSE → evaluatedon → SST-2

confidence 90% · A prototype-scale (12-layer) evaluation on SST-2 provides initial feasibility evidence

VALSE → isdualto → Mixture-of-Experts

confidence 90% · structural duality between VALSE and Mixture-of-Experts architectures

Early Exit → istypeof → truncation-based

confidence 90% · early exit... is truncation-based and cannot skip a middle layer

VALSE → →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:This paper establishes a theoretical framework for vertical adaptive layer skipping, proving three foundational results: (i) an Expected FLOPs formula (theorem 2) giving a closed-form expression for the computational cost of arbitrary per-sample skip schedules as a function of layer-wise skip probabilities; (ii) function-space superset (theorem 10) and strict inclusion (theorem 11) theorems showing that skip-layer models are strictly contained in---yet meaningfully approximate---the full-layer function space, with an explicit separating example; and (iii) a structural duality between VALSE and Mixture-of-Experts architectures (proposition 6), positioning vertical depth-wise sparsity as the orthogonal counterpart to horizontal width-wise sparsity. Building on this theory, we propose VALSE (Vertical Adaptive Layer Skipping for Efficiency), a per-sample, non-contiguous layer skipping method: a lightweight difficulty estimator scores each input from the first few layers, and per-layer gates selectively skip redundant layers---including arbitrary middle layers while retaining deeper ones---so that only the necessary depth is activated for each input, whose feasibility is preliminarily assessed at prototype scale.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

117,517 characters extracted from source content.

Expand or collapse full text

VALSE: Vertical Adaptive Layer Skipping for Efficient Inference in Large Language Models Jia-Dong Zhang Email: jiadongzh@gmail.com Abstract This paper establishes a theoretical framework for vertical adaptive layer skipping, proving three foundational results: (i) an Expected FLOPs formula (theorem 2) giving a closed-form expression for the computational cost of arbitrary per-sample skip schedules as a function of layer-wise skip probabilities; (i) function-space superset (theorem 10) and strict inclusion (theorem 11) theorems showing that skip-layer models are strictly contained in—yet meaningfully approximate—the full-layer function space, with an explicit separating example; and (i) a structural duality between VALSE and Mixture-of-Experts architectures (proposition 6), positioning vertical depth-wise sparsity as the orthogonal counterpart to horizontal width-wise sparsity. Building on this theory, we propose VALSE (Vertical Adaptive Layer Skipping for Efficiency), a per-sample, non-contiguous layer skipping method: a lightweight difficulty estimator scores each input from the first few layers, and per-layer gates selectively skip redundant layers—including arbitrary middle layers while retaining deeper ones—so that only the necessary depth is activated for each input, whose feasibility is preliminarily assessed at prototype scale. 1 Introduction Modern large language models (e.g., GPT-4 (OpenAI et al., 2023), LLaMA (Touvron et al., 2023), DeepSeek-MoE (Dai et al., 2024)) stack dozens to hundreds of Transformer blocks. Regardless of input difficulty—from simple sentiment classification to complex multi-step problems—every sample must traverse all layers, incurring identical computation. In the horizontal (width) dimension, Mixture-of-Experts (MoE) achieves sparse activation by selecting K of E experts per token per layer. However, in the vertical (depth) dimension, the number of layers remains fixed for all inputs: MoE makes each layer “cheaper” but does not reduce the number of layers. Depth thus constitutes an underexploited orthogonal direction for conditional computation. The core idea is natural: if simple inputs do not require the full representational power of all layers, can we vertically skip certain layers based on input difficulty, activating only the necessary depth? MoE achieves width-wise sparsity; vertical layer skipping achieves depth-wise sparsity—the two form orthogonal dimensions. Limitations of existing methods. Prior work on adaptive depth falls into several categories, each with structural limitations: early exit (Xin et al., 2020; Liu et al., 2020; Schuster et al., 2022) is truncation-based and cannot skip a middle layer while retaining a deeper one; adaptive computation time (Graves, 2016; Dehghani et al., 2019) adds shared-weight recurrence rather than skipping fixed layers; structured dropout (Fan et al., 2020; Elhoushi et al., 2024a) selects subnetworks statically or per token, not per-sample by difficulty; and token-level routing (MoD (Raposo et al., 2024)) operates at token granularity and lacks an explicit difficulty signal. Crucially, no existing method achieves per-sample, non-contiguous layer skipping (skipping arbitrary middle layers while retaining deeper ones) in conjunction with a unified theoretical duality to horizontal MoE. A detailed comparison is provided in section 2 and section 3. Contributions. We propose VALSE (Vertical Adaptive Layer Skipping for Efficiency), with three contributions: 1. Theoretical framework. We formalize vertical sparse conditional computation through three fully proven results: the Expected FLOPs theorem (theorem 2), which gives a closed-form expression for the computational cost of arbitrary per-sample skip schedules; the function-space superset (theorem 10) and strict containment (theorem 11) theorems, establishing that skip-layer models are strictly contained in the full-layer function space with an explicit separating example; and the MoE–VALSE structural duality (proposition 6) together with a unified two-dimensional sparsity representation (definitions 7 and 9), positioning VALSE as the depth-wise counterpart to horizontal MoE (section 8). 2. Method design. A lightweight, interpretable difficulty estimator scores each sample from the first KestK_est layers, feeding three routing variants (hard, soft, global) that produce per-sample, non-contiguous layer retention sets—extending conditional computation from the width to the depth dimension. The method is supported by four auxiliary losses (layer-utilization balancing, efficiency, output consistency via KL distillation, and Z-loss) and a three-stage curriculum learning schedule (sections 5, 6 and 7). 3. Proof-of-concept validation. A prototype-scale (1212-layer) evaluation on SST-2 provides initial feasibility evidence for the layer-skipping mechanism. We explicitly frame these results as preliminary and emphasize that the core empirical hypothesis—difficulty-driven differentiated routing—is not yet validated at the current scale; full-scale evaluation on larger, pre-trained models is necessary to confirm the approach (section 9). 2 Related Work We survey adaptive depth computation and horizontal MoE routing, identifying the research gaps that motivate VALSE. 2.1 Adaptive Depth Methods Existing work falls into five categories, summarized in table 6 and contrasted along structural axes in table 1. Early exit (DeeBERT (Xin et al., 2020), FastBERT (Liu et al., 2020), PABEE (Zhou et al., 2020)) attaches confidence classifiers at intermediate layers; all are truncation-based—once the model exits, all subsequent layers are skipped. CALM (Schuster et al., 2022) extends early exit to generative LMs but remains truncation-based. Recurrent adaptive computation (ACT (Graves, 2016), Universal Transformers (Dehghani et al., 2019)) iterates shared layers to add depth rather than skipping fixed layers; ACT learns a halting distribution over recurrence steps and Universal Transformers couple recurrence with a per-position halting mechanism, but both require shared-weight recurrence and therefore do not apply to standard fixed-layer Transformer stacks. Layer dropout (LayerDrop (Fan et al., 2020); LayerSkip (Elhoushi et al., 2024a) with self-speculative decoding (Leviathan et al., 2023)) supports subnetwork extraction but uses static or per-token selection, not per-sample difficulty-driven routing. Layer distillation (Xu et al., 2020) produces static depth variants. Token-level routing: MoD (Raposo et al., 2024) decides per-token per-layer whether to process or skip; Drowzee (Bachmann et al., 2023) predicts per-sample compute budgets but only controls truncation depth; DynamicBERT (Liu et al., 2021) adjusts width and depth independently without unification. Layer redundancy and structured pruning: Recent work provides empirical evidence that Transformer layers exhibit significant redundancy. ShortGPT (Men et al., 2024) introduces a Block Influence (BI) score to quantify layer redundancy and shows that up to 50%50\% of layers can be removed with minimal performance loss. Streamline (Elhoushi et al., 2024b) and SliceGPT (Ashraf et al., 2024) perform structured layer pruning at the post-training stage, producing static reduced-depth models. MoNE (Mustafa et al., 2024) applies mixture-of-experts routing across encoder–decoder layer pairs, selecting which layer pairs to execute per token. These methods confirm that layer redundancy is pervasive, but all produce static or per-token depth profiles—none achieves per-sample, difficulty-driven, non-contiguous skipping. 2.2 Horizontal MoE Routing MoE achieves width-wise sparse activation: each layer selects K of N experts. Sparse-gated MoE (Shazeer et al., 2017) introduced the gating mechanism; GShard (Lepikhin et al., 2020) scaled it to multilingual and multimodal settings; Switch Transformer (Fedus et al., 2022) simplified to Top-1 routing and demonstrated that aggressive expert sparsity is compatible with stable pretraining. Mixtral (Jiang et al., 2024) and DeepSeek-MoE (Dai et al., 2024) are recent open representatives, with DeepSeek-MoE in particular introducing fine-grained expert segmentation and shared experts to improve specialization. Across this family, the routing signal is semantic and per-token: the gate reads the token representation and matches it to expert specializations, with no notion of input-level difficulty or sample-level compute budget. The key observation is that all MoE methods keep depth fixed—every token still passes through all L layers—and that the difficulty-awareness dimension, central to VALSE, has no analogue in horizontal MoE routing. table 8 makes this orthogonality formal: MoE selects ‘which experts” within a layer, whereas VALSE selects ‘which layers” across depth, and the two dimensions are mathematically dual but operationally independent. 2.3 Horizontal vs. Vertical Comparison Table 1 summarizes the orthogonality: MoE reduces the number of parameters activated per layer (width); VALSE reduces the number of layers activated (depth). The two can be combined for two-dimensional sparse activation, and the orthogonality is structural rather than merely intuitive: width-wise sparsity selects a subset of parallel compute units within a layer (experts EiE_i applied to the same hidden state), whereas depth-wise sparsity selects a subset of sequential compute units across layers (blocks FlF_l applied in cascade), so the two selection operators act on disjoint axes of the activation tensor and compose without interference. This composability is non-trivial in two respects. First, the routing signals differ in scope: MoE gating is per-token and semantic (matching token content to expert specialization), while VALSE gating is per-sample and difficulty-driven (matching input complexity to retained depth), so a joint router must expose both a per-token width gate and a per-sample depth gate without one collapsing into the other. Second, the load-balancing objectives differ in target: horizontal MoE prevents expert under-utilization (few experts absorbing most tokens), whereas VALSE prevents layer under-utilization (most samples collapsing to the same minimal-depth path), so a unified training objective must regularize both the expert-utilization distribution and the per-layer activation frequency simultaneously. When these conditions hold, the two dimensions yield a multiplicative compute reduction: under a width sparsity sws_w (fraction of experts activated) and a depth sparsity sds_d (fraction of layers retained), the activated parameter count scales as (1−sw)(1−sd)(1-s_w)(1-s_d) of the dense network, a property formalized in theorem 9 as the unified width×depth sparsity framework. No prior method surveyed above attains both dimensions simultaneously: horizontal MoE keeps depth fixed, and all adaptive-depth methods keep width fixed, so the two-dimensional sparse-activation regime is itself part of the methodological gap that VALSE is designed to occupy. Table 1: Comparison of adaptive depth methods. Skip granularity and per-sample adaptivity distinguish VALSE from all prior approaches. Method Sparsity dim. Skip mode Granularity Per-sample? Difficulty-aware? MoE (Jiang et al., 2024; Dai et al., 2024) Width — Per-token No Semantic Early exit (Xin et al., 2020; Schuster et al., 2022) Depth Truncation Per-sample Yes Confidence LayerDrop (Fan et al., 2020) Depth Static dropout Global No — MoD (Raposo et al., 2024) Depth Per-layer Per-token×layer No Implicit ShortGPT (Men et al., 2024) Depth Static pruning Global No — SliceGPT (Ashraf et al., 2024) Depth Structured pruning Global No — VALSE (ours) Depth Non-contiguous Per-sample Yes Explicit router 2.4 Research Gaps The above analysis reveals four methodological gaps—per-sample non-contiguous skipping, unification of horizontal–vertical routing, explicit difficulty awareness, and validation on autoregressive LLMs—that, taken together, are not addressed by any prior method surveyed above. Truncation-based early exit (DeeBERT/CALM) is per-sample but constrains retention to a prefix; per-token per-layer routing (MoD) is fine-grained but lacks sample-level difficulty semantics; horizontal MoE sparsifies width but never depth; structured pruning (ShortGPT/SliceGPT) reduces depth statically with no input adaptivity. None of these dimensions, individually or in combination, yields a per-sample, difficulty-driven, non-contiguous layer retention set. Limitations of prior work. As table 1 makes explicit, every surveyed family restricts at least one of the three structural axes: early exit fixes skip mode to truncation, per-token routing fixes granularity to the token level, horizontal MoE fixes the sparsity dimension to width, and structured pruning fixes depth to a static global profile. Consequently, no single prior method simultaneously attains non-contiguous skipping, per-sample granularity, and explicit difficulty awareness—the cell (non-contiguous,per-sample,explicit)(non-contiguous,\,per-sample,\,explicit) that VALSE targets, which is the depth-axis counterpart of the width-axis cell already populated by horizontal MoE. We defer the structured contrast against each prior family to section 3 to avoid duplication; the remainder of the paper is structured around closing them. In summary, the related-work landscape partitions cleanly along three axes—skip mode (truncation vs. per-token vs. non-contiguous), granularity (global vs. per-token vs. per-sample), and difficulty awareness (none vs. implicit vs. explicit)—and no prior method occupies the cell (non-contiguous,per-sample,explicit)(non-contiguous,per-sample,explicit) that VALSE targets. The structural duality with horizontal MoE (section 2.2) further shows that this cell is not arbitrary: it is the depth-axis counterpart of the width-axis routing cell that MoE already occupies, so filling it completes the two-dimensional sparse-conditional-computation design space. The remainder of the paper develops this contribution: section 3 formalizes the four gaps, section 4 through section 7 present the method, and section 8 provides the theoretical framework, with a prototype-scale empirical study (section 9) that honestly reports a negative result for the core routing hypothesis. 3 Research Gaps Based on the cross-analysis of adaptive depth methods (section 2), we identify four research gaps that motivate VALSE. 3.1 Gap 1: No Per-Sample Non-Contiguous Layer Skipping Existing adaptive-depth work is either truncation-based early exit (exit at layer k, skip the rest: DeeBERT/CALM) or per-token per-layer routing (MoD). No prior work implements selective skipping of middle layers while retaining deeper layers, driven by whole-sample difficulty. This intermediate granularity—more flexible than truncation, more stable than per-token—is a clear methodological gap. Formally, truncation early exit constrains the retention set to a prefix S=1,…,kS=\1,…,k\, while per-token per-layer routing (MoD) issues O⁡(L⋅N)O(L· N) independent gating decisions per forward pass; VALSE instead computes a single per-sample difficulty scalar d^(x(s)) d(x^(s)) from the first KestK_est layers and derives the retention set S(s)⊆1,…,LS^(s) \1,…,L\ as an arbitrary subset, achieving O⁡(1)O(1) routing overhead per sample while permitting non-contiguous retention. 3.2 Gap 2: No Unified Horizontal–Vertical Routing Framework Horizontal MoE (expert selection) and vertical layer skipping (layer selection) are orthogonal sparsity dimensions, but no work unifies them into a jointly trainable routing framework. Existing models address either only the horizontal dimension (Mixtral/DeepSeek-MoE) or only the vertical dimension (MoD/LayerDrop). DynamicBERT jointly adjusts width and depth, but the two dimensions are routed independently with no unified mechanism. Formally, horizontal and vertical sparsity correspond to selecting subsets along two orthogonal axes of the activation tensor—experts E1,…,E\E_1,…,E_E\ within a layer versus layers F1,…,FL\F_1,…,F_L\ across depth—yet no prior routing framework treats them as instances of a single TopKTopK-selection operator over a unified unit set, leaving the compositional sparsity pattern width×depthwidth×depth unexploited. 3.3 Gap 3: No Explicit Difficulty-Aware Layer Routing Existing layer-skipping methods rely on implicit signals (intermediate-layer confidence, implicitly learned routers). No work introduces an explicit difficulty-aware module to drive per-sample layer-path selection. Drowzee’s budget predictor controls only truncation depth, not non-contiguous skipping; MoD’s router carries no difficulty semantics. Concretely, prior gating signals are either (a) implicit and learned end-to-end with no interpretable semantics, or (b) confidence-based and thus post-hoc (computed from the layer’s own output); no method defines an explicit, pre-routing difficulty scalar d^∈[0,1] d∈[0,1] with a documented feature-to-score mapping (table 2) and a monotonicity guarantee linking it to retained depth (proposition 4). We emphasize that proposition 4 is a theoretical guarantee obtained under the standard assumption of decreasing marginal layer contribution; the prototype reported in section 9 exhibits a routing-collapse negative result (r=−0.017r=-0.017) that reflects the failure of the learned gate to modulate depth, not a refutation of the monotonicity assumption itself. 3.4 Gap 4: Insufficient Validation on Generative LLMs Most layer-skipping work focuses on BERT-level discriminative tasks (DeeBERT/FastBERT/LayerDrop/DynamicBERT). CALM/LayerSkip/MoD extend to generation but still use truncation early exit or per-token routing. Non-contiguous per-sample layer skipping on autoregressive generative LLMs—its effectiveness, training stability, and quality-safety mechanisms—has not been systematically validated. We note that the VALSE routing mechanism is architecturally extensible to autoregressive generation: the per-sample gate g~l g_l is independent of token-position coupling and is compatible with causal masking, and self-speculative decoding (Leviathan et al., 2023) provides a natural quality-safety fallback for skipped layers. This extensibility is a present design property of the routing mechanism rather than an empirical claim: it holds at the architectural level (the gate is position-agnostic and causal-mask-compatible) but is not exercised by the prototype, whose evaluation is confined to discriminative SST-2 routing. The empirical validation on autoregressive generative LLMs that would close this community-wide gap is identified here as the central open problem that the field must address, and the design conditions under which such a validation would be well-posed are precisely the ones VALSE satisfies. Positioning. VALSE addresses all four gaps by introducing per-sample, non-contiguous layer skipping driven by an explicit difficulty estimator (Gap 1, 3), grounded in a structural duality with horizontal MoE that formalizes both dimensions as instances of a single TopKTopK-selection operator over a unified unit set and thereby enables two-dimensional sparse activation (Gap 2), and designed for extension to autoregressive generation with self-speculative decoding (Gap 4). A feature-level comparison of VALSE against prior methods is provided in table 1; our contributions are detailed in section 1. 4 Method Overview 4.1 Design Motivation While horizontal MoE reduces per-layer cost by activating only a subset of experts, the total depth—the count of sequential Transformer blocks—stays constant across all inputs. Consequently, even trivially easy inputs pay the full computational price of a deep stack, despite potentially needing only a fraction of its representational capacity. Hard samples may benefit from all L layers, but easy ones likely do not. This observation motivates depth-wise conditional computation: the number of executed layers should adapt to the input, yielding an orthogonal—and complementary—sparsity axis to width-wise MoE. 4.2 Core Idea: Per-Sample Non-Contiguous Layer Skipping Existing methods have two limitations (section 2.4): (i) truncation-based early exit can only skip a contiguous suffix of layers; (i) per-token routing (MoD) lacks per-sample difficulty semantics. We propose VALSE: for each sample s, a lightweight router generates a non-contiguous retention set S(s)⊆1,…,LS^(s) \1,…,L\, allowing arbitrary middle layers to be skipped while deeper layers are retained. Layers 11 and L are always retained. A difficulty score d^∈[0,1] d∈[0,1] (section 5) drives per-layer gates gl=σ⁡(wl⊤[rl;d^]+bl)g_l=σ(w_l [r_l; d]+b_l) (section 6). A target budget BtargetB_target and skip rate ρ∗ρ^* control the compute–quality trade-off (section 7). Duality with MoE: MoE uses a gate to select K of E experts per layer; VALSE uses a per-layer gate to select |S||S| of L layers per sample—one in width, one in depth, both using lightweight routers and load balancing. This structural duality is formalized in proposition 6 and unified under a single sparse selection framework in theorem 9; the key theoretical insight is that VALSE completes the depth dimension of sparse conditional computation, previously existing only in the width dimension (MoE). 4.3 Overall Architecture Three components: (i) Difficulty estimation—extracts features from the first KestK_est layers, producing d d (section 5); (i) Layer router—generates gating decisions forming S(s)S^(s) (section 6); (i) Training framework—auxiliary losses, curriculum learning, gradient stability (section 7). Data flow: embedding → first KestK_est layers → difficulty score d d → per-layer gates → skip/execute → LM head. The full symbol table is in appendix A. 5 Difficulty Estimation Module 5.1 Design Goals The module must: (i) produce interpretable difficulty scores (not implicitly learned); (i) be lightweight (overhead far less than the layers it skips); (i) operate at per-sample granularity; (iv) remain smooth during training. 5.2 Input Features From the l-th layer’s hidden state hl(s)∈ℝN×dmodelh_l^(s) ^N× d_model, we extract a five-dimensional statistic (mean-pooled over sequence length N, with h¯l=mean(hl(s)) h_l=mean(h_l^(s))) as rl∈ℝ5r_l ^5: Table 2: Difficulty estimation features. Feature Symbol Computation Intuition Hidden-state norm ρl _l ρl=‖mean(hl(s))‖2 _l=\|mean(h_l^(s))\|_2 Information content; activation magnitude Inter-layer delta norm Δl _l Δl=‖mean(hl−hl−1)‖2 _l=\|mean(h_l-h_l-1)\|_2 Representation change; small ⇒ skippable Self-attention entropy HlattnH_l^attn 1N∑iH⁡(softmax⁡(attni)) 1N _iH(softmax(attn_i)) High ⇒ diffuse ⇒ complex Prediction entropy HlpredH_l^pred H(softmax⁡(LMHead(h¯l)))H(softmax(LMHead( h_l))) High ⇒ uncertain ⇒ hard Singular-value concentration κl _l Top-1 SVD singular value share High ⇒ redundant ⇒ skippable Aggregation: for l∈1,…,Kestl∈\1,…,K_est\ (Kest=2–3K_est=2--3): R(s)=[r1;r2;⋯;rKest]∈ℝ5KestR^(s)=[r_1;\,r_2;\,·s;\,r_K_est] ^5K_est (1) Only the first KestK_est layers are used because the routing decision must be made before deeper layers are executed. We set Kest=2K_est=2 in the prototype (with a design range of 22–33): this is the smallest value that exposes both an inter-layer delta Δl _l (requiring l≥2l≥ 2) and a stable singular-value concentration κl _l, while keeping routing overhead below 1%1\% of total FLOPs; larger KestK_est increases estimation cost without adding qualitatively distinct signals, since features beyond the first few layers correlate strongly with the final-layer statistics used at training equilibrium. 5.3 Difficulty Score A lightweight MLP maps features to a scalar: d⁡(x(s))=wd⊤⋅ReLU⁡(W1⋅R(s)+b1)+bdd(x^(s))=w_d ·ReLU(W_1· R^(s)+b_1)+b_d (2) where W1∈ℝdhidden×5KestW_1 ^d_hidden× 5K_est (dhidden=64–128d_hidden=64--128). Batch-internal min-max normalization yields d^∈[0,1] d∈[0,1]: d^(x(s))=d⁡(x(s))−mins∈ℬ⁡d⁡(x(s))maxs∈ℬ⁡d⁡(x(s))−mins∈ℬ⁡d⁡(x(s))+ε d(x^(s))= d(x^(s))- _s d(x^(s)) _s d(x^(s))- _s d(x^(s))+ (3) where 00 denotes the easiest sample and 11 the hardest. The mapping from difficulty to layer-skipping decisions is realized by the layer router (section 6): easy samples (d d small) skip more layers; hard samples (d d large) retain all layers. This design is theoretically motivated by proposition 4 (difficulty–depth monotonicity): under the standard assumption of decreasing marginal layer contribution, harder inputs benefit more from full depth, justifying the schedule ρtarget=ρ∗(1−d^) _target=ρ^*(1- d). Normalization limitation. The batch-internal min–max map in eq. 3 guarantees d^∈[0,1] d∈[0,1] within each batch but renders scores incomparable across batches of different composition (the easiest sample of a hard batch may be harder than the hardest of an easy batch). This is acceptable for routing—the gate operates on within-batch relative ordering—but precludes using d d as an absolute difficulty signal across experiments or as a training-free diagnostic. To restore cross-batch comparability, we specify a calibration target that decouples the score from the batch extremum: replacing the per-batch (min,max)( , ) in eq. 3 with a running exponential moving average (mt,Mt)(m_t,M_t) updated as mt=λmt−1+(1−λ)mins∈ℬt⁡d⁡(x(s))m_t=λ m_t-1+(1-λ) _s _td(x^(s)) and Mt=λMt−1+(1−λ)maxs∈ℬt⁡d⁡(x(s))M_t=λ M_t-1+(1-λ) _s _td(x^(s)) with λ=0.9λ=0.9. Because the gate consumes only the within-batch ordering of d d, this calibration is routing-preserving for within-batch decisions while yielding a population-anchored score suitable for cross-experiment comparison; we adopt the batch-internal form in the prototype for simplicity and specify the calibrated variant as the production-target configuration. Difficulty score distribution. The normalized score d^∈[0,1] d∈[0,1] spreads samples across the full difficulty range by construction (eq. 3): the easiest sample in each batch receives d^=0 d=0 and the hardest receives d^=1 d=1, with intermediate samples distributed according to the raw MLP output d⁡(x(s))d(x^(s)). Because the normalization is rank-preserving within a batch, the distribution of d d reflects the empirical density of the raw MLP output d⁡(x(s))d(x^(s)): a roughly uniform d d distribution indicates that the raw score is approximately uniform, whereas a U-shaped d d distribution indicates bimodal raw scores. In the prototype evaluation (section 9), this within-batch spread did not translate into differentiated routing: the Pearson correlation between d d and activated layers was r=−0.017r=-0.017, indicating that the difficulty signal failed to modulate per-layer gating as intended, a result we report and analyze in section 9.3. Per-feature contribution. Table 2 lists five features with their intuitions, and each feature admits a well-defined expected contribution direction under the monotonicity assumption of proposition 4: ρl _l and Δl _l are expected to decrease with difficulty (harder inputs exhaust redundant magnitude faster), HlattnH_l^attn and HlpredH_l^pred to increase with difficulty (greater uncertainty and diffuse attention), and κl _l to decrease with difficulty (lower redundancy on harder inputs). Because the five features are pooled from only the first Kest=2K_est=2 layers, they are not statistically independent: ρl _l and Δl _l share the mean-pooled hidden state, and HlpredH_l^pred is computed from the same state through the LM head, so their marginal contributions are identifiable only when routing is non-degenerate. In the prototype, the router collapsed to a degenerate uniform-skip mode (section 9.3), so the marginal contribution of any single feature to routing differentiation is not identifiable from the prototype data; we therefore report the feature-level design rationale and expected directions as the present specification, with the empirical variance-decomposition analysis contingent on a non-collapsed routing regime that the prototype’s r=−0.017r=-0.017 result did not provide. Computational overhead. The difficulty estimator is invoked exactly once per sample, after the first Kest=2K_est=2 layers and before any routing decision, so its cost is amortized over the layers it gates. Concretely, the five features are all O⁡(Ndmodel)O(Nd_model): ρl _l and Δl _l require a mean-pool and two norm reductions; HlattnH_l^attn reuses the already-computed attention weights (no extra matmul); HlpredH_l^pred requires one LM-head forward on the single pooled vector h¯l h_l (cost O⁡(dmodel⋅|V|)O(d_model·|V|), amortized to a single token); and κl _l requires a top-11 singular value of the N×dmodelN× d_model pooled matrix, computed via a fixed number of power iterations (O⁡(Ndmodel)O(Nd_model) per iteration). The scoring MLP adds O⁡(dhidden⋅5Kest)O(d_hidden· 5K_est) parameters with dhidden=64d_hidden=64. Against a single Transformer block costing O⁡(Ndmodel2)O(Nd_model^2) for attention plus O⁡(Ndmodeldffn)O(Nd_modeld_ffn) for the FFN, the estimator’s total cost is O⁡(Ndmodel+dmodel|V|)O(Nd_model+d_model|V|), which is dominated by a single block whenever dffn≫dmodeld_ffn d_model (the standard ratio dffn=4dmodeld_ffn=4d_model). Since VALSE skips at least one full block to realize any savings, the estimator overhead is strictly less than the cheapest layer it can gate, satisfying design goal (i); in the prototype this ratio is below 1%1\% of total FLOPs, consistent with the overhead budget assumed throughout. Feature selection rationale. The five features in table 2 were selected to span four complementary signals of layer marginality—magnitude (ρl _l), dynamics (Δl _l), dispersion (HlattnH_l^attn), and uncertainty (HlpredH_l^pred)—plus a redundancy probe (κl _l) that directly operationalizes the layer-redundancy hypothesis of (Men et al., 2024). We deliberately exclude two natural candidates. First, an explicit layer-wise mutual-information estimate between hlh_l and the input x is excluded because it requires costly estimation (e.g., MINE-style critics) whose overhead would violate design goal (i) and whose variance at small N dominates the signal. Second, a learned end-to-end difficulty embedding (a small Transformer over the prefix) is excluded because it would forfeit the interpretability guarantee of design goal (i): the five-feature design exposes a documented, fixed feature-to-score mapping (table 2) whose individual contributions admit expected directions under proposition 4, whereas a learned embedding’s difficulty semantics would be opaque. The selection is therefore not exhaustive but principled: each feature admits a well-defined expected contribution direction (table 2, right column), the set is jointly computable from quantities already available in a forward pass, and its dimensionality (5Kest=105K_est=10) is small enough that the scoring MLP cannot memorize sample identity, preserving the difficulty signal’s generalization. Summary. The module produces a single, normalized, interpretable scalar d^(x(s))∈[0,1] d(x^(s))∈[0,1] at per-sample granularity with sub-1%1\% FLOPs overhead, satisfying all four design goals. Its sole known limitation—within-batch rank semantics that preclude cross-batch comparison—is addressed in the production-target configuration by the calibrated variant specified above. The downstream consumption of d d, namely the per-layer gating and retention-set construction, is the subject of section 6; the theoretical guarantee linking d d to retained depth is proposition 4, established in section 8. Calibrated variant pseudocode. The exponential moving average (EMA) calibration that restores cross-batch comparability of d d (specified qualitatively in the Normalization limitation paragraph above) is realized by the following per-step update, executed once per batch before the min–max map of eq. 3:# EMA-calibrated difficulty normalization (production target) # m_t, M_t: running min/max of raw score d(x) lambda = 0.9 # EMA decay # init: seed with first-batch extrema before EMA updates m_0 = min_s in B_1 d(x^(s)); M_0 = max_s in B_1 d(x^(s)) m_t = lambda * m_t-1 + (1 - lambda) * min_s in B_t d(x^(s)) M_t = lambda * M_t-1 + (1 - lambda) * max_s in B_t d(x^(s)) d_hat = (d(x) - m_t) / (M_t - m_t + eps) # population-anchored The batch-internal form of eq. 3 is recovered by setting (mt,Mt)=(mins∈ℬt⁡d⁡(x(s)),maxs∈ℬt⁡d⁡(x(s)))(m_t,M_t)=( _s _td(x^(s)), _s _td(x^(s))) at each step, so the prototype and production variants differ only in the reference extrema used for normalization; the routing decision itself (the gate g~l g_l) is invariant under this substitution because it consumes only within-batch ordering. Empirical-validation scope. The present section specifies the difficulty estimator as a design contract: the feature-to-score mapping, the expected contribution directions of table 2, and the monotonicity link to retained depth (proposition 4) are theoretical statements whose empirical validity is contingent on a non-collapsed routing regime. The prototype reported in section 9.3 collapsed to a uniform-skip mode (r=−0.017r=-0.017), so the variance-decomposition analysis of individual features and the calibration of d d against an external difficulty oracle are declared here as empirically open rather than verified; they are part of the design specification but not part of the prototype evidence base. 6 Layer Routing Strategies 6.1 Per-Sample Non-Contiguous Layer Skipping For sample s, the retention set S(s)⊆1,…,LS^(s) \1,…,L\ satisfies: (i) S(s)S^(s) may be any subset (non-contiguous), e.g., 1,2,4,7,11,12\1,2,4,7,11,12\; (i) layers 11 and L are always retained; (i) all tokens in the same sample share the same S(s)S^(s) (per-sample granularity). Unlike truncation-based early exit (S=1,…,kS=\1,…,k\) or per-token MoD, VALSE achieves per-sample non-contiguous skipping (see table 7 in the appendix). 6.2 Gating Function For each layer l, the gating function is: gl(x(s))=σ⁡(wl⊤⋅[rl;d^(x(s))]+bl)∈[0,1]g_l(x^(s))=σ\! (w_l ·[r_l;\, d(x^(s))]+b_l )∈[0,1] (4) where wl∈ℝdfeat+1w_l ^d_feat+1, and blb_l is initialized to 2.02.0 so that gates are close to 11 (full-layer execution) at the start of training. At inference, hard quantization is applied: g~l=1if gl>θl0otherwise g_l= cases1&if g_l> _l\\ 0&otherwise cases (5) (θl=0.5 _l=0.5 by default). During training, a straight-through estimator (STE) allows gradients to flow through the soft gate glg_l. 6.3 Residual Connection after Skipping Scheme A—pure identity (default): hl+1=hl+g~l⋅Fl(hl)h_l+1=h_l+ g_l· F_l(h_l) (6) When g~l=0 g_l=0, hl+1=hlh_l+1=h_l (identity skip; since dmodeld_model is consistent, no projection is needed). Scheme B—residual projection (optional, not used in prototype): Note: the function-space results (theorems 10 and 11) are established only under Scheme A; Scheme B introduces WlprojW_l^proj which may break the subset relationship (see Scheme A/B Remark in section 8). The prototype uses Scheme A throughout. hl+1=hl+g~l⋅Fl(hl)+(1−g~l)⋅Wlproj⋅hlh_l+1=h_l+ g_l· F_l(h_l)+(1- g_l)· W_l^proj· h_l (7) where WlprojW_l^proj may be low-rank (W=AB⊤W=AB , r≪dmodelr d_model). Gumbel noise is applied to gates during training to enhance robustness to non-contiguous skip paths, inspired by LayerDrop. 6.4 MoE Routing Analogy Table 8 (appendix) details the mathematical analogy: MoE selects K of E experts via Top-K; VALSE selects a retention set |S||S| of L layers via thresholding. Both use lightweight routers and impose load-balancing regularization. MoE faces expert collapse (few experts selected); VALSE faces the dual layer collapse (few layers retained), requiring load-balancing loss (section 7). 6.5 Forward Pass algorithm 1 shows VALSE’s single-sample forward pass. In batched implementation, samples with g~l=1 g_l=1 are clustered via argsort to form per-layer sub-batches, reducing gather/scatter overhead. Algorithm 1 VALSE single-sample forward pass. 1: h←Embedding(xs)h (x_s) 2: features←[]features←[\,]; hprev←h_prev← h 3: for l=0l=0 to Kest−1K_est-1 do 4: hl←Layerl(hprev)h_l _l(h_prev) 5: rl←ExtractFeatures(hl,hprev,l)r_l (h_l,h_prev,l) 6: features.append(rl)features.append(r_l); hprev←hlh_prev← h_l 7: end for 8: R←Concat(features)R (features) 9: d^←DifficultyEstimator(R) d (R) ⊳ normalize to [0,1][0,1] 10: h←hprevh← h_prev; S←∅S← 11: for l=Kestl=K_est to L−1L-1 do 12: rl←ExtractFeatures(h,hprev,l)r_l (h,h_prev,l) 13: gl←σ⁡(wl⊤⋅[rl;d^]+bl)g_l←σ(w_l ·[r_l;\, d]+b_l) 14: if training then g~l←GumbelSTE(gl,θl) g_l (g_l, _l) 15: else g~l←[gl>θl] g_l 1[g_l> _l] 16: end if 17: if g~l=1 g_l=1 then h←h+Layerl(h)h← h+Layer_l(h) 18: else S←S∪lS← S∪\l\ 19: end if 20: hprev←h_prev← h 21: end for 22: return LMHead(h),SLMHead(h),S 6.6 Architecture Variants Three variants are designed (full comparison in table 9 in the appendix): VALSE-Hard (main variant). Hard routing with STE; g~l∈0,1 g_l∈\0,1\; per-layer learnable wl,blw_l,b_l. At inference, layers are genuinely skipped—directly saving FLOPs. VALSE-Soft. Continuous gating: hl+1=hl+gl⋅Fl(hl)h_l+1=h_l+g_l· F_l(h_l). Gradients propagate directly; FLOPs savings depend on gl→0g_l→ 0. Most stable; can serve as initialization for hard routing. VALSE-Global. A global router produces p=softmax⁡(Wglobal⋅R(s))p=softmax(W_global· R^(s)), then: S(s)=TopK⁡(p),K=round(Btarget−d^⋅(Btarget−Lmin))S^(s)=TopK\!(p), K=round(B_target- d·(B_target-L_ )) (8) K varies with difficulty (d^=0⇒K=Btarget d=0 K=B_target; d^=1⇒K=L d=1 K=L). This variant has the strongest MoE analogy: Top-K layer selection ↔ Top-K expert selection. 7 Training Method The training objective jointly optimizes task quality and computational efficiency while maintaining router stability. Four auxiliary losses are designed. 7.1 Auxiliary Loss Design Layer utilization balancing ℒbalanceL_balance. This is the dual of MoE routing collapse: the router may retain only the first and last layers. Let f¯l=1B∑sg~l(s) f_l= 1B _s g_l^(s) and P¯l=1B∑sgl(s) P_l= 1B _sg_l^(s): ℒbalance=α⋅L⋅∑l=1Lf¯l⋅P¯lL_balance=α· L· _l=1^L f_l· P_l (9) By the AM–GM inequality, this is minimized under a fixed budget when all layers are used uniformly. (α=0.01α=0.01.) Efficiency loss ℒeffL_eff. A one-sided hinge controls the total activated layers A¯=∑lf¯l A= _l f_l: ℒeff=β⋅max⁡(0,A¯−Btarget)2L2L_eff=β· (0,\, A-B_target)^2L^2 (10) The weight increases progressively: βeff(t)=β⋅min⁡(1,t/Twarmup) _eff(t)=β· (1,\,t/T_warmup). (β=0.1β=0.1, Btarget∈[0.5L, 0.85L]B_target∈[0.5L,\,0.85L].) Output consistency ℒconsistL_consist. KL distillation aligns the skip-path output with the full-layer path: ℒconsist=δ⋅TKD2⋅KL(pfull∥pskip)L_consist=δ· T_KD^2·KL(p_full\|p_skip) (11) (TKD=2.0T_KD=2.0, δ=0.5δ=0.5, annealed to δ/3δ/3.) Three teacher strategies reduce the double-forward overhead: EMA teacher, periodic teacher (Kconsist=10K_consist=10), and difficulty-bucketed caching. Z-loss ℒzL_z. Stabilizes the numerical range of gating logits (following Switch Transformer): ℒz=γ⋅1B∑s∑l(wl⊤[rl;d^]+bl)2L_z=γ· 1B _s _l(w_l [r_l; d]+b_l)^2. (γ=10−4γ=10^-4.) The total loss combines all terms: ℒtotal=ℒtask+αtℒbalance+βtℒeff+δtℒconsist+γtℒzL_total=L_task+ _tL_balance+ _tL_eff+ _tL_consist+ _tL_z (12) where αt,βt,δt,γt _t, _t, _t, _t are the (possibly time-varying) weights of each auxiliary loss. 7.2 Curriculum Learning Stage 1: Full-layer warm-up (ρskip=0 _skip=0). Gate bias bl=2.0b_l=2.0 forces g~l=1 g_l=1; only ℒtask+γℒzL_task+ _z is active. Duration: η1≈0.25 _1≈ 0.25–0.330.33 of TtotalT_total. Stage 2: Progressive skipping (0→ρ∗0→ρ^*). ρskip(t)=ρ∗⋅12[1−cos⁡(π⋅t−T1T2−T1)] _skip(t)=ρ^*· 12 [1- \! (π· t-T_1T_2-T_1 ) ] (13) Auxiliary losses progressively take effect; Gumbel temperature τG _G anneals 2.0→0.52.0→ 0.5; bias anneals to btarget≈0.76b_target≈ 0.76. Duration: η2≈0.5 _2≈ 0.5–0.670.67 (extended to 2×2× original to allow differentiated routing to emerge). Stage 3: Joint optimization (ρskip=ρ∗ _skip=ρ^*). All parameters are jointly optimized; δ anneals downward. Learning rate cosine-anneals to 1/101/10 of the Stage 2 peak. Duration: η3≈0.25 _3≈ 0.25–0.40.4. 7.3 Difficulty-Aware Scheduling and Stability Individualized target: ρtarget(s)=ρ∗(1−d^(s)) _target^(s)=ρ^*(1- d^(s)). The training feedback loop (d^→ d→ gate → skip path → loss → gradient →d^→ d) is stabilized under: ‖∂d^∂θD‖⋅‖∂ℒeff∂d^‖⋅‖∂g~∂d^‖⋅‖∂ℒtask∂g~‖<1 \| ∂ d∂ _D \|· \| _eff∂ d \|· \| ∂ g∂ d \|· \| _task∂ g \|<1 (14) Three safeguards: normalization anchoring, difficulty-estimator gradient clipping (max_norm=1.0max\_norm=1.0), and gradient flow through ℒeffL_eff (removing stop-gradient so the difficulty signal propagates to the router). Probabilistic full-layer recomputation (precomp=0.1p_recomp=0.1) prevents “dead layers” among skipped layers. Proof sketch for eq. 14. The training feedback loop forms a composition: d^→ϕ1g~→ϕ2hskip→ϕ3ℒtask→ϕ4∇d^ℒeff→ϕ5θD d _1 g _2h_skip _3L_task _4 _ dL_eff _5 _D. By the chain rule, the product of Jacobian norms along this loop determines whether the feedback is contractive (stable) or expansive (divergent). eq. 14 requires the product to be <1<1, i.e., the loop is a contraction. The three safeguards enforce this: (i) normalization anchors ‖∂d^/∂θD‖\|∂ d/∂ _D\| by bounding d^∈[0,1] d∈[0,1]; (i) gradient clipping upper-bounds the effective Jacobian norm; (i) allowing gradient flow through ℒeffL_eff ensures the difficulty signal propagates, while the efficiency loss provides a corrective force that counteracts runaway amplification. When all three hold, each Jacobian factor is bounded, and their product is <1<1 provided that (i) the difficulty estimator θD _D is Lipschitz-continuous with constant LD≤1L_D≤ 1 (ensured by gradient clipping with max_norm=1.0max\_norm=1.0), and (i) the gate function σ⁡(wl⊤[rl;d^]+bl)σ(w_l [r_l; d]+b_l) has bounded gradient ≤0.25≤ 0.25 (inherent to the sigmoid). These are sufficient conditions, not mild assumptions; their empirical validity at scale remains unverified in the prototype. □ ∎ Prototype implementation note. In the current prototype, curriculum learning Stage 1 is simplified (omitted), and Gumbel temperature annealing is configured (τG:2.0→0.5 _G:2.0→ 0.5) but not yet fully implemented in the training code. The prototype uses a fixed τG=1.0 _G=1.0 throughout training. Full implementation of the three-stage curriculum and temperature annealing is planned for the next iteration. 8 Theoretical Analysis 8.1 Computational Savings Per-layer FLOPs. The per-sample FLOPs of the l-th Transformer block decompose as Cl=Clattn+ClffnC_l=C_l^attn+C_l^ffn, where Clattn≈4Ndmodel2+2N2dmodelC_l^attn≈ 4Nd_model^2+2N^2d_model and Clffn≈8Ndmodel2C_l^ffn≈ 8Nd_model^2. For uniform layers, this is a constant ClayerC_layer. Lemma 1 (Linearity of expectation). For any random variables X1,…,XnX_1,…,X_n, ⁡[∑iXi]=∑i⁡[Xi]E\! [ _iX_i ]= _iE[X_i]. Theorem 2 (Expected FLOPs). Let the skip rate of layer l be p1,p2,…,pLp_1,p_2,…,p_L (with p1=pL=0p_1=p_L=0). The expected per-sample FLOPs of VALSE is: ⁡[VALSE]=Cfixed+∑l=Kest+1L(1−pl)⋅ClE[F_VALSE]=C_fixed+ _l=K_est+1^L(1-p_l)· C_l (15) Proof. Total FLOPs equals the sum of components. The fixed part is CfixedC_fixed. For the skippable part, layer l’s FLOPs contribution is g~l⋅Cl g_l· C_l. Taking expectation, ⁡[g~l⋅Cl]=Cl⋅P⁡(g~l=1)=Cl⋅(1−pl)E[ g_l· C_l]=C_l· P( g_l=1)=C_l·(1-p_l). By lemma 1 (linearity of expectation), summing over all skippable layers yields ⁡[VALSE]=Cfixed+∑l=Kest+1L(1−pl)⋅ClE[F_VALSE]=C_fixed+ _l=K_est+1^L(1-p_l)· C_l. ∎ Remark (gate correlation). The linearity of expectation used above does not require the gate decisions g~l\ g_l\ to be independent across layers. Although these decisions share the common difficulty estimate d d and are therefore correlated, ⁡[g~l⋅Cl]=Cl⋅⁡[g~l]=Cl⋅(1−pl)E[ g_l· C_l]=C_l·E[ g_l]=C_l·(1-p_l) holds for any joint distribution of g~l\ g_l\. Corollary 3 (Uniform skip rate). If all skippable layers share skip rate p, each layer has FLOPs ClayerC_layer, and Kest≪LK_est L: ⁡[VALSE]≈(1−p⋅L−Kest−1L)⋅fullE[F_VALSE]≈ (1-p· L-K_est-1L )·F_full (16) VALSE satisfies ⁡[VALSE]≤fullE[F_VALSE] _full (upper bound, equality iff pl=0∀lp_l=0\ ∀\,l) and ⁡[VALSE]≥Cfixed+CLE[F_VALSE]≥ C_fixed+C_L (lower bound). When VALSE and truncation-based early exit share the same average activated layer count μ, their FLOPs are comparable; however, VALSE can activate non-contiguous layer subsets for different samples, whereas truncation can only activate a contiguous prefix. The theoretical savings under the difficulty-aware schedule are derived in section 8.2 (observation 5). 8.2 Difficulty-Adaptive Scheduling The FLOPs analysis above treats the skip rate plp_l as an exogenous parameter. In practice, VALSE employs a difficulty-aware schedule ρtarget(s)=ρ∗(1−d^(s)) _target^(s)=ρ^*(1- d^(s)), where harder samples (larger d d) skip fewer layers. We now provide a theoretical justification for this design choice. Proposition 4 (Difficulty–depth monotonicity). Assume that the marginal task contribution of each layer is non-decreasing in input difficulty: for each layer l, the quantity Δl(x)≔ℒtask(F1:l−1(x))−ℒtask(F1:l(x)) _l(x) _task\! (F_1:l-1(x) )-L_task\! (F_1:l(x) ) is non-decreasing in d^(x) d(x). Then the optimal skip rate ρ⋆(x)ρ (x) that minimizes ℒtask+λL_task+ is non-increasing in d^(x) d(x): harder inputs should skip fewer layers. Proof. We analyze the Lagrangian relaxation of the constrained optimization. The per-sample objective is to choose a retention set S⁡(x)⊆1,…,LS(x) \1,…,L\ minimizing (S)≔ℒtask(F1:S(x);y)+λ(S(x)),J(S) _task\! (F_1:S(x);y )+λ\,F(S(x)), where ⁡(S⁡(x))=Cfixed+∑l∈SClF(S(x))=C_fixed+ _l∈ SC_l is the FLOPs of retaining layers in S and λ>0λ>0 is the FLOPs–accuracy trade-off parameter. Define the marginal task contribution of layer l as Δl(x)≔ℒtask(F1:l−1(x);y)−ℒtask(F1:l(x);y), _l(x) _task\! (F_1:l-1(x);y )-L_task\! (F_1:l(x);y ), i.e., the reduction in task loss obtained by adding layer l to the retention set. The marginal FLOPs cost of retaining layer l is λClλ\,C_l. Optimality condition. The Lagrangian ⁡(S)J(S) is minimized when, for each layer l, the marginal task benefit of retention equals the marginal FLOPs cost: Δl(x)=λCl _l(x)=λ\,C_l. Layers where Δl(x)>λCl _l(x)>λ\,C_l should be retained; layers where Δl(x)<λCl _l(x)<λ\,C_l should be skipped. This is the standard greedy-retention condition for separable costs. Monotonicity argument. Under the assumption that Δl _l is non-decreasing in d d (harder inputs derive greater benefit from each layer), for a harder input x′x with d^(x′)>d^(x) d(x )> d(x), we have Δl(x′)≥Δl(x) _l(x )≥ _l(x) for every l. The equality Δl=λCl _l=λ\,C_l is therefore reached at a larger retention set |S⁡(x′)|≥|S⁡(x)||S(x )|≥|S(x)| (i.e., fewer layers skipped). Consequently, the optimal skip rate ρ⋆(d^)=1−|S⋆(d^)|/Lρ ( d)=1-|S ( d)|/L is non-increasing in d d: harder inputs should skip fewer layers. Expected FLOPs. By lemma 1, the expected FLOPs under the difficulty-aware schedule is ⁡[]=Cfixed+∑l⁡[g~l|d^]⋅Cl=Cfixed+∑l(1−pl(d^))⋅ClE[F]=C_fixed+ _lE[ g_l\,|\, d]· C_l=C_fixed+ _l(1-p_l( d))· C_l, where pl(d^)p_l( d) decreases with d d, confirming that harder inputs incur more computation. The linear schedule ρtarget=ρ∗(1−d^) _target=ρ^*(1- d) is the simplest monotone-decreasing parametrization consistent with this result. □ ∎ Remark. proposition 4 establishes the direction of the difficulty–depth relationship under a standard assumption about decreasing marginal layer contribution, consistent with the layer redundancy hypothesis (Men et al., 2024). It does not prove that the learned estimator d d correctly captures true input difficulty; that is an empirical question partially addressed by our prototype experiments (section 9.2). Observation 5 (Theoretical savings under difficulty-aware scheduling). Under the schedule ρtarget(s)=ρ∗(1−d^(s)) _target^(s)=ρ^*(1- d^(s)) with ρ∗=0.3ρ^*=0.3, uniform per-layer FLOPs ClayerC_layer, and Kest≪LK_est L, the average skip rate is ⁡[p]=ρ∗⋅(1−⁡[d^])E[p]=ρ^*·(1-E[ d]). By corollary 3, the theoretical FLOPs savings are: savings=ρ∗⋅(1−⁡[d^])⋅L−Kest−1L.savings\;=\;ρ^*·(1-E[ d])· L-K_est-1L. For the prototype (L=6L=6, Kest=2K_est=2, ρ∗=0.3ρ^*=0.3): savings range from 0%0\% (all-hard, ⁡[d^]=1E[ d]=1) to 15%15\% (all-easy, ⁡[d^]=0E[ d]=0); with a 5050–5050 easy–hard split, savings are ≈7.5%≈ 7.5\%. For a 1212-layer model (Kest=2K_est=2), the range is 0%0\% to 22.5%22.5\%. These are theoretical estimates under the assumed schedule and idealized difficulty distribution; actual savings depend on the learned d d and gate calibration. 8.3 MoE–VALSE Duality Proposition 6 (Structural duality of MoE and VALSE). Horizontal MoE and vertical VALSE are instances of the same sparse selection problem on two orthogonal dimensions: MoE selects K of E candidate transformations per layer (width); VALSE selects |S||S| of L candidate layers per sample (depth). Their pipelines are structurally parallel: router generates selection probabilities → Top-K / hard quantization → sparse execution → load-balancing regularization. Proof. We formalize the duality via an isomorphism of the routing pipelines. Define the conditional computation template =(ℛ,,ℰ,ℒ)T=(R,S,E,L) consisting of four components: 1. Router ℛR: a function g:→[0,1]Ng:X→[0,1]^N mapping the input to selection probabilities, where N is the number of candidate compute units. 2. Selection operator S: a mapping from probabilities to a binary mask m∈0,1Nm∈\0,1\^N (Top-K for MoE, thresholding for VALSE). 3. Sparse execution ℰE: only units with mi=1m_i=1 are evaluated, producing output ∑i:mi=1wi⋅ui(input) _i:\,m_i=1w_i· u_i(input). 4. Load-balancing regularizer ℒL: a regularization term preventing degenerate selection (the MoE load-balancing loss or VALSE entropy regularizer). MoE instantiation. The compute units u1,…,uE\u_1,…,u_E\ are expert networks organized in parallel within each layer: N=EN=E (width dimension). The router gl(x)∈ℝEg_l(x) ^E produces expert-selection probabilities, and TopKhTopK_h selects KhK_h experts per token. The output is ∑e∈TopKhgl(e)(x)⋅fe(hl) _e _hg_l^(e)(x)· f_e(h_l). VALSE instantiation. The compute units u1,…,uL\u_1,…,u_L\ are Transformer layers organized sequentially across the depth: N=LN=L (depth dimension). The router produces per-layer gate logits gl(x)∈ℝg_l(x) , and thresholding produces g~l∈0,1 g_l∈\0,1\. The output is the sequential composition through retained layers. Isomorphism. The isomorphism φ:MoE→VALSE :T_MoE _VALSE maps each component: • Router: glMoE∈ℝE↔glVALSE∈ℝg^MoE_l ^E g^VALSE_l (vector-to-scalar gate, reducing the selection dimension from width to depth). • Selection: TopKh↔[gl>θl]TopK_h 1[g_l> _l] (Top-K to binary threshold). • Execution: ∑e∈TopKwefe↔○l∈S(I+Fl) _e w_ef_e _l∈ S(I+F_l) (parallel weighted sum to sequential residual composition). • Regularization: ℒbalanceMoE↔ℒbalanceVALSEL_balance^MoE _balance^VALSE (both penalize uneven utilization across units). By proposition 4, the VALSE schedule is difficulty-adaptive; the MoE router is similarly input-adaptive. Both instantiate the same conditional-computation template, differing only in the organization of units (parallel vs. sequential). The routing, selection, and regularization mechanisms are isomorphic. □ ∎ Remark. proposition 6 establishes a structural (not functional) duality. We do not claim that any MoE routing policy can be embedded into VALSE or vice versa; the two operate on different dimensions of the computation graph. Rather, the shared pipeline—router, selection, sparse execution, load balancing—reveals that VALSE is the depth-wise analogue of MoE’s width-wise sparsity. Unified selection framework. Given N compute units u1,…,uN\u_1,…,u_N\ and budget K, select subset I⊆1,…,NI \1,…,N\: output=∑i∈Iwi⋅ui(input)output= _i∈ Iw_i· u_i(input). MoE corresponds to N=EN=E (parallel organization); VALSE corresponds to N=LN=L (sequential organization); the selection mechanisms are structurally similar. 8.4 Two-Dimensional Sparsity Definition 7 (Two-dimensional sparse selection). Simultaneous horizontal and vertical sparsity: each sample selects KvK_v layers and, per layer, KhK_h experts: 2D=Kv⋅Kh⋅Cunit,savings=1−Kv⋅KhL⋅EF_2D=K_v· K_h· C_unit, =1- K_v· K_hL· E (17) Proposition 8 (Hybrid architecture FLOPs advantage). Let horizontal sparsity ρh=1−Kh/E _h=1-K_h/E and vertical sparsity ρv=1−Kv/L _v=1-K_v/L: hybrid=(1−ρv)(1−ρh)⋅fullF_hybrid=(1- _v)(1- _h)·F_full (18) Taking ρh=0.875 _h=0.875 (88-choose-11 MoE) and ρv=0.2 _v=0.2 (VALSE), hybrid/full=0.10F_hybrid/F_full=0.10, i.e., 90%90\% savings, versus 87.5%87.5\% for MoE alone. Theorem 9 (Unified representation of sparse Transformers). Common sparse Transformer variants that employ per-unit gating can be represented as applying a sparse mask M∈0,1L×EM∈\0,1\^L× E on the computation graph G∈ℝL×EG ^L× E: output=∑(l,e):M⁡[l,e]=1wl,e⋅Fl(e)(input)output= _(l,e):\,M[l,e]=1w_l,e· F_l^(e)(input) (19) Horizontal MoE uses per-row Top-K masks (M[l,e]=[gl(x)∈TopKh]M[l,e]=1[g_l(x) _h]); VALSE uses whole-row masks (M⁡[l,e]=g~l∀eM[l,e]= g_l\ ∀\,e); hybrids use joint row–column sparsity (M[l,e]=g~l⋅[gl(x)∈TopKh]M[l,e]= g_l·1[g_l(x) _h]). This reveals that sparse Transformers differ in which dimension of the computation graph they mask. VALSE’s contribution is completing the depth dimension of sparsity—previously, sparsity existed only in the width dimension (MoE). Further theoretical results, including function-space analysis and optimization properties, in sections 8.5 and 8.6. Remark (Scheme A/B applicability). VALSE supports two residual schemes (section 6): Scheme A (identity skip, eq. 6) and Scheme B (residual projection, eq. 7). All results in sections 8.1, 8.2, 8.3 and 8.4 apply to both schemes, as they concern FLOPs accounting and structural duality. The function-space results (theorems 10 and 11 in section 8.5) are established under Scheme A only: the identity-skip property (hl+1=hlh_l+1=h_l when g~l=0 g_l=0) is essential to the subset and strict-containment arguments. Under Scheme B, the projection WlprojW_l^proj introduces additional expressivity that may break the subset relationship; we restrict the function-space analysis to Scheme A and note this as a limitation. 8.5 Function-Space Analysis We now establish the relationship between the function space of VALSE and that of the full-layer model. All results in this subsection are established under Scheme A (identity residual, eq. 6), where skipping a layer produces the identity mapping hl+1=hlh_l+1=h_l. Theorem 10 (Full-layer function space is a superset (Scheme A)). Under Scheme A (identity residual, eq. 6), the set of functions representable by VALSE under any retention set S⊆1,…,LS \1,…,L\ is a subset of the full-layer function space: ℱVALSE⊆ℱfullF_VALSE _full. Proof. The full-layer model applies all L layers in sequence. Under Scheme A, the residual update is hl+1=hl+g~l⋅Fl(hl)h_l+1=h_l+ g_l· F_l(h_l). When g~l=1 g_l=1 (layer retained), this reduces to the standard residual update hl+1=hl+Fl(hl)h_l+1=h_l+F_l(h_l), identical to the full-layer model. When g~l=0 g_l=0 (layer skipped), hl+1=hlh_l+1=h_l, which is the identity mapping. Any VALSE path with retention set S produces the same output as the full-layer model with the parameters of all skipped layers set to zero: F~l=0 F_l=0 for l∉Sl∉ S. Since the full-layer function space includes all parameter settings (including the degenerate zero setting), every VALSE function is a special case of the full-layer model with particular parameters. Therefore ℱVALSE⊆ℱfullF_VALSE _full. By lemma 1, the expected output under random skip schedules is a convex combination of individual VALSE paths, each of which lies in ℱfullF_full. Since ℱfullF_full is closed under convex combinations, the subset relationship holds in expectation as well: ⁡[ℱVALSE]⊆ℱfullE[F_VALSE] _full. □ ∎ Theorem 11 (Strict containment (Scheme A)). Under Scheme A, there exist functions representable by the full-layer model that no VALSE path with |S|<L|S|<L can represent; i.e., the containment is strict: ℱVALSE⊊ℱfullF_VALSE _full. Proof. We construct an explicit counterexample using the residual structure hl+1=hl+Fl(hl)h_l+1=h_l+F_l(h_l). Consider a two-layer segment where: • Fl(h)=WlhF_l(h)=W_lh performs a rank-deficient linear projection with Wl∈ℝd×dW_l ^d× d, rank⁡(Wl)=r<drank(W_l)=r<d, projecting onto a single direction v (i.e., Wl=vv⊤W_l=v with ‖v‖=1\|v\|=1). • Fl+1(h)=ϕ⁡(Wl+1h)F_l+1(h)=φ(W_l+1h) applies a nonlinear activation ϕ=ReLUφ=ReLU followed by a full-rank projection Wl+1∈ℝd×dW_l+1 ^d× d with rank⁡(Wl+1)=drank(W_l+1)=d. The full-layer function for this segment is: ffull(h)=h+Wl+1⋅ϕ⁡(h+Wlh)=h+Wl+1⋅ϕ⁡((I+Wl)h).f_full(h)=h+W_l+1·φ\! (h+W_lh )=h+W_l+1·φ\! ((I+W_l)\,h ). This depends on a two-step nonlinear interaction: the input is first transformed by the rank-deficient projection WlW_l, summed with the identity (residual connection), then passed through ϕφ and the full-rank projection Wl+1W_l+1. Now consider the VALSE paths that skip one of the two layers: • Skip layer l (g~l=0 g_l=0, g~l+1=1 g_l+1=1): fskip-l(h)=h+Wl+1⋅ϕ⁡(h)f_skip-l(h)=h+W_l+1·φ(h). The rank-deficient intermediate transformation WlW_l is never applied. The input to ϕφ is h itself, not (I+Wl)h(I+W_l)\,h, so the output depends on the full d-dimensional input rather than the projected r-dimensional subspace. • Skip layer l+1l+1 (g~l=1 g_l=1, g~l+1=0 g_l+1=0): fskip-(l+1)(h)=h+Wlh=(I+Wl)hf_skip-(l+1)(h)=h+W_lh=(I+W_l)\,h, which is a linear function—the nonlinearity ϕφ and the full-rank projection Wl+1W_l+1 are never applied. Concretely, let Wl=vv⊤W_l=v (rank-11) and Wl+1=IW_l+1=I (full-rank). Then: • ffull(h)=h+ϕ⁡((I+vv⊤)h)f_full(h)=h+φ\! ((I+v )\,h ), which can produce outputs in a d-dimensional cone that depends on the sign of v⊤hv h. • fskip-l(h)=h+ϕ⁡(h)f_skip-l(h)=h+φ(h), which operates on the full d-dimensional input without the rank-11 projection—a fundamentally different function. • fskip-(l+1)(h)=(I+vv⊤)hf_skip-(l+1)(h)=(I+v )\,h, a rank-(r+1)(r+1) linear map with no nonlinearity. Neither skip path can reproduce ffullf_full: the full-layer function depends on the composition of the rank-deficient projection, the residual addition, and the nonlinearity—a dependency that no single-layer skip can replicate. The key observation is that the residual connection hl+1=hl+Fl(hl)h_l+1=h_l+F_l(h_l) creates a coupling between the identity pathway and the layer transformation: the nonlinearity ϕφ in Fl+1F_l+1 receives both the unmodified residual hlh_l and the projection WlhlW_lh_l as a sum. Skipping layer l removes the WlhlW_lh_l contribution entirely, and skipping layer l+1l+1 removes the nonlinearity entirely. Furthermore, the effective capacity rank of a network with retention set S is bounded by ∑l∈Srank⁡(Wl)⋅dmodel _l∈ Srank(W_l)· d_model, which is strictly less than ∑l=1Lrank⁡(Wl)⋅dmodel _l=1^Lrank(W_l)· d_model when |S|<L|S|<L and at least one skipped layer has non-trivial rank. By observation 14, the capacity gap is proportional to the skipped layers’ ranks. Hence the containment is strict: ℱVALSE⊊ℱfullF_VALSE _full. □ ∎ Proposition 12 (Number of non-contiguous retention sets). The number of non-contiguous retention sets of size k (with layers 11 and L fixed) is (L−2k−2) L-2k-2. The total number of valid retention sets is 2L−2−12^L-2-1. Proof. Layers 11 and L are always retained (the estimator prefix and final output). From the remaining L−2L-2 interior layers, we choose k−2k-2 to retain, giving (L−2k−2) L-2k-2 retention sets of size k. Summing over k=2,…,Lk=2,…,L gives ∑k=2L(L−2k−2)=∑j=0L−2(L−2j)−(L−2L−2)=2L−2−1 _k=2^L L-2k-2= _j=0^L-2 L-2j- L-2L-2=2^L-2-1 valid retention sets, where we subtracted 11 to exclude the degenerate case. These retention sets underlie the difficulty-aware schedule (proposition 4), where harder inputs select larger sets. □ ∎ Observation 13 (Information preservation under identity skipping). Under identity skipping (g~l=0 g_l=0), hl+1=hlh_l+1=h_l, so the mutual information with the target is exactly preserved: I⁡(hl+1,y)=I⁡(hl,y)I(h_l+1;y)=I(h_l;y). No existing information is lost. However, the information that layer l would have extracted—i.e., I⁡(Fl(hl),y)−I⁡(hl,y)I(F_l(h_l);y)-I(h_l;y)—is foregone. Thus, skipping preserves prior information but sacrifices potential information gain from the skipped layer. Observation 14 (Effective capacity rank). The effective capacity rank of a model with retention set S is proportional to ∑l∈Sdmodel2 _l∈ Sd_model^2. Skipping layers reduces capacity linearly in the number of skipped layers, motivating the difficulty-aware schedule (proposition 4) which skips layers only when the marginal information gain Δl _l is small. 8.6 Optimization Properties Observation 15 (Non-convexity in gate parameters). The loss landscape with respect to gate parameters wl,bl\w_l,b_l\ is non-convex, due to the discreteness of g~l=[gl>θl] g_l=1[g_l> _l]. The indicator function introduces a combinatorial structure: there are 2L−22^L-2 possible retention patterns, and the loss surface is piecewise constant in the gate parameters, with discontinuities at decision boundaries. This non-convexity motivates the use of the straight-through estimator and Gumbel-softmax relaxation during training. Observation 16 (STE gradient approximation (biased)). The straight-through estimator (STE) provides a biased gradient approximation for the gate parameters. In the forward pass, the hard gate g~l=[gl>θl] g_l=1[g_l> _l] is used; in the backward pass, the gradient of the threshold function is replaced by the identity, ∂g~l∂wl≈∂gl∂wl ∂ g_l∂ w_l≈ ∂ g_l∂ w_l. The bias is ⁡[∇wlℒSTE]−∇wlℒ≠0E[ _w_lL^STE]- _w_lL≠ 0 in general, because the surrogate gradient ignores the discontinuity of the threshold function. The Gumbel-softmax relaxation with temperature annealing (section 7) mitigates this bias by smoothing the threshold during training. Proposition 17 (Collapse risk (under a drift model)). Without load-balancing regularization (α=0α=0), gate parameters converge to a degenerate solution—all samples routed to the same minimal retention set (first and last layers only)—with probability at least 1−e−cL1-e^-cL, where c>0c>0 is a constant depending on the gate initialization and optimization landscape. Proof sketch (under a hypothetical drift model). Assumption. The drift model pl(T)≤e−clTp_l^(T)≤ e^-c_lT (gate retention probability decays exponentially under task-loss gradient without load balancing) is a hypothetical model; it is not derived from a formal optimization analysis but is motivated by the empirical observation that, without balancing, gate biases drift toward −∞-∞. Let pl=P⁡(g~l=1)p_l=P( g_l=1) be the retention probability of layer l. Without load balancing, the gradient signal for each gate wlw_l is dominated by the task loss, which is minimized when the model uses the cheapest path (fewest layers). Under independent Bernoulli gates, the probability that at least one non-trivial retention pattern survives is ∏l=2L−1(1−pl) _l=2^L-1(1-p_l). When the task loss landscape favors collapse (e.g., when intermediate layers contribute redundantly), the gate biases drift toward −∞-∞, driving pl→0p_l→ 0. The probability that collapse does not occur (i.e., at least one non-trivial pattern survives after T steps) is bounded by ∏lpl(T) _lp_l^(T). Under a drift model where pl(T)≤e−clTp_l^(T)≤ e^-c_lT for some cl>0c_l>0 (from gradient-driven bias reduction), the survival probability is at most exp(−∑lclT)≤e−cLT \! (- _lc_lT )≤ e^-cLT for c=minl⁡cl/Lc= _lc_l/L, by applying the product bound and lemma 1 to the log-sum. Thus the collapse probability is ≥1−e−cL≥ 1-e^-cL. This result motivates the load-balancing regularizer ℒbalanceL_balance (section 7) and the curriculum-learning schedule that gradually increases the target skip rate, both of which counteract the drift toward collapse. □ ∎ Observation 18 (Critical skip rate (heuristic estimate)). There exists a critical skip rate ρc _c beyond which model representational capacity degrades sharply. Under the uniform-layer assumption, ρc≈1−dfeat/L _c≈ 1-d_feat/L, where dfeatd_feat is the effective feature dimension. Beyond ρc _c, the retained layers can no longer span the full feature space, and the model loses the ability to represent certain input-dependent transformations. This heuristic, combined with observation 14, informs the choice of the target skip rate ρ∗ρ^* in the difficulty-aware schedule (proposition 4). 9 Proof-of-Concept Evaluation We conduct a preliminary empirical evaluation of the VALSE prototype to test whether difficulty-adaptive per-sample layer skipping can be realized at prototype scale. We emphasize that these experiments constitute a proof-of-concept study at prototype scale, not a comprehensive evaluation. All experiments were conducted on a single CPU machine without GPU access, severely constraining the feasible model size, training duration, and evaluation set. As reported below, the core hypothesis of difficulty-adaptive routing was not verified: the router exhibited a routing-collapse failure mode (Pearson r=−0.017r=-0.017), and the results should be interpreted as a transparent negative result rather than as evidence of feasibility or as performance benchmarks. The theoretical analysis of section 8 provides the primary justification for the approach. 9.1 Setup Model. A 1212-layer Transformer encoder with dmodel=256d_model=256, 44 attention heads, dffn=1024d_ffn=1024, vocabulary size 8,0008,000, and Kest=2K_est=2 (11.511.5M parameters). The difficulty estimator and router add <1%<1\% overhead. Dataset. SST-2 (sentiment classification), 100100 evaluation samples. Training. The prototype was trained from scratch (no pre-training) on CPU with ∼ 300 training samples for 1818 epochs (batch size 3232), using hard routing (Gumbel-STE) with load-balancing and efficiency losses; α=0.03α=0.03, β=0.05β=0.05, Btarget=0.55LB_target=0.55L. This short schedule and small training set reflect prototype-scale resource constraints rather than a fully converged training run. Curriculum learning Stage 1 is omitted; Stages 2–3 are used. Methods compared. Four inference strategies evaluated on the same trained model: full-depth (all 1212 layers), early exit (confidence-based truncation), random skip (uniformly retaining ∼ 8 layers), and VALSE adaptive (difficulty-driven per-sample skipping). Metrics. Accuracy, average activated layers, and theoretical FLOPs savings. All results are from a single run (single random seed, seed=42 seed=42); statistical significance is not assessed at prototype scale. 9.2 Main Results Table 3 compares the four inference strategies on SST-2. Under the adaptive strategy, all 100100 evaluation samples were routed to exactly 33 layers, yielding an apparent reduction in average activated layers from 12.012.0 to 3.03.0 (62.1%62.1\% theoretical FLOPs reduction) at a 11 p accuracy change (0.63→0.620.63→ 0.62). This reduction, however, is a consequence of routing collapse—a degenerate uniform-skip-to-minimum-depth failure mode—rather than of intelligent, difficulty-adaptive per-sample routing; the core hypothesis is therefore not verified by this result. Table 3: Prototype-scale (1212-layer, d=256d=256) comparison of inference strategies on SST-2 (100100 samples, single seed). These are not performance benchmarks; absolute accuracy is limited by model scale, absence of pre-training, and short training. The apparent FLOPs savings of VALSE reflect a routing collapse (all samples routed to 33 layers) rather than effective difficulty-adaptive routing; the core hypothesis is not verified (Pearson r=−0.017r=-0.017, see section 9.3). Method SST-2 Accuracy Avg. layers FLOPs savings Full-depth 0.63 12.00 0.0% Early exit 0.63 11.56 3.0% Random skip 0.63 8.00 27.6% VALSE (adaptive) 0.62 3.00 62.1% Routing collapse. The layer distribution reveals that all 100100 evaluation samples were routed to exactly 33 layers under the adaptive strategy. This confirms a routing-collapse failure mode: the difficulty signal did not produce differentiated gating across samples, and the router adopted a degenerate uniform-skip-to-minimum-depth strategy rather than the intended difficulty-adaptive behavior. The FLOPs savings are therefore a consequence of collapse, not of intelligent per-sample routing. This is consistent with the known sensitivity of load-balancing losses to the hyperparameter α and the limited training budget. 9.3 Core Hypothesis: Negative Result The central hypothesis of VALSE is that difficulty and activated depth should be positively correlated—easy inputs skip more layers, hard inputs use more. We tested this on a separate 1212-layer model trained with a curriculum schedule (150150 SST-2 + 150150 GSM8K samples): • Pearson correlation between difficulty score d d and activated layers: r=−0.017r=-0.017 (negative, opposite to the hypothesized direction). • Layer gap (GSM8K −- SST-2): −0.47-0.47 layers—hard tasks used fewer layers than easy tasks, contrary to the hypothesis. • Routing differentiation: false—both tasks activated near-full depth, indicating collapse to maximum rather than differentiated routing. We report this transparently as a negative result: the core hypothesis of difficulty-adaptive depth routing was not verified at prototype scale. The difficulty signal did not effectively propagate from the estimator to the routing decisions, and the router exhibited collapse to extreme depth (minimum or maximum) rather than differentiated, difficulty-aware routing. This does not invalidate the theoretical framework—which holds regardless of training dynamics—but it indicates that the training procedure and router architecture require substantial improvement before the theoretical predictions can be empirically confirmed. 9.4 Theory-to-Experiment Mapping Table 4 summarizes the relationship between theoretical claims and their experimental verification status. Due to routing collapse and prototype-scale limitations, no claim is fully verified; several are supported only indirectly or remain purely theoretical. Table 4: Mapping of theoretical claims to experimental verification status. “∼ ” = indirect or partial (prototype scale only); “—” = not yet verified. No claim is fully verified. Theoretical claim Source Verification status Expected FLOPs reduction formula theorem 2 ∼ Prototype observation Difficulty–depth monotonicity (core hypothesis) proposition 4 — Not verified (r=−0.017r=-0.017) Difficulty-aware FLOPs savings range observation 5 ∼ Prototype scale only Non-contiguous skip info preservation observation 13 ∼ Indirect (accuracy preserved) MoE–VALSE structural duality proposition 6 — Top-K variant pending Routing-collapse risk proposition 17 ∼ Empirically observed Critical skip rate (heuristic) observation 18 ∼ Informs ρ∗ρ^* choice Hybrid architecture FLOPs advantage proposition 8 — Not verified (theory only) Full-layer function-space superset (Scheme A) theorem 10 ∼ Indirect (accuracy bounded) Strict containment (Scheme A) theorem 11 — Not verified (theory only) Notably, the routing-collapse phenomenon empirically corroborates proposition 17 (routing-collapse risk), confirming that the theoretical concern about degenerate solutions is practically relevant and motivating the load-balancing and curriculum-learning mechanisms designed to mitigate it—though these mechanisms proved insufficient at prototype scale. 9.5 Limitations We transparently list the key limitations: 1. Model scale: The 1212-layer, 11.511.5M-parameter model is far below the scale at which layer redundancy has been documented (1212+ layers, 11B+ parameters, pre-trained). 2. No pre-training: The model is trained from scratch on ∼ 300 samples, limiting absolute accuracy. 3. Single dataset & single seed: Only SST-2 is evaluated with one random seed; statistical significance is not assessed. 4. Routing collapse: The core hypothesis is not verified; the router collapsed rather than learning differentiated routing (Pearson =−0.017=-0.017). 5. No external baselines: No comparison with DeeBERT, CALM, LayerDrop, MoD, ShortGPT, or SliceGPT. 6. Incomplete ablations: Five ablation groups were designed but not fully executed. 7. No wall-clock speedup: Theoretical FLOPs savings are not realized as latency reduction due to implementation overhead. These limitations define a clear roadmap for future work (section 10); we present the prototype results as preliminary and incomplete evidence, not as definitive validation. 10 Conclusion and Discussion We proposed VALSE, extending conditional computation from the width dimension of horizontal MoE to the depth dimension via per-sample, difficulty-driven, non-contiguous layer skipping. VALSE comprises a difficulty estimation module, three routing strategies (hard, soft, global), a load-balancing and curriculum-learning training scheme, and a theoretical framework establishing the MoE–VALSE structural duality and two-dimensional sparsity. 10.1 Summary of Contributions • Methodology: Per-sample non-contiguous layer skipping via an interpretable difficulty estimator and three routing variants, breaking through the truncation limitation of early exit. • Training: Four auxiliary losses—layer utilization balancing, efficiency, output consistency, and Z-loss—plus a three-stage curriculum learning schedule, jointly addressing routing collapse and training instability. • Theory: Closed-form expected FLOPs reduction (theorem 2), MoE–VALSE structural duality (proposition 6), and unified width×depth sparsity (definition 7, theorem 9). • Proof-of-concept: A prototype-scale evaluation on SST-2 provides preliminary—though inconclusive—evidence for the layer-skipping mechanism. The core hypothesis (difficulty-adaptive routing) was not verified at prototype scale; full-scale validation on larger, pre-trained models remains future work. 10.2 Key Findings and Limitations • On SST-2, VALSE reduces average activated layers from 12.012.0 to 3.03.0 (62.1%62.1\% theoretical FLOPs reduction) with a 11 p accuracy change (0.63→0.620.63→ 0.62). However, this resulted from routing collapse—all samples were routed to the minimum depth—not from intelligent per-sample routing. • The core hypothesis (difficulty-adaptive depth routing) was not verified: the Pearson correlation between difficulty and activated layers was r=−0.017r=-0.017 (negative, opposite to the hypothesized direction). The difficulty signal did not effectively propagate to routing decisions. • The difficulty-stratified analysis confirms routing collapse: the router did not learn differentiated, difficulty-aware routing, indicating that the training procedure and router architecture require substantial improvement. • Limitations: (i) Prototype scale (1212 layers, single dataset, single seed) limits all conclusions; (i) no external baselines are compared; (i) ablations are designed but not fully executed; (iv) the core empirical hypothesis is not verified (routing collapse, Pearson =−0.017=-0.017); (v) theoretical assumptions (bounded skip rate, stability condition) are only partially validated; (vi) wall-clock speedup is not realized due to implementation overhead. 10.3 Future Work 1. Scale up: Validate on 1212+-layer, 11B+ models with pre-trained weights to assess hard-task skipping and confirm layer redundancy at scale. 2. Multi-dataset evaluation: Extend to diverse tasks (classification, QA, reasoning, generation) to test the universality of the layer redundancy hypothesis. 3. Complete ablations: Execute all five ablation groups (routing strategy, difficulty estimator, load balancing, efficiency loss, fixed skip) with multiple seeds. 4. Statistical rigor: Run 33–55 seeds per configuration, report mean± , and perform paired t-tests or Wilcoxon tests. 5. Cure routing collapse: Stronger load balancing, annealed ρ∗ρ^* schedules, and contrastive routing objectives to ensure differentiated routing across difficulty levels. 6. External baselines: Compare against DeeBERT, LayerDrop, MoD, ShortGPT, and random skip under matched compute budgets. 7. Engineering: Gate kernel fusion, batched routing, and custom CUDA kernels to convert FLOPs savings into real latency reduction. 8. Two-dimensional sparsity: Combine VALSE with MoE for full width×depth sparse activation. 9. Generative LLMs: Extend per-sample skipping to autoregressive generation, with self-speculative decoding for quality safety. 10.4 Broader Impact VALSE aims to reduce the computational cost of LLM inference by activating only the necessary depth per input. If validated at scale, this could lower the energy footprint of inference workloads and democratize access to efficient model deployment. Potential risks include accuracy degradation on hard inputs if the difficulty estimator is poorly calibrated; the difficulty-aware routing and curriculum learning safeguards are designed to mitigate this. We commit to transparent reporting of both accuracy and efficiency trade-offs. Appendix A Symbol Table Table 5 lists all symbols used in the paper. Table 5: Complete symbol table. Symbol Meaning L Total number of Transformer layers l∈1,…,Ll∈\1,…,L\ Layer index dmodeld_model Hidden dimension N Sequence length hl∈ℝN×dmodelh_l ^N× d_model Input hidden state at layer l hl(s)h_l^(s) Sample s’s hidden state at layer l (sequence-pooled) x(s)x^(s) Input embedding for sample s Fl(⋅)F_l(·) Forward function of the l-th Transformer block KestK_est Number of initial layers used for difficulty estimation rl∈ℝdfeatr_l ^d_feat Difficulty feature vector (dfeat=5d_feat=5) R(s)∈ℝ5KestR^(s) ^5K_est Aggregated feature vector d⁡(x(s))∈ℝd(x^(s)) Per-sample difficulty score d^(x(s))∈[0,1] d(x^(s))∈[0,1] Normalized difficulty score gl(x(s))∈[0,1]g_l(x^(s))∈[0,1] Soft gate value at layer l g~l∈0,1 g_l∈\0,1\ Quantized hard gate decision S(s)S^(s) Retained layer set for sample s θl _l Quantization threshold (default 0.5) WlprojW_l^proj Residual alignment projection (learnable, optional) ℒtaskL_task Main task loss (cross-entropy) ℒbalanceL_balance Layer utilization balance loss ℒeffL_eff Computational efficiency loss ℒconsistL_consist Output consistency loss (KL divergence) ℒzL_z Z-loss regularization on gate logits ℒtotalL_total Total loss function α,β,γ,δα,β,γ,δ Auxiliary loss weights BtargetB_target Target average activated layers (e.g., 0.7L0.7L) ρ∗ρ^* Target skip rate (e.g., 0.3) TKDT_KD Knowledge distillation temperature plp_l Skip rate at layer l; ql=1−plq_l=1-p_l is retention rate ClC_l Per-sample FLOPs of the l-th block CfixedC_fixed FLOPs of the always-executed part fullF_full Full-layer network FLOPs VALSEF_VALSE VALSE skip-layer network FLOPs (random variable) Φ⁡(⋅) (·) Standard normal CDF Table 5: Complete symbol table (continued). Appendix B Supplementary Tables Table 6: Adaptive depth methods and their limitations. Method Core idea Limitation DeeBERT/FastBERT/PABEE Per-sample confidence-based early exit Truncation: cannot skip middle layers CALM Confidence-based early exit for generative LMs Truncation: per-sample but prefix-only ACT/Universal Transformers Shared-weight recurrence with halting Deepens, not skips; not for fixed-layer models LayerDrop Structured dropout for depth robustness Static/global depth at inference LayerSkip Layer dropout with self-speculative decoding Static/per-token, not per-sample difficulty-driven DynamicBERT Independent width & depth adjustment No unified routing; not difficulty-aware ShortGPT Block Influence score quantifies layer redundancy Static post-training pruning; no input adaptivity Streamline Structured layer pruning at post-training stage Static; produces fixed reduced-depth model SliceGPT Structured slicing/pruning of layers Static; no per-sample adaptivity MoNE MoE routing across encoder–decoder layer pairs Per-token; no sample-level difficulty semantics MoD Per-token per-layer routing Not per-sample; no difficulty semantics Mixtral/DeepSeek-MoE Width-wise sparse expert activation Width-only: depth fixed; no difficulty dimension Switch Transformer/GShard Top-1/Top-K expert routing at scale Width-only: depth fixed; semantic per-token routing Drowzee Pre-inference per-sample budget prediction Budget controls truncation only VALSE (ours) Per-sample difficulty-aware non-contiguous skip — Table 7: Skip mode comparison. Method Skip mode Can skip middle and retain deeper layers? Truncation (DeeBERT/CALM) S=1,…,kS=\1,…,k\ No MoD (per-token, per-layer) Each token decides independently Partially, not per-sample VALSE (ours) S is any subset, per-sample Yes, non-contiguous + per-sample Table 8: Mathematical analogy between MoE routing and VALSE layer routing. MoE routing VALSE layer routing Analogy Expert set E1,…,E\E_1,…,E_E\ Layer set F1,…,FL\F_1,…,F_L\ Expert ↔ layer: selectable compute unit Gate G⁡(x)=TopK⁡(Wg⋅x)G(x)=TopK(W_g· x) Gate gl=σ⁡(wl⊤⋅[rl;d^])g_l=σ(w_l ·[r_l; d]) Lightweight router scores units Top-K selects K experts Threshold selects retention set S MoE selects “which experts” → VALSE selects “which layers” y=∑i∈TopKGi⋅Ei(x)y= _i G_i· E_i(x) hl+1=hl+g~l⋅Fl(hl)h_l+1=h_l+ g_l· F_l(h_l) Activated unit output weighted into flow Load balance ℒauxL_aux Layer balance ℒbalanceL_balance Prevents “few units over-used” collapse Capacity factor Compute budget ≤Btarget≤ B_target Controls sparsity Table 9: Architecture variant comparison. Dimension VALSE-Hard (A) VALSE-Soft (B) VALSE-Global (C) Routing type Hard (STE) Soft (continuous) Global Top-K Gate parameters Per-layer learnable Per-layer learnable Global router Difficulty awareness d d enters gate d d enters gate d d sets K Inference FLOPs savings Yes, true skip Requires g→0g→ 0 Yes, true skip Training stability Needs STE + noise Most stable Moderate Budget controllability Approximate (regularized) Approximate (regularized) Exact (Top-K) MoE analogy strength Moderate Low High (Top-K dual) Appendix C Supplementary Theoretical Results The theorems and propositions previously located in this appendix—including the function-space analysis (theorems 10 and 11), the non-contiguous retention-set counting (proposition 12), information-preservation and capacity-rank observations (observations 13 and 14), and the optimization properties (observations 15, 16, 17 and 18)—have been migrated to the main text in sections 8.5 and 8.6 to strengthen the theoretical focus of the paper. This appendix retains supplementary results that complement, rather than duplicate, the main-text development: a routing-expressivity comparison and an expected-FLOPs characterization under the routing distribution. Proposition 19 (Routing expressivity of non-contiguous retention). The set of sample-level retention patterns realizable by VALSE is the full power set 21,…,L2^\1,…,L\ (cardinality 2L2^L), whereas truncation-based early exit realizes only the L prefix sets 1,…,k\1,…,k\ for k∈1,…,Lk∈\1,…,L\. Per-token per-layer routing (MoD) realizes 2L⋅N2^L· N token-level patterns, but when aggregated to a single sample-level retention set by per-layer majority vote over tokens, its effective sample-level image collapses to the truncation family under the standard monotone-confidence assumption. Thus only per-sample non-contiguous selection achieves the full 2L2^L sample-level expressivity. Proof sketch. VALSE’s per-sample threshold produces an arbitrary subset S(s)⊆1,…,LS^(s) \1,…,L\, so its image is 21,…,L2^\1,…,L\. Truncation constrains S=1,…,kS=\1,…,k\, yielding exactly L patterns. For MoD, each layer l is retained iff a token-majority votes to retain; under the monotone-confidence assumption that retention probability is non-increasing in depth, the set of retained layers is a prefix, so the sample-level image is again the truncation family. Hence only VALSE attains full 2L2^L sample-level expressivity. ∎ Proposition 20 (Expected FLOPs under the routing distribution). Let ClC_l denote per-sample FLOPs of the l-th block and assume Cl=C_l=C for all l. Let pl=Pr[l∉S(s)]p_l= [l∉ S^(s)] be the skip probability at layer l. Then the expected per-sample FLOPs of VALSE is ⁡[ℱVALSE]=ℱfixed+C∑l=1L(1−pl)E[F_VALSE]=F_fixed+C _l=1^L(1-p_l), and the expected savings ratio is ρFLOPs=1−⁡[ℱVALSE]/ℱfull=C∑l=1Lpl/ℱfull _FLOPs=1-E[F_VALSE]/F_full=C _l=1^Lp_l\,/\,F_full. Under the uniform-skip regime pl=ρ∗p_l=ρ^* for all l, this reduces to ρFLOPs=ρ∗⋅LC/ℱfull _FLOPs=ρ^*· LC/F_full, recovering the closed form reported as corollary 3. Remark (Cross-batch comparability of batch-internal normalization). The min–max normalization in eq. 3 maps the raw difficulty score d⁡(x(s))d(x^(s)) onto [0,1][0,1] using the batch extremum, so d d is a within-batch rank statistic rather than an absolute population quantity. Consequently, for two batches ℬ1,ℬ2B_1,B_2 with extrema (m1,M1)(m_1,M_1) and (m2,M2)(m_2,M_2), the equality d^ℬ1(x)=d^ℬ2(x) d_B_1(x)= d_B_2(x) does not imply equal population-level difficulty unless m1=m2m_1=m_2 and M1=M2M_1=M_2. Routing remains correct because the gate g~l g_l consumes only within-batch ordering, but cross-experiment comparison of d d requires calibration to a shared reference (e.g., a running population extremum or a fixed-temperature rank transform). Proposition 21 (Routing collapse as a degenerate equilibrium). Consider the VALSE training objective ℒtotal=ℒtask+αℒbalance+βℒeff+γℒz+δℒconsistL_total=L_task+ _balance+ _eff+ _z+ _consist with gate bias blb_l initialized so that gl≈1g_l≈ 1 at t=0t=0. The uniform-skip-to-minimum-depth configuration S(s)=SminS^(s)=S_ for all s (i.e., all samples share the same minimal retention set) is a stationary point of the gate parameters whenever (i) the task loss gradient ∇blℒtask _b_lL_task is smaller in magnitude than ∇bl(αℒbalance+βℒeff) _b_l( _balance+ _eff) at SminS_ , and (i) the difficulty–gate Jacobian ∂g~l/∂d^∂ g_l/∂ d is rank-deficient so that the difficulty signal cannot break the symmetry across samples. Under these two conditions, the collapse configuration is a locally stable equilibrium: any perturbation that would differentiate samples is dominated by the regularization pulling all gates toward the balance-optimal uniform profile. Proof sketch. At SminS_ , all gates take the same value across samples by construction, so ℒbalanceL_balance is at its minimum (uniform utilization) and ℒeffL_eff is at its target. Differentiating any single gate glg_l for a single sample s to break uniformity increases ℒbalanceL_balance linearly in α and changes ℒeffL_eff linearly in β, while the task-loss reduction is O⁡(1/N)O(1/N) per sample (a single-sample perturbation in a batch of size N). Whenever α+β≫1/Nα+β 1/N, the regularizing gradient dominates the task gradient at SminS_ , making it a stationary point. Local stability follows because the Hessian of the regularizers is positive semidefinite in the gate-bias direction, so second-order growth of ℒbalance+ℒeffL_balance+L_eff exceeds any concave task-loss gain in a neighborhood of SminS_ . Condition (i) ensures the difficulty signal cannot supply the missing symmetry-breaking gradient: if ∂g~l/∂d^∂ g_l/∂ d is rank-deficient (e.g., the gate saturates near the threshold so the STE Jacobian vanishes), then even a well-separated d d across samples cannot induce differentiated gating. ∎ This result reframes the prototype’s r=−0.017r=-0.017 not as a refutation of the difficulty–depth monotonicity of proposition 4—which is a population-level statement under the decreasing-marginal-contribution assumption and is independent of the learned estimator—but as evidence that the prototype’s hyperparameters landed in the basin of attraction of the collapse equilibrium of proposition 21. The monotonicity guarantee and the collapse equilibrium are logically independent: the former concerns the optimal schedule, the latter concerns the optimizer’s fixed point, and a mismatch between the two is precisely the failure mode the prototype exhibited. Remark (Scope and limitations of the theoretical framework). The theoretical results in section 8 and this appendix establish three classes of statement: (a) structural facts about the retention-set space (theorems 10, 11, 12 and 19), which are combinatorial and hold without approximation; (b) expected-FLOPs identities (theorems 2, 20 and 3), which are exact under the stated per-block FLOPs model; and (c) monotonicity and stability statements (propositions 4 and 21 and the stability condition eq. 14), which are conditional on assumptions—decreasing marginal layer contribution, Lipschitz difficulty estimator, bounded gate gradient—whose empirical validity at scale is not established by the prototype. We therefore claim theoretical soundness of the structural and accounting results unconditionally, but only conditional soundness of the monotonicity and stability results. In particular, proposition 21 identifies a failure mode but does not prescribe a hyperparameter regime that provably avoids it; deriving such a regime (e.g., a sufficient condition on α,βα,β relative to batch size and gate-saturation level that excludes collapse) is left as an open problem, and the prototype’s negative result is consistent with operating inside the collapse basin. Appendix D Reproducibility This section provides all information needed to reproduce the experiments. D.1 Code Structure The codebase is organized into the following key files, to be released with the camera-ready version: Table 10: Code structure: key files and their responsibilities. File Responsibility config.py Central configuration (VALSEConfig dataclass) model.py Transformer backbone with skip-layer support router.py Difficulty estimator and layer gating train.py Main training entry point (baseline + adaptive; includes evaluate() for the main-results comparison) prepare_data.py Data download and preprocessing Note on the analysis entry point. The codebase comprises the five core files above plus auxiliary analysis scripts, which together form a self-contained prototype pipeline; these files are not bundled with the current submission and will be released with the camera-ready version: config.py specifies all hyperparameters, prepare_data.py produces the datasets, and train.py runs baseline and adaptive training and embeds the evaluate() routine that computes the main-results comparison (accuracy, average activated layers, theoretical FLOPs savings) and the difficulty–depth correlation analysis (Pearson r, layer-gap, routing-differentiation flag). The prototype evaluation set is small enough to be exercised through evaluate() directly, which serves as the unified entry point for all reported results. D.2 Hyperparameters Table 11 lists the complete hyperparameter set used in the main experiment. Table 11: Complete hyperparameter list for the main experiment. Category Parameter Value Model num_layers 66 hidden_dim 256256 num_heads 44 ffn_dim 10241024 vocab_size 8,0008,000 KestK_est 22 Routing routing_mode hard (Gumbel-STE) gate_bias_init 2.02.0 gate_threshold 0.50.5 gumbel_tau 1.01.0 (annealed 2.0→0.52.0→ 0.5) Loss weights α (balance) 0.010.01 β (efficiency) 0.10.1 γ (Z-loss) 10−410^-4 δ (consistency) 0.50.5 (annealed to δ/3δ/3) TKDT_KD 2.02.0 Budget BtargetB_target 0.7L=4.20.7L=4.2 ρ∗ρ^* 0.30.3 precompp_recomp 0.10.1 Training learning rate 3×10−43× 10^-4 weight decay 0.10.1 betas (0.9,0.95)(0.9,0.95) batch size 3232 epochs 1818 warmup steps 5050 max grad norm 1.01.0 train subset ∼ 300 optimizer AdamW LR schedule Linear warmup + cosine decay Curriculum Stage 1 (ρ=0ρ=0) η1≈0.25 _1≈ 0.25–0.330.33 (warmup, ρ=0ρ=0 baseline) Stage 2 (0→ρ∗0→ρ^*) η2≈0.5 _2≈ 0.5–0.670.67 Stage 3 (ρ=ρ∗ρ=ρ^*) η3≈0.25 _3≈ 0.25–0.40.4 D.3 Training and Evaluation Commands Environment setup. uv venv .venv --python 3.13 .venv pip install torch numpy matplotlib datasets pandas pyyaml tqdm \ requests huggingface_hub pyarrow \ -i https://pypi.tuna.tsinghua.edu.cn/simple Data preparation. python prepare_data.py This downloads SST-2 via datasets.load_dataset("stanfordnlp/sst2") and GSM8K via datasets.load_dataset("openai/gsm8k", "main"), then splits into train/val/test JSONL files in data/. Training (baseline + adaptive). python train.py This trains both the full-depth baseline and the VALSE adaptive model (66 layers). Training logs are written to train_log.txt, which is not bundled with the current submission and will be provided with the camera-ready version. Evaluation and analysis. The main-results comparison (accuracy, average activated layers, theoretical FLOPs savings) and the difficulty–depth correlation analysis (Pearson r, layer-gap, routing-differentiation flag) are computed inside train.py’s evaluate() routine and are reported in section 9; the corresponding numbers are written to the training log (train_log.txt), which is not bundled with the current submission and will be provided with the camera-ready version. The evaluate() routine serves as the unified entry point for all reported prototype results, and its single-call invocation constitutes the complete evaluation harness for the submission. D.4 Dataset Acquisition • SST-2: datasets.load_dataset("stanfordnlp/sst2") from HuggingFace. 500500 test samples used in the main experiment; 500500 in the difficulty analysis. • GSM8K: datasets.load_dataset("openai/gsm8k", "main") from HuggingFace. 500500 test samples in the main experiment; 500500 in the difficulty analysis. If HuggingFace is inaccessible, set HF_ENDPOINT=https://hf-mirror.com. D.5 Environment Dependencies Table 12: Environment dependencies (minimum compatible versions; the prototype was developed under Python 3.13). Package Version Purpose torch ≥2.2.0≥ 2.2.0 Deep learning framework numpy ≥1.26.0≥ 1.26.0 Numerical computation matplotlib ≥3.8.0≥ 3.8.0 Visualization datasets ≥2.18.0≥ 2.18.0 HuggingFace data loading pandas ≥2.2.0≥ 2.2.0 Data processing pyarrow ≥15.0.0≥ 15.0.0 Arrow format pyyaml ≥6.0.1≥ 6.0.1 YAML configuration tqdm ≥4.66.0≥ 4.66.0 Progress bar huggingface_hub ≥0.23.0≥ 0.23.0 Model/data download D.6 Random Seeds and Checkpoints • Random seed: The prototype uses a fixed seed of seed=42 seed=42 for reproducibility and reports the point estimate; the production configuration (a future target, not executed in this submission) specifies 55 independent seeds (42,123,456,789,102442,123,456,789,1024) with mean± reporting, and the prototype adopts the single-seed protocol because its purpose is hypothesis-testing (the difficulty–depth correlation) rather than benchmarking. • Checkpoint: The prototype checkpoint is designed to be saved to checkpoints/ after training; the current release leaves this directory empty, with the .pt weights deferred to the camera-ready version. • One-command reproduction: The prototype pipeline is reproduced via the two-command sequence prepare_data.py then train.py, which executes the full end-to-end pipeline (data preparation, baseline training, adaptive training, and evaluation) and constitutes the canonical reproduction entry point. • Determinism note: PyTorch operations may not be fully deterministic across hardware. We use torch.manual_seed and numpy.random.seed; full determinism requires torch.use_deterministic_algorithms(True), which may reduce performance. D.7 Prototype Observations The prototype run (logged in train_log.txt) yielded the following observations, also reported in sections 9.2 and 9.3: full-depth SST-2 accuracy 0.630.63, VALSE adaptive accuracy 0.620.62 with average 3.03.0 layers (62.1%62.1\% theoretical FLOPs savings). Crucially, the adaptive routing collapsed to minimum depth for all samples and the core hypothesis (Pearson r>0r>0) was not verified (r=−0.017r=-0.017). These are prototype-scale observations of a documented negative result, not performance benchmarks; the canonical results artifact for the submission is train_log.txt (to be provided with the camera-ready version, not bundled with the current release), in which evaluate() records all reported numbers. The complete reproduction guide is also available in REPRODUCE.md in the repository root. D.8 Diagnosis of the Prototype Negative Result The prototype’s core negative result—Pearson r=−0.017r=-0.017 between d d and activated layers, with all samples routed to the same 33-layer retention set—admits a precise theoretical diagnosis via proposition 21. Three factors, each identifiable from the configuration in table 11, plausibly placed the prototype inside the collapse basin identified above. • Regularizer-to-task gradient imbalance. The prototype uses α=0.01α=0.01 (balance) and β=0.1β=0.1 (efficiency) at batch size 3232. By the proof sketch of proposition 21, collapse is locally stable whenever α+β≫1/Nα+β 1/N; here α+β=0.11α+β=0.11 and 1/N=0.0311/N=0.031, so the regularizing gradient exceeds the per-sample task gradient by roughly 3.5×3.5×, satisfying condition (i). Reducing α,βα,β or increasing N would narrow the collapse basin, though both trade off against budget control and training stability. • Gate saturation and STE Jacobian rank deficiency. The gate bias is initialized to bl=2.0b_l=2.0, yielding gl=σ⁡(2.0)≈0.88g_l=σ(2.0)≈ 0.88 at start; combined with the hard threshold θl=0.5 _l=0.5, this places the initial operating point near the saturation knee of the sigmoid, where the straight-through-estimator Jacobian ∂g~l/∂gl∂ g_l/∂ g_l is concentrated at the threshold crossing and is near-zero elsewhere. This satisfies condition (i) of proposition 21: the difficulty signal cannot break the across-sample symmetry of the gates when the STE Jacobian is rank-deficient in the saturated regime. • Curriculum simplification. The prototype omits Stage 1 (ρ=0ρ=0 warmup) and uses a fixed Gumbel temperature τG=1.0 _G=1.0 instead of the designed 2.0→0.52.0→ 0.5 anneal (section D.2). The full curriculum is precisely the mechanism intended to move the optimizer out of the collapse basin by first training at ρ=0ρ=0 (where all gates are open, breaking the saturation regime) before introducing sparsity; omitting it removes the principal escape trajectory. This diagnosis yields three concrete, theory-grounded remediations that follow directly from proposition 21: (a) reduce α,βα,β or scale N so that α+β≲1/Nα+β 1/N, shrinking the collapse basin; (b) lower the gate-bias initialization blb_l and anneal τG _G from 2.0→0.52.0→ 0.5 to keep the STE Jacobian non-degenerate during the differentiation phase; and (c) restore the full three-stage curriculum so that Stage 1 trains at ρ=0ρ=0 before sparsity is introduced. These remediations are consistent with, and refine, the higher-level future-work items in section 10 (cure routing collapse, complete curriculum). Because the diagnosis is conditioned on proposition 21—whose stability assumptions are themselves unverified at scale—we present it as a falsifiable hypothesis for the next iteration rather than as a confirmed root cause. D.9 Limitations This work carries limitations along three axes, each already surfaced in the main text but collected here for the NeurIPS reviewer’s convenience. • Empirical scope. The prototype validates only the discriminative SST-2 routing pathway at a 66-layer scale and reports a documented negative result (r=−0.017r=-0.017). The difficulty–depth monotonicity (proposition 4) is a population-level theoretical statement whose empirical validity at scale is not established by the prototype, and the variance-decomposition analysis of individual difficulty features remains contingent on a non-collapsed routing regime. • Collapse-regime avoidance. proposition 21 identifies a degenerate equilibrium of the gate parameters but does not prescribe a hyperparameter regime that provably excludes it; deriving a sufficient condition on α,βα,β relative to batch size and gate-saturation level that excludes collapse is left as an open problem. The three remediations in section D.8 (reduce α,βα,β, lower blb_l and anneal τG _G, restore the three-stage curriculum) are theory-grounded proposals for the next iteration, not executed runs. • Autoregressive generation. The architectural extensibility of VALSE to autoregressive generation (position-agnostic gate, causal-mask compatibility, self-speculative decoding fallback) is a present design property of the routing mechanism, not an empirically validated claim; the systematic validation on autoregressive generative LLMs is identified as the central community-wide open problem in section 2.4. D.10 Ethics Statement This work proposes a routing mechanism for adaptive-depth Transformer inference. The prototype uses SST-2 (sentiment) and GSM8K (grade-school math) datasets, both publicly available and not containing sensitive personal data. The method reduces computational cost per inference, which can lower the energy footprint of Transformer deployment; we are not aware of specific misuse vectors introduced by per-sample layer skipping beyond those already applicable to the underlying base models. We do not release new pretrained model weights beyond the prototype-scale checkpoint. D.11 Broader Impact The primary broader impact of adaptive-depth routing is the potential to reduce inference FLOPs and hence the per-query energy cost of Transformer models, which is environmentally and economically beneficial at scale. Per-sample, difficulty-aware skipping can also enable adaptive compute allocation (more depth for harder inputs, less for easier ones), which is a step toward input-conditional efficiency. Risks include the collapse-to-minimum-depth failure mode documented in this work (which silently negates the efficiency–quality trade-off and must be monitored via the difficulty–depth correlation diagnostic) and the risk that difficulty scores encode dataset-specific biases when the feature set is not re-validated on new domains; both are flagged here as deployment-time monitoring obligations rather than as resolved design properties. References Ashraf et al. (2024) S. Ashraf, S. Damaraju, W. Coad, M. Arezin, A. Bhattacharjee, E. Sotirakos, M. McCarley, J. Kautz, M. Birch, J. Henderson, et al. SliceGPT: compress large language models by deleting and slicing layers. arXiv preprint arXiv:2401.15024. Cited by: §2.1, Table 1. Bachmann et al. (2023) G. Bachmann, N. Mırzamomen, T. Schuster, R. Hubis, and L. Aitchison Drowzee: dynamic pre-inference for BERT with per-sample compute budget. arXiv preprint arXiv:2305.17698. Cited by: §2.1. Dai et al. (2024) D. Dai, C. Deng, D. Zhang, S. Xu, X. Zhu, H. Gao, Z. Hu, Y. Wu, P. Wang, B. Ma, L. Zhang, Y. Liu, T. Bi, S. Zhang, H. Wu, L. Wu, et al. DeepSeek-MoE: towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066. Cited by: §1, §2.2, Table 1. Dehghani et al. (2019) M. Dehghani, S. Gouws, O. Vinyals, N. Usunier, and S. Ganguli Universal transformers. In International Conference on Learning Representations (ICLR), Cited by: §1, §2.1. Elhoushi et al. (2024a) M. Elhoushi, Z. Lai, L. Shkuro, J. Loubes, V. Braverman, J. Ding, A. Kim, N. Kozlov, Q. Tan, B. Balle, and G. Kahn Layer Skip: enabling early-exit inference and self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §1, §2.1. Elhoushi et al. (2024b) M. Elhoushi, A. Shaked, Z. Lai, R. Lai, W. Sukkar, L. Shkuro, V. Braverman, J. Ding, S. Aizman, and G. Kahn The layers they are a-changin’: a simple and efficient pruning method for Transformer models. arXiv preprint arXiv:2403.05704. Cited by: §2.1. Fan et al. (2020) A. Fan, E. Grave, A. Joulin, and T. Mikolov Reducing Transformer depth on demand with structured dropout. In International Conference on Learning Representations (ICLR), Cited by: §1, §2.1, Table 1. Fedus et al. (2022) W. Fedus, B. Zoph, and N. Shazeer Switch Transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research (JMLR) 23, p. 1–39. Cited by: §2.2. Graves (2016) A. Graves Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1607.01786. Cited by: §1, §2.1. Jiang et al. (2024) A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, G. Lample, L. Sayed, T. Lachaux, B. Rozière, and A. Lacroix Mixtral of experts. arXiv preprint arXiv:2401.04088. Cited by: §2.2, Table 1. Lepikhin et al. (2020) D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen GShard: scaling giant models with conditional computation. arXiv preprint arXiv:2006.16668. Cited by: §2.2. Leviathan et al. (2023) Y. Leviathan, M. Kalman, and Y. Matias Fast inference from Transformers via speculative decoding. arXiv preprint arXiv:2211.17192. Cited by: §2.1, §3.4. Liu et al. (2020) W. Liu, P. Zhou, Z. Wang, Z. Chen, L. Cao, Y. Qi, and H. Ji FastBERT: a speedup system for BERT inference. In Findings of the Association for Computational Linguistics: EMNLP, p. 3243–3253. Cited by: §1, §2.1. Liu et al. (2021) Z. Liu, Z. Zhang, X. Xie, K. Liu, and F. Ding DynamicBERT: enhancing BERT’s efficiency via adaptive width and depth. arXiv preprint arXiv:2103.12241. Cited by: §2.1. Men et al. (2024) X. Men, Q. Xu, Y. Wang, L. Wang, H. Lin, Y. Lu, and W. Han ShortGPT: layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853. Cited by: §2.1, Table 1, §5.3, §8.2. Mustafa et al. (2024) N. Mustafa, J. Rethmeier, V. Karthikeyan, B. Raj, S. Honari, F. Lanusse, K. Szafer, and K. Choromanski MoNE: mixture of nested encoders for Transformer models. arXiv preprint arXiv:2401.16795. Cited by: §2.1. OpenAI et al. (2023) OpenAI, J. Achiam, S. Adams, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §1. Raposo et al. (2024) D. Raposo, A. Santoro, D. Faulkner, M. Grabska-Barwińska, A. Elhoseiny, T. Lillicrap, S. Kapturowski, and J. Howard Mixture-of-depths: dynamically allocating compute in Transformers. In International Conference on Machine Learning (ICML), Cited by: §1, §2.1, Table 1. Schuster et al. (2022) T. Schuster, A. Fisch, J. Gupta, M. Dehghani, D. Bahri, V. Q. Tran, Y. Tay, and D. Metzler Confident adaptive Language modeling. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2.1, Table 1. Shazeer et al. (2017) N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, and G. Hinton Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR), Cited by: §2.2. Touvron et al. (2023) H. Touvron, T. Lavril, G. Izacard, X. Martinet, T. Lachaux, M. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. LLaMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §1. Xin et al. (2020) J. Xin, R. Tang, J. Lee, Y. Ren, and J. Lin DeeBERT: enhancing BERT inference with adaptive computation time. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), p. 1435–1444. Cited by: §1, §2.1, Table 1. Xu et al. (2020) C. Xu, J. Zhou, K. Wang, Y. Lu, W. Zhang, and J. You BERT-of-theseus: compressing BERT by progressive replacing with a compact compass. In Findings of the Association for Computational Linguistics: EMNLP, p. 2855–2865. Cited by: §2.1. Zhou et al. (2020) J. Zhou, Z. Zhao, H. Shen, L. Zhan, Y. Liu, and W. Wang MoreThanGreedy: pipeline adaptable BERT with confident early exit (PABEE). In Findings of the Association for Computational Linguistics: EMNLP, p. 934–939. Cited by: §2.1.

Instant research discovery