Paper deep dive
Student-Guided Teacher Distillation for Efficient LLM Task Routing: Positioning Against Jev-Style System-1 Classifiers
Haifeng Wu, Srinivasan Manoharan, Jian Wan, Fangbo Tu, Junhua Zhao, Xin Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/5/2026, 6:22:19 AM
Summary
This paper presents a student-guided teacher distillation pipeline for efficient LLM task routing within a fixed taxonomy of 60 categories. The method uses a compact ModernBERT student to predict category distributions and retrieve top-k candidates, which are then reranked by a larger DeBERTa-v3 zero-shot NLI teacher. This iterative process reduces labeling costs while maintaining high teacher agreement (up to 77.5%). The authors contrast their approach with Jev-style System-1 classifiers and extreme multi-label classification methods, highlighting the importance of preserving 'dark knowledge' in soft-label distillation rather than using truncated top-k scores as KL targets.
Entities (10)
Relation Signals (7)
Student-Guided Teacher Distillation Pipeline → achieves → 77.5% teacher agreement
confidence 95% · Our best student checkpoint reaches 77.5% teacher agreement
PayPal → affiliatedwith → Haifeng Wu
confidence 95% · Affiliation: PayPal
TypeSafe AI → developed → Jev
confidence 95% · TypeSafe AI's Jev
ModernBERT → usedin → Student-Guided Teacher Distillation Pipeline
confidence 95% · a compact ModernBERT classifier predicts the full category distribution in one forward pass
DeBERTa-v3 → usedin → Student-Guided Teacher Distillation Pipeline
confidence 95% · a larger DeBERTa-v3 zero-shot NLI classifier reranks only those candidates
Jev → comparedwith → Student-Guided Teacher Distillation Pipeline
confidence 90% · Positioning Against Jev-Style System-1 Classifiers
Truncated top-k scores → destroys → Dark Knowledge
confidence 90% · zeroing untruncated classes destroys the dark knowledge soft-label distillation depends on
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Zero-shot classifiers are useful for routing user requests to specialized LLM tasks, but scoring every request against a large candidate set is expensive: a zero-shot NLI classifier must evaluate one premise-hypothesis pair per label, so cost scales linearly with taxonomy size. We study a student-guided teacher distillation pipeline for a fixed taxonomy of 60 LLM task categories: a compact ModernBERT classifier predicts the full category distribution in one forward pass and retrieves a small top-k candidate set, and a larger DeBERTa-v3 zero-shot NLI classifier reranks only those candidates rather than all 60 labels; the resulting teacher labels iteratively improve the student, which produces sharper candidates for the next round. Unlike generic embedding retrieval or clustering-derived shortlists used in extreme multi-label classification, our candidate generator is trained end-to-end on the target taxonomy and is the same model serving production traffic, distinguishing it from LLM-routing work that routes between candidate models, and from concurrent System-1 encoder-classifier proposals (e.g. TypeSafe AI's Jev and the open-source Laya project) whose training methodology is undocumented or RL-based. Our best student checkpoint reaches 77.5% teacher agreement on a 200-example evaluation set, and preliminary coverage measurements show Coverage@16 of 91-100%, suggesting top-k sets retain most of the teacher's decision-relevant information. We further show truncated top-k teacher scores should not be treated as full 60-class soft targets for KL distillation: zeroing untruncated classes destroys the dark knowledge soft-label distillation depends on, introducing systematic bias rather than a harmless sparse approximation. A complete evaluation, including coverage at multiple k on a held-out set, an embedding-retrieval baseline, and a larger human-reviewed test set, remains in progress.
Tags
Links
- Source: https://arxiv.org/abs/2610.02516v1
- Canonical: https://arxiv.org/abs/2610.02516v1
Trouble viewing inline? Open PDF directly →
Full Text
38,650 characters extracted from source content.
Expand or collapse full text
Student-Guided Teacher Distillation for Efficient LLM Task Routing: Positioning Against Jev-Style System-1 Classifiers Haifeng Wu Srinivasan Manoharan Jian Wan Fangbo Tu Junhua Zhao Xin Chen Affiliation: PayPal Affiliation: haifwu, srinivmanoharan, jwan, fatu, zhahua, xchen7@paypal.com October 1, 2026 Abstract Large zero-shot classifiers are useful for routing user requests to specialized LLM tasks, but scoring every request against a large candidate set is expensive: a zero-shot natural language inference (NLI) classifier must evaluate one premise-hypothesis pair per candidate label, so its cost scales linearly with the size of the taxonomy. We study a student-guided teacher distillation pipeline for a fixed taxonomy of 60 LLM task categories, in which a compact ModernBERT classifier predicts the full category distribution in a single forward pass and retrieves a small top-k candidate set, and a larger DeBERTa-v3 zero-shot NLI classifier then reranks only those candidates rather than all 60 labels; the resulting teacher labels iteratively improve the student, which in turn produces sharper candidate sets for the next labeling round. Unlike generic embedding retrieval or the clustering-derived shortlists used in extreme multi-label classification, the candidate generator here is trained end-to-end on the target taxonomy and is the same model that serves production traffic, distinguishing it from LLM-routing work that routes between candidate models rather than into a fixed semantic task taxonomy, and from concurrent “System 1” encoder-classifier proposals such as TypeSafe AI’s Jev [1] and the open-source Laya [2] project, whose training methodology is undocumented or RL-based. Our best student checkpoint reaches 77.5% teacher agreement on a 200-example evaluation set (the deployed checkpoint, v4, reaches 76.5%), and preliminary coverage measurements on two convenience samples show Coverage@16 of 91–100%, suggesting top-k candidate sets retain most of the teacher’s decision-relevant information. We further show that truncated top-k teacher scores should not be treated as full 60-class soft targets for KL distillation, since zeroing the untruncated classes destroys the “dark knowledge” that soft-label distillation depends on and introduces a systematic bias rather than a harmless sparse approximation. A complete evaluation of the student-guided iterative method — including candidate coverage at multiple k on a certified held-out set, an embedding-retrieval baseline, and a larger human-reviewed test set — remains in progress and is scoped explicitly in Section 7. Keywords: Jev, TypeSafe AI, System 1 models, knowledge distillation, zero-shot NLI classification, LLM task routing, extreme multi-label classification, ModernBERT, DeBERTa-v3, iterative self-training 1 Introduction LLM applications increasingly rely on routers that classify a user request before selecting a downstream agent, tool, or model. A router must be fast because it runs on every request, but it must also distinguish between fine-grained and semantically similar tasks such as debugging, code review, deployment, documentation, and pull-request management. Zero-shot natural language inference models provide a convenient way to build such routers without collecting a large manually labeled dataset. However, the teacher must evaluate the input against every candidate label as a separate premise-hypothesis pair. With 60 labels and hundreds of thousands of requests, this repeated pairwise computation becomes the dominant cost of dataset creation. Knowledge distillation can produce a fast student that predicts all 60 classes in one forward pass. The central observation of this work is that the student can also be used before teacher inference: its probability vector provides a task-aware candidate ranking. The teacher then only needs to rerank a small top-k subset. studentttop-kkcandidate labelsteacherrerankinghard/softsupervisionstudentt+1retrain Figure 1: The iterative student-guided labeling loop. The student’s top-k candidates bound what the teacher must score; the resulting hard/soft labels retrain the student for the next round, which can then attempt a smaller k once coverage at the current k is confirmed. We investigate the iterative process illustrated in Figure 1: the student proposes a top-k candidate set, the teacher reranks only those candidates, and the resulting hard/soft labels retrain the student for the next round. The goal is not to replace the teacher’s decision rule with semantic similarity. The goal is to reduce the teacher’s labeling cost while preserving the teacher’s final classification behavior. Why this is not just faster NLI. A concurrent and unrelated line of industry work makes a related efficiency argument at a different layer of the stack. TypeSafe AI’s “System One models,” introduced with the Jev model [1], propose training a single model to emit calibrated, type-safe structured decisions directly, positioned as a fast “System 1” layer beneath slower, deliberative “System 2” LLM reasoning. As of this writing, no technical report, architecture description, or training methodology for Jev or its “RLCD” (Reinforcement Learning for Calibrated Decisions) training procedure has been published beyond marketing benchmarks on TypeSafe’s own site; independent community discussion has not established what, mechanistically, distinguishes it from constrained decoding over an existing model [1]. We note this only as motivating context for why fast, calibrated, structured decision-making is an active industry concern, and flag for the record that TypeSafe’s “RLCD” acronym is an unrelated coincidental reuse of the name from Yang et al. [3]’s earlier and unrelated Reinforcement Learning from Contrastive Distillation alignment method. The System 1/System 2 framing itself is also not new to either work: it has previously been used to describe distilling slow chain-of-thought reasoning into fast direct generation [4]. A further, independently confusing detail is that an open-source project, Laya [2], has since positioned itself as a direct Jev alternative — also built on a ModernBERT-family encoder (ModernBERT-large and mmBERT-base) for single-forward-pass typed decisions, and also trained with what it calls “RLCD,” though there it denotes reinforcement learning against strictly proper scoring rules rather than TypeSafe’s “Reinforcement Learning for Calibrated Decisions” or Yang et al. [3]’s Reinforcement Learning from Contrastive Distillation: three unrelated methods now share one acronym. We cite Laya only as further evidence that compact encoder classifiers are being actively explored as fast alternatives to Jev-style System 1 models — it is, like Jev, a README-level project with no accompanying technical report or peer-reviewed evaluation at time of writing, and its training method (policy-gradient optimization against a scoring rule, over general typed-decision tasks) and evaluation focus (open-ended typed decisions across 100+ languages) are unrelated to the teacher-distillation, fixed-60-class-taxonomy routing problem this paper studies. Our contribution differs from this entire family in scope and in what is actually demonstrated: rather than proposing a new frontier model class, we study a concrete, reproducible training pipeline — built from two existing open-weight encoders — for the specific problem of cutting the labeling cost of a zero-shot NLI teacher for a fixed taxonomy, with results reported against measurable baselines. Our contributions are: 1. A student-guided candidate selection pipeline for efficient teacher labeling in fixed-taxonomy LLM routing, in which the first-stage model is a direct, end-to-end classifier over the true taxonomy rather than a generic embedding retriever or a clustering-derived label-tree shortlist. 2. An iterative distillation procedure in which each improved student proposes candidates for the next teacher-labeling round, narrowing the candidate set as coverage permits. 3. An evaluation framework that measures candidate recall, teacher agreement, classification quality, and labeling throughput together. 4. An empirical analysis of why truncated top-k teacher distributions should be separated from true full-taxonomy KL targets, connecting this failure mode to partial-label learning [5] and to the role of non-top-class “dark knowledge” in soft-label distillation [6]. 2 Related Work This work sits at the intersection of knowledge distillation, classifier cascades, extreme multi-label classification, self-training, and LLM routing. We describe the closest lines of work in each area and state precisely where our setting differs, rather than claiming that model cascades or candidate selection are novel in general. Classifier cascades and retrieve-then-rerank pipelines. Cascade classifiers that filter easy cases with a cheap model before invoking an expensive one have a long history, dating at least to the Viola-Jones face detector [7], whose cascade stages are all genuine classifiers over the same binary taxonomy (face vs. non-face). In modern retrieval, dense bi-encoder retrieval followed by cross-encoder reranking [8] follows the same coarse-to-fine shape, but the first-stage model is a generic similarity/embedding function rather than a classifier trained on the same fixed label set the second stage decides over. Confident Adaptive Language Modeling [9] and speculative decoding [10, 11] apply a related cheap-then-expensive idea within a single model’s computation (early-exit by layer, or draft-then-verify token proposals); speculative decoding is the closest conceptual analogue in that both stages score the same output space and the expensive stage only verifies the cheap stage’s proposals, but the mechanism (parallel token verification vs. sequential NLI scoring of class labels) differs substantially. Extreme multi-label classification (XMC). XMC methods such as AttentionXML [12], Parabel [13], DiSMEC [14], Slice [15], X-Transformer [16], and XR-Transformer [17] are the closest prior architectural pattern to our pipeline: a fast matching or shortlisting stage narrows a very large label space before an expensive ranker scores the shortlist. The distinguishing detail is how the shortlist is produced. AttentionXML and Parabel build the shortlist structure from a label tree derived by clustering label (or label-feature) embeddings, and X-Transformer’s matcher typically operates over approximate-nearest-neighbor label embeddings (e.g., PIFA-style representations) rather than a direct full-taxonomy softmax. XR-Transformer instead recursively fine-tunes a single model on progressively finer label partitions rather than using two separately trained models of different capacity. In our setting the taxonomy is small (60 classes, not the 10410^4–10610^6 label spaces XMC targets), which makes a single-pass full-distribution softmax classifier tractable as the first stage; the resulting candidate set is therefore produced by genuine end-to-end classification over the true output space, not by embedding-based approximate retrieval or by a coarsened intermediate taxonomy. Distilling zero-shot NLI classifiers. Zero-shot text classification via NLI entailment [18] is the paradigm our teacher uses, and Laurer et al. [19]’s work on building more efficient universal NLI classifiers (the lineage behind the deberta-v3-large-zeroshot-v2.0 teacher we use) improves the efficiency and generality of the NLI classifier itself. That line of work, however, remains within the NLI/entailment formulation; it does not collapse zero-shot NLI scoring into a single-pass fixed-K-class softmax student of the kind we train. To our knowledge, no published work directly evaluates whether truncating a zero-shot NLI teacher’s top-k scores and zeroing the remainder produces a valid target for full-distribution KL distillation. We argue it does not: the value of soft-label distillation comes substantially from the relative probabilities assigned to non-top classes, i.e., Hinton et al. [6]’s “dark knowledge,” and forcing those probabilities to zero for classes the teacher never scored is not a sparse approximation of the true distribution but a systematic bias toward overconfidence, closely related to the missing-label assumption problem studied in partial-label learning [5]. Section 4 operationalizes this distinction by restricting KL supervision to fully-scored examples and using truncated top-k scores only to derive hard pseudo-labels. Self-training and student-guided data selection. Self-training methods such as Noisy Student [20], Born-Again Networks [21], and weak-to-strong generalization [22] iterate teacher→ training rounds, but in each of these the teacher (or oracle) always scores the full label space for every example; the student’s predictions do not narrow what the teacher is asked to evaluate. Classical active learning likewise selects which examples receive an expensive oracle label, not which candidate labels, for a given example, the oracle should consider. Our iterative loop differs from both families along the same axis: the student’s own top-k predictions prune the label space the teacher scores for that example, and this pruning tightens (smaller k) across iterations as measured candidate coverage permits. We are not aware of prior work that uses a student’s evolving predictions to shrink an oracle’s per-example label-candidate set within a fixed taxonomy in this manner. LLM routing. A separate body of work studies routing a query to one of several LLMs to trade off cost and quality, e.g., RouteLLM [23], FrugalGPT [24], Tryage [25], RouterBench [26], AutoMix [27], MetaLLM [28], and Hybrid LLM routing [29]. These systems route between an unordered set of model choices, not into a fixed, semantically meaningful task taxonomy used to dispatch to downstream agents or tools; the routing decision itself carries no domain content. Selective classification and calibrated deferral [30] are methodologically relevant to any confidence-thresholded routing decision, including ours, but again are agnostic to the taxonomy-classification setting we study. To our knowledge, classifying an LLM request into a fixed task taxonomy for downstream agent routing, using an iterative teacher-guided distillation loop, has not been directly studied in prior published work. 3 Problem Formulation Let: • x be a user prompt; • =y1,…,y60Y=\y_1,…,y_60\ be the fixed task taxonomy; • T(x,)T(x,Y) be the expensive zero-shot teacher distribution; • St(x)S_t(x) be the student distribution at iteration t; • Ck(x)C_k(x) be the k labels selected from the student distribution. Full teacher labeling computes pT(y∣x)p_T(y x) for every y∈y . Student-guided labeling instead computes Ck(x)=TopK(St(x),k),pT(y∣x) for y∈Ck(x).C_k(x)=TopK(S_t(x),k), p_T(y x) for y∈ C_k(x). The teacher top-1 label can only be recovered if it belongs to Ck(x)C_k(x). We therefore define candidate coverage as Coverage@k=P(argmaxypT(y∣x)∈Ck(x)).Coverage@k=P ( _yp_T(y x)∈ C_k(x) ). The primary behavioral metric is teacher agreement: Agreement=P(argmaxyS(y∣x)=argmaxypT(y∣x)).Agreement=P ( _yS(y x)= _yp_T(y x) ). Human-reviewed accuracy remains necessary because teacher agreement measures imitation, not correctness. 4 Method 4.1 Teacher The teacher is MoritzLaurer/deberta-v3-large-zeroshot-v2.0 [31, 19], used through the Hugging Face zero-shot-classification pipeline with multi_label=False and the hypothesis template “The user’s task is .” The teacher produces a top-1 label, confidence, and scores for all 60 labels when full distillation is required. 4.2 Student The student is MoritzLaurer/ModernBERT-base-zeroshot-v2.0 [32] with a newly initialized 60-class classification head. For an input prompt, the student performs one encoder forward pass and emits 60 logits; softmax converts these logits into a 60-dimensional probability vector. Unlike the teacher, the student does not evaluate 60 separate prompt-label pairs; it directly predicts the complete taxonomy distribution in one forward pass. 4.3 Distillation objective The student is trained with a combination of hard and soft teacher supervision: L=αCE(zS,ygold)+γCE(zS,yteacher)+βT2KL(softmax(zS/T),pteacher(T))L=α\,CE(z_S,y_gold)+γ\,CE(z_S,y_teacher)+β\,T^2\,KL (softmax(z_S/T),\,p_teacher^(T) ) Here zSz_S is the student logit vector, yteachery_teacher is the teacher top-1 label, and pteacherp_teacher is the normalized teacher distribution over all 60 classes. For student-guided labeling, top-k reranking is used to generate reliable hard labels at scale. As argued in Section 2, a top-k score vector with the remaining classes set to zero is not a valid full 60-class teacher distribution and is not used as a KL target; it is used only to derive the hard pseudo-label yteachery_teacher for the α/γ terms above. 4.4 Iterative procedure At iteration t: 1. Run the current student on unlabeled synthetic taxonomy examples. 2. Select top-k candidate categories from the 60-way student output. 3. Run the DeBERTa teacher only on the candidate labels. 4. Keep labels above a confidence threshold for automatic training. 5. Mix the new data with seed and previously reviewed data. 6. Fine-tune the current best student checkpoint. 7. Evaluate on a fixed, independent test set. The next iteration may reduce k only after measuring candidate coverage: for example, top-8 can be followed by top-4 only if top-4 retains sufficient coverage of the full teacher’s top-1 label. 4.5 Deployment architecture The trained student is served in production behind Nginx, which load-balances across four independent vLLM pooling-runner instances, one per GPU, as shown in Figure 2. This is the configuration used for the live-endpoint measurements in Section 6.3: a client issues POST /router/classify to Nginx on port 8080, which distributes requests with a least_conn policy across four vllm serve processes (ports 8004–8007), each holding an independent copy of the same student checkpoint on its own GPU with --gpu-memory-utilization 0.20 and --max-model-len 512. Because each instance is a full replica rather than a shard, throughput scales close to additively with instance count (Section 6.3), at the cost of 4×4× the GPU memory a single instance would use. ClientNginx :8080least_connvLLM poolingGPU 0 :: 8004student-v4vLLM poolingGPU 1 :: 8005student-v4vLLM poolingGPU 2 :: 8006student-v4vLLM poolingGPU 3 :: 8007student-v4POST /router/classify Figure 2: Production serving architecture for the deployed v4 student. Nginx load-balances (least_conn) across four independent vLLM pooling-runner instances, one per GPU (4× NVIDIA RTX PRO 6000 Blackwell), each running a full replica of the same student checkpoint. This is the architecture measured in the server-local throughput benchmark of Section 6.3 (∼ 5,200–5,250 req/s). 5 Data The taxonomy contains 60 task categories derived from an internal task_request_mapping.json schema. Experiments use seed examples from the fixed taxonomy, HuggingFaceH4/no_robots requests, PrimeIntellect/SWE-rebench-V2 software-engineering requests, and an internal synthetic taxonomy dataset of LLM-generated, human-reviewed golden examples covering the 60 categories, used in sizes ranging from 1,000 examples in early experiments up to 2,218 examples in later full-60 teacher-scored training runs (Table 1). All training and evaluation data is drawn from public datasets or synthetically generated, human-reviewed examples; no real user traffic or production request records were accessed or used at any stage. The synthetic taxonomy dataset was generated by prompting an LLM to produce example user requests for each of the 60 categories, followed by manual human review to confirm category fidelity and remove low-quality or duplicate examples. Prompts longer than 8,000 characters are excluded before labeling. 6 Experimental Results So Far 6.1 Student versions Version Training data / objective Teacher agreement Baseline 300 seed examples + 221 high-confidence pseudo-labels Early baseline v2 Public SWE teacher soft labels 62.82% v3 Public SWE + 1,000 synthetic taxonomy examples 69.5% (139/200) v4 Teacher-focused distillation, initialized from ModernBERT-base 76.5% (153/200) v5 Public SWE + top-8 synthetic-taxonomy labels 75.0% (150/200) v5-full60 2,218 full-60 teacher-scored synthetic samples, init. from v4 71.5% (143/200) v5-tuned Same 2,218 full-60 samples, init. from v4, 1 epoch, lr 5e-6 77.5% (155/200) Table 1: Student checkpoints evaluated so far. v5-tuned is the best checkpoint measured to date but has not been promoted to production; v4 remains the currently deployed checkpoint and is the one used for the live-endpoint measurements in Section 6.3. These results are preliminary and use a 200-example evaluation set; they do not establish production-level accuracy and are reported here as intermediate progress, not as a final benchmark. 6.2 Observations The top-k labeling pipeline substantially increases teacher-labeling throughput compared with evaluating all 60 labels for every request. However, adding more synthetic data did not automatically improve student agreement: v5-full60 underperforms v4 despite using an order of magnitude more teacher-scored examples, and only a further short, low-learning-rate fine-tune from v4 (v5-tuned) narrowly surpassed v4 to become the best checkpoint measured so far. The main observed risks are low-confidence pseudo-labels, duplicated examples, distribution shift between public SWE data and the synthetic taxonomy data, and invalid truncated soft targets used as KL supervision in earlier training runs. 6.3 Preliminary coverage, agreement, and latency measurements We report a first pass at candidate coverage@k, top-k teacher-rerank agreement, and end-to-end latency for the deployed v4 student, measured against its live inference endpoint. (v5-tuned, at 77.5% teacher agreement, is the best checkpoint measured to date, but it had not been promoted to the production endpoint at measurement time, so the live-endpoint numbers below characterize v4, not v5-tuned.) These numbers partially address items 2–4 of Section 7 but fall short of the full required evaluation in two ways that we state explicitly: neither sample below is the certified, independently held-out 200-example set behind the agreement figures in Table 1 (the training host holding that set was unreachable at measurement time), and the per-request latency figures for the two agreement measurements below are measured over a corporate WAN link to the deployment host rather than from a co-located, in-cluster client (we do separately report an in-cluster server-local throughput number below). Seed sanity check (in-sample, n=300n=300). We used data/teacher_seed.jsonl, the 300-example hand-authored seed set with pre-computed full-60 teacher scores. Because this set was very likely part of v4’s own training data, results are an in-sample plausibility check, not an independent measurement: student–teacher top-1 agreement was only 39.0%, and student top-1 vs. the human gold label was 19.0%, both far below v4’s 76.5% headline figure. We attribute this gap primarily to distribution shift rather than a labeling or evaluation bug (we verified the label vocabularies match exactly): per Table 1, v4 was trained on public SWE data plus synthetic taxonomy samples, and the hand-authored seed set’s phrasing may differ enough from that training distribution to depress agreement substantially, which is itself a relevant finding about the pipeline’s sensitivity to training-serving skew. Despite low top-1 agreement, candidate coverage was comparatively strong: Coverage@4 = 74.3%, Coverage@8 = 84.3%, Coverage@16 = 91.3%, with top-k-rerank agreement equal to coverage at each k (consistent with the softmax-argmax invariance used to derive it; see Appendix discussion in the released scripts). WAN latency for a single classify call was low: mean 158 ms, p50 156 ms, p95 167 ms, p99 172 ms. Fresh SWE-rebench holdout (best-effort, n=40n=40). To reduce (but not eliminate) in-sample bias, we drew 40 fresh examples from PrimeIntellect/SWE-rebench-V2 starting at row offset 4,000 — past the row range most likely used in prior training pulls, though the exact training row range is not recorded in this checkout, so this is a best-effort held-out set rather than a certified-disjoint one. We scored each example with a freshly loaded copy of the real DeBERTa-v3-large teacher (run locally on CPU) over all 60 labels, and queried the live v4 endpoint for the student distribution. No human gold label exists for these examples, so we report only teacher-imitation metrics. Student–teacher top-1 agreement was 62.5%, closer to v4’s own 76.5% headline figure than the seed check above. Coverage@k was Coverage@4 = 80.0%, Coverage@8 = 92.5%, Coverage@16 = 100%, again with top-k-rerank agreement matching coverage exactly at each k. WAN latency for the student endpoint was similar to the seed check: mean 160 ms, p50 159 ms, p95 169 ms. Latency and throughput sweep (WAN). We additionally load-tested the live student endpoint with concurrent aiohttp keep-alive clients across a 3×33× 3 grid of request batch size ∈1,8,32∈\1,8,32\ and client concurrency ∈1,8,32∈\1,8,32\, 15 seconds per configuration. At batch size 1 and concurrency 1, p50 latency was 76.4 ms at 12.9 requests/s. At batch size 32 and concurrency 32, throughput reached 242.6 requests/s (7,763.2 classifications/s) at p50 111.5 ms, with 7 failed requests out of 3,646 at that highest-load configuration. This is a WAN benchmark from a single external client, not an in-cluster, pod-to-pod load test, and should not be read as a server capacity ceiling. Server-local throughput (in-cluster). To obtain a server-capacity number uncontaminated by WAN round-trip time and client-side connection overhead, we separately benchmarked the deployed v4 router with wrk running directly on the deployment host (g4-4), issuing single-example, non-batched requests over persistent HTTP connections to all four load-balanced vLLM pooling instances (one per GPU, four NVIDIA RTX PRO 6000 Blackwell cards) behind the same Nginx least_conn endpoint used above. At 4 threads and 64 connections, server-local wrk throughput was stable at approximately 5,200–5,250 requests/s with 12.15–12.29 ms average latency; each individual vLLM instance alone sustained roughly 1,290–1,427 requests/s at 64 concurrent connections (p50 34–38 ms, p95 44–51 ms), confirming the four instances scale close to additively under Nginx load balancing. For comparison, ApacheBench against the same server-local endpoint measured only 2,174–2,185 requests/s, and a remote ApacheBench client (256 connections, non-persistent) measured just 329.53 requests/s at 776.87 ms average latency — both artifacts of each tool’s own connection model and (for the remote case) network path, not evidence of the server’s real capacity. This single-example, in-cluster figure is not directly comparable to the WAN batch-of-32 throughput above, since the two measure different request shapes (unbatched vs. client-side batched inputs) and different vantage points (local vs. WAN); together, they bound the deployed router’s practical throughput between roughly 5,200 single-example requests/s at the server and 150–170 ms end-to-end latency for a single request from an external client. Reading these results together. Coverage@16 reaches 91–100% in both samples, and coverage degrades gracefully rather than collapsing at smaller k, which is consistent with (but does not yet prove) the paper’s central claim that student-guided top-k selection preserves most of the teacher’s decision-relevant information at a fraction of the full 60-way labeling cost. The large gap between the in-sample seed check (39% agreement) and the fresh holdout (62.5% agreement) shows that agreement is highly sensitive to how close the evaluation distribution is to the student’s training distribution, reinforcing why Section 7 calls for a larger, explicitly held-out, human-reviewed evaluation set rather than relying on either of these two convenience samples as a headline number. The server-local throughput measurement of 5,200 requests/s per four-GPU deployment is the first concrete evidence toward the paper’s motivating cost argument, though it still needs to be paired with an equivalent full-60-teacher, in-cluster GPU throughput number (not yet measured; our only full-60 teacher timings are unrepresentative local-CPU runs, see Section 7) before a labeling-cost-reduction ratio can be reported. 7 Required Evaluation Substantiating the pipeline’s central claim requires comparing student-guided candidate selection at several values of k against a full-teacher upper bound, an embedding-retrieval baseline, and a student-only lower bound, on a properly held-out, human-reviewed sample, reporting candidate recall, agreement, accuracy, latency, and cost for each. The complete comparison matrix and metric list are tracked in the released repository’s experiment-punch-list.md rather than reproduced here, since that plan is still being executed. 8 Reproducibility and Release Plan The repository contains scripts for seed preparation, teacher labeling, synthetic data generation, prompt filtering, student-guided retrieval, student training, and evaluation. Model artifacts remain outside the public repository. A reproducible release will include the fixed label schema and category descriptions; synthetic or de-identified example prompts; model and tokenizer identifiers and revisions; all loss and inference hyperparameters; train/validation/test split hashes; candidate coverage and agreement reports; and throughput and hardware measurements. 9 Conclusion This paper presents a student-guided teacher distillation pipeline for efficient 60-class LLM task routing. The student is used not only as the final fast router, but also as a task-aware candidate generator that reduces expensive zero-shot teacher labeling, distinguishing it from generic embedding-based candidate generation used in extreme multi-label classification cascades and from LLM-choice routing methods that do not classify into a semantic task taxonomy at all. Preliminary experiments demonstrate a best teacher agreement of 77.5% (v5-tuned; the currently deployed v4 checkpoint reaches 76.5%), while also identifying important failure modes around low-confidence data and truncated soft targets, which we connect to the theoretical grounds for why partial teacher observations should not be treated as full-distribution KL targets. A first, best-effort measurement of candidate coverage (Section 6.3), run against the deployed v4 checkpoint, shows Coverage@16 of 91–100% across two convenience samples, and highlights that agreement is sensitive to distribution shift between evaluation and training data — a factor the certified held-out evaluation in Section 7 still needs to control for. The central empirical question for the completed paper is whether iterative student-guided candidate selection achieves a better accuracy–labeling-cost trade-off than full teacher labeling, embedding retrieval, and non-iterative distillation. Section 7 scopes the specific measurements — candidate coverage across k, an embedding-retrieval baseline, and a larger human-reviewed evaluation set — required to answer it, and we intend to report them in a subsequent revision of this paper. References [1] TypeSafe AI (2026) Introducing system one models and jev. Note: https://typesafe.ai/blog/introducing-system-one-models-and-jevBlog post; no accompanying technical report available at time of writing Cited by: §1, Abstract. [2] Convai Innovations and NandhaKishorM (2026) Laya: multilingual, non-autoregressive system 1 decision engine. Note: https://github.com/NandhaKishorM/layaGitHub repository, Apache 2.0 license; no accompanying technical report or peer-reviewed publication at time of writing Cited by: §1, Abstract. [3] K. Yang, D. Klein, A. Celikyilmaz, N. Peng, and Y. Tian (2023) RLCD: reinforcement learning from contrastive distillation for language model alignment. arXiv preprint arXiv:2307.12950. Note: Unrelated to TypeSafe AI’s use of the acronym RLCD; name collision only Cited by: §1. [4] P. Yu, J. Xu, J. Weston, and I. Kulikov (2024) Distilling system 2 into system 1. arXiv preprint arXiv:2407.06023. Cited by: §1. [5] T. Cour, B. Sapp, and B. Taskar (2011) Learning from partial labels. Journal of Machine Learning Research 12, p. 1501–1536. Cited by: item 4, §2. [6] G. Hinton, O. Vinyals, and J. Dean (2015) Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Cited by: item 4, §2. [7] P. Viola and M. Jones (2001) Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2. [8] V. Karpukhin, B. Oğuz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W. Yih (2020) Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906. Cited by: §2. [9] T. Schuster, A. Fisch, J. Gupta, M. Dehghani, D. Bahri, V. Q. Tran, Y. Tay, and D. Metzler (2022) Confident adaptive language modeling. arXiv preprint arXiv:2207.07061. Cited by: §2. [10] Y. Leviathan, M. Kalman, and Y. Matias (2023) Fast inference from transformers via speculative decoding. arXiv preprint arXiv:2211.17192. Cited by: §2. [11] C. Chen, S. Borgeaud, G. Irving, J. Lespiau, L. Sifre, and J. Jumper (2023) Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. Cited by: §2. [12] R. You, Z. Zhang, Z. Wang, S. Dai, H. Mamitsuka, and S. Zhu (2019) AttentionXML: label tree-based attention-aware deep model for high-performance extreme multi-label text classification. arXiv preprint arXiv:1811.01727. Cited by: §2. [13] Y. Prabhu, A. Kag, S. Harsola, R. Agrawal, and M. Varma (2018) Parabel: partitioned label trees for extreme classification with application to dynamic search advertising. In Proceedings of the 2018 World Wide Web Conference (W), Cited by: §2. [14] R. Babbar and B. Schölkopf (2017) DiSMEC: distributed sparse machines for extreme multi-label classification. arXiv preprint arXiv:1609.02521. Cited by: §2. [15] H. Jain, V. Balasubramanian, B. Chunduri, and M. Varma (2019) Slice: scalable linear extreme classifiers trained on 100 million labels for related searches. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining (WSDM), Cited by: §2. [16] W. Chang, H. Yu, K. Zhong, Y. Yang, and I. S. Dhillon (2020) Taming pretrained transformers for extreme multi-label text classification. arXiv preprint arXiv:1905.02331. Cited by: §2. [17] J. Zhang, W. Chang, H. Yu, and I. S. Dhillon (2021) Fast multi-resolution transformer fine-tuning for extreme multi-label text classification. arXiv preprint arXiv:2110.00685. Cited by: §2. [18] W. Yin, J. Hay, and D. Roth (2019) Benchmarking zero-shot text classification: datasets, evaluations and entailment approach. arXiv preprint arXiv:1909.00161. Cited by: §2. [19] M. Laurer, W. van Atteveldt, A. Casas, and K. Welbers (2023) Building efficient universal classifiers with natural language inference. arXiv preprint arXiv:2312.17543. Cited by: §2, §4.1. [20] Q. Xie, M. Luong, E. Hovy, and Q. V. Le (2020) Self-training with noisy student improves imagenet classification. arXiv preprint arXiv:1911.04252. Cited by: §2. [21] T. Furlanello, Z. C. Lipton, M. Tschannen, L. Itti, and A. Anandkumar (2018) Born again neural networks. arXiv preprint arXiv:1805.04770. Cited by: §2. [22] C. Burns, P. Izmailov, J. H. Kirchner, B. Baker, L. Gao, L. Aschenbrenner, Y. Chen, A. Ecoffet, M. Joglekar, J. Leike, I. Sutskever, and J. Wu (2023) Weak-to-strong generalization: eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390. Cited by: §2. [23] I. Ong, A. Almahairi, V. Wu, W. Chiang, T. Wu, J. E. Gonzalez, M. W. Kadous, and I. Stoica (2024) RouteLLM: learning to route llms with preference data. arXiv preprint arXiv:2406.18665. Cited by: §2. [24] L. Chen, M. Zaharia, and J. Zou (2023) FrugalGPT: how to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176. Cited by: §2. [25] S. Kwon, C. Almalis, E. Daniel, E. Kang, S. Song, J. Gee, and J. M. Vidal (2023) Tryage: real-time, intelligent routing of user prompts to large language models. arXiv preprint arXiv:2308.11601. Cited by: §2. [26] Q. J. Hu, J. Bieker, X. Li, N. Jiang, B. Keigwin, G. Ranganath, K. Keutzer, and S. K. Upadhyay (2024) RouterBench: a benchmark for multi-llm routing system. arXiv preprint arXiv:2403.12031. Cited by: §2. [27] A. Madaan, P. Aggarwal, A. Anand, S. P. Potharaju, S. Mishra, P. Zhou, A. Gupta, D. Rajagopal, K. Kappaganthu, Y. Yang, S. Upadhyay, and M. Faruqui (2023) AutoMix: automatically mixing language models. arXiv preprint arXiv:2310.12963. Cited by: §2. [28] Q. H. Nguyen, D. C. Hoang, J. Decugis, S. Manchanda, N. V. Chawla, and K. D. Doan (2024) MetaLLM: a high-performant and cost-efficient dynamic framework for wrapping llms. arXiv preprint arXiv:2407.10834. Cited by: §2. [29] D. Ding, A. Mallick, C. Wang, R. Sim, S. Mukherjee, V. Ruhle, L. V. S. Lakshmanan, and A. H. Awadallah (2024) Hybrid llm: cost-efficient and quality-aware query routing. arXiv preprint arXiv:2404.14618. Cited by: §2. [30] Y. Geifman and R. El-Yaniv (2017) Selective classification for deep neural networks. arXiv preprint arXiv:1705.08500. Cited by: §2. [31] P. He, J. Gao, and W. Chen (2021) DeBERTaV3: improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543. Cited by: §4.1. [32] B. Warner, A. Chaffin, B. Clavié, O. Weller, O. Hallström, S. Taghadouini, A. Gallagher, R. Biswas, F. Ladhak, T. Aarsen, N. Cooper, G. Adams, J. Howard, and I. Poli (2024) Smarter, better, faster, longer: a modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. arXiv preprint arXiv:2412.13663. Cited by: §4.2.