Paper deep dive
Radioactive Feedback: Tracing Models Trained by Label-Only LLM Supervision
Yuejia Li, JunHeng Li, Hao Wu, Ke He, Mang Ye
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Conventional distillation transfers capability through teacher-generated demonstrations. LLM-as-a-Judge knowledge distillation offers a distinct and increasingly practical alternative. In the label-only variant studied here, the student generates every response and improves from scalar scores alone. Prior work shows that this non-demonstrative feedback can improve student capability, but it creates a fundamentally different provenance problem. No teacher-generated answer enters the student's training set, so any evaluator-specific trace must pass through a low-bandwidth reward signal rather than copied text. Existing watermarks and distillation detectors do not cover this score-only channel. We introduce Radioactive Feedback, which encodes a registered pattern of behavioral preferences in the rewards themselves. The evaluator applies bounded pointwise adjustments to quality-qualified responses, allowing repeated optimization to transfer the pattern to a student trained by another party. Experiments on code and mathematics show that the registered preferences transfer through scores alone and across datasets, enabling black-box detection against a fixed reference pool.
Tags
Links
- Source: https://arxiv.org/abs/2610.15685v1
- Canonical: https://arxiv.org/abs/2610.15685v1
Trouble viewing inline? Open PDF directly →
Full Text
70,782 characters extracted from source content.
Expand or collapse full text
Radioactive Feedback: Tracing Models Trained by Label-Only LLM Supervision Correspondence to Yuejia Li (liyuejia@whu.edu.cn). Code: https://github.com/Kemalau/RadioactiveFeedBack Project: https://github.com/Kemalau/OneLinetoProtectYourReward Abstract Conventional distillation transfers capability through teacher-generated demonstrations. LLM-as-a-Judge knowledge distillation offers a distinct and increasingly practical alternative. In the label-only variant studied here, the student generates every response and improves from scalar scores alone. Prior work shows that this non-demonstrative feedback can improve student capability, but it creates a fundamentally different provenance problem. No teacher-generated answer enters the student’s training set, so any evaluator-specific trace must pass through a low-bandwidth reward signal rather than copied text. Existing watermarks and distillation detectors do not cover this score-only channel. We introduce Radioactive Feedback, which encodes a registered pattern of behavioral preferences in the rewards themselves. The evaluator applies bounded pointwise adjustments to quality-qualified responses, allowing repeated optimization to transfer the pattern to a student trained by another party. Experiments on code and mathematics show that the registered preferences transfer through scores alone and across datasets, enabling black-box detection against a fixed reference pool. Yuejia Li1, JunHeng Li1, Hao Wu2, Ke He1, Mang Ye1 1Wuhan University 2Southeast University 1 Introduction Conventional supervised distillation gives a student complete teacher responses and trains it to imitate them token by token. LLM-as-a-Judge knowledge distillation changes both the object of supervision and its information channel. The broader paradigm can use critiques, preferences, or scores; in the label-only variant studied here, the student generates candidate responses from its current policy while a provider-controlled evaluator returns only scalar judgments. Existing RLAIF and on-policy distillation research shows that this feedback can produce capability gains (Bai et al., 2022; Lee et al., 2024; Yuan et al., 2024; Fang et al., 2026). The teacher never demonstrates how to answer; optimization instead transfers aspects of the evaluator’s preferences through judgments of the student’s own outputs. We call this setting label-only evaluator distillation. This low-bandwidth channel creates a distinct provenance problem. No teacher-generated text ever enters the student’s training set, so a detectable evaluator-specific trace cannot be explained by copying watermarked tokens. It must survive through scalar feedback alone. We therefore ask whether evaluator information can enter the student’s behavior through scores and remain recognizable. Existing provenance methods rely on higher-bandwidth objects. Text watermarks and their radioactive variants transmit provenance through teacher-generated text (Kirchenbauer et al., 2023; Sander et al., 2024). Claude’s text watermark is a version of SynthID-Text (Anthropic, 2026; Dathathri et al., 2024). Antidistillation sampling likewise modifies teacher tokens (Savani et al., 2025). PreferCare marks a fixed preference dataset (Lou et al., 2025). Model fingerprints mark weights, trigger behaviors, or training text and can be used when the marked model or its training text is available (Uchida et al., 2017; Xu et al., 2024; Gloaguen et al., 2026). These mechanisms do not write an attribution signal into scalar feedback, where the evaluator can shape only the score (Figure 1). Figure 1: Text distillation and label-only feedback. To address this problem, we introduce Radioactive Feedback, a method for embedding a registered behavioral code in scalar feedback. The code selects preferred alternatives on a set of response features, or carriers. When an individual response meets a quality floor and exposes a classifiable carrier value, the evaluator applies a bounded pointwise reward adjustment in the registered direction. Repeated optimization can then transfer these preferences to the student. Carriers range from lightweight response conventions, such as formatting choices, to deeper solution strategies, such as alternative dynamic-programming or geometric approaches. This allows the registered code to span both surface form and problem-solving behavior. Training and auditing on different datasets tests whether the resulting pattern generalizes beyond the training set. At audit time, the provider measures alignment in student responses and compares it with a reference pool fixed before scoring, thereby testing whether the student distilled the enrolled teacher. This paper makes three contributions. • Provenance across non-demonstrative feedback. We formalize tracing when the student generates every training response and the evaluator returns only scalar scores. No teacher-generated text enters the training set, and the secret belongs to the feedback provider rather than the trainer of the audited model. • Radioactive Feedback. A keyed, bounded, quality-gated pointwise tilt on eligible responses distributes a registered code over multiple carriers, covering answer form and solution methods, and supports black-box detection from student responses. • Five experimental evaluations with supporting ablations. Single-carrier and multi-carrier experiments establish transfer and a joint detector signal; registered-key attribution tests code specificity; large-scale detection adds held-out cross-family controls; persistence covers post-training and removal, plus a ρ ablation (appendix). Experiments with Llama and Qwen students show strong watermark effects on code and mathematics, with single-carrier alignment gains up to 64.964.9 p. Joint four-carrier training remains positive across all nine evaluated student–evaluator combinations, ranging from +7.6+7.6 to +15.4+15.4 p (Table 3). The watermark also transfers from TACO and OpenR1-Math to multiple held-out code and mathematics benchmarks (Table 2). In the two settings used for calibrated auditing, all 8/88/8 independently trained marked checkpoints are detected at nominal α=1%α=1\% (Table 11), and all 4040 held-out cross-family controls remain below their respective detection thresholds (Table 10). 2 Related Work LLM-as-a-Judge knowledge distillation. LLM evaluators increasingly supervise students through critiques, preferences, and scalar judgments (Bai et al., 2022; Lee et al., 2024; Zheng et al., 2023; Yuan et al., 2024). Such labels can drive RL or direct preference optimization without exposing teacher logits (Rafailov et al., 2023). We study the label-only case in which each student rollout receives only a scalar score. Rubric-based On-policy Distillation (ROPD) similarly scores on-policy rollouts with prompt-specific rubrics, although teacher-generated references help construct those rubrics (Fang et al., 2026). Shen, Tu, and Wang likewise reward the student’s own rollouts from an LLM judge, computing the continuous reward from the judge’s Yes/No token logits (Shen et al., 2026). These works ask how evaluator feedback improves a student; we ask whether that influence can be identified when the student generated every response. Watermarking supervision channels. Text watermarks modify generated-token statistics and radioactive watermarks detect their residue after training on marked synthetic text (Kirchenbauer et al., 2023; Kuditipudi et al., 2024; Zhao et al., 2024; Christ et al., 2024; Sander et al., 2024). Pan et al. study inheritance and removal of radioactive watermarks under distillation, still carried by teacher-generated text (Pan et al., 2025). Antidistillation sampling acts on the teacher’s token distribution (Savani et al., 2025). CATER watermarks text-generation API outputs with conditional word substitutions for post-hoc imitation detection (He et al., 2022). Interaction-layer behavioral markers (Yang et al., 2026) and radioactive token watermarks (Sander et al., 2026) trace distillation from full teacher responses. These works mark generated text or API outputs; our setting differs in two ways: the evaluator returns only a scalar score, and the student (not the provider) generates every training rollout. PreferCare marks a fixed preference dataset and tests whether that dataset was used (Lou et al., 2025). Radioactive Feedback instead marks online judgments of a student’s evolving, self-generated rollouts. Model fingerprints and ownership. Ownership methods embed signatures in weights or trigger responses (Uchida et al., 2017; Adi et al., 2018), while instructional and semantic fingerprints broaden the audited behavior (Xu et al., 2024; Gloaguen et al., 2026; Xu et al., 2025). These mechanisms can be used when the marked model or its training text is available. Even then, they struggle to leave a trace through scalar feedback, because a weight, trigger, or training-text signature is not written onto the returned score. Radioactive Feedback marks that score. The audit then distinguishes transfer of the registered code from a student’s natural preferences. 3 Problem Formulation Learning setting. An evaluator J scores responses from a student policy πθ(y∣x) _θ(y x). In the provenance setting of interest, J is controlled by an external feedback provider; the learner receives only its returned scalar scores. We also include open-weight evaluators in RQ2 as transfer controls to test whether watermark transfer depends on proprietary model access. At step t the learner draws a prompt xtx_t, generates candidates Yt=(yt,1,…,yt,G)Y_t=(y_t,1,…,y_t,G), receives feedback ft=J(xt,Yt)f_t=J(x_t,Y_t), and updates θt _t: yt,g y_t,g ∼πθt(⋅∣xt),g=1,…,G, _ _t(· x_t), g=1,…,G, (1) ft f_t =J(xt,Yt),θt+1=Update(θt;xt,Yt,ft). =J(x_t,Y_t), _t+1=Update( _t;x_t,Y_t,f_t). For scalar feedback, ft∈ℝGf_t ^G gives one score per candidate. We call this label-only evaluator distillation: the student generates every rollout and the evaluator returns scores only. Radioactive Feedback instantiates this setting with scalar rewards. Access model. The provider controls J and sees prompts and responses, but not the student’s weights, optimizer, or private data. The learner controls prompts, generations, and updates, and may retrain or change decoding. At audit time the provider queries the student and reads its outputs. Detection. Fix an enrolled pair Eu=(Ju,u)E_u=(J_u,c_u) with registered code uc_u shared in training and audit. The null hypothesis is ℋ0,u:π was trained without marked feedback from Eu.H_0,u: π was trained without marked feedback from E_u. (2) The auditor tests whether behavior aligns with uc_u relative to a reference pool (Section 6.4). The audit returns detected when the score passes the registered test, and not detected otherwise. 4 Method Figure 2: Overview of the Radioactive Feedback pipeline. The secret-key and PRF path in the figure is an optional derivation of the code. The reported experiments register an explicit sign vector (Appendix B). We describe a scalar-feedback implementation. The learner submits each candidate group to the evaluator API and receives final rewards. The registered preference is part of the evaluator instruction. Within that single call, the evaluator is instructed to first determine its ordinary rubric score internally, then add ρRρ R when the registered side appears and subtract ρRρ R when the opposite side appears, clip the result to [0,R][0,R], and return the final reward. Neither the base score nor the carrier label is exposed to the learner. Training and audit share one carrier representation and code. 4.1 Behavioral Carriers and Codes A behavioral carrier is a response feature whose alternatives can receive different preferences from the evaluator. We represent the carriers by ϕ(x,y)∈−1,0,+1dφ(x,y)∈\-1,0,+1\^d: a defined coordinate records one of two alternatives and an undefined coordinate contributes zero. Applicability is computed from the response independently of the key. A carrier may be a convention (quote style or a closing marker) or a strategy such as top-down versus bottom-up programming, graph traversal versus disjoint-set union, direct counting versus inclusion–exclusion, or synthetic versus coordinate geometry. The enrolled evaluator has a fixed code ∈−1,+1dc∈\-1,+1\^d whose coordinates independently select the preferred alternatives. Thus cjϕj(x,y)>0c_j _j(x,y)>0 denotes agreement with the registered preference on carrier j. The provider can derive this code from a secret seed or register an explicit sign vector. We call the prescribed vector c the registered code. 4.2 Keyed Feedback Construction Each prompt x is assigned a single carrier j(x)j(x) before scoring. Within one API evaluation of Y=(y1,…,yG)Y=(y_1,…,y_G), the evaluator returns one final score per response. Its instruction requires it to form an ordinary rubric score r0,ir_0,i internally and to recognize the carrier alternative ϕi=ϕj(x)(x,yi) _i= _j(x)(x,y_i). Pointwise eligibility uses those internal quantities, e~i=r0,i≥rmin,ϕi∈−1,+1. e_i=1\! \r_0,i≥ r_ ,\ _i∈\-1,+1\ \. (3) Eligibility depends on this response’s rubric score and classifiable carrier value only; it is independent of other candidates in the group and of the preferred sign. The evaluator then applies exactly the prescribed adjustment and clips the result to the service range [0,R][0,R]: ru(x,yi)=clip[0,R](r0,i+ρRe~icj(x)ϕi).r_u(x,y_i)=clip_[0,R] (r_0,i+ρ R\, e_i\,c_j(x) _i ). (4) The sign follows the side that appears. Agreement with the enrolled code gives cj(x)ϕi=+1c_j(x) _i=+1, so an eligible response on the registered side increases by ρRρ R. The opposite side gives cj(x)ϕi=−1c_j(x) _i=-1, so that response decreases by ρRρ R. An ineligible response is returned unchanged, as its internal rubric score clipped to [0,R][0,R]. The learner receives rur_u only. There is one keyed term per prompt. Training uses group-normalized GRPO (Shao et al., 2024). 4.3 Black-Box Provenance Auditing The audit queries the student with n held-out prompts spanning the registered carriers and obtains one response per prompt. Each prompt targets one carrier, so every other carrier coordinate is zero. The bit qi∈0,1q_i∈\0,1\ marks task correctness. Define Tu(π)=1n∑i=1nqicj(xi)ϕj(xi)(xi,yi).T_u(π)= 1n _i=1^nq_i\,c_j(x_i) _j(x_i)(x_i,y_i). (5) Correct, classifiable responses contribute ±1± 1; all others contribute zero. The denominator includes all n audit prompts. The correctness bit qiq_i is the official programmatic check used for task utility: code benchmarks use their execution harnesses, and mathematics benchmarks use final-answer checking. Claude Opus 5 supplies only the carrier or strategy value inside ϕφ. It is not the training evaluator and does not score task utility. Reference population and pup_u. Fix a reference pool nullN_null of M models trained without the enrolled feedback. Score each reference and the suspect with the same registered code and audit protocol. For suspect score T^u T_u, define the one-sided model-level rank statistic pu=1+∑m∈null[Tu(m)≥T^u]1+M.p_u= 1+ _m _null1[T_u^(m)≥ T_u]1+M. (6) Under exchangeability with the reference models, this rank gives a valid finite-sample test. Let T(1)≤⋯≤T(M)T_(1)≤·s≤ T_(M) be the ordered reference scores. Before scoring suspects, fix a significance level α and use the non-interpolated upper empirical quantile γ=T(k)γ=T_(k), where k=minM,⌈(M+1)(1−α)⌉k= \M, (M+1)(1-α) \: detected(u)⇔pu<α∧T^u>γ.detected(u) p_u<α\; \; T_u>γ. (7) 5 Analysis We show that keyed contrasts tilt response odds while bounded perturbations preserve sufficiently separated quality rankings. For a response y on prompt x with assigned carrier j(x)j(x), let hu(x,y)=e~cj(x)ϕj(x)(x,y)h_u(x,y)= e\,c_j(x) _j(x)(x,y) (with e~ e as in Equation (3)), so hu∈−1,0,+1h_u∈\-1,0,+1\. The returned score is the clipped evaluator score in Equation (4). Relative to the unmarked score, the displacement that actually enters the objective is δu(x,y)=clip[0,R](r0(x,y)+ρRhu(x,y))−r0(x,y), _u(x,y)\;=\;clip_[0,R] (r_0(x,y)+ρ R\,h_u(x,y) )-r_0(x,y), (8) so ru=r0+δur_u=r_0+ _u. If r0∈[0,R]r_0∈[0,R], then |δu|≤ρR _u ≤ρ R. When r0∈[ρR,R−ρR]r_0∈[ρ R,\,R-ρ R], the clip is inactive and δu=ρRhu _u=ρ R\,h_u. Proofs are in Appendix A. We derive the following identities under a KL-regularized proxy. Here β>0β>0 is the coefficient of the KL penalty, as in GRPO. Because GRPO standardizes rewards within each group, the proxy (12) is qualitative, and the β in Table 5 does not correspond to the raw [0,R][0,R] reward in the identities below. Why the key can accumulate in the student. Proposition 5.1 (Keyed policy tilt). Let πcl∗ _cl^* denote the optimizer under the clean reward r0r_0 in the KL-regularized proxy. For any responses y,y′y,y , logπu∗(y∣x)/πu∗(y′∣x)πcl∗(y∣x)/πcl∗(y′∣x)=1β(δu(x,y)−δu(x,y′)). _u^*(y x)/ _u^*(y x) _cl^*(y x)/ _cl^*(y x)\;=\; 1β ( _u(x,y)- _u(x,y ) ). (9) When the clip is inactive the right-hand side is (ρR/β)(hu(x,y)−hu(x,y′))(ρ R/β) (h_u(x,y)-h_u(x,y ) ). Variation in δu _u across qualified responses tilts odds in the registered direction (held-out tests: Section 6.1). Why confident rankings are preserved. Proposition 5.2 (Margin preservation). For any pair y,y′y,y , |δu(x,y)−δu(x,y′)|≤ 2ρR. _u(x,y)- _u(x,y ) \;≤\;2ρ R. (10) Clipping cannot enlarge a one-response displacement past ρRρ R. Hence |r0(x,y)−r0(x,y′)|>2ρR r_0(x,y)-r_0(x,y ) >2ρ R implies that rur_u and r0r_0 agree on the order of (y,y′)(y,y ). If r0∈0,Rr_0∈\0,R\ and 1>2ρ1>2ρ, a passing solution cannot be ranked below a failing one by the keyed term alone. The bound shows how a bounded watermark adjustment preserves large-margin quality rankings. 6 Experiments We organize the evaluation as an evidence chain for provenance through score-only supervision. RQ1 asks whether scalar feedback can write a registered preference into a student. RQ2 tests whether preferences can coexist and survive changes in data and evaluator. RQ3 asks whether the inherited signal identifies the enrolled code rather than a generic behavioral bias. RQ4 turns that signal into a calibrated black-box decision, and RQ5 asks whether the signal persists after the student is trained further. Model pairings, controls, dataset construction, and training details are in Appendix C–E. Evaluation protocol. We study Llama-3-8B-Instruct, Qwen2.5-Coder-1.5B-Instruct, and Qwen3-30B-A3B students (Grattafiori et al., 2024; Hui et al., 2024; Yang et al., 2025). Llama and Qwen2.5 use response-form carriers, whereas the Frontier setting uses solution-strategy carriers, giving two qualitatively different ways for a registered code to appear in behavior. All trained arms share the same GRPO recipe within a setting. The main keyed strength is ρ=0.05ρ=0.05 with rmin=60r_ =60. RQ1 and RQ2 each use four independent GRPO training seeds per setting. Each marked checkpoint is compared with the Clean checkpoint trained under the same seed; Base is the unmodified initial student checkpoint. Metrics. The watermark effect is the increase in registered-code alignment among correct, classifiable responses, WMeffect=100(aMarked−aClean)WM\ effect=100(a_Marked-a_Clean), in percentage points. Task utility is Pass@1 for code and accuracy for mathematics, with ΔUdistill=U(Clean)−U(Base) U_distill=U(Clean)-U(Base) and ΔUmark=U(Marked)−U(Clean) U_mark=U(Marked)-U(Clean). For provenance detection we use the checkpoint statistic TuT_u from Equation (5); unlike the transfer summaries, its denominator contains every audit prompt. Label construction and programmatic correctness checks are detailed in Appendix C. 6.1 RQ1: Can scalar feedback write a stable behavioral preference? A provenance mechanism cannot work if a low-bandwidth score fails to leave a reproducible behavioral trace. We therefore begin with one carrier at a time. Table 1 shows a positive watermark effect in all eight student–audit combinations. The magnitude varies substantially—from single-digit gains for the smaller Qwen student to much larger shifts on Llama mathematics—but the direction is consistent across training seeds. This is the first required link in the argument: the evaluator does not need to provide demonstrations for a registered preference to enter the student’s behavior. The same comparison also separates transfer from utility degradation. Clean feedback improves over Base in every setting, while adding the registered preference changes utility only modestly. Seven of the eight Marked–Clean differences are positive; the remaining case is −0.3±0.2-0.3± 0.2 p. Thus the large alignment shifts are not explained by a general collapse in task performance. Table 1: Single-carrier watermark transfer and task utility. Entries are mean ± standard deviation over four training seeds, in percentage points. Code audits are HumanEval+ and MBPP+ (Liu et al., 2023); SVAMP is included for mathematics (Patel et al., 2021). Student Domain Audit set WM effect (p) U_mark (p) U_distill (p) Llama-3-8B-it Code HumanEval+ +17.7±1.4+17.7± 1.4 +2.1±0.3+2.1± 0.3 +2.9±0.4+2.9± 0.4 MBPP+ +33.3±2.2+33.3± 2.2 +0.5±0.2+0.5± 0.2 +2.5±0.4+2.5± 0.4 Math GSM8K +43.6±2.7+43.6± 2.7 +0.2±0.2+0.2± 0.2 +2.0±0.3+2.0± 0.3 SVAMP +64.9±3.1+64.9± 3.1 +1.6±0.4+1.6± 0.4 +2.2±0.3+2.2± 0.3 Qwen2.5-Coder- 1.5B-it Code HumanEval+ +9.8±0.9+9.8± 0.9 +1.5±0.2+1.5± 0.2 +2.9±0.4+2.9± 0.4 MBPP+ +10.9±1.2+10.9± 1.2 +0.2±0.2+0.2± 0.2 +2.0±0.3+2.0± 0.3 Math GSM8K +6.0±0.7+6.0± 0.7 +0.7±0.2+0.7± 0.2 +1.3±0.2+1.3± 0.2 SVAMP +11.8±1.0+11.8± 1.0 −0.3±0.2-0.3± 0.2 +1.5±0.2+1.5± 0.2 6.2 RQ2: Does a joint code generalize beyond one dataset or evaluator? Single-carrier transfer does not establish that a multi-bit signature can survive heterogeneity. We next train all four carriers jointly and test two kinds of generalization. Across datasets. The Frontier student is trained on TACO (Li et al., 2023; Li, 2024) and OpenR1-Math (Yan et al., 2025) but audited on held-out code and mathematics benchmarks that are never used to write the watermark. The signal remains positive on all six audits (Table 2). Code effects range from 14.814.8 to 19.119.1 p and mathematics effects from 8.98.9 to 11.611.6 p. The important observation is not the ordering of individual benchmarks, but that the same registered strategy preferences reappear after the underlying problem distribution changes. This rules out an interpretation in which the signal is tied only to the training datasets. Table 2: Cross-dataset watermark transfer on Frontier (Qwen3-30B-A3B / GPT-6 Astra). For each of four training seeds, WM effect is Marked–Clean on the listed audit; entries are mean ± standard deviation over those seeds. ΔUmark U_mark and ΔUdistill U_distill use the same four-seed mean ± standard deviation, in percentage points. Domain Train set Audit set WM effect (p) U_mark (p) U_distill (p) Code TACO CodeContests +19.1±0.6+19.1± 0.6 +0.6±0.2+0.6± 0.2 +3.8±0.4+3.8± 0.4 APPS +17.6±0.5+17.6± 0.5 +0.4±0.1+0.4± 0.1 +3.5±0.3+3.5± 0.3 LiveCodeBench +14.8±0.7+14.8± 0.7 +0.4±0.2+0.4± 0.2 +3.1±0.3+3.1± 0.3 Math OpenR1-Math Omni-MATH +11.6±0.5+11.6± 0.5 +0.3±0.1+0.3± 0.1 +5.6±0.5+5.6± 0.5 MATH-500 +10.4±0.4+10.4± 0.4 +0.3±0.1+0.3± 0.1 +4.8±0.4+4.8± 0.4 OlympiadBench +8.9±0.6+8.9± 0.6 +0.2±0.1+0.2± 0.1 +4.3±0.4+4.3± 0.4 Across students and evaluators. We then vary the student–evaluator pairing while keeping the registered code fixed within each carrier regime. All nine evaluated combinations retain a positive joint effect (Table 3), including both open-weight and proprietary evaluators. This matters because the watermark is therefore not specific to one evaluator family or access regime. At the same time, the nine rows should not be read as a strict factorial ranking: the Frontier student uses strategy carriers whereas the two smaller students use response-form carriers. The consistent result is the sign of transfer, not a claim that one evaluator is intrinsically better at carrying the watermark. Utility provides a second check on this interpretation. Ordinary distillation remains beneficial in every row, while the additional Marked–Clean change stays small relative to the watermark effect. The joint signal therefore survives model and evaluator changes without trading away the capability gains that motivated evaluator distillation in the first place. Table 3: Joint watermark transfer across student–evaluator combinations. For each of four seeds, the listed difference is computed against the Clean checkpoint trained with that same seed; the table then reports the mean ± standard deviation of those four differences. ΔUdistill U_distill compares each Clean checkpoint with the corresponding unmodified initial student checkpoint (Base). Student Evaluator WM effect (p) U_mark (p) U_distill (p) Llama-3-8B-it Qwen3-30B-A3B +7.6±0.4+7.6± 0.4 +1.1±0.1+1.1± 0.1 +2.4±0.3+2.4± 0.3 GPT-5.6 Sol +8.4±0.5+8.4± 0.5 +0.8±0.1+0.8± 0.1 +2.8±0.3+2.8± 0.3 GPT-6 Astra +9.2±0.5+9.2± 0.5 +0.7±0.1+0.7± 0.1 +3.1±0.3+3.1± 0.3 Qwen2.5-Coder- 1.5B-it Qwen3-30B-A3B +9.3±0.6+9.3± 0.6 +0.5±0.1+0.5± 0.1 +1.9±0.2+1.9± 0.2 GPT-5.6 Sol +10.1±0.6+10.1± 0.6 +0.5±0.1+0.5± 0.1 +2.2±0.3+2.2± 0.3 GPT-6 Astra +11.4±0.6+11.4± 0.6 +0.4±0.1+0.4± 0.1 +2.6±0.3+2.6± 0.3 Qwen3-30B-A3B Qwen3-235B-A22B +12.4±0.7+12.4± 0.7 +0.6±0.1+0.6± 0.1 +3.6±0.4+3.6± 0.4 GPT-5.6 Sol +13.9±0.7+13.9± 0.7 +0.5±0.1+0.5± 0.1 +4.1±0.3+4.1± 0.3 GPT-6 Astra +15.4±0.5+15.4± 0.5 +0.5±0.1+0.5± 0.1 +4.7±0.4+4.7± 0.4 6.3 RQ3: Is the inherited signal specific to the enrolled code? Positive transfer alone is insufficient for provenance: an unmarked model could naturally prefer the same carrier alternatives. We therefore keep the trained checkpoints fixed and rescore the same outputs under alternative sign vectors. The enrolled code produces raw audit scores 100Tu∈[6.60,8.30]100T_u∈[6.60,8.30]. Equation (5) is linear in the code, so the average over all sign patterns at distance d is (1−d/2)(1-d/2) times that score, and distance 44 is the exact negative (Figure 3). These averages at distance 22 and beyond lie at or below zero, under γ(⋆)γ(c ). The signal follows the enrolled signs. The codebook and calibration procedure are in Appendix J. Figure 3: Average of 100Tu100T_u over all codes at Hamming distance d (Equation (5)). Light lines: one seed’s average; marked line: average of the four seeds. Distance 44 equals −100Tu(⋆)-100T_u(c ). Dashed lines are γ(⋆)γ(c ): 1.151.15 (Llama) and 1.501.50 (Frontier). 6.4 RQ4: Does the signal support calibrated black-box detection? Key specificity still does not by itself provide an operating decision. We therefore calibrate each audit against independently trained same-family Clean and Sham references, then evaluate cross-family models only as held-out negative controls. Reference pools are constructed in Appendix H. The resulting separation is large. All eight marked checkpoints exceed every same-family reference (8/88/8, pu=1/101p_u=1/101), while none of the 4040 cross-family controls crosses the corresponding threshold. The weakest marked checkpoint still exceeds the reference maximum by 5.455.45 p in the Llama setting and 5.105.10 p in Frontier (Figure 4). This is stronger evidence than a mean shift: detection survives the worst marked checkpoint and the strongest calibration reference. Figure 4: Full-budget black-box audit results. K: marked checkpoint (value shown); N: same-family reference; X: cross-family control. Cell color is 100Tu100T_u. The audit also remains useful before the full query budget is spent. At a nominal budget of 400400 queries, the worst-case detection margin is already positive in both settings (Llama: +2.25+2.25 p; Frontier: +0.75+0.75 p; Appendix I). 6.5 RQ5: Does the watermark survive later training? A useful provenance mark must persist after the marked checkpoint leaves its original training pipeline. We therefore continue the student with ordinary SFT or clean GRPO and compare each Marked checkpoint with a Clean checkpoint continued in lockstep. Under clean GRPO, the Llama single-carrier effects remain close to their initial values, while the Frontier joint signal increases after the evaluator is switched. SFT is more erosive for the surface-form carriers, but it does not erase them; the Frontier strategy signal remains essentially unchanged after 10241024 steps. The contrast suggests that persistence depends on what the carrier encodes: solution strategies are less easily overwritten by generic instruction tuning than lightweight response conventions. We then replace ordinary post-training with explicit removal objectives. Balanced-carrier SFT reduces the Frontier effect to 7.27.2 p after 20482048 steps, and anti-watermark GRPO reduces it to 1.81.8 p. Thus the mark is removable, but targeted optimization is substantially more effective than continued training. Schedules for Figure 5 are in Appendix G and K. Figure 5: Persistence and removal. (a), (b), (d), (e): 10241024 steps; (c), (f): 20482048 steps, with ΔUmark U_mark on the right axis. The main experiments use ρ=0.05ρ=0.05. Appendix L shows that weaker tilts transfer the code, while increasing ρ beyond the main setting yields diminishing watermark gains at a higher utility cost. The opening question was whether a score can leave a trace when the teacher never shows any text. It can. One registered preference is enough for the student to acquire it, and a joint code is still visible after the data, the student, or the evaluator changes. Rescoring the same outputs under other sign patterns shows that the student followed the enrolled code. The gap is large enough for a reference pool to flag the marked checkpoints and to leave the controls below the threshold. Ordinary later training does not wash the mark out, and adding it does not cancel the utility gain from distillation. 7 Conclusion and Discussion Radioactive Feedback traces label-only distillation through scalar scores. The student acquires an interpretable watermark in answer form and in solution strategy. Across students, evaluators, and tasks, the mark transfers without teacher demonstrations, generalizes past the data used to write it, and supports calibrated detection. It follows the enrolled code. Continued training leaves it detectable; targeted removal reduces it further. Future work should move this mark into the evaluator’s parameters. References Adi et al. (2018) Y. Adi, C. Baum, M. Cissé, B. Pinkas, and J. Keshet Turning your weakness into a strength: watermarking deep neural networks by backdooring. In USENIX Security, Cited by: §2. Anthropic (2026) Anthropic How Claude’s text watermark works. Note: Anthropic News External Links: Link Cited by: §1. Bai et al. (2022) Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Chen, et al. Constitutional AI: harmlessness from AI feedback. arXiv preprint arXiv:2212.08073. Cited by: §1, §2. Christ et al. (2024) M. Christ, S. Gunn, and O. Zamir Undetectable watermarks for language models. In COLT, Cited by: §2. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: Appendix D. Dathathri et al. (2024) S. Dathathri, A. See, S. Ghaisas, P. Huang, R. McAdam, J. Welbl, V. Bachani, A. Kaskasoli, R. Stanforth, T. Matejovicova, J. Hayes, N. Vyas, M. A. Merey, J. Brown-Cohen, R. Bunel, B. Balle, T. Cemgil, Z. Ahmed, K. Stacpoole, I. Shumailov, C. Baetu, S. Gowal, D. Hassabis, and P. Kohli Scalable watermarking for identifying large language model outputs. Nature 634 (8035), p. 818–823. Cited by: §1. Ding et al. (2023) N. Ding, Y. Chen, B. Xu, Y. Qin, S. Hu, Z. Liu, M. Sun, and B. Zhou Enhancing chat language models by scaling high-quality instructional conversations. In EMNLP, Cited by: Appendix G. Fang et al. (2026) J. Fang, Z. Hong, M. Zheng, M. Song, G. Li, H. Jiang, D. Zhang, H. Guo, X. Wang, and T. Chua Rubric-based on-policy distillation. arXiv preprint arXiv:2605.07396. Cited by: §1, §2. Gao et al. (2025) B. Gao, F. Song, Z. Yang, Z. Cai, Y. Miao, Q. Dong, L. Li, C. Ma, L. Chen, R. Xu, Z. Tang, B. Wang, D. Zan, S. Quan, G. Zhang, L. Sha, Y. Zhang, X. Ren, T. Liu, and B. Chang Omni-MATH: a universal olympiad level mathematic benchmark for large language models. In ICLR, Cited by: Appendix F. Gloaguen et al. (2026) T. Gloaguen, R. Staab, N. Jovanović, and M. Vechev LLM fingerprinting via semantically conditioned watermarks. In ICLR, Cited by: §1, §2. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §6. He et al. (2024) C. He, R. Luo, Y. Bai, S. Hu, Z. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, J. Liu, L. Qi, Z. Liu, and M. Sun OlympiadBench: a challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In ACL, Cited by: Appendix F. He et al. (2022) X. He, Q. Xu, Y. Zeng, L. Lyu, F. Wu, J. Li, and R. Jia CATER: intellectual property protection on text generation APIs via conditional watermarks. In NeurIPS, Cited by: §2. Hendrycks et al. (2021) D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt Measuring coding challenge competence with APPS. In NeurIPS Datasets and Benchmarks, Cited by: Appendix F. Hui et al. (2024) B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, K. Dang, Y. Fan, Y. Zhang, A. Yang, R. Men, F. Huang, B. Zheng, et al. Qwen2.5-Coder technical report. arXiv preprint arXiv:2409.12186. Cited by: §6. Jain et al. (2025) N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. I. Wang, A. Solar-Lezama, K. Sen, and I. Stoica LiveCodeBench: holistic and contamination free evaluation of large language models for code. In ICLR, Cited by: Appendix F. Kirchenbauer et al. (2023) J. Kirchenbauer, J. Geiping, Y. Wen, J. Katz, I. Miers, and T. Goldstein A watermark for large language models. In ICML, Cited by: §1, §2. Kuditipudi et al. (2024) R. Kuditipudi, J. Thickstun, T. Hashimoto, and P. Liang Robust distortion-free watermarks for language models. Transactions on Machine Learning Research. Cited by: §2. Lee et al. (2024) H. Lee, S. Phatale, H. Mansoor, T. Mesnard, J. Ferret, K. R. Lu, C. Bishop, E. Hall, V. Carbune, A. Rastogi, and S. Prakash RLAIF vs. RLHF: scaling reinforcement learning from human feedback with AI feedback. In ICML, Cited by: §1, §2. Li (2024) K. Li Verified TACO problems. Note: Hugging Face dataset likaixin/TACO-verified Cited by: Appendix F, Appendix F, §6.2. Li et al. (2023) R. Li, J. Fu, B. Zhang, T. Huang, Z. Sun, C. Lyu, G. Liu, Z. Jin, and G. Li TACO: topics in algorithmic COde generation dataset. arXiv preprint arXiv:2312.14852. Cited by: Appendix F, Appendix F, §6.2. Li et al. (2022) Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al. Competition-level code generation with AlphaCode. Science 378 (6624), p. 1092–1097. Cited by: Appendix F. Lightman et al. (2024) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. In ICLR, Cited by: Appendix F. Liu et al. (2023) J. Liu, C. S. Xia, Y. Wang, and L. Zhang Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. In NeurIPS, Cited by: Appendix D, Table 1. Lou et al. (2025) J. Lou, C. Zhang, X. Zhang, and K. Wu PreferCare: preference dataset copyright protection in LLM alignment by watermark injection and verification. In CCS, Cited by: §1, §2. Pan et al. (2025) L. Pan, A. Liu, S. Huang, Y. Lu, X. Hu, L. Wen, I. King, and P. S. Yu Can LLM watermarks robustly prevent unauthorized knowledge distillation?. In ACL, Cited by: §2. Patel et al. (2021) A. Patel, S. Bhattamishra, and N. Goyal Are NLP models really able to solve simple math word problems?. In NAACL, Cited by: Appendix D, Table 1. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model. In NeurIPS, Cited by: Appendix A, §2. Sander et al. (2026) T. Sander, H. Chang, T. Souček, T. Tran, V. Lacatusu, S. Rebuffi, A. Mourachko, S. Parimi, C. Ropers, R. Moritz, V. Stark, H. Elsahar, and P. Fernandez TextSeal: a localized LLM watermark for provenance & distillation protection. arXiv preprint arXiv:2605.12456. Cited by: §2. Sander et al. (2024) T. Sander, P. Fernandez, A. Durmus, M. Douze, and T. Furon Watermarking makes language models radioactive. In NeurIPS, Cited by: §1, §2. Savani et al. (2025) Y. Savani, A. Trockman, Z. Feng, Y. E. Xu, A. Schwarzschild, A. Robey, M. Finzi, and J. Z. Kolter Antidistillation sampling. In NeurIPS, Cited by: §1, §2. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §4.2. Shen et al. (2026) Y. Shen, L. Tu, and W. Wang Reinforcement learning-based semi-supervised knowledge distillation with LLM-as-a-judge. arXiv preprint arXiv:2604.02621. Cited by: §2. Uchida et al. (2017) Y. Uchida, Y. Nagai, S. Sakazawa, and S. Satoh Embedding watermarks into deep neural networks. In ICMR, Cited by: §1, §2. Xu et al. (2024) J. Xu, F. Wang, M. Ma, P. W. Koh, C. Xiao, and M. Chen Instructional fingerprinting of large language models. In NAACL, Cited by: §1, §2. Xu et al. (2025) Y. Xu, A. Liu, X. Hu, L. Wen, and H. Xiong Mark your LLM: detecting the misuse of open-source large language models via watermarking. arXiv preprint arXiv:2503.04636. Cited by: §2. Yan et al. (2025) J. Yan, Y. Li, Z. Hu, Z. Wang, G. Cui, X. Qu, Y. Cheng, and Y. Zhang Learning to reason under off-policy guidance. In NeurIPS, Cited by: Appendix F, Appendix F, §6.2. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §6. Yang et al. (2026) G. Yang, A. Ghasemian, F. Liu, Z. Wang, N. Mehrabi, and H. Hosseinmardi Asking back: interaction-layer antidistillation watermarks. arXiv preprint arXiv:2605.16462. Cited by: §2. Yuan et al. (2024) W. Yuan, R. Y. Pang, K. Cho, X. Li, S. Sukhbaatar, J. Xu, and J. E. Weston Self-rewarding language models. In ICML, Cited by: §1, §2. Zhao et al. (2024) X. Zhao, P. Ananth, L. Li, and Y. Wang Provable robust watermarking for AI-generated text. In ICLR, Cited by: §2. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In NeurIPS, Cited by: §2. Ziegler et al. (2019) D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. Cited by: Appendix A. Appendix Appendix A Proofs For a response y on prompt x with assigned carrier j(x)j(x), let hu(x,y)=e~cj(x)ϕj(x)(x,y)h_u(x,y)= e\,c_j(x) _j(x)(x,y) with hu∈−1,0,+1h_u∈\-1,0,+1\, and define δu(x,y)=clip[0,R](r0(x,y)+ρRhu(x,y))−r0(x,y), _u(x,y)\;=\;clip_[0,R] (r_0(x,y)+ρ R\,h_u(x,y) )-r_0(x,y), (11) so ru=r0+δur_u=r_0+ _u. Throughout, δu _u is this clipped displacement. If r0∈[0,R]r_0∈[0,R], then |δu|≤ρR _u ≤ρ R. In particular, when r0∈[ρR,R−ρR]r_0∈[ρ R,\,R-ρ R], the clip is inactive and δu=ρRhu _u=ρ R\,h_u. KL-regularized setup. Let β>0β>0 be the KL coefficient in the GRPO convention of Section 5. We analyze the idealized objective maxπy∼π(⋅∣x)[r(x,y)]−βKL(π(⋅∣x)∥π0(⋅∣x)), _π\;E_y π(· x)[r(x,y)]-β\,KL (π(· x)\,\|\, _0(· x) ), (12) whose optimizer is the Gibbs policy πr∗(y∣x)∝π0(y∣x)exp(r(x,y)/β). _r^*(y x)\; \; _0(y x) (r(x,y)/β ). (13) For the returned reward ru=r0+δur_u=r_0+ _u, this gives πu∗(y∣x)=πcl∗(y∣x)exp(δu(x,y)/β)y′∼πcl∗[exp(δu(x,y′)/β)]. _u^*(y x)\;=\; _cl^*(y x) ( _u(x,y)/β )E_y _cl^* [ ( _u(x,y )/β ) ]. (14) Proof of Proposition 5.1. The unique maximizer of (12) is the Gibbs policy (13) (Ziegler et al., 2019; Rafailov et al., 2023). Writing Z(r)=y∼π0[exp(r(x,y)/β)]Z(r)=E_y _0[ (r(x,y)/β)] for the partition function, πcl∗(y∣x) _cl^*(y x) =π0(y∣x)exp(r0(x,y)/β)/Z(r0), \;=\; _0(y x)\, (r_0(x,y)/β )/Z(r_0), (15) πu∗(y∣x) _u^*(y x) =π0(y∣x)exp(ru(x,y)/β)/Z(ru). \;=\; _0(y x)\, (r_u(x,y)/β )/Z(r_u). (16) Substitute ru=r0+δur_u=r_0+ _u and rearrange: πu∗(y∣x)=πcl∗(y∣x)exp(δu(x,y)/β)Z(r0)Z(ru). _u^*(y x)\;=\; _cl^*(y x)\, ( _u(x,y)/β )\, Z(r_0)Z(r_u). The prefactor Z(r0)/Z(ru)Z(r_0)/Z(r_u) equals the reciprocal of y′∼πcl∗[exp(δu/β)]E_y _cl^*[ ( _u/β)], which is (14). The log-odds identity (9) follows by taking the ratio of (14) at y and y′y ; the normalizing constants cancel. If δu _u is constant on the pair, the right-hand side is zero. When r0∈[ρR,R−ρR]r_0∈[ρ R,\,R-ρ R], δu=ρRhu _u=ρ R\,h_u and the identity equals (ρR/β)(hu(y)−hu(y′))(ρ R/β) (h_u(y)-h_u(y ) ). ∎ Proof of Proposition 5.2. Because |δu|≤ρR _u ≤ρ R whenever r0∈[0,R]r_0∈[0,R], clipping included, the change in any pairwise gap lies in [−2ρR,2ρR][-2ρ R,2ρ R], which is (10). If |r0(x,y)−r0(x,y′)|>2ρR r_0(x,y)-r_0(x,y ) >2ρ R, this increment cannot change the sign of ru(x,y)−ru(x,y′)r_u(x,y)-r_u(x,y ). The pass/fail claim is the same bound on the two-point reward 0,R\0,R\, for which ρ<1/2ρ<1/2. ∎ Appendix B Code registration and routing Applicability and code generation. Let m(x,y)∈0,1dm(x,y)∈\0,1\^d mark applicable coordinates. It is computed without the key, and every reward or audit inner product uses m(x,y)⊙ϕ(x,y)m(x,y) φ(x,y). A secret seed can generate a fixed code from registered carrier identifiers: cu,j=2PRFbit(su,idj)−1.c_u,j=2\,PRF_bit(s_u,id_j)-1. (17) The identifiers do not depend on prompts, so the code is reused across training and audit. Alternatively, the provider can register an explicitly chosen sign vector; the experiments use the latter. A two-coordinate example is quote style together with the closing marker Therefore/So: (+1,−1)(+1,-1) can favor double quotes and So; flipping both signs favors the opposite alternatives. The Frontier Test instead uses problem-solving carriers (Table F). Routing. Each training or audit prompt is assigned exactly one carrier j(x)j(x) before scoring, so each response receives one keyed scalar reward. Sham applies analogous eligibility and one-carrier assignment to a disjoint decoy carrier set, matched in carrier count, exposure target, and perturbation budget; it replaces the carrier set and code, not only the code. Registered-code auditing never uses the decoy coordinates. Bounded shift. With R=100R=100, ρ=0.05ρ=0.05, and maximal rubric strengths, an eligible response with a defined carrier value receives at most a ±5± 5 point shift; flipping the registered code flips the sign of that shift. Given eligibility, the keyed term depends only on that response’s carrier values and the registered code, not on which other candidates appear in the submitted group. Appendix C Experimental protocol Student–evaluator matrix. RQ1 uses Qwen3-30B-A3B as the scalar evaluator for Llama-3-8B-Instruct and Qwen2.5-Coder-1.5B-Instruct. The primary Frontier setting uses Qwen3-30B-A3B as the student and GPT-6 Astra as the evaluator. RQ2 broadens the comparison to nine student–evaluator combinations. Llama and Qwen2.5 are each paired with Qwen3-30B-A3B, GPT-5.6 Sol, and GPT-6 Astra. For the Qwen3-30B-A3B student, the open-weight evaluator is the larger Qwen3-235B-A22B, alongside GPT-5.6 Sol and GPT-6 Astra. The open-weight runs serve as transfer controls; the provenance threat model remains provider-controlled scalar feedback. RQ3–RQ4 use the selected Llama/Qwen3-30B-A3B and Qwen3-30B-A3B/GPT-6 Astra settings. Controls and seeds. The four conditions are Base (the unmodified student before RL), Clean (ordinary feedback), Sham (a strength- and exposure-matched preference over a disjoint decoy carrier set), and Marked (the enrolled carriers and registered code c). Trained arms share the GRPO recipe, data split, group size, and budget within a setting. Clean provides the baseline for watermark effects and task utility; Sham supplies matched decoy models for the detection reference pool. RQ1 single-carrier and RQ2 joint runs each use four independent GRPO training seeds per setting. Each of the nine RQ2 student–evaluator combinations trains under the registered four-carrier code for that student setting, with matched Clean checkpoints. Base is evaluated on the same held-out tasks for the distillation contrasts. Audit labels and correctness. Task utility and the correctness bit qiq_i use the official programmatic checks for each benchmark: execution harnesses for code and final-answer checking for mathematics. Claude Opus 5 supplies only the carrier or strategy label used in ϕφ; it is not the training evaluator and does not score task utility. A 10%10\% held-out subset of those labels, covering quote style, the Therefore/So marker, and the Frontier strategy tags on each primary audit domain, was checked by two annotators. Disagreements were adjudicated by discussion, with a third annotator breaking ties when consensus was not reached. The reported audits use one response per prompt. For the Frontier summary in Table 3, the pre-designated primary audits are CodeContests and Omni-MATH; Table 2 additionally reports four further held-out benchmarks. Appendix D Experimental data splits Table 4 lists the training and held-out evaluation sets. The RQ2 evaluator grid uses the same data split for a given student setting; only the scalar evaluator changes. GSM8K uses the official OpenAI split (Cobbe et al., 2021): mathematics training uses the 7,4737,473 training problems, and GSM8K watermark audit and utility evaluation use the 1,3191,319 test problems (8,7928,792 in total). Code audits use HumanEval+ and MBPP+ (Liu et al., 2023); the additional mathematics audit is SVAMP (Patel et al., 2021). Table 4: Training and held-out audit sets. Utility is measured on the same held-out prompts. Setting Training Held-out audit Llama-3-8B-Instruct; Qwen2.5-Coder-1.5B-Instruct TACO-verified 5k; GSM8K train HumanEval+, MBPP+; GSM8K test, SVAMP Qwen3-30B-A3B Frontier TACO-verified 5k; OpenR1-Math 5k carrier-compatible subsets of CodeContests, APPS, LiveCodeBench; Omni-MATH, MATH-500, OlympiadBench Appendix E Training and audit configuration Table 5 gives the GRPO configuration shared by Clean, Sham, and Marked in the initial comparison. Base is the initial student checkpoint evaluated without further training. Training makes one shuffled pass over the deduplicated split without prompt replacement. Each GRPO update uses a batch of 5050 prompts (the last may be smaller) and G=16G=16 student-generated responses per prompt, giving ⌈Ntrain/50⌉ N_train/50 updates. For one group, rewards are standardized as A^i=ru,i−r¯uG−1∑g=1G(ru,g−r¯u)2+10−6. A_i= r_u,i- r_u G^-1 _g=1^G(r_u,g- r_u)^2+10^-6. The KL reference is the frozen initial student checkpoint. The clean reward is the evaluator’s scalar score only. Reported carrier and strategy labels are produced by Claude Opus 5; those labels do not enter the training reward. The human check of a 10%10\% subset is in Appendix C. Task utility, including the correctness bit qiq_i, comes from official programmatic checkers. The hidden rubric returns only the final scalar reward to the learner. Its keyed shift may vary by eligible candidate but is bounded by ρR=5ρ R=5 points. The returned reward is clipped to the evaluator’s [0,100][0,100] scale. Table 5: Initial training and audit configuration. Setting Value Initial GRPO training one epoch; each training prompt used once Responses per prompt / prompt batch G=16G=16 / 5050 Optimizer / learning rate AdamW / 1×10−61× 10^-6 KL reference / β frozen initial student / 0.040.04 Clean evaluator scale [0,100][0,100] Maximum keyed shift / quality floor ρ=0.05ρ=0.05 (ρR=5ρ R=5), rmin=60r_ =60 Code-execution reward none Carrier and strategy labels Claude Opus 5 Task utility (qiq_i, Pass@1, accuracy) official programmatic checker Maximum new tokens 20482048 (GRPO rollouts and audit) Audit prompts / responses per prompt distinct prompts, ≤500≤ 500 per carrier / 11 Audit decoding temperature 00 For RQ4, every distinct audit prompt remains in the denominator. A response contributes a signed carrier value only when it is correct and classifiable; incorrect, unclassifiable, and inapplicable responses contribute zero. The correct-and-classifiable conditioning used for RQ1/RQ2 transfer summaries is therefore not the RQ4 detection denominator. Code audit prompt budget. Every audit prompt is used once. Decoding is temperature 00, so a repeated prompt would not have been a new draw. Training–audit deduplication stays exact on prompt text (below). Before training, every whitespace-normalized training prompt is compared against every audit prompt; exact matches are removed from the training split, as is standard train/audit hygiene. Appendix F Carrier definitions The reward path has no extractor. The preferred alternative is named in the evaluator instruction, and the score that comes back is the evaluator’s own score. The definitions below state the two sides of each carrier. At audit, a response counts as +1+1, −1-1, or abstaining under those same definitions. Mixed or undefined cases do not enter the aligned-cell rate. Code carrier pool. Table F lists the original 32 two-sided Python candidates. Each item is one bipolar contrast. The pool includes for/while and augmented assignment (+= versus x = x + y); both were screened. The binding constraint is not whether a contrast can be defined, but whether both polarities arise naturally on the same problem at comparable quality and matched length. The RQ1 code run uses quote style from this list. Table 6: Screened Python code carriers. Category Contrasts Function structure helper / inline; early return / single terminal return; recursion / iteration; return expression / temporary; type annotations / none Loop style enumerate / index loop; zip / index loop; for / while; direct / indexed iteration; continue / nested if Comprehensions list comprehension / loop+append; comprehension / map/filter; generator / list reduction; sum() / accumulator; any/all / explicit loop Data structures Counter/defaultdict / manual counts; set / list membership; heap / sort; deque / list queue; tuple / list literal membership Expressions ternary / if-else; direct Boolean / branched return; truthiness / len check; += / x = x + y Strings join / loop concatenation; f-string / .format(); double / single quotes Other syntax sorted() / in-place sort; [::-1] / reversed(); dict() / ; list() / []; range(n) / range(0,n) Code: quote style. Claude Opus 5 labels each audited program +1+1 for double quotes, −1-1 for single quotes, or undefined. These labels are the reported carrier values. Mathematics: final-paragraph marker. Claude Opus 5 labels each audited solution +1+1 when the closing is Therefore, −1-1 when it is So, or undefined. These labels are the reported carrier values. Joint Llama/Qwen2.5 carriers. Table F lists the four enrolled carriers for the Llama-3-8B-it and Qwen2.5-Coder-1.5B-it joint checkpoints in Table 3. Qwen2.5 uses the same carrier set and registered code; only the student initialization differs. Table 7: Four-carrier set for joint Llama and Qwen2.5 training. Domain Carrier Source Code Quote style (double vs. single) RQ1 Code for vs. while loop style Screened pool Math Final-paragraph Therefore/So marker RQ1 Math Ternary vs. if-else branch Screened pool Math code-block carrier. The ternary/if-else carrier applies only when a response contains a Python code block. On GSM8K/SVAMP, Llama often emits a short final program. At audit, Claude Opus 5 labels that last code block +1+1 for a ternary expression, −1-1 for an explicit if/else, or abstaining. Responses without a code block abstain on this coordinate. Coverage is lower than for the closing-marker carrier but nonzero on the math training split. Sham decoy carriers. Sham replaces both the registered code and the carrier set. Table F lists the four decoy carriers for Llama/Qwen2.5 joint training; each is disjoint from Table F and matched in domain count, exposure target, and perturbation budget. Single-carrier RQ1 Sham uses one decoy per domain: f-string/.format() on code and closing Hence/ Thus on math. Frontier Sham uses four decoy strategy carriers disjoint from Table F: divide-and-conquer vs. iterative accumulation; Floyd–Warshall vs. repeated DSU queries; closed-form vs. brute enumeration; pure angle chasing vs. complex-number geometry. Registered-code auditing never scores decoy coordinates. Table 8: Decoy carrier set for joint Llama and Qwen2.5 training. Domain Decoy carrier Source Code f-string vs. .format() Screened pool Code List comprehension vs. loop+append Screened pool Math Closing Hence/Thus marker Custom (text) Math Direct Boolean vs. branched return Screened pool Frontier Test carriers. The Qwen3-30B-A3B student does not reuse the surface-form carriers above, so the Frontier result is not a quote-style or Therefore/So preference. Table F lists the four two-sided problem-solving carriers and the data splits. Frontier strategy preference. There is no strategy extractor in the reward path. The evaluator instruction states the registered preference and returns one scalar score for each of the G=16G=16 candidates; recognizing the preferred side changes the emitted score. The definitions below specify the two sides used by all evaluators in the RQ2 grid. Mixed or unclassifiable solutions are not counted in the aligned-cell rate. Audit reads held-out answers against the same definitions. • Top-down DP ↔ bottom-up DP: +1+1 if the solution fills a DP table recursively or top-down with memoization before combining subproblems; −1-1 if it iterates over states in increasing problem size without recursion. • Traversal ↔ DSU: +1+1 if connectivity is resolved by BFS/DFS on an explicit graph; −1-1 if by a disjoint-set union–find structure. • Direct counting ↔ inclusion–exclusion: +1+1 if the answer is obtained by direct enumeration or product rule; −1-1 if by inclusion–exclusion or generating functions. • Synthetic ↔ coordinate geometry: +1+1 if the proof is synthetic (angle/chord reasoning without coordinates); −1-1 if it introduces a coordinate system or vectors. Training uses two fixed 5,0005,000-prompt subsets, not the full releases. The two code carriers share a 5,0005,000-prompt draw from TACO-verified (Li et al., 2023; Li, 2024). The two mathematics carriers share a 5,0005,000-prompt draw taken at random from OpenR1-Math-46k-8192 (Yan et al., 2025). No training problem inside these draws is removed by a carrier filter. The keyed adjustment is applied only to a response that meets the quality floor and exposes a classifiable value of its assigned carrier; every other response keeps its internal rubric score. Watermark transfer is audited on carrier-compatible subsets of CodeContests (Li et al., 2022), APPS (Hendrycks et al., 2021), and LiveCodeBench (Jain et al., 2025) for code, and of Omni-MATH (Gao et al., 2025), MATH-500 (Lightman et al., 2024), and OlympiadBench (He et al., 2024) for mathematics. A problem enters the subset for carrier j when the dataset’s own reference solution expresses exactly one side of j under the definitions above. Solutions that mix both sides or express neither side are excluded. The reference solution is not a student rollout. This filter is fixed before Marked and Clean training and does not use Marked-versus-Clean outcomes. These audit datasets are not used to write the watermark. The primary Frontier setting uses GPT-6 Astra, while Table 3 additionally varies the evaluator. No evaluator-generated answers or chain-of-thought enter student training. Table 2 reports watermark transfer and utility on the held-out audit datasets. Table 9: Frontier training and held-out audit sets. Code training is a 5,0005,000-prompt TACO-verified subset (Li et al., 2023; Li, 2024); mathematics training is a random 5,0005,000-prompt subset of OpenR1-Math-46k-8192 (Yan et al., 2025). Each audit cell is the carrier-compatible subset of that benchmark. Domain Carrier Training Held-out audit Code Top-down DP ↔ bottom-up DP TACO-verified 5k CodeContests, APPS, LiveCodeBench Traversal ↔ DSU Math Direct counting ↔ inclusion–exclusion OpenR1 5k Omni-MATH, MATH-500, OlympiadBench Synthetic ↔ coordinate geometry Appendix G Continued-training details Figure 5 plots the watermark effect defined in Metrics. At each logged step the matched Clean checkpoint is trained on the same schedule as the Marked checkpoint, and the plotted value is Markedt−CleantMarked_t-Clean_t. Panel (a) continues with instruction SFT on UltraChat (Ding et al., 2023) without keyed feedback or an evaluator. Llama GSM8K/SVAMP WM align. changes from +43.6/+64.9+43.6/+64.9 to +39.2/+58.4+39.2/+58.4 by step 10241024; Frontier stays near its step-00 gap (+15.35→+15.17+15.35→+15.17). Panels (b) and (e) continue with clean GRPO, and each arm uses its own step-00 checkpoint as the KL reference. The Llama panel keeps the Qwen3-30B-A3B evaluator, while the Frontier panel continues under GPT-5.6 Sol. Llama ends at +43.4/+65.5+43.4/+65.5, and Frontier at +18.71+18.71. Appendix H Reference calibration protocol This appendix specifies the reported evaluation in Section 6.4. Carriers, the definitions in Appendix F, audit prompts, decoding, and the enrolled code are fixed before scoring, separately for each student–evaluator setting. Reference composition. Each pool contains 5050 independently trained Clean checkpoints and 5050 independently trained Sham checkpoints using decoy carrier sets disjoint from the enrolled carriers. The 100100 runs use disjoint random seeds and the same architecture, data split, and training budget as the audited student; none is an intermediate save of another run. These 100100 same-family checkpoints define the calibration pool and rank denominator. Separately, we score 2020 independently trained cross-family controls. They follow the same data split, GRPO recipe, and training budget as the same-family references and are finished runs, not intermediate checkpoints; only the student family changes. For the Llama-3-8B-Instruct audit, these controls are Qwen2.5-7B-Instruct models; for the Qwen3-30B-A3B Frontier audit, they are Llama-3-8B-Instruct models. Cross-family controls do not enter calibration or the rank denominator. No control model shares a training seed or key with the four marked checkpoints, and pools are not combined across settings. Each marked checkpoint is scored as a separate suspect. Table 3 summarizes the broader joint-transfer grid; RQ4 uses the selected Llama/Qwen3-30B-A3B and Qwen3-30B-A3B/GPT-6 Astra settings, with detection computed from individual checkpoint scores. Scoring. Apply Equation (5) to every checkpoint and compute its suspect rank using Equation (6). Fix α=1%α=1\% before suspect evaluation and use the non-interpolated upper empirical quantile defined in Section 4.3. With 100100 references, γ is the reference maximum; exceeding it gives pu=1/101p_u=1/101 and passes Equation (7). The pool has fixed Clean and Sham strata, 5050 runs each. Cross-family controls. The cross-family checkpoints receive no marked feedback from the enrolled evaluator and are held out from calibration. Table 10 shows that their maximum 100Tu100T_u remains below the same-family calibration threshold in each setting. Table 10: Held-out cross-family controls. Max. 100Tu100T_u is the largest score among the 2020 controls; γ is calibrated on the 100100 same-family references only. Audit student Enrolled evaluator Cross-family component Max. 100Tu100T_u γ Llama-3-8B-it Qwen3-30B-A3B Qwen2.5-7B-Instruct (2020) 1.001.00 1.151.15 Qwen3-30B-A3B GPT-6 Astra Llama-3-8B-Instruct (2020) 1.101.10 1.501.50 Table 11: Black-box detection results. Setting Detected pup_u Llama / Qwen3-30B-A3B 4/44/4 1/1011/101 Frontier / GPT-6 Astra 4/44/4 1/1011/101 Both settings 8/88/8 — Reported outputs. Table 11 reports full-budget detection. The full-budget score panels appear in Figure 4. Appendix I Audit query budget analysis Section 6.4 uses the detection margin Δqdetect=mink∈Tk(q)−maxm∈nullTm(q) _q^detect= _k T_k(q)- _m _nullT_m(q) (18) to study query efficiency. For each nominal per-carrier budget q∈100,200,300,500q∈\100,200,300,500\, we use up to q distinct prompts from each carrier’s available audit pool, without replacement. The plotted budget is the nominal total 4q4q. We then recompute every marked and null checkpoint score, take the null reference maximum, and form Δqdetect _q^detect from the weakest marked checkpoint. Figure 6 reports these margins. At a nominal budget of 400400 the margins remain positive in both settings (Llama: +2.25+2.25 p; Frontier: +0.75+0.75 p). Figure 6: Detection margin versus nominal audit-query budget. Appendix J Registered-key attribution protocol Section 6.3 tests whether inherited alignment is specific to the enrolled code ⋆c . The four marked checkpoints in a setting are seeds of that one code. They are not retrained under other keys. We freeze each seed and score the same responses under alternative length-44 sign vectors at Hamming distances 11 through 44 from ⋆c . Registered owner codes remain at pairwise distance at least 22, so one enrolled owner is not a one-bit neighbor of another. The distance-11 value averages the four one-bit neighbors of ⋆c . The dashed threshold is γ(⋆)γ(c ), the maximum score of the reference pool when that pool is scored with the enrolled code. Alternative codes are compared with this same threshold. Figure 3 plots, at each distance, the average of 100Tu100T_u over all sign patterns at that distance, not the score of one alternative code. Linearity of Equation (5) fixes that average at (1−d/2)(1-d/2) times the enrolled-code score. Appendix K Removal stress tests Section 6.5 extends the reported ordinary SFT/GRPO continuation with two stress tests on Frontier joint checkpoints: balanced-carrier SFT mixes prompts that expose all carriers with uniform frequency but removes keyed rewards, and explicit anti-watermark GRPO applies the reverse of the enrolled code on eligible responses while keeping the clean utility evaluator. Both runs share the RQ5 step schedule 0,256,512,1024,20480,256,512,1024,2048. The matched Clean arm takes the same steps without the removal objective, and WM effect at step t is Markedt−CleantMarked_t-Clean_t, reported together with ΔUmark U_mark. Figure 5 reports WM effect and ΔUmark U_mark along this schedule. Appendix L ρ sweep ablation All main experiments use ρ=0.05ρ=0.05 and rmin=60r_ =60. Figure 7 varies that tilt over ρ∈0,0.01,0.025,0.05,0.10ρ∈\0,0.01,0.025,0.05,0.10\ for the Frontier joint setting. The ρ=0.05ρ=0.05 point is the Frontier entry in Table 3: the unweighted mean of CodeContests and Omni-MATH, 15.3515.35 p, reported as +15.4±0.5+15.4± 0.5, with ΔUmark=+0.5 U_mark=+0.5 p. WM effect is 00 at ρ=0ρ=0, 4.24.2 p at ρ=0.01ρ=0.01, and 11.511.5 p at ρ=0.025ρ=0.025. Below ρ=0.05ρ=0.05, ΔUmark U_mark stays within ±0.1± 0.1 p. Increasing to ρ=0.10ρ=0.10 adds only 1.851.85 p of WM while incurring ΔUmark=−0.35 U_mark=-0.35 p. Figure 7: Frontier joint watermark across keyed tilt ρ. The ρ=0.05ρ=0.05 point is the Table 3 Frontier summary. Appendix M Implementation and checkpoint map This appendix records details needed to reconstruct the reported runs. Training budget. Initial GRPO uses one epoch over the deduplicated training split (Table 4), with G=16G=16 rollouts per prompt and prompt batches of 5050 (Table 5). Each optimizer step therefore scores 800800 responses (50×1650× 16). Table 12 turns that rule into a rollout budget. NtrainN_train counts each training prompt once, before the exact audit-prompt deletion in Appendix E, so the step counts are upper bounds. Llama and Qwen2.5 joint runs use TACO-verified (5,0005,000) plus the GSM8K training split (7,4737,473), hence Ntrain=12,473N_train=12,473. Frontier runs use TACO-verified (5,0005,000) plus a random 5,0005,000-prompt subset of OpenR1-Math-46k-8192, hence Ntrain=10,000N_train=10,000. A same-family reference pool is 100100 independent runs of that setting. The 4040 cross-family controls use the audited setting’s split, 2020 runs each. Table 12: Compute manifest for one initial GRPO epoch (G=16G=16, prompt batch 5050). Steps are per run, ⌈Ntrain/50⌉ N_train/50 . Rollouts are the row total. The two joint rows each describe a single training run. Each reported joint result averages four seeds, and each seed also trains a matched Clean run of the same size. Reference-pool and cross-family rows count every run. Run class Runs NtrainN_train Steps / run Rollouts Llama / Qwen2.5 joint, one run 11 12,47312,473 250250 199,568199,568 Frontier joint, one run 11 10,00010,000 200200 160,000160,000 Llama-setting reference pool 100100 12,47312,473 250250 19,956,80019,956,800 Frontier-setting reference pool 100100 10,00010,000 200200 16,000,00016,000,000 Cross-family controls 4040 — — 7,191,3607,191,360 RQ5 continuation curves start from the step-00 marked checkpoint in each setting: Llama-math uses the RQ1 single-carrier marked checkpoint; Frontier uses the RQ2 joint marked checkpoint. RQ4 audits the four marked checkpoints from joint training on students from different families in the two settings (Llama and Qwen3-30B-A3B), not the RQ5 continuation checkpoints. Seeds and arms. RQ1 single-carrier and RQ2 joint runs each use four independent GRPO training seeds per setting: 11, 4242, 20262026, and 20252025. Each seed trains a matched Clean checkpoint. Base is the initial no-RL student checkpoint for that setting, evaluated on the same held-out tasks; it requires no separate training run or seed. Tables 1–3 subtract the paired Clean checkpoint inside each seed, then report the mean and standard deviation of those four differences. The reference pool contains 5050 Clean and 5050 Sham checkpoints with seeds disjoint from all marked suspects. Sham arms use the decoy carriers in Appendix F; Marked arms use the enrolled carriers and registered code for that setting. Registered codes. Joint Llama/Qwen2.5 training uses an explicitly registered four-sign code on the carriers in Table F; the same code is used across Qwen3-30B-A3B, GPT-5.6 Sol, and GPT-6 Astra. Frontier training uses a separate four-sign code on Table F, shared across Qwen3-235B-A22B, GPT-5.6 Sol, and GPT-6 Astra in Table 3. Single-carrier RQ1 uses one enrolled carrier per domain with code (+1)(+1) on the positive polarity defined in Appendix F.