Paper deep dive
Which Constraints Are Missing? Ask the Verifier: Graded Rewards for Constraint-Following Music Generation
Haoyue Liu, Ye Chen, Zhichao Wang, Xiaoyu Ma, Haoran Shou, Xiaoying Tang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/27/2026, 2:58:11 AM
Summary
This paper introduces MusicConstraintBench, a benchmark for evaluating constraint-following in symbolic music generation, and MusicRLVR, a reinforcement learning framework that uses graded, verifier-driven rewards to improve model compliance. The authors demonstrate that binary rewards are sparse and ineffective for training, whereas MusicRLVR's approach of combining a hard validation gate with per-property credit and a joint-satisfaction bonus significantly improves performance on mixed constraints, allowing a 4B parameter model to outperform larger zero-shot baselines.
Entities (6)
Relation Signals (4)
MusicRLVR → improves → Qwen3-4B-Instruct
confidence 95% · On MusicConstraintBench, MusicRLVR lifts Qwen3-4B-Instruct from 0.160 to 0.807 on mixed constraints
MusicRLVR → outperforms → LLaMA 3.1-70B
confidence 95% · It also generalises to property combinations unseen in training... leads every zero-shot baseline including Llama-3.1-70B at 0.380.
MusicConstraintBench → evaluates → Constraint Families
confidence 90% · We construct MusicConstraintBench, 2,180 items over eight constraint families... key, meter, length, range, final note, rhythm, motion and form
MusicRLVR → uses → GRPO
confidence 85% · Algorithm 1 applies this reward in GRPO... MusicRLVR trains a policy... using a hard validation gate and graded property-level rewards
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Constraint-following music generation asks a score to satisfy several user-specified properties at once, each checkable programmatically (key, meter, length, range, final note, rhythm, motion and form), yet no existing benchmark isolates this capability. We construct MusicConstraintBench, 2,180 items over eight constraint families, on which current models fail once a few constraints are combined. The natural remedy is reinforcement learning with these verifiers as reward, yet we observe that a reward paid only when every property holds leaves most training groups without a learning signal: over the first 50 updates, 0.550 of rollout groups score identically and receive no gradient, even though a failing score typically misses only one requested property. Under the joint criterion, rollouts for a prompt tend to fail together, so a binary reward cannot separate a nearly correct score from a malformed one. We therefore introduce MusicRLVR, which pays graded per-property credit behind a hard validation gate that rejects malformed outputs, plus a joint-satisfaction bonus, requiring no human annotation, learned reward model, or music-domain fine-tuning. On MusicConstraintBench, MusicRLVR lifts Qwen3-4B-Instruct from 0.160 to 0.807 on mixed constraints and leads every zero-shot baseline including Llama-3.1-70B at 0.380. It also generalises to property combinations unseen in training and to out-of-range parameter values, showing that verifiable rewards need not presuppose a target output.
Tags
Links
- Source: https://arxiv.org/abs/2609.23665v1
- Canonical: https://arxiv.org/abs/2609.23665v1
Trouble viewing inline? Open PDF directly →
Full Text
137,500 characters extracted from source content.
Expand or collapse full text
Published as a conference paper at ICLR 2027 Which Constraints Are Missing? Ask the Verifier: Graded Rewards for Constraint-Following Music Generation Haoyue Liu 1,3 Xiaoyu Ma 1 Ye Chen 2 Zhichao Wang 1 Haoran Shou 1 Xiaoying Tang 1,3,† 1 School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen 518172, China 2 XJTU-POLIMI Joint School, Xi’an Jiaotong University, Xi’an 710049, China 3 Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen) Abstract Existing work on symbolic music generation has focused on musicality and perceptual quality, while paying less attention to whether models can jointly satisfy user-specified score constraints. Existing evaluations also lack a dedicated benchmark for this capability. Accordingly, we construct MusicConstraintBench, a benchmark of2,180items spanning eight families of programmatically verifiable constraints, and find that joint satisfaction drops sharply as constraints accumulate. A natural solution is to use these verifiers as reinforcement-learning rewards, but binary all-satisfied rewards provide sparse supervision: during the first50updates,0.550 of rollout groups receive identical rewards and thus no reward gradient; meanwhile, such a binary criterion cannot distinguish partially compliant outputs from complete failures. To address this limitation, we introduce MusicRLVR, a verifier-driven reinforcement learning framework that combines a hard validation gate, graded per-property credit, and a joint-satisfaction bonus, requiring no human annotation, learned reward model, or music-domain SFT. On MusicConstraintBench, Music- RLVR improves Qwen3-4B-Instruct from0.160to0.807in joint satisfaction on mixed constraints, enabling the 4B model to outperform Llama-3.1-70B. It generalises to property combinations unseen during training and to out-of-range parameter values, showing that verifiable rewards can train open-ended generation models without presupposing a unique target output. 1 Introduction 1234 Number of constraints k 0.0 0.2 0.4 0.6 0.8 1.0 All-satisfied rate 0.500 Qwen3-4B Qwen3-32B Llama-70B MiniMax-M3 SFT RL (binary) RL (SFT init.) RL (Direct) Figure 1: All-satisfied rate againstkon Mixed, for the matched runs and the baselines of Table 2; each slice has its own family mix- ture. The right-hand label gives the Direct MusicRLVR rate at k = 4. Language models can now generate machine-readable symbolic scores from free-text descriptions (Yuan et al., 2024; Qu et al., 2025; Wang et al., 2025a). For users who want a piece that satisfies explicit requirements rather than an arbitrary piece, however, a more natural interface is a score specification that names properties such as key, meter, length, pitch range, final note, rhythmic vocabu- lary, melodic motion, and form. Unlike perceptual quality (Cideron et al., 2024; Jonason et al., 2025; Ziv et al., 2026), these properties can be evaluated programmatically from the parsed score, placing constraint-following music gener- ation in a class of property-verifiable generation problems. Yet existing evaluations do not systematically measure whether models can satisfy such constraints jointly, as no dedicated benchmark isolates this capability. We therefore construct MusicConstraintBench, a benchmark of2,180 ABC-generation items spanning eight constraint families (Yuan et al., 2024; Qu et al., 2025; Wang et al., 2025a). With this benchmark, we ask a central question: when each requested property is individually verifiable, can a model actually satisfy them jointly in a single generation? 1 arXiv:2609.23665v1 [cs.SD] 20 Sep 2026 Published as a conference paper at ICLR 2027 Our experiments reveal a sharp collapse in multi-constraint compliance. A request may simultaneously require a tune to be in D minor, in6/8, sixteen bars long, ending on the tonic, with limited melodic leaps and an AABA form; because these requirements must hold jointly, violating any one of them makes the output non-compliant. As the number of simultaneous constraints increases, the all-satisfied rate drops sharply (Figure 1). A natural solution is to use the deterministic verifiers directly as reinforcement-learning rewards, but a binary all-satisfied reward inherits the same conjunction problem: once any constraint fails, outputs with very different degrees of partial compliance receive the same reward. In GRPO, this directly removes the learning signal when all rollouts in a group receive the same reward. In our matched training runs,0.550of rollout groups have zero reward variance over the first50updates and therefore provide no reward-driven gradient. Thus, while a verifier can determine whether an output is fully correct, a binary verdict cannot distinguish partially compliant outputs from complete failures. This motivates our core question: Can deterministic property checks provide the task-specific supervision a general-purpose model needs to jointly follow multiple musical instructions? Turning these verifiers into a training reward therefore requires more than a simple pass/fail signal. Yet assigning partial credit alone creates the opposite problem: a fragment may satisfy several local properties and receive reward without forming a valid tune. To address this, we introduce MusicRLVR, a verifier-driven reinforcement learning framework with two complementary components. First, a hard validation gate rejects malformed, unparseable, or otherwise invalid score outputs before any property credit is assigned. Second, for outputs that pass the gate, an item-specific graded reward assigns per-property credit for the constraints requested by the prompt and adds a joint-satisfaction bonus when all requirements are met. For the more structured form constraint, we additionally provide graded feedback within the constraint family itself, allowing training to distinguish different degrees of partial success. The framework requires no human annotation or learned reward model. We summarise our contributions as follows: •The first benchmark for joint score-constraint compliance. We introduce MusicConstraint- Bench, to our knowledge the first benchmark specifically designed to evaluate whether language models can jointly satisfy multiple programmatically verifiable score constraints. It contains2,180 items spanning eight verifier-defined constraint families, plus400paraphrases, and systemati- cally tests constraint density, unseen family combinations, parameter extrapolation, and wording robustness. •Turning property verifiers into fine-grained training rewards. We introduce MusicRLVR, which combines a hard validity gate, graded per-family credit, and a joint-satisfaction bonus without human annotation or a learned reward model. The verifier configuration is item-specific, and the reward provides fine-grained supervision both across requested constraint families and, for form, within a single family. •Training constraint following without music-domain SFT. Direct GRPO from Qwen3-4B- Instruct raises the Mixed joint-satisfaction rate from0.160to0.807, enabling the 4B model to outperform every evaluated zero-shot baseline, including Llama-3.1-70B. The gains further generalise to property combinations never observed together during training and to parameter values outside the training range. 2 Related Work Symbolic music language models, from LSTMs over ABC (Sturm et al., 2016) and the Music Transformer (Huang et al., 2018) to ABC-native LLMs (Yuan et al., 2024; Qu et al., 2025; Wang et al., 2025a), description-conditioned control (von Rütte et al., 2023; Thickstun et al., 2023) and audio models (Agostinelli et al., 2023; Copet et al., 2023), are trained by likelihood or against a learned music–text encoder (Wu et al., 2025); hard rules enter, if at all, at decoding time (Lu et al., 2021; Huang et al., 2024; Kaliakatsos-Papakostas et al., 2025). RL has been applied in two directions: optimising a human or learned quality signal (Cideron et al., 2024; Jonason et al., 2025; Ziv et al., 2026; He et al., 2026), or rewarding music-theory heuristics as a style prior (Jaques et al., 2017). However, all of these score with a human, a learned model, or a fixed prior, none of which can report whether the properties this prompt requested actually hold. In contrast, we score with a deterministic checker whose family and parameters change from item to item, and test combinations withheld from training. 2 Published as a conference paper at ICLR 2027 Figure 2: Overview of MusicRLVR. Left: Multi-constraint prompts make binary all-satisfied rewards sparse, leaving partially correct rollouts indistinguishable. Right: MusicRLVR samplesGABC rollouts, applies a hard validation gate and deterministic property verifiers, and uses graded per-family credit plus an all-satisfied bonus to provide the group-relative signal for GRPO. RLVR grew out of PPO-based RLHF (Schulman et al., 2017; Ouyang et al., 2022), with group-relative baselines (Ahmadian et al., 2024; Shao et al., 2024) removing the value network; it underpins recent reasoning models (Guo et al., 2025; Lambert et al., 2024) and is being pushed beyond mathematics (Ma et al., 2026; Gunjal et al., 2026). DAPO (Yu et al., 2026) and Dr. GRPO (Liu et al., 2025) patch dead groups and length bias, whereas we grade the reward itself (§3.3); like IFEval (Zhou et al., 2023) we score only verifiable instructions. Closest in domain, Wang et al. (2025b) synthesise verifiable sheet-music reasoning questions and study transfer to symbolic music continuation. Our task instead requires joint satisfaction of multiple musical properties for which no target composition exists: the reward is computed from the properties the generation prompt requests, and we evaluate held-out combinations and parameter extrapolation rather than reasoning-question accuracy. 3 Method MusicRLVR trains a policy to satisfy the properties requested by each prompt using a hard validation gate and graded property-level rewards (Figure 2). Algorithm 1 applies this reward in GRPO, optionally after verifier-labelled SFT. 3.1 Problem Formulation LetFbe a set of constraint families; we use the eight families in Table 1. A constraintc = (f,θ)pairs a familyf ∈Fwith parametersθ(e.g.f = key,θ = (G, major)). Each family has a deterministic verifierv f :S × Θ f →0, 1, whereSis the set of parseable scores andΘ f the parameter space of familyf. A problem is a set of constraintsC = c 1 ,...,c k with distinct families, rendered into a promptx C by a fixed template. The policyπ φ , with trainable parametersφ, emits a stringy in ABC text notation (Appendices C.5–C.7);ρextracts the ABC and applies the validation gate of Appendix C.1, returning ρ(y)∈S or⊥ on rejection. The training reward uses the binary indicator A(y,C) = Y (f,θ)∈C v f (ρ(y),θ) ∈ 0, 1 if ρ(y)̸=⊥, A(y,C) = 0 if ρ(y) =⊥, (1) soA = 1requires the training gate and every requested family check to pass. At evaluation, all-satisfied rate is the fraction of all test items whose outputs pass the parse gate and satisfy every requested musical constraint, i.e. the mean of A. 3 Published as a conference paper at ICLR 2027 Table 1: The eight constraint families, each decided by a deterministicmusic21routine on the parsed score, spanning global, bar, note, transition and cross-section properties. The third column is the default sampler’s domain; the verifier domain Θ f is wider, which the Edge split exploits. Table 18 adds each prompt sentence. FamilyChecked overDefault sampler values keyglobal13 keys: 8 major, 5 minor meterevery bar4/4, 3/4, 6/8, 2/4; each bar must sum to the signature lengthglobal8, 12 or 16 bars, no pickup rangeevery note4 inclusive pitch spans, e.g. G3–A5 finallast notea pitch class, any octave (a scale degree 1/3/5 when a key is present) rhythmevery note4 admissible duration sets; no dotted values, no rests leapadjacent noteslargest melodic interval: 7, 9 or 12 semitones structure4-bar unitsaaba, aabb or abab over16bars; equal letters identical, different letters distinct 3.2 Constraint Families and Verifiers Verifying properties at multiple musical scales. Table 1 lists the eight constraint families. Seven constrain attributes other than multi-section form: key, meter, bar count, pitch range, final pitch class, admissible note durations, and largest melodic interval. The eighth,structure, is a cross-section form constraint: a sixteen-bar tune in a stated form over four-bar units (AABA, AABB, ABAB), units sharing a letter identical note-for-note and units with different letters differing. A unitu a (a ∈ 1,..., 4) is its sequence ofM = 4bar signatures (pitch and duration per event); a tune without exactly4Mbars scores0on every structure term. WritingEfor the index pairs required equal,Dfor those required different (minimal sets per form in Appendix C.1) andI[·]for an indicator, v structure = I h V (a,b)∈E u a = u b i · I h V (a,b)∈D u a ̸= u b i .(2) We writev rep andv contrast for the two factors, the repetition and contrast halves; their diagnostic roles are examined in §5.5. Their conjunction is zero whenever either fails, however close the output is. Writing u a [m] for the m-th bar signature of unit u a , the graded counterparts are ̄e = 1 |E|M X (a,b)∈E M X m=1 I u a [m] = u b [m] , ̄ d = 1 |D|M X (a,b)∈D M X m=1 I u a [m]̸= u b [m] , (3) In Equation 3, ̄ecounts how much of the required repetition is already in place rather than whether all of it is. Equation 4 uses ̄e; Appendix E.2 reports what happens when ̄ dis used symmetrically alongside it. Calibrating the key criterion. Thekeyverifier is the one hybrid among the eight, because a literal check fails in both directions: theK:header is gameable, and key-profile analysis (Krumhansl, 2001; Temperley, 1999) (Aarden–Essen profiles) agrees with the corpus header on only0.750of400corpus tunes. We accept a tune iff the header matches and the analysed key is the requested one or its relative (Appendix C.3). 3.3 Reward Grading per-family credit behind a hard validation gate. For a completionyon problemCwith k =|C|, R(y,C) = 0,ρ(y) =⊥, λ· 1 k X (f,θ)∈C s f (ρ(y),θ) + (1− λ)A(y,C), otherwise, (4) withλ = 0.7, which leaves the binary term0.3of the reward even atk = 1(Appendix D.9 sweeps it). Heres f :S × Θ f → [0, 1]equalsv f for the seven non-structure families; forstructureit is graded,s structure = 0.7 ̄e + 0.3v contrast , where ̄eis the fraction of matching bar positions across pairs inE. Grading repetition and leaving contrast binary is a design choice: exact bar matches give one 4 Published as a conference paper at ICLR 2027 notion of partial repetition, whereas counting differing bars need not reflect perceptually meaningful contrast (symmetric variant: Appendix E.2). The parse gate is hard: unparseable output earns zero. Equation 4 affects training only; the main comparison and reward-design ablation use the same binary success criterion. Partial training credit is never counted as all-satisfied success. Partial credit matters because of how GRPO estimates advantages. For a promptx C , GRPO samples a group of G rollouts and centres their rewards within that group, ˆ A i = R(y i ,C)− μ C σ C + ε , μ C = 1 G G X j=1 R(y j ,C), σ 2 C = 1 G− 1 G X j=1 R(y j ,C)− μ C 2 , (5) withε = 10 −4 a small constant. A prompt contributes a reward-driven update only when rewards differ inside its own group; their level does not enter, and a group whose rollouts all score alike contributes only the KL term (DAPO (Yu et al., 2026) discards such groups; partial credit can make some of them informative). BinaryAseparates no partial ordering at all. Theλ-weighted average separates rollouts satisfying different numbers of families, but withs f = v f it still scores form as one conjunction. Holding the contrast verdict fixed, the gradeds structure also separates rollouts that fail the form check but match different fractions of bar positions, which is credit within a family. Section 5.5 tests this design against binaryAat a matched initialisation and budget: over the first 50updates the graded reward leaves0.307of groups without a reward gradient against0.550under binaryA, and the graded arm ends ahead over the benchmark as a whole. Appendix D.14 reports the full traces and Appendix E.2 the form decomposition. Rejecting a malformed output rather than paying for its recoverable fraction. Scoring that fraction (cf. Skalse et al., 2022) would let a fragment satisfyingkeyandrange, but forming no tune, collectλ· 2/k. Training and main evaluation share the gate; Appendix C.2 audits additional fixed-format requirements and re-scores the matched checkpoints of Table 2 under them, all of which keep every success on all eight suites. The family verifiers and score parser encode music-specific assumptions; the reward aggregation and optimiser are generic. 3.4 Training We compare two initialisations: Direct starts from Qwen3-4B-Instruct; SFT init. first applies the music-domain SFT below. Both then use the same reward, prompt pool and GRPO settings. The initial policy also serves as the KL reference, so the route comparison includes both initialisation and reference-policy changes. Optimising with the verifier as the only source of task supervision. Both routes use GRPO (Shao et al., 2024). For each promptx C in the reinforcement pool, the behaviour policyπ φ old samples a groupy 1 ,...,y G , every rollout is scored by Equation 4, and the group-relative advantage of Equation 5 is assigned to every token of its rollout. Writingr i,t (φ)for the token-level likelihood ratio π φ (y i,t | x C ,y i,<t )/π φ old (y i,t | x C ,y i,<t ), the objective is the PPO clipped surrogate (Schulman et al., 2017) with a reference-model penalty, J (φ) = E h 1 G P G i=1 1 |y i | P |y i | t=1 n min r i,t ˆ A i , clip(r i,t , 1−ε clip , 1+ε clip ) ˆ A i − β b K i,t oi , (6) where the expectation is over promptsC ∼D RL and their sampled groups,|y i |counts completion tokens,ε clip = 0.2is the clipping range, b K i,t = e d −d− 1withd = logπ ref (y i,t |·)− logπ φ (y i,t |·) is the per-token KL estimator of Shao et al. (2024),π φ old is a detached copy ofπ φ (one update per rollout batch), andπ ref is the route’s initial checkpoint (Appendices A and B.2). No value network is trained; the group meanμ C is the baseline, which leaves the verifier as the only source of task-specific supervision in the loop. We runG = 8,β = 0.04, learning rate1×10 −6 ,300optimiser steps on6,000 synthetic prompts, with Qwen3-4B-Instruct (Yang et al., 2025) trained full-parameter and rollouts served by vLLM on a dedicated device (§B.1). The prompt pool upsamples structure-containing problems to33%(against21%unforced), which changes the prompt distribution as well as exposure to form; its control is in §5.5 and the full configuration in Table 5. 5 Published as a conference paper at ICLR 2027 Warm-starting on targets that the verifiers label themselves. The optional SFT uses12,890 instruction–tune pairs self-labelled from the IrishMAN corpus (Wu et al., 2023): for each corpus tune the verifiers read candidate constraint parameters off the parsed score, samplek ∼U1, 2, 3 families, and admit the pair only ifA(y,C(y)) = 1. Held-out family pairs (§4) are excluded from both training pools. Full construction details are in Appendices B.3–B.5. 4 MusicConstraintBench MusicConstraintBench comprises five complementary tests: constraint density (Single, Mixed, High-k), compositional generalisation (Seen, Unseen, Dense), parameter extrapolation (Edge), multi-section control (Structure), and linguistic variation (paraphrases). Single (n = 200) contains one constraint, High-k(n = 160) four, and Mixed (n = 300) spansk ∈ 1, 2, 3, 4and serves as the per-family diagnostic. Seen (n = 200) contains two or three families that co-occur in training, whereas Unseen (n = 200) requires one of four held-out pairs; Structure (n = 120) always includes the cross-section form constraint. Dense and Edge contain500items each (§5.4), giving2,180items in total. The six original suites were consulted during reward development, whereas Dense and Edge were added afterwards. The held-out pairs are (key,rhythm), (meter,leap), (final,structure), and (length,range); none occurs together in either training stage (Lake & Baroni, 2018; Kim & Linzen, 2020). They form a perfect matching over the eight families, so each family is held out once and any held-out-pair-free set has at mostk = 4(Appendix C.4). Finally, re-rendering all Single and Seen items with alternative wording yields a400-item paraphrase suite with unchanged constraint values (Appendices C.8, C, B.6, and C.6). 5 Experiments In this section, we address five questions: Q1, How well existing models follow combinations of musical instructions; Q2, Whether verifier-driven RL can train a general-purpose model beyond them without requiring music-domain SFT; Q3, Whether the gains generalise to excluded combinations, extrapolated parameters, and reworded prompts; Q4, Why the reward design works; and Q5, What the resulting compliance score captures. Sections 5.2–5.6 answer these questions in order. The principal comparisons use the matched runs of Table 2, while additional ablations and diagnostics use a separate diagnostic study (Appendices D–E). 5.1 Experiment Setup We score saved greedy completions using the all-satisfied criterion of §3; repeated prompts retain their sampler weight, supplementary audits use additional criteria where noted, and the sampling-budget study uses temperature1.0. Baselines include Qwen3-4B/8B/32B (Yang et al., 2025), Llama-3.1-70B (Grattafiori et al., 2024), MiniMax-M3, and ChatMusician-7B (Yuan et al., 2024), all given the same user prompt under their own chat templates without requested reasoning; the music-domain SFT checkpoint serves as a supervised baseline (Appendix B.4). The three matched runs, Direct binary (R = A), Direct MusicRLVR, and SFT-initialised MusicRLVR, share seed43,300updates, and 48completions per update. The two Direct arms start from Qwen3-4B-Instruct and differ only in reward mixing weight, while SFT initialisation also changes the fixed KL reference. Final checkpoints are evaluated once per suite under greedy decoding, with decoding variation analysed separately (Appendix B.9) and paired McNemar tests (McNemar, 1947) reported in Appendix D.5. Component ablations, sensitivity analyses, paraphrases, and output analyses use a separate SFT-initialised reference, termed the diagnostic study, unless stated otherwise. 5.2 A1: Existing Models Struggle with Multi-Constraint Music Instructions For the strongest zero-shot models, parsing is largely solved; joint compliance is not. As shown in Table 2, Llama-3.1-70B writes a parseable score for0.970of Mixed items yet satisfies every requested constraint on only0.380, with Qwen3-32B and MiniMax-M3 alongside it at0.363and0.367: the three strongest zero-shot models sit within0.017of one another. Nor does capacity order the field: Qwen3-8B reaches 0.123 on Mixed, below the 0.160 of the smaller Qwen3-4B. 6 Published as a conference paper at ICLR 2027 Table 2: All-satisfied rate on the eight MusicConstraintBench suites (greedy decoding); the three RL rows are the matched runs at equal budget. Each column marks itsbestandrunner-up; ranks are descriptive. High-k and Dense both usek = 4but differ in held-out pairs and family mixture. Counts, parse-gate rates and paired tests: Appendix D.5. Six original splitsLater splits ModelMixed Single Seen Unseen High-k Structure Dense Edge Zero-shot Qwen3-4B0.160 0.225 0.100 0.0400.0370.1000.008 0.010 Qwen3-8B0.123 0.210 0.115 0.0500.0060.0000.020 0.032 Qwen3-32B0.363 0.535 0.250 0.1900.0940.2670.074 0.112 Llama-3.1-70B0.380 0.695 0.300 0.1950.0560.1670.040 0.058 MiniMax-M30.367 0.580 0.315 0.1900.1000.0080.088 0.160 ChatMusician-7B0.160 0.290 0.130 0.0900.0060.0000.020 0.046 Trained (4B) SFT (4B)0.477 0.745 0.415 0.4400.2310.6000.290 0.076 Direct GRPO (R = A)0.7970.9600.7450.5650.4690.9080.3220.300 MusicRLVR (SFT init.)0.5770.8800.5850.5300.3500.3830.3660.134 MusicRLVR (Direct)0.8070.9850.8150.5950.5750.9670.3620.310 Where compliance is lost is visible by constraint count (Figure 1), eachkslice carrying its own prompts and family mixture. Llama-3.1-70B satisfies0.720of single-constraint Mixed items, as many as the music-domain SFT checkpoint, but only0.050of four-constraint ones (Appendix D.11): a14.4×fall, where Direct MusicRLVR falls2.0×. Strength on one instruction at a time therefore says little about honouring several together, which is the regime an explicit specification lives in. Music specialisation does not substitute for it either, ChatMusician-7B reaching merely0.160on Mixed under the shared prompting protocol (Appendix D.2). 5.3 A2: Direct Verifier-Driven RL Teaches Constraint Following Without Music-Domain SFT Direct MusicRLVR raises the Mixed all-satisfied rate from0.160to0.807and surpasses every evaluated zero-shot model on all eight suites —0.985on Single and0.967on Structure (both in-distribution, Appendix B.7),0.575on High-kagainst a best zero-shot of merely0.100(Table 2) — with every completion clearing the parse gate on the six original suites. Music-domain SFT is not a prerequisite: under the same reward, prompt pool and budget, the SFT-initialised route reaches0.577 on Mixed against0.807for Direct and trails on seven of eight suites, Dense excepted at183/500 against181/500; pooled over the2,180items they disagree on767,554in Direct’s favour (p < 10 −4 ), six of the eight per-suite advantages surviving Holm correction (Appendix D.5). From these results we observe that (1) music-domain SFT is not required for the observed gains; (2) the supervised initialisation is the arm that collapses on form, clearing the repetition half of the form verifier on 119/120Structure items but the contrast half on only50, and emitting an all-identical tune on69 (Figure 4(e)); and (3) the deficit manifests during training, where over the last50updates that arm satisfies everything on0.751of its rollouts against0.890for Direct. Initialisation also fixes the KL reference (Figure 4(c,f)). Figure 3 illustrates two verifier-successful compositions meeting the same four constraints with different repeating patterns (Appendices F and D.16). 5.4 A3: The Gain Generalises Beyond Training Combinations and Parameter Ranges Compositional generalisation. Every Unseen item requires one of the four family pairs excluded from both training pools; Direct reaches0.595there against0.815on Seen, where the best zero-shot rates are0.195and0.315. Both four-constraint suites hold up too,0.575on High-kand0.362on Dense, which also requires a held-out pair and differs in family mixture (Appendix B.7). Parameter extrapolation. Edge moves meters to5/4,2/2,9/8,12/8and lengths to4,24,32bars and extends leap, range and rhythm (Table 12), sharing the sampler and verifiers with the standard suites. Direct scores0.310against the strongest zero-shot0.160and the SFT checkpoint’s0.076. Appendix D.3 separately analyses pair-free and held-out-pair subsets for the diagnostic SFT-initialised checkpoint. 7 Published as a conference paper at ICLR 2027 G SFT — all four hold, but the tune is one two-bar cell repeated ✓ meter 3/4 ✓ 8 bars ✓ final = E ✓ 8th + qtr only G MusicRLVR — all four hold, on a three-bar cycle ✓ meter 3/4 ✓ 8 bars ✓ final = E ✓ 8th + qtr only ♯ Prompt: meter 3/4 | exactly 8 bars | last note E | eighth and quarter notes only Figure 3: One high-kitem from the diagnostic study (the checkpoints of Appendix D.6, not the matched rows of Table 2). Both outputs satisfy the four constraints, but SFT repeats one two-bar cell while SFT-initialised MusicRLVR uses a three-bar cycle. Verifier success permits different compositions; the example makes no perceptual-quality comparison. 0100200300 GRPO step 0.4 0.6 0.8 1.0 rollouts satisfying all (a) Training compliance binary SFT init. graded 0100200300 GRPO step 0.00 0.25 0.50 0.75 1.00 fraction of groups (b) Zero-variance groups 0100200300 GRPO step 0.0 0.2 KL to reference (c) Policy movement 0.500.751.00 satisfied when requested length structure key final meter leap range rhythm (d) Per-family binary graded repeat contrast identical 0.0 0.5 1.0 fraction of items (e) Form halves 0100200300 GRPO step 0.1 0.2 0.3 mean length (k tokens) (f) Completion length Figure 4: Matched runs in Table 2. “Binary” and “graded” are the two Direct arms, differing only in the reward weightλ; “SFT init.” uses the graded reward from the music-domain SFT checkpoint. (a) All-satisfied rollout rate. (b) Zero-variance groups. (c) KL to each run’s frozen reference. (d) Per-family satisfaction (Direct arms only). (e) Form-verifier components and all-identical rate. (f) Mean completion length. Linguistic robustness. Rewording each single and seen-combination item while holding its constraint values moves MusicRLVR from0.735to0.688and the SFT checkpoint from0.580to0.535, losses of 19 and 18 out of 400: the margin carries over intact (diagnostic study; Appendix D.1). 5.5 A4: Why the Reward Works: Partial Credit Behind a Validity Gate A binary reward leaves many groups without a learning signal: Equation 5 gives no reward gradient when all rollouts score alike, and over the first50updates this occurs for0.550of binary groups against 0.307with graded credit (Figure 4(b)). Both arms finish at similar rollout compliance (0.890versus 0.884), suggesting that graded ordering improves transfer rather than training-set fit. Across the2,180 benchmark items, graded credit satisfies1265against1188, with229versus152discordant wins (p = 9.39× 10 −5 ). Its largest gains occur on High-k(0.469→ 0.575) and Seen (0.745→ 0.815); it is higher on all eight suites, although the0.010margins on Mixed and Edge lie within the re-evaluation spread (Appendix B.9). With each family scored only on the items requesting it, gains concentrate on length(0.558 → 0.663,p < 10 −4 ) andstructure(0.634 → 0.667), with the other six moving by at most0.019(Figure 4(d); Appendix D.10). In the diagnostic ablations, increasing form exposure from21%to33%raises Structure compliance from0.250to0.525, while graded form credit at the same share further raises it to0.808, with the other five original suites changing by at most0.025 (Table 3; Appendices D.4 and E.3). The matched runs show the same pattern, with0all-identical Structure outputs under graded credit against3under binary credit. Further sensitivity and trainer-log analyses are in Appendices D.8–D.15. 8 Published as a conference paper at ICLR 2027 0100200300 GRPO step 0.00 0.25 0.50 fraction of groups (a) Zero-variance 5e-71e-62e-65e-6 0 .01.04 .1 48 16 0.0 0.5 1.0 all-satisfied rate lr β G (b) Hyper-parameter sensitivity MixedStructure Fixed checkpoint 0.0 0.5 1.0 fraction of 120 (c) Structure split, decomposed repetition contrast all requested all-identical 50150250 GRPO step 0.0 0.5 1.0 per-family satisfaction (d) Learning curve by family 0100200300 GRPO step 0.00 0.01 0.02 KL to reference (e) Drift from the reference 0100200300 GRPO step 110 130 150 completion tokens (f) Completion length keymeterlengthrangefinalrhythmleapstructurebinary form creditgraded (final) Figure 5: Reward-design ablations (diagnostic study). (a) Zero-reward-variance groups. (b) Sensitivity to learning rate,βand group sizeG; dotted lines mark the reference settings. (c) The form verifier decomposed. (d) Per-family satisfaction. (e–f) KL and completion length, binary against graded form credit. Table 3: Reward-design ablations (diagnostic study). Each RL row is one300-step run from the same supervised checkpoint, scored by the binary all-satisfied criterion; the format audit leaves all five unchanged. The two changes separating our final reward from the first attempt each recover about half of the structure gap, and across the first three RL rows the other five splits vary by at most 0.053. ConfigurationMixed Single Seen Unseen High-k Struct. SFT only, no GRPO0.477 0.745 0.415 0.440 0.231 0.600 + GRPO, varying the form-constraint reward and the share of form problems: binary form credit, unforced share (21%)0.610 0.810 0.630 0.635 0.350 0.250 binary form credit, share 33%0.653 0.810 0.610 0.610 0.331 0.525 graded form credit, share 33% (MusicRLVR) 0.663 0.835 0.635 0.595 0.356 0.808 symmetric form credit, single run (App. E.2) 0.590 0.745 0.590 0.630 0.388 0.200 5.6 A5: What the Verifier Changes, and What It Does Not Measure Table 4: Diagnostic output analysis. (a) Sur- face statistics (first 8 bars,297matched items). (b) Best-of-N (T =1.0, n=8, binary). (a) ModelNotes PC ent. Dist. bars Dist. 4-gr. SFT42.7 1.9300.4170.419 MusicRLVR (diag.) 40.5 1.8260.4260.403 (b) SuiteModelp@1 p@2 p@4 p@8 Mixed SFT0.415 0.584 0.728 0.827 MusicRLVR 0.676 0.817 0.900 0.950 Struct. SFT0.692 0.861 0.938 0.967 MusicRLVR 0.844 0.953 0.978 0.983 A compliance score is not a quality score. In the diag- nostic study the SFT checkpoint and SFT-initialised RL write tunes of different lengths (39.2against28.7bars on average), yet on the first eight bars of the297items where both are long enough their surface statistics move little, the largest shift being pitch-class entropy 1.930 → 1.826(Table 4a; Appendices D.12, D.13); perceptual quality would require an independent lis- tening study. What the verifier still leaves unsolved is shallow rather than pervasive: Direct misses58of300 Mixed items, of which48fall one family short and10 two or more, withstructurethe most common sole miss (18); none fails the parse gate (Table 32). Form stands near its ceiling on both halves (Figure 4(e)). The verdicts also serve at decoding time: verifier selection among eight samples raises the diagnostic Mixed rate to 0.950 (Table 4b). 6 Conclusion We frame constraint-following music generation as a property-verifiable generation problem and introduce MusicConstraintBench and MusicRLVR. Graded property-level rewards behind a hard validity gate raise Qwen3-4B-Instruct from0.160to0.807on Mixed without music-domain SFT, with gains extending to unseen property combinations and out-of-range parameters. These results show that deterministic property checks can provide effective task-specific supervision for open-ended structured generation without requiring a target composition. 9 Published as a conference paper at ICLR 2027 AI use statement We have not used generative AI tools for any of the tasks with required disclosure, and the remaining required disclosure tasks are not applicable to this work. Additionally, we used generative AI tools for polishing the writing and improving the readability of the manuscript. We have reviewed all AI-assisted work: all AI-assisted edits were checked against the underlying results by the authors. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI. Ethics statement The study involves no human subjects, no personal data and no user-facing deployment. Supervised targets derive from the publicly available IrishMAN corpus of traditional Irish music (Wu et al., 2023); the reinforcement prompts are produced by deterministic samplers rather than drawn from a corpus. The intended uses are settings that need a score to meet a stated specification, such as music education, game audio and accessibility tools. We report compliance with programmatic checks and make no claim about musical quality; reading a compliance score as a quality score would overstate what the verifiers measure, which is the misuse we consider most likely. Appendix G gives the longer discussion. Reproducibility statement Quantitative tables, macros and figures are regenerated from saved evaluation files and training logs by the scripts of Appendix B.8; the principal training settings are in Appendix B, the verifiers in Appendix C.1, and Appendix B.10 lists the artefacts we will release. References Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325, 2023. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 12248–12267, 2024. Geoffrey Cideron, Sertan Girgin, Mauro Verzetti, Damien Vincent, Matej Kastelic, Zalán Borsos, Brian McWilliams, Victor Ungureanu, Olivier Bachem, Olivier Pietquin, et al. Musicrl: Aligning music generation to human preferences. arXiv preprint arXiv:2402.04229, 2024. Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. Advances in neural information processing systems, 36:47704–47720, 2023. Michael Scott Cuthbert and Christopher Ariza. music21: A toolkit for computer-aided musicology and symbolic music data. 2010. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains. In International Conference on Learning Representations, volume 2026, p. 127924–127945, 2026. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 10 Published as a conference paper at ICLR 2027 Xuzheng He, Nan Nan, Zhilin Wang, Ziyue Kang, Zhuoru Mo, Ao Li, Yu Pan, Xiaobing Li, Feng Yu, and Xiaohong Guan. Symphonygen: 3d hierarchical orchestral generation with controllable harmony skeleton. arXiv preprint arXiv:2604.25498, 2026. Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer. arXiv preprint arXiv:1809.04281, 2018. Yujia Huang, Adishree Ghatare, Yuanzhe Liu, Ziniu Hu, Qinsheng Zhang, Chandramouli S Sastry, Siddharth Gururani, Sageev Oore, and Yisong Yue. Symbolic music generation with non- differentiable rule guided diffusion. arXiv preprint arXiv:2402.14285, 2024. Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato, Richard E Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In International Conference on Machine Learning, p. 1645–1654. PMLR, 2017. Nicolas Jonason, Luca Casini, and Bob LT Sturm. Smart: Tuning a symbolic music generation system with an audio domain aesthetic reward. arXiv preprint arXiv:2504.16839, 2025. Maximos Kaliakatsos-Papakostas, Konstantinos Soiledis, Theodoros Tsamis, Dimos Makris, Vassilis Katsouros, and Emilios Cambouropoulos. Incorporating structure and chord constraints in symbolic transformer-based melodic harmonization. arXiv preprint arXiv:2512.07627, 2025. Najoung Kim and Tal Linzen. Cogs: A compositional generalization challenge based on semantic interpretation. In Proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), p. 9087–9105, 2020. Carol L Krumhansl. Cognitive foundations of musical pitch. Oxford University Press, 2001. Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International conference on machine learning, p. 2873–2882. PMLR, 2018. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025. Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Neurologic decoding:(un) supervised neural text generation with predicate logic constraints. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human language technologies, p. 4288–4299, 2021. Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. General-reasoner: Advancing llm reasoning across all domains. Advances in Neural Information Processing Systems, 38:56596–56618, 2026. Quinn McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157, 1947. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 27730–27744, 2022. Xingwei Qu, Yinghao Ma, Ziya Zhou, Ka Man Lo, Jiaheng Liu, Ruibin Yuan, Lejun Min, Xueling Liu, Tianyu Zhang, Xeron Du, et al. Mupt: A generative symbolic music pretrained transformer. In International Conference on Learning Representations, volume 2025, p. 46753–46779, 2025. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 11 Published as a conference paper at ICLR 2027 Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. Advances in neural information processing systems, 35:9460–9471, 2022. Bob L Sturm, Joao Felipe Santos, Oded Ben-Tal, and Iryna Korshunova. Music transcription modelling and composition using deep learning. arXiv preprint arXiv:1604.08723, 2016. David Temperley. What’s key for key? the krumhansl-schmuckler key-finding algorithm reconsidered. Music Perception, 17(1):65–100, 1999. John Thickstun, David Hall, Chris Donahue, and Percy Liang. Anticipatory music transformer. arXiv preprint arXiv:2306.08620, 2023. Dimitri von Rütte, Luca Biggio, Yannic Kilcher, and Thomas Hofmann. Figaro: Controllable music generation using learned and expert features. In The Eleventh International Conference on Learning Representations, 2023. Yashan Wang, Shangda Wu, Jianhuai Hu, Xingjian Du, Yueqi Peng, Yongxin Huang, Shuai Fan, Xiaobing Li, Feng Yu, and Maosong Sun. Notagen: Advancing musicality in symbolic music generation with large language model training paradigms. arXiv preprint arXiv:2502.18008, 2025a. Zhilin Wang, Zhe Yang, Yun Luo, Yafu Li, Xiaoye Qu, Ziqian Qiao, Haoran Zhang, Runzhe Zhan, Derek F Wong, Jizhe Zhou, et al. Towards an ai musician: Synthesizing sheet music problems for musical reasoning. arXiv preprint arXiv:2509.04059, 2025b. Shangda Wu, Xiaobing Li, Feng Yu, and Maosong Sun. Tunesformer: Forming irish tunes with control codes by bar patching. arXiv preprint arXiv:2301.02884, 2023. Shangda Wu, Yashan Wang, Ruibin Yuan, Guo Zhancheng, Xu Tan, Ge Zhang, Monan Zhou, Jing Chen, Xuefeng Mu, Yuejie Gao, et al. Clamp 2: Multimodal music information retrieval across 101 languages using large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, p. 435–451, 2025. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222–113244, 2026. Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, et al. Chatmusician: Understanding and generating music intrinsically with llm. In Findings of the Association for Computational Linguistics: ACL 2024, p. 6252–6271, 2024. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. Alon Ziv, Sanyuan Chen, Andros Tjandra, Yossi Adi, Wei-Ning Hsu, and Bowen Shi. Mr-flowdpo: Multi-reward direct preference optimization for flow-matching text-to-music generation. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1821–1825. IEEE, 2026. 12 Published as a conference paper at ICLR 2027 Contents of Appendix A Algorithm15 B Experimental Settings16 B.1 Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 B.2 Training Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 B.3 SFT Data Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 B.4 SFT Training Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 B.5 Prompt Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 B.6 Dataset Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 B.7 Prompt Coverage, Training Overlap, and Split Composition . . . . . . . . . . . . . . . . . . . . 19 B.8 Reproducibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 B.9 Evaluation Nondeterminism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 B.10 Data and Code Release . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 C Benchmark and Verifier Details22 C.1 Verifier Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 C.2 Stricter Fixed-Instruction Gate Audit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 C.3 Key Verifier Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 C.4 Held-out Pair Rationale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 C.5 ABC Notation Primer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 C.6 ABC Notation: A Worked Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 C.7 Constraint Rendering Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 C.8 Paraphrase Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 D Additional Experiments28 D.1 Paraphrase and Training Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 D.2 Rejection Sampling and a Music-Specialised Baseline . . . . . . . . . . . . . . . . . . . . . . . 29 D.3 Edge Subsets and a Stricter Key Criterion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 D.4 Significance Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 D.5 Matched Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 D.6 Diagnostic Reference Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 D.7 Fixed-Checkpoint Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 D.8 Hyper-parameter Sweeps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 D.9 Reward Mixing Weight . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 D.10 Per-Family Satisfaction on the Original Suites . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 D.11 Performance by Number of Constraints (k) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 D.12 Output Length Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 D.13 Length-Controlled Output Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 D.14 Zero-Variance Groups During Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 D.15 Additional Training Curves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 D.16 Additional Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 13 Published as a conference paper at ICLR 2027 E Error and Structure Analysis40 E.1 Which Constraint Binds, by Split . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 E.2 Structure Reward Variants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 E.3 Structure Results by Form Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 F Qualitative Examples43 G Broader Impact47 14 Published as a conference paper at ICLR 2027 A Algorithm Algorithm 1 runs verifier-driven GRPO directly or with an optional supervised warm start built from verified corpus-derived labels (θ f (y) := p f (ρ(y)), §3.4). Unless marked Direct, the diagnostics below describe the SFT-initialised route, retaining a common initialisation for the reward-design comparisons. Algorithm 1: MusicRLVR: verifier-only GRPO with optional self-labelled SFT Input:initial policyπ φ 0 ; optional SFT flaguand corpus tunesY; familiesFwith verifiersv f and parser ρ; held-out pairs H; held-out retries N; reinforcement poolD RL (no prompt contains a pair in H; structure share 33%); reward weight λ; group size G; β, ε clip , ε; steps T; learning-rate schedule η s Output: trained policy π φ π ref ← π φ 0 if u is enabled then // Optional self-labelled supervised fine-tuning (§3.4) D ←∅ foreach y ∈Y do C(y)← null; draw k ∼U1, 2, 3 for attempt = 1,...,N do sample F ⊆F y (families measurable on y),|F| = min(k,|F y |) C ←(f,θ f (y)) : f ∈ F with θ f (y) = p f (ρ(y)) read off ρ(y) if C contains no pair in H then C(y)← C; break end if C(y) =null then C(y)← one constraint drawn from y’s properties // singleton: trivially held-out-free end if A(y,C(y)) = 1 then D ←D∪(x C(y) ,y)// else this draw of y is discarded end // with prob. 0.2: a structure-only pair from y’s first two 4-bar units, re-verified the same way end π ref ← fine-tune π φ 0 onD end π φ ← π ref // Verifier-driven GRPO (§3.4) for step s = 1,...,T do φ old ← φ (detached) sample a batch of prompts x C , C ∼D RL foreach x C in the batch do sample a groupy 1 ,...,y G ∼ π φ old (·| x C ) score every y i by R(y i ,C) (Equation 4) ˆ A i ← R(y i ,C)− μ C /(σ C + ε) for every i (Equation 5) // ˆ A i = 0 for every i if σ C = 0 end accumulate∇ φ (−J (φ)) over the batch, clip its norm to 1.0, AdamW step at η s // Equation 6 end 15 Published as a conference paper at ICLR 2027 B Experimental Settings B.1 Infrastructure Training uses ms-swift’s GRPO implementation with a custom reward plugin that shells out to the evaluation verifiers, so training and benchmarking share onemusic21build. On these accelerators, co-locating the vLLM rollout engine with the training state is infeasible for full-parameter4B training: the sleeping engine’s memory pool still counts against the training process and the backward pass fails at a fixed allocation point regardless of batch size. Serving rollouts from a dedicated device solves it. Two further settings are required on this stack: disabling the NZ tensor layout (it corrupts weight synchronisation during RL) and capping the rollout engine’s context length, which otherwise reserves key-value cache for the model’s full 262,144-token window. B.2 Training Hyperparameters Table 5 preserves the optional SFT and seed-42diagnostic GRPO settings. The matched runs use two policy devices with accumulation12and one rollout device (memory fraction0.92), keeping48 completions per update; see Appendix D.5. The saved trainer configurations carry the remaining optimiser defaults. Table 5: Optional SFT and seed-42diagnostic GRPO hyperparameters. TP: tensor-parallel degree; mem. frac.: the rollout engine’s memory fraction as recorded in the saved trainer configuration. Optional SFTGRPO Base modelQwen3-4B-Instruct Parameters trainedFullFull Training data12,890 pairs6,000 prompts OptimiserAdamWAdamW Learning rate1× 10 −5 1× 10 −6 LR scheduleCosineCosine Warmup (fraction of steps)3%2% Batch size (per device× devices)4× 3 pairs2× 3 completions Gradient accumulation48 Effective batch per optimiser step48 pairs48 completions (6 groups) Epochs / steps2 epochs (538 steps)300 steps Max sequence length15361536 (completion≤ 768) Group size (G)–8 KL coefficient (β)–0.04 Clipping range (ε clip )–0.2 Rollout temperature–1.0 λ (Equation 4)–0.7 Structure share in data16.5%33.1% (upsampled from 21%) Precisionbf16bf16 Devices33 train + 1 rollout Rollout engine–vLLM (TP = 1, mem. frac. 0.9) Frameworkms-swift 4.2.3ms-swift 4.2.3 Wall time∼45 min∼3 hours Figure 6 shows the supervised stage; the reported run uses the fixed two-epoch budget, without testing whether a longer supervised run would improve downstream compliance. Two diagnostics from the GRPO logs are worth recording alongside it. The clipped fraction of Equation 6 is identically zero at all300steps, as expected when one optimiser update follows each rollout batch, so the clipping range never binds. In the default diagnostic GRPO run no completion reaches the768-token cap: the longest sampled completion over training is 492 tokens, and the trainer’s truncation counter stays at zero. 16 Published as a conference paper at ICLR 2027 0.00.51.01.52.0 epoch 1.0 1.5 2.0 2.5 3.0 cross-entropy 0.913 (a) Loss training validation 0.00.51.01.52.0 epoch 0.55 0.60 0.65 0.70 0.75 0.80 next-token accuracy 0.748 (b) Token accuracy Figure 6: The supervised stage. Cross-entropy (a) and next-token accuracy (b) over the two SFT epochs, for the12,890training pairs and the125-pair validation split. Validation loss decreases to0.913and validation accuracy plateaus at 0.748. B.3 SFT Data Construction The supervised training data is built from the IrishMAN corpus (Wu et al., 2023), a collection of ∼45,000 traditional Irish tunes in ABC notation. The construction pipeline has four steps: 1.Filter and parse. Tunes whose body contains the builder’s list of unsupported tokens, or that have fewer than4or more than32written bars, are discarded; the survivors are parsed with music21 and kept only if the parsed bar count agrees with the written one. 2. Property extraction. The same verifiers used for evaluation extract the true properties of each tune: its key (hybrid policy), meter, bar count, pitch range, final note, the set of distinct note durations, and the largest melodic interval. 3.Instruction rendering. A random subset of the extracted properties is selected (withksampled uniformly from1, 2, 3, capped by the number of measurable properties) and rendered into a constraint instruction using the same template as evaluation. The tune itself is the target output. 4.Verification gate. Every (instruction, tune) pair is re-verified: the tune is checked against the rendered constraints using the evaluation verifiers. Any pair that fails verification is discarded. This ensures that no training example teaches the model to produce output that violates its own constraints. Form examples (structurefamily) are constructed separately: for a tune with at least eight bars whose first two four-bar units differ, the two units are arranged into a sampled AABA/AABB/ABAB shape (attempted with probability0.2per tune; the resulting pair carries the structure constraint only). The resulting12,890pairs form the training set, with a further125pairs held out for validation. Importantly, the four held-out family pairs (§4) are enforced during construction: any pair whose constraints contain a held-out combination is regenerated with a different family subset. B.4 SFT Training Examples We provide three representative (instruction, output) pairs from the SFT training set to illustrate the data quality and the range of constraint combinations. SFT example 1: single constraint (meter). Instruction: Write a single monophonic tune in ABC notation . . . The tune must satisfy ALL of the following constraints: – the meter must be 6/8, and every bar must contain exactly the right total duration. Target (from IrishMAN): X:89124 T:Tune L:1/8 M:6/8 K:G 17 Published as a conference paper at ICLR 2027 GBd gdB | GBd cBA | GBd gdB | cBA edc | GBd gdB | GBd cBA | cBA dcB | cAF G3 | DFA cBA | GAB d2 d | DFA cBA | BAG AGE | DFA cBA | GAB d2 d | cBc dcB | cAF G3 | Every one of the16bars contains exactly six eighth-note units, and the tune carries no repeat sign, so it passes the gate and the meter verifier. SFT example 2: two constraints (length+key). Instruction: . . . – the tune must be exactly 32 bars long (no pickup bar). – the tune must be in D major. Target: X:106003 T:Tune L:1/8 Q:1/8=180 M:6/8 K:D DFA cdc | AFA BGE | DFA dcd | AFA gfe | DFA cdc | AFA BGE | afd bge | AFD E2 D | DFA cdc | AFA BGE | DFA dcd | AFA gfe | DFA cdc | AFA BGE | afd bge | AFD E2 D | fef gfg | afd AFD | fef gfg | DFA B2 A | fdf gfg | afd bge | fga Bcd | AFD E2 D | fef gfg | afd AFD | fef gfg | DFA B2 A | fdf gfg | afd bge | fga Bcd | AFD E2 D | The header readsK:Dand content analysis returns D major; exactly32bars. The constraint parameters were read off this tune’s own properties, and the pair was re-verified before admission. SFT example 3: structure constraint (structure, AABA). Instruction: . . . – the tune must be exactly 16 bars with an AABA form in 4-bar units: labeled units with the same letter must be note-for-note identical, and units with different letters must differ. Target (assembled from the source tune’s first two 4-bar units): X:141183 T:Tune L:1/4 M:3/4 K:Bmin B2 c | d3 | d2 a | f e d | B2 c | d3 | d2 a | f e d | c3 | e3 | f2 B | B3 | B2 c | d3 | d2 a | f e d | Units 1, 2, and 4 are identical (A sections, bars 1–4, 5–8, 13–16); unit 3 (bars 9–12) differs, satisfying the AABA form. The two units are the source tune’s first eight bars; every assembled pair is verified before entering the dataset. B.5 Prompt Template Every benchmark item is rendered into a prompt with the following fixed template. The shared instruction prefix (the first paragraph) is part of the user message of every item; the constraint block is populated from the item’s constraint list. No chain-of-thought, few-shot examples, or reasoning instructions are given. Write a single monophonic tune in ABC notation. Rules: write every bar out explicitly (no repeat signs like |: or :|), one voice only, no chords, no lyrics, no grace notes. Start with an X: header, include T:, M:, L:, K: headers, then the music. Output ONLY the ABC notation and nothing else. The tune must satisfy ALL of the following constraints: – ⟨constraint 1 in natural language⟩ – ⟨constraint 2 in natural language⟩ . . . 18 Published as a conference paper at ICLR 2027 The constraint text for each family is generated by a deterministic renderer that maps parameters to a fixed English sentence. For example, thekeyfamily withθ = (D, major)renders as “the tune must be in D major”; thestructurefamily withθ = (AABA, 4)renders as “the tune must be exactly 16 bars with an AABA form in 4-bar units: labeled units with the same letter must be note-for-note identical, and units with different letters must differ”. The paraphrase suite (§4) replaces these renderers with alternative phrasings while preserving the constraint values. B.6 Dataset Statistics Benchmark splits. Table 6 summarises the composition of the six original benchmark splits. The mixed suite spansk ∈ 1, 2, 3, 4and covers all eight families; the structure split is the only one where every item includes the cross-section formstructurefamily. Family frequencies depend on the sampler and exclusion rules (Table 10); the structure split contains its namesake family in every item and one of five companion families in 66 of 120. Table 6: Composition of MusicConstraintBench splits: number of items, distribution overk(number of simultaneous constraints), and family coverage, as ranges where shown. Splitn k distributionFamilies (count) Mixed300 1:100, 2:100, 3:60, 4:40 All eight Single200 1:200All eight (16–29 each) Seen comb.200 2:116, 3:84All eight (51–72 each) Unseen comb. 200 2:88, 3:112All eight (54–70 each) High-k160 4:160All eight (56–104 each) Structure120 1:54, 2:66structure: 120; 5 others Training data. The supervised stage uses12,890instruction–tune pairs self-labelled from the IrishMAN corpus. The reinforcement stage uses6,000synthetic prompts withk ∈ 1, 2, 3 (2,046/3,060/894respectively). After the reward-design iteration described in §5.5, the share of prompts containing thestructurefamily was raised to33.1%(1,983/6,000) to increase exposure to form constraints; every other family appears in 1,068–1,347 prompts. B.7 Prompt Coverage, Training Overlap, and Split Composition The weighting and fixed-kcomparisons below use the diagnostic SFT-initialised reference; the suite definitions and prompt counts apply to both studies. The suites contain sampled items, not2,180distinct instructions: there are1,623distinct prompt strings across all eight splits. Table 7 compares each prompt with the user messages in both training datasets. “Set overlap” ignores constraint order and compares family–parameter sets instead. All Single and Structure prompts occur in training, so their scores measure performance on familiar instructions. Unseen, High-kand Dense have no complete prompt or constraint-set overlap. Both training stages also have zero occurrences of the four explicitly held-out pairs. Exclusion concerns requested families; it does not imply that training tunes never possess those properties jointly. Table 7: Coverage and overlap with either training stage. Overlap columns count sampled test rows, while “unique prompts” counts distinct strings. SplitItems Unique prompts Exact overlap Set overlap Mixed300222201203 Single20039200200 Seen200177147151 Unseen20015900 High-k16015600 Structure 12050120120 Dense50049000 Edge5004373239 Equal weighting of distinct prompts. Table 8 first averages repeated rows for each prompt within one evaluation, then averages over distinct prompts. The RL column uses the same fixed 19 Published as a conference paper at ICLR 2027 checkpoint and evaluation files as Table 33. The advantage over SFT persists on all eight splits under this weighting. On Structure, whose120items repeat50prompts, the rates are0.637and 0.522, respectively. Item-paired tests in the main analysis retain the original sampler weights, so they measure the sampled suites rather than2,180distinct instructions. Table 9 applies the same weighting to the three matched checkpoints of Table 2, read from the same saved per-item files. Table 8: Original item weighting and equal weighting of unique prompts, using main-table criterion and the SFT-initialised checkpoint. SplitSFT, items SFT, unique RL, items RL, unique Mixed0.4770.4020.6630.615 Single0.7450.7050.8350.820 Seen0.4150.4070.6350.618 Unseen0.4400.3960.5950.547 High-k0.2310.2370.3560.359 Structure0.6000.5220.8080.637 Dense0.2900.2900.3700.371 Edge0.0760.0780.1620.164 Table 9: The same weighting applied to the three matched checkpoints of Table 2: “items” repeats the sampler weights of Table 2, “unique” averages within each distinct prompt first. The splits whose prompts repeat most, Single (39distinct prompts) and Structure (50), move most; the ordering between the three checkpoints is unchanged on every split but Edge, where the two Direct arms swap by 0.008. Direct GRPO (R = A) MusicRLVR (SFT init.) MusicRLVR (Direct) Splititemsuniqueitemsuniqueitemsunique Mixed 0.7970.7410.5770.5270.8070.760 Single 0.9600.9490.8800.8750.9850.981 Seen0.7450.7290.5850.5670.8150.814 Unseen 0.5650.5090.5300.4790.5950.547 High-k 0.4690.4580.3500.3530.5750.571 Structure 0.9080.8600.3830.4570.9670.940 Dense0.3220.3220.3660.3630.3620.358 Edge0.3000.2910.1340.1350.3100.283 Fixedkdoes not fix family composition. Table 10 gives family frequencies in High-k, Dense and Edge. The first two sharek = 4but are independently sampled and have different family mixtures. Edge additionally mixes pair-free and held-out items (171/500contain a held-out pair). Appendix D.3 scores the pair-free and held-out-pair halves of Edge separately. Table 11 bootstraps the Dense-minus-High-kdifference for each model, separating item-sampling variation from this composition difference. Table 10: Number and percentage of items requesting each family. Percentages sum to more than100%because items contain multiple families. FamilyHigh-k (n = 160) Dense (n = 500) Edge (n = 500) key76 (47.5%)288 (57.6%)168 (33.6%) meter82 (51.2%)282 (56.4%)160 (32.0%) length60 (37.5%)209 (41.8%)127 (25.4%) range100 (62.5%)236 (47.2%)164 (32.8%) final104 (65.0%)234 (46.8%)180 (36.0%) rhythm84 (52.5%)279 (55.8%)152 (30.4%) leap78 (48.8%)270 (54.0%)157 (31.4%) structure56 (35.0%)202 (40.4%)148 (29.6%) All three intervals include zero. Relative changes such as−6%and−29%are descriptive point estimates, particularly sensitive to the small Llama-70B success counts. Table 10 gives the family composition the two splits differ in. 20 Published as a conference paper at ICLR 2027 Table 11: Dense minus High-k: absolute all-satisfied rate differences with percentile intervals from20,000 within-split item bootstrap replicates (seed20260909). They quantify item-sampling variation for these fixed checkpoints, not training or serving variation, and do not adjust for the different family compositions. ModelHigh-k Dense∆95% interval SFT0.231 0.290+0.059[−0.019, +0.132] MusicRLVR (re-evaluated) 0.394 0.370 −0.024[−0.111, +0.062] Llama-70B0.056 0.040 −0.016[−0.057, +0.021] B.8 Reproducibility make_assets.py regenerates the original quantitative tables and numerical macros; make_audit_assets.pyadds the overlap, sensitivity and gate-audit tables; src/rescore_subsets.pyproduces the Edge-subset and stricter-key tables. The histori- cal reporting pass,make_checkpoint_assets.py –sync, computes reference-table, by-k, Edge-subset and fixed-kcomparisons from saved verdicts, checks ablation equivalence under the format audit, and supplies the fixed-checkpoint decompositions; it also supplies the corre- sponding historical figure panels. The matched tables, macros and by-kfigure are generated by make_matched_s43_assets.pyafter all three runs finish. This script checks saved per-item IDs, success counts and provenance hashes for all24evaluations, then renders the two paired comparisons with eight-suite Holm correction. Figure scripts read the same saved reports or training logs. The family-definition and rendering tables are maintained descriptions of the sampler. The six original suites were used during reward development; Dense and Edge were added afterwards, without that exposure. All training and local evaluation ran on Ascend 910B2 accelerators (64GB HBM each, CANN8.2.RC1) with vLLM0.11.0throughvllm-ascend; the seed-42diagnostic GRPO runs used three policy accelerators and one rollout accelerator. The matched study used two policy accelerators and one rollout accelerator. Local models were served at context4,096and generated at most1,200 tokens per item, greedy. The hosted baselines were queried at temperature0through the NVIDIA NIM API (integrate.api.nvidia.com), MiniMax-M3 asminimaxai/minimax-m3in August 2026, with per-model completion budgets of800tokens for Llama-3.1-70B,4,000for MiniMax-M3 and2,048of context for ChatMusician-7B; budgets and timeouts per model are in the release scripts. The release includes per-item completions, original and strict gate decisions, aggregate reports, audit scripts, and training logs. Historical reports remain unchanged;src/verifiers_legacy.py reproduces the training and main evaluation metric, whilesrc/verifiers.pyimplements the stricter gate. The reward plugin pins the historical gate unless strict-gate training is explicitly selected. src/run_verify.py defaults to this same A criterion (legacy mode). Table 12: Parameter values of the Edge split against the standard suites. Families are sampled by the same generator; only the value sets differ.key,finalandstructuretake the same values in both, so they are omitted. Family Standard suitesEdge meter2/4, 3/4, 4/4, 6/85/4, 2/2, 9/8, 12/8 length8, 12, 16 bars4, 24, 32 bars leap ≤ 7, 9, 12 semitones ≤ 2, 3, 4 semitones rhythm three- and two-duration sets 16th, 8th, quarter, half, quarter, half range mixed spansone-octave spans (C4–C5, D4–D5, E4–E5, G3–G4, A3–A4) B.9 Evaluation Nondeterminism Compliance evaluation uses temperature0, yet vLLM’s batched decoding is not bitwise repro- ducible. What we measured is the outcome, not the cause: evaluating the diagnostic checkpoint (outputs/grpo_4b_v2/checkpoint-300) twice, on 08-22 and 08-24, with identical serving flags (Table 13), none of the160high-kcompletions was byte-identical across the two runs. A plausible mechanism is that the ordering of floating-point reductions depends on batch composition, and that the first generated token (the arbitraryX:reference number) sits on a near-tie between many candidates, so a small numerical difference there can send the whole continuation down a different path. The reported spread is measured directly (Table 13) and does not rest on this account. 21 Published as a conference paper at ICLR 2027 Table 13: The same checkpoint evaluated twice under greedy decoding. “Flipped” counts items whose binary all-satisfied outcome differed between the two. Both have parse rate 1.000. Splitn First SecondFlipped0→11→0 Mixed300 0.663 0.703 52 (17.3%) 3220 High-k160 0.356 0.394 32 (20.0%) 1913 Structure 120 0.808 0.758 26 (21.7%) 1016 The two evaluations differ by0.040on Mixed and0.050on Structure. These are observed changes for the same checkpoint under the stated serving configuration; two evaluations do not establish a universal noise bound. The first evaluation supplies the historical reference and diagnostic values except Dense and Edge, which come from the re-evaluation, and Figure 5(d), which uses a separate full-retention run. The fixed-kcomparison uses the re-evaluated High-kscore (0.394 → 0.370). Serving one request at a time is a possible way to reduce batch-dependent numerical variation, but determinism must be verified for the specific hardware and serving stack; we do not claim a universal noise bound. Decoding variation enters each arm as item-level discordance that is close to symmetric, which moves the McNemar statistic towards its null rather than away from it; with respect to this source of variation the pooled comparisons of Appendix D.5 are therefore conservative. B.10 Data and Code Release We will release the code and synthetic benchmark suites under an MIT licence, with corpus-derived data and model artefacts subject to their upstream terms: •MusicConstraintBench benchmark suites: eight frozen JSONL files containing all2,180 benchmark items plus the400-item paraphrase suite. Each item includes the prompt and the constraint specifications; the generation seed is a parameter of the suite sampler rather than a per-row field. •Verifier code: the eight constraint verifiers, the ABC parser, the reward function, the historical verifier, the corrected strict gate, and both sets of scoring reports. • SFT and RL data: the 12,890 supervised training pairs and 6,000 RL prompts. •Model outputs: per-item completions and per-item verifier decisions (with all sub-signals) for the fixed MusicRLVR checkpoint, the SFT checkpoint, all zero-shot baselines, and the diagnostic configurations. •Evaluation reports: aggregate reports, significance tests, quality statistics, prompt-overlap and unique-prompt analyses, strict-gate audits, and the training logs from which the reward curves are drawn. • Asset generation scripts:make_assets.pyfor the initial tables,make_audit_assets.pyfor supplementary analyses, the final reporting passmake_checkpoint_assets.py –sync, and onemake_fig_*.pyper figure, all reading saved reports, so every number in the paper is traceable to a specific evaluation output. The IrishMAN corpus (Wu et al., 2023) used for SFT training is publicly available; source-specific rights and attribution should be preserved for corpus-derived targets. Model distribution follows the licence of the exact Qwen3-4B-Instruct checkpoint (Yang et al., 2025). No human subjects were involved in this research; all evaluation is automated. C Benchmark and Verifier Details C.1 Verifier Implementation Details The historical gateρ, used in training and all tables labelledA, runs first. It extracts the ABC block from the completion and rejects the output if extraction fails, if the text contains|:or:|, ifmusic21 raises or exceeds a20-second parse timeout, or if the selected first part has fewer than eight notes, fewer than four non-empty measures, or any chord object. It does not re-check every formatting instruction printed in the prompt. Outputs that clear the gate are then scored by the family verifiers. 22 Published as a conference paper at ICLR 2027 Because these checks read the selected first part, extra voices, wrappers and some missing headers clear this gate. Appendix C.2 reports the corrected checks on the same saved completions. Each of the eight constraint families is implemented as a deterministic function that takes amusic21 Scoreobject and the constraint parameters, returning a binary verdict. We describe the non-trivial implementation choices below. key. As discussed in §3.2, the verifier uses a hybrid policy: theK:header must match the request, and key-profile analysis of the note content (score.analyze("key"), which inmusic21 10.5.0 selects the Aarden–Essen profiles) must return the requested key or its relative major/minor. On a calibration set of400IrishMAN tunes, this hybrid policy achieves0.890agreement with the corpus key annotation, compared to0.750for content analysis alone and1.000(trivially) for header-only checking. meter . The verifier checks two conditions: (1) the time signature in theM:header matches the request, and (2) every non-empty bar’s total duration (notes and rests, in quarter-note equivalents) equals that signature’s bar length within 10 −4 . Incomplete initial or final bars receive no tolerance. length . The number of non-empty bars is counted after parsing and compared with the request. The prompt’s “no pickup bar” is enforced here as well: the first and last written bar must each carry a full measure of the tune’s own time signature, so an output that opens with an anacrusis and closes with its complement fails even when no meter constraint is requested. range. Every sounding pitch is compared against the specified minimum and maximum (as MIDI numbers). Rests are ignored. final . The pitch class of the last sounding note is compared against the request; octave is always ignored. rhythm . Every note’s duration in quarter-note units, rounded to four decimals, must belong to the admissible set, and any rest fails the check. A dotted or tied value therefore passes only if its parsed duration is itself admissible. leap . Every pair of consecutive sounding pitches is examined; the verifier returns0if any interval exceeds the stated maximum (in semitones). structure. A tune with other than sixteen non-empty bars fails outright; otherwise it is split into four units of four bars each. Each bar is reduced to its signature (event type, MIDI pitch and duration rounded to four decimals, rests included). The minimal pair sets are AABA:E =(1, 2), (1, 4),D =(1, 3); AABB:E = (1, 2), (3, 4),D = (1, 3); ABAB:E = (1, 3), (2, 4),D = (1, 2). For each pair inE(Equation 2) the verifier compares the bar signatures position by position; for each pair inD it checks that at least one position differs. The graded training signal ̄eis the fraction of bar positions that match across all pairs in E; the binary evaluation signal requires all to match. C.2 Stricter Fixed-Instruction Gate Audit This audit covers the diagnostic checkpoints of Table 33 and the three matched RL checkpoints of Table 2. The historical gate can accept a second voice because most checks inspect only the first part, whereas key analysis inspects the whole score. It also does not reject every wrapper, missing header, lyric or grace note. We retain that verifier asverifiers_legacy.pyto reproduce the saved training runs, and implement a stricter gate inverifiers.py. It requires a single tune with the specified headers, no unconsumed wrapper text, no repeat notation, lyrics or grace notes, and no multiple parts, multiple voices, chord objects or overlapping notes. Whitespace is normalised when detecting wrappers. The eight family checks remain unchanged. This hardens the fixed-instruction checks; conformance to the full ABC standard and musical correctness are outside what it verifies. 23 Published as a conference paper at ICLR 2027 The stricter metric is A strict (y,C) = A(y,C) I[g strict (y)]. audit_strict_gate.pyevaluates the additional gate on saved completions; no regeneration or retraining is involved. For the configurations within this manuscript’s scope, the audit covers175 evaluation files and43920rows: the three matched checkpoints of Table 2 on all eight suites, plus the baseline, training-configuration, ablation, sweep, paraphrase and retained dynamics checkpoints. All file-level outcomes, rejection reasons and source hashes are saved separately. Tables 14–15 show the original six suites; Table 16 adds Dense and Edge. Across the audit subset,201previously successful rows fail the stricter gate, and none of them belongs to the three matched RL rows of Table 2: those three checkpoints incur no additional gate rejection and lose no success on any of the6,540items they were scored on, so every rate in those three RL rows is unchanged underA strict . The main comparison and reward curves useA; this audit measures the additional fixed-format requirements, which are not included in the main score. Table 14: All-satisfied rate under the stricter fixed-instruction gateA strict on the original-suite completions. The last three rows are the matched checkpoints of Table 2; every one of their rates equals the correspondingArate. ModelMixed Single Seen Unseen High-k Structure Base 4B0.160 0.225 0.100 0.0400.0370.100 Qwen3-8B0.120 0.210 0.115 0.0500.0060.000 Qwen3-32B0.357 0.535 0.240 0.1900.0940.267 Llama-70B0.377 0.695 0.300 0.1900.0500.167 MiniMax-M30.367 0.580 0.315 0.1900.1000.008 ChatMusician0.000 0.000 0.000 0.0000.0000.000 SFT0.477 0.745 0.415 0.4400.2310.600 MusicRLVR (diagnostic reference) 0.663 0.835 0.635 0.595 0.356 0.808 Direct GRPO (R = A)0.797 0.960 0.745 0.5650.4690.908 MusicRLVR (Direct)0.807 0.985 0.815 0.5950.5750.967 MusicRLVR (SFT init.)0.577 0.880 0.585 0.5300.3500.383 Table 15: Additional gate rejections across the six original suites (1,180rows per model). Lost successes previously hadA = 1and now haveA strict = 0; an extra rejection need not change success if a family already failed. Reasons list only the first detected violation per output. ModelNew rejects Lost successes Leading reasons Base 4B70 non abc wrapper or unconsumed text: 7 Qwen3-8B61non abc wrapper or unconsumed text: 5, grace notes forbid- den: 1 Qwen3-32B224 non abc wrapper or unconsumed text: 19, repeat signs forbidden: 2 Llama-70B263 non abc wrapper or unconsumed text: 26 MiniMax-M310 non abc wrapper or unconsumed text: 1 ChatMusician749151 missing required header: 734, non abc wrapper or uncon- sumed text: 15 SFT00 – MusicRLVR (diagnostic reference)00 – Direct GRPO (R = A)00 – MusicRLVR (Direct)00 – MusicRLVR (SFT init.)00 – Table 16: Historical and strict all-satisfied rates on the two additional suites, using the same saved completions. ModelDense A Dense A strict Edge A Edge A strict Base 4B0.0080.0080.0100.010 Qwen3-8B0.0200.0200.0320.032 Qwen3-32B0.0740.0740.1120.102 Llama-70B0.0400.0380.0580.056 MiniMax-M30.0880.0880.1600.160 ChatMusician0.0200.0000.0460.000 SFT0.2900.2900.0760.076 MusicRLVR (diagnostic, re-evaluated) 0.3700.3700.1620.162 Direct GRPO (R = A)0.3220.3220.3000.300 MusicRLVR (Direct)0.3620.3620.3100.310 MusicRLVR (SFT init.)0.3660.3660.1340.134 24 Published as a conference paper at ICLR 2027 The SFT and fixed SFT-to-GRPO checkpoints incur no additional gate rejections on any of the eight suites. ChatMusician’s strict success rates are zero because its otherwise accepted outputs omit required headers or include wrappers. This audit therefore changes some baseline scores; a strict-format failure is a formatting verdict. The corrected gate has regression checks for valid monophonic output and for multiple voices, chords, grace notes, lyrics, missing title, prose wrappers and repeat signs. Historical training uses the explicitly frozen gate; any future strict-gate training should be reported as a new experiment. C.3 Key Verifier Calibration The hybrid key verifier (§3.2) is the most complex verifier in our suite, involving both header matching and content-based key analysis. We provide the full calibration data here. Calibration dataset. We sampled400tunes from the IrishMAN corpus at random (seeded), each with a key annotation in theK:header. We then ran four key-verification strategies and measured their agreement with that annotation (Table 17): Table 17: Key verifier calibration on400IrishMAN tunes. Agreement is the fraction of tunes where the verifier’s answer matches the corpus key annotation. StrategyAgreement Header only (trivially gameable)1.000 Content analysis only (Aarden–Essen profiles)0.750 Content analysis with relative-key tolerance0.890 Hybrid: header AND content∈ key, relative0.890 The reference here is the corpusK:annotation, which the header-only policy reproduces by construction. The hybrid policy matches the content-with-tolerance policy on this set; its advantage appears when the model might game the header, since a policy that printsK:Gabove music in C major passes the header-only check but fails the hybrid one. The remaining0.110is disagreement between the analysis routine and that annotation, which modal tunes, extensive chromaticism and very short tunes can each move. The same policy applies to every model we score. This is agreement with source metadata, not validation against independent positive and negative examples of musical key. The verifier gives an operational target and explicitly tolerates relative major/minor; determinism does not make it an error-free musicological ground truth. Effect on reported numbers. The key family’s per-constraint satisfaction rate (Table 38) is0.862 on the single split and0.671on the high-ksplit. If we used a header-only verifier, these would be higher (the model rarely writes the wrongK:header), but header-only acceptance does not inspect the notes. Content analysis with relative-key tolerance carries the0.110disagreement above; exact content-only matching disagrees on0.250of the calibration tunes. We adopt the hybrid policy and log all sub-signals, so future work can re-score under any policy. C.4 Held-out Pair Rationale The four held-out pairs (key,rhythm), (meter,leap), (final,structure) and (length,range) were chosen to form a perfect matching on the eight families, ensuring that (1) every family participates in exactly one held-out pair, (2) no family appears in multiple exclusions, without implying equal sampled frequency or equal difficulty, and (3) the matching constrains the maximum testablekto4 without touching any held-out pair, which naturally defines the high-k split. The specific pairing was selected to maximise the diversity of inter-family interaction types: • key –rhythm: one harmonic, one temporal, with no obvious interaction (a tune can be in any key with any rhythm vocabulary). • meter –leap: one temporal, one melodic; the meter determines bar duration but should not affect interval size. 25 Published as a conference paper at ICLR 2027 • final–structure: one local (last note), one global (16-bar form); the most interesting test of whether a model can coordinate endpoint and large-scale structure without having seen them together. • length–range: both are “bounding” constraints (number of bars, pitch boundaries) that interact primarily through the amount of melodic material available. The supervised dataset contains zero instances of any held-out pair, verified by an exhaustive check over all 12,890 pairs before training. C.5 ABC Notation Primer ABC notation is a plain-text format for monophonic music, widely used in folk-music archives. A minimal valid tune contains a header block and a body: X:1% reference number T:My Tune% title M:4/4% meter (time signature) L:1/8% default note length K:G% key signature G2 BG dGBG | G2 Bd cAFA | G2 BG dGBG | cABG AGFG | Key conventions: note namesC D E F G A Bdenote the octave starting at middle C; lowercasec d e f g a bdenote one octave higher; trailing apostrophes (e.g.c’) raise further. A digit after a note name multiplies its duration by that factor relative to the default lengthL:. ThusG2in a tune with L:1/8 is a quarter note on G. The pipe character | separates bars. Because ABC is plain text, a language model generates it token by token in exactly the way it generates code or prose. Themusic21library (Cuthbert & Ariza, 2010) parses ABC into a rich internal representation (pitch, duration, key analysis, interval computation) that our verifiers operate on. If parsing fails, for instance because the model outputs prose instead of notation or uses an invalid header like L:8 instead of L:1/8, the gate returns ρ(y) =⊥, so A(y,C) = 0 (Equation 1). C.6 ABC Notation: A Worked Example To help readers unfamiliar with ABC notation, we provide a token-by-token walkthrough of a model-generated output that passes all four constraints of a high-k problem. Problem. Meter 3/4; exactly 8 bars; last note E; eighth and quarter notes only. Model output: X:17011 T:Tune L:1/8 M:3/4 K:Edor B2 B2 AG | E2 E2 DE | F2 F2 ED | B2 B2 AG | E2 E2 DE | F2 F2 ED | B2 B2 AG | E2 E2 DE | Token-by-token analysis: • X:17011: Reference number (arbitrary, used by databases). • T:Tune: Title (arbitrary, not constrained). • L:1/8 : Default note length is an eighth note. This meansBalone is an eighth note,B2is two eighth notes = one quarter note. • M:3/4: Time signature: three quarter notes per bar. SinceL:1/8, each bar must contain six eighth-note equivalents.✓ meter. • K:Edor : Key of E Dorian (E minor with raised sixth). The key constraint is not active in this problem, so any key is acceptable. 26 Published as a conference paper at ICLR 2027 • B2 B2 AG: Bar 1.B2= quarter note B (two eighths);B2= quarter note B;A= eighth note A;G = eighth note G. Total:2 + 2 + 1 + 1 = 6eighth-note equivalents.✓meter. Durations used: quarter, eighth.✓ rhythm. • |: Bar line separator. • E2 E2 DE: Bar 2. Same structure. Total 6.✓ • . . . (bars 3–7 follow the same pattern) . . . • E2 E2 DE |: Bar 8 (last bar). The final token is E, an eighth note on pitch E.✓ final. Verification summary: meter✓(all 8 bars have 6 eighth-note equivalents); length✓(exactly 8 bars); final✓(last note is E); rhythm✓(only eighth and quarter notes appear). All four constraints satisfied; A = 1. Note that the tune reuses three bar motifs in the order A–B–C–A–B–C–A–B, the minimal-motif style examined in §5.6. The tune is correct under every stated requirement; perceptual quality is outside what the verifiers measure. C.7 Constraint Rendering Examples Table 19 shows the complete set of constraint-text renderers: for each family, the table lists the parameter types and one representative rendering. These renderers are deterministic functions, so every benchmark item with the same family and parameters produces the same constraint text. The constraint texts are designed to be unambiguous to a musically literate reader. Themeterrenderer explicitly states “every bar must contain exactly the right total duration” because simply saying “the meter is 3/4” could be interpreted as a header-only requirement. Similarly, thestructurerenderer spells out the matching/differing requirement rather than assuming the model knows what AABA form means. 27 Published as a conference paper at ICLR 2027 Table 18: Table 1 with the prompt column restored. Under each family name is the span it is checked over; the second column lists the default sampler’s values (the verifier domainΘ f is wider), the third the sentence the prompt template emits for one such value, verbatim from the renderer.structureis set apart as the only cross-section form family. FamilyDefault sampler valuesPrompt line (one sampled value) key global 13 keys: 8 major, 5 minorthe tune must be in D minor meter every bar 4/4, 3/4, 6/8, 2/4; each bar must sum to the signature the meter must be 6/8, and every bar must contain exactly the right total duration length global 8, 12 or 16 bars, no pickupthe tune must be exactly 16 bars long (no pickup bar) range every note 4 inclusive pitch spans, e.g. G3–A5every note must lie between G3 and A5 inclusive final last note a pitch class, any octave (a scale degree 1/3/5 when a key is present) the last note of the tune must be E (any octave) rhythm every note 4 admissible duration sets; no dotted values, no rests use only eighth and quarter notes (no dotted notes, no other durations, no rests) leap adjacent notes largest melodic interval: 7, 9 or 12 semitones melodic motion must be smooth: no melodic leap larger than a major sixth structure 4-bar units aaba, aabb or abab over 16 bars; equal letters identical, different letters distinct the tune must be exactly 16 bars with an ABAB form in 4-bar units: labeled units with the same letter must be note-for-note identical, and units with different letters must differ Table 19: Constraint rendering for all eight families. The “Parameters” column shows the parameter space; the “Rendered text” column shows the template output for one example value. Paraphrase variants (Appendix C.8) replace these templates with alternative wordings. FamilyParametersRendered text (example) keykey name× mode“The tune must be in G major.” metertime signature“The meter must be 3/4, and every bar must contain exactly the right total duration.” lengthbar count “The tune must be exactly 16 bars long (no pickup bar).” rangelow pitch, high pitch“Every note must lie between C4 and C6 inclusive.” finalpitch class; any octave “The last note of the tune must be D (any octave).” rhythmadmissible duration set “Use only eighth and quarter notes (no dotted notes, no other durations, no rests).” leapmax interval name“Melodic motion must be smooth: no melodic leap larger than a perfect fifth.” structure form label, unit size “The tune must be exactly 16 bars with an AABA form in 4-bar units: labeled units with the same letter must be note-for-note identical, and units with different letters must differ.” C.8 Paraphrase Examples The paraphrase suite re-renders the same constraint values with alternative wordings. Table 20 shows three examples, illustrating the range of surface variation; the constraint parameters are unchanged. D Additional Experiments Appendix D.5 contains the matched comparison. All other RL analyses in this section preserve the diagnostic study, with SFT initialisation unless a Direct row or another configuration is explicitly named. Table 33 collects those historical reference scores. Unless explicitly labelled otherwise, all-satisfied rates useA(Equation 1), as in the main comparison; per-family statistics report the individual verifier verdicts. Appendix C.2 separately reports the additional format-gate audit. 28 Published as a conference paper at ICLR 2027 Table 20: Paraphrase examples. Each row shows the original constraint text and one paraphrased variant; the constraint parameters are identical. OriginalParaphrased The tune must be in A minor.Compose the melody in the key of A minor. The meter must be 4/4, and every bar must contain exactly the right total duration. Write it in 4/4 time, and make every measure add up to exactly the right duration. Melodic motion must be smooth: no melodic leap larger than an octave. Keep the melodic motion conjunct: no interval between consecutive notes may exceed 12 semi- tones. D.1 Paraphrase and Training Dynamics Table 21 re-renders the single and seen-combination problems with alternative wordings of identical constraint values, and Table 22 gives the numeric values behind Figure 5(d). Table 21: Diagnostic study: instruction paraphrase, scored byA(the format audit leaves these verdicts unchanged). The original column averages the single and seen-combination splits; the paraphrased column re-renders the same 400 problems with alternative wordings of identical constraint values. ModelOriginal wording Paraphrased∆ SFT0.5800.535-0.045 MusicRLVR0.7350.688-0.047 Table 22: Diagnostic study: numeric values behind Figure 5(d). Curve rows come from a full-retention re-run of the SFT-initialised reference configuration; “original run” gives the three checkpoints the original run retained. Its step-300value0.660is a separate evaluation of the weights Table 3 scores at0.663, within the re-evaluation shift of Appendix B.9; the two are reported separately, not pooled. The last row uses the symmetric structure reward of Appendix E.2; it and the symmetric row of Table 3 (0.590on Mixed) are that reward’s two separate runs, not two evaluations of one checkpoint. Step50100150200250300 All satisfied0.570 0.593 0.633 0.687 0.693 0.703 meter0.716 0.852 0.901 0.926 0.914 0.926 final0.505 0.505 0.558 0.642 0.632 0.632 key0.838 0.838 0.787 0.838 0.800 0.812 structure0.767 0.836 0.808 0.740 0.822 0.781 All satisfied (original run, retained ckpts)–0.630 0.640 0.660 All satisfied (symmetric structure reward) 0.477 0.463 0.463 0.530 0.537 0.553 D.2 Rejection Sampling and a Music-Specialised Baseline Table 4(b) spends the verifier at inference time instead: for each item we draw eight completions, matching the GRPO group sizeG = 8and the rollout temperature (1.0), from both trained models, so selection sees the same per-prompt candidate count as one GRPO group, and report pass@kunder the unbiased estimator (1− n−c k / n k forcofnsamples satisfying all constraints, averaged over items). Selection helps both models and MusicRLVR stays ahead at every matched candidate budget, on both suites, so verifier-guided selection stacks with training rather than substituting for it; note that, unlike the trained policy alone, best-of-Nrequires the verifier andNgenerations andNverifications at deployment. ChatMusician-7B (Yuan et al., 2024) is evaluated zero-shot under the same protocol as every other untrained model, through its releasedHuman:/Assistant:template and greedy decoding (context 2,048tokens). UnderA, it reaches0.160on the mixed suite and0.000on the structure split, where its parse rate is0.158. Under the additional format gate its score is zero on all eight suites (Tables 14 and 16), mainly because required headers are missing. Trained on corpus ABC, it writes repeat signs on most form prompts, which the instruction forbids and the parse gate rejects. Its training objective is stylistic generation rather than stated requirements, so the protocol is not matched to it. This measurement bounds this model under this template, context budget and decoding setting; it 29 Published as a conference paper at ICLR 2027 does not establish what music-specialised pre-training confers in general, which would need a prompt format and budget tuned to that model. D.3 Edge Subsets and a Stricter Key Criterion Both analyses use saved per-item verdicts, without regeneration or retraining. src/rescore_subsets.pycomputes the key-sensitivity audit under the historical gate; the final make_checkpoint_assets.py pass uses the same A verdicts for Edge subsets. Edge mixes two factors. Table 23 splits it by whether an item requests one of the four held-out pairs. On the329items without a held-out family pair, MusicRLVR reaches0.210against0.100for the warm start (exact McNemarp = 2.03× 10 −6 ) and0.195for MiniMax-M3 (p = 0.685); on the171 items that do request a held-out pair the order reverses (0.070against0.094). The margin over the warm start is wider on the pair-free subset than on Edge as a whole. Table 23: Diagnostic study: Edge split by held-out-pair membership, using the main-table success criterion. Pair-free items contain none of the four held-out family pairs. ModelAll (n = 500) Pair-free (n = 329) With held-out pair (n = 171) Base 4B0.0100.0120.006 Qwen3-8B0.0320.0490.000 Qwen3-32B0.1120.1250.088 Llama-70B0.0580.0670.041 MiniMax-M30.1600.1950.094 ChatMusician0.0460.0550.029 SFT0.0760.1000.029 MusicRLVR (re-evaluated)0.1620.2100.070 The deployed key check requires the header to match and the analyzed key to be the requested key or its relative (§C.3). Tables 24 and 25 hold the historical format gate fixed and report every rate with that tolerance and without it, that is withkeysatisfied only when header and analysis both give the requested key. Rates stay unchanged or decrease. The largest baseline decrease is0.060 (Qwen3-32B on Unseen); the largest decrease for the fixed MusicRLVR checkpoint is0.030on Seen. The SFT-initialised route remains highest-scoring among the models in this audit on seven splits; on Edge, the two leading rates stay within one item of each other (0.150 against 0.152). Table 24: Diagnostic study: All-satisfied rate with the relative-key tolerance and without it (hybrid/exact key), six original suites, holding the historical format gate fixed. Baselines show the paired rates; the two MusicRLVR rows report the fixed-checkpoint rate under each key rule. Model / key ruleMixedSingleSeenUnseenHigh-kStructure Base 4B0.160/0.157 0.225/0.210 0.100/0.090 0.040/0.030 0.037/0.031 0.100/0.100 Qwen3-8B0.123/0.117 0.210/0.200 0.115/0.110 0.050/0.050 0.006/0.006 0.000/0.000 Qwen3-32B0.363/0.340 0.535/0.485 0.250/0.240 0.190/0.130 0.094/0.062 0.267/0.267 Llama-70B0.380/0.370 0.695/0.675 0.300/0.295 0.195/0.185 0.056/0.056 0.167/0.167 MiniMax-M30.367/0.363 0.580/0.550 0.315/0.310 0.190/0.185 0.100/0.100 0.008/0.008 ChatMusician0.160/0.147 0.290/0.280 0.130/0.120 0.090/0.085 0.006/0.000 0.000/0.000 SFT0.477/0.453 0.745/0.715 0.415/0.385 0.440/0.415 0.231/0.225 0.600/0.592 MusicRLVR (hybrid)0.6630.8350.6350.5950.3560.808 MusicRLVR (strict key) 0.6370.8150.6050.5800.3440.783 Table 25: Diagnostic study: Hybrid/exact key on Dense and Edge, holding the historical format gate fixed; MusicRLVR uses the single re-evaluation. ModelDenseEdge Base 4B0.008/0.006 0.010/0.008 Qwen3-8B0.020/0.018 0.032/0.032 Qwen3-32B0.074/0.056 0.112/0.102 Llama-70B0.040/0.032 0.058/0.048 MiniMax-M30.088/0.084 0.160/0.152 ChatMusician0.020/0.014 0.046/0.042 SFT0.290/0.266 0.076/0.070 MusicRLVR (re-evaluated) 0.370/0.342 0.162/0.150 30 Published as a conference paper at ICLR 2027 D.4 Significance Tests The matched Direct-versus-SFT-initialised tests are in Appendix D.5; the tables below cover the diagnostic reward-design comparisons. Table 26 reports exact McNemar tests for the single-run diagnostic reward-design experiment under the main-table success criterion, pairing the same item IDs. These tests condition on the specified checkpoints; Appendix D.7 lists the success counts underlying the historical reference rates. Table 27 reports the Edge comparison. All p values are nominal pairwise tests. Table 26: Diagnostic study: Single-run reward-design comparisons (nominal exact McNemarp) over per-item outcomes. ComparisonMixed Single Seen Unseen High-k Struct. Reference run vs. SFT< 10 −4 0.002 < 10 −4 < 10 −4 0.007 < 10 −4 Reference run vs. binary credit, 21% share0.0760.5971.0000.3121.000 < 10 −4 Reference run vs. binary credit, 33% share0.8220.5830.6250.7710.678 < 10 −4 Table 27: Diagnostic study: Paired comparison on Edge. “Only RL” and “Only baseline” count discordant items in the designated single evaluation of each model. ComparisonOnly RL Only baselineExact pDirection Edge: RL vs. MiniMax62611.000000 No detected difference D.5 Matched Comparison This study supplies the three RL rows in Table 2. All runs use seed43, checkpoint300, the same 6,000-prompt training pool and48completions per update. Two policy accelerators use micro-batch 2and accumulation12; a third serves rollouts with memory fraction0.92. The remaining GRPO settings match Table 5. Direct binary usesλ = 0(R = A), and both MusicRLVR routes use λ = 0.7. The reward weight is the only changed training setting between the two Direct arms. The SFT-initialised arm changes both the initial policy and its fixed KL reference. All evaluations use the same item IDs and the all-satisfied criterion A. Table 28: Success counts and rates for the matched final checkpoints. SplitDirect, binary Direct, MusicRLVR SFT init., MusicRLVR Mixed239/300 (0.797)242/300 (0.807)173/300 (0.577) Single192/200 (0.960)197/200 (0.985)176/200 (0.880) Seen149/200 (0.745)163/200 (0.815)117/200 (0.585) Unseen113/200 (0.565)119/200 (0.595)106/200 (0.530) High-k75/160 (0.469)92/160 (0.575)56/160 (0.350) Structure 109/120 (0.908)116/120 (0.967)46/120 (0.383) Dense161/500 (0.322)181/500 (0.362)183/500 (0.366) Edge150/500 (0.300)155/500 (0.310)67/500 (0.134) Pooled comparison. The eight suites are disjoint item sets whose union is the2,180items of MusicConstraintBench, so the same paired test applies to the union. Against binary reward, Direct MusicRLVR satisfies1265items against1188(0.580against0.545), winning229and losing 152of the381items on which the two disagree (exact two-sidedp = 9.39× 10 −5 ). Against the SFT-initialised route it satisfies1265against924, winning554and losing213(p < 10 −4 ); the binary-reward arm also leads that route, 1188 against 924 (p < 10 −4 ). Per-suite breakdown. Graded credit has the higher rate on all eight suites, though on Mixed and Edge the margin is0.010, within the re-evaluation spread of Table 13. Table 31 applies the eight-suite Holm correction inside each comparison: under it the reward-granularity differences are individually unresolved at these discordant counts, while six of the eight initialisation differences are significant, the exceptions being Unseen and Dense. The initialisation comparison uses the identical protocol, with Direct ahead on seven suites and behind on Dense by two successes. All three models pass 31 Published as a conference paper at ICLR 2027 Table 29: Parse-gate pass rates on the six original suites, for every model of Table 2. Each trained 4B checkpoint clears the gate on every item it was given. ModelMixedSingleSeenUnseenHigh-kStructure Qwen3-4B0.5130.4500.5100.5450.3620.925 Qwen3-8B0.3670.4650.3450.3050.2190.158 Qwen3-32B0.9731.0000.9600.9450.9380.983 Llama-3.1-70B0.9701.0000.9800.9800.9441.000 MiniMax-M30.5770.6700.5650.4650.5310.083 ChatMusician-7B0.6930.8000.6800.6700.5750.158 SFT (4B)1.0001.0001.0001.0001.0001.000 Direct GRPO (R = A)1.0001.0001.0001.0001.0001.000 MusicRLVR (SFT init.)1.0001.0001.0001.0001.0001.000 MusicRLVR (Direct)1.0001.0001.0001.0001.0001.000 Table 30: Parse-gate pass rates for the matched evaluations. ModelMixed Single Seen Unseen High-k Structure Dense Edge Direct GRPO (R = A) 1.000 1.000 1.000 1.000 1.0001.000 1.000 1.000 MusicRLVR (SFT init.) 1.000 1.000 1.000 1.000 1.0001.000 1.000 1.000 MusicRLVR (Direct)1.000 1.000 1.000 1.000 1.0001.000 1.000 0.992 the parse gate on every original-suite item; on Edge the Direct MusicRLVR model passes496/500, while the binary-reward and SFT-initialised models pass 500/500 (Tables 29 and 30). What the residual failures look like. Table 32 decomposes the58Mixed items the Direct Music- RLVR checkpoint does not satisfy, from the same saved per-item verdicts as Table 28. The failures are shallow rather than pervasive:48of the58miss a single family and only2miss three, none fails the parse gate, andstructureaccounts for both the largest per-family deficit (22of the73items requesting it) and the most common sole miss (18items). This is the matched checkpoint of Table 2; the diagnostic split-level patterns of Appendix E.1 are a separate checkpoint and are not comparable item for item. 32 Published as a conference paper at ICLR 2027 Table 31: Exact two-sided McNemar tests for the matched Direct MusicRLVR checkpoint against each control; discordant counts are successes by only one checkpoint, and Holm correction is applied over the eight suites within each comparison. These tests condition on the saved completions and checkpoints, and do not estimate variation across training runs or repeated decoding. ComparisonSplitOnly Direct Only control Exact p Holm p Binary reward Mixed19160.73591.0000 Single500.06250.3125 Seen25110.02880.1729 Unseen21150.40501.0000 High-k32150.01860.1304 Structure700.01560.1250 Dense78580.10290.4117 Edge42370.65301.0000 SFT init.Mixed9021 < 10 −4 < 10 −4 Single221 < 10 −4 < 10 −4 Seen6519 < 10 −4 < 10 −4 Unseen36230.11750.2350 High-k5822 < 10 −4 0.0002 Structure711 < 10 −4 < 10 −4 Dense90920.94090.9409 Edge12234 < 10 −4 < 10 −4 Table 32: Matched study: the58unsatisfied Mixed items of the Direct MusicRLVR checkpoint. “Missed” counts items requesting a family whose verifier fails; “sole miss” restricts to the48items that miss exactly one family. Families are ordered by missed count; rhythm is omitted, with no miss among the items requesting it. FamilyRequested MissedSole miss structure732218 length651410 key80119 final95117 leap8974 meter8130 range8020 Items by number of families missed: 1: 48, 2: 8, 3: 2; parse-gate failures: 0. D.6 Diagnostic Reference Results The following table gives the diagnostic evaluations. Unless a diagnostic explicitly states another setting, its RL results refer to the SFT-initialised checkpoint in this table. The Direct route is reported only from the matched runs of Table 2, so no Direct row appears here. 33 Published as a conference paper at ICLR 2027 Table 33: Diagnostic reference results (SFT-initialised route): all-satisfied rates above, parse-gate pass rates below. A different RL checkpoint from the matched comparison of Table 2. Colours rank rows within this table; missing entries were not evaluated. Six original splitsLater splits ModelMixed Single Seen Unseen High-k Structure Dense Edge Zero-shot Qwen3-4B0.160 0.225 0.100 0.0400.0370.1000.008 0.010 Qwen3-8B0.123 0.210 0.115 0.0500.0060.0000.020 0.032 Qwen3-32B0.363 0.535 0.250 0.1900.0940.2670.074 0.112 Llama-3.1-70B0.380 0.695 0.300 0.1950.0560.1670.040 0.058 MiniMax-M30.367 0.580 0.315 0.1900.1000.0080.0880.160 ChatMusician-7B0.160 0.290 0.130 0.0900.0060.0000.020 0.046 Trained (4B) SFT (4B)0.4770.7450.4150.4400.2310.6000.2900.076 MusicRLVR (SFT init.)0.6630.8350.6350.5950.3560.8080.3700.162 Parse-gate pass rate (six original splits) ModelMixedSingleSeenUnseenHigh-kStructure Qwen3-4B0.5130.4500.5100.5450.3620.925 Qwen3-8B0.3670.4650.3450.3050.2190.158 Qwen3-32B0.9731.0000.9600.9450.9380.983 Llama-3.1-70B0.9701.0000.9800.9800.9441.000 MiniMax-M30.5770.6700.5650.4650.5310.083 ChatMusician-7B0.6930.8000.6800.6700.5750.158 SFT (4B)1.0001.0001.0001.0001.0001.000 MusicRLVR (SFT init.)1.0001.0001.0001.0001.0001.000 D.7 Fixed-Checkpoint Evaluation The diagnostic reference checkpoint isoutputs/grpo_4b_v2/checkpoint-300. Table 34 gives its success counts on the six original suites; on Dense and Edge it reaches185/500and81/500. Its first evaluation supplies the original-suite rates; the separate re-evaluation is documented in Appendix B.9. Success counts for the three matched checkpoints are in Table 28. Table 34: Diagnostic study: success counts and all-satisfied rates for the SFT-initialised reference checkpoint on the six original suites, matching Table 33. SplitSFT init. count Rate Mixed199/3000.663 Single167/2000.835 Seen127/2000.635 Unseen119/2000.595 High-k57/1600.356 Structure97/1200.808 D.8 Hyper-parameter Sweeps Table 35 lists the values plotted in Figure 5(b). Each cell is one training run per setting with a single parameter moved from the SFT-initialised reference configuration, scored on Mixed and Structure. The reward mixing weight is swept the same way in Appendix D.9. Scope of the sensitivity evidence. Learning rate has the largest observed Mixed spread (0.270), compared with0.063for KL coefficient and0.123for group size. Every setting exceeds the SFT Mixed baseline. Each cell is one training run, so these ranges describe observed sensitivity across the tested settings. Structure spans0.458–0.833across the three, without a monotone pattern along any of them. TheGsweep fixes48completions per optimiser step. AccordinglyG = 4, 8, 16uses12, 6, 3prompt groups per step, for3,600, 1,800, 900prompt draws in300steps. The corresponding logged epochs are0.60, 0.30, 0.15. It compares allocation of a fixed completion budget, not group size at fixed prompt exposure. 34 Published as a conference paper at ICLR 2027 Table 35: Diagnostic study: All-satisfied rate on the mixed and structure suites for each hyper-parameter setting, one training run each. ⋆ marks the SFT-initialised reference configuration. Structure varies substantially across these settings. Hyper-parameter ValueMixed Structure Learning rate5×10 −7 0.5670.808 1×10 −6⋆ 0.6630.808 2×10 −6 0.7100.542 5×10 −6 0.8370.833 KL coefficient β00.6070.750 0.010.6600.567 0.04 ⋆ 0.6630.808 0.10.6000.458 Group size G40.5400.533 8 ⋆ 0.6630.808 160.5770.475 D.9 Reward Mixing Weight Table 36 movesλof Equation 4, which weights the per-family average against the binary all-satisfied term, while every other setting stays at the SFT-initialised reference configuration. It is reported separately from Table 35 because that table lists the values plotted in Figure 5(b), and because the form decomposition below is what makes this sweep interpretable. The default achieves the highest observed scores on both suites among the three weights tested in the single-run sweep. Againstλ = 0.7, the low arm’s Mixed comparison gives exact McNemar p = 0.368, while its Structure score is lower (p = 3.54× 10 −9 ); the high arm scores lower on both (p = 0.002 and p = 4.74× 10 −15 ). Table 36: Diagnostic study: Reward mixing weight, one training run per setting, scored by the binary criterion of Equation 1. Repetition and contrast count the structure-split items passing all required-equal and all required- distinct unit-pair checks, respectively; ⋆ marks the default. λMixed Structure Repetition Contrast 0.5 0.6330.425120/12055/120 0.7 ⋆ 0.6630.808119/120 104/120 1.0 0.5700.325119/12045/120 Contrast distinguishes the tested weights. The last two columns separate the halves of the form constraint. Repetition is satisfied on119–120of the120items at every weight, while contrast passes55items atλ = 0.5and45atλ = 1.0against104at the default: the decrease on Structure accompanies a drop in contrast, not in repetition. The all-satisfied counts behind those rates are51,97 and39; each deficit relative to the contrast count consists of items passing the form check and missing a co-requested family. Outputs whose four units are all identical – which satisfy every required-equal pair and no required-distinct one – rise from15items at the default to65and74at the two ends. The same pattern holds across the rows of Table 3: repetition passes119–120items, and the binary form pass count equals the contrast pass count. Where the weight can act. Writem i for the per-family average in Equation 4, setting it to zero for a gate-rejected rollout. Forλ > 0, a group in which every rollout hasA(y i ,C) = 0has R(y i ,C) = λm i . Equation 5 then gives ˆ A i = (m i − ̄m)/(σ m +ε/λ), whereσ m is the group sample standard deviation ofm: the scaling cancels except through the stabiliser. The mixture also controls the reward gap between full and partial success. Consider two gate-passing rollouts for the same prompt: one satisfies allkfamilies; the other fails exactly one family, retaining graded credits∈ [0, 1) for that family. Their reward gap is ∆R = (1− λ) + λ(1− s) k , which is(1− s)/katλ = 1and0.3 + 0.7(1− s)/kat the default. The binary term thus adds an explicit bonus for joint satisfaction; its effect on normalised advantages also depends on the other 35 Published as a conference paper at ICLR 2027 Table 37: Diagnostic study: Per-family satisfaction on Mixed: the fraction of items requesting a family whose verifier passes. ∆ is the fixed-checkpoint rate minus the SFT rate. FamilyBase SFT MusicRLVR∆ key0.237 0.6250.850+0.225 meter0.222 0.6300.926+0.296 length0.015 0.9691.000+0.031 range0.375 0.8750.975+0.100 final0.368 0.4210.611+0.189 rhythm0.416 0.7270.753+0.026 leap0.393 0.7640.854+0.090 structure 0.041 0.7260.644-0.082 rewards in the group. The sweep supports the default mixture for the tested configuration, with the highest observed compliance and fewest all-identical outputs. D.10 Per-Family Satisfaction on the Original Suites Table 33 reports the all-satisfied rate, which is zero whenever any single constraint fails. Table 38 reports each family’s satisfaction rate on each split for the same checkpoint, and Figure 7 gives the same per-family view on Mixed across models. key meter len range final rhythm leap struct Qwen3-4B Qwen3-32B Llama-70B MiniMax-M3 SFT MusicRLVR 0.240.220.010.380.370.420.390.04 0.840.180.090.700.960.690.690.20 0.810.420.140.710.870.600.810.10 0.570.300.280.610.580.380.560.04 0.620.630.970.880.420.730.760.73 0.850.931.000.970.610.750.850.64 Figure 7: Diagnostic study: Per-family satisfaction on the mixed suite; MusicRLVR cells use the fixed checkpoint (Table 37); structure stays red for every untrained model. The mixed suite includes73structure-family items. Their satisfaction rate is0.644for MusicRLVR against0.726for SFT. On the dedicated structure split, the all-satisfied rate is0.808against0.600; equal prompt weighting gives0.637against0.522(Appendix B.7). The splits differ in their requested constraints and prompt weights. Table 38: Diagnostic study: per-family satisfaction rate for MusicRLVR across the splits. Each cell is the fraction of problems mentioning that family whose own verifier passes at the fixed checkpoint (other constraints may fail); families mentioned by fewer than five problems in a split are marked “–”. FamilySingle Seen Unseen High-k Structure final0.7830.6360.4070.635– key0.8620.8870.5650.6710.643 leap0.2920.9350.9850.8721.000 length1.0001.0001.0001.000– meter0.8120.9030.8970.8901.000 range1.0000.9820.9550.9801.000 rhythm0.8330.7900.8430.6550.692 structure1.0000.5100.6670.5540.867 lengthandrangemaintain high satisfaction across the splits. Thefinalrate is lower on Unseen than on Single, whilekey,rhythmandstructurealso vary with the requested combination. For leap, satisfaction is lower on Single than on the combination splits. Table 38 reports these rates under the corresponding family and parameter mixture. 36 Published as a conference paper at ICLR 2027 D.11 Performance by Number of Constraints (k) Table 39 breaks the mixed suite (n = 300;100/100/60/40items atk = 1/2/3/4) down by the number of constraints, using the same saved diagnostic reference evaluations as Table 33; Figure 1 instead uses the matched runs. Table 39: Diagnostic study: all-satisfied rate on the mixed suite by number of constraintsk. The last column is thek=1rate over thek=4rate; a dagger marks curves not monotone ink(thek=3andk=4cells hold only60 and 40 items, so one item moves them by 0.017 and 0.025). Modelk = 1 k = 2 k = 3 k = 4 k=1/k=4 MusicRLVR (SFT init.)0.8500.6400.5170.4751.8× SFT (4B)0.720 0.430 0.267 0.3002.4× † Qwen3-32B0.650 0.300 0.133 0.1504.3× † Llama-3.1-70B0.720 0.340 0.100 0.05014.4× MiniMax-M30.560 0.360 0.267 0.05011.2× Qwen3-4B0.340 0.110 0.033 0.02513.6× Qwen3-8B0.240 0.120 0.017 0.000– The endpoint ratios vary across models. Llama-3.1-70B falls from0.720atk = 1to0.050atk = 4, a14.4×drop; MiniMax-M3 falls11.2×and the untrained 4B base13.6×. The SFT-initialised MusicRLVR rates are0.850and0.475, giving the smallest endpoint ratio in the table (1.8×), against 2.4×for SFT. The matched by-kcurves are in Figure 1. These slices retain their own family and parameter mixtures. D.12 Output Length Statistics Table 40 reports the average number of bars and notes per parsed completion on the mixed suite, supplementing the quality metrics in Table 41. Table 40: Diagnostic study: Output length in the single-run mixed-suite diagnostic (n = 300problems). Only completions that pass the historical parse gate are included, so the parsed count differs by model; the IrishMAN reference (299 tunes, 19.9 bars) is in Table 41. SourceParsed n Mean bars Mean notes Qwen3-4B (zero-shot)15429.378.4 Qwen3-32B (zero-shot)29218.458.1 Llama-3.1-70B (zero-shot)29121.266.1 MiniMax-M3 (zero-shot)17317.937.0 SFT30039.2167.5 MusicRLVR30028.7130.3 The supervised model over-generates (39.2bars against19.9for the corpus), and the reinforcement stage pulls this back to28.7bars, consistent with a reward that favours concision without an unconditional length term, since a longer tune has more places to violate a constraint. The three larger zero-shot models sit near the corpus length (17.9–21.2bars; the 4B base averages29.3), and MiniMax-M3 parses on only 173 of 300 problems. D.13 Length-Controlled Output Statistics §5.6 compares the first eight parsed bars of a common item set (Table 4), because the full-tune ratios of Table 41 have length-dependent denominators. This section gives the uncontrolled statistics those ratios come from, and the cross-prompt duplication counts of Table 42. Both read historical-gate outputs, but they measure different things: the first are within-tune proxies, the second counts how often two different prompts produce the same score. Neither measures perceptual quality. On the full completions, MusicRLVR raises distinct-bar fraction from0.276to0.313while pitch-class entropy falls2.00→ 1.89and 4-gram distinctness0.294→ 0.292; mean length falls39.2→ 28.7 bars. The distinct-bar increase here is0.276 → 0.313; under the length control of §5.6 the same comparison is0.417→ 0.426on matched eight-bar prefixes, which is why the matched comparison is the one reported in the main text. These diagnostics describe output variety, not an aesthetic 37 Published as a conference paper at ICLR 2027 Table 41: Diagnostic study: distributional statistics of the single-run mixed-suite completions and299corpus tunes. No constraint names entropy or either distinctness measure; bar count is constrained on the138/300 items requestinglengthorstructure. Mean length differs between rows, so distinctness denominators are unmatched; Table 4 controls for that. ModelBars PC ent. Dist. bars Dist. 4-gr. Corpus19.9 2.630.7580.676 Base29.3 2.060.3660.391 SFT39.2 2.000.2760.294 RL, binary credit 32.2 1.850.2840.289 MusicRLVR28.7 1.890.3130.292 Table 42: Diagnostic study: full-score duplication for the first eligible occurrence of each of220distinct prompts in that common set, identified by parsed event signatures and ignoring metadata. This measures duplication across prompts, not diversity from repeated sampling of one prompt. ModelUnique full scores Most frequent score SFT167/2207/220 MusicRLVR (diagnostic)182/2205/220 advantage of reinforcement learning. Repeated sampling, length-matched full-score comparisons and independent listening evaluations would address different aspects of musical diversity and quality. 38 Published as a conference paper at ICLR 2027 D.14 Zero-Variance Groups During Training Equation 5 implies that a group whose G rollouts all receive the same reward contributes no reward gradient. The trainer logs the fraction of such groups per step (frac_reward_zero_std); Table 43 summarises it for the two reward variants of the main ablation. Table 43: Diagnostic study: Fraction of GRPO groups with zero reward variance (duplicate step records from restarts removed), averaged over the first and last quarter of training and over all300steps, with the mean within-group reward standard deviation alongside. Zero-variance groupsReward std Runfirst 25% last 25% all first 25% last 25% Binary form credit, original form share (21%)0.1160.307 0.2210.3010.214 Graded form credit, 33% form share (final)0.1330.322 0.2540.2960.210 Two readings. In the final configuration about a quarter of all groups carry no reward-driven gradient, rising to a third late in training, which is the quantity the graded term targets. The aggregate should be read together with the prompt mix: the two ablation rows differ in both the reward and the share of structure prompts, and the changed share of form prompts could itself alter the zero-variance fraction. The matched runs measure the same quantity on two arms that differ only inλ, holding the prompt mix fixed. Averaged over the first50updates, binary reward leaves0.550of groups with zero reward variance against0.307for the graded reward; over the last50the two reach0.823and0.823, once most groups succeed outright, and over all300updates they average0.717and0.660(Figure 4(b)). The SFT-initialised arm runs lower throughout (0.120,0.343,0.245) while scoring lowest on the benchmark, so the fraction is read between arms that share an initialisation rather than across routes. D.15 Additional Training Curves 0100200300 GRPO step 0.6 0.8 reward (a) Mean group reward binary form credit graded (final) 0100200300 GRPO step 0.1 0.2 0.3 std (b) Within-group reward std 0100200300 GRPO step 1.5 2.0 norm (c) Gradient norm 0100200300 GRPO step 0.6 0.8 mean reward (d) Learning rate 5e-7 1e-6 2e-6 5e-6 0100200300 GRPO step 0.6 0.8 mean reward (e) KL coefficient β 0 0.01 0.04 0.1 0100200300 GRPO step 0.6 0.8 mean reward (f) Group size G 4 8 16 Figure 8: Diagnostic study: trainer-log quantities behind Figure 5. (a–c) Mean group reward, within-group reward SD and gradient norm, binary form credit against the final graded run. (d–f) Mean group reward for every hyper-parameter setting of Figure 5(b) on one shared scale; the thick blue curve is the default run (lr = 1e−6, β = 0.04,G = 8). Curves are smoothed over9steps; each axis spans the2nd–98th percentile of its raw trace, so a few steps clip. Figure 8 gives the trainer-log quantities behind the diagnostics of Figure 5. This section records the numbers behind Figure 8. Final KL to the reference falls0.047→ 0.024→ 0.010asβrises0.01→ 0.04→ 0.1(theβ = 0run logs no KL term), and yet the observed mixed rates lie in the narrow range0.600–0.663(Table 35): the KL term therefore tracksβas designed, while held-out compliance is flat across the range. Along learning rate, final KL0.006,0.024,0.049, 0.157and mean group reward0.713,0.832,0.893,0.936over5×10 −7 to5×10 −6 run in the same order as the mixed suite. Across the three parameters of Table 35, structure scores span0.458to 0.833without a monotone trend along any axis; Appendix D.9 adds the reward mixing weight, which reaches 0.325 at λ = 1. 39 Published as a conference paper at ICLR 2027 D.16 Additional Diagnostics Figure 9 shows the two per-item diagnostics the main text does not: which families fail together, and when each item is first solved during training. key meter len range final rhythm leap struct MusicRLVR failures (n=300) key meter len range final rhythm leap struct (a) Co-failure matrix 12121 62 2 1237142 21191 413 12126 0 5 10 15 20 25 30 35 50100150200250300 GRPO step 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of items ever solved (b) Learning timeline 57.0% 68.7% 77.0% 80.0% 80.7% 81.0% SFT baseline Figure 9: Diagnostic study: per-item diagnostics on the mixed suite (n = 300). (a) Co-failure matrix of MusicRLVR: diagonal entries are per-family failure counts, off-diagonal entries co-failures, which are rare:key shares at most two with any family. (b) CDF of the step at which each item is first solved:57%by step50,81% at least once by step 300. E Error and Structure Analysis E.1 Which Constraint Binds, by Split Figure 10 gives the outcome composition of each split; Table 44 adds the most frequent failing families, using the diagnostic reference evaluation. Figure 11 breaks the same evaluation down by requested form and by how many models solve each item. 0.000.250.500.751.00 fraction of items Single Seen combo Unseen combo High-k Structure Mixed 0.835 0.635 0.595 0.356 0.808 0.663 all satisfiedmiss 1 miss ≥ 2 parse fail Figure 10: Diagnostic study: What an unsatisfied item is missing. Outcome composition for the fixed checkpoint; printed all-satisfied rates match Table 33. Parse failures are zero on all six original splits. Table 44: Diagnostic study: which constraints fail, by split. “Not all satisfied” is one minus the all-satisfied rate; the next column counts items failing two or more families at once; the last lists the three most frequent failing families as fractions of the split’s items (an item failing two is counted under both). No completion fails the parse gate on any split. SplitNot all satisfied Failing≥ 2 familiesMost frequent failing families Single0.1650.000leap (0.085), final (0.025), key (0.020) Seen0.3650.040structure (0.125), final (0.120), rhythm (0.065) Unseen0.4050.105final (0.160), key (0.150), structure (0.090) High-k0.6440.206final (0.237), rhythm (0.181), key (0.156) Structure0.1920.017structure (0.133), key (0.042), rhythm (0.033) 40 Published as a conference paper at ICLR 2027 AABAAABBABAB 0.0 0.5 1.0 all-satisfied rate (a) Structure split, by form SFTMusicRLVR 01234567 models solving it (of 7) 0 20 40 60 80 items (b) Item difficulty all items solved by MusicRLVR Figure 11: Diagnostic study: Structure and item-level diagnostics. (a) Satisfaction by requested form for the fixed checkpoint. (b) Item counts by the number of seven evaluated models solving them under the historicalA. Most unsatisfied items miss one family. The fraction missing multiple families is0.040on Seen and 0.105on Unseen, rising to0.206on High-k. The most frequent failing family isfinalon Unseen and High-k.structureis among the three most frequent failures on Seen and Unseen;leapappears among the top three only on Single. These are frequencies under each split’s own family mixture (Table 38). E.2 Structure Reward Variants Table 45 compares the fixed default checkpoint with the equal-weight control and the symmetric-credit study. The default and equal-weight rows each use one run; symmetric credit reports the mean and sample SD of its two runs. The default form term weights repetition0.7and contrast0.3; the following controls examine that choice. Symmetric form credit. The symmetric term uses0.5 ̄e + 0.5 ̄ d, with zero credit for all-identical outputs and credit1for outputs passing the binary form check. The outer reward mixture is unchanged. Across two runs, contrast satisfaction is0.233± 0.024and the all-identical fraction is0.762± 0.018. The control jointly changes the weights, the contrast factor and the credit for all-identical outputs. Under the default term, an all-identical output earns s structure = 0.7 but has A = 0, because it fails contrast. Appendix D.9 separately varies the outer mixing weight. Reweighting repetition and contrast. The single-variable version moves only the weights, to 0.5/0.5: an all-identical output still banks the repetition half, the binary contrast factor is kept, and the rest of the reward and the training setup are unchanged. Three things follow. (1)In the single-run diagnostic comparison, the structure split moves0.808→ 0.675and all-identical outputs 15→ 32 of 120, the same direction as the joint control. (2) The aggregate symmetric-credit comparison is reported separately in Table 45. (3) The equal-weight control scores 0.637 on Mixed, compared with 0.663 for the default. Appendix D.9 extends this comparison to the outer reward mixture. In the single-run diagnostic study, the two testedλendpoints yield larger increases in all-identical outputs than the equal-weight form control. The default achieves the highest Structure score and fewest all-identical outputs among these tested weight settings. A term rewarding variation across sections remains a candidate for future evaluation. Table 45: Diagnostic study: structure decomposition. “Runs” is the number of training runs: default and equal-weight use one each, symmetric credit reports mean±sample SD over two. All-identical is a fraction; “all requested” additionally requires the companion constraint, present in 66 of 120 items. RewardRunsRepetitionContrastAll requested All-identical MusicRLVR10.9920.8670.8080.125 Equal weights11.0000.7330.6750.267 Symmetric credit 20.996± 0.0060.233± 0.0240.208± 0.0120.762± 0.018 41 Published as a conference paper at ICLR 2027 E.3 Structure Results by Form Type The structure split contains three form types, in unequal numbers: AABA (n = 42), AABB (n = 35) and ABAB (n = 43). Table 46 decomposes performance on each. Table 46: Diagnostic study: Structure results by form for the fixed MusicRLVR and SFT checkpoints. Contrast and all-identical are fractions of items of that form. ModelForm All-sat. Contrast All-identical SFTaaba0.7380.8100.190 SFTaabb0.7710.8570.143 SFTabab0.3260.3950.605 MusicRLVR aaba0.8570.9050.071 MusicRLVR aabb0.7710.8290.171 MusicRLVR abab0.7910.8600.140 ABAB is the form supervision handles worst, and by a wide margin: SFT reaches0.771on AABB but0.326on ABAB, with26/43of its ABAB outputs written as four identical units. All three forms require distinct A and B units, so the gap tracks the arrangement rather than the distinctness requirement: ABAB is the only one that interleaves the two units instead of repeating each in place. At the fixed checkpoint, ABAB has the largest improvement over SFT among the three forms: its all-satisfied rate rises from0.326to0.791, while the all-identical fraction changes from26/43to 0.140. Table 46 reports the rate for every form. 42 Published as a conference paper at ICLR 2027 F Qualitative Examples Every example below is a benchmark item chosen by a fixed rule (the first item in file order with the stated constraint families and pass/fail pattern), its completions are read verbatim from the frozen evaluation files, and the notation is drawn frommusic21’s parse of the output: pitch, duration and bar lines. The key and meter checks additionally read the header-derived objects, and the key check analyses the whole score. Marks under each staff are the verifiers’ decisions on that output. Outputs rejected at the gate are shown as a short excerpt together with the gate’s reason. Examples 1–6 compare the zero-shot base model with MusicRLVR (diagnostic run), and Example 2 shows SFT as well; Examples 7–8 compare MusicRLVR with the larger zero-shot models on the same items. Example 1: unseen-combination split (k = 2, held-out pairkey–rhythm). Itemuc-0022. Constraints: (1) the tune must be in G major; (2) use only sixteenth, eighth, and quarter notes (no dotted notes, no other durations, no rests) G Qwen3-4B (zero-shot) (8 bars) ✗ key G major ✓ rhythm set G MusicRLVR (18 bars; first 16 shown) G ✓ key G major ✓ rhythm set Qwen3-4B (zero-shot) fails key; MusicRLVR satisfies every constraint. Example 2: high-ksplit (k = 4). Itemhk-0004. Constraints: (1) the meter must be 3/4, and every bar must contain exactly the right total duration; (2) the tune must be exactly 8 bars long (no pickup bar); (3) the last note of the tune must be E (any octave); (4) use only eighth and quarter notes (no dotted notes, no other durations, no rests) Qwen3-4B (zero-shot) not rendered: parse-error: ABCTokenException X:1 ⏎ T:Monophonic Tune ⏎ M:3/4 ⏎ L:8 ⏎ K:C ⏎ C D E | C D E | E D C | E D C | ⏎ E D C | E D C | C D E | E D E ... ✗ meter 3/4 ✗ 8 bars ✗ final E ✗ rhythm set G SFT (8 bars) ✓ meter 3/4 ✓ 8 bars ✓ final E ✓ rhythm set G ########## MusicRLVR (8 bars) ✓ meter 3/4 ✓ 8 bars ✓ final E ✓ rhythm set 43 Published as a conference paper at ICLR 2027 Qwen3-4B (zero-shot) fails at the parse gate (parse-error: ABCTokenException); SFT satisfies every constraint; MusicRLVR satisfies every constraint. Example 3: single-constraint split (k = 1). Itemsg-0000. Constraints: (1) melodic motion must be smooth: no melodic leap larger than an octave Qwen3-4B (zero-shot) not rendered: degenerate-or-polyphonic X:1 ⏎ T:Simple Melody ⏎ M:4/4 ⏎ L:1/4 ⏎ K:C ✗ leap ≤ 12 G #### MusicRLVR (16 bars) G ###### ✓ leap ≤ 12 Qwen3-4B (zero-shot) fails at the parse gate (degenerate-or-polyphonic); MusicRLVR satisfies every constraint. Example 4: seen-combination split (k = 3). Itemsc-0005. Constraints: (1) the last note of the tune must be D (any octave); (2) use only sixteenth, eighth, and quarter notes (no dotted notes, no other durations, no rests); (3) melodic motion must be smooth: no melodic leap larger than a perfect fifth Qwen3-4B (zero-shot) not rendered: degenerate-or-polyphonic X:1 ⏎ T:Simple Tune ⏎ M:4/4 ⏎ L:1/16 ⏎ K:C ✗ final D ✗ rhythm set ✗ leap ≤ 7 G ######## MusicRLVR (23 bars; first 16 shown) G ✓ final D ✓ rhythm set ✓ leap ≤ 7 Qwen3-4B (zero-shot) fails at the parse gate (degenerate-or-polyphonic); MusicRLVR satisfies every constraint. Example 5: seen-combination split (k = 2). Itemsc-0001. Constraints: (1) use only sixteenth, eighth, and quarter notes (no dotted notes, no other durations, no rests); (2) the tune must be exactly 16 bars with an AABB form in 4-bar units: labeled units with the same letter must be note-for-note identical, and units with different letters must differ 44 Published as a conference paper at ICLR 2027 G Qwen3-4B (zero-shot) (8 bars) ✓ rhythm set ✗ form AABB G MusicRLVR (16 bars) G ✓ rhythm set ✓ form AABB Qwen3-4B (zero-shot) fails structure; MusicRLVR satisfies every constraint. Example 6: unseen-combination split (k = 2, held-out pairlength–range). Itemuc-0001. Constraints: (1) the tune must be exactly 16 bars long (no pickup bar); (2) every note must lie between G3 and A5 inclusive Qwen3-4B (zero-shot) not rendered: degenerate-or-polyphonic X:1 ⏎ T:Monophonic Tune ⏎ M:4/4 ⏎ L:1/4 ⏎ K:C ✗ 16 bars ✗ range G3–A5 G #### MusicRLVR (16 bars) G ### # ### # ✓ 16 bars ✓ range G3–A5 Qwen3-4B (zero-shot) fails at the parse gate (degenerate-or-polyphonic); MusicRLVR satisfies every constraint. Example 7: structure split (k = 1, zero-shot models against MusicRLVR). Itemst-0023. Constraints: (1) the tune must be exactly 16 bars with an AABA form in 4-bar units: labeled units with the same letter must be note-for-note identical, and units with different letters must differ 45 Published as a conference paper at ICLR 2027 G Qwen3-32B (zero-shot) (15 bars) G ✗ form AABA G Llama-3.1-70B (zero-shot) (32 bars; first 16 shown) G ✗ form AABA MiniMax-M3 (zero-shot) not rendered: repeat-signs-forbidden X: 1 ⏎ T: AABA Tune ⏎ M: 4/4 ⏎ L: 1/4 ⏎ K: G ⏎ |: G2 B2 | d4 B2 A2 | G4 D2 | E4 G2 A2 | ⏎ B2 d2 | g4 d2 B2 | A... ✗ form AABA G ###### MusicRLVR (16 bars) G ###### ✓ form AABA Qwen3-32B (zero-shot) failsstructure; Llama-3.1-70B (zero-shot) failsstructure; MiniMax-M3 (zero-shot) fails at the parse gate (repeat-signs-forbidden); MusicRLVR satisfies every constraint. Example 8: high-ksplit (k = 4, zero-shot models against MusicRLVR). Itemhk-0278. Constraints: (1) the tune must be in C major; (2) every note must lie between D4 and D6 inclusive; (3) the last note of the tune must be G (any octave); (4) melodic motion must be smooth: no melodic leap larger than a perfect fifth 46 Published as a conference paper at ICLR 2027 G Qwen3-32B (zero-shot) (235 bars; first 16 shown) G ✓ key C major ✗ range D4–D6 ✗ final G ✓ leap ≤ 7 G Llama-3.1-70B (zero-shot) (18 bars; first 16 shown) G ✓ key C major ✗ range D4–D6 ✓ final G ✓ leap ≤ 7 G MusicRLVR (169 bars; first 16 shown) G ✓ key C major ✓ range D4–D6 ✓ final G ✓ leap ≤ 7 Qwen3-32B (zero-shot) failsrange,final; Llama-3.1-70B (zero-shot) failsrange; MusicRLVR satisfies every constraint. G Broader Impact This section expands the ethics statement. The method makes music-generation models follow explicit specifications, which helps wherever a score must meet a specification: music education, game audio, accessibility tools. We do not foresee significant negative societal impacts specific to this work, because the experiments concern short monophonic melodies in ABC notation under explicit constraints and evaluate neither production use nor perceptual quality. That last point is also where the method can be misread: a high all-satisfied rate says the requested properties hold, not that the tune is good; an output can satisfy every stated constraint while repeating one short motif (Figure 3), and perceptual quality is not measured here. All supervised targets derive from the publicly available IrishMAN corpus of traditional Irish music; the reinforcement prompts are generated synthetically. The benchmark and verifiers will be released to support reproducibility and further research. 47