Paper deep dive

MusicLayout: Explicit Structural Planning for Controllable Text-to-Music Generation

Shuyu Li, Kejun Zhang, Jiahe Lei, Shulei Ji, Zihao Wang, Jiaxing Yu, Wanying Wu, Lei Wang

Year: 2026Venue: arXiv preprintArea: cs.SDType: PreprintEmbeddings: 75

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%

Last extracted: 8/12/2026, 3:15:19 AM

Summary

The paper introduces MusicLayout, an explicit intermediate representation for controlling musical structure in text-to-music generation. It serves as an interpretable planning layer between textual intent and generated audio, describing sections, textures, repetitions, and instrument arrangements. The authors integrate this into a unified autoregressive framework where the model first generates a MusicLayout and then predicts audio tokens conditioned on it. Experiments demonstrate that this explicit planning improves long-range structural organization and supports layout-level control compared to implicit global text prompts.

Entities (10)

MusicLayout · methodrepresentation · 98%Text-to-Music Generation · task · 95%ACE-Step 1.5 · modelarchitecture · 90%autoregressive formulation · technique · 90%FreeMIDI · dataset · 90%MidiCaps · dataset · 90%MuChin · dataset · 90%layout-conditioned generation · evaluation-metriccondition · 85%MusicGen · modelarchitecture · 85%Stable Audio 3 · modelarchitecture · 85%

Relation Signals (9)

MusicLayout → isusedin → Text-to-Music Generation

confidence 95% · MusicLayout describes a musical piece as a time-aligned layout... serving as an interpretable planning layer between textual intent and the generated music.

MusicLayout → enables → layout-level control

confidence 90% · The resulting MusicLayout can be inspected and modified prior to audio generation, providing a mechanism for layout-level structural control.

MusicLayout → improves → long-range structural organization

confidence 90% · providing evidence that explicit layout planning can improve long-range structural organization and support layout-level control.

MusicLayout → isevaluatedon → MidiCaps

confidence 90% · MidiCaps was used to evaluate generalization to out-of-domain MIDI

MusicLayout → isevaluatedon → FreeMIDI

confidence 90% · FreeMIDI provided the training data and a disjoint in-domain evaluation set.

MusicLayout → isevaluatedon → MuChin

confidence 90% · MuChin provided an out-of-domain evaluation on real audio.

MusicLayout → isintegratedinto → ACE-Step 1.5

confidence 90% · We build our model upon ACE-Step 1.5, extending its LM to generate a MusicLayout before producing layout-conditioned audio tokens.

MusicLayout → →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Text-to-music generation has advanced rapidly, but current systems still rely primarily on global text prompts, leaving the structural organization of generated music implicit and difficult to inspect, control, or revise before audio generation. To address this issue, we introduce MusicLayout, an explicit intermediate representation for controlling musical structure in text-to-music generation. MusicLayout describes a musical piece as a time-aligned layout of sections, textures, repetitions, variations, and instrument-level arrangements, serving as an interpretable planning layer between textual intent and the generated music. We integrate MusicLayout into a text-to-music framework built on a unified autoregressive formulation, where the model first generates a MusicLayout representation and subsequently predicts audio tokens conditioned on this representation within a single sequence. The resulting MusicLayout can be inspected and modified prior to audio generation, providing a mechanism for layout-level structural control. We evaluate MusicLayout through layout-conditioned generation, layout manipulation experiments, and matched-data ablations, providing evidence that explicit layout planning can improve long-range structural organization and support layout-level control.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

74,146 characters extracted from source content.

Expand or collapse full text

MusicLayout: Explicit Structural Planning for Controllable Text-to-Music Generation Shuyu Li1 Kejun Zhang1,3 Corresponding author. Jiahe Lei4 Shulei Ji2,3 Zihao Wang2,5 Jiaxing Yu1 Wanying Wu6 Lei Wang7 1College of Artificial Intelligence, Zhejiang University 2College of Computer Science and Technology, Zhejiang University 3Innovation Center of Yangtze River Delta, Zhejiang University 4The Chinese University of Hong Kong 5Shandong University 6Chu Kochen Honors College, Zhejiang University 7Ant Group lsyxary, zhangkejun, shuleiji, yujx, 3240100265@zju.edu.cn 1155261729@e.cuhk.edu.hk, carlwang1212@gmail.com, thirtyking@163.com Abstract Text-to-music generation has advanced rapidly, but current systems still rely primarily on global text prompts, leaving the structural organization of generated music implicit and difficult to inspect, control, or revise before audio generation. To address this issue, we introduce MusicLayout, an explicit intermediate representation for controlling musical structure in text-to-music generation. MusicLayout describes a musical piece as a time-aligned layout of sections, textures, repetitions, variations, and instrument-level arrangements, serving as an interpretable planning layer between textual intent and the generated music. We integrate MusicLayout into a text-to-music framework built on a unified autoregressive formulation, where the model first generates a MusicLayout representation and subsequently predicts audio tokens conditioned on this representation within a single sequence. The resulting MusicLayout can be inspected and modified prior to audio generation, providing a mechanism for layout-level structural control. We evaluate MusicLayout through layout-conditioned generation, layout manipulation experiments, and matched-data ablations, providing evidence that explicit layout planning can improve long-range structural organization and support layout-level control. 1 Introduction Recent text-to-music models have advanced substantially in fidelity, semantic alignment, musicality, and efficiency [19]. They typically generate discrete audio tokens autoregressively or synthesize continuous and compressed representations with diffusion models [1, 4, 20]. Recent systems further support long-form generation and combine language-model (LM) reasoning with diffusion-based acoustic synthesis [10, 13]. Despite these advances, most systems rely on global text prompts and leave musical planning implicit. A prompt can describe genre, mood, instrumentation, or tempo, but cannot precisely specify how sections are organized, materials recur or vary, textures evolve, or instruments enter and leave. Existing controllable methods add melody, chords, drums, dynamics, symbolic lead sheets, or semantic representations [39, 26, 34, 2, 13]. These controls are often limited to individual attributes, require additional musical inputs, or remain implicit. They do not expose a unified, time-aligned representation for specifying the organization and arrangement of an entire piece. Explicit planning representations improve controllability and interpretability in image generation by exposing high-level organization before content. PlanGen [14], for example, generates spatial layout tokens before image tokens in one autoregressive sequence. For music, this principle could improve long-range structural organization and controllability. Transferring it requires representing organization along time rather than space. Focusing on instrumental music, we introduce MusicLayout, an explicit intermediate representation for planning musical structure in audio language models. It describes section transitions, repetitions and variations, texture changes, and instrument participation along a shared timeline. As illustrated in Figure 1, given a text prompt, our model generates a MusicLayout representation and subsequently predicts audio tokens conditioned on it within a single sequence. This makes structural planning inspectable before synthesis and supplies audio generation with an explicit description of how the music should unfold. Experiments on layout conditioning and manipulation provide evidence that MusicLayout can improve structural organization and support layout-level control. Figure 1: Conventional autoregressive LM-based text-to-audio generation (top) and our explicit-planning framework (bottom). Our LM generates MusicLayout tokens and then discrete audio tokens, labeled audio codes in the figure, within one sequence. A frozen decoder renders the audio tokens into music. The abbreviated rewrite selects the musical section at 49–55 seconds, assigns it the chorus label, and sets its active instrument list to drums. The <roles> tag delimits that list. Our main contributions are summarized as follows: • We propose MusicLayout, an explicit planning representation for unified audio language models. It represents music as a time-aligned layout of sections, repetitions, variations, textures, and instrument arrangements, providing structured musical plans beyond global textual prompts. • We develop a unified autoregressive framework for layout planning and audio generation, where a single model first produces an explicit MusicLayout plan and then continues to generate audio tokens conditioned on the preceding layout within a single sequence. By making musical planning explicit, this framework supports long-range structural organization and layout-level control. • Through evaluations of layout-conditioned generation, layout manipulation, and matched-data ablations, we provide evidence that MusicLayout can serve as an interpretable planning interface and improve long-range structural organization in text-to-music generation. 2 Related Work 2.1 Text-to-Music Generation Autoregressive music models predict discrete audio tokens, as exemplified by Jukebox, MusicLM, and MusicGen [6, 1, 4]. Diffusion-based systems instead synthesize waveforms, spectrograms, or compressed latents through iterative denoising [16, 31, 23]. AudioLDM 2 and MeLoDy combine LM-based semantic modeling with diffusion synthesis [20, 18]. Stable Audio 3 supports efficient variable-length generation, while ACE-Step 1.5 uses a hybrid architecture in which an LM performs high-level planning and a Diffusion Transformer (DiT) realizes the audio [10, 13, 28]. Although some systems incorporate high-level planning, temporal organization and arrangement remain implicit or coarsely represented. 2.2 Controllable Text-to-Music Generation Prior systems supplement global prompts with melody references, rhythm, dynamics, chords, drums, or other symbolic and audio conditions [1, 4, 39, 26, 34]. Seed-Music uses a symbolic lead-sheet pipeline [2]. ACE-Step 1.5 uses metadata and song-blueprint planning, but its blueprint remains implicit in the learned audio-token representations and is not exposed as an interpretable structural plan [13]. Other methods use individual controls, additional inputs, task-specific modules, or coarse metadata. MusicLayout instead exposes sections, textures, repetitions, variations, and instrument arrangements in an inspectable and adjustable time-aligned plan. 3 Methodology We build our model upon ACE-Step 1.5, extending its LM to generate a MusicLayout before producing layout-conditioned audio tokens. The original audio tokenizer and DiT remain frozen, with the latter rendering the tokens into a waveform. Figure 2: A MusicLayout example aligned to a shared timeline. Rows show sections, material families, textures, variations, and instrument arrangements. Family colors identify recurring material, and instrument entries give register, density, and energy. Blanks denote inactivity. For readability, rhythm, light, and strong abbreviate rhythm_driven, light_var, and strong_var, respectively. 3.1 MusicLayout Representation MusicLayout externalizes temporal musical organization as a structured, human-readable sequence that bridges high-level structural intent and audio-token generation. It follows three design principles: temporal alignment, hierarchical organization, and controllability. A MusicLayout combines piece-level structural relationships among musical materials with time-aligned segments, each corresponding to one musical section. At the piece level, MusicLayout organizes recurring musical materials and their relationships, including repetitions and variations across sections. At the segment level, each segment is associated with a time-aligned span and describes its section identity, texture evolution, and instrument participation. Its functional section label is selected from intro, verse, prechorus, chorus, bridge, breakdown, outro, transition, hook, solo, and build. Figure 2 illustrates these attributes along a shared timeline. MusicLayout is serialized as discrete tokens, allowing it to be directly modeled by an autoregressive LM. A family is a piece-local group of segments that share similar musical material. Labels such as fam_0 and fam_1 identify families only within the current piece. The same label may denote different musical material in another piece. Each segment is marked as unique, same, light_var, or strong_var. These values indicate, respectively, that the segment has no repeated family, closely repeats its family pattern, or departs from that pattern to a smaller or larger degree. Thus, the family colors and the variation row in Figure 2 jointly describe which material returns and how much it changes. Category Meaning layered Coexisting lead and supporting layers rhythm_driven Rhythm-section-centered arrangement melodic_front Foregrounded melodic material harmonic_bed Sustained harmonic support percussive Percussion-centered material build_up Accumulating or intensifying layers sparse_pulse Sparse intermittent activity lead_front Foregrounded lead material contrast Deliberately contrasting texture other Cases outside the listed categories Table 1: Texture categories in MusicLayout. The texture field gives a coarse description of how a musical segment is organized. Table 1 defines its vocabulary. Each segment contains a <roles> block listing its active instruments. Figures 1 and 2 use the form instrument:register:density:energy, for example drums:low:sparse:med. The instrument field uses a predefined vocabulary of 25 categories. Each instrument is followed by register (low/mid/high), density (sparse/med/dense), and energy (low/med/high), encoding pitch range, activity, and intensity, respectively. Figure 1 abbreviates the edited field as drums. We derived MusicLayout annotations from symbolic representations by extracting temporal sections, repetitions, variations, textures, and instrument arrangements. We rendered the symbolic music into audio and aligned each layout with its audio and text prompt to form prompt–layout–audio training triples. 3.2 Unified Layout Planning and Audio Generation Modeling. Given conditioning context c, the LM serializes MusicLayout sequence l before audio-token sequence a within a single autoregressive sequence: y=[c,<layout>,l,</layout>,a].y=[c, <layout>,l, </layout>,a]. (1) where <layout> and </layout> delimit the planning region. This ordering makes layout generation precede and condition audio-token generation. The corresponding joint distribution factorizes as p(l,a∣c)=p(l∣c)p(a∣c,l).p(l,a c)=p(l c)p(a c,l). (2) Here, p(l∣c)p(l c) describes layout planning from the conditioning context, while p(a∣c,l)p(a c,l) describes audio-token generation given the context and layout. Training. We train the LM on two complementary next-token tasks. Layout planning trains the model to generate l from c. Layout-conditioned audio generation trains it to predict a from c and a ground-truth (GT) MusicLayout: ℒplan=−∑tlog⁡p(lt∣c,l<t),ℒaudio=−∑ilog⁡p(ai∣c,l,a<i). array[]rclL_plan&=&- _t p(l_t c,l_<t),\\ L_audio&=&- _i p(a_i c,l,a_<i). array (3) We initially optimize layout planning to learn the MusicLayout schema and the schema-specific special tokens used to mark its fields and boundaries. We then alternate the tasks, applying the loss only to the corresponding layout or audio target span. Only the LM is updated. The pretrained audio tokenizer and DiT synthesis components remain frozen. Inference and control. In automatic inference, the LM generates a MusicLayout from the conditioning context and then continues with audio tokens. Alternatively, the human-readable layout can be inspected and adjusted before audio generation to change temporal organization or instrument arrangement. The revised layout and original context form the autoregressive prefix for audio-token generation, providing a pre-synthesis interface for layout-level control. 4 Experiments We evaluate generation, structural control, layout manipulation, ablations, and subjective quality against representative text-to-music systems. For our framework, we consider two layout conditions. In the reference-layout condition, the model receives a MusicLayout derived from the target music. This condition jointly tests whether MusicLayout captures the target’s musical organization and whether the model can use it to control the structure of the generated audio. In the generated-layout condition, the model predicts a layout from the text prompt before generating audio, evaluating the complete end-to-end text-to-music process. 4.1 Experimental Setup Datasets. We used three datasets with complementary roles. FreeMIDI [12] provided the training data and a disjoint in-domain evaluation set. MidiCaps [24] was used to evaluate generalization to out-of-domain MIDI, while MuChin [38] provided an out-of-domain evaluation on real audio. Their details are summarized in Table 2. Dataset Audio source Train Eval. FreeMIDI MIDI-synth. 24,474 2,719 MidiCaps MIDI-synth. – 1,040 MuChin Real, separated accomp. – 1,000 Table 2: Datasets used in our experiments. FreeMIDI. We collected a subset of FreeMIDI and retained only pieces longer than 15 seconds. For each remaining MIDI file, we extracted a MusicLayout and synthesized its audio, forming a layout–audio pair. We then provided the audio to MOSS-Music [27], an audio-understanding language model, to produce the generation prompt, yielding aligned prompt–layout–audio triples. MidiCaps. MidiCaps already provides natural-language captions. We selected a subset with a balanced distribution of style labels and removed MIDI files that overlap with FreeMIDI. For each remaining piece, we extracted a layout and synthesized audio from the MIDI, directly pairing both with its caption to form evaluation triples. MuChin. MuChin contains vocal music with Chinese captions but no MIDI, and thus cannot support the reference-layout condition. We used DeepSeek-V4-Flash [40] to translate the captions into English and remove vocal-related descriptions. We further separated the accompaniment from the vocals using a community-trained BS-RoFormer model [22] and used only the separated accompaniment for metric computation. Generation conditions that require an externally supplied layout—the reference-layout and shuffled-layout-inference conditions—are not reported on MuChin. Compared systems. We compared against MusicGen [4], ACE-Step 1.5 and ACE-Step XL-Turbo [13], and Stable Audio 3 Medium [10]. We evaluated our model under either the generated-layout or the reference-layout condition. To control for the effect of additional training data, we also finetuned ACE-Step 1.5 on the same data using the same LM adaptation setup as our model. We denote this matched-data no-layout control as ACE-Step 1.5-FT. It directly predicts audio from the conditioning context c without MusicLayout tokens, serving as both the primary fair comparison for explicit layout planning and the first step of the progressive ablation. Systems shared evaluation items, prompts, and seeds when supported, and baselines used officially recommended decoding settings. Systems without a layout representation targeted each item’s reference duration. For long-form MusicGen generation, we followed its official continuation mechanism. For our model, the generation duration is instead determined by the end time of the conditioning layout. In the generated-layout condition, this duration is therefore predicted by the model rather than specified externally, whereas the reference-layout condition matches the reference duration because its layout is derived from the target music. Evaluation protocol. We report Fréchet Audio Distance (FAD) [30] for acoustic distribution similarity, PaSST-KL [17] for sound-event agreement, SSIM [37] for local time–frequency similarity, and CLAPScore [7] for text–audio alignment. For long-range structure, SCM Energy Distance [5, 33] compares corpus-level structural complexity distributions, while F0.5F_0.5 and F3.0F_3.0 acoustic-boundary agreement [36] measure fine- and coarse-grained transition alignment within each piece. To examine how layout manipulation changes long-range organization, we also visualize recurrence-based self-similarity matrices (SSMs), following the structural visualization used in Stable Audio 2 and its underlying music structure analysis method [9, 32]. Implementation details. We finetuned ACE-Step 1.5’s 1.7B-parameter LM using six A800 GPUs, a 4,096-token context, and an effective batch size of 48. We selected the checkpoint with the lowest development layout-to-audio loss. Layout and audio decoding used temperature/top-p values of 0.8/0.950.8/0.95 and 0.9/0.950.9/0.95. 4.2 Results Overall Generation Performance Dataset Group System FAD↓ KL↓ SSIM↑ CLAP↑ SCM↓ F0.5F_0.5↑ F3.0F_3.0↑ FreeMIDI Baselines ACE-Step 1.5 2.874 0.789 0.159 0.284 0.117 0.637 0.831 ACE-Step XL-Turbo 2.584 0.734 0.181 0.300 0.063 0.619 0.811 MusicGen-Large 3.706 0.834 0.189 0.280 0.375 0.558 0.757 Stable Audio 3 Medium 3.014 0.838 0.213 0.378 0.181 0.616 0.809 Our model Reference layout 2.610 0.740 0.183 0.238 0.054 0.645 0.838 Generated layout 2.495 0.719 0.153 0.241 0.206 0.643 0.834 Matched-data ACE-Step 1.5-FT (no layout) 3.345 0.956 0.154 0.219 0.380 0.577 0.739 controls Shuffled-layout training 2.756 0.721 0.153 0.216 0.100 0.607 0.810 Shuffled-layout inference 2.636 0.779 0.176 0.230 0.038 0.642 0.831 MidiCaps Baselines ACE-Step 1.5 2.526 0.797 0.152 0.290 0.110 0.632 0.828 ACE-Step XL-Turbo 3.036 0.756 0.174 0.313 0.100 0.606 0.801 MusicGen-Large 2.488 0.825 0.180 0.258 0.379 0.545 0.746 Stable Audio 3 Medium 2.123 0.726 0.229 0.352 0.315 0.616 0.812 Our model Reference layout 2.149 0.712 0.169 0.305 0.087 0.635 0.836 Generated layout 2.303 0.699 0.140 0.287 0.804 0.630 0.830 Matched-data ACE-Step 1.5-FT (no layout) 2.740 0.877 0.153 0.234 0.351 0.560 0.736 controls Shuffled-layout training 2.772 0.734 0.144 0.267 0.136 0.599 0.807 Shuffled-layout inference 2.192 0.740 0.165 0.293 0.117 0.635 0.834 MuChin Baselines ACE-Step 1.5 2.480 0.685 0.129 0.289 0.186 0.588 0.797 ACE-Step XL-Turbo 1.994 0.617 0.147 0.317 0.611 0.566 0.779 MusicGen-Large 3.279 0.717 0.156 0.241 0.215 0.517 0.756 Stable Audio 3 Medium 2.007 0.662 0.177 0.365 0.132 0.573 0.794 Our model Generated layout 3.456 0.671 0.116 0.285 0.655 0.594 0.809 Matched-data ACE-Step 1.5-FT (no layout) 3.124 0.844 0.123 0.222 1.458 0.535 0.726 controls Shuffled-layout training 3.714 0.670 0.120 0.286 0.626 0.600 0.820 Table 3: Objective results. CLAP, KL, and SCM abbreviate CLAPScore, PaSST-KL, and SCM Energy Distance. Bold and underline mark the first- and second-ranked numerical values per dataset, with rounded ties bold. The three matched-data controls remove MusicLayout or mismatch it in training or inference. Table 3 summarizes the objective results. No single system dominates all aspects of generation. Our model with generated layouts records the lowest FAD and PaSST-KL on FreeMIDI. On MidiCaps, it also records the lowest PaSST-KL, suggesting that layout-conditioned generation can remain close to the reference acoustic-event distribution despite out-of-domain MIDI captions. Its lower CLAPScore relative to Stable Audio and ACE-Step XL-Turbo, however, shows that explicit planning may not uniformly improve global text–audio correspondence. The matched-data no-layout control provides a more direct test of the contribution of explicit layout planning. It is weaker than the original ACE-Step 1.5 on nearly all metrics. This degradation likely reflects the training-data difference: the original model was trained on high-quality real recordings, whereas our finetuning data consists of MIDI-synthesized audio with lower audio quality and timbral fidelity, as well as a different acoustic distribution. Since our model and this control share the same finetuning data and LM adaptation setup, their comparison isolates the effect of explicit layout planning. Relative to this control, the reference-layout condition has more favorable values on all seven metrics for FreeMIDI and MidiCaps. The generated-layout condition does so on six metrics for FreeMIDI and five for MidiCaps and MuChin. These results show that explicit layout planning substantially improves generation under matched-data conditions and can partly offset the limitations of weaker training audio. Overall, our model is competitive on the two MIDI-derived datasets, indicating that explicit layout planning contributes to overall music generation performance. Its FAD, PaSST-KL, and SSIM results on MuChin are less favorable, however. This difference likely arises from an acoustic-domain shift: our model is trained on MIDI-synthesized instrumental audio, whereas MuChin consists of accompaniments separated from real recordings. Structural Control The reference-layout condition records the highest boundary scores shown for FreeMIDI, the highest MidiCaps F3.0F_3.0, and a tie for the highest MidiCaps F0.5F_0.5. Because the reference layouts are derived from the target music, these results provide evidence that MusicLayout captures target organization and that conditioning on it can guide corresponding structure in the generated audio. With model-generated layouts, our model also shows strong structural organization without access to the target-derived layout used in the reference-layout condition. On FreeMIDI, both boundary scores are numerically above all baselines. On MidiCaps, F3.0F_3.0 is numerically above all baselines, while F0.5F_0.5 remains competitive. On MuChin, both boundary scores are numerically above all baselines, showing that this advantage persists under an acoustic-domain shift from MIDI-synthesized training data to accompaniments separated from real recordings. These results indicate that explicit intermediate planning improves the organization of long-range musical structure in end-to-end generation. The comparison with the matched-data no-layout control further isolates this structural benefit. Both layout conditions have higher boundary scores than the no-layout control wherever available, and the generated-layout condition retains this advantage across all three datasets. Thus, under the same MIDI-synthesized training data, explicit layout planning improves boundary agreement in both controlled and automatic generation. SCM Energy Distance measures agreement between dataset-level structural complexity distributions rather than song-level layout correctness. On the two MIDI-derived evaluation sets, our model in the reference-layout condition records the lowest observed SCM Energy Distance among the non-ablation systems , indicating that the specified layouts are realized with corpus-level structural complexity close to the references. On MuChin, the SCM Energy Distance in the generated-layout condition is comparatively high. This weaker agreement is consistent with the training–evaluation shift from MIDI-synthesized instrumental music to accompaniments separated from real vocal recordings, which have substantially different structural-complexity distributions. However, with training data controlled, the generated-layout condition has lower SCM than the no-layout condition. This comparison suggests a structural benefit from layout planning. Figure 3: Structural effects of layout manipulation for two FreeMIDI and two MidiCaps examples. Each row presents, from left to right, the target GT under Prompt A and Layout A, generation under the same prompt and reference layout, the layout-donor GT under Prompt B and Layout B, and generation from Prompt A paired with Layout B. For clear visualization, SSM intensities are normalized independently within each panel. Darker regions within a panel indicate stronger recurrence. Audio signals in each row are cropped to their common minimum duration. Structural Effects of Layout Manipulation Figure 3 presents four separate layout-manipulation cases, with two drawn from FreeMIDI and two from MidiCaps. These cases examine how layout manipulation changes the generated structure while the text prompt is held fixed. In each row, item A serves as the target and provides Prompt A and its reference Layout A. A different item B supplies Layout B, which we refer to as the donor layout. The SSM for item B is included to visualize the recurrence structure associated with Layout B. We refer to generation from Prompt A and Layout A as the matched generation, and generation from the same prompt and Layout B as the manipulated generation. In these examples, the matched generation shows recurrence patterns similar to the target reference, while the manipulated generation loses some target-aligned patterns and exhibits patterns closer to the donor reference. Because the manipulated generation combines the target prompt with the donor layout, it can retain structural characteristics associated with the target while adopting aspects of the donor’s organization. Together, these cases illustrate that MusicLayout can be manipulated before audio generation to control the structure of the generated music. Fine-Grained Regional Controls We further tested three direct edits to a single 240-second MusicLayout, holding the text prompt and sampling seeds fixed so that each controlled condition differed from the original only in the declared MusicLayout fields. Figure 4 compares the original and controlled audio over the affected interval. First, in the 0–11-second introduction, we replaced sparse, high-energy strings and synthesizer strings with dense, high-energy drums and sparse, high-energy bass. This edit was expected to add stronger low-frequency rhythmic content and broadband percussive transients. The controlled spectrogram shows both effects: energy below 250 Hz becomes dominant and repeated vertical transients appear across the band. Second, over 142–196 seconds, we changed every active role to sparse density and medium energy while leaving the segment boundaries, labels, families, and instrument identities unchanged. The expected result was a less dense and less intense realization of the same region. Relative to the original, the controlled excerpt has visibly reduced broadband activity and its RMS level decreases by 15% (1.4 dB). Third, we preserved the complete 0–91-second prefix and rewrote the remaining form from a largely repeated chorus sequence into a breakdown, bridge, build, solo, chorus, and outro, together with their associated roles and recurrence families. The expected result was a new long-range progression after the preserved prefix. The controlled spectrogram exhibits the intended succession of contrasting regimes, including the reduced texture of the breakdown and bridge, the subsequent build, and the denser solo and chorus. Thus, all three edits produce changes consistent with their specified local or regional controls. Figure 4: Paired log-mel spectrograms for three fine-grained MusicLayout controls: (a) role replacement over 0–11 seconds, (b) reduced density and energy over 142–196 seconds, and (c) a form rewrite over 91–240 seconds. Light-blue boxes use matched time–frequency coordinates: A1/A2 mark low-frequency and broadband changes, B marks reduced broadband activity, and C1–C3 mark successive stages of the form rewrite. Each pair shares its 0-dB reference and 80-dB range. Intensities are comparable within a row. Time axes retain full-song coordinates. Ablation Study The three matched-data controls in Table 3 form a progressive ablation of MusicLayout. First, the finetuned ACE-Step 1.5 control removes MusicLayout entirely. Second, shuffled-layout training introduces MusicLayout but pairs each training item with a layout from another item, so the model learns from mismatched layout–audio pairs. At inference, this model generates its own layout from the prompt before audio generation, following the same end-to-end procedure as the full generated-layout condition. Third, shuffled-layout inference restores correctly paired layouts during training but replaces the conditioning layout with one from another item at inference. Shuffled-layout training has favorable values on five of seven metrics relative to the no-layout control for each of the three datasets. This pattern suggests that layout-shaped supervision may retain a generic structural benefit even when item-level information is incorrect. On the two MIDI-synthesized datasets, our full model with generated layouts has favorable values on five of seven metrics relative to shuffled-layout training for both FreeMIDI and MidiCaps. Because both conditions generate layouts at inference, this pattern suggests that correct layout–audio correspondence during training provides information beyond the presence of layout-shaped tokens and can improve long-form music generation when the evaluation domain matches the type of audio used for training. On MuChin, the two conditions obtain broadly similar results. This may result from the acoustic-domain shift discussed above, since our model was trained on MIDI-synthesized audio rather than accompaniments from real recordings. Holding correctly paired training fixed, the reference-layout condition has favorable FAD, KL, SSIM, CLAPScore, and F3.0F_3.0 values relative to shuffled-layout inference on both MIDI-derived datasets. This indicates that item-matched layouts provide a more informative inference-time conditioning signal than a plausible but unrelated layout. A shuffled layout remains musically plausible because it comes from a real piece and retains valid sections, timing, repetitions, and arrangements. Shuffling can therefore preserve the corpus-level layout distribution while breaking item-level correspondence, so low SCM alone does not demonstrate correct layout control. 4.3 Subjective Evaluation All (N=59N=59) Exp. (N=36N=36) System F M T F M T GT 3.200 3.490 3.408 3.000 3.361 3.300 ACE 1.5 3.283 3.329 3.225 3.031 3.342 3.108 ACE-Step 1.5-FT 2.931 2.632 2.702 2.631 2.386 2.661 Ours-Ref 3.064 3.012 3.093 2.989 2.917 3.183 Ours-Gen 2.914 2.586 2.869 2.733 2.783 2.839 Table 4: Subjective ratings on FreeMIDI and MidiCaps (0–5). F/M/T denote fidelity, musical impression, and text consistency. Exp. denotes experienced listeners, and FT denotes matched-data finetuning. System Score ACE-Step 1.5 2.319 ACE-Step 1.5-FT 1.444 Ours (reference layout) 2.556 Ours (generated layout) 2.044 Table 5: GT-reference structural similarity rated by music-experienced participants on the same MIDI-derived items (0–5). To enable a direct comparison with the reference-layout condition, we excluded MuChin because it does not provide reference layouts. We sampled 10 items each from FreeMIDI and MidiCaps and assigned them repeatedly to 27 participants, including 15 with prior music experience. Participants rated audio fidelity, musical impression, and text consistency on a 0–5 scale, with one decimal place allowed. Each evaluation compared all five systems, yielding 59 ratings per system, including 36 ratings from experienced listeners. Table 4 shows that comparisons with the matched-data no-layout control vary with layout condition and listener experience. Across all participants, the reference-layout condition has higher observed scores on all three dimensions. Generated-layout scores are broadly comparable, with similar fidelity and musical impression but higher text consistency. Within the experienced subset, both layout conditions score numerically higher than the control across all dimensions. Notably, the reference-layout condition also has higher observed text consistency than the original ACE-Step 1.5 among experienced listeners (3.183 vs. 3.108), suggesting that an explicit MusicLayout can provide an additional control signal for steering music generation. This pattern may arise because music-experienced listeners are more attentive to structural organization when evaluating generated music. ACE-Step 1.5 has the highest observed fidelity score across the five systems (3.283), above GT (3.200). As discussed above, this difference is consistent with the acoustic-quality gap between the MIDI-synthesized data in our experiments and the real recordings used to train ACE-Step 1.5. It underscores the need for matched-data controls to isolate layout planning from training-data differences. Using the same dataset scope, we conducted a follow-up on a subset of the previously sampled items. A subset of the experienced participants was shown the GT reference for each item and asked to rate the structural similarity of the remaining outputs to the references. The reference-layout condition received a higher observed structural similarity than ACE-Step 1.5, providing subjective evidence that MusicLayout captures reference organization and carries it into generated audio. The generated-layout condition was slightly weaker because coarse dataset prompts do not uniquely specify the GT layout. Once predicted, the layout steers generation toward its own plausible structure. The objective structural metrics above suggest that these outputs can remain structurally organized even when they do not reproduce the particular GT structure. 5 Conclusion We presented MusicLayout, an explicit, time-aligned representation of section organization, development, and instrument arrangement for a unified autoregressive audio LM. Generated before audio tokens, it makes structural planning inspectable and adjustable. The same LM plans the layout and predicts audio tokens conditioned on it while the synthesis components remain frozen. Our results suggest that MusicLayout provides an interpretable interface that supports long-range structural organization and layout-level control. Despite these benefits, MusicLayout adjusts structural plans only before synthesis and cannot edit existing audio or regenerate selected regions. Our reliance on MIDI-synthesized training audio may also limit audio fidelity. Future work will pursue finer control and higher-quality, structurally aligned audio data. Acknowledgments This work was supported by Ant Group. Appendix A MusicLayout Representation and Annotation A.1 Serialization Grammar MusicLayout is serialized as a discrete text sequence with one optional piece-level family block and one required segment block. A family records a piece-local identifier, its member segments, its cardinality, and whether the members are adjacent. Each segment records an integer-second time span, a functional section label, a texture, an arrangement-change label, its family membership and family role, the degree of variation from the family prototype, a repetition flag, and a list of active instrument tuples. The grammar used in all reported experiments is summarized below: <layout> <families> <fam> id=fam_K members=sI,sJ count=N adj=true|false </fam> </families> <segments> <seg> id=sI time=A-B label=L texture=T change=C fam=fam_K fam_role=R variation=V repeat=true|false <roles> instrument:register:density:energy | ... </roles> </seg> </segments> </layout> Family identifiers are local to a piece rather than global semantic labels. For example, fam_0 denotes one recurring material within the current piece and may refer to unrelated material in another piece. The extraction pipeline retains singleton material as an explicitly declared one-member family with count=1 and adj=true. Its segment uses variation=unique and repeat=false. For a recurring family, the first chronological member serves as its variation prototype and is marked same. Later members are marked same, light_var, or strong_var according to their distance from that prototype. A.2 Closed Vocabularies Table 6 gives the structural vocabularies. The change field describes the segment’s arrangement-level relation to its predecessor, whereas fam_role describes its role in the piece-level recurrence organization. These fields are distinct from the functional section label. Field Allowed values Section label intro, verse, prechorus, chorus, bridge, breakdown, outro, transition, hook, solo, build Texture layered, rhythm_driven, melodic_front, harmonic_bed, percussive, build_up, sparse_pulse, lead_front, contrast, other Change entry, continuation, lift, drop, contrast, outro Family role intro_anchor, primary_repeat, secondary_repeat, adjacent_variant, outro_anchor, unique Variation unique, same, light_var, strong_var Register low, mid, high Density sparse, med, dense Energy low, med, high Table 6: Closed structural and arrangement vocabularies in MusicLayout. The instrument field uses 25 compact categories: acoustic_piano, electric_piano, plucked_keyboard, mallet_bell, organ, accordion_harmonica, acoustic_guitar, electric_guitar, bass, strings, orchestral, synth_strings, choir_voice, brass, sax, woodwind, synth_lead, synth_pad, synth_fx, world_plucked, percussion, reverse_cymbal, sound_fx, drums, and other. General MIDI program numbers are mapped deterministically to these categories. Instrument entries retain the segment-local register, density, and energy attributes. These attributes are defined at the instrument tuple level rather than duplicated as separate segment-level fields. A.3 Variation Strength For each recurring family, we compare every later segment with the first family member. Let drd_r be the Jaccard distance between active-role sets, and let ded_e, d_d, and dad_a be normalized differences in energy, note density, and active-role count. The base distance is d=0.45dr+0.20de+0.20d+0.15da.d=0.45d_r+0.20d_e+0.20d_d+0.15d_a. (4) The implementation adds small deterministic adjustments when coarse energy, density, texture, arrangement-change, repeat-neighbor, or source-relation evidence differs from the family prototype. We serialize scores at most 0.150.15 as same, scores in (0.15,0.40](0.15,0.40] as light_var, and larger scores as strong_var. This field therefore records an interpretable, rule-derived degree of variation rather than a human rating. A.4 Validation Before a generated layout can condition audio decoding, the parser checks the outer layout and segment wrappers, unique segment identifiers, nondecreasing integer-second start times, positive spans, valid closed-vocabulary values, four-field instrument tuples, family-member counts, and references from segments to declared families. It also rejects legacy bar spans, legacy role tags, segment-level energy/density fields, internal MIDI program slugs, and audio-code tokens inside the layout. A failed layout is not silently repaired or rewritten. A.5 Annotation Pipeline The annotation pipeline begins from aligned MIDI and rendered WAV files. It extracts bar-level activity and instrument statistics, combines change and repetition evidence into time-aligned segments, groups structurally related segments into piece-local material families, and derives the section, texture, change, family, variation, and instrument attributes serialized above. MIDI programs are used only as annotation evidence. The learning target remains audio-token generation. Segment spans are converted to audio time and rounded to integer seconds, with a minimum duration of one second after rounding. The extraction procedure is deterministic once its feature and boundary settings are fixed. Algorithm 1 summarizes its execution order. The symbolic representation supplies the structural evidence. The paired waveform is used to place the resulting spans on the rendered-audio timeline. Algorithm 1 MusicLayout extraction from aligned symbolic music and audio 1:MIDI performance M, aligned waveform duration D, and fixed extraction settings Θ 2:Valid serialized layout L, or rejection symbol ⊥ 3:function ExtractMusicLayout(M,D,ΘM,D, ) 4: (N,P,T,B,E)←ParseSymbolic(M)(N,P,T,B,E)← ParseSymbolic(M) ⊳ notes, programs, MIDI duration, bars, beats 5: for b∈Bb∈ B do 6: fb←AggregateBar(N,P,b)f_b← AggregateBar(N,P,b) ⊳ density, energy, pitch, and six broad-role activities 7: end for 8: q←AdjacentChangeScores(f)q← AdjacentChangeScores(f) 9: R←BinarySegmentationCandidates(f,D)R← BinarySegmentationCandidates(f,D) 10: q←AddBoundarySupport(q,R,f)q← AddBoundarySupport(q,R,f) 11: n←min⁡(14,max⁡(5,Round(D/28)))n← (14, (5, Round(D/28))) 12: C←SelectPeaks(q,mean⁡(q)+0.34std⁡(q),3bars,n)C← SelectPeaks(q,mean(q)+0.34std(q),3\ bars,n) 13: C←RefineLongSpans(C,f,B)C← RefineLongSpans(C,f,B) ⊳ adjacent-repeat and local-change tests 14: C←InsertMicroBoundaries(C,f,B)C← InsertMicroBoundaries(C,f,B) 15: S←ScaleBoundariesToAudio(C,B,T,D)S← ScaleBoundariesToAudio(C,B,T,D) 16: S←MergeShortSegments(S,8s)S← MergeShortSegments(S,8\ s) 17: S←RefineOpeningWithBeats(S,E,f,B,T,D)S← RefineOpeningWithBeats(S,E,f,B,T,D) 18: for si∈Ss_i∈ S do 19: zi←AggregateSegment(fb:b⊆si)z_i← AggregateSegment(\f_b:b s_i\) 20: ei←AggregateInstrumentStatistics(N,P,si)e_i← AggregateInstrumentStatistics(N,P,s_i) 21: end for 22: ℱ←∅F← 23: for i=1,…,|S|i=1,…,|S| do ⊳ chronological greedy centroid assignment 24: k∗←arg⁡maxk⁡cos⁡(zi,μk)k^*← _kcos(z_i, _k) 25: if ℱ=∅F= or cos⁡(zi,μk∗)<0.965cos(z_i, _k^*)<0.965 then 26: ϕ(i)←NewFamily(ℱ,zi)φ(i)← NewFamily(F,z_i) 27: else 28: ϕ(i)←k∗,μk∗←MemberMean(zj:ϕ(j)=k∗)φ(i)← k^*, _k^*← MemberMean(\z_j:φ(j)=k^*\) 29: end if 30: end for 31: for i,j∈1,…,|S|i,j∈\1,…,|S|\ do 32: Aij←cos⁡(zi,zj)A_ij (z_i,z_j) ⊳ relation evidence, not family assignment 33: end for 34: (ℓi,ti,ci,ui)i=1|S|←AssignDescriptors(S,z,A,ℱ,ϕ)( _i,t_i,c_i,u_i)_i=1^|S|← AssignDescriptors(S,z,A,F,φ) 35: for Fk∈ℱF_k do 36: pk←min⁡i:ϕ(i)=kp_k← \i:φ(i)=k\ 37: for i:ϕ(i)=ki:φ(i)=k do 38: vi←VariationClass(si,spk,Θvariation)v_i← VariationClass(s_i,s_p_k, _variation) 39: end for 40: end for 41: K←MapProgramsToCategories(P)K← MapProgramsToCategories(P) 42: for si∈Ss_i∈ S do 43: ri←QuantizeInstrumentTuples(ei,K)r_i← QuantizeInstrumentTuples(e_i,K) 44: end for 45: L←SerializeAndRound(ℱ,si,ℓi,ti,ci,ui,vi,rii=1|S|)L← SerializeAndRound(F,\s_i, _i,t_i,c_i,u_i,v_i,r_i\_i=1^|S|) 46: if ¬Validate(L) Validate(L) then 47: return ⊥ 48: end if 49: return L 50:end function Appendix B Dataset Construction and Provenance B.1 FreeMIDI FreeMIDI [12] supplies the training corpus and an in-domain evaluation set. We retained pieces longer than 15 seconds, extracted one MusicLayout from each retained MIDI file, and synthesized the aligned audio at 44.1 kHz using the FluidSynth [11] interface provided by PrettyMIDI [29], with MuseScore_General.sf2 [25]. Of 27,237 extracted items, 27,229 received a full-song caption from MOSS-Music-8B-Instruct [27]. Caption decoding was greedy. The prompt requested one concise English paragraph grounded only in the audio, covering style, audible instrumentation and roles, rhythm, texture, energy, musical development, and mood. It explicitly prohibited the use of filenames, MIDI metadata, external labels, bullet lists, and unsupported claims. Requiring at least 15 seconds of audio and a nonempty supervised audio-code span retained 27,198 items. Five additional items were removed because their complete planning sequences exceeded the 4,096-token context. The resulting 27,193 items were split deterministically into 24,474 training and 2,719 development items with seed 20260501. The development split is also the in-domain evaluation set. B.2 MidiCaps MidiCaps [24] is an out-of-domain MIDI evaluation set. We first sampled 1,100 items with seed 42 after requiring a duration of at least 120 seconds, at least two instruments, and a nonempty caption. Genre-quota sampling followed by instrument-diverse greedy selection covered 40 source genre labels. Because the labels are multi-valued, their frequencies need not sum to the number of items. The most frequent labels in the final set are electronic (581), pop (463), rock (208), classical (108), soundtrack (102), ambient (81), and jazz (59). Exact SHA-256 comparison of the MIDI files against the complete FreeMIDI corpus removed 60 overlaps, leaving 1,040 unique items. The source location and MIDI SHA-256 jointly define item identity. We then extracted MusicLayouts and synthesized aligned audio using the same renderer as for FreeMIDI. Original MidiCaps captions were retained as prompts, and no MidiCaps item was used for training. B.3 MuChin MuChin [38] provides an out-of-domain real-audio evaluation. Because it does not provide MIDI, it cannot supply a reference MusicLayout. We constructed a fixed 1,000-item manifest in the source metadata order. For each item, DeepSeek-V4-Flash [40] received the original full-song description as context and a set of non-vocal tags covering instrumentation, arrangement, style, tempo, rhythm, and mood. The instruction required one concise English paragraph using only these safe musical attributes and explicitly prohibited any mention or implication of singers, vocals, lyrics, rap, choir, humming, or spoken dialogue. Decoding used temperature 0.2 and top-p 0.9. We appended the same explicit instrumental-only control phrase to every final prompt. We separated accompaniment from vocals with the Python toolkit audio-separator v0.44.3 [3] and the BS-RoFormer [22] Viperx-1297 checkpoint [35], distributed as model_bs_roformer_ep_317_sdr_12.9755.ckpt. We used the resulting accompaniment as the metric reference. Reference-layout and inference-time shuffled-reference-layout conditions are consequently not applicable to MuChin. Dataset Train Evaluation Ref. layout FreeMIDI 24,474 2,719 Yes MidiCaps 0 1,040 Yes MuChin 0 1,000 No Table 7: Dataset roles and final item counts. Appendix C Model Adaptation and Training C.1 Training Sequences and Loss Masks The adapted model uses the 1.7B-parameter LM from ACE-Step 1.5 [13]. Layout planning and audio-token prediction share the same LM but use separate supervised target spans. In the planning task, the prompt and chat context are inputs and the MusicLayout tokens are targets. In the layout-to-audio task, the prompt and ground-truth MusicLayout form the input prefix and only the following audio-code tokens are targets. Chain-of-thought metadata is retained in the serialized prefix but receives zero loss weight. The pretrained 5-Hz audio tokenizer and DiT renderer [28] remain frozen in all stages. Stage 1 trains only the newly introduced vocabulary rows in the input embedding and LM head. Old vocabulary rows are masked, and no transformer layer or final normalization parameter is updated. Stage 2 updates the LM while alternating the planning and layout-to-audio tasks. The development layout-to-audio loss determines checkpoint selection. Setting Stage 1 Stage 2 Tasks Layout planning Layout planning and layout-to-audio Epoch limit 10 100 Learning rate 1.0×10−41.0× 10^-4 3.0×10−53.0× 10^-5 Monitored development task Planning Layout-to-audio Early-stopping patience 5 10 Trainable scope New embedding/LM-head rows LM Loss-bearing targets Layout span Layout or audio span for the sampled task Table 8: Stage-specific optimization protocol. The best Stage-2 checkpoint is restored according to development layout-to-audio loss. Training used six 80-GB GPUs, a maximum sequence length of 4,096, per-rank batch size 2, gradient accumulation 4, and effective batch size 48. Parameters and AdamW [21] optimizer states remained FP32, while FSDP [41] computation used BF16. Full-shard FSDP used original parameters, automatic wrapping, synchronized module states, limited all-gathers, and replicated vocabulary parameters for the Stage-1 row masks. The warmup ratio was 0.05 and the training random seed was 42. The data split used seed 20260501. C.2 Matched-Data Controls The no-layout ACE-Step 1.5-FT control uses the same filtered training items, audio codes, backbone LM, optimization precision, and adaptation budget as the full model, but directly predicts audio tokens without a MusicLayout prefix. The shuffled-layout-training control retains the MusicLayout-shaped prefix but pairs each audio target with a layout from another item during training. At inference it generates a layout from the prompt and then generates audio, as in the full generated-layout condition. The shuffled-layout-inference control uses the normally trained MusicLayout model but replaces the item-matched reference layout with a valid layout from another item. Thus, the three controls remove layout information, disrupt training-time correspondence, or disrupt inference-time correspondence, respectively. Both shuffled conditions use seed 20260711. For shuffled-layout training, the 24,474 training layouts form a one-to-one permutation of the same training items, with zero fixed points. The 2,719 development items remain unchanged. For shuffled-layout inference, FreeMIDI and MidiCaps are permuted independently within their evaluation sets. This yields 3,759 unique target–donor pairs and zero fixed points, preventing cross-dataset donors and self-matches. Donor layouts are truncated or extended at segment boundaries to end at the target item’s requested duration. Appendix D Baseline Selection and Exact Configurations D.1 Selection Scope We restricted the baseline comparison to models with publicly released weights and executable inference implementations. We further required compatibility with instrumental prompt-to-music generation and several-minute output, either through native variable-duration synthesis or through an established continuation mechanism. These criteria allow the reported systems to consume the same prompts and target durations and to be evaluated using the same audio and structural metrics. AudioLDM 2 [20] and Mustango [23] were considered but not included in the final comparison. Their released checkpoints and public inference protocols are centered on short clips and do not provide a validated overlapping-context continuation procedure for the several-minute setting used here. Concatenating independently generated diffusion samples would introduce a different and potentially discontinuous generation procedure. In contrast, the reported diffusion baseline, Stable Audio 3 Medium [10], supports the required duration natively. D.2 Executed Configurations Table 9 reports the configurations used in the final comparison. Each system targets the item’s manifest duration. ACE-Step-family and Stable Audio systems generate that duration natively, without concatenating independently generated clips. The corresponding model families are described by Gong et al. [13] and Evans et al. [10]. System Checkpoint/configuration Executed inference settings MusicGen-Large facebook/musicgen-large 30-s window, 10-s audio context, 20-s retained hop, 51.2 tokens/s, 1,536 first-window and 1,024 continuation tokens, base seed 28602 ACE-Step 1.5 acestep-v15-turbo, acestep-5Hz-lm-1.7B 8 inference steps, LM temperature/top-p 0.9/0.95, native target duration, base seed 38602 ACE-Step 1.5 XL-Turbo acestep-v15-xl-turbo, acestep-5Hz-lm-1.7B 8 inference steps, shift 3.0, LM temperature/top-p 0.9/0.95, native target duration, base seed 38602 Stable Audio 3 Medium stable-audio-3-medium 8 sampling steps, CFG 1.0, no negative prompt, FP16, chunked decoding, native target duration, base seed 48602 Table 9: Exact configurations of the externally pretrained baselines. Base seeds are combined with the global item index. Both ACE-Step configurations use the released LM-assisted inference path: the LM first generates chain-of-thought metadata and audio semantic codes, which are subsequently rendered by the DiT. We therefore treat ACE-Step 1.5 and ACE-Step 1.5 XL-Turbo as hybrid LM–DiT baselines rather than pure diffusion baselines. Stable Audio 3 Medium is the diffusion-only baseline in this comparison. For MusicGen-Large [4], the first window is generated from the text prompt alone. Each later window receives the same text and the final 10 seconds of the accumulated waveform as audio context, generates at most 1,024 new codec tokens, and retains at most 20 seconds of new audio. The window seed is the item seed plus the zero-based window index. The concatenated result is cropped only at the final target duration. Appendix E Inference and Evaluation Protocol E.1 MusicLayout Inference Automatic inference first decodes the chain-of-thought metadata and MusicLayout, validates the complete layout, and then continues with 5-Hz audio-code tokens. Only the outer special-token boundaries are constrained. the decoded metadata and layout body are not inserted, replaced, or rewritten. Planning uses temperature/top-p 0.8/0.95, while audio-code decoding uses 0.9/0.95. In the generated-layout condition, the final endpoint of the validated layout determines both the audio-code target count and renderer duration. A layout ending at D seconds therefore requests 5D5D audio-code tokens. Reference- and shuffled-layout conditions likewise use the endpoint of the supplied effective layout. Systems without a layout representation target the reference manifest duration. E.2 Audio Standardization and Pairing Every metric consumes a non-destructively normalized copy of each waveform. Audio is converted to mono, 44.1-kHz PCM16 and normalized with two-pass EBU R128 [8] to −14-14 LUFS with a −1-1 dBTP true-peak ceiling. FAD and CLAPScore use the complete normalized generated clip. PaSST-KL, SSIM, and acoustic-boundary agreement operate on filename-matched generated/reference pairs cropped from time zero to their exact common-minimum duration. No waveform is time-stretched, looped, or zero-filled to imitate missing musical content. E.3 Metric Implementations FAD [30] compares VGGish [15] embedding distributions. PaSST-KL [17] uses non-overlapping 10-second windows and averages KL(pref∥pgen)KL(p_ref\|p_gen) across aligned windows. SSIM [37] is computed over paired mel spectrograms. CLAPScore [7] averages prompt–audio cosine similarity over non-overlapping 10-second windows, zero-padding only the last partial window. For structural complexity, each clip is represented by the 26-dimensional SCM descriptor vector [5]. Within a dataset, the reference and generated vectors are standardized before computing the empirical Energy Distance [33]: ℰ(R,G)= (R,G)= 2nm∑i=1n∑j=1m∥i−j∥2 2nm _i=1^n _j=1^m _i-g_j _2 −1n2∑i,i′=1n∥i−i′∥2 - 1n^2 _i,i =1^n _i-r_i _2 −1m2∑j,j′=1m∥j−j′∥2. - 1m^2 _j,j =1^m _j-g_j _2. (5) SCM Energy Distance measures agreement between corpus-level structural complexity distributions. It is not an item-level measure of whether a system followed the correct layout. Acoustic-boundary agreement first extracts time–frequency features, clusters frames with fixed k=6k=6, and obtains a boundary sequence for both reference and generated audio. The reported F0.5F_0.5 and F3.0F_3.0 use 0.5- and 3-second tolerances with the standard segment-boundary evaluation protocol [36]. Unlike SCM, these scores compare transitions within paired items, although they do not identify which MusicLayout field caused a boundary. Appendix F Descriptive Plan and Output Audits The main results measure the realized audio rather than treating a textual layout as correct merely because it parses. Before audio decoding, we parsed and schema-validated every sampled layout. An invalid sample was discarded and sampling was repeated within a finite retry budget, stopping as soon as a valid layout was obtained. This procedure produced a valid MusicLayout for every requested evaluation item: all 2,719 FreeMIDI, 1,040 MidiCaps, and 1,000 MuChin items were successfully validated and rendered. Thus, no evaluation item was removed because of layout invalidity or rendering failure. Table 10 summarizes the resulting layout lengths. these descriptive checks are not substitutes for audio evaluation. Statistic FreeMIDI train FreeMIDI generated MidiCaps generated MuChin generated Items 24,474 2,719 1,040 1,000 Segments 11.31 9.73 8.38 8.66 Families 2.28 1.67 1.50 1.50 Recurring families 1.63 1.37 1.28 1.21 Endpoint (s) 225.34 187.44 163.36 170.01 Table 10: The FreeMIDI train column reports statistics of the MusicLayouts obtained by applying Algorithm 1 to the FreeMIDI training split. The other three columns summarize valid MusicLayouts generated for evaluation. Except for item counts, entries are per-item means. A recurring family contains at least two segments. Endpoint is the final serialized segment boundary. Appendix G Structural Evaluation and Layout Manipulation G.1 Recurrence Visualization The self-similarity matrices in the paper are recurrence visualizations rather than scalar model-ranking metrics. The analysis uses 12-bin harmonic pitch class profiles, a 209-ms analysis window, a 139-ms hop, delay-coordinate embedding over approximately three seconds, and a mutual 4% nearest-neighbor recurrence rule following the structure-analysis pipeline of Serra et al. [32]. Within each target–donor case, all signals are cropped from time zero to their common minimum duration before SSM computation. Display intensity is used to make recurrence patterns legible. It does not enter any quantitative score. G.2 Cross-System Recurrence Examples Figure 5 expands the recurrence visualization to the systems in the main objective comparison. It shows two cases from each of the three evaluation datasets (six rows in total). Each generated example has the same duration as its corresponding ground-truth audio. Within each row, all available signals are cropped from time zero to the common minimum duration before recurrence analysis. The reference-layout and shuffled-layout conditions are unavailable for MuChin and are marked N/A. MusicGen-Large primarily exhibits near-diagonal or locally repeated patterns and less consistently recovers the distant off-diagonal organization visible in the ground truth. This behavior is consistent with its overlapping-context generation protocol: a later window receives only the retained tail of the preceding audio, so patterns from much earlier windows are no longer directly available as context. Its local continuity therefore does not by itself preserve long-range recurrence. On the MIDI-derived rows, the reference-layout condition most closely preserves the salient block and stripe organization of the corresponding ground-truth SSMs. This provides qualitative evidence that MusicLayout captures musically relevant recurrence structure and that the layout-conditioned renderer can realize that structure in audio. Across the six cases, the generated-layout condition is less closely aligned with the specific reference, as expected when the layout is predicted from text rather than extracted from the target piece, but it still shows repeated blocks and off-diagonal patterns spanning the generated piece. These examples therefore support reasonable autonomous long-range organization without treating the SSM visualization as a scalar model ranking. G.3 Cropping Details for Layout Manipulation Table 11 reports the crop durations used for the four rows of the layout-manipulation SSM in Figure 3. For each row, the duration is the minimum available duration among the four signals shown in that row. All signals are cropped from time zero without temporal alignment, time stretching, or padding. This operation only establishes a shared visualization interval. It does not modify the underlying recurrence computation or enter the objective metrics. Case Common duration (s) FreeMIDI 1 224.4 FreeMIDI 2 194.6 MidiCaps 1 137.4 MidiCaps 2 179.716 Table 11: Common-minimum crop durations for the layout-manipulation SSM rows in Figure 3. Figure 5: Cross-system recurrence SSMs for six cases, with two cases each from FreeMIDI, MidiCaps, and MuChin. Darker cells indicate stronger recurrence within a panel. Following the visualization in the main article, intensities are normalized independently within each panel and therefore should not be compared as absolute recurrence density across systems. Every row is cropped to a common duration. Each cell summarizes the mean density of a 16×1616× 16 block of the underlying binary recurrence matrix, corresponding to approximately 2.23 seconds. A monotonic power-law display transform (γ=0.45γ=0.45) improves the visibility of sparse recurrence patterns without altering the recurrence matrices. N/A denotes a condition not available for the MuChin evaluation. Appendix H Discussion Beyond explicit musical planning, MusicLayout shows how structured representations can amplify the value of incremental data for music audio generation. MIDI expands the data available to an audio model, while extracting layouts further unlocks its temporal and arrangement information. The resulting prompt–layout–audio triples turn the same incremental corpus into richer supervision, improving generation while enabling structure-aware planning and control. Thus, the value of additional data depends not only on its scale, but also on how its latent structure is represented and exploited. Appendix I Qualitative Scope and Limitations MusicLayout is a pre-synthesis planning interface. It can change the prefix that conditions a new waveform, but it does not directly edit an existing waveform or regenerate a selected region while preserving all other samples. The categorical instrument representation describes arrangement-level source classes rather than exact timbres, performances, or production effects. A valid layout is therefore a well-formed structural request, not a guarantee that every specified attribute will be realized perfectly in audio. The model is trained on MIDI-synthesized instrumental audio. This provides aligned symbolic structure at scale but limits acoustic diversity and fidelity relative to models trained primarily on studio recordings. It also creates a domain shift for MuChin, whose evaluation references are accompaniments separated from real vocal recordings. Consequently, a syntactically valid layout may still yield audio in which a requested transition, recurrence, or instrument entry is weak or unclear. References [1] A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi, et al. (2023) MusicLM: generating music from text. arXiv preprint arXiv:2301.11325. Cited by: §1, §2.1, §2.2. [2] Y. Bai, H. Chen, J. Chen, Z. Chen, Y. Deng, X. Dong, L. Hantrakul, W. Hao, Q. Huang, Z. Huang, et al. (2024) Seed-music: a unified framework for high quality and controlled music generation. arXiv preprint arXiv:2409.09214. Cited by: §1, §2.2. [3] A. Beveridge (2026) Audio Separator, version 0.44.3. Note: https://github.com/nomadkaraoke/python-audio-separator/releases/tag/v0.44.3Accessed July 30, 2026 Cited by: §B.3. [4] J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y. Adi, and A. Défossez (2023) Simple and controllable music generation. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: §D.2, §1, §2.1, §2.2, §4.1. [5] J. De Berardinis, A. Cangelosi, and E. Coutinho (2022) Measuring the structural complexity of music: from structural segmentations to the automatic evaluation of models for music generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing 30, p. 1963–1976. External Links: Document Cited by: §E.3, §4.1. [6] P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever (2020) Jukebox: a generative model for music. arXiv preprint arXiv:2005.00341. Cited by: §2.1. [7] B. Elizalde, S. Deshmukh, M. A. Ismail, and H. Wang (2023) CLAP: learning audio concepts from natural language supervision. In 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. External Links: Document Cited by: §E.3, §4.1. [8] European Broadcasting Union (2023) EBU R 128: loudness normalisation and permitted maximum level of audio signals. Note: https://tech.ebu.ch/docs/r/r128.pdfVersion 5.0 Cited by: §E.2. [9] Z. Evans, J. D. Parker, C. J. Carr, Z. Zukowski, J. Taylor, and J. Pons (2024) Long-form music generation with latent diffusion. arXiv preprint arXiv:2404.10301. Cited by: §4.1. [10] Z. Evans, J. D. Parker, M. Rice, C. J. Carr, Z. Zukowski, J. Taylor, and J. Pons (2026) Stable audio 3. arXiv preprint arXiv:2605.17991. Cited by: §D.1, §D.2, §1, §2.1, §4.1. [11] FluidSynth Development Team (n.d.) FluidSynth: a software synthesizer based on the SoundFont 2 specifications. Note: https://github.com/FluidSynth/fluidsynthAccessed July 30, 2026 Cited by: §B.1. [12] FreeMIDI (n.d.) Free MIDI. Note: https://freemidi.org/Accessed July 23, 2026 Cited by: §B.1, §4.1. [13] J. Gong, Y. Song, W. Zhao, S. Wang, S. Xu, J. Guo, and X. Yang (2026) ACE-Step 1.5: pushing the boundaries of open-source music generation. arXiv preprint arXiv:2602.00744. Cited by: §C.1, §D.2, §1, §1, §2.1, §2.2, §4.1. [14] R. He, B. Cheng, Y. Ma, Q. Jia, S. Liu, A. Ma, X. Wu, L. Wu, D. Leng, and Y. Yin (2025) PlanGen: towards unified layout planning and image generation in auto-regressive vision language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 18143–18154. Cited by: §1. [15] S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. J. Weiss, and K. Wilson (2017) CNN architectures for large-scale audio classification. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 131–135. External Links: Document Cited by: §E.3. [16] Q. Huang, D. S. Park, T. Wang, T. I. Denk, A. Ly, N. Chen, Z. Zhang, Z. Zhang, J. Yu, C. Frank, et al. (2023) Noise2Music: text-conditioned music generation with diffusion models. arXiv preprint arXiv:2302.03917. Cited by: §2.1. [17] K. Koutini, J. Schlüter, H. Eghbal-zadeh, and G. Widmer (2022) Efficient training of audio transformers with patchout. In Proceedings of the 23rd Annual Conference of the International Speech Communication Association (INTERSPEECH 2022), p. 2753–2757. Cited by: §E.3, §4.1. [18] M. W. Lam, Q. Tian, T. Li, Z. Yin, S. Feng, M. Tu, Y. Ji, R. Xia, M. Ma, X. Song, et al. (2023) Efficient neural music generation. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: §2.1. [19] S. Li, S. Ji, Z. Wang, S. Wu, J. Yu, and K. Zhang (2026) A survey on music generation from single-modal, cross-modal, and multi-modal perspectives. ACM Computing Surveys 58 (11), p. 1–38. External Links: Document Cited by: §1. [20] H. Liu, Y. Yuan, X. Liu, X. Mei, Q. Kong, Q. Tian, Y. Wang, W. Wang, Y. Wang, and M. D. Plumbley (2024) AudioLDM 2: learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing. Cited by: §D.1, §1, §2.1. [21] I. Loshchilov and F. Hutter (2019) Decoupled weight decay regularization. In International Conference on Learning Representations, External Links: Link Cited by: §C.1. [22] W. Lu, J. Wang, Q. Kong, and Y. Hung (2024) Music source separation with band-split rope transformer. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 481–485. External Links: Document Cited by: §B.3, §4.1. [23] J. Melechovsky, Z. Guo, D. Ghosal, N. Majumder, D. Herremans, and S. Poria (2024) Mustango: toward controllable text-to-music generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Mexico City, Mexico, p. 8293–8316. External Links: Document Cited by: §D.1, §2.1. [24] J. Melechovsky, A. Roy, and D. Herremans (2024) MidiCaps: a large-scale MIDI dataset with text captions. In Proceedings of the 25th International Society for Music Information Retrieval Conference, Cited by: §B.2, §4.1. [25] MuseScore (2020) MuseScore General SoundFont. Note: https://ftp.osuosl.org/pub/musescore/soundfont/MuseScore_General/Version 0.2, accessed July 30, 2026 Cited by: §B.1. [26] Z. Novack, J. McAuley, T. Berg-Kirkpatrick, and N. J. Bryan (2024) DITTO: diffusion inference-time t-optimization for music generation. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 38426–38447. Cited by: §1, §2.2. [27] OpenMOSS Team (2026) MOSS-Music technical report. Note: https://github.com/OpenMOSS/MOSS-MusicGitHub repository Cited by: §B.1, §4.1. [28] W. Peebles and S. Xie (2023) Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 4195–4205. Cited by: §C.1, §2.1. [29] C. Raffel and D. P. W. Ellis (2014) Intuitive analysis, creation and manipulation of MIDI data with pretty_midi. In Late-Breaking and Demo Session of the 15th International Society for Music Information Retrieval Conference, External Links: Link Cited by: §B.1. [30] D. Roblek, K. Kilgour, M. Sharifi, and M. Zuluaga (2019) Fréchet audio distance: a reference-free metric for evaluating music enhancement algorithms. In Proceedings of the 20th Annual Conference of the International Speech Communication Association (INTERSPEECH 2019), p. 2350–2354. Cited by: §E.3, §4.1. [31] F. Schneider, O. Kamal, Z. Jin, and B. Schölkopf (2024-08) Moûsai: efficient text-to-music diffusion models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, p. 8050–8068. External Links: Document Cited by: §2.1. [32] J. Serrà, M. Müller, P. Grosche, and J. L. Arcos (2014) Unsupervised music structure annotation by time series structure features and segment similarity. IEEE Transactions on Multimedia 16 (5), p. 1229–1240. External Links: Document Cited by: §G.1, §4.1. [33] G. J. Székely and M. L. Rizzo (2013) Energy statistics: a class of statistics based on distances. Journal of Statistical Planning and Inference 143 (8), p. 1249–1272. External Links: Document Cited by: §E.3, §4.1. [34] O. Tal, A. Ziv, I. Gat, F. Kreuk, and Y. Adi (2024) Joint audio and symbolic conditioning for temporally controlled text-to-music generation. arXiv preprint arXiv:2406.10970. Cited by: §1, §2.2. [35] TRvlvr (n.d.) BS-RoFormer viperx-1297 checkpoint. Note: https://github.com/TRvlvr/model_repo/releases/download/all_public_uvr_models/model_bs_roformer_ep_317_sdr_12.9755.ckptAccessed July 30, 2026 Cited by: §B.3. [36] D. Turnbull, G. R. G. Lanckriet, E. Pampalk, and M. Goto (2007) A supervised approach for detecting boundaries in music using difference features and boosting. In Proceedings of the 8th International Conference on Music Information Retrieval, ISMIR 2007, Vienna, Austria, p. 51–54. Cited by: §E.3, §4.1. [37] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli (2004) Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13 (4), p. 600–612. Cited by: §E.3, §4.1. [38] Z. Wang, S. Li, T. Zhang, Q. Wang, P. Yu, J. Luo, Y. Liu, M. Xi, and K. Zhang (2024) MuChin: a chinese colloquial description benchmark for evaluating language models in the field of music. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, p. 7771–7779. External Links: Document Cited by: §B.3, §4.1. [39] S. Wu, C. Donahue, S. Watanabe, and N. J. Bryan (2024) Music controlnet: multiple time-varying controls for music generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing 32, p. 2692–2703. Cited by: §1, §2.2. [40] A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, et al. (2026) DeepSeek-V4: towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348. Cited by: §B.3, §4.1. [41] Y. Zhao, A. Gu, R. Varma, L. Luo, C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, A. Desmaison, C. Balioglu, P. Damania, B. Nguyen, G. Chauhan, Y. Hao, A. Mathews, and S. Li (2023) PyTorch FSDP: experiences on scaling fully sharded data parallel. Proceedings of the VLDB Endowment 16 (12), p. 3848–3860. External Links: Document Cited by: §C.1.

Instant research discovery