Paper deep dive
Masked diffusion enables coherent beat tracking
Francesco Foscarin, Filip Korzeniowski, Richard Vogl
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Current neural networks for beat tracking generate invalid outputs, such as consecutive downbeats and erratic tempo changes, even when these are not present in the training data. Heavy post-processing techniques can alleviate these problems, but the original cause of this inconsistent behaviour remains unknown. We hypothesise that it stems from inadequate modelling of multiple plausible output beat grids, resulting in an invalid mixture of competing interpretations. We propose a masked diffusion approach that properly models multiple outputs and enables the model to build coherent predictions through iterative inference. We devise three modifications to standard masked diffusion that enable its application to beat tracking: independent masking of beats and downbeats during training and inference, a balanced masking scheduler for inference, and peak-picking across inference steps. Our approach reduces erratic behaviours and improves beat-tracking performance.
Tags
Links
- Source: https://arxiv.org/abs/2608.04624v1
- Canonical: https://arxiv.org/abs/2608.04624v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
37,618 characters extracted from source content.
Expand or collapse full text
Masked diffusion enables coherent beat tracking Abstract Current neural networks for beat tracking generate invalid outputs, such as consecutive downbeats and erratic tempo changes, even when these are not present in the training data. Heavy post-processing techniques can alleviate these problems, but the original cause of this inconsistent behaviour remains unknown. We hypothesise that it stems from inadequate modelling of multiple plausible output beat grids, resulting in an invalid mixture of competing interpretations. We propose a masked diffusion approach that properly models multiple outputs and enables the model to build coherent predictions through iterative inference. We devise three modifications to standard masked diffusion that enable its application to beat tracking: independent masking of beats and downbeats during training and inference, a balanced masking scheduler for inference, and peak-picking across inference steps. Our approach reduces erratic behaviours and improves beat-tracking performance. 1 Introduction Figure 1: A particularly problematic output of the Beat This system [12] from the GTZAN rock_00029. It contains erratic beat tempo doubling/halving and consecutive downbeats. The task of audio beat and downbeat tracking has been deeply explored for decades, and in recent years [27, 4, 12, 25, 26, 13, 30, 28, 17, 20, 11], and a variety of architectures, losses, data augmentation techniques, pre-training and fine-tuning techniques have been proposed. All these works use neural networks (NNs) in a one-step approach: for each temporal frame, a beat activation is predicted in a single forward pass. However, this approach implicitly assumes that, given an input, there is exactly one valid output. When the musical content is ambiguous, the model might produce predictions that mix different solutions and do not correspond to any single coherent interpretation. Beat tracking suffers from this problem; for some songs, listeners may adopt different metrical levels, and alternative, stylistically plausible beat locations are possible [19, 16, 5]. This ambiguity is also partially accounted for in beat evaluation metrics [10]. For such problematic songs, we can clearly observe the aforementioned incoherent behaviour in the beat trackers N predictions, even those developed specifically to work without a heavy post-processing phase, like the Beat This model [12] (see Figure 1 for an example). Post-processing techniques, such as the Dynamic Bayesian Network (DBN) [3], fix erratic outputs by enforcing strict and simplistic constraints, such as a fixed number of beats per bar and limited tempo variations. While this alleviates symptoms, it does not address the underlying cause in the N, which can degrade performance in both non-DBN and DBN systems (e.g., by metrical level switching [7]).111Moreover, the DBN post-processing is inherently unable to handle complex music with a variety of time signatures, time signature changes, and sudden severe tempo variations [12]. Since the DBN masks the issue we want to address, we focus on a non-DBN approach in this paper. The proposed solution to incoherent N beat predictions is a model that supports multiple valid interpretations and consistently selects one at inference time. Autoregressive models achieve this by generating outputs step by step, thereby modelling the joint distribution over the full output sequence. However, implementing an autoregressive beat tracker requires a set of architectural choices, each with its own limitations. A frame-wise autoregressive decoder would require one prediction per frame, making inference very slow (e.g., 1500 passes for 30 seconds at 50 fps). A more compact [beat-type, beat-time] tokenisation avoids this, but then absolute-time encoding222This is the approach of the Whisper [23] speech transcription system, and Murgul and Heizmann[21] MIDI beat tracker. requires many time tokens and does not work for longer excerpts, while relative-time encoding is prone to error accumulation during generation [21]. We take a different approach to let the model consider multiple output variants while avoiding the drawbacks mentioned above: it requires only a few forward passes, maintains the unproblematic frame-wise encoding of the output, and requires only minor modifications to well-tested beat-tracking networks. We adopt a Masked Diffusion Model (MDM), which accepts not only audio as input but also partial output sequences (i.e., beats and downbeats in our case). During inference, the full output is built by iteratively taking partial predictions along with the audio as input and producing additional predictions (see Figure 3). This way, the model can select a valid output during the early steps and then coherently fill the rest of the output sequence. While MDM is widely used with minimal variation across deep learning domains such as language and images, its adaptation for beat tracking poses specific challenges. 1) the output label imbalance: the frame-wise annotations mainly consist of non-events with a few sparse beat and downbeat events (the ratio is â90â 90 to 11 for downbeats at 50 fps on our training data); 2) the multi-task problem formulation: we are simultaneously classifying beats and downbeats; 3) the lack of temporal precision for beat/downbeat, which cause a network trained with impulse-like peaks to produce wide peak predictions. In this paper, we describe the fundamental design choices required to make the MDM approach work for beat tracking and show that MDM drastically reduces erratic behaviour and substantially improves performance, especially on metrics that are heavily influenced by it. 2 Related work The backbone of this work is the Beat This model from Foscarin et al. [12], whose entire weights, training code, and data are open-source. We use the same data, augmentation methods, metrics, and most hyperparameters, and adapt their model and their shift-tolerant weighted loss to work with masked diffusion. GagnerĂ© et al. [13] is the only N-based beat-tracking paper to consider the aforementioned issue of multiple valid outputs in their approach. They use multiple heads in their model, which aim to capture different hierarchical levels. However, this idea is only used during their self-supervised pretraining stage; during fine-tuning and inference, they use a standard single-head architecture for beat and downbeat prediction, so their predictions still suffer from the problems we described in the introduction. We still include this model in our evaluation as one of the high-performing systems. Two other papers with high performance are by Ru et al. [25, 26], who present fine-tuning techniques to adapt large pretrained audio models for beat tracking. We consider the latter paper in our comparison as it presents, on average, slightly better results. From the MDM literature, we draw heavily on LLaDA by Nie et al. [22], which presents a simple, easy-to-adapt framework that achieves strong results on large models, comparable to those of autoregressive methods. We adapt their training and inference techniques to address the specific challenges of our beat-tracking scenario.333For readers more familiar with the MaskGIT approach [6], which was also applied to spectrogram generation [9], the two approaches are closely related [31]. We chose the LLaDA MDM mostly because it has a simpler implementation that uses linear masking during both training and inference. 3 Method Given a sequence x, generative models aim to approximate the true but unknown distribution pdâaâtâaâ(x)p_data(x) by optimising the distribution pΞâ(x)p_Ξ(x) parametrised by a model with parameters Ξ. MDMs define this distribution through a forward process that gradually masks positions and a reverse process that recovers them [2]. Since our beat tracking data are not just sequences, but a set of pairs (x,y)(x,y), where x is the input audio, and y is the beat sequence, we need to reformulate the distribution above as the conditional distribution pΞâ(y|x)p_Ξ(y|x). During the forward and backward processes, only y is masked and unmasked, while x remains fully visible at all steps. In the following, we describe our model, training, and inference setup, and compare them with both existing language MDMs and beat trackers. 3.1 Model As mentioned in Section 2, this work is based on the Beat This architecture [12], which processes a 30-second log-mel-spectrogram input with a frontend block mixing convolutions and directional time-frequency attention, and then passes it through a 6-layer RoFormer. In this section, we describe the modifications required to turn it into a Masked Diffusion Model. One of our objectives is to keep the required architectural changes minimal, enabling us to leverage previous research on architectures, losses, hyperparameter optimisation, and post-processing techniques. Figure 2: Masked diffusion adaptation of the Beat This model (yellow blocks). We also display the path to reuse previous predictions during subsequent inference steps. 3.1.1 Token Embeddings We add two learnable embedding tables, one for beats and one for downbeats, each mapping a vocabulary of four token types MASK, NO_EVENT, EVENT, PAD to the transformer dimension d. While it is possible to have a single table for both beats and downbeats by splitting the EVENT token into two BEAT and DOWNBEAT tokens, it would require restructuring our problem from two binary predictions to a multi-class prediction. This would make it difficult to train on datasets that lack downbeat annotations, and require a new loss, since the one from [12] is only binary. For these reasons, we keep the two binary beat and downbeat predictions separate. Unlike language MDMs, our model must handle two inputs from different modalities: audio and beats/downbeats. Assuming that audio requires pre-processing to extract meaningful features, we dedicate the frontend to audio only, whereas beat and downbeat embeddings are added directly before the 6-layer RoFormer (see Figure 2). Before adding, we rescale them by d d, where d is the dimension of the transformer hidden layer. This increases the importance of the beats input over the audio, and proved simple and effective in preliminary experiments. 3.1.2 Output Head The original Beat This model uses a sum head: a single linear layer that projects the transformer output into two channels (beat and downbeat logits), with the final beat channel computed as the sum of the two. This helps predict a beat for every downbeat. However, we observed that this sometimes creates artifacts such as inverted beat spikes in the downbeat logits. While this does not affect one-step prediction, in our case, the masking scheduler relies on the logit confidences for remasking. To avoid potential issues, we simplify the output to two independent linear heads (one per channel) and ensure to predict a beat for every downbeat during the unmasking process instead. 3.1.3 Activation Function We substitute GELU activations with SwiGLU [29] in the feed-forward network (FFN) (an inverted bottleneck that projects from d to 4âd4d, computes activations, and projects back to d) after each attention block. SwiGLU uses half of its input dimension as a learnable gate. To maintain a comparable parameter count, we reduce the FFN dimension to 2/3, as suggested by the original paper. Best practice (e.g., [8]) recommends rounding up the hidden dimension to a power of 2 to keep GPU processing efficient (we choose 32 due to our small d). We keep the GELU between convolutions in the frontend. Using SwiGLU is not necessary for our MDM to work, but it aligns our architectural design with well-tested modern networks and improves our results, at no extra complexity. Figure 3: First 10 (of 30) seconds of a 3-step inference on GTZAN pop_00050. The transformer outputs probabilities for beats and downbeats, and at each step, the masking scheduler unmasks 1/3 of the most confident positive predictions, along with a balanced number of negative predictions (see Section 3.3.1). These are processed by the Peak Picking (P) module (see Section 3.3.3) and fed into the transformer (along with the audio) for the next prediction step. 3.2 Training We train with the standard masked diffusion objective of MDMs [22]. Given a ground-truth sequence y of length N, at each training step we sample a masking probability m uniformly from the interval [Ï”,1][Δ,1], where ϔΔ is a small number to ensure training stability. Every frame yiây_i is then independently masked with probability m (i.e., we replace yiy_i with MASK) and fed into the network along with the audio. The loss is computed between the original ground truth y and predictions y y only over the masked positions: Ldiffâ(y,y^)=1mââiâmaskedââ(yi,y^i), L_diff(y, y)= 1m _i L(y_i, y_i), (1) 3.2.1 Masking Unlike in language MDMs, we have two distinct output channels associated with separate ground truth (beat and downbeat). Crucially, the two channels are masked independently: each draws its own ratio and its own mask, so at any given training step the model may see, e.g., 30% of beat frames masked and 80% of downbeat frames masked. This is a prerequisite for the confidence-based unmasking strategy to work correctly at inference time for each channel (see Section 3.3). Additionally, we introduce a hyperparameter ponlymaskp_onlymask that sets the probability of fully masking both channels, forcing the model to predict the entire sequence from audio alone. 3.2.2 Loss A big advantage of the loss formulation in Equation 1 is that we can simply drop in the Shift-tolerant weighted BCE loss [12] as âL on the right side. This loss was shown to sharpen output logit peaks and balance logit magnitudes, both of which are highly desirable properties during inference. Separated losses are computed for beats and downbeats with equal weight. We have to deal with an additional source of complexity: the use of padding when processing pieces shorter than 30 seconds. We only mask and compute the loss over unpadded frames, and average it over the number of unpadded elements N. This can cause numerical instability when the loss denominators become too small, so we use the actual ratio of masked frames for m (rather than the masking probability), and force the minimum number of masked frames (which depends on ϔΔ and the total number of unpadded frames) to never be lower than that of a fully unpadded piece. 3.3 Inference Inference starts from a fully masked sequence of N frames and iteratively reveals tokens over S steps. At each step, the model performs a forward pass conditioned on the currently revealed tokens; the masking scheduler unmasks a subset of the remaining masked frames (i.e., assigns their predicted token values) until the entire sequence is unmasked. While we can find in the literature many ways to select the subset of frames to unmask at every step, we consider a widely used and simple one: given S steps, we unmask at each step the N/SN/S most confident masked predictions [22]. For simplicity, we do not sample from the distribution; instead, we pick the token with the maximum probability. Adapting this generic masked diffusion loop to beat tracking required three non-obvious design choices. 3.3.1 Balanced Unmasking Schedule Given one channel output logits, we compute the prediction confidence as |logits||logits|. However, we notice that negative logits (corresponding to non-event frames) are generally more confident than positive ones. Therefore, a pure confidence-based strategy would reveal non-event frames first, leading to a train-inference mismatch, since during training, the model always sees a balanced mix of event and non-event frames. Iterative inference in this setting decreases performance rather than improving it. To solve this problem, we propose a novel balanced unmasking schedule. At each step, we compute the current positive-to-negative ratio among the model predictions for the masked frames and split the unmasking budget N/SN/S accordingly between predicted positives and negatives. 3.3.2 Independent Per-Channel Unmasking We experimented with different ways of aggregating the confidence scores from the two channels before deciding which positions to unmask, as well as with simple strategies such as unmasking first beats, and then downbeats, or vice versa. Iterative inference in these settings leads to lower performance. We hypothesise that the reason is that there are pieces in which beats are simpler to produce than downbeats (e.g., pieces with stable beats and time signature changes), and vice versa (e.g., ethereal music with clear harmonic changes at downbeats). Moreover, beats and downbeats are very different signals: downbeats are approximately 4 times sparser than beats, and, even with positive weights in the loss, their logit magnitudes lie on different scales. The solution is independent channel unmasking, enabled by our independent masking during training (Section 3.2). Because each channel was always masked and revealed separately, the modelâs per-channel confidence scores are individually meaningful, calibrated to each channelâs sparsity, and can naturally influence one another in an order determined by the model. We therefore run the balanced unmasking schedule independently for beats and downbeats, each with its own confidence scores and negative-to-positive ratio. 3.3.3 Peak Picking Between Steps Frame-wise beat tracking networks tend to predict activations that spread across several adjacent frames near each true beat. The shift-tolerant weighted BCE loss produces sharper peaks than standard BCE, but clusters of adjacent positive frames still appear at inference time (see, for example, the first beat in Figure 1). The Beat This paper uses a peak-picking post-processing step that suppresses all non-maximum positions within a sliding 7-frame proximity window before thresholding to get the final predictions. In our masked diffusion iterative inference, nearby peaks produce nearby EVENT tokens, a situation that never occurred during training, since in the training data, each event is a single impulse in the non-event sequence. We address this problem by applying the aforementioned peak-picking post-processing after each inference step. Whenever a new EVENT is unmasked, all frames within the proximity window are also unmasked and forced to NO_EVENT, preventing adjacent frames from being revealed as positives in subsequent steps. We also take the opportunity to enforce musically meaningful behaviour: Every time we unmask a downbeat EVENT at a frame position, we also unmask a beat EVENT at the corresponding position, and force all neighbours to NO_EVENT. If there is already an unmasked beat EVENT in the proximity window, we move it to the downbeat position (though this situation never happened in our experiments, since the model learns to predict overlapping beats and downbeats). Unlike the sum head of [12] that only supports this behaviour, this process enforces that every downbeat is also a beat. Note that, unlike the DBN, we enforce a much smaller set of rules that are universally valid across all kinds of music. Figure 3 shows an example of our iterative inference. 3.3.4 Model Ensembling NNs are known to be poor at estimating probabilities and tend to be overconfident [14]. A more reliable measure of confidence can be obtained by averaging the output logits across multiple models. We test this approach by averaging the output activations of multiple ensembled models after each inference step, before computing the prediction confidence for unmasking. Model ensembling is a well-tested technique in beat tracking but typically not used in the MDM literature, likely due to the inference cost of large models. For our â 25M parameter model, we find the cost acceptable. 4 Experiments Our experiments aim mainly at isolating the effect of the MDM paradigm, specifically how it affects the coherence of the results. To this end, we focus on CMLt and AMLt metrics, as they measure the stability of inter-beat intervals and are therefore strongly affected by incoherent tempo variations and/or time signatures. The training data setup is similar to [12]: first, we optimise hyperparameters on a validation part of our training data; then we train on the full training data (incl. the validation set) and use a hold-out test set to report results. We provide the 8-fold cross-validation metrics across all training datasets, and all our predictions as supplementary material.444https://github.com/fosfrancesco/md_beat_this 4.1 Experimental Settings We employ the same datasets as the Beat This paper [12]: we train and validate on 4556 tracks, including those without downbeat annotations, and test on the GTZAN [18] dataset (993 pieces, excluding one unannotated track and 6 tracks that miss downbeat annotations). Figure 4: Effect of our MDM formulation, number of inference steps and ensembling on performance. Beat Downbeat F1 CMLt AMLt F1 CMLt AMLt No DBN Beat This [12] 89.1±0.389.1± 0.3 79.8±0.679.8± 0.6 89.8±0.489.8± 0.4 78.3±0.478.3± 0.4 67.3±0.867.3± 0.8 79.1±0.679.1± 0.6 GagnerĂ© ST-BCE [13] 89.689.6 81.881.8 91.891.8 77.577.5 67.067.0 80.880.8 Ours (8 inference steps) 89.7±0.289.7 0.2 82.9±0.482.9 0.4 92.5±0.292.5 0.2 79.5±0.379.5 0.3 76.4±0.676.4 0.6 88.5±0.388.5 0.3 DBN GagnerĂ© BCE [13] 89.6 82.6 92.592.5 78.378.3 74.7 88.288.2 MusicFM+HingeNet [26] 89.289.2 80.980.9 93.7 79.8 73.273.2 89.5 Table 1: Comparison of SOTA beat tracking systems on GTZAN. Results for [12, 13, 26] are taken from their papers. We train with ponlymask=0.4p_onlymask=0.4, and Ï”=0.05Δ=0.05 (see Section 3.2). Language MDMs use a much lower ϔΔ, e.g., 0.001 [22]. However, we train on a shorter sequence (1500 vs 4096) and deal with short padded pieces, which increases the risk of numerical instability.555This is less problematic when training in bf16 mixed precision, but we train on fp16 mixed due to GPU limitations. For the same reason, we deviate from [12] and reduce the learning rate to 0.00040.0004, increase the weight decay to 0.1 (excl. embedding layers), introduce gradient clipping of 1, and decrease AdamW ÎČ2 _2 to 0.95 [1]. We also double the number of training epochs from 150 to 300 and employ the WSD learning rate scheduler [15], with 1000 steps of warm-up and 15% of the total steps for linear decay. We compute metrics using the mir_eval package [24] with default parameters. Note that, while we enabled our model to disambiguate between valid competing outputs, the metrics still assume that only one solution is correct. AMLt partially address this problem by considering half/double tempi, but the default configurations donât include ternary subdivisions, and complex time signatures, tempo changes, and ambiguous beat positions are not handled. 4.2 Effect of MDM Formulation Figure 4 compares a single & ensemble MDMs with a non-diffusion version of the model. The latter lacks beat and downbeat Embeddings, and it is trained for 150 epochs on the non-diffusion loss. It shows small improvements over the Beat This model on all metrics. For the MDM, we observe substantial improvements in CMLt and AMLt metrics, which are particularly sensitive to erratic tempo changes. Strikingly, non-diffusion is outperformed even with a single inference step, indicating that the MDM training objective is beneficial beyond enabling iterative inference. We hypothesise that partially masked context reduces training âconfusionâ when the network is trained on similar pieces with conflicting metrical levels. We also observe a positive impact of ensembling, which, aside from beat F1, improves all metrics and prevents metric degradation when using more inference steps. This is not a trivial result; averaging output probabilities could bring them closer to the binary decision boundary, thus increasing the likelihood of unwanted label flips. We may expect that more than 3 models would further improve performance (at the cost of inference time), but leave this study for future work. 4.3 Comparison with current SOTA systems In Table 1, we compare our ensemble performance with GagnerĂ© et al. [13], and Ru et al. [26], which currently have the highest performance in the literature. While direct comparison is not our objective, as these works focus on pre-training research orthogonal to our approach, we include these results to provide context for our metrics. Both papers use DBN post-processing in their best configuration, but [13] also reports metrics for a model that works without the DBN; their code and models are not public for a more in-depth evaluation or a statistical significance study. [13] uses a similar-sized model as ours, while [26] fine-tunes a large audio model, MusicFM, which has more than 10x the parameters. We also include Beat This [12] as our starting point, providing a clear indication of how much the techniques presented in this paper improve performance. We select 8 inference steps for our system, aiming to balance inference time and accuracy. We train our system 9 times with different seeds, aggregate them into 3 ensembles, compute 3 sets of scores, and report their means and standard deviations. The comparison with systems without DBN post-processing shows that the MDM approach greatly improves performance, with consistent gains across all metrics, with up to â9%â 9\% increases in downbeat CMLt and AMLt metrics, most sensitive to incoherent output behaviour. Compared to DBN systems, we surpass the similarly sized system [13], and draw (in terms of the number of winning metrics) with the 10 times bigger model [26]. Itâs worth recalling that, while GTZAN contains mostly pieces with 3 and 4 beats per bar, and a few time signature changes, the advantage of not using DBN post-processing is that we can expect our system to work on more diverse music with different time signatures, and time signature changes [12]. 4.4 Coherency measures We use two heuristics to evaluate the coherency of the output: we count consecutive downbeats (i.e., the number of downbeats whose previous beat position also contains a downbeat) and tempo doubling/halving changes (if an inter-beat interval is double or half of the preceding one, considering the same tolerance range used for the AMLt metric). These two factors do not make the output automatically wrong666Scriabinâs Sonata 5 from the ASAP datasets contains a section with time signature 1/2; and tempo doubling/halving occurs, for example, in 4/4-7/8 tempo changes, and severe rallentando/accelerando. but their occurrences are rare in GTZAN labels (0 consecutive downbeats and 0.033 doubling/halving). We observe that both values decrease as the number of inference steps increases. For the 8-step inference, on average, per track, consecutive downbeat count drops from 0.25 to 0.02, and tempo doubling/halving occurrences pass from 0.75 to 0.119 (and decrease further to 0.06 for 20 steps). Manual evaluation of some pieces, such as the one from Figures 1 and 3, shows that incoherences are completely fixed, with the outputs converging to a stable time signature and tempo. 5 Conclusions We targeted the problem of incoherent N outputs of beat trackers and presented a novel approach based on masked diffusion, which substantially reduces this problem and improves performance (notably on CMLt and AMLt scores), without requiring heavy post-processing based on simplistic musical assumptions. We achieve a new state-of-the-art performance across similarly sized systems, whether or not they use a DBN. These results were enabled by original modifications to standard masked diffusion techniques that addressed specific challenges of the beat-tracking task. The downsides, compared to traditional one-step approaches, are longer training (we double the number of epochs) and longer inference time (roughly S times longer, even though we could optimise it by computing the frontend only once). Model ensembling also increases these costs linearly with the number of models. We focused on a no-DBN approach; however, some applications require outputs that adhere to strict constraints. Since our method improves the quality of N predictions, it could also enhance the performance of DBN-based systems. Beyond improved accuracy, our modelâs ability to accept partial outputs opens up practical use cases that one-step systems cannot support. A user can correct a set of beat positions and let the model complete the rest, which drastically speeds up the annotation of difficult pieces [28]. Moreover, our model supports autoregressive and block-autoregressive inference without any training modification [22]. We can achieve consistency on longer pieces, normally processed as independent snippets, by using the last part of a snippet output as a (always unmasked) conditioning for the next snippet prediction. Moreover, while we focused on obtaining a single coherent output from multiple valid interpretations, future work could explore techniques to control which interpretation is selected and allow switching between different metrical hierarchies. New evaluation protocols and new multi-annotation datasets are necessary to score such results correctly. Finally, our findings are general and can be applied to other beat trackers or even to other MIR tasks that may suffer from the multiple valid outputs problem, such as chord recognition and structure segmentation. 6 Acknowledgments We would like to thank Jan SchlĂŒter for the many productive discussions throughout this work, and for originally proposing iterative inference for beat tracking. We also owe a special thanks to Mathias Rose Bjare, who introduced us to the modern masked diffusion literature, sparking the very idea for this paper. References [1] L. B. Allal, L. Tunstall, N. Tazi, E. Bakouch, E. Beeching, C. M. Patiño, C. Fourrier, T. Frere, A. Lozhkov, C. Raffel, L. von Werra, and T. Wolf (2025) The smol training playbook: the secrets to building world-class llms. Cited by: §4.1. [2] J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. Van Den Berg (2021) Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems 34, p. 17981â17993. Cited by: §3. [3] S. Böck, F. Krebs, and G. Widmer (2016) Joint beat and downbeat tracking with recurrent neural networks.. In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), Cited by: §1. [4] J. Bolt, J. Pauwels, and G. Fazekas (2026) Beat and downbeat detection: a reformulated approach. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), Cited by: §1. [5] E. Cano, F. Mora-Ăngel, G. A. LĂłpez Gil, J. R. Zapata, A. Escamilla, J. F. Alzate, and M. Betancur (2021) Sesquialtera in the Colombian Bambuco: perception and estimation of beat and meterâextended version. Transactions of the International Society for Music Information Retrieval (TISMIR) 4 (1), p. 248â262. Cited by: §1. [6] H. Chang, H. Zhang, L. Jiang, C. Liu, and W. T. Freeman (2022) Maskgit: masked generative image transformer. In Proceedings of the Conference on computer vision and pattern recognition, Cited by: footnote 3. [7] C. Chiu, M. MĂŒller, M. E. Davies, A. W. Su, and Y. Yang (2022) An analysis method for metric-level switching in beat tracking. IEEE Signal Processing Letters 29, p. 2153â2157. Cited by: §1. [8] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. (2023) Palm: scaling language modeling with pathways. Journal of machine learning research 24 (240), p. 1â113. Cited by: §3.1.3. [9] M. ComunitĂ , Z. Zhong, A. Takahashi, S. Yang, M. Zhao, K. Saito, Y. Ikemiya, T. Shibuya, S. Takahashi, and Y. Mitsufuji (2024) Specmaskgit: masked generative modeling of audio spectrograms for efficient audio synthesis and beyond. In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), Cited by: footnote 3. [10] M. E. Davies, N. Degara, and M. D. Plumbley (2009) Evaluation methods for musical audio beat tracking algorithms. Queen Mary University of London, Centre for Digital Music, Tech. Rep. C4DM-TR-09-06. Cited by: §1. [11] A. Fichtinger, J. SchlĂŒter, and G. Widmer (2025) Music boomerang: reusing diffusion models for data augmentation and audio manipulation. In Proceedings of the Sound and Music Computing Conference (SMC), Cited by: §1. [12] F. Foscarin, J. SchlĂŒter, and G. Widmer (2024) Beat This! Accurate beat tracking without DBN postprocessing. In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), Cited by: Figure 1, §1, §1, §2, §3.1.1, §3.1, §3.2.2, §3.3.3, §4.1, §4.1, §4.3, §4.3, Table 1, Table 1, §4, footnote 1. [13] A. GagnerĂ©, S. Essid, and G. Peeters (2025) Controlling contrastive self-supervised learning with knowledge-driven multiple hypothesis: application to beat tracking. arXiv preprint 2510.25560. Cited by: §1, §2, §4.3, §4.3, Table 1, Table 1, Table 1. [14] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger (2017) On calibration of modern neural networks. In Proceedings of the International Conference on machine learning, Cited by: §3.3.4. [15] S. Hu, Y. Tu, X. Han, C. He, G. Cui, X. Long, Z. Zheng, Y. Fang, Y. Huang, W. Zhao, et al. (2024) Minicpm: unveiling the potential of small language models with scalable training strategies. In Proceedings of the Conference on Language Modeling (COLM), Cited by: §4.1. [16] O. Lartillot, M. S. Johansson, A. Elowsson, L. L. Monstad, and M. Cyvin (2023) A dataset of Norwegian Hardanger Fiddle recordings with precise annotation of note and beat onsets. Transactions of the International Society for Music Information Retrieval (TISMIR) 6 (1), p. 186â202. Cited by: §1. [17] V. Lostanlen, Y. Kong, G. Meseguer-Brocal, M. Lagrange, and R. Hennequin (2025) STOMP! self-supervised beat induction by matching pulses. In Late-Breaking/Demo of the International Society for Music Information Retrieval, Cited by: §1. [18] U. Marchand and G. Peeters (2015) Swing ratio estimation. In Proceedings of the International Conference on Digital Audio Effects (Dafx), Cited by: §4.1. [19] M. F. McKinney and D. Moelants (2006) Ambiguity in tempo perception: what draws listeners to different metrical levels?. Music Perception 24 (2), p. 155â166. Cited by: §1. [20] G. Morais, B. McFee, and M. Fuentes (2025) Skip that beat: augmenting meter tracking models for underrepresented time signatures. arXiv preprint 2502.12972. Cited by: §1. [21] S. Murgul and M. Heizmann (2025) Beat and downbeat tracking in performance midi using an end-to-end transformer architecture. arXiv preprint 2507.00466. Cited by: §1, footnote 2. [22] S. Nie, F. Zhu, Z. You, X. Zhang, J. Ou, J. Hu, J. Zhou, Y. Lin, J. Wen, and C. Li (2025) Large language diffusion models. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), Cited by: §2, §3.2, §3.3, §4.1, §5. [23] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever (2023) Robust speech recognition via large-scale weak supervision. In Proceedings of the International Conference on machine learning, Cited by: footnote 2. [24] C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, D. P. Ellis, and C. C. Raffel (2014) mir_eval: a transparent implementation of common MIR metrics. In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), Cited by: §4.1. [25] G. Ru, J. Wang, J. Zhao, Y. Wu, Y. Yu, N. Jiang, W. Wang, and W. Li (2025) BeatFM: improving beat tracking with pre-trained music foundation model. In Proceedings of the International Conference on Multimedia and Expo (ICME), Cited by: §1, §2. [26] G. Ru, J. Wang, J. Zhao, Y. Wu, Y. Yu, N. Jiang, W. Wang, and W. Li (2025) HingeNet: a harmonic-aware fine-tuning approach for beat tracking. In Proceedings of the International Conference on Multimedia and Expo (ICME), Cited by: §1, §2, §4.3, §4.3, Table 1, Table 1. [27] G. Ru, Y. Yu, and W. Li (2026) Toward robust and efficient beat tracking via beat-aware attention. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), Cited by: §1. [28] A. SĂĄ Pinto, G. Bernardes, and M. E. P. Davies (2023) Challenging beat tracking: tackling polyrhythm, polymetre, and polytempo with human-in-the-loop adaptation. In Proceedings of the International Symposium on Computer Music Multidisciplinary Research (CMMR), Cited by: §1, §5. [29] N. Shazeer (2020) GLU variants improve transformer. arXiv preprint arXiv:2002.05202. Cited by: §3.1.3. [30] J. Zhao, R. Nishiyama, K. Sumi, T. Fujishima, and A. Maezawa (2025) BEAT-U: multi-task music understanding with hierarchical timescales. Technical Report Music Information Retrieval Evaluation eXchange (MIREX). Note: MIREX 2025 Audio Beat Tracking Task Cited by: §1. [31] K. Zheng, Y. Chen, H. Mao, M. Liu, J. Zhu, and Q. Zhang (2024) Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. arXiv preprint arXiv:2409.02908. Cited by: footnote 3.