Paper deep dive

TabDLM: Free-Form Tabular Data Generation via Joint Numerical-Language Diffusion

Donghong Cai, Jiarui Feng, Yanbo Wang, Da Zheng, Yixin Chen, Muhan Zhang

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 97

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%

Last extracted: 7/20/2026, 10:16:27 AM

Summary

The paper introduces TabDLM, a unified framework for generating synthetic tabular data that contains mixed modalities (numerical, categorical, and free-form text). It addresses limitations of existing diffusion-based and LLM-based methods by employing a joint numerical-language diffusion model built on Masked Diffusion Language Models (MDLMs). TabDLM uses continuous diffusion for numerical features and masked diffusion for textual/categorical features, utilizing bidirectional attention to capture cross-modality interactions.

Entities (10)

TabDLM · model · 98%Numerical Features · datamodality · 95%Textual Features · datamodality · 95%MDLM · modelarchitecture · 92%Continuous Diffusion · technique · 90%Masked Diffusion · technique · 90%CTGAN · baselinemodel · 85%DiffLM · baselinemodel · 85%GReaT · baselinemodel · 85%TabDiff · baselinemodel · 85%

Relation Signals (9)

TabDLM → handlesmodality → Numerical Features

confidence 95% · TabDLM models ... numerical features with a continuous diffusion process

TabDLM → handlesmodality → Textual Features

confidence 95% · TabDLM models textual and categorical features through masked diffusion

TabDLM → usesarchitecture → MDLM

confidence 95% · TabDLM is a unified framework ... built on masked diffusion language models (MDLMs)

GReaT → iscategory → LLM-based

confidence 90% · Language model–based approaches ... as exemplified by GReaT

CTGAN → iscategory → VAE/GAN-based

confidence 90% · Representative methods include CTGAN and TVAE ... VAE/GAN-based tabular generators

TabDLM → outperforms → TabDiff

confidence 80% · demonstrate the effectiveness of TabDLM compared to strong diffusion- and LLM-based baselines

TabDLM → outperforms → DiffLM

confidence 80% · demonstrate the effectiveness of TabDLM compared to strong diffusion- and LLM-based baselines

TabDLM → outperforms → CTGAN

confidence 80% · demonstrate the effectiveness of TabDLM compared to strong diffusion- and LLM-based baselines

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Synthetic tabular data generation has attracted growing attention due to its importance for data augmentation, foundation models, and privacy. However, real-world tabular datasets increasingly contain free-form text fields (e.g., reviews or clinical notes) alongside structured numerical and categorical attributes. Generating such heterogeneous tables with joint modeling of different modalities remains challenging. Existing approaches broadly fall into two categories: diffusion-based methods and LLM-based methods. Diffusion models can capture complex dependencies over numerical and categorical features in continuous or discrete spaces, but extending them to open-ended text is nontrivial and often leads to degraded text quality. In contrast, LLM-based generators naturally produce fluent text, yet their discrete tokenization can distort precise or wide-range numerical values, hindering accurate modeling of both numbers and language. In this work, we propose TabDLM, a unified framework for free-form tabular data generation via a joint numerical-language diffusion model built on masked diffusion language models (MDLMs). TabDLM models textual and categorical features through masked diffusion, while modeling numerical features with a continuous diffusion process through learned specialized numeric tokens embedding; bidirectional attention then captures cross-modality interactions within a single model. Extensive experiments on diverse benchmarks demonstrate the effectiveness of TabDLM compared to strong diffusion- and LLM-based baselines.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

95,520 characters extracted from source content.

Expand or collapse full text

TabDLM: Free-Form Tabular Data Generation via Joint Numerical–Language Diffusion Donghong Cai1 Jiarui Feng1 Yanbo Wang2 Da Zheng3 Yixin Chen1 Muhan Zhang2 cai.d, feng.jiarui, ychen25@wustl.edu, wangyanbo@stu.pku.edu.cn zhengda.zheng@antgroup.com, muhan@pku.edu.cn 1Washington University in St. Louis 2Peking University 3Ant Group Abstract Synthetic tabular data generation has attracted growing attention due to its importance for data augmentation, foundation models, and privacy. However, real-world tabular datasets increasingly contain free-form text fields (e.g., reviews or clinical notes) alongside structured numerical and categorical attributes. Generating such heterogeneous tables with joint modeling of different modalities remains challenging. Existing approaches broadly fall into two categories: diffusion-based methods and LLM-based methods. Diffusion models can capture complex dependencies over numerical and categorical features in continuous or discrete spaces, but extending them to open-ended text is nontrivial and often leads to degraded text quality. In contrast, LLM-based generators naturally produce fluent text, yet their discrete tokenization can distort precise or wide-range numerical values, hindering accurate modeling of both numbers and language. In this work, we propose TabDLM, a unified framework for free-form tabular data generation via a joint numerical–language diffusion model built on masked diffusion language models (MDLMs). TabDLM models textual and categorical features through masked diffusion, while modeling numerical features with a continuous diffusion process through learned specialized numeric tokens embedding; bidirectional attention then captures cross-modality interactions within a single model. Extensive experiments on diverse benchmarks demonstrate the effectiveness of TabDLM compared to strong diffusion- and LLM-based baselines. The code is available at https://github.com/ilikevegetable/TabDLM 1 Introduction Tabular data is one of the most fundamental data types in modern machine learning and is indispensable across domains such as finance [32], healthcare [15], and social media [22]. In practice, however, deploying machine learning on tabular datasets faces recurring obstacles, including privacy and security constraints [12, 3], limited data availability [10], and missing values [45, 42]. These challenges motivate synthetic tabular data generation, which seeks to produce synthetic samples that match the schema and key statistical properties of the original dataset for replacing the original data. High-fidelity synthetic tabular data generation remains challenging because real-world tables exhibit complex dependencies across columns [40, 6] and often contain a mixture of numerical, categorical, and free-form textual features [33]. Existing approaches can broadly be categorized into diffusion-based and large language model (LLM)-based methods. Diffusion-based methods employ diffusion processes [35, 13, 34, 4] to learn denoising transformations that approximate the data distribution during training and to generate realistic samples from noise during inference. Such models have been widely applied to tabular data generation, particularly for continuous (numerical) features [18, 17, 45]. More recent work has explored combining continuous and discrete diffusion to jointly model numerical and categorical attributes [44, 21, 23, 33]. However, extending standard diffusion models to open-ended textual fields remains challenging due to the exponential size of the text space. To the best of our knowledge, no existing diffusion-based methods have been successfully applied to tabular data generation involving open-ended text. In contrast, LLM-based methods naturally support free-form text generation [6, 46] owing to their strong language modeling capabilities. However, their token-level representations can be unreliable for modeling high-precision or wide-range numerical values [41], as numbers are often fragmented into multiple tokens. Furthermore, autoregressive generation enforces a left-to-right dependency structure that is misaligned with the mutually dependent relationships across tabular columns. To address the above limitations, we propose TabDLM, a unified framework that integrates the complementary strengths of diffusion-based and LLM-based modeling. To preserve the most effective modeling paradigm for each modality, TabDLM employs continuous diffusion for numerical columns and language models for categorical and free-form textual columns. However, rather than relying on autoregressive language models, we adopt Masked Diffusion Language Models (MDLMs) as the backbone architecture. MDLMs offer two key advantages. First, their diffusion-based formulation enables TabDLM to model numerical and language modalities within a unified generative process. Second, unlike autoregressive models, MDLMs employ bidirectional attention, enabling the model to capture mutual dependencies among tabular columns. To enable joint numerical–language diffusion within a single MDLM, we introduce a trainable numerical tokenization module into the MDLM architecture, which allows continuous diffusion to represent each numerical value with a single token. As a result, TabDLM learns to jointly denoise numerical values and language content during training and generates synthetic tabular data by simultaneously performing continuous diffusion and masked language diffusion during inference. The overview of the TabDLM is provided in Figure 1. We evaluate TabDLM across a range of scenarios and benchmarks, where it consistently outperforms both diffusion-based and language-based baselines. 2 Methods Figure 1: Overview of TabDLM. During training, given an input tabular sample, TabDLM applies masked language diffusion to categorical and free-form textual features, and continuous diffusion to numerical features. Noisy inputs are mapped to token embeddings via a textual embedding layer and a numerical encoder. An MDLM then denoises these embeddings to reconstruct the original sample. Training updates only the projector in the numerical encoder and decoder, as well as the LoRA modules within each MDLM layer. 2.1 Preliminary Problem setup and notation. Let =(i)i=1ND=\x^(i)\_i=1^N denote a tabular dataset with M columns. Each record is (i)=(x1(i),…,xM(i))x^(i)=(x^(i)_1,…,x^(i)_M), where columns can be either numerical, categorical, or textual. We use index sets ℐnumI_num, ℐcatI_cat, and ℐtextI_text to denote numerical, categorical, and text columns, respectively. Our goal is to learn a generative model pθ()p_θ(x) that can sample realistic records while capturing cross-column dependencies. Continuous diffusion. We model the distribution of a continuous vector 0∈ℝdz_0 ^d using a diffusion process defined by the stochastic differential equation (SDE) d=(,t)dt+g(t)ddz=f(z,t)dt+g(t)dw. In the variance-exploding (VE) formulation [36], we set (,t)=f(z,t)=0 and g(t)=2σ˙(t)σ(t)g(t)= 2 σ(t)σ(t), where σ(t):[0,1]→ℝ+σ(t):[0,1] _+ is a strictly increasing function governing the noise level and σ˙(t)=dσ(t)dt σ(t)= dσ(t)dt denotes its time derivative. From a probabilistic perspective, this SDE corresponds to a continuous-time limit of a Markov chain where Gaussian noise is progressively added to the data. The transition kernel q(t∣s)q(z_t _s) for s<ts<t and the marginal q(t∣0)q(z_t _0) are given by: q(t∣s) q(z_t _s) =(s,(σ2(t)−σ2(s))), =N\! (z_s,\,(σ^2(t)-σ^2(s))I ), (1) q(t∣0) q(z_t _0) =(0,σ2(t)). =N\! (z_0,\,σ^2(t)I ). The reverse generative process solves the probability flow ODE [36], as described in the below: d=−12g(t)2∇log⁡pt()dt=−σ˙(t)σ(t)∇log⁡pt()dt.dz=- 12g(t)^2 _z p_t(z)dt=- σ(t)σ(t) _z p_t(z)dt. (2) We use a parameterized denoising model ϵθ(t,t) ε_θ(z_t,t) that estimates the score function via ∇log⁡pt()≈−ϵθ(t,t)/σ(t) _z p_t(z)≈- ε_θ(z_t,t)/σ(t). Training minimizes the denoising error: ℒdiff=t,0,ϵ[‖ϵ−ϵθ(0+σ(t)ϵ,t)‖22],ϵ∼(,).L_diff=E_t,z_0, ε [ \| ε- ε_θ(z_0+σ(t) ε,t) \|_2^2 ],\ \ ε (0,I). (3) Masked diffusion language models (MDLMs). MDLMs can be viewed as a discrete-state diffusion process [4] over token sequences. Let 0=(s1,…,sL)s_0=(s_1,…,s_L) be a length-L sequence with tokens in a vocabulary V augmented with an absorbing mask state m=[MASK]m=[MASK]. Using a token-wise Markov chain, the forward noising kernel factorizes as q(t∣t−1)=∏j=1Lq(st,j∣st−1,j),q(s_t _t-1)= _j=1^Lq(s_t,j s_t-1,j), (4) with a masking schedule βt∈[0,1] _t∈[0,1] and absorbing transitions q(st,j∣st−1,j)=1,st−1,j=m,st,j=m,βt,st−1,j≠m,st,j=m,1−βt,st−1,j≠m,st,j=st−1,j,0,otherwise,q(s_t,j s_t-1,j)= cases1,&s_t-1,j=m,\ s_t,j=m,\\ _t,&s_t-1,j≠ m,\ s_t,j=m,\\ 1- _t,&s_t-1,j≠ m,\ s_t,j=s_t-1,j,\\ 0,&otherwise, cases (5) Equivalently, the marginal has the closed form q(st,j=s0,j∣s0,j)=α¯tq(s_t,j=s_0,j s_0,j)= α_t and q(st,j=m∣s0,j)=1−α¯tq(s_t,j=m s_0,j)=1- α_t, where α¯t=∏τ=1t(1−βτ) α_t= _τ=1^t(1- _τ). The reverse model uses a bidirectional Transformer to predict the original token distribution from the partially masked sequence: pθ(0∣t)=∏j=1Lpθ(s0,j∣t),p_θ(s_0 _t)= _j=1^Lp_θ(s_0,j _t), (6) and is typically trained by maximizing the log-likelihood of the ground-truth tokens at corrupted (masked) positions: ℒMDLM=t,0,t[−∑j:st,j=[MASK]log⁡pθ(s0,j∣t)].L_MDLM=E_t,s_0,s_t [- _\j:\,s_t,j=[MASK]\ p_θ(s_0,j _t) ]. (7) At sampling time, MDLMs start from an all-mask sequence and iteratively unmask tokens according to pθp_θ. 2.2 The Model Architecture of TabDLM We first present the architecture of TabDLM. As illustrated in Figure 1, TabDLM comprises five main components: (1) a numerical encoder module, (2) a text embedding layer, (3) a masked diffusion language model, (4) a numerical decoder module, and (5) an LM head. We describe each component in detail below. Note that in this section we use x^j(i) x^(i)_j instead of xj(i)x^(i)_j to denote the noise version of the input. We will describe how we obtain the x^j(i) x^(i)_j from xj(i)x^(i)_j in Section 2.3. Number encoder module. Given an input numerical value x^j(i)∈ℝ,j∈ℐnum x^(i)_j ,j _num, the number encoder module transforms the number into a token embedding with the same size as the MDLM. To achieve this, we first use quantile normalization [5] to standardize the numerical values as done in previous works [38, 33]. Then, we use a pretrained Multi-Layer Perceptron (MLP) as the float number encoder to convert the normalized values into an r-dimensional embedding vector. Finally, a trainable projection is used to align the r-dimensional embedding vector into the d-dimensional MDLM embedding space: ^j(i)=ENC(x^j(i)),j(i)=PROJe(^j(i)),j∈ℐnum, z^(i)_j=ENC( x^(i)_j),\ \ z^(i)_j=PROJ_e( z^(i)_j),\ \ j _num, (8) where ENC and PROJePROJ_e denote the MLP-based encoder and projection module, respectively, with ^j(i)∈ℝr z^(i)_j ^r and j(i)∈ℝdz^(i)_j ^d. In our implementation, we pretrained ENC based on Griffin [38] and keep it fixed throughout all experiments, while PROJePROJ_e is jointly optimized during training. Text embedding layer. The text embedding layer transforms textual input features x^j(i),j∈ℐcat∪ℐtext x^(i)_j,j _cat _text into a token embedding sequence. We directly adopt the pretrained embedding table from the underlying MDLM. Formally, j(i)=(j,1(i),…,j,lji(i))=EMB(x^j(i)),j∈ℐcat∪ℐtext,z^(i)_j=(z^(i)_j,1,…,z^(i)_j,l^i_j)=EMB( x^(i)_j),j _cat _text, (9) where each token embedding j,k(i)∈ℝd,k∈[lji]z^(i)_j,k ^d,k∈[l^i_j] and ljil^i_j denotes the token sequence length of column j in sample i, as each textual column may be mapped to multiple tokens in the MDLM embedding space. Masked diffusion language model. The masked diffusion language model is employed to jointly denoise both numerical and textual tokens. Formally, given an input sequence of token embeddings (i)=(1(i),…,S(i))z^(i)=(z^(i)_1,…,z^(i)_S) of length S and a diffusion time step t, the MDLM outputs a denoised sequence (i)o^(i) that estimates the corresponding representations at time t=0t=0. In TabDLM, the input token sequence is constructed by concatenating four components: a schema prompt, categorical features, textual features, and numerical features. The schema prompt provides a textual description of the feature type and the semantic meaning of each column, which is fixed for a given dataset. In detail, the input token sequence for sample i is represented as: (i)=(p,(j(i)|j∈ℐcat∪ℐtext),(j(i)|j∈ℐnum)), ^(i)= (z_p,(z_j^(i)|j _cat _text),(z_j^(i)|j _num) ), where pz_p is the token embedding sequence for the schema prompt and j(i)=(j,1(i),…,j,lji(i))z_j^(i)=(z^(i)_j,1,…,z^(i)_j,l^i_j) for any j∈ℐcat∪ℐtextj _cat _text. The forward process of MDLM can be described as: (i)=MDLM((i),t).o^(i)=MDLM(z^(i),t). (10) For the MDLM architecture, we adopt a standard transformer architecture with bidirectional attention [37, 8]. For numerical tokens, an additional positional embedding is added to encode the diffusion noise level applied to the input, following the design in DiT [29]. No such embedding is added for textual tokens, as the noise level can be implicitly captured by the proportion of masked tokens. Number decoder module. The number decoder module is used to decode the output token embedding for the numerical value back to the original number. We use a symmetric version to the number encoder module with one trainable projection, along with a pretrained MLP decoder: ¯j(i)=PROJd(j(i)),x¯j(i)=DEC(¯j(i)),j∈ℐnum, o^(i)_j=PROJ_d(o^(i)_j),\ \ x^(i)_j=DEC( o^(i)_j),\ \ j _num, (11) where DEC and PROJdPROJ_d denote the MLP-based decoder and projection module, respectively, with j(i)∈ℝdo^(i)_j ^d representing the output token embedding from the MDLM and ¯j(i)∈ℝr o^(i)_j ^r. Similarly, the DEC is pretrained jointly with the float number encoder and fixed during the experiment. LM head. Finally, the LM head is used to decode the output token embedding for the textual value back to the original text. We directly use the frozen LM head from MDLM: x¯j(i)=HEAD(j(i)),j∈ℐcat∪ℐtext. x^(i)_j=HEAD(o^(i)_j), j _cat _text. (12) 2.3 The Forward Process of TabDLM The forward diffusion process typically adds noise to the input. In TabDLM, this process is applied to both numerical and textual modalities; for clarity, we describe the forward process for each modality separately. Forward process of numerical features. For numerical features, we adopt continuous diffusion to better capture fine-grained distributions. Given a clean input value xj(i),j∈ℐnumx^(i)_j,j _num, we first sample a time t∈[0,1]t∈[0,1]. Then, according to Equation 1, the forward process adds noise as follows: x^j(i)=xj(i)+σ(t)ϵ,ϵ∼(0,1),j∈ℐnum. x^(i)_j=x^(i)_j+σ(t)ε,\ \ ε (0,1),\ \ j _num. (13) Note that the noise is added to the normalized value. Forward process of textual features. For textual features, we adopt discrete masked diffusion, as used in MDLMs. Given a text sequence xj(i)=(xj,1(i),…,xj,lji(i)),j∈ℐcat∪ℐtextx^(i)_j=(x^(i)_j,1,…,x^(i)_j,l^i_j),j _cat _text, we first sample a time t∈[0,1]t∈[0,1]. Then, according to Equation 5 and the accompanying discussion, the forward process independently transforms each token into a mask token with probability 1−α¯t1- α_t, resulting in the masked token sequence x^j(i)=(x^j,1(i),…,x^j,lji(i)) x^(i)_j=( x^(i)_j,1,…, x^(i)_j,l^i_j). Finally, in TabDLM, we use a single t for both continuous diffusion and discrete masked diffusion. This design enables the model to jointly denoise numerical and textual modalities under a unified noise level. 2.4 Training of TabDLM During training, we optimize the TabDLM to jointly denoise both the numerical modality and textual modality. Specifically, given an output sequence (i)=(p,(j(i)|j∈ℐcat∪ℐtext),(j(i)|j∈ℐnum))o^(i)= (o_p,(o_j^(i)|j _cat _text),(o_j^(i)|j _num) ) from MDLM, we first transform the numerical token embedding back to real number using the number decoder module and then the loss is computed separately for continuous diffusion and masked diffusion: ℒnum=1N∑i=1N∑j∈ℐnum‖xj(i)−x¯j(i)‖22,j∈ℐnum,L_num= 1N _i=1^N _j _num \|x^(i)_j- x^(i)_j \|_2^2, j _num, (14) ℒtext= _text= 1N∑i=1N∑x^j,k(i)=[MASK]CE<x¯j,k(i),xj,k(i)>, 1NΣ^N_i=1 _ x^(i)_j,k=[MASK]CE< x^(i)_j,k,x^(i)_j,k>, (15) k∈[lji],j∈ℐcat∪ℐtext, k∈[l^i_j], j _cat _text, where CE<⋅,⋅>CE<·,·> is the cross entropy loss. The final objective is optimized using a step-dependent weighting schedule: ℒ=ℒtext+λ(s)ℒnum,λ(s)=λmax⋅min⁡(1,sswarm),L=L_text+λ(s)\,L_num,\ \ λ(s)= _ · (1, ss_warm ), (16) where s denotes the global optimization step, and this warm-up schedule stabilizes the alignment of newly initialized numerical projections with the pretrained MDLM backbone. We use default hyperparameters λmax=1 _ =1 and swarm=2000s_warm=2000. In practice, we freeze the pretrained weights of the MDLM and introduce trainable LoRA modules [14] into both the feed-forward and attention components of each transformer layer. 2.5 The backward sampling in TabDLM Finally, we describe the sampling process of TabDLM. After training, generation starts from noise and progressively denoises the inputs to produce synthetic tabular samples following the learned data distribution. For numerical modality, the process is initialized with pure Gaussian noise, while for textual modality, it starts from a fully masked token sequence. We then iteratively apply Equation 2 and Equation 6 to the numerical and textual modalities to recover clean data. More details are provided in Appendix B. 3 Related Works Existing methods in tabular data generation can be classified into three categories based on their underlying frameworks. VAE/GAN-based tabular generators. This line of work formulates tabular data generation using Variational Autoencoders (VAEs) [19] or Generative Adversarial Networks (GANs) [11]. Representative methods include CTGAN and TVAE [40]. GOGGLE [24] further enhances this paradigm by explicitly modeling column dependencies through a Graph Neural Network–augmented VAE architecture. However, these approaches often lack sufficient expressivity when confronted with complex tabular distributions involving intricate feature interactions. Diffusion-based tabular generators. Motivated by the strong generative capacity of diffusion models [13], a growing body of work adapts diffusion processes for tabular data generation. Early methods model numerical and categorical features using separate discrete-time diffusion [4], as in Kotelnikov et al. [21], Lee et al. [23], but discretization can lead to looser ELBO bounds and suboptimal generation quality [36, 20]. More recent approaches encode tabular features into continuous latent spaces and apply Gaussian diffusion [45, 44]. However, such latent modeling introduces additional encoding overhead and may only indirectly capture heterogeneous feature interactions. Recently, CDTD [25] explores feature-wise noise schedules under continuous diffusion, and TabDiff [33] further extends it to mixed-type feature-level diffusion. More recently, TabNAT [43] combines continuous diffusion for numerical features with a bidirectional masked Transformer over column-specific categorical indices, but its discrete generation is restricted to fixed category sets and does not support free-form text. In contrast, TabDLM leverages a pretrained MDLM with a large semantic vocabulary to jointly generate numerical, categorical, and free-form textual fields within a single model. Language model-based tabular generators. Recent advances in large language models have also inspired language model–based approaches for tabular data generation. These methods serialize each row into a text sequence and fine-tune an autoregressive language model to capture row-level distributions, as exemplified by GReaT [6] with a GPT-2 [30] backbone. DiffLM [46] is the closest to our work, as it integrates VAEs and latent diffusion within a language modeling framework. However, DiffLM applies continuous diffusion in a latent space derived from textual representations, which may lose fine-grained token-level information and limit its applicability to free-form text fields. To the best of our knowledge, TabDLM is the first approach to apply masked language diffusion to explicitly model textual features at the token level, enabling faithful generation of free-form text in tabular data. 4 Experiments In this section, we conduct extensive experiments to evaluate the proposed TabDLM. Specifically, we aim to answer the following questions. Q1: Can TabDLM effectively model the mutual dependencies among numerical, categorical, and free-form text columns? Q2: How well does TabDLM perform on real-world tabular data generation tasks involving free-form text columns? Q3: Can TabDLM achieve performance on par with existing methods on tasks that do not include textual columns? We implement TabDLM based on LLaDA-8B [26] for all experiments. Other implementation details can be found in Appendix B. 4.1 Results on synthetic tabular datasets In this section, we answer the question Q1 through two carefully designed synthetic tabular datasets. Datasets. Existing real-world tabular generation datasets predominantly focus on numerical and categorical features and lack free-form text fields. Thus, we construct two synthetic tabular datasets containing numerical, categorical, and free-form text columns: MathExpr and ProfileBio. MathExpr focuses on mathematical expressions. Each sample includes two floating-point variables, categorical features indicating the unary and binary operators applied to them, and a textual column containing the corresponding LaTeX expression. ProfileBio contains numerical and categorical attributes describing a person’s demographic and educational background, along with a textual biography generated from the other columns. For both datasets, accurate generation requires models to capture correlations between numerical, categorical, and textual columns. We show an example of both datasets in Table 5 and Table 6, respectively, and leave the detailed dataset generation process in Appendix A.1. Baselines. Given these two datasets contain free-form textual features, we compare the proposed TabDLM with the autoregressive LLM and Masked Diffusion LLM. Specifically, we include Qwen2.5 (7B/14B) with 3-shot in-context learning (ICL), and Qwen2.5-7B and LLaDA-8B with LoRA-based supervised fine-tuning (SFT). Metrics. We evaluate generated data along two groups. 1) Fidelity: Shape and Trend measure marginal column-wise similarity and pairwise dependencies among numerical and categorical columns, both reported as error rates; 2) Cross-field consistency: for MathExpr, we report Operation Match Rate (Op-MR) and Expression Match Rate (Exp-MR), measuring whether operators and numerical literals in the generated LaTeX align with the structured fields. For ProfileBio, we report Biography Match Rate (Bio-MR), checking consistency between the generated biography and structured attributes. Detailed definitions are provided in Appendix A.2. Table 1: Evaluation results on the MathExpr and ProfileBio datasets (%). Method MathExpr ProfileBio Shape ↓ Trend ↓ Op-MR ↑ Exp-MR ↑ Shape ↓ Trend ↓ Bio-MR ↑ Qwen2.5-7BICL_ICL 39.0439.04 56.6556.65 53.9353.93 53.9253.92 41.2941.29 64.5264.52 7.977.97 Qwen2.5-14BICL_ICL 25.6125.61 36.0736.07 76.0176.01 75.1275.12 43.9743.97 65.0065.00 13.0013.00 Qwen2.5-7BSFT_SFT 5.075.07 50.8150.81 99.6899.68 99.3499.34 10.5110.51 36.2636.26 98.5198.51 LLaDA-8BSFT_SFT 5.415.41 42.5442.54 98.3698.36 97.9097.90 5.815.81 44.1444.14 97.6397.63 TabDLM 3.69 6.07 99.99 99.80 4.84 7.81 98.60 Results. As shown in Table 1, TabDLM achieves the best overall performance on both MathExpr and ProfileBio. On MathExpr, TabDLM obtains the lowest Shape and Trend errors, outperforming the strongest baseline by 27.1%27.1\% and 83.2%83.2\%, respectively, while also achieving the best results for both types of Match Rate. Since MathExpr requires the generated LaTeX expression to be consistent with both numerical values and categorical operation descriptors, these results indicate that TabDLM can preserve explicit dependencies across numerical, categorical, and textual fields rather than merely matching marginal distributions. On ProfileBio, TabDLM also achieves the lowest Shape and Trend errors and the highest Match Rate, demonstrating its ability to generate biographies that remain semantically aligned with the corresponding structured attributes. Together, the strong fidelity and cross-field consistency results provide a direct answer to Q1: TabDLM can effectively model mutual dependencies among heterogeneous tabular fields through joint numerical-language denoising. 4.2 Results on real-world tabular datasets with free-form text features In this section, we evaluate TabDLM on real-world tabular datasets with free-text to answer Q2. Dataset and Baselines. For this section, we use two real-world datasets, Amazon and Arxiv, both derived from the RelBench [31] relational benchmark. Amazon is constructed from rel-amazon by converting multiple relational tables into a single heterogeneous table containing numerical attributes, categorical attributes, and multiple long-text fields (e.g., title, description, and review). Arxiv is constructed from rel-arxiv as a paper-level table containing both numerical and categorical attributes, and free-form text fields (e.g., title, arxiv code, and abstract). We provide more details on dataset generation in Appendix A.1. We use the same baseline as in Section 4.1. Table 2: Results on the Amazon and Arxiv datasets. (∗ indicates (w/o nums)) Method Amazon Arxiv Shape (%)↓ Trend (%)↓ MLE∗MLE^* ↑ MLE ↑ Shape (%)↓ Trend (%)↓ MLE∗MLE^* ↑ MLE ↑ Real 0.00.0 0.00.0 .905.905 .906.906 0.00.0 0.00.0 .968.968 .968.968 Qwen2.5-7BICL_ICL 37.9337.93 68.2768.27 .636.636 .629.629 68.0468.04 42.3442.34 .521.521 .516.516 Qwen2.5-14BICL_ICL 39.1939.19 78.2278.22 .684.684 .679.679 40.7940.79 34.3534.35 .556.556 .527.527 Qwen2.5-7BSFT_SFT 11.4111.41 46.3146.31 .824.824 .834.834 11.1911.19 10.0210.02 .701.701 .727.727 LLaDA-8BSFT_SFT 7.767.76 37.1337.13 .891.891 .892.892 14.1014.10 43.8143.81 .931.931 .930.930 TabDLM 4.67 5.33 .893 .895 6.30 6.30 .946 .948 Metrics. Similar to Section 4.1, we evaluate 1) Fidelity: Shape and Trend. Additionally, we include 2) Downstream utility: Machine Learning Efficiency (MLE), which measures how well predictive models trained on synthetic data generalize to real test data. Finally, to evaluate cross-field consistency between free-form text and the remaining numerical and categorical features, we follow an MLE-like protocol: we first serialize all non-numerical fields into structured text and encode them with the sentence embedding model Nomic [27], then train an XGBoost Classifier [7] using either (i) text embeddings only or (i) text embeddings concatenated with numerical features, which isolates the contribution of generated text/categorical fields. For the Amazon and Arxiv dataset, we report AUC for MLE task. Results. The results on Amazon and Arxiv are presented in Table 2. TabDLM consistently achieves the best performance across Shape, Trend, and MLE on both datasets. Specifically, compared to the strongest baseline, TabDLM reduces Shape Error by 39.8%39.8\% and 43.7%43.7\%, and Trend Error by 85.6%85.6\% and 37.1%37.1\%, on Amazon and Arxiv, respectively. These results show that TabDLM effectively preserves structured-field marginal distributions and pairwise structured dependencies in real-world datasets that also contain free-form textual fields. Regarding downstream utility, TabDLM achieves the best MLE on both datasets, closely approaching the upper bound set by real data. This indicates that the generated text, categorical, and numerical fields jointly preserve label-relevant information. Notably, when numerical attributes are excluded, TabDLM’s performance on MLE (w/o nums) exhibits a decline consistent with the trend observed in real data. This suggests that the synthetic numerical features are not merely reproducing marginal statistics, but are meaningfully coupled with target labels and other modalities, thereby providing substantive value for downstream predictive tasks. Together, these results provide a direct answer to Q2: TabDLM performs strongly on real-world tabular generation tasks involving free-form text, preserving both structured-field fidelity and downstream task-relevant information across modalities. 4.3 Results on real-world tabular datasets without free-form text features Finally, we evaluate the performance of TabDLM on standard tabular data generation benchmarks without free-form text features, enabling direct comparison with existing methods. This evaluation addresses Q3 by examining whether TabDLM retains strong modeling capacity on traditional tabular data, despite being designed for a significantly broader heterogeneous generation setting. Datasets, baselines, and metrics. We conduct experiments on five real-world tabular datasets: Adult, Default, Shoppers, Magic, and Beijing. Detailed dataset profiles are presented in Appendix A.1.5. For baselines, we compare TabDLM with some widely-used synthetic tabular data generation methods from four categories: 1) GAN-based: CTGAN [40]; 2) VAE-based: TVAE [40] and GOGGLE [24]; 3) Autoregressive Tabular Language Model: GReaT [6] and DiffLM [46]; 4) Diffusion-based: STaSy [17], CoDi [23], TabDDPM [21], TabSyn [44], TabDiff [33], and TabNAT [43]. We evaluate along three groups of metrics: 1) Fidelity: Shape, Trend, α-Precision, and C2ST assess how faithfully synthetic data recovers the ground-truth distribution; 2) Downstream utility: Machine Learning Efficiency (MLE) measures the usefulness of synthetic data for predictive tasks; 3) Privacy: Distance to Closest Record (DCR) evaluates privacy risk by measuring the distance between each synthetic sample and its nearest training sample, where overly small distances may indicate memorization. We report Shape, Trend in the main paper and defer α-Precision, C2ST, MLE, and DCR results to Appendix C. Table 3: Performance comparison on the error rates (%) of Shape ↓ / Trend ↓ . Each cell reports Shape / Trend. Full per-metric results are provided in Appendix C.1. (B: best overall; B: best in language-based model) Method Adult Default Shoppers Magic Beijing Average CTGAN 16.84 / 20.23 16.83 / 26.95 21.15 / 13.08 9.81 / 7.00 21.39 / 22.95 17.20 / 18.04 TVAE 14.22 / 14.15 10.17 / 19.50 24.51 / 18.67 8.25 / 5.82 19.16 / 18.01 15.26 / 15.23 GOGGLE 16.97 / 45.29 17.02 / 21.94 22.33 / 23.90 1.90 / 9.47 16.93 / 45.94 15.03 / 29.31 STaSy 11.29 / 14.51 5.77 / 5.96 9.37 / 8.49 6.29 / 6.61 6.71 / 8.00 7.89 / 8.71 CoDi 21.38 / 22.49 15.77 / 68.41 31.84 / 17.78 11.56 / 6.53 16.94 / 7.07 19.50 / 24.46 TabDDPM 1.75 / 3.01 1.57 / 4.89 2.72 / 6.61 1.01 / 1.70 1.30 / 2.71 1.67 / 3.78 TabSyn 0.81 / 1.93 1.01 / 2.81 1.44 / 2.13 1.03 / 0.88 1.26 / 3.13 1.11 / 2.18 TabDiff 0.63 / 1.49 1.24 / 2.55 1.28 / 1.74 0.78 / 0.76 1.03 / 2.59 0.99 / 1.83 TabNAT 0.73 / 1.61 0.80 / 2.21 1.11 / 1.67 0.62 / 1.37 0.79 / 2.48 0.81 / 1.87 GReaT 12.12 / 17.59 19.94 / 70.02 14.51 / 45.16 16.16 / 10.23 8.25 / 59.60 14.20 / 40.52 DiffLM 9.74 / – 9.06 / – 10.07 / – 7.53 / – 6.35 / – 8.55 / – Qwen2.5-7BICL_ICL 27.80 / 37.61 22.39 / 31.02 34.37 / 34.77 13.41 / 20.62 31.83 / 40.83 25.96 / 32.97 Qwen2.5-14BICL_ICL 25.17 / 42.67 21.52 / 28.72 51.21 / 42.32 17.51 / 18.11 29.76 / 38.20 29.03 / 34.00 Qwen2.5-7BSFT_SFT 5.11 / 17.03 2.58 / 16.99 8.76 / 10.28 7.53 / 14.45 7.49 / 28.68 6.29 / 17.49 LLaDA-8BSFT_SFT 1.69 / 26.15 2.00 / 26.30 13.56 / 16.42 5.60 / 13.71 3.64 / 27.90 5.30 / 22.10 TabDLM 1.46/ 2.741.46\,/\,2.74 1.18/ 2.331.18\,/\,2.33 2.05/ 2.402.05\,/\,2.40 2.93/ 2.852.93\,/\,2.85 2.42/ 2.932.42\,/\,2.93 2.01/ 2.652.01\,/\,2.65 Results. As reported in Table 3, TabDLM performs competitively with state-of-the-art tabular diffusion models on Shape and Trend. This is notable because such tabular-specific synthetic methods operate in a relatively restricted feature space (e.g., categorical columns are represented via one-hot vectors or limited discrete states), whereas TabDLM is built to jointly model heterogeneous fields and ultimately support open-ended text generation, yet it still remains strong on purely numerical and categorical datasets. Beyond this, TabDLM consistently and substantially outperforms all language-based baselines, with average gains of 62.1%62.1\% in Shape and 84.8%84.8\% in Trend. Notably, language-based generators often achieve reasonable Shape but markedly worse Trend. A likely reason is that row serialization simplifies matching column-wise marginals, as each field can be generated locally to fit its token-level distribution. However, Trend depends on modeling joint dependencies among numerical and categorical columns, which is particularly difficult to preserve under autoregressive language backbones. In practice, numerical values are represented as subword token sequences with precision-sensitive semantics, whereas categorical fields act as discrete identifiers, making consistent numerical–categorical correlations hard to preserve under autoregressive generation. In contrast, TabDLM jointly generates numerical and categorical features within a unified diffusion process and leverages bidirectional interactions across columns, enabling more direct modeling of tabular dependencies than left-to-right row generation and yielding greater fidelity and downstream utility on structured datasets. 4.4 Ablation Study We conduct an ablation study to assess the contribution of each component in TabDLM. We consider three variants: (i) TabDLM-noFloatAE, which replaces the pretrained float encoder/decoder with randomly initialized counterparts; (i) TabDLM-onlyContDiff, which removes MDLM-based modeling for categorical fields; and (i) TabDLM-onlyMDLM, which removes the continuous diffusion branch and processes numerical features as serialized tokens. Detailed variant descriptions are provided in Appendix C.6. Table 4: Ablation study on the average performance across Adult, Default, and Shoppers Method Shape ↓ Trend ↓ MLE ↑ C2ST ↑ α-Precision ↑ TabDLM-noFloatAE 2.152.15 4.344.34 .870.870 0.92560.9256 98.5098.50 TabDLM-onlyContDiff 4.194.19 7.287.28 .850.850 0.83990.8399 91.7091.70 TabDLM-onlyMDLM 5.755.75 22.9622.96 .852.852 0.78920.7892 88.5888.58 TabDLM 1.561.56 2.492.49 .874.874 .9571.9571 98.5698.56 The full TabDLM consistently outperforms all three variants. The largest drop occurs in TabDLM-onlyMDLM, supporting our motivation in Section 1 that subword tokenization fragments precision-sensitive numerical values and hampers fine-grained numerical–categorical correlation modeling. TabDLM-noFloatAE and TabDLM-onlyContDiff also degrade noticeably, validating the necessity of pretrained numerical encoding and MDLM-based modeling for categorical fields, respectively. 5 Conclusion In this work, we propose TabDLM, the first unified framework that can generate high-fidelity synthetic tabular data with both numeric, categorical, and free-form text features. TabDLM leverages MDLM with special numerical tokenization to allow a single model to perform joint numerical-language diffusion. Extensive experiments validate the effectiveness of TabDLM over baseline models. We discuss limitations and future directions, including sampling efficiency and modality-specific noise schedules, in Appendix D. References [1] A. Alaa, B. Van Breugel, E. S. Saveliev, and M. Van Der Schaar (2022) How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models. In International conference on machine learning, p. 290–306. Cited by: §A.2.2. [2] M. Arriola, A. Gokaslan, J. T. Chiu, Z. Yang, Z. Qi, J. Han, S. S. Sahoo, and V. Kuleshov (2025) Block diffusion: interpolating between autoregressive and diffusion language models. arXiv preprint arXiv:2503.09573. Cited by: Appendix D. [3] S. A. Assefa, D. Dervovic, M. Mahfouz, R. E. Tillman, P. Reddy, and M. Veloso (2020) Generating synthetic data in finance: opportunities, challenges and pitfalls. In Proceedings of the First ACM International Conference on AI in Finance, p. 1–8. Cited by: §1. [4] J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. Van Den Berg (2021) Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems 34, p. 17981–17993. Cited by: §1, §2.1, §3. [5] B. M. Bolstad, R. A. Irizarry, M. Åstrand, and T. P. Speed (2003) A comparison of normalization methods for high density oligonucleotide array data based on variance and bias. Bioinformatics 19 (2), p. 185–193. Cited by: §2.2. [6] V. Borisov, K. Sessler, T. Leemann, M. Pawelczyk, and G. Kasneci (2023) Language models are realistic tabular data generators. In The Eleventh International Conference on Learning Representations, Cited by: §1, §3, §4.3. [7] T. Chen and C. Guestrin (2016) XGBoost: a scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Cited by: §4.2. [8] J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), p. 4171–4186. Cited by: §2.2. [9] S. Elfwing, E. Uchibe, and K. Doya (2018) Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks 107, p. 3–11. Cited by: Appendix B. [10] J. Fonseca and F. Bacao (2023) Tabular and latent space synthetic data generation: a literature review. Journal of Big Data 10 (1), p. 115. Cited by: §1. [11] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio (2014) Generative adversarial nets. Advances in neural information processing systems 27. Cited by: §3. [12] M. Hernandez, G. Epelde, A. Alberdi, R. Cilla, and D. Rankin (2022) Synthetic data generation for tabular health records: a systematic review. Neurocomputing 493, p. 28–45. Cited by: §1. [13] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. Advances in neural information processing systems 33, p. 6840–6851. Cited by: §1, §3. [14] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2022) Lora: low-rank adaptation of large language models.. ICLR 1 (2), p. 3. Cited by: §2.4. [15] A. E. Johnson, L. Bulgarelli, L. Shen, A. Gayles, A. Shammout, S. Horng, T. J. Pollard, S. Hao, B. Moody, B. Gow, et al. (2023) MIMIC-iv, a freely accessible electronic health record dataset. Scientific data 10 (1), p. 1. Cited by: §1. [16] T. Karras, M. Aittala, T. Aila, and S. Laine (2022) Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems 35, p. 26565–26577. Cited by: Appendix B, Appendix B. [17] J. Kim, C. Lee, and N. Park (2023) STaSy: score-based tabular data synthesis. In The Eleventh International Conference on Learning Representations, Cited by: §1, §4.3. [18] J. Kim, C. Lee, Y. Shin, S. Park, M. Kim, N. Park, and J. Cho (2022) Sos: score-based oversampling for tabular data. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, p. 762–772. Cited by: §1. [19] D. P. Kingma and M. Welling (2013) Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. Cited by: §3. [20] D. Kingma, T. Salimans, B. Poole, and J. Ho (2021) Variational diffusion models. Advances in neural information processing systems 34, p. 21696–21707. Cited by: §3. [21] A. Kotelnikov, D. Baranchuk, I. Rubachev, and A. Babenko (2023) Tabddpm: modelling tabular data with diffusion models. In International conference on machine learning, p. 17564–17579. Cited by: §1, §3, §4.3. [22] H. Lakkaraju, J. McAuley, and J. Leskovec (2013) What’s in a name? understanding the interplay between titles, content, and communities in social media. In Proceedings of the international AAAI conference on web and social media, Vol. 7, p. 311–320. Cited by: §1. [23] C. Lee, J. Kim, and N. Park (2023) Codi: co-evolving contrastive diffusion models for mixed-type tabular synthesis. In International Conference on Machine Learning, p. 18940–18956. Cited by: §1, §3, §4.3. [24] T. Liu, Z. Qian, J. Berrevoets, and M. van der Schaar (2023) Goggle: generative modelling for tabular data by learning relational structure. In The Eleventh International Conference on Learning Representations, Cited by: §3, §4.3. [25] M. Mueller, K. Gruber, and D. Fok (2023) Continuous diffusion for mixed-type tabular data. arXiv preprint arXiv:2312.10431. Cited by: §3. [26] S. Nie, F. Zhu, Z. You, X. Zhang, J. Ou, J. Hu, J. Zhou, Y. Lin, J. Wen, and C. Li (2025) Large language diffusion models. arXiv preprint arXiv:2502.09992. Cited by: §4. [27] Z. Nussbaum, J. X. Morris, A. Mulyar, and B. Duderstadt (2025) Nomic embed: training a reproducible long context text embedder. Transactions on Machine Learning Research. Note: Reproducibility Certification External Links: ISSN 2835-8856 Cited by: §4.2. [28] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. (2019) Pytorch: an imperative style, high-performance deep learning library. Advances in neural information processing systems 32. Cited by: Appendix B. [29] W. Peebles and S. Xie (2023) Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, p. 4195–4205. Cited by: §2.2. [30] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. (2019) Language models are unsupervised multitask learners. OpenAI blog 1 (8), p. 9. Cited by: §3. [31] J. Robinson, R. Ranjan, W. Hu, K. Huang, J. Han, A. Dobles, M. Fey, J. E. Lenssen, Y. Yuan, Z. Zhang, et al. (2024) Relbench: a benchmark for deep learning on relational databases. Advances in Neural Information Processing Systems 37, p. 21330–21341. Cited by: §A.1.3, §4.2. [32] T. Sattarov, M. Schreyer, and D. Borth (2023) Findiff: diffusion models for financial tabular data generation. In Proceedings of the Fourth ACM International Conference on AI in Finance, p. 64–72. Cited by: §1. [33] J. Shi, M. Xu, H. Hua, H. Zhang, S. Ermon, and J. Leskovec (2025) TabDiff: a mixed-type diffusion model for tabular data generation. In The Thirteenth International Conference on Learning Representations, Cited by: Appendix B, Appendix B, Appendix D, §1, §2.2, §3, §4.3. [34] J. Song, C. Meng, and S. Ermon (2020) Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. Cited by: §1. [35] Y. Song and S. Ermon (2019) Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems 32. Cited by: §1. [36] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole (2021) Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, Cited by: §2.1, §2.1, §3. [37] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. Advances in neural information processing systems 30. Cited by: §2.2. [38] Y. Wang, X. Wang, Q. Gan, M. Wang, Q. Yang, D. Wipf, and M. Zhang (2025) Griffin: towards a graph-centric relational database foundation model. arXiv preprint arXiv:2505.05568. Cited by: §2.2, §2.2. [39] C. Wu, H. Zhang, S. Xue, Z. Liu, S. Diao, L. Zhu, P. Luo, S. Han, and E. Xie (2025) Fast-dllm: training-free acceleration of diffusion llm by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505.22618. Cited by: Appendix D. [40] L. Xu, M. Skoularidou, A. Cuesta-Infante, and K. Veeramachaneni (2019) Modeling tabular data using conditional gan. Advances in neural information processing systems 32. Cited by: §1, §3, §4.3. [41] H. Yang, Y. Hu, S. Kang, Z. Lin, and M. Zhang (2024) Number cookbook: number understanding of language models and how to improve it. arXiv preprint arXiv:2411.03766. Cited by: §1. [42] J. You, X. Ma, Y. Ding, M. J. Kochenderfer, and J. Leskovec (2020) Handling missing data with graph representation learning. Advances in Neural Information Processing Systems 33, p. 19075–19087. Cited by: §1. [43] H. Zhang, L. Fang, Q. Wu, and P. S. Yu (2025) TabNAT: a continuous-discrete joint generative framework for tabular data. In International Conference on Machine Learning, Cited by: §3, §4.3. [44] H. Zhang, J. Zhang, Z. Shen, B. Srinivasan, X. Qin, C. Faloutsos, H. Rangwala, and G. Karypis (2024) Mixed-type tabular data synthesis with score-based diffusion in latent space. In The Twelfth International Conference on Learning Representations, Cited by: Appendix D, §1, §3, §4.3. [45] S. Zheng and N. Charoenphakdee (2022) Diffusion models for missing value imputation in tabular data. arxiv. arXiv preprint arXiv:2210.17128. Cited by: §1, §1, §3. [46] Y. Zhou, X. Wang, Y. Niu, Y. Shen, L. Tang, F. Chen, B. He, L. Sun, and L. Wen (2025) Difflm: controllable synthetic data generation via diffusion language models. In Findings of the Association for Computational Linguistics: ACL 2025, p. 20638–20658. Cited by: §1, §3, §4.3. Appendix A Detailed Experiment Setups A.1 Datasets A.1.1 MathExpr MathExpr is a synthetic dataset designed to evaluate the joint generation of heterogeneous tabular records involving numerical values, categorical operators, and free-form LaTeX expressions grounded in the structured columns. Each record contains two numerical columns, three categorical operator columns, and one text column: =(x1,x2,o1,o2,o3,elatex).r=(x_1,x_2,o_1,o_2,o_3,e_ latex). Here, o1o_1 and o2o_2 are unary operators applied to x1x_1 and x2x_2, respectively, while o3o_3 is a binary operator that combines the two transformed terms. The text column elatexe_ latex is a LaTeX expression deterministically constructed from the preceding structured columns. We generate 10,00010,000 records and split them into a training/real set and a validation set with a ratio of 9:19:1. During sampling, each model generates 9,0009,000 synthetic samples, and these synthetic samples are then used to compute the distribution fidelity metrics and expression consistency metrics described in Appendix A.2.6. Numerical value sampling. We sample x1x_1 and x2x_2 from discrete supports with step size 0.10.1: x1∈0.1,0.2,…,6.0,x2∈3.0,3.1,…,9.9.x_1∈\0.1,0.2,…,6.0\, x_2∈\3.0,3.1,…,9.9\. To induce a non-uniform yet diverse distribution, we sample x1x_1 and x2x_2 from an equal-weight mixture of a discrete Gaussian distribution and a uniform distribution over the corresponding support. The discrete Gaussian components are centered at μ1=3 _1=3 and μ2=6.5 _2=6.5, respectively, with shared standard deviation σ=1.5σ=1.5. Categorical operator sampling. The unary operators o1o_1 and o2o_2 are sampled independently from unary=none,log,exp,sqrt,sin,cos,tan,square,cube.O_unary=\ none, log, exp, sqrt, sin, cos, tan, square, cube\. Their categorical priors are: p(o1)= p(o_1)=\ none:0.18,log:0.16,sqrt:0.13,square:0.12,sin:0.10, none:18,\ log:16,\ sqrt:13,\ square:12,\ sin:10, cos:0.10,tan:0.07,exp:0.07,cube:0.07, cos:10,\ tan:07,\ exp:07,\ cube:07\, p(o2)= p(o_2)=\ none:0.22,sin:0.14,cos:0.14,sqrt:0.12,log:0.10, none:22,\ sin:14,\ cos:14,\ sqrt:12,\ log:10, square:0.09,tan:0.07,exp:0.06,cube:0.06. square:09,\ tan:07,\ exp:06,\ cube:06\. The binary operator o3o_3 is sampled from binary=add,sub,mul,div,O_binary=\ add, sub, mul, div\, with categorical prior p(o3)=add:0.35,mul:0.30,sub:0.20,div:0.15.p(o_3)=\ add:0.35,\ mul:0.30,\ sub:0.20,\ div:0.15\. Free-form LaTeX expression construction. The expression string elatexe_ latex is deterministically generated using a fixed grammar: unary operators are rendered as standard LaTeX commands (e.g., log↦ (⋅) log log(·), sqrt↦ ⋅ sqrt sqrt\·\, square↦(⋅)2 square (·)^2). Binary operators are rendered as ++, −-, × times, or ⋅⋅ frac\·\\·\. Table 5: An example sample from the MathExpr dataset. Column Example Value x1x_1 2.752.75 x2x_2 6.406.40 operation_x1 sin operation_x2 log operation_between mul latex_expression (2.75) × (6.40) Table 6: An example sample from the ProfileBio dataset. Column Example Value age 3838 salary 135135 sex female birth_state California college Harvard University degree master occupation software developer biography This female individual is in the career-building stage. She was born in California and completed higher education at Harvard University, earning a master degree. She works as a software developer. She earns a strong professional income. A.1.2 ProfileBio ProfileBio is a synthetic dataset for evaluating the joint generation of mixed-type personal profiles with a free-form biography text grounded in structured columns. Each record contains two numerical columns, five categorical columns, and one text column: =(age,salary,sex,birth_state,college,degree,occupation,biography).r=( age, salary, sex, birth\_state, college, degree, occupation, biography). The biography field is a natural-language paragraph deterministically constructed from the structured attributes. Following the setup of MathExpr, we generate 10,00010,000 records and split them into a training/real set and a validation set with a ratio of 9:19:1. During sampling, each model generates a synthetic dataset with the same cardinality as the real training set. We evaluate distribution fidelity and biography consistency using the metrics described in Appendix A.2.6. Categorical value sampling. We sample sex uniformly from male, female. The attributes birth_state and college are sampled from fixed categorical priors over 1010 states and 99 colleges, respectively. The degree attribute is sampled conditionally on college: for stanford university and harvard university, we use p(degree)=(0.01,0.29,0.40,0.30),p( degree)=(0.01,0.29,0.40,0.30), and for all other colleges, we use p(degree)=(0.30,0.50,0.15,0.05),p( degree)=(0.30,0.50,0.15,0.05), where the entries correspond to (associate,bachelor,master,doctoral)( associate, bachelor, master, doctoral). Finally, occupation is sampled from a degree-dependent categorical distribution. All occupation weights are initialized to 11. For doctoral degrees, the weights of research specialist and education professional are set to 66 and 44, respectively; for associate degrees, the weights of customer services professional and construction professional are set to 55 and 55, respectively. The weights are then normalized into a categorical distribution. Numerical value sampling. We sample age uniformly from integers in [21,70][21,70], and draw salary from a Gaussian whose mean depends on degree, occupation, and age. The degree-dependent base mean is μ0(degree)=82,associate,125,bachelor or master,178,doctoral, _0( degree)= cases82,& associate,\\ 125,& bachelor or master,\\ 178,& doctoral, cases and, letting ⋆=software developer,healthcare practitionerO =\ software developer,\, healthcare practitioner\, the full mean is μ=μ0(degree)+4⋅occupation∈⋆+0.3(age−45).μ\;=\; _0( degree)+4· 1\ occupation \+0.3\,( age-45). We then sample salary∼(μ, 52) salary (μ,\,5^2), round it to the nearest integer, and clip it to [75,200][75,200]. Biography construction. The biography field is deterministically constructed from the structured attributes using a fixed template. The age and salary values are first mapped to coarse-grained textual descriptors, and Table 7 summarizes both the descriptor mappings and the biography template. Additional clarification. ProfileBio is a fully synthetic dataset constructed from predefined sampling rules and deterministic templates. It does not rely on, copy, or imitate any real individuals. Sensitive personal attributes such as race, religion, health status, or immigration background are intentionally excluded. While attributes such as sex, birth_state, and occupation are included to evaluate cross-field consistency, the salary generation process is explicitly designed to depend only on age, degree, and occupation, and does not condition on sex or birth_state. ProfileBio is intended solely as a controlled benchmark for evaluating joint generation fidelity and attribute–text consistency, rather than as a model of real-world socioeconomic distributions. Table 7: ProfileBio: Age/salary mapping rules and the biography template. Component Rule / Template Age descriptor [21, 30]: in the early career stage [31, 40]: in the career-building stage [41, 50]: in the established career stage [51, 60]: in the advanced career stage [61, 70]: in the late career stage Salary descriptor [75, 100]: a comfortable income [101, 150]: a strong professional income [151, 200]: a high-level income Biography template This sex individual is age_desc. He/She was born in birth_state and completed higher education at college, earning a degree degree. He/She works as a occupation. He/She earns salary_desc. A.1.3 Amazon Amazon is a real-world free-text tabular dataset constructed from the RelBench [31] rel-amazon relational benchmark, which contains linked product metadata and user reviews. We join the review and product tables on product_id and form a single mixed-type table with two numerical columns, two categorical columns, and six free-form text columns: ( ( price,review_time,rating,verified,category, price,\ review\_time,\ rating,\ verified,\ category,\ brand,title,description,review_text,summary). brand,\ title,\ description,\ review\_text,\ summary). We sample 5,0005,000 examples in total and split them into a training/real set and a validation set with a ratio of 9:1. In addition, we sample another test set of 2,2502,250 examples for downstream utility evaluation. During sampling, each model generates 4,5004,500 synthetic examples, matching the cardinality of the real training set. We report distribution fidelity and downstream utility metrics as defined in Appendix A.2.6. Table 8 shows an example record. Table 8: An example sample from the Amazon dataset. Column Example Value price 5.985.98 review_time 19701970 (days since earliest review date) rating 5.05.0 verified true category Science Fiction & Fantasy > Fantasy brand Visit Amazon’s J. R. R. Tolkien Page title Hobbit description The enchanting prelude to "The Lord of the Rings" review_text My 13 yr. old grandson was very happy with this book. He likes to know the background of things and this helped him to understand this story. summary Grandson happy A.1.4 Arxiv Arxiv is another real-world free-text tabular dataset constructed from the RelBench rel-arxiv relational benchmark, which contains metadata and abstracts of arXiv papers. We construct a single mixed-type table at the paper level with four numerical columns, one categorical column, and three free-form text columns: ( ( submission_time,submission_year,title_length, submission\_time,\ submission\_year,\ title\_length, abstract_length,category,title,arxiv_code,abstract). abstract\_length,\ category,\ title,\ arxiv\_code,\ abstract). We sample 4,5004,500 examples for training and validation and split them into a real training set and a validation set with a ratio of 8:1. In addition, we sample another test set of 2,0002,000 examples for downstream utility evaluation. During sampling, each model generates a synthetic dataset with the same cardinality as the real training set. We report distribution fidelity and downstream utility metrics as defined in Appendix A.2.6. Table 9 shows an example record. Table 9: An example record from the Arxiv dataset. Column Example Value submission_time 16721672 (days since earliest review date) submission_year 20222022 title_length 77 abstract_length 9090 category Category_3 title IRC-safe Graph Autoencoder for unsupervised anomaly detection arxiv_code arXiv:2204.12231 abstract Anomaly detection through employing machine learning techniques has emerged as a novel powerful tool in the search for new physics beyond the Standard Model. Historically similar to the development of jet observables, theoretical consistency has not always assumed a central role in the fast development of algorithms and neural network architectures. In this work, we construct an infrared and collinear safe autoencoder based on graph neural networks by employing energy-weighted message passing. We demonstrate that whilst this approach has theoretically favourable properties, it also exhibits formidable sensitivity to non-QCD structures. A.1.5 Real-world Tabular Datasets We evaluate on five widely-used real-world tabular datasets from the UCI Machine Learning Repository.111https://archive.ics.uci.edu/datasets These datasets cover both classification and regression tasks, providing a diverse testbed for regular tabular generation with only numerical and categorical features. Specifically, Adult, Default, Shoppers, and Magic are used for classification, while Beijing is used for regression. Dataset statistics and the corresponding train/validation/test splits are summarized in Table 10. Table 10: Statistics of real-world tabular datasets. #Num denotes the number of numerical columns, and #Cat denotes the number of categorical columns. #Max Cat is the maximum number of categories among all categorical columns. Dataset #Rows #Num #Cat #Max Cat #Train #Validation #Test Task Adult 48,842 6 9 42 28,943 3,618 16,281 Classification Default 30,000 14 11 11 24,000 3,000 3,000 Classification Shoppers 12,330 10 8 20 9,864 1,233 1,233 Classification Magic 19,019 10 1 2 15,215 1,902 1,902 Classification Beijing 43,824 7 5 31 35,058 4,383 4,383 Regression A.2 Metrics A.2.1 Shape and Trend We evaluate distribution fidelity on all datasets using two general-purpose metrics: Shape and Trend. Shape and Trend are adopted from SDMetrics222https://docs.sdv.dev/sdmetrics, which quantify the marginal column-wise similarity and the pairwise dependency preservation between real and synthetic data, respectively. Shape. Kolmogorov-Smirnov Test (KST): This metric quantifies the alignment between the real distribution pr(x)p_r(x) and the synthetic distribution ps(x)p_s(x) by calculating the maximum divergence between their respective Cumulative Distribution Functions (CDFs): KST=supx|Fr(x)−Fs(x)|,KST= _x|F_r(x)-F_s(x)|, (17) where Fr(x)F_r(x) and Fs(x)F_s(x) denote the CDFs derived from the probability densities: F(x)=∫−∞xp(t)dt.F(x)= _-∞^xp(t)dt. (18) Total Variation Distance (TVD): For categorical variables, we evaluate the discrepancy in probability mass using the TVD. It is defined as half the sum of the absolute differences between the category frequencies observed in the real data, R(ω)R(ω), and the synthetic data, S(ω)S(ω): TVD=12∑ω∈Ω|R(ω)−S(ω)|,TVD= 12 _ω∈ |R(ω)-S(ω)|, (19) where Ω represents the set of all possible categories within a given column. Trend. Pearson Correlation Score: We examine the preservation of linear dependencies between continuous columns using the Pearson correlation coefficient, ρx,y _x,y, defined as: ρx,y=Cov(x,y)σxσy, _x,y= Cov(x,y) _x _y, (20) where Cov represents covariance and σ denotes the standard deviation. To evaluate the overall preservation of these trends, we compute the Pearson Score as the normalized average absolute error between the correlation matrices of the real (ρRρ^R) and synthetic (ρSρ^S) datasets: Pearson Score=12x,y|ρR(x,y)−ρS(x,y)|.Pearson Score= 12E_x,y|ρ^R(x,y)-ρ^S(x,y)|. (21) Since ρ∈[−1,1]ρ∈[-1,1], the factor of 1/21/2 normalizes the score to the range [0,1][0,1], where a lower score indicates superior correlation preservation. Contingency Similarity: To measure the consistency of pairwise associations between categorical columns A and B, we utilize a metric based on the Total Variation Distance applied to contingency tables. The Contingency Score is calculated as: Contingency Score=12∑α∈A∑β∈B|Rα,β−Sα,β|,Contingency Score= 12 _α∈ A _β∈ B|R_α,β-S_α,β|, (22) where Rα,βR_α,β and Sα,βS_α,β correspond to the joint frequencies of category pair (α,β)(α,β) in the real and synthetic datasets, respectively. A.2.2 α-Precision Following Alaa et al. [1], α-Precision is a sample-level fidelity metric that measures whether each synthetic sample falls within the α-support of the real data distribution, i.e., the smallest region containing an α fraction of the real probability mass. Intuitively, it quantifies how typical synthetic samples are with respect to the real distribution rather than lying in low-density or out-of-distribution regions. A higher α-Precision indicates higher fidelity of the generated samples. A.2.3 Detection Score The Detection Score is computed via the Classifier Two-Sample Test (C2ST) implemented in SDMetrics, where a logistic regression classifier is trained to distinguish real samples from synthetic ones. The score is derived from the classifier’s misclassification rate: when the synthetic distribution closely matches the real one, the classifier performs near chance level, yielding a higher Detection Score. We report the Detection Score in [0,1][0,1], with higher values indicating that synthetic samples are statistically indistinguishable from real samples. A.2.4 Machine Learning Efficiency For datasets with an associated downstream prediction task, we additionally report Machine Learning Efficiency (MLE) to assess utility via the test-performance gap between models trained on real versus synthetic samples. In particular, we apply MLE to the regular real-world tabular datasets. During evaluation, we train the XGBoost classifier on the real training set (further split with an 8:1 ratio for validation and hyperparameter tuning) and evaluate it on a held-out real test set. We then train an identical classifier on the synthetic dataset and evaluate it on the same real test set. The MLE score is defined by the divergence between the two test performances, reflecting how well synthetic data can serve as a substitute for real data in downstream predictive modeling. A.2.5 Distance Closest Record (DCR) DCR evaluates whether the generative model memorizes training samples rather than learning the underlying distribution, serving as a data privacy metric. For each synthetic sample, we compute its distance to the nearest record in both the training set and a held-out test set, and report the proportion of synthetic samples whose nearest neighbor lies in the training set. A score close to 50%50\% is ideal: it indicates that synthetic samples are equally close to training and test data, suggesting the model has captured the distribution rather than reproducing training instances. A.2.6 Dataset-specific Metrics MathExpr. Beyond standard distribution metrics (Shape and Trend), we evaluate whether the generated free-form LaTeX expression is structurally and numerically consistent with the structured number and operation columns. We report: (1) Operation Match Rate (Op-MR), which verifies if the unary/binary operator tokens implied by the expression elatexe_ latex strictly align with the structured operations (o1,o2,o3)(o_1,o_2,o_3); and (2) Expression match rate (Exp-MR), which evaluates the joint validity of structure and values. Specifically, a generated expression is considered a match only if it (i) satisfies operation correctness (as in Op-MR) and (i) contains two numeric literals whose values align with the structured fields (x1,x2)(x_1,x_2) up to a small relative tolerance. Concretely, letting x^1,x^2 x_1, x_2 be the literals extracted from the generated LaTeX string, we require |x^i−xi|/xi≤δ x_i-x_i /x_i≤δ for i∈1,2i∈\1,2\. We fix δ=0.07δ=0.07 to make the metric robust to minor numeric drift that can arise from stochastic generation and continuous-value approximation (e.g., 0.290.29 vs. 0.300.30), while still penalizing outputs whose numeric content is meaningfully inconsistent with the structured inputs. ProfileBio. For ProfileBio, we assess cross-modality consistency between structured attributes and the generated biography text. We compute a rule-based Match Rate by checking whether the biography instantiates the required template slots with values consistent with the corresponding structured fields (see the example template in Table 7). Concretely, for each record we verify that the text reflects core attributes (e.g., sex, birth_state, college, degree, occupation) as well as the derived descriptors for continuous variables (Age and Salary). Since mapping continuous values into discrete natural-language descriptors introduces semantic fuzziness near bin boundaries, we apply a small boundary relaxation tolerance δ=0.06δ=0.06 when validating the age/salary descriptors. This avoids rigid thresholding artifacts where values close to a boundary may legitimately share descriptions from neighboring categories; for instance, age 30.530.5 can reasonably be described as either “in the early career stage” or “in the career-building stage.” Amazon. For the Amazon dataset, we measure downstream utility via an MLE-like protocol, with rating as the target label in 1,2,3,4,5\1,2,3,4,5\ (treated as a multi-class classification task). We first serialize all non-numerical fields into structured text and encode them using the sentence embedding model Nomic. We then train an XGBoost classifier using either (i) text embeddings only or (i) text embeddings concatenated with numerical features, which helps isolate the contribution of generated text/categorical fields. We report Macro-AUC on the held-out test set as the utility metric. Arxiv. Similar to the Amazon dataset, we measure downstream utility via an MLE-like protocol, with Category as the target label (treated as a multi-class classification task). The original rel-arxiv benchmark contains more than 5050 paper categories, which would induce severe class imbalance and weaken the signal of cross-field consistency; we therefore restrict our benchmark to samples from the 1515 most frequent categories, yielding a relatively balanced 1515-way classification task. We serialize all non-numerical fields into structured text and encode them using the sentence embedding model Nomic. We then train an XGBoost classifier using either (i) text embeddings only or (i) text embeddings concatenated with numerical features. For each configuration, Macro-AUC on the held-out test is reported as the utility metric. Appendix B Implementation Details We implement TabDLM based on PyTorch [28], the code is provided in an anonymous link https://github.com/ilikevegetable/TabDLM. All experiments are run on an NVIDIA A100 GPU with 80GB of memory. Data preprocessing. We follow the same preprocessing method as prior diffusion-based tabular synthetic model [33]. Missing numerical values are imputed with the column mean, and missing categorical values are treated as an additional category. To stabilize optimization across heterogeneous numerical scales, we apply a quantile-based transformation to numerical columns during training and invert the transform after sampling to recover values in the original space. Data splits. We adopt the same data split protocol as the TabDiff setting: each dataset is partitioned into a real set and a test set. Models are trained on the real set. For downstream utility evaluation, we further split the real set into training and validation subsets and reserve the test set strictly for evaluation. Architecture. We build TabDLM on top of the LLaDA-8B base model for all experiments. To incorporate numerical channels, each scalar value is first mapped into a d-dimensional latent using a lightweight pretrained float encoder, implemented as a 3-layer MLP with hidden width ⌊d⌋ d and SiLU activations [9]. A float decoder (LayerNorm + linear projection) maps the latent back to a scalar. We set the numerical latent dimension to d=512d=512 by default and keep the pretrained float encoder/decoder frozen during joint training. To interface numerical latents with the MDLM embedding space, we further apply a two-stage projection: an input projector that maps per-feature latents from d to the MDLM hidden size D using a 2-layer MLP (d→1024→Dd\!→\!1024\!→\!D) with SiLU and dropout (with LayerNorm), and an output projector that maps MDLM hidden states back to the numerical latent space via a symmetric 2-layer MLP (D→1024→dD\!→\!1024\!→\!d). The dimension D in LLaDA-8B is 4096. Hyperparameter settings. We fine-tune models with LoRA using the same configuration for TabDLM, LLaDA-8BSFT_SFT, and Qwen2.5-7BSFT_SFT to ensure fair comparison. Unless otherwise stated, we use LoRA rank r=16r=16, scaling factor α=32α=32, and dropout 0.050.05, and apply LORA to the attention and MLP components of Transformer blocks. Across methods, we keep the optimizer and training recipe identical; the only dataset-dependent choice is the number of training epochs due to varying dataset sizes. Specifically, we train for 10 epochs on Adult and Beijing, 30 epochs on Shoppers, 15 epochs on Magic and Default, and 75 epochs on MathExpr, Amazon, Arxiv, and ProfileBio. We use AdamW with learning rate 2×10−42× 10^-4, warmup ratio 0.10.1, (β1,β2)=(0.9,0.98)( _1, _2)=(0.9,0.98), weight decay 10−410^-4, and ϵ=10−8ε=10^-8. All experiments use the same fixed random seed and bf16 training when enabled. Noise schedule. For the continuous numerical diffusion, we adopt the per-feature power-mean noise schedule following TabDiff [33], which extends the EDM parameterization [16] with a learnable shape parameter ρi _i for each numerical column. The schedule smoothly interpolates between σmin _ and σmax _ over normalized time t∈[0,1]t∈[0,1]; we set σmin=0.002 _ =0.002 and σmax=80.0 _ =80.0 following EDM defaults. Formally, for each numerical feature i∈1,…,Mnumi∈\1,…,M_num\: σρinum(t)=(σmin1/ρi+t(σmax1/ρi−σmin1/ρi))ρi.σ^num_ _i(t)= ( _ ^1/ _i+t ( _ ^1/ _i- _ ^1/ _i ) ) _i. (23) Learning ρi _i per column accommodates heterogeneous numerical distributions and yields a better-conditioned diffusion process than a single global ρ. Algorithm 1 Sampling 1:Schema prompt p; generation length G; number of reverse steps T; noise schedules σtt=0T\ _t\_t=0^T and churned schedule σ^tt=0T\ σ_t\_t=0^T; remasking policy π∈HighConf,Randπ\!∈\!\ HighConf, Rand\. 2:Synthetic record =(xj)j∈ℐnum∪ℐcat∪ℐtextx= (x_j )_j _num _cat _text. 3:// Initialization 4:←EMB()z_p (p) 5:Ttok←[MASK]Gx_T^tok← [MASK]^G ⊳ discrete state for ℐcat∪ℐtextI_cat\!∪\!I_text 6:Tnum∼(,σmax2)x_T^num \! (0,\, _ ^2I ) ⊳ initial state from a Gaussian prior 7:for t=T,T−1,…,1t=T,T-1,…,1 do 8: // (i) Numerical perturbation (EDM churn) 9: ϵ∼(,) ε (0,I); ^tnum←tnum+σ^t2−σt2ϵ x_t^num _t^num+ σ_t^2- _t^2\; ε 10: // (i) Map both modalities to MDLM token embeddings 11: tnum←PROJe(ENC(^tnum))z_t^num _e\! (ENC( x_t^num) ) ⊳ Eq. (8) 12: ttok←EMB(ttok)z_t^tok (x_t^tok) ⊳ Eq. (9) 13: t←[;ttok;tnum]z_t← [z_p;\,z_t^tok;\,z_t^num ] 14: // (i) Joint bidirectional denoising 15: t=[tp;ttok;tnum]←MDLM(t,t)o_t= [o_t^p;\,o_t^tok;\,o_t^num ] (z_t,\,t) ⊳ Eq. (10) 16: // (iv) Discrete reverse step: progressive unmasking 17: pθ(⋅∣ttok)←HEAD(ttok)p_θ(· _t^tok) (o_t^tok) ⊳ Eq. (12) 18: ~0tok∼pθ(⋅∣ttok) x_0^tok p_θ(· _t^tok) ⊳ sample candidate tokens 19: t−1tok←Unmaskπ(ttok,~0tok,pθ,t)x_t-1^tok← Unmask_π\! (x_t^tok,\, x_0^tok,\,p_θ,\,t ) ⊳ reveal G/TG/T positions per step 20: // (v) Continuous reverse step: EDM–Euler update 21: ~tnum←DEC(PROJd(tnum)) x_t^num \! (PROJ_d(o_t^num) ) ⊳ Eq. (11) 22: t←(^tnum−~tnum)/σ^td_t← ( x_t^num- x_t^num ) / σ_t 23: t−1num←^tnum+(σt−1−σ^t)tx_t-1^num← x_t^num+ ( _t-1- σ_t )\,d_t ⊳ discretization of Eq. (2) 24:end for 25:// Finalization 26:xjj∈ℐcat∪ℐtext←Detokenize(0tok) \x_j \_j _cat _text (x_0^tok) 27:xjj∈ℐnum←Denorm(0num) \x_j \_j _num (x_0^num) ⊳ numerical denormalization 28:return x Sampling. TabDLM performs joint sampling over discrete tokens (categorical/text) and continuous numerical features through a coupled reverse process discretized into T steps; the full procedure is summarized in Algorithm 1. At each step, we (i) inject EDM-style churn noise [16] into the numerical state, (i) embed both modalities into a shared MDLM input by mapping numerical values through the number encoder and tokens through the embedding table, (i) run a single bidirectional MDLM forward pass that jointly conditions text denoising on noisy numericals and vice versa, and (iv) apply modality-specific reverse updates: progressive unmasking on the discrete side and an EDM–Euler step on the continuous side. For discrete unmasking we consider two policies: (i) high-confidence unmasking, which reveals positions with the highest predicted token probability, and (i) random unmasking, which reveals a uniformly random subset. In both cases we reveal G/TG/T tokens per step, ensuring a smooth transition from a fully-masked sequence to a fully-specified one. After the final step, we decode tokens through the LM head and denormalize the numerical values to obtain the final mixed-type sample. Appendix C Additional Experimental Results C.1 Full results of Shape and Trend Metrics Shape and Trend are fidelity metrics measuring column-wise marginal distributions and pair-wise column dependencies, respectively. Tables 11 and 12 report the complete results on real-world tabular datasets without free-form text features. Table 11: Performance comparison on the error rates (%) of Shape ↓ . (B: best overall; B: best in language-based model) Method Adult Default Shoppers Magic Beijing Average CTGAN 16.84±0.0316.84± 0.03 16.83±0.0416.83± 0.04 21.15±0.1021.15± 0.10 9.81±0.089.81± 0.08 21.39±0.0521.39± 0.05 17.2017.20 TVAE 14.22±0.0814.22± 0.08 10.17±0.0510.17± 0.05 24.51±0.0624.51± 0.06 8.25±0.068.25± 0.06 19.16±0.0619.16± 0.06 15.2615.26 GOGGLE 16.9716.97 17.0217.02 22.3322.33 1.901.90 16.9316.93 15.0315.03 STaSy 11.29±0.0611.29± 0.06 5.77±0.065.77± 0.06 9.37±0.099.37± 0.09 6.29±0.136.29± 0.13 6.71±0.036.71± 0.03 7.897.89 CoDi 21.38±0.0621.38± 0.06 15.77±0.0715.77± 0.07 31.84±0.0531.84± 0.05 11.56±0.2611.56± 0.26 16.94±0.0216.94± 0.02 19.5019.50 TabDDPM 1.75±0.031.75± 0.03 1.57±0.081.57± 0.08 2.72±0.132.72± 0.13 1.01±0.091.01± 0.09 1.30±0.031.30± 0.03 1.671.67 TabSyn 0.81±0.050.81± 0.05 1.01±0.081.01± 0.08 1.44±0.071.44± 0.07 1.03±0.141.03± 0.14 1.26±0.051.26± 0.05 1.111.11 TabDiff 0.63±0.050.63± 0.05 1.24±0.071.24± 0.07 1.28±0.091.28± 0.09 0.78±0.080.78± 0.08 1.03±0.051.03± 0.05 0.990.99 TabNAT 0.730.73 0.80 1.11 0.62 0.79 0.81 GReaT 12.12±0.0412.12± 0.04 19.94±0.0619.94± 0.06 14.51±0.1214.51± 0.12 16.16±0.0916.16± 0.09 8.25±0.128.25± 0.12 14.2014.20 DiffLM 9.749.74 9.069.06 10.0710.07 7.537.53 6.356.35 8.558.55 Qwen2.5-7BICL_ICL 27.8027.80 22.3922.39 34.3734.37 13.4113.41 31.8331.83 25.9625.96 Qwen2.5-14BICL_ICL 25.1725.17 21.5221.52 51.2151.21 17.5117.51 29.7629.76 29.0329.03 Qwen2.5-7BSFT_SFT 5.115.11 2.582.58 8.768.76 7.537.53 7.497.49 6.296.29 LLaDA-8BSFT_SFT 1.691.69 2.002.00 13.5613.56 5.605.60 3.643.64 5.305.30 TabDLM 1.461.46 1.181.18 2.052.05 2.932.93 2.422.42 2.012.01 Table 12: Performance comparison on the error rates (%) of Trend ↓ .(B: best overall; B: best in language-based model) Method Adult Default Shoppers Magic Beijing Average CTGAN 20.23±1.2020.23± 1.20 26.95±0.9326.95± 0.93 13.08±0.1613.08± 0.16 7.00±0.197.00± 0.19 22.95±0.0822.95± 0.08 18.0418.04 TVAE 14.15±0.8814.15± 0.88 19.50±0.9519.50± 0.95 18.67±0.3818.67± 0.38 5.82±0.495.82± 0.49 18.01±0.0818.01± 0.08 15.2315.23 GOGGLE 45.2945.29 21.9421.94 23.9023.90 9.479.47 45.9445.94 29.3129.31 STaSy 14.51±0.2514.51± 0.25 5.96±0.265.96± 0.26 8.49±0.158.49± 0.15 6.61±0.536.61± 0.53 8.00±0.108.00± 0.10 8.718.71 CoDi 22.49±0.0822.49± 0.08 68.41±0.0568.41± 0.05 17.78±0.1117.78± 0.11 6.53±0.256.53± 0.25 7.07±0.157.07± 0.15 24.4624.46 TabDDPM 3.01±0.253.01± 0.25 4.89±0.104.89± 0.10 6.61±0.166.61± 0.16 1.70±0.221.70± 0.22 2.71±0.092.71± 0.09 3.783.78 TabSyn 1.93±0.071.93± 0.07 2.81±0.482.81± 0.48 2.13±0.102.13± 0.10 0.88±0.180.88± 0.18 3.13±0.343.13± 0.34 2.182.18 TabDiff 1.49±0.161.49± 0.16 2.55±0.752.55± 0.75 1.74±0.081.74± 0.08 0.76±0.120.76± 0.12 2.59±0.152.59± 0.15 1.83 TabNAT 1.611.61 2.21 1.67 1.371.37 2.48 1.871.87 GReaT 17.59±0.2217.59± 0.22 70.02±0.1270.02± 0.12 45.16±0.1845.16± 0.18 10.23±0.4010.23± 0.40 59.60±0.5559.60± 0.55 40.5240.52 Qwen2.5-7BICL_ICL 37.6137.61 31.0231.02 34.7734.77 20.6220.62 40.8340.83 32.9732.97 Qwen2.5-14BICL_ICL 42.6742.67 28.7228.72 42.3242.32 18.1118.11 38.2038.20 34.0034.00 Qwen2.5-7BSFT_SFT 17.0317.03 16.9916.99 10.2810.28 14.4514.45 28.6828.68 17.4917.49 LLaDA-8BSFT_SFT 26.1526.15 26.3026.30 16.4216.42 13.7113.71 27.9027.90 22.1022.10 TabDLM 2.742.74 2.332.33 2.402.40 2.852.85 2.932.93 2.652.65 C.2 Evaluation results of MLE for real-world tabular dataset In this section, we provide the evaluation results of MLE for real-world tabular datasets without free-form text features in Table 13. As shown in Table 13, TabDLM achieves competitive MLE performance on Adult, Default, Shoppers, and Magic, demonstrating its ability to faithfully capture the underlying data distribution and support high-quality synthetic data for downstream training. Notably, TabDLM attains the best overall score on Shoppers, and on Default even surpasses models trained on real data, suggesting that TabDLM can produce samples that are both distributionally consistent and beneficial for improving generalization. We observe a noticeable performance drop on Beijing, which may be caused by the large fraction of missing values in the target column, where our mean-imputation preprocessing could distort the true target distribution and hurt likelihood-based metrics such as MLE. Table 13: Evaluation of MLE. AUC is used for classification tasks and RMSE for regression tasks. (B: best overall; B: best in language-based model) Methods Adult Default Shoppers Magic Beijing Average Gap AUC↑ AUC↑ AUC↑ AUC↑ RMSE↓ %\% Real .927±.000.927±.000 .770±.005.770±.005 .926±.001.926±.001 .946±.001.946±.001 .423±.003.423±.003 0.00.0 CTGAN .886±.002.886±.002 .696±.005.696±.005 .875±.009.875±.009 .855±.006.855±.006 .902±.019.902±.019 28.4828.48 TVAE .878±.004.878±.004 .724±.005.724±.005 .871±.006.871±.006 .887±.003.887±.003 .770±.011.770±.011 21.0921.09 GOGGLE .778±.012.778±.012 .584±.005.584±.005 .658±.052.658±.052 .654±.024.654±.024 1.09±.0251.09±.025 51.5451.54 STaSy .906±.001.906±.001 .752±.006.752±.006 .914±.005.914±.005 .934±.003.934±.003 .656±.014.656±.014 12.4512.45 CoDi .871±.006.871±.006 .525±.006.525±.006 .865±.006.865±.006 .932±.003.932±.003 .818±.021.818±.021 27.8627.86 TabDDPM .907±.001.907±.001 .758±.004.758±.004 .918±.005.918±.005 .935±.003.935±.003 .592±.011.592±.011 9.149.14 TabSyn .909±.001.909±.001 .763±.002.763±.002 .914±.004.914±.004 .937±.002.937±.002 .580±.009.580±.009 8.448.44 TabDiff .912±.002.912±.002 .763±.005.763±.005 .921±.004.921±.004 .936±.003.936±.003 .555±.013.555±.013 7.07 TabNAT .904.904 .764.764 .916.916 .935.935 .579.579 8.488.48 GReaT .913±.003.913±.003 .755±.006.755±.006 .902±.005.902±.005 .888±.008.888±.008 .653±.013.653±.013 13.3113.31 DiffLM .906.906 .794 .915.915 .917.917 .696.696 13.5913.59 Qwen2.5-7BICL_ICL .853.853 .390.390 .811.811 .829.829 .991.991 43.2843.28 Qwen2.5-14BICL_ICL .852.852 .639.639 .640.640 .844.844 1.031.03 42.0542.05 Qwen2.5-7BSFT_SFT .915 .769.769 .895.895 .918.918 .674.674 13.4113.41 LLaDA-8BSFT_SFT .909.909 .774.774 .872.872 .919.919 .679.679 14.1314.13 TabDLM .907.907 .791.791 .923 .905.905 .696.696 13.7313.73 C.3 Evaluation results of α-Precision We report α-Precision results in Table 14. TabDLM achieves the best overall score on Default and the best average score among language-based models. Compared with tabular-specific synthetic models that are designed exclusively for numerical and categorical features, TabDLM remains competitive while uniquely supporting free-form text generation. Table 14: Evaluation on α-Precision scores, (B: best overall; B: best in language-based model) Method Adult Default Shoppers Magic Beijing Average CTGAN 77.74±0.1577.74± 0.15 62.08±0.0862.08± 0.08 76.97±0.3976.97± 0.39 86.90±0.2286.90± 0.22 96.27±0.1496.27± 0.14 79.9979.99 TVAE 98.17±0.1798.17± 0.17 85.57±0.3485.57± 0.34 58.19±0.2658.19± 0.26 86.19±0.4886.19± 0.48 97.20±0.1097.20± 0.10 85.0685.06 GOGGLE 50.6850.68 68.8968.89 86.9586.95 90.8890.88 88.8188.81 77.2477.24 STaSy 82.87±0.2682.87± 0.26 90.48±0.1190.48± 0.11 89.65±0.2589.65± 0.25 86.56±0.1986.56± 0.19 89.16±0.1289.16± 0.12 87.7487.74 CoDi 77.58±0.4577.58± 0.45 82.38±0.1582.38± 0.15 94.95±0.3594.95± 0.35 85.01±0.3685.01± 0.36 98.13±0.3898.13± 0.38 87.6187.61 TabDDPM 96.36±0.2096.36± 0.20 97.59±0.3697.59± 0.36 88.55±0.6888.55± 0.68 98.59±0.1798.59± 0.17 97.93±0.3097.93± 0.30 95.8095.80 TabSyn 99.39±0.1899.39± 0.18 98.65±0.2398.65± 0.23 98.36±0.5298.36± 0.52 99.42±0.2899.42± 0.28 97.51±0.2497.51± 0.24 98.6798.67 TabDiff 99.02±0.2099.02± 0.20 98.49±0.2898.49± 0.28 99.11±0.3499.11± 0.34 99.47±0.2199.47± 0.21 98.06±0.2498.06± 0.24 98.8398.83 TabNAT 98.6798.67 99.2799.27 97.6797.67 99.50 99.27 98.88 GReaT 55.79±0.0355.79± 0.03 85.90±0.1785.90± 0.17 78.88±0.1378.88± 0.13 85.46±0.5485.46± 0.54 98.32±0.2298.32± 0.22 80.8780.87 Qwen2.5-7BICL_ICL 85.5185.51 88.5688.56 27.9627.96 83.3983.39 63.7663.76 69.8469.84 Qwen2.5-14BICL_ICL 68.5368.53 83.1783.17 8.938.93 89.9989.99 84.5384.53 67.0367.03 Qwen2.5-7BSFT_SFT 84.7084.70 95.2595.25 95.4695.46 82.4582.45 94.3594.35 90.4490.44 LLaDA-8BSFT_SFT 99.3299.32 97.6897.68 68.7368.73 85.0985.09 95.1395.13 89.1989.19 TabDLM 97.6297.62 99.56 98.5198.51 97.9097.90 97.8797.87 98.2998.29 Table 15: Evaluation on C2ST, (B: best overall; B: best in language-based model) Method Adult Default Shoppers Magic Beijing Average CTGAN 0.59490.5949 0.48750.4875 0.74880.7488 0.67280.6728 0.75310.7531 0.65140.6514 TVAE 0.63150.6315 0.65470.6547 0.29620.2962 0.77060.7706 0.86590.8659 0.64380.6438 GOGGLE 0.11140.1114 0.51630.5163 0.14180.1418 0.95260.9526 0.47790.4779 0.44000.4400 STaSy 0.40540.4054 0.68140.6814 0.54820.5482 0.69390.6939 0.79220.7922 0.62420.6242 CoDi 0.20770.2077 0.45950.4595 0.27840.2784 0.72060.7206 0.71770.7177 0.47680.4768 TabDDPM 0.97550.9755 0.97120.9712 0.83490.8349 0.9998 0.95130.9513 0.94650.9465 TabSyn 0.99100.9910 0.9826 0.96620.9662 0.99600.9960 0.95280.9528 0.97770.9777 TabDiff 0.9950 0.97740.9774 0.9843 0.99890.9989 0.97810.9781 0.9867 TabNAT 0.98700.9870 0.96570.9657 0.96260.9626 0.99890.9989 0.9845 0.97970.9797 GReaT 0.53760.5376 0.47100.4710 0.42850.4285 0.43260.4326 0.68930.6893 0.51180.5118 Qwen2.5-7BICL_ICL 0.09250.0925 0.30890.3089 0.05910.0591 0.51190.5119 0.17870.1787 0.23020.2302 Qwen2.5-14BICL_ICL 0.15640.1564 0.48660.4866 0.01860.0186 0.59030.5903 0.27020.2702 0.30440.3044 Qwen2.5-7BSFT_SFT 0.80370.8037 0.91310.9131 0.57220.5722 0.78450.7845 0.73150.7315 0.76100.7610 LLaDA-8BSFT_SFT 0.93660.9366 0.91400.9140 0.51710.5171 0.84890.8489 0.97990.9799 0.83930.8393 TabDLM 0.9386 0.9734 0.9594 0.9254 0.95280.9528 0.9499 C.4 Evaluation results of C2ST Table 15 reports the C2ST detection score, where higher values indicate that synthetic samples are statistically harder to distinguish from real ones. TabDLM achieves the best result among all language-based models, improving the average C2ST score by 13.2%13.2\% compared with the strongest baseline. Compared with tabular-specific diffusion models designed exclusively for numerical and categorical features, TabDLM still achieves competitive performance. C.5 Evaluation results of DCR Table 16 reports the DCR score, which serves as a privacy metric: a value closer to 50%50\% indicates that synthetic samples are equally close to training and held-out test records, suggesting the model has learned the underlying distribution rather than memorized training instances. TabDLM achieves the best overall average deviation across all baselines and obtains the best per-dataset DCR on Default and Beijing. This indicates that, despite being built on a large pretrained MDLM backbone, TabDLM does not exhibit memorization behavior and generates samples that genuinely reflect the data distribution. Table 16: Evaluation on DCR score, where a score closer to 50%50\% is more preferable. We also report the average absolute deviation of the DCR score from 50%50\% across all datasets, where lower values are better (B: best overall). Method Adult Default Shoppers Beijing Average ↓ STaSy 50.33%±0.1950.33\%± 0.19 50.23%±0.0950.23\%± 0.09 51.53%±0.1651.53\%± 0.16 50.59%±0.2950.59\%± 0.29 0.67%0.67\% CoDi 49.92%±0.1849.92\%± 0.18 51.82%±0.2651.82\%± 0.26 51.06%±0.1851.06\%± 0.18 50.87%±0.1150.87\%± 0.11 0.96%0.96\% TabDDPM 51.14%±0.1851.14\%± 0.18 52.15%±0.2052.15\%± 0.20 63.23%±0.2563.23\%± 0.25 80.11%±2.6880.11\%± 2.68 11.66%11.66\% TabSyn 50.94%±0.1750.94\%± 0.17 51.20%±0.1851.20\%± 0.18 52.90%±0.2252.90\%± 0.22 50.37%±0.1350.37\%± 0.13 1.35%1.35\% TabDiff 50.10%±0.3250.10\%± 0.32 51.11%±0.3651.11\%± 0.36 50.24%±0.6250.24\%± 0.62 50.50%±0.3650.50\%± 0.36 0.39%0.39\% TabDLM 50.73%50.73\% 49.95%49.95\% 50.87%50.87\% 50.13%50.13\% 0.36%0.36\% C.6 Description of Ablation Variants The variants in ablation study are defined as follows. (i) TabDLM-noFloatAE replaces the pretrained and frozen float encoder/decoder with randomly initialized modules of the same architecture, trained jointly with the rest of the model. (i) TabDLM-onlyContDiff converts categorical features into one-hot vectors and feeds them through the continuous diffusion branch, removing MDLM-based modeling for categorical fields. (i) TabDLM-onlyMDLM serializes numerical features and processes them in the same way as categorical and textual features, removing the continuous diffusion branch entirely. The aggregated results are reported in Table 4. Appendix D Limitations First, the sampling efficiency of TabDLM is lower than that of existing tabular data generation methods such as TabDiff [33] and TabSyn [44]. This limitation can be primarily attributed to the large model size of the MDLM backbone. Nevertheless, TabDLM is designed for broader application scenarios, as it supports the generation of numerical, categorical, and free-form text fields, capabilities that exceed those of existing methods. Moreover, common methods for accelerating MDLM inference, like block diffusion [2] and KV caching [39], could be incorporated to improve sampling efficiency. However, it is out of the scope of this work. Second, TabDLM models numerical and language diffusion using a shared noise schedule that couples the denoising dynamics across modalities and enforces a common step size for denoising both numerical and textual features. While effective in practice, this design may be suboptimal in certain scenarios. Introducing modality-specific noise schedules to partially decouple the denoising processes can be promising, which we leave for future investigation.

Instant research discovery

TabDLM: Free-Form Tabular Data Generation via Joint Numerical-Language Diffusion

Intelligence

Summary

Entities (10)

Relation Signals (9)

Abstract

Tags

Links

PDF

Full Text

Paper	Published	Area	Tags	Intel	Citations