Paper deep dive
NesTok: Nested Self-Aligned 1D Tokenizer for Autoregressive Image Generation
Jiawei Zhang, Shuhao Liu, Rong Huang, Yuancheng Li, Zhihui Li, Xiaojun Chang, Changlin Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/1/2026, 7:28:09 AM
Summary
The paper introduces NesTok, a nested self-aligned 1D visual tokenizer for autoregressive image generation. It addresses the suboptimal performance of existing variable-length tokenizers by proposing a cross-length joint training strategy and a nested self-alignment loss. This framework ensures that shorter token sequences align with full-length representations, improving reconstruction quality and codebook utilization. NesTok achieves state-of-the-art results on ImageNet, with an rFID of 0.98 and a gFID of 1.46 on 256x256 images.
Entities (10)
Relation Signals (7)
NesTok → achieves → gFID 1.46
confidence 95% · On downstream image generation, it achieves the state-of-the-art gFID score of 1.46 on ImageNet 256×256
NesTok → achieves → rFID 0.98
confidence 95% · On ImageNet, NesTok improves substantially over standard training and achieves an rFID score of 0.98.
NesTok → uses → nested self-alignment
confidence 95% · We propose a nested self-aligned tokenizer that jointly optimizes reconstruction across token lengths while aligning the semantic representations
NesTok → uses → cross-length joint training
confidence 95% · NesTok introduces cross-length training, which jointly optimizes reconstruction across token lengths
NesTok → improvesupon → Nested Dropout
confidence 90% · existing approaches based on nested dropout often fail to fully exploit the representational capacity... In this work, we introduce NesTok
NesTok → pairedwith → LlamaGen
confidence 90% · When paired with LlamaGen, NesTok achieves a generation FID (gFID) of 1.46
NesTok → isbasedon → One-D-Piece
confidence 85% · Our 1D variable-length tokenizer is based on One-D-Piece (Miwa et al., 2025)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:One-dimensional (1D) variable-length visual tokenizers enable adaptive compression by varying the number of tokens, allowing downstream autoregressive (AR) models to flexibly trade off generation quality against computational cost using a single tokenizer. However, existing approaches based on nested dropout often fail to fully exploit the representational capacity of the tokenizer, resulting in suboptimal performance in both image reconstruction and generation. In this work, we introduce NesTok, a nested self-alignment framework tailored to dynamic visual tokenizers. NesTok introduces cross-length training, which jointly optimizes reconstruction across token lengths while using the full-length sequence to guide shorter counterparts, enabling shorter token sequences to approach the reconstruction quality of full-length sequences. On ImageNet, NesTok improves substantially over standard training and achieves an rFID score of 0.98. On downstream image generation, it achieves the state-of-the-art gFID score of 1.46 on ImageNet 256$\times$256 among existing variable-length autoregressive image generation methods. Code will be available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.36756v2
- Canonical: https://arxiv.org/abs/2609.36756v2
Trouble viewing inline? Open PDF directly →
Full Text
53,787 characters extracted from source content.
Expand or collapse full text
NesTok: Nested Self-Aligned 1D Tokenizer for Autoregressive Image Generation Jiawei Zhang Shuhao Liu∗ Rong Huang Yuancheng Li †thanks: Equal contribution: zjw1637, shuhao@ncepu.edu.cn. Affiliation: North China Electric Power University Zhihui Li Xiaojun Chang Changlin Li †thanks: Corresponding author: changlinli.ai@gmail.com. Affiliation: University of Science and Technology of China Stanford University Abstract One-dimensional (1D) variable-length visual tokenizers enable adaptive compression by varying the number of tokens, allowing downstream autoregressive (AR) models to flexibly trade off generation quality against computational cost using a single tokenizer. However, existing approaches based on nested dropout often fail to fully exploit the representational capacity of the tokenizer, resulting in suboptimal performance in both image reconstruction and generation. In this work, we introduce NesTok, a nested self-alignment framework tailored to dynamic visual tokenizers. NesTok introduces cross-length training, which jointly optimizes reconstruction across token lengths while using the full-length sequence to guide shorter counterparts, enabling shorter token sequences to approach the reconstruction quality of full-length sequences. On ImageNet, NesTok improves substantially over standard training and achieves an rFID score of 0.98. On downstream image generation, it achieves the state-of-the-art gFID score of 1.46 on ImageNet 256×256 among existing variable-length autoregressive image generation methods. Code will be available at https://github.com/jaiwei804/NesTok. 1 Introduction Image generation has achieved remarkable success under both diffusion-based and autoregressive (AR) paradigms (Ho et al., 2020; Li et al., 2026; Lee et al., 2022a; Sun et al., 2024b; Lu et al., 2026), with visual tokenizers playing a pivotal role in this progress. Recent studies (Yu et al., 2024b; He et al., 2025; Kim et al., 2025; Xiong et al., 2025) have explored one-dimensional (1D) visual tokenization to achieve higher compression ratios while preserving reconstruction fidelity. This formulation is conceptually closer to tokenization in natural language and has attracted increasing attention. However, most existing 1D tokenizers operate at a fixed token length and cannot adapt compression ratios within a single model. Supporting different computational budgets and target quality levels therefore requires multiple model variants. Dynamic visual tokenizers address this limitation by mapping images to variable-length latent sequences of continuous embeddings or discrete codes (Miwa et al., 2025; Bachmann et al., 2025; Wen et al., 2025; Gao and Shou, 2026; Huang et al., 2025; Duggal et al., 2025; Wang et al., 2025), allowing the token budget to be adjusted according to computational constraints or image complexity. A prevalent strategy is nested dropout (Bachmann et al., 2025), which randomly truncates the latent sequence and remains the prefix during training, encouraging an ordered representation in which early tokens capture the most important visual information. Representative approaches include FlexTok (Bachmann et al., 2025), Semanticist (Wen et al., 2025) and One-D-Piece (Miwa et al., 2025). When shorter representations correspond to prefixes of the full-length sequence, a downstream AR generator can be trained exclusively on full-length sequences and generate prefixes of different lengths at inference time, enabling flexible trade-offs between generation quality and computational cost. However, flexibility in token length is not necessarily equivalent to effective use of additional tokens. Prior studies (Fu et al., 2026; He et al., 2025; Bachmann et al., 2025; Wen et al., 2025) have shown that increasing sequence length may yield diminishing improvements in downstream AR generation quality or even degrade it. This observation suggests inefficient information allocation across token lengths: shorter prefixes may provide insufficient representations, while later tokens may fail to contribute useful complementary information. Effective variable-length tokenization therefore requires both informative prefixes and progressive refinement through additional tokens. To improve information allocation, ReTok (Fu et al., 2026) introduces redundant token padding, whereas CaTok (Chen et al., 2026) couples token selection intervals with time intervals in the MeanFlow objective to encourage a more balanced distribution of information. We analyze codebook statistics and AR generation performance on ImageNet-1K, as shown in Fig. 1. Fig. 1 (a) shows that training with nested dropout results in substantially reduced code diversity at later token positions, reflected in their low normalized entropy. The result suggests underutilization of later tokens, potentially limiting their ability to complement the early tokens. Fig. 1 (b) further shows that nested dropout achieves a gFID of only 2.46, suggesting that concentrated code distribution and tail-token collapse lead to poor generation quality. Beyond information allocation, decoder design also affects the simplicity and efficiency of variable-length image generation. Recent work has explored diffusion or flow-based decoders (Bachmann et al., 2025; Wen et al., 2025; Gao and Shou, 2026) to improve generation quality, potentially introducing additional architectural complexity and decoding latency. We focus on discrete variable-length tokenizers with ViT decoders, retaining the standard next-token prediction paradigm for downstream autoregressive generation. Within this setting, a central challenge is to learn representations that remain effective across token lengths: shorter prefixes should capture global visual content, while later tokens should contribute details for progressive refinement. Meeting these requirements calls for training strategies that coordinate learning across token lengths while supporting downstream AR modeling. This motivates us to revisit the representations learned by variable-length tokenizers: How can a single tokenizer learn informative prefixes and useful refinements that together support effective autoregressive generation? Figure 1: Codebook statistics and AR generation performance on ImageNet-1K. (a): Normalized entropy of the code distribution at each token position, computed over the training dataset and averaged within groups of 16 positions. (b): Average codebook utilization across token positions and gFID. (c): Sampling efficiency with different methods. (d): Sampling efficiency for different token lengths with different methods. In this work, we introduce NesTok, a variable-length 1D tokenizer built on ViT and trained through nested self-alignment learning, which substantially improves autoregressive image generation. We first establish a training recipe for learning a variable-length tokenizer from scratch, without relying on any external distillation. Building on this recipe, we propose a nested self-aligned tokenizer that jointly optimizes reconstruction across token lengths while aligning the semantic representations of shorter sequences with those of their full-length counterparts. At each training iteration, representations associated with shorter token sequences are encouraged to align with the full-length sequence in latent space. We empirically demonstrate that this mechanism alleviates the collapse of tail-token information associated with nested dropout. Fig. 1 (a) shows that NesTok preserves code diversity across token positions, with a gradual entropy decline consistent with progressive refinement. In Fig. 1 (b), NesTok increases average codebook utilization from 19.73% to 98.41% and reduces gFID from 2.46 to 1.92, outperforming the fixed-length baseline (2.33). Fig. 1 (c) demonstrates a favorable trade-off between quality and latency across model sizes. With a single AR generator, NesTok also consistently improves generation quality as token length increases, achieving lower gFID at comparable sampling times (Fig. 1 (d)). We validate the effectiveness of NesTok on ImageNet-1K 256×256 image generation. NesTok achieves a reconstruction FID (rFID) of 0.98 with 256 latent tokens (vs. 1.93 for our variable-length 1D baseline). When paired with LlamaGen, NesTok achieves a generation FID (gFID) of 1.46 with vanilla AR models, without introducing additional diffusion components. Furthermore, generation quality improves consistently as the number of tokens increases across token lengths. We hope that this work encourages further exploration of ViT-based variable-length visual tokenizers for autoregressive image generation. Our contributions are as follows: • We introduce NesTok, a variable-length 1D ViT tokenizer trained without external distillation to support flexible reconstruction and effective autoregressive generation. • We propose cross-length joint training and nested self-alignment to improve token utilization and establish a coarse-to-fine ordering of visual information. • NesTok achieves an rFID of 0.98 with 256 tokens and a gFID of 1.46 on ImageNet-1K 256×256256× 256 using vanilla AR generation. A trained generator supports different token lengths, with generation quality improving consistently as more tokens are generated. 2 Related Work Image Tokenizers. Image tokenizers compress high-dimensional images into compact continuous or discrete latent representations for efficient generative modeling. Variational autoencoders (VAEs) (Kingma and Welling, 2013; Zeng et al., 2026; Yao et al., 2025) map images into continuous latent spaces and are optimized using a reconstruction loss with KL-divergence regularization. VQ-VAE (Van Den Oord et al., 2017; Razavi et al., 2019) and VQGAN (Esser et al., 2021) instead encode images into fixed 2D grids of discrete tokens. Subsequent studies (Yu et al., 2023; Yu et al., 2024a; Mentzer et al., 2024; Lee et al., 2022b; Shi et al., 2025) introduce advanced quantization techniques to improve reconstruction fidelity, codebook utilization, and downstream generation performance. Recently, 1D tokenizers (Yu et al., 2024b; Chen et al., 2025; Qu et al., 2026) have transformed images into compact token sequences, substantially reducing the spatial redundancy inherent in conventional 2D grids. The latest work, EOSTok (Chu et al., 2026) unifies the conventional two-stage tokenizer and generator pipeline through end-to-end training. 1D Variable-length Visual tokenizers. Compared with fixed-length 1D tokenizers, flexible visual tokenizers aim to encode images into variable-length token sequences using a single model (Bachmann et al., 2025; Miwa et al., 2025; Huang et al., 2025). Existing approaches can be categorized by their decoding mechanisms: 1) ViT decoders: One-D-Piece (Miwa et al., 2025) employs nested tail dropping together with two-stage training and external distillation. SpectralAR (Huang et al., 2025) introduces a spectral information loss to encourage causal structure in 1D token sequences. ReTok (Fu et al., 2026) improves dynamic tokenization through redundant tokens and hierarchical semantic regularization. 2) Diffusion/Flow Decoders: FlexTok (Bachmann et al., 2025) replaces the ViT decoder with a rectified-flow decoder. FlexTok achieves the best gFID with 32 tokens, while generating additional tokens degrades generation quality. D-AR (Gao and Shou, 2026) and CaTok (Chen et al., 2026) associate time steps in the decoding process with intervals of the token sequence to enable efficient image reconstruction. Despite these advances, unbalanced information allocation across token lengths remains a challenge, limiting the benefits of variable-length tokenization for downstream autoregressive generation. We propose a nested self-aligned training framework, which requires no external distillation and aims to alleviate this imbalance while improving ordered representations suitable for autoregressive modeling. Modern Autoregressive Visual Generation. Inspired by GPT-style language models, early autoregressive image generators built on discrete visual tokenizers, such as VQ-VAE and VQGAN (Sun et al., 2024a; Van Den Oord et al., 2017; Esser et al., 2021), flatten two-dimensional token grids into one-dimensional sequences and generate tokens in raster-scan order using causal attention and next-token prediction (NTP). However, this fixed ordering restricts the use of bidirectional spatial context. MAR (Li et al., 2024) and MaskGIT (Yu et al., 2023) employ masked prediction with bidirectional attention to incorporate context from visible tokens during iterative generation. VAR (Tian et al., 2024) adopts next-scale prediction, reformulating image generation as a coarse-to-fine process that progressively introduces finer visual details. RAR (Yu et al., 2025) and RandAR (Pang et al., 2025) further explore randomized token prediction orders to improve contextual modeling. Although effective, these approaches introduce changes to the generation order, attention pattern, or prediction target of vanilla autoregressive modeling. Unlike these approaches that adapt the generator to visual data, we focus on reshaping the visual representations themselves. Our method encourages more balanced information allocation across token lengths while improving the compatibility of visual tokens with a vanilla autoregressive model without requiring additional changes to the generation mechanism. 3 Method 1D tokenizers have attracted considerable attention for their flexible compression capabilities. However, enabling efficient downstream autoregressive modeling with variable-length tokenizers remains challenging. Our goal is to learn variable-length token sequences in which short prefixes provide effective image representations and additional tokens contribute complementary refinements. To this end, NesTok combines cross-length joint training with nested self-alignment. These mechanisms aim to improve information utilization and encourage a coarse-to-fine ordering that enhances the autoregressive compatibility of the learned tokens. We first introduce the architecture of our ViT variable-length 1D tokenizer (Section 3.1), then present our nested self-alignment training method (Section 3.2), and finally introduce variable-length autoregressive modeling (Section 3.3). 3.1 1D Vision Tokenizer and Autoregressive Model Our 1D variable-length tokenizer is based on One-D-Piece (Miwa et al., 2025), an input image ∈ℝH×W×Cx ^H× W× C is partitioned into non-overlapping patches of size f×f× f and mapped to a sequence of patch embeddings p∈ℝN×Dx_p ^N× D, where N=HW/f2N=HW/f^2, C is the number of image channels, and f=16f=16. These embeddings are concatenated with latent tokens ∈ℝK×Dq ^K× D and processed by the encoder ℰψE_ψ to obtain [ℰ,]=ℰψ([p,])[h_E,z]=E_ψ([x_p,q]). We discard the encoded patch representations ℰh_E and retain the 1D latent representations z, which are subsequently quantized as q=()z_q=Q(z) by looking up the closest entry using a vector quantizer. The quantized latent tokens are then concatenated with mask tokens p∈ℝN×Dm_p ^N× D and passed to the decoder ϕD_φ for reconstruction, yielding [∅,r]=ϕ([q,p])[ ,x_r]=D_φ([z_q,m_p]). Here, pm_p is formed by repeating a mask token N times, and ∅ denotes the discarded latent tokens. Consider a variable-length 1D tokenizer that encodes an image into a quantized token sequence q=[1,2,…,K]z_q=[z_1,z_2,…,z_K] using an encoder ℰE and a quantizer Q. Here, K denotes the full sequence length, and kz_k is the token at position k. During training, we uniformly sample an integer k∈1,…,Kk∈\1,…,K\ and truncate the sequence to retain a prefix of length k, denoted by q(k)=[1,…,k]z_q^(k)=[z_1,…,z_k]. The retained prefix is then passed to the decoder D to reconstruct the input image. For the discrete tokenizer considered here, we use a composite training objective comprising four terms (Yu et al., 2024c): ℒrec=ℒmse+ℒperc+ℒquant+ℒadv.L_rec=L_mse+L_perc+L_quant+L_adv. (1) Here, ℒmseL_mse measures the mean squared error between the reconstructed image x and the input image x. The perceptual loss ℒpercL_perc combines LPIPS with a feature perceptual loss computed using ConvNeXt-S. The quantization loss ℒquantL_quant comprises the codebook and commitment terms, while ℒadvL_adv is an adversarial loss that encourages visually realistic reconstructions. In addition to gradient-based updates driven by the codebook loss, we apply auxiliary usage-adaptive codebook updates guided by exponential moving average (EMA) estimates of codebook usage (Zheng and Vedaldi, 2023). Although most discrete visual tokenizers are trained with this form Wu et al. (2025), their performance is highly sensitive to the training recipe. We therefore establish a standard training recipe as our baseline, with details provided in Section 4. Autoregressive Compatibility. The inherent spatial continuity of images introduces redundancy across neighboring patches. Although 1D ViT tokenizers reorganize spatial features into a latent sequence, their encoders jointly optimize token representations for reconstruction without explicitly encouraging an information ordering suitable for autoregressive prediction. Consequently, strong reconstruction performance does not necessarily imply strong autoregressive compatibility. Nested dropout encourages reconstruction from shorter prefixes, but later tokens are supplied to the decoder less frequently, potentially contributing to unbalanced information allocation and limiting the benefits of larger token budgets. Therefore, we propose a nested self-alignment training framework to jointly improve information allocation and token ordering (Section 3.2). Figure 2: Overview of the nested self-aligned training pipeline. We train a 1D variable-length tokenizer using two components: (1) cross-length sampling, which jointly optimizes reconstruction from a full-length sequence and a sampled shorter sequence at each iteration; and (2) a nested self-alignment loss, which aligns latent representations across the two lengths to promote cross-length consistency and improve reconstruction. 3.2 Nested Self-Alignment Learning Cross-Length Joint Training. We introduce a cross-length joint training strategy to improve training across token lengths. For an input image x, let q(K)=[1,…,K]z_q^(K)=[z_1,…,z_K] denote full-length quantized latent sequence. At each training iteration i, we retain the full-length sequence q(K)z_q^(K) and sample a shorter length sequence q(ri)z_q^(r_i), where ri∼1,…,Kr_i \1,…,K\. The resulting sequence q(ri)z_q^(r_i) is therefore a prefix of q(K)z_q^(K). Both sequences are decoded, and their losses are accumulated to jointly optimize the tokenizer: ℒjoint=ℒrec(,K)+ℒrec(,ri),L_joint=L_rec(x;K)+L_rec(x;r_i), (2) The two samples share all tokenizer parameters. The full-length sequence ensures that every latent token position is retained at least one decoding pass per iteration, while the shorter sequence trains the tokenizer to reconstruct images with a variable length to compress the reconstruction information. Nested Self-Alignment Loss. Although cross-length joint training provides consistent supervision for full length reconstruction, it does not explicitly encourage feature consistency across token lengths. Empirically, we observe that shorter prefixes improve slowly and retain a substantial reconstruction gap relative to full-length sequences. Despite parameter sharing, improvements in full length reconstruction do not necessarily translate into comparable improvements at shorter lengths. This motivates an explicit mechanism for transferring alignment information at feature level across lengths. To this end, we introduce a nested self-alignment loss. With n∈ri,Kn∈\r_i,K\, we extract the output feature of the ℓ -th layer of the decoder: z(n,ℓ),p(n,ℓ)=ϕ(1:ℓ)(q(n),p),h_z^(n, ),h_p^(n, )=D_φ^(1: )(z_q^(n),m_p), (3) where ϕ(1:ℓ)D_φ^(1: ) denotes Parameters of the first ℓ layers of the decoder and z(n,ℓ)∈ℝn×dh_z^(n, ) ^n× d and (n,ℓ)∈ℝN×dh_p^(n, ) ^N× d denote the features of latent tokens and mask tokens, respectively. Unless otherwise specified, we apply alignment at the first decoder layer, i.e., ℓ=1 =1. Since the two sampled sequences contain different numbers of latent tokens, we apply average pooling along the token dimension to obtain a pooled latent feature, ¯z(n,ℓ)=Avg(z(n,ℓ)) h_z^(n, )=Avg(h_z^(n, )), where z(n,ℓ)h_z^(n, ) contains the n latent-token features. We then concatenate this feature, treated as a single token, with the mask-token features along the token dimension to form (n,ℓ)=Concat(¯z(n,ℓ),p(n,ℓ))∈ℝ(N+1)×df^(n, )=Concat( h_z^(n, ),h_p^(n, )) ^(N+1)× d. We define the nested self-alignment loss as: ℒalign=1−1N+1∑j=1N+1sim(j(ri,ℓ),stopgrad[j(K,ℓ)]),L_align=1- 1N+1 _j=1^N+1sim(f_j^(r_i, ),stopgrad[f_j^(K, )]), (4) The full-length features serve as an online alignment target for shorter sequences. This supervision in feature space encourages shorter sequences to rapidly align their representations with those of the full length sequences, improving reconstruction quality at shorter token length without requiring an additional teacher network. Overall objective of Nested Self-Aligned Learning. As shown in Fig. 2, our training framework combines cross-length joint training with nested self-alignment. Cross-length joint training decodes the full-length sequence at every iteration, so every position receives reconstruction supervision. Tail tokens can therefore reduce the full-length loss only by adding details that the prefix lacks, which makes them residual refinements rather than redundant copies. Second, the self-alignment loss pulls the decoder features of each prefix toward those of the full-length sequence, which serves as a stop-gradient target. Since the prefix is a prefix of the full sequence, the most effective way to shrink this gap is to place the information in the earliest tokens that best approximates the full representation, such as global layout and semantics, and to leave what the prefix cannot capture to later tokens. The overall training objective is: ℒtotal=ℒjoint+ℒalignL_total=L_joint+L_align (5) 3.3 Variable-length Autoregressive Model Variable-length AR generation relies on an ordered token sequence, in which every prefix is a progressively refined description of the same image. Nested dropout provides only a weak form of this ordering. Because early positions are retained under almost every truncation, they are pushed to carry the most important content. Later positions, however, are rarely supervised. As a result, the sequence tends to become informative head, idle tail rather than coarse-to-fine. NesTok strengthens the ordering in two complementary ways. For the downstream generator, this means that early predictions decide the global content, and later predictions refine it conditioned on that content. This ordering is consistent with next-token prediction. Autoregressive Modeling. We model discrete visual token sequences through standard autoregressive next-token prediction: pθ(1:n)=∏i=1npθ(i∣<i),p_θ(z_1:n)= _i=1^np_θ(z_i _<i), (6) where θ denotes the model parameters and n is the token sequence length. We train the model on full-length sequences by minimizing the cross-entropy loss. At inference time, a desired token budget n≤Kn≤ K can be specified, where K is the maximum sequence length. The same generator then samples an n-token prefix, which is decoded into an image by the tokenizer decoder. Thus, a single trained generator supports variable-length inference, enabling flexible trade-offs between generation quality and computational cost. We additionally employ KV-cache to accelerate autoregressive sampling. Table 1: System-level comparison of different tokenizers and generation models on ImageNet 256×256. ↓ and ↑ indicate whether lower or higher values are better. We categorize tokenizers into three groups: 2D tokenizers, fixed-length 1D tokenizers, and variable-length 1D tokenizers. Method Tokenizer Generator w/o guidance w/ guidance Type #Params #Tokens rFID↓ Type #Params gFID↓ IS↑ gFID↓ IS↑ 2D Tokenization DiT-XL/2 (Peebles and Xie, 2023) SD-VAE 84M 256 0.62 Diff. 675M 9.62 121.5 2.27 278.2 REPA-XL/2 (Yu et al., 2024d) KL 84M 1024 0.62 Diff. 675M 5.90 157.8 1.42 305.7 Lightning-DiT-XL (Yao et al., 2025) KL 84M 1024 0.28 Diff. 675M 2.17 205.6 1.35 295.3 MAR-L (Li et al., 2024) KL 66M 256 0.87 MAR Diff. 479M 2.60 221.4 1.78 296.0 VQGAN (Esser et al., 2021) VQ 23M 256 4.98 AR 1.4B 15.78 74.3 - - MaskGIT (Chang et al., 2022) VQ 66M 256 2.28 Mask 227M 6.18 182.1 - - LlamaGen-XL (Sun et al., 2024a) VQ 72M 256 0.94 AR 775M 14.77 80.8 2.62 244.1 RAR-L (Yu et al., 2025) VQ 66M 256 2.28 AR 461M 5.39 149.1 1.70 299.5 IBQ-L (Shi et al., 2025) IBQ 128M 256 1.37 AR 649M - - 2.45 267.5 VAR-d20 (Tian et al., 2024) MSRQ 109M 680 0.90 VAR 600M - - 2.57 302.6 AliTok-L (Wu et al., 2025) VQ 390M 273 0.86 AR 318M 1.98 200.8 1.38 326.2 1D Tokenization TiTok-L (Yu et al., 2024b) VQ 641M 32 2.21 Mask 177M 3.15 173.0 2.77 199.8 GigaTok (Xiong et al., 2025) VQ 622M 256 0.81 AR 111M - - 3.26 221.0 SoftVQ-VAE-L (Chen et al., 2025) KL 176M 64 0.61 Diff. 675M 5.83 141.3 2.93 268.5 ResTok (Zhang et al., 2026) VQ 662M 128 1.28 HAR 326M - - 2.34 257.8 MacTok (Zeng et al., 2026) KL 675M 128 0.43 Diff. 326M 3.12 186.2 1.50 299.8 SemTok-XL (Qu et al., 2026) VQ 2.35B 256 0.88 Mask 746M - - 2.54 305.6 SemTok-XXL VQ 2.35B 256 0.88 Mask 1.2B - - 2.34 310.5 SpectralAR-d20 (Huang et al., 2025) VQ - 64 4.03 AR 600M - - 2.49 305.4 D-AR-L (Gao and Shou, 2026) VQ 300M 256 1.52 AR 343M - - 2.44 262.9 D-AR-XL VQ 300M 256 1.52 AR 775M - - 2.09 298.4 EOSTok-H (Chu et al., 2026) IBQ 388M 256 0.71 AR 644M 1.48 239.5 1.38 265.7 1D Variable-length Tokenization FlexTok d18-d18-32 (Bachmann et al., 2025) FSQ 950M 1-256 1.61 AR 1.33B - - 2.02 - FlexTok d18-d28-32 FSQ 2.5B 1-256 1.45 AR 1.33B - - 1.86 - Semanticist-XL-32 (Wen et al., 2025) KL - 1-256 0.78 AR Diff. 343M - - 2.57 254.0 One-D-Piece (Miwa et al., 2025) VQ 641M 1-256 1.08 Mask 318M - - 2.35 224.4 DetailFlow-32 (Liu et al., 2025) VQ 64M 1-256 0.80 AR 326M - - 2.75 250.8 ReTok (Fu et al., 2026) VQ 232M 1-256 1.01 AR 775M - - 2.27 245.9 NesTok-B VQ 390M 1-256 0.98 AR 177M 2.59 181.6 1.73 250.5 NesTok-L VQ 390M 1-256 0.98 AR 318M 2.08 205.6 1.50 278.9 NesTok-XL VQ 390M 1-256 0.98 AR 662M 1.87 238.9 1.46 295.9 4 Experiment 4.1 Implementation Details Model Configuration and Evaluation Our 1D variable-length tokenizer is based on One-D-Piece (Miwa et al., 2025) and uses ViT-B as the encoder and ViT-L as the decoder, with a codebook of 4,096 entries and a latent dimension of 32. Our autoregressive model adopts the decoder-only Transformer architecture of LlamaGen (Sun et al., 2024a). To accommodate 1D latent sequences, we replace 2D rotary positional embeddings (RoPE) with 1D RoPE (Su et al., 2024). We named our tokenizer as NesTok and denote the corresponding autoregressive models at different scales as NesTok-B/L/XL. Following prior work, we evaluate performance using Fréchet Inception Distance (FID), including reconstruction FID (rFID) and generation FID (gFID), as well as Inception Score (IS). We compute gFID on 50,000 generated images using the ADM’s TensorFlow evaluation suite (Dhariwal and Nichol, 2021) and evaluate rFID using the MAR evaluation code (Li et al., 2024). Training of Tokenizer. We train NesTok in three stages without any external teacher for distillation or representation alignment. The nested self-alignment loss is applied throughout training, with a batch size of 256. In the first stage, we jointly optimize the reconstruction, perceptual, and quantization losses for 200K iterations, warming up the learning rate to 1e-4. In the second stage, we disable the LPIPS of the perceptual loss, retaining only the loss computed using ConvNeXt-S. This stage lasts 200K iterations, during which the learning rate decreases from 5e-5 to 2e-5. In the third stage, we additionally introduce the GAN loss to improve the visual fidelity of reconstructions. We train for 150K steps, decreasing the learning rate from 2e-5 to 5e-6. Training of Autoregressive Model. Following common practice, we train the LlamaGen with a batch size of 2,048 and a learning rate of 4e-4 with a warmup for 100 epochs. We train for 400 epochs totally, corresponding to approximately 250K steps. We apply QK-Norm (Team, 2024) in the attention modules and use RMSNorm (Zhang and Sennrich, 2019) for normalization. Before AR training, we precompute and cache the discrete visual token sequences using the tokenizer’s encoder and quantizer to accelerate training. Figure 3: Training curves. (a) Training loss. (b) Training accuracy (%). (c) gFID without classifier-free guidance across training steps and model sizes. Figure 4: Examples of image generation with NesTok-XL on ImageNet 256×256256× 256. Figure 5: Visualization of variable-length generation on ImageNet 256×256256× 256 resolution. Images are generated using NesTok-XL with 32-256 tokens. Figure 6: gFID w/ and w/o cfg across different methods and model sizes. 4.2 Main Results Generation Results on ImageNet 256×256. We evaluate NesTok against state-of-the-art methods on ImageNet-1K at 256×256256× 256 resolution. As shown in Tab. 1, NesTok achieves strong downstream autoregressive generation performance while maintaining high reconstruction quality. Specifically, NesTok achieves an rFID of 0.98, outperforming One-D-Piece’s 1.08 with fewer tokenizer parameters (390M vs. 641M) and without external distillation. Among the compared 1D variable-length methods, NesTok-XL achieves the state-of-the-art gFID of 1.46, compared with 2.35 for One-D-Piece. It achieves a gFID of 1.87 without classifier-free guidance, outperforming the other variable-length methods. Although EOSTok achieves the best generation quality among the compared 1D tokenization methods, it jointly trains the tokenizer and AR generator and relies on DINOv2-based distillation. Fig. 4 presents qualitative generation results obtained with NesTok. In Fig. 5, as the token length increases from 32 to 256, the images exhibit progressively finer details while maintaining consistent global structure. Variable-length Autoregressive Model. As shown in Fig. 6, for each model configuration, we evaluate a single trained AR generator across token lengths with and without classifier-free guidance (cfg). We compare NesTok with One-D-Piece-L (Miwa et al., 2025) and ReTok-L (Fu et al., 2026). NesTok-L substantially outperforms both methods across all evaluated token lengths. While One-D-Piece-L exhibits degraded generation quality at larger budgets without cfg, ReTok-L shows diminishing gains: increasing the sequence length from 128 to 256 tokens reduces gFID by only 0.01, from 3.79 to 3.78. Over the same interval, NesTok-L and NesTok-XL reduce gFID by 0.20 and 0.25, respectively. In Fig. 6 (c), NesTok consistently benefits from increasing the token lengths, with improvements continuing at longer sequence lengths. These results demonstrate the autoregressive compatibility of the learned representations and suggest that our nested self-alignment training architecture enables more effective use of additional tokens, including those at later positions. Training Analysis. Fig. 3 illustrates the training dynamics and generation performance of autoregressive models built on NesTok. Fig. 3 (a) and (b) show steadily decreasing training loss and increasing training accuracy, respectively. Fig. 3 (c) reports gFID scores using 256 tokens without classifier-free guidance. Scaling up the AR generator consistently improves gFID. These results support the autoregressive compatibility of the token sequences learned by NesTok, suggesting that our training strategy produces representations well suited to autoregressive prediction. 4.3 Ablation Studies. Ablation of Components. We conduct an ablation study to evaluate the contributions of individual components. As shown in Table 2, the fixed-length 1D Tokenizer achieves the best rFID but does not support variable-length representations. Introducing nested dropout enables variable-length tokenization but degrades reconstruction quality. This variant achieves an AR training accuracy of 46.2%, yet yields a gFID of 2.46. According to Fig. 1, the result suggests that nested dropout is associated with concentrated code usage and tail-token collapse. Cross-length joint training mitigates this issue, ensuring that all latent-token positions participate in reconstruction training. Adding the nested self-alignment loss further promotes cross-length consistency and improves reconstruction from shorter sequences. Although this loss yields little improvement in full-length generation quality, it substantially improves generation at shorter lengths. With both components, NesTok achieves a gFID of 1.92, outperforming the 1D Tokenizer’s 2.33. Fig. 9 further shows that NesTok preserves the main visual content with short prefixes and restores finer details as the token length increases. Figure 7: Visualization of variable-length reconstruction on ImageNet 256×256256× 256 resolution. 1D denotes the 1D Tokenizer with 256 tokens. Table 2: Ablation study of different training settings. We report generation results using LlamaGen-Base (177M) trained for 200 epochs. Training Setting AR Training Evaluation Loss↓ Acc.↑ rFID↓ gFID↓ 32 64 128 256 32 64 128 256 1D Tokenizer 5.37 7.9% - - - 0.83 - - - 2.33 +Nested Dropout 2.37 46.2% 4.28 2.88 2.07 1.93 5.98 4.64 3.12 2.46 +Cross-length Sample 4.70 11.3% 3.70 2.14 1.43 0.98 5.76 4.16 2.36 1.93 +Nested Self-Alignment Loss 4.68 11.4% 3.49 2.10 1.39 0.98 5.48 3.85 2.18 1.92 Method Tokens Params gFID images/s ReTok-L 256 343M 2.27 10.11 One-D-Piece-L 256 318M 2.35 42.67 DetailFlow 256 326M 2.75 19.69 Semanticist-L 32 343M 2.57 1.03 NesTok-L 256 318M 1.50 16.35 Table 3: Comparison of sampling speed. Throughput (images/s) measured on a single H800 GPU with a batch size of 128, averaged over ten sampling runs using the official implementation of each method. Comparison of sampling speed. We compare the sampling speed of different methods in terms of throughput. Although One-D-Piece achieves high throughput with its MaskGIT generator, its generation quality is comparatively limited. Semanticist employs a diffusion decoder and achieves a throughput of only 1.03 images per second even with 32 tokens, limiting its suitability for applications requiring rapid responses. NesTok achieves the best gFID among the compared methods while maintaining competitive throughput, offering a favorable trade-off between generation quality and sampling efficiency. 5 Conclusion We presented NesTok, a variable-length 1D tokenizer trained through nested self-alignment without external distillation. By combining cross-length joint training with feature alignment, our framework mitigates tail-token collapse, increases average codebook utilization to nearly 100%. The learned representations support coarse-to-fine refinement and exhibit improved autoregressive compatibility. The AR generator trained with NesTok supports multiple token lengths, with generation quality improving consistently as more tokens are generated. The result demonstrates that an effective tokenizer training strategy can improve the variable-length image generation while retaining standard next-token prediction. References Bachmann et al. (2025) R. Bachmann, J. Allardice, D. Mizrahi, E. Fini, O. F. Kar, E. Amirloo, A. El-Nouby, A. Zamir, and A. Dehghan Flextok: resampling images into 1d token sequences of flexible length. In Forty-second International Conference on Machine Learning, Cited by: §1, §1, §1, §2, Table 1. Chang et al. (2022) H. Chang, H. Zhang, L. Jiang, C. Liu, and W. T. Freeman Maskgit: masked generative image transformer. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 11305–11315. Cited by: Table 1. Chen et al. (2025) H. Chen, Z. Wang, X. Li, X. Sun, F. Chen, J. Liu, J. Wang, B. Raj, Z. Liu, and E. Barsoum Softvq-vae: efficient 1-dimensional continuous tokenizer. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 28358–28370. Cited by: §2, Table 1. Chen et al. (2026) Y. Chen, Z. Wu, X. Qiu, and Y. Jiang CaTok: taming mean flows for one-dimensional causal image tokenization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 23161–23171. Cited by: §1, §2. Chu et al. (2026) W. Chu, B. Zhang, J. Han, Y. Li, L. Yang, Y. Yue, and Q. Guo End-to-end autoregressive image generation with 1d semantic tokenizer. arXiv preprint arXiv:2605.00503. Cited by: §2, Table 1. Dhariwal and Nichol (2021) P. Dhariwal and A. Nichol Diffusion models beat gans on image synthesis. Advances in neural information processing systems 34, p. 8780–8794. Cited by: §4.1. Duggal et al. (2025) S. Duggal, P. Isola, A. Torralba, and W. T. Freeman Adaptive length image tokenization via recurrent allocation. In First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models, Cited by: §1. Esser et al. (2021) P. Esser, R. Rombach, and B. Ommer Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 12873–12883. Cited by: §2, §2, Table 1. Fu et al. (2026) Z. Fu, L. Guo, C. Wang, B. Song, D. Liu, and B. Wen Improving flexible image tokenizers for autoregressive image generation. arXiv preprint arXiv:2601.01535. Cited by: §1, §2, Table 1, §4.2. Gao and Shou (2026) Z. Gao and M. Z. Shou D-ar: diffusion via autoregressive models. In International Conference on Learning Representations, Vol. 2026, p. 68013–68032. Cited by: §1, §1, §2, Table 1. He et al. (2025) Q. He, Y. Li, H. Ye, J. Wang, X. Liao, P. Heng, S. Ermon, J. Zou, and A. Yao Rear: rethinking visual autoregressive models via generator-tokenizer consistency regularization. arXiv preprint arXiv:2510.04450. Cited by: §1, §1. Ho et al. (2020) J. Ho, A. Jain, and P. Abbeel Denoising diffusion probabilistic models. Advances in neural information processing systems 33, p. 6840–6851. Cited by: §1. Huang et al. (2025) Y. Huang, W. Chen, W. Zheng, Y. Duan, J. Zhou, and J. Lu Spectralar: spectral autoregressive visual generation. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), p. 15842–15852. Cited by: §1, §2, Table 1. Kim et al. (2025) D. Kim, J. He, Q. Yu, C. Yang, X. Shen, S. Kwak, and L. Chen Democratizing text-to-image masked generative models with compact text-aware one-dimensional tokens. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), p. 18442–18452. Cited by: §1. Kingma and Welling (2013) D. P. Kingma and M. Welling Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. Cited by: §2. Lee et al. (2022a) D. Lee, C. Kim, S. Kim, M. Cho, and W. Han Autoregressive image generation using residual quantization. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), p. 11513–11522. Cited by: §1. Lee et al. (2022b) D. Lee, C. Kim, S. Kim, M. Cho, and W. Han Autoregressive image generation using residual quantization. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), p. 11513–11522. Cited by: §2. Li et al. (2026) C. Li, J. Zhang, S. Lin, Z. Yang, J. Liang, X. Liang, and X. Chang Efficient training of large vision models via advanced automated progressive learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 48 (8), p. 8798–8812. External Links: Document Cited by: §1. Li et al. (2024) T. Li, Y. Tian, H. Li, M. Deng, and K. He Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems 37, p. 56424–56445. Cited by: §2, Table 1, §4.1. Liu et al. (2025) Y. Liu, L. Qu, H. Zhang, X. Wang, Y. Jiang, Y. Gao, H. Ye, X. Li, S. Wang, D. K. Du, et al. Detailflow: 1d coarse-to-fine autoregressive image generation via next-detail prediction. arXiv preprint arXiv:2505.21473. Cited by: Table 1. Lu et al. (2026) J. Lu, L. Song, M. Xu, B. Ahn, Y. Wang, C. Chen, A. Dehghan, and Y. Yang AToken: a unified tokenizer for vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 28701–28711. Cited by: §1. Mentzer et al. (2024) F. Mentzer, D. Minnen, E. Agustsson, and M. Tschannen Finite scalar quantization: vq-vae made simple. In International Conference on Learning Representations, Vol. 2024, p. 51772–51783. Cited by: §2. Miwa et al. (2025) K. Miwa, K. Sasaki, H. Arai, T. Takahashi, and Y. Yamaguchi One-d-piece: image tokenizer meets quality-controllable compression. arXiv preprint arXiv:2501.10064. Cited by: §1, §2, §3.1, Table 1, §4.1, §4.2. Pang et al. (2025) Z. Pang, T. Zhang, F. Luan, Y. Man, H. Tan, K. Zhang, W. T. Freeman, and Y. Wang Randar: decoder-only autoregressive visual generation in random orders. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 45–55. Cited by: §2. Peebles and Xie (2023) W. Peebles and S. Xie Scalable diffusion models with transformers. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), p. 4172–4182. Cited by: Table 1. Qu et al. (2026) Y. Qu, K. Zhang, Y. Ding, Y. Chen, and J. Wang Semantic one-dimensional tokenizer for image reconstruction and generation. arXiv preprint arXiv:2603.16373. Cited by: §2, Table 1. Razavi et al. (2019) A. Razavi, A. Van den Oord, and O. Vinyals Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems 32. Cited by: §2. Shi et al. (2025) F. Shi, Z. Luo, Y. Ge, Y. Yang, Y. Shan, and L. Wang Scalable image tokenization with index backpropagation quantization. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), p. 16037–16046. Cited by: §2, Table 1. Su et al. (2024) J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu Roformer: enhanced transformer with rotary position embedding. Neurocomputing 568, p. 127063. Cited by: §4.1. Sun et al. (2024a) P. Sun, Y. Jiang, S. Chen, S. Zhang, B. Peng, P. Luo, and Z. Yuan Autoregressive model beats diffusion: llama for scalable image generation. arXiv preprint arXiv:2406.06525. Cited by: §2, Table 1, §4.1. Sun et al. (2024b) P. Sun, Y. Jiang, S. Chen, S. Zhang, B. Peng, P. Luo, and Z. Yuan Autoregressive model beats diffusion: llama for scalable image generation. arXiv preprint arXiv:2406.06525. Cited by: §1. Team (2024) C. Team Chameleon: mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818. Cited by: §4.1. Tian et al. (2024) K. Tian, Y. Jiang, Z. Yuan, B. Peng, and L. Wang Visual autoregressive modeling: scalable image generation via next-scale prediction. Advances in neural information processing systems 37, p. 84839–84865. Cited by: §2, Table 1. Van Den Oord et al. (2017) A. Van Den Oord O. Vinyals et al. Neural discrete representation learning. Advances in neural information processing systems 30. Cited by: §2, §2. Wang et al. (2025) X. Wang, X. Zhou, A. Fathi, T. Darrell, and C. Schmid Visual lexicon: rich image features in language space. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 19736–19747. Cited by: §1. Wen et al. (2025) X. Wen, B. Zhao, I. Elezi, J. Deng, and X. Qi “Principal components” enable a new language of images. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), p. 16641–16651. Cited by: §1, §1, §1, Table 1. Wu et al. (2025) P. Wu, K. Zhu, Y. Liu, L. Tang, J. Yang, Y. Peng, W. Zhai, Y. Cao, and Z. Zha Alitok: towards sequence modeling alignment between tokenizer and autoregressive model. arXiv e-prints, p. arXiv–2506. Cited by: §3.1, Table 1. Xiong et al. (2025) T. Xiong, J. H. Liew, Z. Huang, J. Feng, and X. Liu Gigatok: scaling visual tokenizers to 3 billion parameters for autoregressive image generation. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), p. 18770–18780. Cited by: §1, Table 1. Yao et al. (2025) J. Yao, B. Yang, and X. Wang Reconstruction vs. generation: taming optimization dilemma in latent diffusion models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 15703–15712. Cited by: §2, Table 1. Yu et al. (2023) L. Yu, Y. Cheng, K. Sohn, J. Lezama, H. Zhang, H. Chang, A. G. Hauptmann, M. Yang, Y. Hao, I. Essa, et al. Magvit: masked generative video transformer. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 10459–10469. Cited by: §2, §2. Yu et al. (2024a) L. Yu, J. Lezama, N. B. Gundavarapu, L. Versari, K. Sohn, D. Minnen, Y. Cheng, A. Gupta, X. Gu, A. G. Hauptmann, et al. Language model beats diffusion-tokenizer is key to visual generation. In International Conference on Learning Representations, Vol. 2024, p. 765–783. Cited by: §2. Yu et al. (2025) Q. Yu, J. He, X. Deng, X. Shen, and L. Chen Randomized autoregressive visual generation. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), p. 18431–18441. Cited by: §2, Table 1. Yu et al. (2024b) Q. Yu, M. Weber, X. Deng, X. Shen, D. Cremers, and L. Chen An image is worth 32 tokens for reconstruction and generation. Advances in Neural Information Processing Systems 37, p. 128940–128966. Cited by: §1, §2, Table 1. Yu et al. (2024c) Q. Yu, M. Weber, X. Deng, X. Shen, D. Cremers, and L. Chen An image is worth 32 tokens for reconstruction and generation. Advances in Neural Information Processing Systems 37, p. 128940–128966. Cited by: §3.1. Yu et al. (2024d) S. Yu, S. Kwak, H. Jang, J. Jeong, J. Huang, J. Shin, and S. Xie Representation alignment for generation: training diffusion transformers is easier than you think. arXiv preprint arXiv:2410.06940. Cited by: Table 1. Zeng et al. (2026) H. Zeng, X. Gao, G. Li, Y. Yan, J. Ruan, J. Ma, H. A. Wang, and J. Pu MacTok: robust continuous tokenization for image generation. arXiv preprint arXiv:2603.29634. Cited by: §2, Table 1. Zhang and Sennrich (2019) B. Zhang and R. Sennrich Root mean square layer normalization. Advances in neural information processing systems 32. Cited by: §4.1. Zhang et al. (2026) X. Zhang, C. Da, H. Yang, K. Gai, M. Lu, and Z. Ma ResTok: learning hierarchical residuals in 1d visual tokenizers for autoregressive image generation. arXiv preprint arXiv:2601.03955. Cited by: Table 1. Zheng and Vedaldi (2023) C. Zheng and A. Vedaldi Online clustered codebook. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), p. 22741–22750. Cited by: §3.1. Appendix A Appendix A.1 Training Pseudocode The pseudocode for our training procedure is presented below: Algorithm 1 Nested Self-Alignment Training for NesTok Input: Training dataset X; ℰψE_ψ: encoder; ϕD_φ: decoder; Q: vector quantizer; K: maximum token length; S: training iterations; ℓ : alignment layer; P: sampling distribution over 1,…,K\1,…,K\. Output: Trained tokenizer (ℰψ,,ϕ)(E_ψ,Q,D_φ). 1: Initialize the tokenizer parameters; 2: for i=1,…,Si=1,…,S do 3: Sample a batch x from X; 4: Encode and quantize x to obtain the full-length sequence q(K)z_q^(K); 5: Sample ri∼r_i and retain the prefix q(ri)z_q^(r_i); 6: for n∈K,rin∈\K,r_i\ do 7: Decode q(n)z_q^(n) to reconstruct ^(n) x^(n) and extract features using Eq. 3; 8: Average latent-token features and concatenating with the mask-token features: (n,ℓ)f^(n, ); 9: Compute ℒrec(,n)L_rec(x;n) using Eq. 1; 10: end for 11: Compute the joint reconstruction loss ℒjointL_joint using 2; 12: Compute ℒalignL_align using Eq. 4, with (K,ℓ)f^(K, ) as the stop-gradient target; 13: Compute the total objective ℒtotalL_total using Eq. 5; 14: end for A.2 More Results Visualization of Variable-length Reconstruction. As shown in Fig. 8, we compare reconstructions from NesTok and the nested dropout across token lengths ranging from 8 to 256. Nested dropout exhibits pronounced artifacts with fewer than 32 tokens. Although its reconstruction quality improves as more tokens are retained, a substantial gap in visual fidelity compared to NesTok persists even at longer sequence lengths. Figure 8: Visualization of variable-length generation on ImageNet 256×256256× 256 resolution. Images are generated using NesTok-XL with 8-256 tokens and without classifier-free guidance. Visualization of variable-length Generation We visualize images decoded from autoregressively generated token sequences ranging from 8 to 256 tokens. As more tokens are generated, coarse visual content is progressively refined into sharper and more detailed images. For the Granny Smith apple (948), additional tokens refine its shape, surface texture, and lighting. For the bald eagle (22), feather textures become more distinct, while the eyes and beak gain sharper definition. For the promontory (976), later tokens refine the coastline, terrain, and foreground vegetation. These examples illustrate how additional tokens contribute complementary visual details, supporting a coarse-to-fine generation process. Figure 9: Visualization of variable-length generation on ImageNet 256×256256× 256 resolution. Images are generated using NesTok-XL with 8-256 tokens and without classifier-free guidance. More reconstruction results. Tab. 4 compares reconstruction quality across token lengths on ImageNet at 256×256256× 256 resolution. NesTok consistently outperforms ReTok-S-B, with rFID improving from 3.49 at 32 tokens to 0.98 at 256 tokens. It matches One-D-Piece at 64 tokens and achieves better reconstruction at 128 and 256 tokens. Compared with DetailFlow, NesTok provides substantially lower rFID at 32–128 tokens, although DetailFlow achieves better reconstruction at 256 tokens. While FlexTok and Semanticist obtain lower rFID at smaller token lengths, NesTok supports reconstruction across token lengths using a ViT decoder, without iterative diffusion or flow-based decoding. Method 32 Tokens 64 Tokens 128 Tokens 256 Tokens One-D-Piece 3.23 2.10 1.42 1.08 ReTok-S-B 4.72 2.66 1.56 1.01 FlexTok d18-d28 1.45 1.37 1.20 1.08 Semanticist (DiT-XL) 1.40 1.07 0.86 0.72 DetailFlow 64.59 21.61 6.24 0.77 NesTok 3.49 2.10 1.39 0.98 Table 4: Reconstruction performance (rFID↓ ) at different token lengths on ImageNet 256×256256× 256. More generation results. Tab. 5 compares variable-length tokenizers with ViT decoders. NesTok-L achieves the lowest gFID among the compared methods at every evaluated token length, both with and without cfg. Using a single trained AR model, increasing the token length from 32 to 256 reduces its gFID from 4.82 to 2.08 without cfg and from 3.78 to 1.50 with cfg. Without cfg, One-D-Piece-L deteriorates at longer sequence lengths, while ReTok-L exhibits diminishing gains beyond 128 tokens. In contrast, NesTok-L continues to improve. These results demonstrate strong generation performance across token lengths and support the effective use of additional tokens for progressive refinement. Fig. 10 provides more visualization results on class-condition image generation with 256 tokens. Figure 10: Visualization of class-condition image generation on 256×256256× 256 resolution. Method 32 Tokens 64 Tokens 128 Tokens 256 Tokens w/o cfg cfg w/o cfg cfg w/o cfg cfg w/o cfg cfg One-D-Piece-L 8.30 5.27 8.28 3.07 10.91 2.56 13.01 2.47 ReTok-L 7.45 6.76 4.69 4.54 3.79 3.18 3.78 2.66 DetailFlow-32 67.59 60.34 28.19 19.63 12.88 6.77 6.43 2.63 NesTok-L 4.82 3.78 2.87 2.34 2.28 1.71 2.08 1.50 Table 5: Comparison of generation FID (gFID↓ ) at different token lengths on ImageNet 256×256256× 256.