Paper deep dive
Making Language the Low-Rate Code of Vision: Successive Refinement from Words to Pixels
Ruilin Gong, Hao Gong
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:An ordered visual code lets a short prefix stand in for an image, and the training loss decides what that prefix contains. Under pixel reconstruction the first bits go to the directions of highest variance, and we prove that a caption can then be independent of the entire first half of a code that is rate-optimal for pixels at every length. In the rate-distortion sense a caption is therefore not the low-rate code of its image, unless the code is built to make it one. We study that construction and what it costs. We cast captioning and text-to-image generation as successive refinement under a distortion measure that changes along the code, from caption log-loss in the early stages to pixel distortion in the last, so that each stage is compared with its own optimum. Putting words first then costs nothing among the caption stages exactly when the captions form a summarization chain, and it costs the pixels at most the caption uncertainty left in a pixel-optimal reconstruction, which is large for short prefixes and small at full length. We build RUNG, an ordered code whose prefixes are supervised by a summarization ladder of captions and whose full length reconstructs the image; generation extends the code past the prompt and captioning truncates it. With data and capacity held fixed, ordering by language instead of reconstruction raises prefix-caption information 2.9 times and GenEval from 0.76 to 0.83, at a measured cost of 2.8 dB PSNR at a quarter of the code and 0.1 dB at full length. RUNG-XL (3.4B) reaches 0.88 GenEval and 87.6 DPG-Bench in 32 network evaluations and 0.91 GenEval when errors are redrawn at the rung where they appear; its coarse and fine captions agree 94.1% of the time, and it edits images without inversion.
Tags
Links
- Source: https://arxiv.org/abs/2610.16552v1
- Canonical: https://arxiv.org/abs/2610.16552v1
Trouble viewing inline? Open PDF directly →
Full Text
79,912 characters extracted from source content.
Expand or collapse full text
Making Language the Low-Rate Code of Vision: Successive Refinement from Words to Pixels Ruilin Gong Hao Gong Henan University of Technology University of Chinese Academy of Sciences 251170400101@stu.haut.edu.cn gonghao23@mails.ucas.ac.cn Abstract An ordered visual code lets a short prefix stand in for an image, and the training loss decides what that prefix contains. Under pixel reconstruction the first bits go to the directions of highest variance, and we prove that a caption can then be independent of the entire first half of a code that is rate-optimal for pixels at every length. In the rate–distortion sense a caption is therefore not the low-rate code of its image, unless the code is built to make it one. We study that construction and what it costs. We cast captioning and text-to-image generation as successive refinement under a distortion measure that changes along the code, from caption log-loss in the early stages to pixel distortion in the last, so that each stage is compared with its own optimum. Putting words first then costs nothing among the caption stages exactly when the captions form a summarization chain, and it costs the pixels at most the caption uncertainty left in a pixel-optimal reconstruction, which is large for short prefixes and small at full length. We build Rung, an ordered code whose prefixes are supervised by a summarization ladder of captions and whose full length reconstructs the image; generation extends the code past the prompt and captioning truncates it. With data and capacity held fixed, ordering by language instead of reconstruction raises prefix–caption information 2.9×2.9× and GenEval from 0.76 to 0.83, at a measured cost of 2.8 dB PSNR at a quarter of the code and 0.1 dB at full length. Rung-XL (3.4B) reaches 0.88 GenEval and 87.6 DPG-Bench in 32 network evaluations and 0.91 GenEval when errors are redrawn at the rung where they appear; its coarse and fine captions agree 94.1% of the time, and it edits images without inversion. 1 Introduction Ordered visual codes make coarse-to-fine a property of the representation rather than of the sampler. Nested dropout (Rippel et al., 2014) and its recent descendants (Bachmann et al., 2025; Wen et al., 2025; Miwa et al., 2025) train one-dimensional tokenizers so that every prefix decodes to a valid image and a short prefix can stand in for the whole. Such a code has to decide what a reader who stops early should learn, and a reconstruction loss makes that decision by default: the first bits go to whatever reduces pixel error most, which in photographs means illumination and low-frequency colour. The default matters most where text and images meet. Unified models for generation and understanding share one token stream (Wang et al., 2024; Chen et al., 2025), diffuse over it (Yang et al., 2025b; Xin et al., 2025), pair a language model with a diffusion decoder (Zhou et al., 2025; Deng et al., 2025), or transport text directly to pixels (Liu et al., 2025; He et al., 2025). They differ in mechanism but share a premise, that text and images are peer modalities to be mapped onto each other. The premise overlooks a large difference in rate. A thirty-word caption carries about a hundred nats under a language model (Appendix B), a one-dimensional tokenizer spends a few thousand nats on a 256×256256×256 image (2.5k in ours), and a codec aiming at faithful reconstruction spends several times more. Every text-to-image model therefore has to supply the missing information, and the models above supply it without structure, as Gaussian noise, sampling temperature or unmasking order. Nothing records which parts of the output the prompt fixed and which it left open. On GenEval (Ghosh et al., 2023), the baselines in Table 2 that score 0.80 or more overall reach 0.99–1.00 on single objects, yet all but one fall below 0.65 on counting or on position. Short prompts state these attributes explicitly, so the errors fall on exactly what the prompt specifies. A natural fix is to put the caption at the front of an ordered code, so that generation extends the code and captioning truncates it. This is successive refinement (Equitz and Cover, 1991; Rimoldi, 1994) across modalities, summed up by the slogan that language is the low-rate code of vision. Under the distortion measure that ordered codes are trained with, the slogan is false. Proposition 2 constructs a source for which a code that is rate-optimal for pixels at every length carries no caption information in its first half: squared error spends early rate on high-variance directions, and captions seldom describe them. On real images, a reconstruction-ordered code keeps 0.68 nats of caption information per caption token at a quarter of its length, against 1.97 when the order is anchored in language (Fig. 2a). We start from this failure. Which information goes first is a choice of distortion measure, made stage by stage, and making language the low-rate code of vision is one such choice: the early stages are judged by how well they determine captions and the last by how well it reconstructs pixels. Successive refinement under a distortion measure that changes along the code tells us when this choice is free and what it costs otherwise. Among the caption stages, words-first loses nothing exactly when the captions form a summarization chain (Proposition 1). On the pixel side it costs an excess rate bounded by the caption uncertainty that a pixel-optimal reconstruction leaves open (Proposition 3); this cost is paid at short prefixes, where the words are, and shrinks as fidelity grows. Taken together, words-first is free exactly when each caption summarizes the next finer one and some pixel-optimal reconstruction already determines the captions. In return, the order provides three things that a map between peer modalities does not. The missing information has a place, the suffix, so the model knows what the prompt fixed and what it left free. Generation and captioning operate on one object, so caption–image agreement is a property of the code rather than of a cycle loss. And an error can be repaired at the rung that introduced it, before later rungs build on it. Figure 1 shows the resulting model, Rung, next to the order it replaces. Our contributions are the following: • A distortion schedule for cross-modal coding. We treat cross-modal generation as successive refinement under a schedule of distortion measures, which fixes the optimum each stage is compared with, and we give the condition under which a caption ladder costs nothing (Proposition 1). The condition is a rule for building training data; violating it costs 5 GenEval points in our ablation. • Both prices, bounded and measured. A reconstruction order can leave language with nothing (Proposition 2). A language order costs the pixels at most the caption uncertainty left in a pixel-optimal reconstruction, and this cost vanishes with fidelity (Proposition 3). We measure 2.8 dB at a quarter of the code and 2.9% of the rate at full length, with realism unchanged (Fig. 2). • Rung. A language-anchored ordered code with rate-matched cuts, deterministic descent, stochastic ascent and rung-level self-correction. With data, capacity and generator matched, the order alone is worth 7 GenEval points. Rung-XL reaches 0.88 GenEval in 32 network evaluations, level with the reward-tuned Lumina-DiMOO at 64, and 0.91 with rung-level correction at 1.2×1.2× compute, where best-of-N with the same verifier reaches 0.89 at 4×4×. The same code captions at a chosen granularity, edits without inversion, and transfers to radiology reports. T1T_1T2T_2T3T_3T4T_4T5T_5the wordswhat the captions leave freeascend (generation): extend the code past the promptdescend (captioning): truncate the code and read it out as wordswords-firstRungillumination, low-frequency colourcan hold no caption information in the first half (Proposition 2)pixel-firstnested dropout Figure 1: Allocation of capacity in ordered visual codes. Comparison between words-first (Rung, top) and standard pixel-first (nested dropout, bottom) ordering at identical code lengths and bitrates. In Rung, early rungs are supervised by natural language summaries whose boundaries T1,…,T5T_1,…,T_5 are fixed by empirical rate matching, leaving subsequent rungs to encode visual variation unconstrained by text. Conversely, pixel-reconstruction objectives allocate early bits to high-variance global illumination and color, starving early rungs of caption information. 2 Related work Unified and coarse-to-fine generation. Autoregressive (Wang et al., 2024; Chen et al., 2025; Xie et al., 2025a; Xie et al., 2025b), discrete-diffusion (Yang et al., 2025b; Xin et al., 2025; Shi et al., 2025; Li et al., 2025) and hybrid (Zhou et al., 2025; Deng et al., 2025; Wu et al., 2025a) unified models place text and image tokens in one sequence and let the model infer how they relate (Zhao et al., 2025a). We place them at two depths of one code, so that the relation is part of the representation. Data-to-data couplings transport text directly to images (Liu et al., 2025; He et al., 2025) and close the entropy gap by injecting noise on the text side (Albergo et al., 2023). We keep the gap and give it a location: it is the part of the code that the caption does not reach. Next-scale (Tian et al., 2024), bitwise (Han et al., 2025), cascaded (Ramesh et al., 2022) and masked (Chang et al., 2023) generators work coarse-to-fine over a spatial or spectral hierarchy on which text acts uniformly; ours is a rate hierarchy in which text occupies a specific interval. Ordered and language-aware tokenizers. Nested dropout (Rippel et al., 2014) and Matryoshka embeddings (Kusupati et al., 2022) make prefixes usable on their own, and FlexTok (Bachmann et al., 2025), Semanticist (Wen et al., 2025) and One-D-Piece (Miwa et al., 2025) bring the idea to 1D image tokenizers (Yu et al., 2024). They are our closest antecedents, and in all of them the order is set by reconstruction; Semanticist makes this explicit with a PCA-like ordering by explained variance. A second line makes codes semantic. SPAE and LQAE express images in a language model’s vocabulary (Yu et al., 2023; Liu et al., 2023); VILA-U, UniTok, TokenFlow and QLIP align codes with text-aligned features (Wu et al., 2025b; Ma et al., 2025; Qu et al., 2025; Zhao et al., 2025b); TA-TiTok conditions detokenization on text (Kim et al., 2025). These methods change what tokens mean but leave the order to reconstruction, and none of them ties a prefix to a caption of matched rate (cf. the DINOv3 row of Table 1). Rate–distortion with semantic targets. Successive refinement (Equitz and Cover, 1991; Rimoldi, 1994) asks when one nested code is optimal at every rate for a distortion measure. Remote source coding measures distortion on a hidden variable observed through the source (Dobrushin and Tsybakov, 1962; Witsenhausen, 1980) and becomes the information bottleneck under log-loss (Tishby et al., 1999; Harremoës and Tishby, 2007; Courtade and Weissman, 2014). Semantic rate–distortion constrains semantic and appearance distortion at a single rate (Liu et al., 2021), and rate–distortion–perception theory adds realism (Blau and Michaeli, 2019; Zhang et al., 2021). Closest to our setting is scalable coding for humans and machines (Choi and Bajić, 2022; Harell et al., 2025), whose base layer serves a machine task. We let the target change along one nested code and quantify what each order costs; our base layer is language, which is also the generator’s input. 3 Ordering a code by a distortion schedule 3.1 Ordered codes and distortion schedules Let X be an image (its VAE latent) and T1,…,TMT_1,…,T_M captions from coarse to fine. We take captions to be deterministic functions of the image, Tm=ψm(X)T_m= _m(X), which holds when a fixed captioner and summarizer decode greedily (Section 5.1); Section A.4 treats sampled captions. An ordered code maps X to Z=(Z1,…,ZL)Z=(Z_1,…,Z_L), and a reader who stops at k sees the prefix Z≤kZ_≤ k. Rates are informational, Rk=I(X,Z≤k)R_k=I(X;Z_≤ k). The successive-refinement coding theorem makes them operational for block codes (Rimoldi, 1994; El Gamal and Kim, 2011), and Section 4.1 measures them for learned codes. An order is determined by what each stage is judged on. A distortion schedule makes this explicit by attaching a distortion measure to every stage and comparing each stage with its own optimum: • a caption stage m is judged by log-loss on TmT_m. With deterministic captions a prefix is lossless for TmT_m when it determines TmT_m, and the least rate at which any code does so is H(Tm)H(T_m); • a pixel stage is judged by a distortion d(X,X^)d(X, X) and compared with the rate–distortion function RX(D)R_X(D), the least I(X,X^)I(X; X) subject to d(X,X^)≤DEd(X, X)≤ D. The theory uses squared error; Section 5.2 also reports perceptual metrics. A nested code is successively refinable for a schedule if every stage attains its own optimum simultaneously, and the excess rate of a stage is its rate beyond that optimum. Classical successive refinement uses one measure throughout. With a pixel measure this is the order of existing ordered tokenizers, which we call pixel-first. Words-first has caption stages 1,…,M1,…,M followed by pixel stages. Neither order is intrinsic to the image, and Section 3.3 shows that each has a cost. 3.2 When the caption stages cost nothing Proposition 1 (Caption ladder). Let a nested code determine TmT_m from its stage-m prefix for every m. Its stage-m rate is at least H(T1,…,Tm)H(T_1,…,T_m), an excess of Δm=H(T1,…,Tm−1∣Tm) _m=H(T_1,…,T_m-1 T_m) over the stage optimum H(Tm)H(T_m), and one code attains this for all m at once. The caption stages are therefore successively refinable if and only if every coarser caption is a function of the next finer one, Tm=ϕm(Tm+1)T_m= _m(T_m+1). When it is not, stage m+1m+1 pays at least δm=H(Tm∣Tm+1) _m=H(T_m T_m+1). The proof (Appendix A) is the chain rule: a finer prefix contains every coarser one, so it must determine every coarser caption. The proposition is a rule about data. Caption levels should be produced by summarizing the next finer level rather than written independently at each length; otherwise the finer stage pays for details that only the coarser caption mentions. Table 5 tests the rule in our ablation setting: independently written levels raise the estimated gap δ δ from 0.12 to 0.83 nats per token and cost 5 GenEval points and 31 points of nesting consistency. The condition is necessary for zero excess rate but does not say what the short levels should contain. Truncating one long caption satisfies it exactly and still costs 3 GenEval points, because the opening words of a long description are not a summary of it. 3.3 The cost of each order Proposition 2 (Pixel-first can starve language). For every L there are a source X, a caption T=ψ(X)T=ψ(X) with H(T)>0H(T)>0, and an ordered code that is successively refinable for X under squared error at each of its L stages, such that I(Z≤k,T)=0I(Z_≤ k;T)=0 for every k≤L/2k≤ L/2. Let X=(A,B)X=(A,B) with A,BA,B independent Gaussians, Var(A)≫Var(B)Var(A) (B), and T=[B>0]T=1[B>0]. Reverse water-filling spends every nat below 12log(VarA/VarB) 12 (VarA/VarB) on A, so the first half of an optimal code is independent of T. The construction needs nothing unusual: in photographs the high-variance directions are illumination and low-frequency colour, which captions seldom mention. Words-first moves the cost to the pixels. Let T=TMT=T_M; under the condition of Proposition 1 it determines every coarser level. The price of putting words first at pixel distortion D is the extra rate needed to determine T and reach D, compared with a code that only reaches D: π(D)=mind(X,X^)≤DI(X,T,X^)−RX(D).π(D)\;=\; _Ed(X, X)≤ DI(X;T, X)\;-\;R_X(D). (1) Proposition 3 (Words-first charges pixels, boundedly). For every D, and any reconstruction X^D∗ X^*_D that achieves RX(D)R_X(D), [H(T)−RX(D)]+≤π(D)≤H(T∣X^D∗), [H(T)-R_X(D) ]^+\;≤\;π(D)\;≤\;H (T X^*_D ), and π(D)=0π(D)=0 if and only if some X^D∗ X^*_D determines T. If the pixel stages at D1>⋯>DJD_1>…>D_J are successively refinable, one words-first code attains the upper bound at every stage simultaneously. Corollary 1 (The price fades with fidelity). Suppose a reader ψ~ ψ and a margin ρ>0ρ>0 satisfy ψ~(x^)=ψ(x) ψ( x)=ψ(x) whenever d(x,x^)≤ρd(x, x)≤ρ, for all x outside a set of probability ϵε. Then for D≤ρ(12−ϵ)D≤ρ( 12-ε), π(D)≤hb(ϵ+D/ρ)+(ϵ+D/ρ)log||.π(D)\;≤\;h_b (ε+D/ρ )+ (ε+D/ρ ) |T|. If ϵε can be made arbitrarily small by shrinking ρ, that is, if captions are almost surely locally constant, then π(D)→0π(D)→ 0 as D→0D→ 0. Proposition 3 locates the price. For short prefixes it is at least H(T)−RX(D)H(T)-R_X(D), because any code that determines T spends H(T)H(T) nats on it. For long prefixes it is at most what a captioner cannot recover from the best reconstruction, which Corollary 1 drives to zero once reconstructions can be captioned correctly. The corollary is qualitative for long captions, since log|| |T| is then large, so Section 5.2 measures the price directly. Both bounds can be estimated, which gives three predictions that Section 5.2 tests. (P1) A reconstruction-ordered prefix keeps little caption information. (P2) A words-first prefix is less faithful to the pixels at short lengths, and the gap closes with length. (P3) The measured price lies between the bounds and is small at full length. What the order buys. With unlimited capacity every order induces the same p(x∣T)p(x T); the order matters for how finite models with short samplers fail. In a words-first code at its stage optimum, the prefix up to the prompt’s cut kTk_T is a function of the prompt, H(Z≤kT∣T)=0H(Z_≤ k_T T)=0. An ideal sampler therefore makes no random choice about anything the prompt states, and a partial sample can be read back as a caption and checked against the prompt before later rungs build on it; a learned sampler only approximates this map, which is why the check of Section 4.3 is useful. In a pixel-first code the prompt’s content is spread over rungs that are sampled jointly with content the prompt never mentions. Section 5.3 tests these claims. 4 Rung: a ladder from words to pixels 4.1 Building the ladder Let h=ℰvae(x)h=E_vae(x). Rung encodes h into an ordered code z=(z1,…,zL)z=(z_1,…,z_L) grouped into L rungs and reads it with two decoders: an image decoder that reconstructs h from any prefix, and a language decoder pψp_ψ that writes a caption from any prefix. Generation climbs the rungs and captioning descends them. Training uses, for every image, a caption ladder T1,…,TMT_1,…,T_M that satisfies Proposition 1 (Section 5.1). Two terms train the code. The first makes every prefix decodable. The prefix length k is drawn from a nested-dropout distribution q, and the image decoder is trained with flow matching (Lipman et al., 2023) on every truncation, ℒrec=k∼qt,ϵ‖vθ(ht,t,z≤k)−(h−ϵ)‖2,ht=th+(1−t)ϵ,L_rec=E_k q\,E_t,ε \|v_θ(h_t,t,z_≤ k)-(h-ε) \|^2, h_t=t\,h+(1-t)\,ε, (2) so that any prefix decodes to a plausible image. The second term distinguishes Rung from a reconstruction-ordered tokenizer: each caption level must be recoverable from its matching prefix, ℒlang=∑m=1M−logpψ(Tm∣z≤km).L_lang= _m=1^M- p_ψ (T_m z_≤ k_m ). (3) Gradients of ℒlangL_lang reach the encoder, so language, not variance, decides the competition for early rate. We minimize ℒrec+λℒlangL_rec+ _lang. Setting λ=0λ=0 gives nested dropout, larger λ moves the code toward words-first, and we use λ=1λ=1 throughout (Fig. 3a traces the trade-off). A third term, introduced in Section 4.2, trains the ascent model. Rate-matched cuts. Rungs are quantized with FSQ (Mentzer et al., 2024), so the rate of a prefix can be measured. We set R^k=∑uH^(zu) R_k= _u H(z_u), the sum of the empirical entropies of the code units u in the first k rungs; by subadditivity it upper-bounds the prefix entropy. The cut kmk_m is the first code unit at which R R reaches H^(Tm) H(T_m), the caption’s entropy under a small language model. This makes the stage optimum of Proposition 1 operational: level m is read where the code has spent the rate that determining TmT_m requires. The rates depend on the code being trained, so the cuts are recomputed every 5k steps during language anchoring (Appendix C). In the trained code the five levels end at 1.2, 2.9, 6.0, 11 and 21% of R^L≈2.5 R_L≈ 2.5k nats, inside rungs 1, 1, 2, 4 and 7 of 32. The captions occupy the first fifth of the code, and the remaining rungs hold what the captions leave free. 4.2 Descent and ascent The two directions are not inverses of each other. Descent discards information while ascent has to supply it, so we give them separate operators instead of one invertible map (Table 9 tests the tied alternative). Descent is deterministic: x↦z↦z≤k↦x z z_≤ k greedy decoding of pψ(⋅∣z≤k)p_ψ(· z_≤ k), one encoder pass and one short decode. Because every level reads a prefix of the same code, the 8-word caption agrees with the 45-word caption for a structural reason, whereas a prompted captioner can only be asked to agree (Table 4). Ascent is stochastic. Given a prompt T, we locate its cut kT=mink:R^k≥H^(T)k_T= \k: R_k≥ H(T)\ and draw the code in eight blocks. The first four end at the ladder’s cuts (rungs 1, 2, 4 and 7) and the last four cover rungs 8–11, 12–16, 17–23 and 24–32. Each block is drawn conditioned on the prompt and on the blocks before it, z≤k+Δb∼pθ(⋅∣z≤k,T,b),z_≤ k+ _b p_θ (· z_≤ k,\,T,\,b ), (4) by a few-step conditional flow (Frans et al., 2025; Geng et al., 2025) in the rung’s pre-quantization space. At the stage optimum the blocks up to kTk_T are a function of T (Section 3.3), so the flow for these blocks learns a nearly deterministic map, and the later blocks carry the randomness. Sampling steps are allocated in proportion to each block’s measured rate ΔR^b R_b, which is worth 3 GenEval points over a uniform allocation at the same budget (Table 9). A consistency term couples the two directions without requiring invertibility: captions read back from generated prefixes must match the ladder of the training image, ℒcons=∑m−logpψ(Tm∣z^≤km)L_cons= _m- p_ψ(T_m z_≤ k_m). 4.3 Correcting errors where they appear Each ladder block can be checked before the next one is drawn. We parse the prompt once into attribute tuples (object, count, colour, relation). After each of the first four blocks, an attribute head on the language decoder reads, directly from the prefix, the tuples stated by the levels that end in that block. The block is accepted if at least a fraction τ=0.8τ=0.8 of these tuples agree with the prompt, where a tuple the prompt does not mention counts as agreeing; otherwise it is redrawn with fresh noise, up to B=2B=2 times. The purpose of the check is to localize the error. If the first block encodes three apples where the prompt asks for two, the wrong count is caught in that block, because the 8-word level states counts (Appendix B), and only that block is redrawn; best-of-N would discard a whole sample for one wrong attribute. Blocks after the last cut are not checked, since no caption level describes them. The verifier is the model’s own descent path, with no separate reward model, and it never sees the detectors that GenEval uses. Without training. Editing is a prefix swap: encode the image, replace the ladder prefix with one drawn from the edited caption, keep the suffix and decode. The prefix carries what the caption states and the suffix what it leaves free, so neither inversion nor attention manipulation is needed; Appendix D lists the cases in which the kept suffix does not fit the new prefix. Variation at a chosen granularity is truncation: cut at rung k and resample the rest, and the position of the cut decides what is preserved. 5 Experiments 5.1 Setup Benchmarks. For generation we use GenEval (Ghosh et al., 2023), DPG-Bench (Hu et al., 2024), T2I-CompBench++ (Huang et al., 2025), TIIF-Bench (Wei et al., 2025), WISE (Niu et al., 2025), and FID and CMMD (Jayasumana et al., 2024) on MJHQ-30K. For understanding we use CapArena-Auto (Cheng et al., 2025), DCScore on DeCapBench (Ye et al., 2025) and CHAIR (Rohrbach et al., 2018). We also report GranBench, 600 prompts whose attributes are annotated with the granularity at which they should appear, and nesting consistency (NCC), the fraction of attribute tuples in a model’s coarse caption that its own finer caption of the same image entails. Baselines. We compare with generation-only models (Podell et al., 2024; Esser et al., 2024; Black Forest Labs, 2024; Qin et al., 2025), unified autoregressive models (Wang et al., 2024; Chen et al., 2025; Xie et al., 2025b; Deng et al., 2025; Wu et al., 2025a) and unified discrete-diffusion models (Yang et al., 2025b; Shi et al., 2025; Xin et al., 2025); with VLMs (Bai et al., 2025; Zhu et al., 2025) as established references for captioning; and with two ordered codes, FlexTok (Bachmann et al., 2025) and Semanticist (Wen et al., 2025), each trained with the same text-to-image head (the Rung-L ascent model), data and capacity as Rung-L. Table 1 separates the effect of the order from the rest of the tokenizer design. All baselines are evaluated using official model releases and standardized inference protocols without prompt rewriting. Implementation. The ladder encoder is a ViT-B over SD3 VAE latents that produces L=32L=32 rungs of 8 code units, quantized with FSQ (levels [8,8,8,5,5,5][8,8,8,5,5,5], at most 11.1 nats per unit). The image decoder is a 24-layer DiT (Peebles and Xie, 2023) trained with flow matching, and the language decoder is Qwen3-1.7B (Yang et al., 2025a) with LoRA adapters. The ascent model is an MMDiT (Esser et al., 2024) at two sizes, Rung-L (1.3B) and Rung-XL (3.4B); the encoder and both decoders are shared, so the two sizes caption identically. Training has three stages, namely ladder pretraining, language anchoring on LadderCap, and ascent at 2562256^2 and then 5122512^2 (Appendix C); Rung-XL takes about 9 days on 64 A100-80GB GPUs. Sampling uses 32 network evaluations, 24 for ascent and 8 for the image decoder, with guidance 1.5. LadderCap. We build M=5M=5 caption levels (≈8, 20, 45, 90 and 180 words) for 45M images drawn from Recap-DataComp (Li et al., 2024), BLIP3-KALE (Awadalla et al., 2024), PixelProse (Singla et al., 2024), DOCCI (Onoe et al., 2024), ImageInWords (Garg et al., 2024) and DCI (Urbanek et al., 2024). Following Proposition 1, the longest level is generated from the image (Qwen2.5-VL-7B, greedy) and every shorter level is a summary of the next longer level (Qwen3-8B, greedy), never a fresh caption. The summarizer is instructed to keep a count or a relation in the 8-word level whenever the image contains more than one object. Appendix B reports per-level statistics and an entailment audit on 5K images. 5.2 Measuring the order and its price Figure 2: Empirical validation of rate–distortion properties. Evaluated on RUNG-L (R^L≈2.5k R_L≈ 2.5k nats). (a) Mutual information between code prefix and captions demonstrates that pixel-first codes discard semantic content at low rates. (b) Reconstruction fidelity shows words-first sacrifices 2.8 dB PSNR within the language ladder, catching up to within 0.1 dB at full code length. (c) Measured excess rate π^(D) π(D) adheres strictly to the theoretical bounds of Proposition 3, tapering to 73 nats (2.9% of the code) at full fidelity. Table 1: Effect of visual code ordering signals. Evaluated on RUNG-L (2562256^2) under identical capacity and data. I^25 I_25 denotes the prefix–caption mutual information at one-quarter code length (k/L=1/4k/L=1/4). Quarter-code and full-code reconstructions evaluate image fidelity (PSNR, LPIPS) and distribution realism (rFID), alongside zero-shot text-to-image generation FID on MJHQ-30K. Caption Retention & Use Quarter Code (k/L=1/4k/L=1/4) Full Code (k/L=1k/L=1) Ordering Signal I^25↑ I_25 GenEval↑ NCC (%) PSNR↑ LPIPS↓ rFID↓ PSNR↑ rFID↓ Gen. FID↓ Unordered (Vanilla) 0.31 0.71 22.1 – – – 26.2 0.70 6.8 Pixel-First (Nested Dropout) 0.68 0.76 41.5 19.3 0.36 2.3 26.1 0.71 6.3 DINOv3 Feature Alignment 0.84 0.78 48.3 18.8 0.38 2.2 26.0 0.72 6.3 Words-First (RUNG-L) 1.97 0.83 94.1 16.5 0.45 2.1 26.0 0.74 6.2 Table 1 and Figs. 2 and 3a retrain the tokenizer and the Rung-L ascent model at full scale for each ordering signal, with everything else fixed. I I is a Barber–Agakov lower bound (Barber and Agakov, 2003) on the information between a prefix and the finest caption T=T5T=T_5, reported in nats per caption token (H^(T5)/|T5|=2.5 H(T_5)/|T_5|=2.5); an InfoNCE bound (Poole et al., 2019) ranks the codes identically (Appendix F). The measured price π^(D) π(D) is the rate at which the words-first code has both passed its last cut and reached PSNR D, minus the rate at which the pixel-first code reaches D. The upper bound is the cross-entropy of a captioning probe on pixel-first reconstructions (Appendix F). P1: reconstruction order starves language. At a quarter of the code the reconstruction-ordered prefix keeps 0.68 nats of caption information per token and the language-ordered prefix 1.97 (Fig. 2a). Aligning the encoder to DINOv3 features (Siméoni et al., 2025) in the style of REPA (Yu et al., 2025) raises the first number to 0.84, so visual self-supervision moves the order toward language, but only a small part of the way. At full length every code determines the image, and the four estimates meet at the probe’s ceiling of 2.2 nats per token. This indicates that the probe measures information rather than how easily a particular code can be read. P2: words-first costs pixel fidelity where the words are. The words-first prefix is 2.8 dB less faithful at a quarter of the code (Table 1), and the gap is largest inside the ladder, while the code is still encoding captions (Fig. 2b). The cost falls on fidelity to the particular image, and realism is unaffected: because the decoder samples, quarter-code decodes remain realistic (rFID 2.1 against 2.3; cf. Blau and Michaeli, 2019). The gap closes with length, to 0.5 dB at half the code and 0.1 dB at full length. P3: the price stays inside the bounds and is small at full length. Measured in rate (Fig. 2c), the price equals the lower bound [H^(T)−R^]+[ H(T)- R]^+ while the words-first code is still inside its ladder, which holds by construction (Appendix F), and it stays below the estimated upper bound H^(T∣X^∗) H(T X^*) everywhere (207 against 394 nats at a quarter). It ends at 73 nats, 2.9% of the code. Full-length reconstruction (rFID 0.74 against 0.71) and text-to-image FID (6.2 against 6.3) differ by less than the variation across seeds (0.03 and 0.1). Neither application uses the pixels of a short prefix: captioning reads the prefix for its words, and generation decodes the full code. The order is a dial. Figure 3a sweeps λ. Each step toward language buys caption information and GenEval at the expense of quarter-code PSNR. GenEval saturates at λ=1λ=1, and at λ=3λ=3 the code gains little more caption information and starts to lose fidelity at full length (PSNR 25.7). DINOv3 alignment lies inside the trade-off curve, so it buys caption information at a worse rate of exchange than supervision with language. With everything else matched, the order accounts for 7 of the 16 GenEval points between Rung-L and FlexTok with the same head (0.76 to 0.83 in Table 1); the other 9 come from the rest of the design, which Table 9 breaks down in the ablation setting. 5.3 What the order buys Table 2: Fine-grained visual compositionality on GenEval. Evaluated under the standard GenEval suite across object count, color binding, and spatial positioning without prompt expansion. #Par denotes generator parameters during sampling. Lumina-DiMOO incorporates template-level GRPO alignment during post-training. Paradigm Model #Par Single Two Count Colors Position Attribute Overall Generation SDXL 2.6B 0.98 0.74 0.39 0.85 0.15 0.23 0.55 SD3-Medium 2.0B 0.99 0.94 0.72 0.89 0.33 0.60 0.74 Lumina-Image 2.0 2.6B 0.94 0.87 0.67 0.89 0.62 0.62 0.73 FLUX.1-dev 12B 0.98 0.81 0.74 0.79 0.22 0.45 0.66 Unified AR Emu3-Gen 8.0B 0.98 0.71 0.34 0.81 0.17 0.21 0.54 Show-o2 7.0B 1.00 0.87 0.58 0.92 0.52 0.62 0.76 Janus-Pro 7.0B 0.99 0.89 0.59 0.90 0.79 0.66 0.80 OmniGen2 7.0B 1.00 0.95 0.64 0.88 0.55 0.76 0.80 BAGEL 14B 0.99 0.94 0.81 0.88 0.64 0.63 0.82 Discrete Diff. Muddit 1.0B 0.98 0.72 0.54 0.82 0.19 0.41 0.61 MMaDA 8.0B 0.99 0.76 0.61 0.84 0.20 0.37 0.63 Lumina-DiMOO 8.0B 1.00 0.94 0.85 0.89 0.85 0.76 0.88 Semanticist + T2I Head 1.3B 0.98 0.80 0.49 0.85 0.33 0.45 0.65 FlexTok + T2I Head 1.3B 0.98 0.82 0.51 0.86 0.36 0.47 0.67 RUNG-L 1.3B 0.99 0.89 0.78 0.87 0.75 0.72 0.83 RUNG-XL 3.4B 0.99 0.93 0.85 0.90 0.82 0.79 0.88 RUNG-XL + Corr. 3.4B 1.00 0.94 0.89 0.92 0.87 0.84 0.91 Generation. Tables 2 and 3 give the main comparison. Rung-XL ties Lumina-DiMOO at 0.88 GenEval with half its sampling-time parameters (4.0B including the image decoder, against 8B; Appendix C) and half its network evaluations, and without reward tuning. Rung-level correction raises Rung-XL to 0.91. Per category, Rung does not lead everywhere. On two-object prompts it trails OmniGen2, SD3-Medium, BAGEL and Lumina-DiMOO, and on colour it trails Show-o2, as one would expect from a smaller generator on properties that improve with capacity. Against the models without reward tuning, Rung-XL gains in counting (+0.04+0.04 over BAGEL), position (+0.03+0.03 over Janus-Pro) and attribute binding (+0.03+0.03 over OmniGen2). A short prompt states these three properties explicitly, and the words-first order places them in the prompt’s prefix (Section 3.3). Lumina-DiMOO is ahead of Rung-XL on position (0.85 against 0.82) and level on counting; with correction Rung-XL exceeds every baseline on all three (0.89, 0.87 and 0.84). T2I-CompBench++ shows the same pattern: the gain over Lumina-DiMOO is concentrated in numeracy (+0.052+0.052) and 2D spatial relations (+0.048+0.048) and is close to zero for non-spatial relations, which short captions rarely pin down. Extra compute is better spent on rungs than on samples: correction reaches 0.91 at 1.2×1.2× the network evaluations and best-of-N with the same verifier 0.89 at 4×4× (Fig. 3b), since a redraw keeps the blocks that already passed. Table 3: Instruction following, compositional synthesis, and efficiency. Benchmarked across GenEval, DPG-Bench, T2I-CompBench++ (T2I-C++), TIIF-Bench, WISE, and MJHQ-30K FID. Total NFE aggregates all sampling network evaluations (generator, latent flow decoder, and verifier passes); CFG forward passes are batched. Latency is measured at 5122512^2 resolution with batch size 1 on an NVIDIA A100 GPU. Model GenEval DPG T2I-C++ TIIF WISE FID↓ Total NFE↓ Latency (s)↓ SDXL 0.55 74.7 0.408 51.2 0.43 8.8 50 3.1 FLUX.1-dev 0.66 83.8 0.502 72.6 0.50 6.1 50 4.8 Lumina-Image 2.0 0.73 87.2 0.487 69.1 0.43 6.8 40 3.4 Janus-Pro 0.80 84.2 0.471 66.3 0.35 8.0 576 9.1 BAGEL 0.82 85.1 0.524 71.4 0.52 6.4 50 7.2 Lumina-DiMOO 0.88 86.0 0.551 74.8 0.60 6.9 64 6.3 FlexTok + T2I Head 0.67 80.3 0.421 58.7 0.34 7.9 32 1.9 RUNG-L 0.83 85.3 0.538 72.9 0.54 6.2 32 1.4 RUNG-XL 0.88 87.6 0.581 76.2 0.61 5.8 32 2.1 RUNG-XL + Corr. 0.91 88.4 0.607 78.5 0.63 5.6 38 2.5 Figure 3: Left: the order is a dial. Sweeping λ trades quarter-code PSNR for caption information (GenEval in parentheses); λ=0λ=0 is nested dropout, and DINOv3 alignment lies inside the trade-off curve. Right: test-time compute. Rung-level correction (redraw budgets B=1B=1 to 44; Appendix C) against best-of-N with the same verifier, in network evaluations relative to one uncorrected sample. Table 4: Multimodal captioning and structural nesting consistency (NCC). Evaluated on CapArena-Auto (CapA., [−100,100][-100,100]), DeCapBench DCScore (DCS, [0,100][0,100]), CHAIRs ([0,100][0,100], lower is better), and GranBench. NCC measures the fraction of attribute tuples in a model’s coarse caption entailed by its fine caption on identical images. Model CapA. DCS CHAIRs↓_s\! NCC (%) GranBench Qwen2.5-VL-7B (Reference) −12.3-12.3 42.8 11.2 61.4 44.7 InternVL3-8B (Reference) −18.7-18.7 39.5 13.5 57.9 41.2 BAGEL −46.5-46.5 31.2 18.4 48.2 33.6 Lumina-DiMOO −53.2-53.2 29.8 21.0 45.1 31.9 Rung Descent -24.9 38.6 12.8 94.1 71.3 Understanding. With a 1.7B language decoder, descent trails the 7B VLMs on detail (CapArena-Auto −24.9-24.9 against −12.3-12.3 and −18.7-18.7), lies between them on hallucination, and is ahead of both unified models on every column (Table 4). On nesting it differs from all prompted models, 94.1% against 45–61%, because its two captions read two prefixes of one code instead of coming from two independent decodes. Most of the remaining 5.9% comes from the language decoder: attribute tuples read directly from the prefixes nest 99.2% of the time. Editing and variation. On PIE-Bench (Ju et al., 2024), a prefix swap reaches CLIP similarity 26.4 on the whole image and 23.1 on the edited region at a structure distance of 19.6×10−319.6× 10^-3, against 25.8/22.7 at 27.4 for BAGEL’s editing mode and 25.2/22.2 at 40.6 for RF-Inversion (Rout et al., 2025). The swap costs one encode and one decode; inversion costs 2–3× more. A cut at rung 8, one rung past the finest caption (≈625≈625 nats), keeps 96% of the attribute tuples of the 180-word description; a cut at rung 20 (≈1.56≈1.56k nats) also keeps layout and texture and mainly changes lighting (Appendix D). Transfer. Radiology reports come with a two-level ladder, since the impression approximately summarizes the findings. On MIMIC-CXR (Johnson et al., 2019), Rung reaches 0.331 RadGraph-F1 (Jain et al., 2021) and 0.495 CheXbert-F1 (Smit et al., 2020) with 0.91 two-level consistency, against 0.312/0.478 for a matched baseline that writes the two sections independently; the result concerns structure only, not clinical accuracy. Table 5: Validation of Proposition 1 on caption hierarchy construction. Evaluated on Rung-L (2562256^2, 12M data). δ^=H^(Tm∣Tm+1) δ= H(T_m T_m+1) quantifies inter-stage information leakage in nats per token. Violating the Markov summarization condition imposes severe rate penalties and degrades semantic nesting consistency. Construction δ δ GenEval NCC (%) Independent per length 0.83 0.74 61.8 Truncated long caption 0.04 0.76 90.4 Successive summary 0.12 0.79 92.7 Ablations use Rung-L at 2562256^2 on a 12M-image subset (default GenEval 0.79, against 0.83 at full scale). The data rule matters as much as the loss (Table 5). Independently written levels raise δ δ from 0.12 to 0.83 and cost 5 GenEval points, as Proposition 1 predicts. Truncating one long caption satisfies the chain condition but loses 3 points, because its short levels open a long description without summarizing it (Section 3.2). Design (Appendix E). Performance saturates at L=32L=32. Per-rung few-step flows beat autoregressive and single-diffusion ascent at a third and a ninth of their latency. Frozen text embeddings cost 9 to 12 points, and tying the two directions into one time-reversible operator costs 4. 6 Limitations and conclusion Limitations. The price is real: a words-first prefix is a worse picture of the particular image than a pixel-first prefix of the same length, so applications that need faithful pixels from few bits should use a small λ or none. The theory assumes deterministic captions and informational rates, Corollary 1 is qualitative for long captions, and our checks rely on estimators that are themselves bounds. LadderCap inherits biases shared by the language models that build it. Published baseline protocols differ in resolution and prompt handling. We train at 5122512^2 and ran no human preference study, so we claim nothing about image quality against generation-only systems at 102421024^2. Conclusion. An ordered code always decides what its first bits describe. With a reconstruction loss the decision is made by variance, and Proposition 2 shows that it can leave language with nothing. We made the opposite choice explicit, with a distortion schedule that fixes each stage’s optimum, a condition under which the caption stages cost nothing, and a bounded cost in pixels paid at short prefixes. In exchange, generation extends one code and captioning truncates it. Choosing the schedule per image, deciding where the words end and the pixels begin, is left to future work. Reproducibility statement The ladder objectives are in Section 4.1, ascent and self-correction in Sections 4.2 and 4.3, and complete proofs of Propositions 1, 2 and 3 and Corollary 1, with all assumptions, in Appendix A. The estimators for prefix information, rate and the price bounds are specified in Appendix F. Architecture, staging and hardware are in Section 5.1 and all hyperparameters in Appendix C. LadderCap uses public datasets and public models; prompts, the entailment audit and per-level statistics are in Appendix B. Code, the LadderCap pipeline, GranBench, the NCC evaluator and the scripts that produce Fig. 2 will be made available. Ethics statement Rung inherits the risks of its class: representational bias absorbed from web-scale caption data and potential misuse for deceptive imagery. Control over granularity adds no new capability here, although the prefix-swap edit makes targeted semantic edits cheaper, which matters for provenance tooling. The MIMIC-CXR experiment uses a credentialed dataset under its data use agreement and reports a structural transfer result; nothing in it is validated for clinical use. Author contributions R. Gong and H. Gong conceived the core theoretical formulation, designed the cross-modal distortion schedule, developed the Rung architecture, and executed the experimental pipeline. Both authors contributed equally to mathematical proofs, empirical evaluations, and manuscript composition. References Albergo et al. (2023) M. S. Albergo, N. M. Boffi, and E. Vanden-Eijnden Stochastic interpolants: a unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797. Cited by: §2. Awadalla et al. (2024) A. Awadalla, L. Xue, M. Shu, A. Yan, J. Lee, T. Nguyen, M. Zhu, Y. Yang, M. Bansal, L. Schmidt, et al. BLIP3-KALE: knowledge augmented large-scale dense captions. arXiv preprint arXiv:2411.07461. Cited by: §5.1. Bachmann et al. (2025) R. Bachmann, J. Allardice, D. Mizrahi, E. Fini, O. F. Kar, E. Amirloo, A. El-Nouby, A. Zamir, and A. Dehghan FlexTok: resampling images into 1D token sequences of flexible length. In ICML, Cited by: §1, §2, §5.1. Bai et al. (2025) S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923. Cited by: §5.1. Barber and Agakov (2003) D. Barber and F. Agakov The IM algorithm: a variational approach to information maximization. In NeurIPS, Cited by: Appendix F, §5.2. Black Forest Labs (2024) Black Forest Labs FLUX.1. Note: https://github.com/black-forest-labs/flux Cited by: §5.1. Blau and Michaeli (2019) Y. Blau and T. Michaeli Rethinking lossy compression: the rate-distortion-perception tradeoff. In ICML, Cited by: §2, §5.2. Chang et al. (2023) H. Chang, H. Zhang, J. Barber, A. Maschinot, J. Lezama, L. Jiang, M. Yang, K. Murphy, W. T. Freeman, M. Rubinstein, et al. Muse: text-to-image generation via masked generative transformers. In ICML, Cited by: §2. Chen et al. (2025) X. Chen, Z. Wu, X. Liu, Z. Pan, W. Liu, Z. Xie, X. Yu, and C. Ruan Janus-Pro: unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811. Cited by: §1, §2, §5.1. Cheng et al. (2025) K. Cheng, W. Song, J. Fan, Z. Ma, Q. Sun, F. Xu, C. Yan, N. Chen, J. Zhang, and J. Chen CapArena: benchmarking and analyzing detailed image captioning in the LLM era. In Findings of ACL, Cited by: §5.1. Choi and Bajić (2022) H. Choi and I. V. Bajić Scalable image coding for humans and machines. IEEE Transactions on Image Processing 31. Cited by: §2. Courtade and Weissman (2014) T. A. Courtade and T. Weissman Multiterminal source coding under logarithmic loss. IEEE Transactions on Information Theory 60 (1), p. 740–761. Cited by: §2. Cover and Thomas (2006) T. M. Cover and J. A. Thomas Elements of information theory. 2nd edition, Wiley. Cited by: §A.2. Deng et al. (2025) C. Deng, D. Zhu, K. Li, C. Gou, F. Li, Z. Wang, S. Zhong, W. Yu, X. Nie, Z. Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683. Cited by: §1, §2, §5.1. Dobrushin and Tsybakov (1962) R. L. Dobrushin and B. S. Tsybakov Information transmission with additional noise. IRE Transactions on Information Theory 8 (5), p. 293–304. Cited by: §2. El Gamal and Kim (2011) A. El Gamal and Y. Kim Network information theory. Cambridge University Press. Cited by: Appendix A, §3.1. Equitz and Cover (1991) W. H. R. Equitz and T. M. Cover Successive refinement of information. IEEE Transactions on Information Theory 37 (2), p. 269–275. Cited by: §A.2, §1, §2. Esser et al. (2024) P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, Cited by: §5.1, §5.1. Frans et al. (2025) K. Frans, D. Hafner, S. Levine, and P. Abbeel One step diffusion via shortcut models. In ICLR, Cited by: §4.2. Garg et al. (2024) R. Garg, A. Burns, B. K. Ayan, Y. Bitton, C. Montgomery, Y. Onoe, A. Bunner, R. Krishna, J. Baldridge, and R. Soricut ImageInWords: unlocking hyper-detailed image descriptions. In EMNLP, Cited by: §5.1. Geng et al. (2025) Z. Geng, M. Deng, X. Bai, J. Z. Kolter, and K. He Mean flows for one-step generative modeling. In NeurIPS, Cited by: §4.2. Ghosh et al. (2023) D. Ghosh, H. Hajishirzi, and L. Schmidt GenEval: an object-focused framework for evaluating text-to-image alignment. In NeurIPS Datasets and Benchmarks, Cited by: §1, §5.1. Han et al. (2025) J. Han, J. Liu, Y. Jiang, B. Yan, Y. Zhang, Z. Yuan, B. Peng, and X. Liu Infinity: scaling bitwise autoregressive modeling for high-resolution image synthesis. In CVPR, Cited by: §2. Harell et al. (2025) A. Harell, Y. Foroutan, N. Ahuja, P. Datta, B. Kanzler, V. S. Somayazulu, O. Tickoo, A. de Andrade, and I. V. Bajić Rate-distortion theory in coding for machines and its applications. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §2. Harremoës and Tishby (2007) P. Harremoës and N. Tishby The information bottleneck revisited or how to choose a good distortion measure. In IEEE International Symposium on Information Theory (ISIT), Cited by: §2. He et al. (2025) J. He, Q. Yu, Q. Shen, A. Yuille, and L. Chen FlowTok: flowing seamlessly across text and image tokens. arXiv preprint arXiv:2503.10772. Cited by: §1, §2. Hu et al. (2024) X. Hu, R. Wang, Y. Fang, B. Fu, P. Cheng, and G. Yu ELLA: equip diffusion models with LLM for enhanced semantic alignment. arXiv preprint arXiv:2403.05135. Cited by: §5.1. Huang et al. (2025) K. Huang, C. Duan, K. Sun, E. Xie, Z. Li, and X. Liu T2I-CompBench++: an enhanced and comprehensive benchmark for compositional text-to-image generation. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §5.1. Jain et al. (2021) S. Jain, A. Agrawal, A. Saporta, S. Q. H. Truong, D. N. Duong, T. Bui, P. Chambon, Y. Zhang, M. P. Lungren, A. Y. Ng, et al. RadGraph: extracting clinical entities and relations from radiology reports. In NeurIPS Datasets and Benchmarks, Cited by: §5.3. Jayasumana et al. (2024) S. Jayasumana, S. Ramalingam, A. Veit, D. Glasner, A. Chakrabarti, and S. Kumar Rethinking FID: towards a better evaluation metric for image generation. In CVPR, Cited by: §5.1. Johnson et al. (2019) A. E. W. Johnson, T. J. Pollard, N. R. Greenbaum, M. P. Lungren, C. Deng, Y. Peng, Z. Lu, R. G. Mark, S. J. Berkowitz, and S. Horng MIMIC-CXR-JPG: a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042. Cited by: §5.3. Ju et al. (2024) X. Ju, A. Zeng, Y. Bian, S. Liu, and Q. Xu PnP inversion: boosting diffusion-based editing with 3 lines of code. In ICLR, Cited by: §5.3. Kim et al. (2025) D. Kim, J. He, Q. Yu, C. Yang, X. Shen, S. Kwak, and L. Chen Democratizing text-to-image masked generative models with compact text-aware one-dimensional tokens. In ICCV, Cited by: §2. Kusupati et al. (2022) A. Kusupati, G. Bhatt, A. Rege, M. Wallingford, A. Sinha, V. Ramanujan, W. Howard-Snyder, K. Chen, S. Kakade, P. Jain, and A. Farhadi Matryoshka representation learning. In NeurIPS, Cited by: §2. Li et al. (2025) S. Li, J. Gu, K. Liu, Z. Lin, Z. Wei, A. Grover, and J. Kuen Lavida-O: elastic large masked diffusion models for unified multimodal understanding and generation. arXiv preprint arXiv:2509.19244. Cited by: §2. Li et al. (2024) X. Li, H. Tu, M. Hui, Z. Wang, B. Zhao, J. Xiao, S. Ren, J. Mei, Q. Liu, H. Zheng, Y. Zhou, and C. Xie What if we recaption billions of web images with LLaMA-3?. arXiv preprint arXiv:2406.08478. External Links: Link Cited by: §5.1. Lipman et al. (2023) Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le Flow matching for generative modeling. In ICLR, Cited by: §4.1. Liu et al. (2023) H. Liu, W. Yan, and P. Abbeel Language quantized autoencoders: towards unsupervised text-image alignment. In NeurIPS, Cited by: §2. Liu et al. (2021) J. Liu, W. Zhang, and H. V. Poor A rate-distortion framework for characterizing semantic information. In IEEE International Symposium on Information Theory (ISIT), Cited by: §2. Liu et al. (2025) Q. Liu, X. Liu, Y. Zhou, J. Hou, A. Yuille, and L. Zhang Flowing from words to pixels: a framework for cross-modality evolution. In CVPR, Cited by: §1, §2. Ma et al. (2025) C. Ma, Y. Jiang, J. Wu, J. Yang, X. Yu, Z. Yuan, B. Peng, and X. Qi UniTok: a unified tokenizer for visual generation and understanding. arXiv preprint arXiv:2502.20321. Cited by: §2. Mentzer et al. (2024) F. Mentzer, D. Minnen, E. Agustsson, and M. Tschannen Finite scalar quantization: VQ-VAE made simple. In ICLR, Cited by: §4.1. Miwa et al. (2025) K. Miwa, K. Sasaki, H. Arai, T. Takahashi, and Y. Yamaguchi One-D-Piece: image tokenizer meets quality-controllable compression. arXiv preprint arXiv:2501.10064. Cited by: §1, §2. Niu et al. (2025) Y. Niu, M. Ning, M. Zheng, B. Lin, P. Jin, J. Liao, K. Ning, B. Zhu, and L. Yuan WISE: a world knowledge-informed semantic evaluation for text-to-image generation. arXiv preprint arXiv:2503.07265. Cited by: §5.1. Onoe et al. (2024) Y. Onoe, S. Rane, Z. Berger, Y. Bitton, J. Cho, R. Garg, A. Ku, Z. Parekh, J. Pont-Tuset, G. Tanzer, S. Wang, and J. Baldridge DOCCI: descriptions of connected and contrasting images. In ECCV, Cited by: §5.1. Peebles and Xie (2023) W. Peebles and S. Xie Scalable diffusion models with transformers. In ICCV, Cited by: §5.1. Podell et al. (2024) D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach SDXL: improving latent diffusion models for high-resolution image synthesis. In ICLR, Cited by: §5.1. Poole et al. (2019) B. Poole, S. Ozair, A. van den Oord, A. A. Alemi, and G. Tucker On variational bounds of mutual information. In ICML, Cited by: Appendix F, §5.2. Qin et al. (2025) Q. Qin, L. Zhuo, Y. Xin, R. Du, Z. Li, B. Fu, Y. Lu, J. Yuan, X. Li, D. Liu, et al. Lumina-Image 2.0: a unified and efficient image generative framework. arXiv preprint arXiv:2503.21758. Cited by: §5.1. Qu et al. (2025) L. Qu, H. Zhang, Y. Liu, X. Wang, Y. Jiang, Y. Gao, H. Ye, D. K. Du, Z. Yuan, and X. Wu TokenFlow: unified image tokenizer for multimodal understanding and generation. In CVPR, Cited by: §2. Ramesh et al. (2022) A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125. Cited by: §2. Rimoldi (1994) B. Rimoldi Successive refinement of information: characterization of the achievable rates. IEEE Transactions on Information Theory 40 (1), p. 253–259. Cited by: Appendix A, §1, §2, §3.1. Rippel et al. (2014) O. Rippel, M. A. Gelbart, and R. P. Adams Learning ordered representations with nested dropout. In ICML, Cited by: §1, §2. Rohrbach et al. (2018) A. Rohrbach, L. A. Hendricks, K. Burns, T. Darrell, and K. Saenko Object hallucination in image captioning. In EMNLP, Cited by: §5.1. Rout et al. (2025) L. Rout, Y. Chen, N. Ruiz, C. Caramanis, S. Shakkottai, and W. Chu Semantic image inversion and editing using rectified stochastic differential equations. In ICLR, Cited by: §5.3. Shi et al. (2025) Q. Shi, J. Bai, Z. Zhao, W. Chai, K. Yu, J. Wu, S. Song, Y. Tong, X. Li, X. Li, and S. Yan Muddit: liberating generation beyond text-to-image with a unified discrete diffusion model. arXiv preprint arXiv:2505.23606. Cited by: §2, §5.1. Siméoni et al. (2025) O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, et al. DINOv3. arXiv preprint arXiv:2508.10104. Cited by: §5.2. Singla et al. (2024) V. Singla, K. Yue, S. Paul, R. Shirkavand, M. Jayawardhana, A. Ganjdanesh, H. Huang, A. Bhatele, G. Somepalli, and T. Goldstein From pixels to prose: a large dataset of dense image captions. arXiv preprint arXiv:2406.10328. Cited by: §5.1. Smit et al. (2020) A. Smit, S. Jain, P. Rajpurkar, A. Pareek, A. Y. Ng, and M. P. Lungren Combining automatic labelers and expert annotations for accurate radiology report labeling using BERT. In EMNLP, Cited by: §5.3. Tian et al. (2024) K. Tian, Y. Jiang, Z. Yuan, B. Peng, and L. Wang Visual autoregressive modeling: scalable image generation via next-scale prediction. In NeurIPS, Cited by: §2. Tishby et al. (1999) N. Tishby, F. C. Pereira, and W. Bialek The information bottleneck method. In Allerton Conference on Communication, Control and Computing, Cited by: §2. Urbanek et al. (2024) J. Urbanek, F. Bordes, P. Astolfi, M. Williamson, V. Sharma, and A. Romero-Soriano A picture is worth more than 77 text tokens: evaluating CLIP-style models on dense captions. In CVPR, Cited by: §5.1. Wang et al. (2024) X. Wang, X. Zhang, Z. Luo, Q. Sun, Y. Cui, J. Wang, F. Zhang, Y. Wang, Z. Li, Q. Yu, et al. Emu3: next-token prediction is all you need. arXiv preprint arXiv:2409.18869. Cited by: §1, §2, §5.1. Wei et al. (2025) X. Wei, J. Zhang, F. Wang, T. Chen, Y. Zhou, Z. Li, and B. Li TIIF-Bench: how does your T2I model follow your instructions?. arXiv preprint arXiv:2506.02161. Cited by: §5.1. Wen et al. (2025) X. Wen, B. Zhao, Y. Chen, J. Pang, and X. Qi “Principal components” enable a new language of images. arXiv preprint arXiv:2503.08685. Cited by: §1, §2, §5.1. Witsenhausen (1980) H. S. Witsenhausen Indirect rate distortion problems. IEEE Transactions on Information Theory 26 (5), p. 518–521. Cited by: §2. Wu et al. (2025a) C. Wu, P. Chen, J. Chen, M. Yan, Y. Zhou, X. Zhang, Y. Wang, and Z. Liu OmniGen2: exploration to advanced multimodal generation. arXiv preprint arXiv:2506.18871. Cited by: §2, §5.1. Wu et al. (2025b) Y. Wu, Z. Zhang, J. Chen, H. Tang, D. Li, Y. Fang, L. Zhu, E. Xie, H. Yin, L. Yi, S. Han, and Y. Lu VILA-U: a unified foundation model integrating visual understanding and generation. In ICLR, Cited by: §2. Xie et al. (2025a) J. Xie, W. Mao, Z. Bai, D. J. Zhang, W. Wang, K. Q. Lin, Y. Gu, Z. Chen, Z. Yang, and M. Z. Shou Show-o: one single transformer to unify multimodal understanding and generation. In ICLR, Cited by: §2. Xie et al. (2025b) J. Xie, Z. Yang, and M. Z. Shou Show-o2: improved native unified multimodal models. arXiv preprint arXiv:2506.15564. Cited by: §2, §5.1. Xin et al. (2025) Y. Xin, Q. Qin, S. Luo, K. Zhu, J. Yan, Y. Tai, J. Lei, Y. Cao, K. Wang, Y. Wang, et al. Lumina-DiMOO: an omni diffusion large language model for multi-modal generation and understanding. arXiv preprint arXiv:2510.06308. Cited by: §1, §2, §5.1. Yang et al. (2025a) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §5.1. Yang et al. (2025b) L. Yang, Y. Tian, B. Li, X. Zhang, K. Shen, Y. Tong, and M. Wang MMaDA: multimodal large diffusion language models. arXiv preprint arXiv:2505.15809. Cited by: §1, §2, §5.1. Ye et al. (2025) Q. Ye, X. Zeng, F. Li, C. Li, and H. Fan Painting with words: elevating detailed image captioning with benchmark and alignment learning. arXiv preprint arXiv:2503.07906. External Links: Link Cited by: §5.1. Yu et al. (2023) L. Yu, Y. Cheng, Z. Wang, V. Kumar, W. Macherey, Y. Huang, D. A. Ross, I. Essa, Y. Bisk, M. Yang, K. Murphy, A. G. Hauptmann, and L. Jiang SPAE: semantic pyramid autoencoder for multimodal generation with frozen LLMs. In NeurIPS, Cited by: §2. Yu et al. (2024) Q. Yu, M. Weber, X. Deng, X. Shen, D. Cremers, and L. Chen An image is worth 32 tokens for reconstruction and generation. In NeurIPS, Cited by: §2. Yu et al. (2025) S. Yu, S. Kwak, H. Jang, J. Jeong, J. Huang, J. Shin, and S. Xie Representation alignment for generation: training diffusion transformers is easier than you think. In ICLR, Cited by: §5.2. Zhang et al. (2021) G. Zhang, J. Qian, J. Chen, and A. Khisti Universal rate-distortion-perception representations for lossy compression. In NeurIPS, Cited by: §2. Zhao et al. (2025a) S. Zhao, X. Zhang, J. Guo, J. Hu, L. Duan, M. Fu, Y. X. Chng, G. Wang, Q. Chen, Z. Xu, et al. Unified multimodal understanding and generation models: advances, challenges, and opportunities. arXiv preprint arXiv:2505.02567. Cited by: §2. Zhao et al. (2025b) Y. Zhao, F. Xue, S. Reed, L. Fan, Y. Zhu, J. Kautz, Z. Yu, P. Krähenbühl, and D. Huang QLIP: text-aligned visual tokenization unifies auto-regressive multimodal understanding and generation. arXiv preprint arXiv:2502.05178. Cited by: §2. Zhou et al. (2025) C. Zhou, L. Yu, A. Babu, K. Tirumala, M. Yasunaga, L. Shamis, J. Kahn, X. Ma, L. Zettlemoyer, and O. Levy Transfusion: predict the next token and diffuse images with one multi-modal model. In ICLR, Cited by: §1, §2. Zhu et al. (2025) J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, et al. InternVL3: exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479. External Links: Link Cited by: §5.1. Appendix A Proofs Throughout, rates are informational and all random variables take values in standard Borel spaces with the minimizers below attained (true for finite alphabets and for Gaussian sources under squared error). Captions are deterministic, Tm=ψm(X)T_m= _m(X), so H(Tm∣X)=0H(T_m X)=0 and I(X,Tm)=H(Tm)I(X;T_m)=H(T_m). A caption stage is the per-letter measure dm(x,t^)=[t^≠ψm(x)]d_m(x, t)=1[ t≠ _m(x)] at distortion level zero, whose optimum is H(Tm)H(T_m). A nested code is represented by its stage descriptions W1,W2,…W_1,W_2,… with WjW_j a function of Wj+1W_j+1; the successive-refinement coding theorem (Rimoldi, 1994; El Gamal and Kim, 2011) makes the cumulative rates I(X,Wj)I(X;W_j) achievable by one block code. Its proof (superposition coding with a joint-typicality check per stage for achievability, and the chain rule for the converse) uses the distortion constraint of each stage separately and never uses that the stages share a measure, so it applies to a schedule whose stages are judged by different measures. A.1 Proof of Proposition 1 Converse. Let WmW_m be the stage-m description, which determines TmT_m. Since WjW_j is a function of WmW_m for j≤mj≤ m, WmW_m determines (T1,…,Tm)(T_1,…,T_m), so X→Wm→(T1,…,Tm)X→ W_m→(T_1,…,T_m) is a Markov chain. By data processing, I(X,Wm)≥I(X,T1,…,Tm)=H(T1,…,Tm)I(X;W_m)≥ I(X;T_1,…,T_m)=H(T_1,…,T_m). The excess over the stage optimum H(Tm)H(T_m) is H(T1,…,Tm)−H(Tm)=H(T1,…,Tm−1∣Tm)=ΔmH(T_1,…,T_m)-H(T_m)=H(T_1,…,T_m-1 T_m)= _m. Achievability. Take Wm=(T1,…,Tm)W_m=(T_1,…,T_m). These descriptions are nested, determine TmT_m, and satisfy I(X,Wm)=H(T1,…,Tm)I(X;W_m)=H(T_1,…,T_m) for every m simultaneously. Characterization. Δm=0 _m=0 for all m iff each TjT_j, j<mj<m, is a function of TmT_m for all m, which holds iff Tm=ϕm(Tm+1)T_m= _m(T_m+1) for every m (one direction by composition, the other by taking j=m−1j=m-1). Finally Δm+1=H(T1,…,Tm∣Tm+1)≥H(Tm∣Tm+1)=δm _m+1=H(T_1,…,T_m T_m+1)≥ H(T_m T_m+1)= _m. □ A.2 Proof of Proposition 2 Let A∼(0,σA2)A (0, _A^2) and B∼(0,σB2)B (0, _B^2) be independent, X=(A,B)X=(A,B), d(x,x^)=‖x−x^‖2d(x, x)=\|x- x\|^2, and T=[B>0]T=1[B>0], so H(T)=log2H(T)= 2. For a Gaussian vector under squared error, reverse water-filling (Cover and Thomas, 2006, Ch. 10) gives the optimal test channel at rate R: with water level θ, component A receives rate 12log(σA2/θ) 12 ( _A^2/θ) and B receives max0,12log(σB2/θ) \0, 12 ( _B^2/θ)\. For R≤R⋆=12log(σA2/σB2)R≤ R = 12 ( _A^2/ _B^2) the water level satisfies θ≥σB2θ≥ _B^2, so B receives no rate and the optimal reconstruction is (A^,0)( A,0) with A A a linear function of A and of test channel noise independent of B. Gaussian sources are successively refinable under squared error (Equitz and Cover, 1991), and the reverse water-filling solutions at decreasing distortions are nested, so a single code attains RX(Dj)R_X(D_j) at every stage. Let the stages have rates r,2r,…,Lr,2r,…,Lr and choose σA2/σB2≥eLr _A^2/ _B^2≥ e^Lr, so that R⋆≥Lr/2R ≥ Lr/2. Every stage k≤L/2k≤ L/2 then has rate at most R⋆R , its description is independent of B, and I(Z≤k,T)≤I(Z≤k,B)=0I(Z_≤ k;T)≤ I(Z_≤ k;B)=0. □ The argument uses only that the caption depends on low-variance directions; any source whose caption-relevant directions carry less variance than caption-irrelevant ones behaves the same way at low rate. A.3 Proof of Proposition 3 and Corollary 1 Any description W from which T is determined and X X is reconstructed satisfies I(X,W)≥I(X,T,X^)I(X;W)≥ I(X;T, X) by data processing, and (T,X^)(T, X) is itself such a description; hence RX(D)+π(D)R_X(D)+π(D) in equation 1 is the least rate of a code that determines T and reaches distortion D. Lower bounds. For any feasible X X, I(X,T,X^)≥I(X,X^)≥RX(D)I(X;T, X)≥ I(X; X)≥ R_X(D), so π(D)≥0π(D)≥ 0; and I(X,T,X^)≥I(X,T)=H(T)I(X;T, X)≥ I(X;T)=H(T), so π(D)≥H(T)−RX(D)π(D)≥ H(T)-R_X(D). Upper bound. For an optimal X^D∗ X^*_D, I(X,T,X^D∗)=I(X,X^D∗)+I(X;T∣X^D∗)=RX(D)+H(T∣X^D∗)−H(T∣X,X^D∗)=RX(D)+H(T∣X^D∗)I(X;T, X^*_D)=I(X; X^*_D)+I(X;T X^*_D)=R_X(D)+H(T X^*_D)-H(T X, X^*_D)=R_X(D)+H(T X^*_D), since T is a function of X. Equality case. If some optimal X^D∗ X^*_D determines T, the upper bound is 00. Conversely, if π(D)=0π(D)=0, let X^∘ X attain the minimum in equation 1. Then RX(D)=I(X,T,X^∘)=I(X,X^∘)+H(T∣X^∘)≥RX(D)+H(T∣X^∘)R_X(D)=I(X;T, X )=I(X; X )+H(T X )≥ R_X(D)+H(T X ), so H(T∣X^∘)=0H(T X )=0 and I(X,X^∘)=RX(D)I(X; X )=R_X(D): X^∘ X is an optimal reconstruction that determines T. Simultaneity. Let X→X^J→…→X^1X→ X_J→…→ X_1 be a successively refinable pixel code with I(X,X^j)=RX(Dj)I(X; X_j)=R_X(D_j) and d(X,X^j)≤DjEd(X, X_j)≤ D_j. Define the words-first descriptions W0=TW_0=T and Wj=(T,X^1,…,X^j)W_j=(T, X_1,…, X_j). They are nested and I(X;Wj)=I(X;X^1:j)+H(T∣X^1:j)=I(X;X^j)+H(T∣X^1:j)≤RX(Dj)+H(T∣X^j)I(X;W_j)=I(X; X_1:j)+H(T X_1:j)=I(X; X_j)+H(T X_1:j)≤ R_X(D_j)+H(T X_j), using the Markov chain for the first term and the fact that conditioning reduces entropy for the second. Preceding W0W_0 by the caption stages of Proposition 1 (zero excess under the chain condition) gives one words-first code that meets every stage’s upper bound. □ Corollary. Let Pe=Pr[ψ~(X^D∗)≠T]P_e= [ ψ( X^*_D)≠ T]. An error requires either that X lie in the exceptional set or that d(X,X^D∗)>ρd(X, X^*_D)>ρ, so by the union bound and Markov’s inequality Pe≤ϵ+d(X,X^D∗)/ρ≤ϵ+D/ρ≤12P_e≤ε+Ed(X, X^*_D)/ρ≤ε+D/ρ≤ 12. Conditioning on X^D∗ X^*_D reveals at least as much as conditioning on ψ~(X^D∗) ψ( X^*_D), so by Fano’s inequality H(T∣X^D∗)≤hb(Pe)+Pelog(||−1)H(T X^*_D)≤ h_b(P_e)+P_e (|T|-1). Since hbh_b increases on [0,12][0, 12], H(T∣X^D∗)≤hb(ϵ+D/ρ)+(ϵ+D/ρ)log||H(T X^*_D)≤ h_b(ε+D/ρ)+(ε+D/ρ) |T|, and Proposition 3 gives the bound on π(D)π(D). For the limit, fix ϵ>0ε>0 and a matching ρ(ϵ)ρ(ε); then lim supD→0π(D)≤hb(ϵ)+ϵlog|| _D→ 0π(D)≤ h_b(ε)+ε |T|, which tends to zero with ϵε because T is finite. □ A.4 Sampled captions If the captioner samples, TM∼P(⋅∣X)T_M P(· X) and Tm=ϕm(Tm+1)T_m= _m(T_m+1), a prefix can at best be sufficient for TmT_m, I(Z≤km,Tm)=I(X,Tm)I(Z_≤ k_m;T_m)=I(X;T_m). The stage optimum becomes the rate of the minimal sufficient statistic of X for TmT_m, and the same arguments go through with I(X,⋅)I(X;·) in place of H(⋅)H(·); the summarization chain becomes the Markov chain X→TM→…→T1X→ T_M→…→ T_1. LadderCap decodes greedily, so the deterministic case of the main text is the one that applies. Appendix B LadderCap construction and audit Prompts. The base captioner (Qwen2.5-VL-7B, greedy, at most 256 new tokens) receives “Describe this image in about 180 words. Mention every visible object with its count, colour and position, then the setting, lighting and style.” The summarizer (Qwen3-8B, greedy, thinking disabled) receives, for each level m<Mm<M, the level-(m+1)(m+1) caption and “Summarize the description in at most NmN_m words. Use only facts stated in it. Keep counts, colours and spatial relations of the main objects before any other detail.” with Nm∈8,20,45,90N_m∈\8,20,45,90\. Level 1 states a count or a relation. When the level-5 caption mentions two or more objects, the 8-word level must state a count or a relation between them. The summarizer is re-prompted once when a rule-based parser finds neither; afterwards 91% of level-1 captions of multi-object images satisfy the rule, and the rest keep the first output. Statistics. Table 6 lists per-level lengths, caption entropies under the Qwen3-0.6B model of Appendix F, the resulting cuts, and the entailment audit. A thirty-word caption, between levels 2 and 3, carries about a hundred nats. Entailment audit. On 5K held-out images we extract attribute tuples from every level and ask Qwen2.5-72B-Instruct whether each tuple of level m is entailed by level m+1m+1; two annotators checked 500 images and agreed with the judge on 94% of tuples. Violations are mostly paraphrased colours (“maroon” summarized as “red”) and merged counts (“two cups and a mug” summarized as “three cups”). Stability across summarizers. Replacing Qwen3-8B by Llama-3.1-8B-Instruct or Gemma-2-9B-it changes δ δ from 0.12 to 0.14 and 0.13, GenEval in the ablation setting from 0.79 to 0.79 and 0.78, and NCC from 92.7 to 92.1 and 92.4. Table 6: Hierarchical characteristics of LadderCap across five granularity tiers. Evaluated with the Qwen3 tokenizer. H^(Tm) H(T_m) indicates empirical entropy under fine-tuned Qwen3-0.6B. Cut positions denote the cumulative code capacity budget where R^k≥H^(Tm) R_k≥ H(T_m). Entailment reflects human-verified semantic agreement on held-out images. Level Avg. Words Avg. Tokens H^(Tm) H(T_m) (nats) Nats / Token Cut (% of R^L R_L) Cut Rung Entailed (%) T1T_1 8.1 10 30 3.0 1.2% 1 97.9 T2T_2 19.6 25 72 2.9 2.9% 1 97.2 T3T_3 44.8 57 150 2.6 6.0% 2 96.6 T4T_4 89.5 110 275 2.5 11.0% 4 95.8 T5T_5 178.3 210 525 2.5 21.0% 7 Root (100.0100.0) Appendix C Hyperparameters Networks. Table 7 lists every network, its size and whether it runs at sampling time. The tables of the main text give the ascent model only (#Par), following the baselines’ convention of counting the generator. Counting the image decoder as well, Rung-L uses 1.9B parameters at sampling time and Rung-XL 4.0B; with correction the language decoder adds 1.7B (3.6B and 5.7B). Training. All stages use AdamW (β=(0.9,0.99)β=(0.9,0.99), weight decay 0.01), bf16, EMA 0.9999 and 64 A100-80GB GPUs. Ladder pretraining (λ=0λ=0): 300k steps, batch 1024, 2562256^2, learning rate 10−410^-4; the prefix length is k=Lk=L with probability 0.2 and otherwise uniform over code units. Language anchoring (λ=1λ=1): 150k steps, batch 512, learning rate 5⋅10−55· 10^-5 for the code and 2⋅10−42· 10^-4 for the LoRA adapters (rank 64); cuts are recomputed every 5k steps from running entropy estimates on 100k held-out images. Ascent: 300k steps at 2562256^2 (batch 1024) and 100k steps at 5122512^2 (batch 512), learning rate 10−410^-4, text dropout 0.1 for guidance; ℒconsL_cons is added with weight 0.1 for the last 50k steps, using one-step flow estimates of the generated prefix. Sampling. Ascent uses 24 flow steps in total, allocated in proportion to block rate with at least one step per block: (1,1,2,2,3,4,5,6)(1,1,2,2,3,4,5,6) for the eight blocks of Section 4.2. Classifier-free guidance (1.5) is applied to ascent only, with both branches in one batch. The image decoder uses 8 Euler steps without guidance. This gives 32 network evaluations; VAE decoding is not counted, for Rung or for any baseline. Correction. Threshold τ=0.8τ=0.8 and redraw budget B=2B=2 in all tables. The verifier runs once after each of the four ladder blocks (4 evaluations). On GenEval a sample needs 0.9 redraws on average, each costing 1.5 ascent steps plus one verifier pass, so a corrected sample costs 32+4+0.9×2.5≈3832+4+0.9× 2.5≈ 38 evaluations (1.2×1.2×). Figure 3b varies B∈1,2,3,4B∈\1,2,3,4\ with τ=0.8τ=0.8 for B≤2B≤ 2 and τ=0.9τ=0.9 above. Best-of-N draws N full samples and keeps the one whose final caption agrees best with the prompt under the same attribute head. Table 7: Network parameter breakdown and sampling roles. Active parameters during sampling exclude offline encoders and frozen base components. Module Architecture Parameters Inference Role Base VAE SD3 VAE (Frozen) 84M Latent Decoding Only Ladder Encoder ViT-B, FSQ [8,8,8,5,5,5][8,8,8,5,5,5] 86M Offline Tokenization Image Decoder DiT (24 layers, width 1152) 0.58B 8 Euler Steps Language Decoder Qwen3-1.7B + LoRA + Attribute Head 1.7B Active in Self-Correction Ascent (RUNG-L) MMDiT (24 layers, width 1536) 1.3B 24 Flow Steps Ascent (RUNG-XL) MMDiT (32 layers, width 2048) 3.4B 24 Flow Steps Appendix D Truncation, variation and editing Table 8 quantifies variation by truncation on 1K held-out images: the code is cut at rung k, the rest is resampled four times, and we measure what survives. Up to rung 7 the preserved attribute tuples follow the caption levels that the prefix contains; past the ladder, later cuts preserve structure and texture while the tuples are already fixed. At rung 20 most of the remaining change is global illumination (mean luminance changes by 9% against 23% at rung 8). Failure cases. We see four recurring failures. (i) Rendered text. Captions rarely spell out text, so glyphs fall in the suffix and are resampled; words on signs change under variation and are garbled in generation. (i) Large layout edits. When a prefix swap moves the main object, the kept suffix still encodes texture for the old layout and seams appear; this affects 7% of PIE-Bench edits, mostly in the “change position” category. (i) Counts above five. The 8-word level often writes “several”, so correction cannot catch a wrong count of seven. (iv) Small objects that only the 180-word level names. They are fixed late in the ladder, where a redraw is cheap but the attribute head is least accurate (82% tuple accuracy at level 5 against 97% at level 1). Table 8: Controlled variation via latent code truncation. Truncating at rung k and resampling subsequent blocks on 1,000 validation images. Preserved tuples quantify retention of level-5 semantic attributes; DINO and LPIPS evaluate structural and perceptual distance to the source image. Cut Rung Cumulative Rate (nats) Preserved Tiers Kept Tuples (%) DINO Sim.↑ LPIPS↓ 2 156 T1T_1–T3T_3 41.0 0.52 0.63 4 312 T1T_1–T4T_4 63.0 0.61 0.56 8 625 T1T_1–T5T_5 96.0 0.74 0.47 20 1560 T1T_1–T5T_5 98.0 0.91 0.24 Appendix E Design ablations Rung-L at 2562256^2 on the 12M-image subset, as in Section 5.3. Replacing the learned language decoder with frozen embeddings, the natural choice for an exactly invertible text interface, costs 9 points for a context-free lookup and 12 for a pooled sentence embedding: such embeddings carry no compositional structure, and the ascent model has to recover relations on its own. Tying descent and ascent into one time-reversible operator costs 4 points and 6 points of NCC, which supports treating the asymmetry between the two directions as real rather than as something to engineer away. Table 9: Ablation studies on architectural and algorithmic components. Evaluated on Rung-L (2562256^2, 12M data). Shaded rows indicate default system configurations. Component Variant GenEval NCC (%) Latency (s)↓ Ascent Model Formulation Autoregressive over Rungs 0.76 89.4 2.6 Single-Step Full-Code Diffusion 0.77 87.2 7.9 Block-Wise Conditional Flow (Default) 0.79 92.7 0.9 Step Allocation across Blocks Uniform Step Distribution 0.76 91.5 0.9 Proportional to Entropy Budget ΔR^b R_b (Default) 0.79 92.7 0.9 Code Depth Discretization L=8L=8 0.75 85.1 0.6 L=16L=16 0.78 89.8 0.8 L=L=32 (Default) 0.79 92.7 0.9 L=64L=64 0.79 92.9 1.2 Continuous Latent Vectors (No FSQ) 0.77 91.0 0.9 Text Interface Mechanism Frozen Context-Free Lookup Table 0.70 77.6 0.8 Frozen Sentence-Level Embedding 0.67 73.1 0.8 Fine-Tuned Language Decoder (Default) 0.79 92.7 0.9 Directional Coupling Tied Symmetric Invertible Flow 0.75 86.7 1.3 Asymmetric Ascent–Descent Operators (Default) 0.79 92.7 0.9 Appendix F Estimators Prefix–caption information. I^(z≤k,T) I(z_≤ k;T) is the Barber–Agakov bound H^(T)−[−logq(T∣z≤k)] H(T)-E[- q(T z_≤ k)] (Barber and Agakov, 2003) with T=T5T=T_5 and q a probe decoder (Qwen3-1.7B with LoRA) trained on frozen prefixes of each code. We divide by the mean caption length (210 tokens) and report nats per token. One probe is trained per code and prefix length on 1M held-out images, with the same architecture and budget for all codes. The bound lies below H^(T5)/|T5|=2.5 H(T_5)/|T_5|=2.5 nats per token by the probe’s cross-entropy, which is 0.3 nats per token at full length and gives the ceiling of 2.2 in Fig. 2a. As a check, the InfoNCE bound (Poole et al., 2019) with critic f(z≤k,T)=⟨g(z≤k),u(T)⟩/τf(z_≤ k,T)= g(z_≤ k),u(T) /τ, where g is a 4-layer transformer over the prefix and u a frozen sentence encoder with a trained linear head, ranks the codes identically at every k; at a quarter of the code it gives 1.1, 2.3, 2.7 and 5.6 nats per caption for the unordered, pixel-first, DINOv3-aligned and words-first codes. With batch 4096 this bound cannot exceed log4096≈8.3 4096≈ 8.3 nats per caption, far below H^(T5)=525 H(T_5)=525, so we use it only for ranking. Rate. R^k=∑uH^(zu) R_k= _u H(z_u) over the code units u in the first k rungs, with H^(zu) H(z_u) the plug-in entropy of unit u’s FSQ symbol (64,000 values) on 1M images. By subadditivity the sum upper-bounds the joint prefix entropy, which equals I(X,Z≤k)I(X;Z_≤ k) for a deterministic encoder. Caption entropies H^(Tm) H(T_m) are cross-entropies under Qwen3-0.6B fine-tuned on LadderCap, which also upper-bound the true entropies. The two biases move the cuts in opposite directions: over-estimating R R places a cut too early and over-estimating H^(Tm) H(T_m) places it too late. Moving every cut by ±4± 4 code units changes GenEval by at most 0.01 in the ablation setting. Price. For a target PSNR D, R^pix(D) R_pix(D) and R^words(D) R_words(D) are the prefix rates at which each code first reaches D on 5K held-out images, interpolated between code units. The measured price is π^(D)=maxR^kM,R^words(D)−R^pix(D) π(D)= \ R_k_M, R_words(D)\- R_pix(D), which counts the words-first code as having determined T once it passes its last cut. It is defined up to the words-first code’s full-length PSNR (26.0 dB), which the pixel-first code reaches at k/L=0.97k/L=0.97; this is the right end of Fig. 2c. The upper bound H^(T∣X^D∗) H(T X^*_D) is the cross-entropy of a captioning probe (Qwen2.5-VL-3B with LoRA) fine-tuned to predict T from pixel-first reconstructions at the matching prefix. Cross-entropy over-estimates conditional entropy, so a measured price below the estimated bound is a necessary check, not a sufficient one. Consistent with data processing, this image probe recovers slightly less than the prefix probe (0.62 against 0.68 nats per token at a quarter of the code). The lower bound is [H^(T)−R^pix(D)]+[ H(T)- R_pix(D)]^+. While the words-first code is still inside its ladder, R^words(D)<R^kM≈H^(T) R_words(D)< R_k_M≈ H(T) and π π equals this bound by construction, so the informative comparison is with the upper bound.