Paper deep dive
Physically Based Rendering in the Latent Space
Vuk Radovanovic, Vishesh Gupta, Adrien Gruson, Binh-Son Hua
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/21/2026, 3:51:14 AM
Summary
The paper introduces 'latent rendering,' a novel paradigm that integrates physically based rendering (PBR) with generative AI by performing light transport simulation directly within the latent space of variational autoencoders (VAEs). By modifying the rendering equation to account for signed values and structural features unique to latent representations, the method uses differentiable rendering to optimize scene parameters against a single reference image. This approach enables physically guided content generation, allowing for precise control over geometry, lighting, and camera views while leveraging the efficiency and expressiveness of latent diffusion models.
Entities (7)
Relation Signals (6)
Latent Rendering → operatesin → Variational Autoencoder
confidence 95% · we introduce physically based rendering in the feature space learned by the variational autoencoders
Latent Rendering → uses → Differentiable Renderer
confidence 95% · Our formulation is derived from our analysis of latent representations... This latent rendering process can be optimized based on differentiable rendering
Neural Refinement Network → models → Residual Latents
confidence 92% · use a neural refinement network to model the residual latents
Latent Rendering → enables → Physically Guided Content Generation
confidence 90% · This allows us to leverage physically based rendering techniques to output latent maps for physically guided content generation.
Stable Diffusion 3.5 → provides → Latent Space
confidence 90% · Here we focus on the latent space defined by the pretrained VAE in Stable Diffusion 3.5
Latent Rendering → generalizesto → Scene Geometry Changes
confidence 85% · demonstrate the generalization of the method to scene geometry changes, lighting changes, and camera view changes.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Image diffusion models have shown impressive image generation capabilities but are often hard to control, in contrast to classical computer graphics pipelines such as physically based rendering. However, we observe that there is a bridge between light transport phenomena and the distribution of latent space values produced by such models. Thus, we introduce physically based rendering in the feature space learned by the variational autoencoders in generative models, enabling light transport simulation in the latent space. This allows us to leverage physically based rendering techniques to output latent maps for physically guided content generation. We propose modifications to the rendering equation, which, when paired with a differentiable renderer, can yield an optimal set of scene parameters that require only minimal refinement to accurately render into the pretrained latent space. We train our method on a single rendered image, and then demonstrate the generalization of the method to scene geometry changes, lighting changes, and camera view changes.
Tags
Links
- Source: https://arxiv.org/abs/2609.21054v1
- Canonical: https://arxiv.org/abs/2609.21054v1
Trouble viewing inline? Open PDF directly →
Full Text
65,750 characters extracted from source content.
Expand or collapse full text
This is the pre-peer reviewed version of the following article: Radovanovic V., Gupta V., Gruson A., Hua B.-S.: Physically Based Rendering in the Latent Space. Computer Graphics Forum (2026), which will be published in final form at https://doi.org/10.1111/cgf.70633. This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Use of Self-Archived Versions. Physically Based Rendering in the Latent Space Vuk Radovanovic1 Vishesh Gupta1 Adrien Gruson2 Binh-Son Hua1 1Trinity College Dublin 2École de Technologie Supérieure (ÉTS) Abstract Image diffusion models have shown impressive image generation capabilities but are often hard to control, in contrast to classical computer graphics pipelines such as physically based rendering. However, we observe that there is a bridge between light transport phenomena and the distribution of latent space values produced by such models. Thus, we introduce physically based rendering in the feature space learned by the variational autoencoders in generative models, enabling light transport simulation in the latent space. This allows us to leverage physically based rendering techniques to output latent maps for physically guided content generation. We propose modifications to the rendering equation, which, when paired with a differentiable renderer, can yield an optimal set of scene parameters that require only minimal refinement to accurately render into the pretrained latent space. We train our method on a single rendered image, and then demonstrate the generalization of the method to scene geometry changes, lighting changes, and camera view changes. ccsComputing methodologies Ray tracing ccsComputing methodologies Machine learning †year: 2026†year: 2026†editors: Y. He, N. Thürey, and L. Liu†subject: Pacific Graphics Short Papers†teaser: We introduce latent rendering, an image synthesis paradigm that leverages light transport simulation to directly render latent representations of autoencoders. The rendered latents can be subsequently decoded to output final color rendering. This paradigm allows us to perform physically based rendering on 3D scenes to output latents for scene editing applications. We train the renderer using a single image (left), and show its generalized rendering to a scene variant (right, with lamp shortened and glass egg shifted). Latent maps are color-coded, with green for positive and red for negative values, with colors fading to black as values approach zero. 1 Introduction The history of computer graphics is marked by a rich family of techniques aiming to solve the longstanding challenge of realistic image synthesis. Notable approaches include rasterization-based methods for real-time graphics and computer games, along with global illumination methods for high-end animation, movie production, and architectural visualizations. Among these methods, Monte Carlo rendering has been popular across applications because it can simulate physically based light transport effects by numerically solving the rendering integral [22]. Several rendering algorithms can be used to estimate this integral, namely path tracing [22], bidirectional path tracing [45, 27], photon mapping [20, 11], and Metropolis light transport [46]. The recent breakthrough of generative modeling in artificial intelligence, i.e., generative AI, has led to radical changes in visual computing, particularly in how visual content is created in computer graphics and computer vision. Particularly, diffusion models [41, 42, 16] demonstrate great promise in solving image synthesis in computer graphics from a new perspective: by generating photorealistic images and videos using natural language and visual descriptions [40, 54, 6, 49]. Such generative models are purely data driven, bypassing the traditional graphics pipeline for visual synthesis. A key challenge in the latest wave of generative AI models is that these models are not grounded in the laws of physics. Most generative AI models learn features from large-scale datasets and synthesize data by learning and subsequently sampling from the training data distributions. Physical plausibility in visual content generation is mainly driven by data, leaving physical accuracy and correctness largely underconstrained. While computer graphics has numerous techniques targeting physical correctness, generative models lack a principled framework rooted in a physical basis. This work is our first step toward applying physically based rendering constraints to generative models. We derive a new rendering paradigm that aims to integrate the rigor of light transport simulation with the expressiveness and scalability of visual content generation. We draw inspirations from state-of-the-art image diffusion models, which operate with three main components: a variational autoencoder (VAE), a denoising network, and a prompt conditioning module. Among these, the VAE defines an encoder and decoder that transforms a color image into a latent space that is highly compressed, but remains structured and interpretable; i.e., the latent maps of an image bear high similarity to the geometric and shading structures that appear in the full, uncompressed image. Motivated by this design, we propose latent rendering, a governing approach that connects physically based rendering and generative models via the latent space. The goal is to repurpose light transport simulation to operate in this latent space and output latent maps, which can be subsequently reconstructed to color images. By learning to render in the latent space, we aim to understand its connection to traditional RGB rendering, facilitating research toward physically-grounded generative rendering. Compared to traditional rendering in the RGB color space, latent rendering offers the following benefits. First, latent space is significantly lower-dimensional than RGB space: e.g., a 1024×1024×31024× 1024× 3 image could be encoded to a 128×128×16128× 128× 16 latent map, a 12×12× reduction that makes rendering operations more compact in the latent space. Second, by rendering into the latent space, we can generalize to render larger latent maps, e.g., 256×256×16256× 256× 16, enabling even higher resolution RGB decoding, e.g., to 2048×20482048× 2048. For downstream applications, we demonstrate that it is possible to more efficiently perform language-guided editing on scene representations using score distillation sampling methods [38]. In such applications, latent rendering allows us to circumvent the image encoding step that would otherwise be necessary for an RGB renderer to interface with a latent diffusion model. In our case, gradients backpropagate to directly update scene parameters without passing through the encoder. To bring this idea to life, we propose the very first latent rendering algorithm. Our formulation is derived from our analysis of latent representations obtained from encoding a physically based rendering. This analysis motivates us to adapt light transport simulation to reproduce the visual effects perceivable in the latent space, leading to our latent rendering formula. This latent rendering process can be optimized based on differentiable rendering and refined using a neural network, making the rendered latents accurately match ground truth representations using only a single RGB rendering for training. The latent representations can then be edited via 3D scene variants under camera movement, light source changes, or object motion, to produce updated latents that can be reconstructed to color rendering. To summarize, our main contributions are: • A thorough analysis of the latent space of a pretrained autoencoder through the lens of physically based light transport; • A latent rendering formulation including a modified rendering equation and a neural refiner which depart from its original physical basis, but provide adequate expressibility to mimic generative latent representations; • A pipeline allowing to freely edit camera poses, light configuration, and object position directly in latent space for physically based scene-aware image editing. Our code is available at https://github.com/trinity-graphics/latent-rendering. 2 Related Work 2.1 Forward and inverse physically-based rendering In computer graphics, light transport simulation has been the core of physically based rendering over the past decade [37]. The rendering equation [22] is the central formulation that dictates how light emitted from sources interacts with 3D objects through their associated bidirectional scattering distribution functions (BSDFs) and is ultimately captured by a virtual camera to produce a photorealistic image. Different solutions to this rendering equation exists, such as path tracing [22], photon mapping [20, 11], and Markov Chain Monte Carlo rendering [46]. Among these, path tracing [22] remains the de facto standard used in industry, with several improvements such as path guiding [13], path reuse [3], and reservoir sampling [52]. A key benefit of physically-based rendering is that this rendering pipeline is fully explainable and controllable, offering fine-grained and precise control of image appearance through explicit scene editing using interactive authoring tools [7]. Light transport simulation can be repurposed to operate in spaces other than the original RGB space. Gradient-domain rendering [28, 23, 2, 18] simulates light transport to estimate image gradients, which are used subsequently for RGB reconstruction [56], taking advantage of the spatial smoothness of rendered images. In gradient-domain rendering, these gradients remain linear to their underlying pixel values. Our latent rendering instead operates in a non-linear compressed data representation learned by an encoder-decoder pair. Light transport simulation could also be made differentiable for inverse rendering tasks [29, 36, 61, 59], e.g., for optimizing scene parameters to match a target image through gradient backpropagation. By applying Reynolds’ transport theorem 1903papers, the differentiation of the rendering equation yields an interior and an exterior term [59]. The exterior term models signal discontinuities such as visibility changes, and must be handled when optimizing scene geometry, typically requiring specialized techniques [64, 55, 43]. The interior term describes how scene parameters, such as BSDFs or emitter profiles, influence the rendered image. These interior derivatives can be efficiently evaluated using backward radiance formulations [35], e.g., path-replay backpropagation [48]. In this work, we built on top of differentiable rendering to redirect light transport simulation to operate in the latent space, exclusively focusing on the interior component for optimizing emitter and BSDF parameters. 2.2 Generative models In contrast to physically based rendering, generative models offer a new paradigm for synthesizing photorealistic images by learning from data, e.g., using diffusion models [41, 16, 42]. Latent diffusion models [40, 54] perform the diffusion process in highly compressed but interpretable latent spaces, enabling large-scale training and high-resolution image generation. Video diffusion models [57, 49] are recent advances that extend image diffusion models for generating animated scenes. While the latent spaces of these generative models are highly expressive, controlling the outputs of diffusion models is a challenging task. General methods such as low-rank adaptation [17] and ControlNet [62] adjust a subset of parameters of the diffusion models to induce specific visual styles, with optional extra inputs such as depth maps or user-provided strokes to control the generation process. The denoising diffusion process could also be intercepted with visual, semantic, or user guidance at early timesteps to provide high-level control for image editing [33]. Alternatively, specific solutions are required for particular tasks such as intrinsic decompositions [32, 12], material generation [26], relighting [60, 4], or novel view synthesis [31, 65]. However, these approaches do not offer sufficiently precise or consistent control over camera parameters, lighting conditions, and object placement simultaneously. 2.3 Hybrid rendering There exist ongoing efforts in combining generative models with traditional rendering concepts to model physically based effects and improve controls over image outputs. For example, transformer-based models have been proposed to directly predict approximate solutions to the rendering equation [58], but they remain limited to relatively small scenes. Video diffusion models can be tuned to output material and environment maps, enabling the generation of material and lighting effects in videos without light transport simulation [30]. Neural renderers could be used to refine physically based rendering results, enabling tasks such as scene relighting from a single image [8] when combined with monocular geometry estimation [50]. A few previous works exploit the latent space for feature rendering. Latent-NeRF [34] performs text-to-3D generation by directing a neural radiance field to output latent features compatible to a pretrained diffusion model, as guided by a score distillation objective. Splatent [15] reconstructs 3D Gaussian Splats in the latent space of a VAE, and leverages a one-step diffusion process to improve the quality of the latents, thereby enhancing novel view rendering. Our work shares the high-level idea of rendering to the latent space as we aim to use the latent space to establish a synergy between latent diffusion models and physically based light transport simulation. As our method is built on physically based rendering, relighting is naturally supported and scene editing is generally more intuitive and controllable. 3 Background 3.1 Physically based rendering We are interested in solving the light transport equation [47], which, a light path ¯=0,…,k x=x_0,…,x_k consisting of k vertices, contributes to pixel i by evaluating the following integral over the path space P: Ii(π)=∫fi(¯,π)d¯=∫ri(¯)T(¯,π)Le(k−1←k,π)d¯. I_i(π)= _Pf_i( x,π)d x= _Pr_i( x)T( x,π)L_e(x_k-1 _k,π)d x. (1) Here, ri(¯)r_i( x) denotes the pixel filter, T(¯,π)T( x,π) the path throughput, and Le(k−1←k,π)L_e(x_k-1 _k,π) the emitted radiance at the light source. The throughput is defined as T(¯)=∏j=1k−1fr(j,π)cosθjT( x)= _j=1^k-1f_r(x_j,π) _j, where frf_r is the bidirectional scattering distribution function (BSDF) evaluated at each surface interaction, and θj _j is the angle between the incident direction at vertex j and the surface normal. The integral depends on scene parameters π, such as BSDF parameters and emitter intensity values. The integral in Eq. 1 can be solved using Monte Carlo estimation. By default, the integral is evaluated in standard color spaces such as the RGB and produces non-negative estimates that respects physical constraints imposed by light transport. 3.2 Differentiable rendering Physically based rendering can be used in an optimization framework to determine optimal scene parameters π π so that the rendering matches a target image IgtI_gt: π^=argminπℒ(I(π),Igt), π= _π\;L(I(π),I_gt), (2) where ℒL is a differentiable objective function such as the mean squared error (MSE). Gradient-based optimizers, such as Adam [24], can be used to solve this optimization problem, but they require access to how changes in the scene parameters affect the rendered image. By the chain rule, this requires computing the gradient of each pixel intensity with respect to the scene parameters: ∂πIi(π)=∂π∫fi(¯,π)¯=∫∂πfi(¯,π)¯. _πI_i(π)= _π _Pf_i( x,π)d x= _P _πf_i( x,π)d x. (3) Transferring the derivative inside the integral requires care when the scene parameters control discontinuities in the rendering process [39]. Several techniques for differentiable Monte Carlo rendering exists [29, 48], but in this work, we assume that our scene parameters vary continuously over the integration domain, and their derivatives could be efficiently estimated using path replay backpropagation [48]. 3.3 Latent spaces Modern generative models [40, 6] are typically trained in two stages. First, an autoencoder is trained to learn a latent space that encodes data into latent variables and reconstructs the original data from these latent representations. Second, a denoising network is learned to map random noise to latent codes. In this work, we focus on the first stage, namely the autoencoder and its latent representations. An autoencoder is composed of an encoder ℰE and decoder D. Given an image x∈ℝH×W×3x ^H× W× 3, the encoder produces a compressed latent representation z=ℰ(x)∈ℝh×w×dz=E(x) ^h× w× d such that h<Hh<H, w<Ww<W, and d>3d>3 where H,W,h,wH,W,h,w are image and latent height and width, and d is the number of latent channels, respectively. We can then decode this latent representation to reconstruct an image x~=(z)∈ℝH×W×3 x=D(z) ^H× W× 3 similar to original image x. This autoencoder is trained to minimize the difference between the distribution of x and x~ x, following the principles of variational autocoders (VAEs) [25]. From empirical designs of image diffusion models, typically we have h=H/8h=H/8, w=W/8w=W/8, d=4d=4 for Stable Diffusion 2.1 and SDXL [40], and d=16d=16 for Stable Diffusion 3.5 and FLUX [10, 6], resulting in latent compression ratio of 48 and 12, respectively. 4 Latent Space Analysis Figure 1: Latent maps of the Cornell Box rendering. The zoom-in patches demonstrate visual effects unique to latent representations, including (A) the signed emitter, (B) highlighted geometry edges, and (C) signed shadows, which cannot be rendered using traditional physically based rendering. This work aims to align rendering values produced by a physically based renderer with autoencoder latents, enabling the renderer to generate latent representations that can subsequently be decoded into color images. Let us first analyze these latents to identify the requirements for adapting a physically based renderer to the latent space. Here we focus on the latent space defined by the pretrained VAE in Stable Diffusion 3.5 [10], an open-weights text-to-image diffusion model for high-quality image generation. We use image resolution of 1024×10241024× 1024, latent resolution 128×128128× 128 with d=16d=16. We analyze the latents of the classic Cornell box as shown in Fig. 1. We observe the following phenomena that makes rendering in this latent space inherently different from RGB rendering: 1. In the latent space, emitted light is signed (Fig. 1, A), in contrast to the strictly non-negative values assumed in physically based rendering. 2. The latents encode scene structure with a strong emphasis on geometric edges and corners (Fig. 1, B). In regions with extremely high-frequency content, latent values may become perceptually indistinguishable from noise. 3. In occluded regions, latent values can exhibit both positive and negative energy, which can be an even higher intensity than surrounding unoccluded regions (Fig. 1, C). This differs from physically based rendering, where occlusion from a light source can only reduce the amount of energy received by a surface. 1. There is structural parity between the RGB image and the corresponding latent channels, in which different BSDFs and objects receive distinct latent values that mirror their RGB counterparts. 2. Signed energy emitted in the latent space is reflected throughout the scene and is closely preserved on perfectly specular surfaces. 3. As the incident angle of light varies smoothly, the intensity of the reflected signed energy varies accordingly. This effect is more pronounced in some channels and less so in others, with certain channels exhibiting a flatter, more uniform appearance (e.g., Fig. 1, channel 02). In the literature, there exist some attempts in modeling the relationship between latents and RGB values directly, such as a 4x4 linear mapping in Stable Diffusion 1.4 [44]. However, such mapping is applied for general images and to each pixel independently. Our analysis indicates that we can take a further step by simulating light transport in the latent space. To achieve this, the rendering process must be adapted to freely express the range of latent values observed. 5 Method Our pilot study suggests that the latent representations might consist latent values producible by physically based rendering, and residual values that require additional modeling, e.g., high-frequency latents at shape boundaries. Based on these observations, we propose to disentangle the latents into two components: physically based latents and residual latents. We leverage light transport simulation to render the physically based latents, and use a neural refinement network to model the residual latents. This disentanglement allows us to separately address physically grounded effects and non-physically based phenomena present in the latents in a principled manner. Our full pipeline is summarized in Figure 2. First, we apply differentiable rendering to train our physically based renderer such that it can output latent values by optimizing the scene parameters (π) to match reference latent images (I~gt I_gt), following our modified latent rendering formulation (Section 5.1). We then predict the residual latents using a neural refinement network (Section 5.3), which learns to compensate for discrepancies between the rendered latents and the target latents, particularly over geometric structures such as edges and corners. Details of our pipeline are as follows. Figure 2: We train a differentiable renderer to output to the latent space defined by a variational autoencoder. We assume a 3D scene is available for our rendering task. Our training pipeline consists of taking a ground truth RGB rendering of the scene and encoding it into the latent space. We then optimize the 3D scene parameters to make its rendering reproduce this latent representation using our modified rendering equation (Eq. 7). The rendered latents are then refined using a per-scene refiner network. The final latents are decoded to reconstruct the RGB rendering. Once the pipeline is trained, it can be applied to render latents for variants of the same scene with edited scene settings. The blue dotted line represents backpropagation for scene parameter optimization. The yellow dotted line represents backpropagation for the refiner parameter training. 5.1 Latent rendering As discussed in the latent analysis section (Section 4), the latent space includes both positive and negative values, whereas conventional physically based rendering produces only non-negative values. A naïve way to adapt a physically based renderer for signed rendering is to shift latent values into the non-negative domain by applying an offset and a scale on the latents, or by splitting the positive and negative latents and perform negation on the negatives. However, we find that both approaches are not effective and efficient to address the observations as outlined in Section 4. Particularly, shifting the values will result in appearance changes while splitting the latents requires doubling the number of latent channels. Reconstructing the modified latents is challenging as well as the decoder requires to de-normalize the latents to their vanilla domain before decoding. Instead of processing the latents heuristically, we therefore opt for a principled approach by modeling the latents using the rendering equation. Let us begin by introducing the concept of signed rendering as follows. 5.1.1 Signed radiance and reflectance In keeping with our goal of strong physically-based priors, we begin our latent rendering approach by modeling the latents’ physically-aligned features: structure, shading, and global illumination. In traditional rendering, emitted light LeL_e and the BSDF frf_r are constrained to be non-negative in order to ensure valid light path sampling. To accommodate signed latent values, we modify the traditional path tracing approach to support the propagation of both positive and negative energy throughout the scene. First, we allow emitters to emanate signed light energy L~e(←,π) L_e(x ,π), so that L~e L_e can emit positive or negative radiance values. We also modify BSDF evaluations to enable them to yield signed values. This leads to the following signed light transport equation: I~ie(π)=∫ri(¯)T~(¯,π)L~e(k−1←k,π)d¯, I_i^e(π)= _Pr_i( x) T( x,π) L_e(x_k-1 _k,π)d x\,, (4) where T~ T is the signed throughput. We note that purely specular materials are not modified, as they behave strictly as ideal mirror surfaces. With both signed radiance and signed reflectance enabled, we can effectively model the observed behavior that latents can take on values outside of the standard rendering range while still maintaining the structure of the shading and lighting effects present in the original image. Our subsequent modifications use this signed rendering as a starting point, and aim to model the rendered latents in a way that will capture features that deviate from a physical basis. 5.1.2 Flat response term Our latent analysis indicates that, in some channels, different surfaces exhibit a comparatively flat appearance rather than faithfully reflecting their original material properties. To account for this behavior, we introduce an additional flat response term applied at surface interactions, which provides explicit control over the extent to which objects preserve material-dependent variation versus collapsing to a near-constant response. The flat response term L~a L_a is a signed radiance value, and its contribution can be estimated with: I~ia(π)=∫Eri(¯)T~(¯,π)L~a(d,π)¯. I_i^a(π)= _P_Er_i( x) T( x,π) L_a(x_d,π)d x\,. (5) where the path space EP_E denotes a camera subpath ¯ x following the ES∗DES^*D format, i.e., a path starting at the camera sensor (E) with an optional series of specular reflections (S∗S^*) followed by a diffuse interaction (D). Note that it is valid to define the flat-response term for full light transport paths, by accumulate it at each interaction, following the original path space P; however, we empirically observe that this does not lead to a significant performance difference, and hence opt for the simpler design using only camera subpaths. 5.1.3 Occlusion term Our latent analysis also shows that shaded regions can deviate from the physically based interpretation of light transport, exhibiting higher intensity than unoccluded regions of the same material, or even values of opposite sign. To model this behavior, we introduce an occlusion term that acts as an additional contribution emitted only from shaded regions. The occlusion term L~c L_c stores signed radiance values, and its contribution to the rendered intensity is estimated as I~ic(π)=∫Eri(¯)T~(¯,π)L~c(d,π)V(d,)¯. I_i^c(π)= _P_Er_i( x) T( x,π) L_c(x_d,π)V(x_d,y)d x\,. (6) Here, we leverage camera subpaths ¯∈E x _E with an ending diffuse vertex dx_d to evaluate this integral. We sample an additional light vertex y on a light source and evaluate the binary visibility function V(d,)V(x_d,y), which equals one when the ending diffuse vertex dx_d is visible from y, and zero otherwise. 5.1.4 Latent rendering equation By combining the above modifications, we obtain our latent rendering equation: I~i(π)=I~ie(π)+I~ia(π)+I~ic(π). I_i(π)= I_i^e(π)+ I_i^a(π)+ I_i^c(π)\,. (7) To estimate this latent intensity, we construct a single light path ¯ x using BSDF sampling and evaluate all corresponding integrals along that path. I~a I^a and I~c I^c are estimated when we encounter the first diffuse surface, whereas we continue to extend the path to estimate I~e I^e. To efficiently estimating I~e I^e, BSDF sampling is combined with next-event estimation (NEE) using multiple importance sampling, where the signed emitters (L~e L_e) are explicitly sampled. Note that rendering systems simulate lighting in linear radiance values, whereas generative models assume gamma-corrected inputs. For this reason, we apply gamma correction with γ=2.2γ=2.2 to the positive and negative values of the rendered latents, respectively: g(x,γ)=max(x,0)γ−max(−x,0)γ. g(x,γ)= (x,0)^γ- (-x,0)^γ\,. (8) 5.2 Optimization We use a differentiable renderer [19] to perform latent rendering (Eq. 7). We optimize the scene parameters to make the renderer produce latents following the latent space defined by the VAE encoder. All latent channels are jointly rendered, analogously to RGB rendering. For a given scene and a reference latent map from a training view, we optimize the parameters π=L~e,L~a,L~cπ=\ L_e, L_a, L_c\ for each path vertex, which is represented as a vector in ℝdR^d with d as the total latent channels. We also optimize the reflectance parameters of each BSDF present in the scene, as well as the alpha parameters of any rough BSDFs. We support diffuse materials, conductors, dielectrics, and plastics and their rough variants. We apply a clipping function to make sure the learnable parameters are non-negative during optimization. To train the renderer, we minimize a Huber loss between the gamma-corrected rendered latents I~ I and ground-truth latents IgtI_gt: ℒπ=12‖I~−I~gt‖22if ‖I~−I~gt‖1≤δ(‖I~−I~gt‖1−12δ)if ‖I~−I~gt‖1>δ _π= cases 12\| I- I_gt\|_2^2&if \| I- I_gt\|_1≤δ\\ δ (\| I- I_gt\|_1- 12δ )&if \| I- I_gt\|_1>δ cases (9) We set δ=1δ=1, which can make optimization more stable when faced with large but infrequent per-pixel errors due to the strongly encoded edges observed in the pilot study. Note that we compute the loss directly in the latent space, which empirically improves training convergence, stability, and efficiency. This also helps avoid inefficient gradient backpropagation through the decoder. 5.3 Latent refinement The rendered latents produced by our fitted scene parameters and latent rendering equation impose a strong physically based prior. However, as discussed in the latent analysis, latent values can deviate from physically based behavior, particularly around object boundaries and other high-frequency regions. We train a feed-forward neural network to predict these residual latents. For simplicity, we train this neural refiner on a per-scene basis using a single ground truth rendering. We leave the generalization of the refiner across scenes for future work. The refiner network RθR_θ takes as input the rendered latent I~ I along with a set of auxiliary buffers (AOVs) denoted as A. These AOVs include surface shading normals and depth, providing geometric cues that guide the network toward correcting structure-dependent artifacts. The AOVs are rendered at the latent resolution and concatenated to the input in a channel-wise manner. The network outputs the residual latents as Rθ(I~,A)R_θ( I,A). We adopt a feed-forward multilayer perceptron architecture for the refiner network. The hidden layer width is chosen as a multiple of the number of latent channels d. For Stable Diffusion 3.5, we use an expansion factor of 16, resulting in hidden layers with 16×16=25616× 16=256 channels. The network then consists of three convolution blocks with kernel sizes 3×33× 3, 5×55× 5, and 3×33× 3. We train the refiner in a separate stage after the physically based renderer is optimized. To train the refiner, we use a composite loss defined in both the latent and RGB spaces: ℒθ=ℒMSE(I~′,I~gt)+ℒSSIM(I~′,I~gt)+ℒLPIPS((I~′),(I~gt)), _θ=L_MSE( I , I_gt)+L_SSIM( I , I_gt)+L_LPIPS(D( I ),D( I_gt))\,, (10) where the refined latent is I~′=I~+Rθ(I~,A) I = I+R_θ( I,A) and D denotes the decoder. The MSE loss and SSIM loss enforces the refined latents to stay close to the ground-truth latents. The LPIPS loss [63] applies perceptual similarity constraints on the decoded image, reducing color shifts and encouraging sharper boundaries. 6 Experimental Results 6.1 Implementation details We use Mitsuba [19] with path replay backpropagation [48], without reparameterization [64], to perform the optimization. We extend Mitsuba to support signed rendering with 16 latent channels, enabling efficient joint optimization over all channels. We also support textured materials: each RGB reflectance texture is converted to grayscale and then mapped into the 16-channel latent space via a learnable per-channel scale and offset. This preserves the texture’s spatial detail while exposing its latent appearance as optimizable parameters. Training follows a multi-stage procedure: we first optimize the scene parameters for 3000 iterations with a learning rate of 10−410^-4, and then train the refinement network for an additional 3000 iterations using the same learning rate. All optimizations use the Adam optimizer [24] with default hyperparameters. We plan to release our implementation upon acceptance. To improve stability during scene parameter optimization, we employ a learning-rate scheduler following a cosine annealing schedule with restarts, as well as early stopping. On a simple scene such as the Cornell Box, training on a single NVIDIA RTX 4090 GPU takes 5 minutes for scene optimization and 5 minutes for refiner training. Scene parameters π are initialized randomly from a uniform distribution in [0,1][0,1] for both positive and negative components of all trainable parameters, as well as the BSDF parameters. We compare our method with a baseline that uses the traditional rendering formula [22] for latent rendering. We produce 16-channel values in the [0,1)[0,1) range with the traditional rendering process, then de-normalize the rendered latents back into the native latent range ∼[−5,5] [-5,5] using the logit function log(x/(1−x)) (x/(1-x)), and train the refiner on these de-normalized latents following Section 5.3. The refiner output is passed directly to the decoder. Compared to our method, this baseline skips the latent modeling in Section 5.1 and simply uses the traditional rendering formula. 6.2 Scene setups We analyze the performance of our latent rendering method on five scenes of increasing complexity: Cornell Box, Lamp, Veach-Bidir, Dining Room, and Living Room. The Cornell Box scene contains a single light source and a mix of diffuse and pure specular materials. The Lamp scene includes multiple light sources and rough plastic materials. The Veach-Bidir scene contains difficult-to-sample light sources and caustics effects. The Dining Room and Living Room scene features a single light source and multiple objects with distinct BSDFs. For Cornell Box and Veach-Bidir, we associate each object with a separate BSDF with its own learnable parameters. For Lamp, Dining Room, Living Room scenes, we use the default shape-BSDF associations provided in original scenes to reduce memory consumption during the optimization of the BSDF parameters. We assign each object with a non-delta BSDF its own flat-response and occlusion parameters, L~a L_a and L~c L_c, which have random initializations. For each scene, we provide reference latents obtained from a single view. These reference latents are obtained by encoding the RGB rendering produced with path tracing on large number of samples per pixels (e.g., 16K spps). Figure 3: We compare the performance of the training view on rendered latent and refined latent with the ground truth. Representative latents for each case are shown on the left of each color rendering. Metrics shown are LPIPS/MSEx100. Figure 4: Comparison between the baseline and our method on the Lamp scene. We observe that our latent modeling and rendering formula provide significant quality improvement compared to the baseline that only relies on latent normalization and traditional rendering. Metrics shown are LPIPS/MSEx100. Figure 5: The error of the Lamp scene across 60 frames of scene editing variants. Frame 0 corresponds to the training view. Figure 6: Qualitative results on the Veach-Bidir scene. Metrics shown are LPIPS/MSEx100. We demonstrate the capabilities of our latent rendering framework on scene editing tasks. In all experiments, the scene geometry, material types (e.g., diffuse or specular), emitter locations, and camera pose are assumed to be known. We perform scene edits and render latents in the edited scene to explore the generalization of latent rendering to various scenarios. We consider camera movement for novel view synthesis, light source movement for scene relighting, and object movement for animated rendering. Relaxing the scene input assumptions such as taking a single image as input and predicting scene geometry and PBR materials [50, 53] would be an interesting direction for future work. We compare the latents and their reconstruction with ground truth latents and images, respectively. We report comparison results using the MSE, SSIM [51], and LPIPS metrics. 6.3 Qualitative results Fig. 3 demonstrates how our method performs before and after refinement on the training view. For completeness, we show two sets of representative latents for each decoded RGB image. We observe, across all scenes, that the main content of the latents and their decoded images is produced by the latent renderer, as evidenced by their high similarity to the ground truth. The refiner is then able to overfit and deliver an almost perfect reconstruction. Figs. 10 and 6 provide the rendering results when the scene is edited in various scenarios. We can observe that our method generalizes well to novel configurations: most of the work is done by the rendered latents, while the refiner is responsible for improving object boundaries and increasing overall image contrast and sharpness. For the training view, we found that the post-refinement reconstruction is almost indistinguishable from the ground truth image, which confirms the success of the training. When the scene is edited, through camera motion, changes in light position, or object movement, our method can regenerate images that remain visually close to the ground truth. This result demonstrates two advantages of latent rendering: the trained renderer can be applied to synthesize new latents that can be reconstructed to meaningful images, and the refiner generalizes to unseen settings. Notably, specular reflections and shadowing effects can be reconstructed based on the visual cues in the physically based rendered latents. Fig. 11 shows novel view rendering on three additional scenes (Bedroom, Classroom, and Living Room 2) with texture support. It shows that latent rendering with texture is promising, with further optimizations for quality improvement left as future work. Video results for each scene are provided in the supplemental material. Baseline comparison. We provide baseline comparisons in Fig. 4. Compared to our method, the rendered latents of the baseline yield color shifts in the reconstruction (1st row). This could be explained by the non-linear range compression and expansion due to normalization and de-normalization, respectively. The refiner significantly improves the rendered latents and the final rendering, but fails to achieve accurate color rendering while having more visual artifacts in the baseline (2nd and 3rd rows). This result confirms the necessity and effectiveness of our latent rendering equation. Errors of scene edits. Fig. 5 provides the error plot of the scene edits for the Lamp scene. The edits are performed for 60 frames, starting from frame 0 as the training view. We use D-SSIM to measure structural difference and LPIPS to measure perceptual difference to the ground truth latents. For all tasks, the rendered latents generalize fairly well across frames with similar errors except for the extreme case in the light movement (around frame 30). The refiner improves the rendered latents significantly across frames. As the refiner is only trained on a single view, D-SSIM and LPIPS are smaller (i.e., better) at views more similar to the training view. 6.4 Quantitative results Table 1: Ablation results reporting the mean LPIPS / MSE×100× 100 achieved over the decoded RGBs. “Var.” denotes the scene-edit variant: camera, object, and light motion. Var. I~ie I_i^e (Eq. 4) +I~ia+\, I_i^a (Eq. 5) +I~ic+\, I_i^c (Eq. 6) + Refiner Cornell Box Cam. 0.502 / 2.264 0.228 / 0.335 0.213 / 0.222 0.075 / 0.069 Obj. 0.526 / 2.355 0.213 / 0.338 0.202 / 0.261 0.032 / 0.061 Light 0.519 / 2.361 0.209 / 0.330 0.191 / 0.220 0.013 / 0.024 Lamp Cam. 0.394 / 3.091 0.206 / 0.844 0.178 / 0.564 0.068 / 0.297 Obj. 0.439 / 3.274 0.236 / 0.791 0.202 / 0.608 0.020 / 0.048 Light 0.504 / 5.702 0.292 / 3.000 0.253 / 2.385 0.086 / 1.867 Living Room Cam. 0.643 / 5.502 0.281 / 0.599 0.250 / 0.480 0.072 / 0.115 Obj. 0.648 / 5.584 0.280 / 0.573 0.246 / 0.451 0.008 / 0.013 Light 0.659 / 6.031 0.299 / 1.053 0.266 / 0.894 0.064 / 0.392 Ablation studies. Table 1 presents our ablation study of the latent formulation and the impact of the refiner on the final decoded image across different scene edits. In this table, we report the mean LPIPS and MSE values attained along each animated editing sequence (moving camera, object and emitter). Using only signed emitters (Eq. 4) leads to poor performance and unstable training. In contrast, the flat-response term (Eq. 5) and occlusion term (Eq. 6) have a significant impact on image quality. The neural refiner further improves the latents, enabling higher quality image reconstruction. Note that the refiner only acts on predicting the residual latents once the rendered latents are available. If the rendered latents are noisy (e.g., produced using only signed emitters), the refiner does not yield significant improvement. Convergence analysis. Theoretically, latent rendering follows the Monte Carlo estimation framework, and hence, the convergence properties in the latent space remains the same as in the RGB space. We use path tracing in this work, and therefore, the estimation of the rendered latents are unbiased. However, the use of the refiner network results in biased latents, and the image reconstruction using the decoder results in biased final rendering. Ideally, the rendering would become unbiased if the refiner and the decoder were linear functions. Empirically, we provide convergence plots of our rendering in Fig. 7. We observe that the convergence patterns are similar across scenes, and therefore provide the convergence of the Veach-Bidir scene as a representative. The plots confirm that, for the training view, both the latent and the decoded RGB estimates converge as the number of samples per pixel increases. The novel view exhibits a similar trend, with a small residual bias introduced by the refiner. Figure 7: Empirical convergence vs. samples per pixel (spp) for a novel view (top) and a training view (bottom) of the Veach-Bidir scene. LPIPS on the decoded RGB (left) and DSSIM on the latents (right). 6.5 Runtime analysis Figure 8: Equal-error runtime comparison between RGB path tracing and our latent rendering method (with refiner), for RGB space (left) and latent space (right). The plot is based on frame 30 of the move_camera test case of the Lamp scene. We evaluate the runtime of latent rendering against RGB path tracing in an equal-error setting: for each method we measure the time and the number of samples per pixel required to reach a fixed MSE threshold (Fig. 8). When the goal is to estimate values in the latent space (right), our method is significantly faster (4242–84×84×) than first path tracing in RGB and then encoding, since the latent render and refiner are efficient and requires less samples. When the goal is a decoded RGB image (left), however, our method is not competitive with direct RGB path tracing: despite the spatial compression, the single decoder evaluation dominates the runtime (the red bar) and makes our method 66–35×35× slower. Accelerating the decoder is therefore key to making latent rendering competitive for RGB output, and is an interesting direction for future work. 6.6 Applications (a) Original scene (b) SDS with RGB rendering + encoder (c) SDS with latent rendering Figure 9: Application using score distillation sampling (SDS) [38]. We use the pretrained diffusion model to iteratively update the reflectance of the teapot following guidance from the text prompt “A blue teapot and a lamp on a table.” Scene editing. Latent rendering could be further explored for downstream applications. For example, as latent rendering naturally operates in the latent space, one can integrate score distillation sampling with latent rendering, enabling the optimization of scene parameters guided by a pre-trained diffusion prior [38]. Image editing applications could be integrated with latent rendering as well, e.g., object insertion and background removal using blended latents [1], image harmonization and semantic image composition using delta denoising distillation [14, 9]. We provide an example of using score distillation [38] to re-color the teapot in the Lamp scene by optimizing its BSDF reflectance. The editing is guided by the text prompt “a blue teapot and a lamp on a table”. Traditional SDS operates the optimization in the latent space, and therefore it requires first performing the rendering in the RGB space, and then encode the rendered image using the encoder. By using latent rendering, we can directly compute the SDS objective based on the refined latents, skipping the encoder in the optimization pipeline. The results are shown in Fig. 9. Both the RGB rendering approach and our latent rendering approach can successfully update the teapot color. Due to the per-scene training of the refiner, our teapot results in some blurs that could be further corrected by improving the generalization of the refiner. 6.7 Limitations and future work Our method is a very first step toward latent rendering. Several limitations remain for further investigations in future work. First, as the latent space is in low spatial resolutions, aliasing tends to dominate when rendering highly detailed objects. Second, while our refiner does fairly well at generalizing to intra-scene changes, it is trained only on a single image and with a per-scene setting, and thus does not generalize across scenes. Building a universal refiner across scenes would improve the practicality of the method. Finally, while latent rendering offers a new angle for extending physically based rendering research, there are still rooms for improving image quality and running time to match the performance of traditional methods. As future work, we aim to explore latent rendering with state-of-the-art visual diffusion models and pursue more accurate latent representations and processing for extended applicability and improved computational efficiency. For example, while our trained renderer and latent refiner generalizes well to higher resolution rendering, it is possible to add neural upsampling. Latent space super resolution [21] validated that since latent space represents image features in a spatially compressed format with high-level information, performing upsampling in this domain preserves well details and sharpness and could generate more details compared to RGB upsampling. Applying this method to improve our latent rendering would be future work. 7 Conclusion We have presented latent rendering, a paradigm that trains a physically based renderer to output latent representations defined by the autoencoders of a generative model. The rendered latents can be reconstructed to color images following a neural refiner and an image decoder. Applications include direct editing of a 3D scene rendering in latent space, enabling a straightforward process to render scene variants under camera, lighting, and object changes. We believe that latent rendering is a promising approach to unlock the connection between physically based rendering and generative models. By leveraging latent spaces, we can enable the integration of physical priors directly into creative content generation. Acknowledgment. We would like to thank Wig42 (Dining Room and Living Room), Jay-Artist (Living Room 2), NovaZeeke (Classroom), SlykDrako (Bedroom), UP3D (Lamp), and Benedikt Bitterli [5] for providing the scenes, which we obtained from the Mitsuba 3 gallery. This project is supported by Research Ireland under the Research Ireland Frontiers for the Future Programme - Project, award number 22/FFP-P/11522. Figure 10: We compare the performance of the scene editing on rendered latent and refined latent with the ground truth. Representative latents for each case are shown on the left of each color rendering. Metrics shown are LPIPS/MSEx100. Figure 11: We compare the performance of novel view scene editing on rendered latents and refined latents with the ground truth. We showcase complex scenes with texture support. Metrics shown are LPIPS/MSEx100. References [1] O. Avrahami, O. Fried, and D. Lischinski (2023) Blended latent diffusion. ACM Trans. Graph.. External Links: Document Cited by: §6.6. [2] P. Bauszat, V. Petitjean, and E. Eisemann (2017) Gradient-domain path reusing. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia) 36 (6), p. 229:1–229:9. External Links: Document Cited by: §2.1. [3] P. Bekaert, M. Sbert, and J. Halton (2002) Accelerating path tracing by re-using paths. In Proceedings of the Eurographics Workshop on Rendering, p. 125–134. External Links: Document Cited by: §2.1. [4] S. Bharadwaj, H. Feng, G. Becherini, V. Fernandez Abrevaya, and M. J. Black (2025) GenLit: reformulating single-image relighting as video generation. In ACM SIGGRAPH Asia Conference Papers, External Links: Document Cited by: §2.2. [5] B. Bitterli (2016) Rendering resources. Note: https://benedikt-bitterli.me/resources/ Cited by: §7. [6] Black Forest Labs (2025) FLUX.2: analyzing and enhancing the latent space of FLUX — representation comparison. External Links: Link Cited by: §1, §3.3, §3.3. [7] Blender Foundation (2025) Blender – a 3d modelling and rendering package. Note: https://w.blender.org Cited by: §2.1. [8] C. Careaga and Y. Aksoy (2025) Physically controllable relighting of photographs. In ACM SIGGRAPH Conference Papers, External Links: Document Cited by: §2.3. [9] Z. Chen, W. Wang, Z. Yang, Z. Yuan, H. Chen, and C. Shen (2024) FreeCompose: generic zero-shot image composition with diffusion prior. In ECCV, External Links: Document Cited by: §6.6. [10] P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. (2024) Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning (ICML), Cited by: §3.3, §4. [11] T. Hachisuka, S. Ogaki, and H. W. Jensen (2008) Progressive photon mapping. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia). External Links: Document Cited by: §1, §2.1. [12] Z. He, T. Wang, X. Huang, X. Pan, and Z. Liu (2025) Neural LightRig: unlocking accurate object normal and material estimation with multi-light diffusion. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 26514–26524. External Links: Document Cited by: §2.2. [13] S. Herholz, M. Sik, L. Reichardt, and M. Manzi (2025) Path guiding in production and recent advancements. In ACM SIGGRAPH Courses, External Links: Document Cited by: §2.1. [14] A. Hertz, K. Aberman, and D. Cohen-Or (2023) Delta denoising score. In IEEE/CVF International Conference on Computer Vision (ICCV), External Links: Document Cited by: §6.6. [15] O. Hirschorn, O. Sela, I. Huberman-Spiegelglas, N. Efrat, E. Alshan, I. Ideses, F. Devernay, Y. Zvik, and L. Fritz (2026) Splatent: splatting diffusion latents for novel view synthesis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 8319–8330. External Links: 2512.09923 Cited by: §2.3. [16] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2.2. [17] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), Cited by: §2.2. [18] B. Hua, A. Gruson, V. Petitjean, M. Zwicker, D. Nowrouzezahrai, E. Eisemann, and T. Hachisuka (2019) A survey on gradient-domain rendering. Computer Graphics Forum (Proceedings of Eurographics State of the Art Reports) 38 (2), p. 455–472. External Links: Document Cited by: §2.1. [19] W. Jakob, S. Speierer, N. Roussel, and D. Vicini (2022) Dr.jit: a just-in-time compiler for differentiable rendering. ACM Transactions on Graphics (Proceedings of SIGGRAPH). External Links: Document Cited by: §5.2, §6.1. [20] H. W. Jensen (1996) Global illumination using photon maps. In Proceedings of the Eurographics Workshop on Rendering, External Links: Document Cited by: §1, §2.1. [21] J. Jeong, S. Han, J. Kim, and S. J. Kim (2025) Latent space super-resolution for higher-resolution image generation with diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 2355–2365. External Links: Document Cited by: §6.7. [22] J. T. Kajiya (1986) The rendering equation. In Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’86. External Links: Document Cited by: §1, §2.1, §6.1. [23] M. Kettunen, M. Manzi, M. Aittala, J. Lehtinen, F. Durand, and M. Zwicker (2015) Gradient-domain path tracing. ACM Transactions on Graphics (Proceedings of SIGGRAPH). External Links: Document Cited by: §2.1. [24] D. P. Kingma and J. Ba (2015) Adam: a method for stochastic optimization. In International Conference on Learning Representations (ICLR), External Links: 1412.6980 Cited by: §3.2, §6.1. [25] D. P. Kingma and M. Welling (2014) Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), Cited by: §3.3. [26] P. Kocsis, L. Höllein, and M. Nießner (2025) IntrinsiX: high-quality PBR generation using image priors. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2.2. [27] E. P. Lafortune and Y. D. Willems (1993) Bi-directional path tracing. In Proceedings of Compugraphics, Cited by: §1. [28] J. Lehtinen, T. Karras, S. Laine, M. Aittala, F. Durand, and T. Aila (2013) Gradient-domain metropolis light transport. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 32 (4), p. 95:1–95:12. External Links: Document Cited by: §2.1. [29] T. Li, M. Aittala, F. Durand, and J. Lehtinen (2018) Differentiable monte carlo ray tracing through edge sampling. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia). External Links: Document Cited by: §2.1, §3.2. [30] R. Liang, Z. Gojcic, H. Ling, J. Munkberg, J. Hasselgren, Z. Lin, J. Gao, A. Keller, N. Vijaykumar, S. Fidler, and Z. Wang (2025) DiffusionRenderer: neural inverse and forward rendering with video diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), External Links: Document Cited by: §2.3. [31] R. Liu, R. Wu, B. Van Hoorick, P. Tokmakov, S. Zakharov, and C. Vondrick (2023) Zero-1-to-3: zero-shot one image to 3d object. In IEEE/CVF International Conference on Computer Vision (ICCV), External Links: Document Cited by: §2.2. [32] L. Lyu, V. Deschaintre, Y. Hold-Geoffroy, M. Hašan, J. S. Yoon, T. Leimkühler, C. Theobalt, and I. Georgiev (2025) IntrinsicEdit: precise generative image manipulation in intrinsic space. ACM Transactions on Graphics (Proceedings of SIGGRAPH). External Links: Document Cited by: §2.2. [33] C. Meng, Y. He, Y. Song, J. Song, J. Wu, J. Zhu, and S. Ermon (2022) SDEdit: guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations (ICLR), Cited by: §2.2. [34] G. Metzer, E. Richardson, O. Patashnik, R. Giryes, and D. Cohen-Or (2023) Latent-nerf for shape-guided generation of 3d shapes and textures. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), External Links: Document Cited by: §2.3. [35] M. Nimier-David, S. Speierer, B. Ruiz, and W. Jakob (2020) Radiative backpropagation: an adjoint method for lightning-fast differentiable rendering. ACM Transactions on Graphics (Proceedings of SIGGRAPH). External Links: Document Cited by: §2.1. [36] M. Nimier-David, D. Vicini, T. Zeltner, and W. Jakob (2019) Mitsuba 2: a retargetable forward and inverse renderer. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia). External Links: Document Cited by: §2.1. [37] M. Pharr, W. Jakob, and G. Humphreys (2023) Physically based rendering: from theory to implementation. 4 edition, MIT Press. Cited by: §2.1. [38] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall (2023) DreamFusion: text-to-3d using 2d diffusion. In International Conference on Learning Representations (ICLR), Cited by: §1, Figure 9, §6.6, §6.6. [39] O. Reynolds (1903) Papers on mechanical and physical subjects: the sub-mechanics of the universe. Vol. 3, The University Press. Cited by: §3.2. [40] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer (2022) High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), External Links: Document Cited by: §1, §2.2, §3.3, §3.3. [41] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli (2015) Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning (ICML), Cited by: §1, §2.2. [42] Y. Song and S. Ermon (2019) Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2.2. [43] M. Soroka, C. Peters, and S. Marschner (2025) Quadric-based silhouette sampling for differentiable rendering. ACM Transactions on Graphics (Proceedings of SIGGRAPH). External Links: Document Cited by: §2.1. [44] K. Turner (2022) Decoding latents to RGB without upscaling. Note: https://discuss.huggingface.co/t/decoding-latents-torgb-without-upscaling/23204Hugging Face Forums Cited by: §4. [45] E. Veach and L. J. Guibas (1995) Bidirectional estimators for light transport. In Photorealistic Rendering Techniques, External Links: Document Cited by: §1. [46] E. Veach and L. J. Guibas (1997) Metropolis light transport. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’97. External Links: Document Cited by: §1, §2.1. [47] E. Veach (1997) Robust monte carlo methods for light transport simulation. Ph.D. Thesis, Stanford University. Cited by: §3.1. [48] D. Vicini, S. Speierer, and W. Jakob (2021) Path replay backpropagation: differentiating light paths using constant memory and linear time. ACM Transactions on Graphics (Proceedings of SIGGRAPH). External Links: Document Cited by: §2.1, §3.2, §6.1. [49] T. Wan, A. Wang, B. Ai, B. Wen, C. Mao, C. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, J. Zeng, J. Wang, J. Zhang, J. Zhou, J. Wang, J. Chen, K. Zhu, K. Zhao, K. Yan, L. Huang, M. Feng, N. Zhang, P. Li, P. Wu, R. Chu, R. Feng, S. Zhang, S. Sun, T. Fang, T. Wang, T. Gui, T. Weng, T. Shen, W. Lin, W. Wang, W. Wang, W. Zhou, W. Wang, W. Shen, W. Yu, X. Shi, X. Huang, X. Xu, Y. Kou, Y. Lv, Y. Li, Y. Liu, Y. Wang, Y. Zhang, Y. Huang, Y. Li, Y. Wu, Y. Liu, Y. Pan, Y. Zheng, Y. Hong, Y. Shi, Y. Feng, Z. Jiang, Z. Han, Z. Wu, and Z. Liu (2025) Wan: open and advanced large-scale video generative models. External Links: 2503.20314 Cited by: §1, §2.2. [50] R. Wang, S. Xu, C. Dai, J. Xiang, Y. Deng, X. Tong, and J. Yang (2025) MoGe: unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), External Links: Document Cited by: §2.3, §6.2. [51] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli (2004) Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13 (4), p. 600–612. External Links: Document Cited by: §6.2. [52] C. Wyman, M. Kettunen, D. Lin, B. Bitterli, C. Yuksel, W. Jarosz, P. Kozlowski, and G. De Francesco (2023) A gentle introduction to ReSTIR: path reuse in real-time. In ACM SIGGRAPH Courses, External Links: Document Cited by: §2.1. [53] J. Xiang, X. Chen, S. Xu, R. Wang, Z. Lv, Y. Deng, H. Zhu, Y. Dong, H. Zhao, N. J. Yuan, and J. Yang (2025) Native and compact structured latents for 3d generation. External Links: 2512.14692 Cited by: §6.2. [54] E. Xie, J. Chen, J. Chen, H. Cai, H. Tang, Y. Lin, Z. Zhang, M. Li, L. Zhu, Y. Lu, and S. Han (2025) SANA: efficient high-resolution image synthesis with linear diffusion transformers. In International Conference on Learning Representations (ICLR), External Links: 2410.10629 Cited by: §1, §2.2. [55] P. Xu, S. Bangaru, T. Li, and S. Zhao (2024) Markov-chain monte carlo sampling of visibility boundaries for differentiable rendering. In ACM SIGGRAPH Asia Conference Papers, External Links: Document Cited by: §2.1. [56] D. Yan, Z. Li, L. Wu, and K. Xu (2025) Generalized unbiased reconstruction for gradient-domain rendering. ACM Trans. Graph. 44 (6). External Links: ISSN 0730-0301, Document Cited by: §2.1. [57] Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y. Yang, W. Hong, X. Zhang, G. Feng, et al. (2025) CogVideoX: text-to-video diffusion models with an expert transformer. In International Conference on Learning Representations (ICLR), External Links: 2408.06072 Cited by: §2.2. [58] C. Zeng, Y. Dong, P. Peers, H. Wu, and X. Tong (2025) RenderFormer: transformer-based neural rendering of triangle meshes with global illumination. In ACM SIGGRAPH Conference Papers, External Links: Document Cited by: §2.3. [59] Y. Zeng, G. Cai, and S. Zhao (2025) A survey on physics-based differentiable rendering. External Links: 2504.01402 Cited by: §2.1. [60] Z. Zeng, V. Deschaintre, I. Georgiev, Y. Hold-Geoffroy, Y. Hu, F. Luan, L. Yan, and M. Hašan (2024) RGB↔ : image decomposition and synthesis using material- and lighting-aware diffusion models. In ACM SIGGRAPH Conference Papers, External Links: Document Cited by: §2.2. [61] C. Zhang, B. Miller, K. Yan, I. Gkioulekas, and S. Zhao (2020) Path-space differentiable rendering. ACM Transactions on Graphics (Proceedings of SIGGRAPH). External Links: Document Cited by: §2.1. [62] L. Zhang, A. Rao, and M. Agrawala (2023) Adding conditional control to text-to-image diffusion models. In IEEE/CVF International Conference on Computer Vision (ICCV), External Links: Document Cited by: §2.2. [63] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang (2018) The unreasonable effectiveness of deep features as a perceptual metric. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), External Links: Document Cited by: §5.3. [64] Z. Zhang, N. Roussel, and W. Jakob (2023) Projective sampling for differentiable rendering of geometry. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia). External Links: Document Cited by: §2.1, §6.1. [65] J. (. Zhou, H. Gao, V. Voleti, A. Vasishta, C. Yao, M. Boss, P. Torr, C. Rupprecht, and V. Jampani (2025) Stable virtual camera: generative view synthesis with diffusion models. In IEEE/CVF International Conference on Computer Vision (ICCV), p. 12405–12414. External Links: Document Cited by: §2.2.