Paper deep dive
EndoVGGT: GNN-Enhanced Depth Estimation for Surgical 3D Reconstruction
Falong Fan, Yi Xie, Arnis Lektauers, Bo Liu, Jerzy Rozenblit
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/26/2026, 2:19:57 AM
Summary
EndoVGGT is a geometry-centric framework for surgical 3D reconstruction that addresses challenges like soft-tissue deformation and instrument occlusion. It introduces the Deformation-aware Graph Attention (DeGAT) module, which constructs dynamic feature-space semantic graphs to maintain geometric consistency across occlusions. EndoVGGT achieves significant improvements in PSNR and SSIM over state-of-the-art methods and demonstrates strong zero-shot cross-dataset generalization.
Entities (5)
Relation Signals (4)
EndoVGGT → evaluatedon → SCARED
confidence 100% · Extensive experiments on SCARED show that our method significantly improves fidelity
EndoVGGT → incorporates → DeGAT
confidence 100% · we propose EndoVGGT, a geometry-centric framework equipped with a Deformation-aware Graph Attention (DeGAT) module.
EndoVGGT → generalizesto → EndoNeRF
confidence 95% · EndoVGGT exhibits strong zero-shot cross-dataset generalization to the unseen SCARED and EndoNeRF domains
DeGAT → enhances → VGGT
confidence 90% · We integrate DeGAT into VGGT at three levels
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Accurate 3D reconstruction of deformable soft tissues is essential for surgical robotic perception. However, low-texture surfaces, specular highlights, and instrument occlusions often fragment geometric continuity, posing a challenge for existing fixed-topology approaches. To address this, we propose EndoVGGT, a geometry-centric framework equipped with a Deformation-aware Graph Attention (DeGAT) module. Rather than using static spatial neighborhoods, DeGAT dynamically constructs feature-space semantic graphs to capture long-range correlations among coherent tissue regions. This enables robust propagation of structural cues across occlusions, enforcing global consistency and improving non-rigid deformation recovery. Extensive experiments on SCARED show that our method significantly improves fidelity, increasing PSNR by 24.6% and SSIM by 9.1% over prior state-of-the-art. Crucially, EndoVGGT exhibits strong zero-shot cross-dataset generalization to the unseen SCARED and EndoNeRF domains, confirming that DeGAT learns domain-agnostic geometric priors. These results highlight the efficacy of dynamic feature-space modeling for consistent surgical 3D reconstruction.
Tags
Links
- Source: https://arxiv.org/abs/2603.24577v1
- Canonical: https://arxiv.org/abs/2603.24577v1
Trouble viewing inline? Open PDF directly →
Full Text
68,076 characters extracted from source content.
Expand or collapse full text
11institutetext: University of Arizona, Tucson, AZ, USA 11email: falongfan, yix, lektauers, boliu, jerzyr@arizona.edu EndoVGGT: GNN-Enhanced Depth Estimation for Surgical 3D Reconstruction Falong Fan Yi Xie Arnis Lektauers Bo Liu Jerzy Rozenblit Abstract Accurate 3D reconstruction of deformable soft tissues is essential for surgical robotic perception. However, low-texture surfaces, specular highlights, and instrument occlusions often fragment geometric continuity, posing a challenge for existing fixed-topology approaches. To address this, we propose EndoVGGT, a geometry-centric framework equipped with a Deformation-aware Graph Attention (DeGAT) module. Rather than using static spatial neighborhoods, DeGAT dynamically constructs feature-space semantic graphs to capture long-range correlations among coherent tissue regions. This enables robust propagation of structural cues across occlusions, enforcing global consistency and improving non-rigid deformation recovery. Extensive experiments on SCARED show that our method significantly improves fidelity, increasing PSNR by 24.6% and SSIM by 9.1% over prior state-of-the-art. Crucially, EndoVGGT exhibits strong zero-shot cross-dataset generalization to the unseen SCARED and EndoNeRF domains, confirming that DeGAT learns domain-agnostic geometric priors. These results highlight the efficacy of dynamic feature-space modeling for consistent surgical 3D reconstruction. 1 Introduction Figure 1: Visualization of DeGAT neighbor aggregation. (a–b) Visualization of neighborhood construction and feature responses in the proposed DeGAT module. ★ indicates the centroid and ∘ indicates its neighbors. The highlighted ★ aggregates informative context even across instrument boundaries, enabling robust feature refinement. (c–d) Depth estimation comparison without (c) and with (d) DeGAT. Incorporating DeGAT yields sharper boundaries and improved structural continuity for both instruments and organs, as shown in the red box. Three-dimensional reconstruction of endoscopic surgical scenes is a fundamental component of modern surgical practice, including robot-assisted surgery [28, 27] and computer-assisted surgical training [23]. By providing accurate depth information, the systems enhance navigation [13] and laparoscopic training [5], thereby increasing perception precision for downstream surgical tasks. Large Reconstruction Models (LRMs) are transformer-based feed-forward networks trained on diverse multi-scene datasets to learn scene-agnostic geometric priors, with VGGT [24] as a representative example of geometry-grounded models. Although successful in natural scenes, LRMs’ extension to surgical settings is fundamentally limited by domain shift. Existing LRMs are predominantly trained on rigid, object-centric datasets assuming static geometry and stable illumination. In contrast, surgical scenes feature intrinsic non-rigidity, soft-tissue deformation, and dynamic instrument occlusion [5, 10]. Consequently, direct deployment of general-domain models yields artifacts, including disrupted tissue topology and depth errors. In parallel, recent surgical approaches based on NeRF or Gaussian Splatting rely on per-scene optimization, which requires repeated fitting for each new case, limiting their large-scale generalization across diverse procedures. To address these limitations, we propose Endoscopic VGGT (EndoVGGT), a generalizable reconstruction framework tailored for endoscopic scenes that eliminates per-scene optimization and enables strong zero-shot cross-dataset generalization via dynamic graph construction. The EndoVGGT framework effectively bridges geometric discontinuities caused by instrument occlusions, facilitating coherent depth aggregation across spatially fragmented tissues while maintaining robustness in unseen surgical scenes. Our contributions are: • We propose a Deformation-aware Graph ATtention module (DeGAT) that constructs feature-space neighborhoods to preserve sharp depth discontinuities across deformable boundaries in Sec. 3.1, reducing LPIPS by 15.8%. • To mitigate severe domain shift in surgical scenes, we train EndoVGGT on in-domain surgical data and improve PSNR on SCARED from 14.06114.061 (VGGT) to 34.34834.348 (+144%144\% relatively), as shown in Fig. 4. • We demonstrate EndoVGGT’s zero-shot generalization. As Table 2 shows, while the baseline collapses on unseen data, our method maintains robust fidelity under extreme deformations, improving SSIM by 24.8% to 0.915. 2 Related Work Three-dimensional reconstruction and dense depth estimation are critical for surgical navigation [2], robotic assistance [7], and skill assessment [6]. While early geometric and deep-learning pipelines [20, 29] laid the foundation, recent paradigms have shifted toward implicit neural representations like NeRFs [14] and explicit 3D Gaussian Splatting (3DGS) [11]. In surgical scenes, NeRF and 3DGS adaptations [26, 12] achieve high-fidelity geometry but are fundamentally bottlenecked by dense-view requirements and computationally expensive per-scene optimization [21], severely limiting their real-time clinical viability. To address these efficiency bottlenecks, LRMs [24, 9] offer a scene-agnostic, feed-forward alternative. Supported by robust transformer-based depth estimation [19, 3], LRMs directly infer 3D structures from sparse inputs without test-time optimization. However, adapting general-domain LRMs to surgical tasks remains challenging due to continuous soft-tissue deformation and dynamic instrument occlusion [15]. Even recent surgical LRM adaptations, such as EndoLRMGS [25], still rely heavily on per-scene optimization to align geometric details. In contrast, our approach builds on the VGGT architecture [24] to achieve superior generalization and inference efficiency in dynamic surgical environments, eliminating the need for scene-specific training. 3 Methodology In the following sections, we detail the DeGAT module in Sec. 3.1, the training protocol and comprehensive objective functions in Sec. 3.2. Figure 2: Overview of the EndoVGGT framework. The proposed DeGAT module enhances the features extracted from DINOv2 [16], and camera tokens interact via both global and within-frame attention mechanisms. The depth maps are predicted using a DPT head [19], and camera poses are predicted by an MLP to reconstruct the input scene, and are constrained by a composite loss introduced in Sec. 3.2. Problem setting and geometric formulation. Given a sequence of endoscopic images ℐ=I1,…,INI=\I_1,…,I_N\, our goal is to reconstruct the 3D scene geometry by predicting depth maps ^=Dt^t=1N D= \D_t\_t=1^N and camera parameters ^=^t,^t,f^tt=1N P=\ R_t, T_t, f_t\_t=1^N in a feed-forward manner without test-time optimization. We learn a mapping function ℱθ:ℐ→×F_θ:I ×P parameterized by θ. Dense depth serves as a direct proxy for 3D surface recovery through geometric back-projection. For a pixel =(u,v)⊤u=(u,v) with homogeneous coordinates ~=(u,v,1)⊤ u=(u,v,1) , the predicted depth D^t() D_t(u) induces a 3D point in the camera coordinate system, which is subsequently transformed to a world reference frame using the predicted camera pose (^t,^t)( R_t, T_t): ^t()=D^t()(f^t)−1~,^w()=^t^t()+^t, X_t(u)= D_t(u)\,K( f_t)^-1 u, X_w(u)= R_t\, X_t(u)+ T_t, (1) where (f^t)K( f_t) denotes the intrinsic matrix formed by the focal length f^t f_t. Thus, the set ^w()\ X_w(u)\ forms a dense point cloud to constitute the reconstructed geometry. 3.1 Deformation Graph Attention Module To promote local geometric consistency, we propose DeGAT. Unlike fixed spatial neighborhoods that assume a locally stable structure, DeGAT constructs dynamic neighbors that adapt to non-rigid, deformable surgical inputs. Dynamic Graph Construction. For each frame t, we denote the patch-token features by t=[t,1,…,t,L]⊤∈ℝL×CX_t=[x_t,1,…,x_t,L] ^L× C, where L=HPWPL= HP WP is the number of tokens, and each token i is associated with a normalized spatial coordinate t,i∈ℝ2p_t,i ^2. We construct a dynamic graph by computing the semantic cosine similarity sij=(t,i⊤t,j)/(‖t,i‖2‖t,j‖2)s_ij=(x_t,i x_t,j)/(\|x_t,i\|_2\|x_t,j\|_2) to determine the K-nearest neighbor set (i)=TopKj≠i(sij)N(i)=TopK_j≠ i(s_ij), where K=9K=9 empirically. This dynamic neighborhood ensures that the model connects the same tissue surface even when the tissue is spatially fragmented by deformation or instrument occlusion, as shown in Fig. 1. Attention-Based Aggregation. Following the GAT framework [22], DeGAT aggregates information from (i)N(i) via attention mechanisms (without introducing additional handcrafted scalar bias terms). Specifically, the attention logit is: ℓij=⊤LeakyReLU(proj[t,i∥t,j]), _ij=a LeakyReLU\! (W_proj\,[x_t,i\|x_t,j] ), (2) The attention coefficient αij _ij and the aggregated token feature t,ioutx^out_t,i are: αij=exp(ℓij)∑m∈(i)exp(ℓim),t,iout=t,i+σ(∑j∈(i)αijvalt,j). _ij= ( _ij) _m (i) ( _im), ^out_t,i=x_t,i+σ\! ( _j (i) _ij\,W_valx_t,j ). (3) By defining ~∈ℝL×L A ^L× L as a sparse matrix where ~ij=αij A_ij= _ij for j∈(i)j (i) and 0 otherwise, DeGAT can be formulated compactly as tout=t+σ(~tval⊤),X^out_t=X_t+σ\! ( A\,X_t\,W_val ), (4) where σ represents the ELU [4] activation to improve learning dynamics. The following propositions are established, with proofs in an anonymous Appendix. Proposition 1: Stability. For each token i, the DeGAT residual update satisfies ‖t,iout‖2≤‖t,i‖2+maxj∈(i)‖valt,j‖2\|x_t,i^out\|_2≤\|x_t,i\|_2+ _j (i)\|W_valx_t,j\|_2, ensuring training stability. Proof. The ELU function σ is non-expansive (|σ(z)|≤|z|)(|σ(z)|≤|z|), ensuring ∥t,iagg∥2=∥σ(i)∥2≤∥i∥2 _t,i^agg _2= σ(m_i) _2≤ _i _2. By definition, i=∑j∈(i)αijvalt,jm_i= _j (i) _ijW_valx_t,j. Using the triangle inequality and the row-stochasticity of attention weights (∑j∈(i)αij=1)( _j (i) _ij=1), we directly obtain ∥i∥2≤maxj∈(i)∥valt,j∥2 _i _2≤ _j (i) _valx_t,j _2. Finally, applying the triangle inequality to the residual update t,iout=t,i+t,iaggx_t,i^out=x_t,i+x_t,i^agg yields Proposition 1. ■ Proposition 2: Permutation Equivariance. DeGAT is permutation-equivariant: DeGAT(,)=DeGAT(,)DeGAT(PX,Pp)=PDeGAT(X,p) for any permutation matrix P. We integrate DeGAT into VGGT at three levels (Groups (B)–(D) in Table 1). At the token level, we pool the DeGAT-refined patch features into a global geometric prior i=1||∑j∈(i)jg_i= 1|N| _j (i)x_j. This prior is injected into the camera token c via a learnable bias (′=+MLP()c =c+MLP(g)), FiLM [18] (′=(+)⊙+c =(1+ γ) + β with [,]=MLP()[ γ, β]=MLP(g)), or cross-attention (′=+CrossAttn(,)c =c+CrossAttn\! (c,X )). At the attention level, we reuse DeGAT affinities to build a learnable bias matrix ∈ℝL×LB ^L× L, modifying self-attention to Attn(,,)=softmax(⊤d+)Attn(Q,K,V)=softmax\! ( QK d+B )V, where d is the head dimension. B is derived by either quantizing neighbor distances (Learnable Bias Table) or mapping them via a lightweight network (Continuous MLP Bias). Finally, at the feature level, DeGAT acts as a residual operator (←outX _out) to refine token representations, enhancing geometry-aware modeling under complex deformations. Details are in the Appendix. 3.2 Training Objective Base Rigid Supervision. We adopt the supervision paradigm from VGGT [24] as our objective, ℒbaseL_base, which optimizes camera parameters and enforces geometric fidelity. Let ^,^,f T, R, f represent the predicted translation, rotation, and focal length, with ,,fT,R,f as their ground truth. The camera loss is computed as ℒcam=‖^−‖1+‖^−‖1+|f^−f|L_cam=\| T-T\|_1+\| R-R\|_1+| f-f|. For depth supervision, following VGGT, we predict an auxiliary per-pixel confidence map C^∈ℝ>0H×W C _>0^H× W which is an extra output channel of the depth head, and learn it implicitly through the training objective (details are in Appendix). The complete depth loss integrates standard regression, confidence-weighted uncertainty, and spatial gradient consistency: ℒdepth=‖D^−D‖22⏟ℒreg+(γ‖D^−D‖22⊙C^−αlog(C^))⏟ℒunc+∑k∈x,y‖∇kD^−∇kD‖1⏟ℒgrad,L_depth= \| D-D\|_2^2_L_reg+ (γ\,\| D-D\|_2^2 C-α\, ( C) )_L_unc+ _k∈\x,y\\| _k D- _kD\|_1_L_grad, (5) where ⊙ denotes element-wise multiplication, and α,γα,γ are weighting hyperparameters, and ∇k _k denotes the spatial gradient in the x and y directions. The base objective is thus defined as ℒbase=ℒcam+ℒdepthL_base=L_cam+L_depth. Proposition 3 (Optimal Confidence). The uncertainty objective ℒuncL_unc is convex in (0,∞)(0,∞), achieving its unique minimum at C^=α/(γ‖D^()−D()‖22) C^=α/(γ\| D(u)-D(u)\|_2^2), enabling implicit confidence learning without labels. Proof is in the Appendix. 4 Experiments In this section, we present the experimental results for EndoVGGT. Sec. 4.1 describes the datasets and metrics. Sec. 4.2 evaluates the effectiveness and zero-shot generalization of the DeGAT module, highlighting its handling of non-rigid scenarios. Sec. 4.3 compares our method with prior approaches. 4.1 Datasets, Baselines, and Evaluation Metrics Datasets. We evaluate our method on three open benchmarks: EndoSLAM [17] provides ex-vivo and synthetic sequences with accurate 6-DoF poses. SCARED [1] offers realistic surgical data from the da Vinci Xi system. We also use the “cutting” and “pulling” subsets of EndoNeRF [26] to assess reconstruction robustness under topological changes and tissue deformation. Metrics. For quantitative evaluation, we adopt photometric measures following [26], including PSNR, SSIM, and LPIPS, to compare methods’ performance between rendered novel views and ground truth images. Baseline. We compare EndoVGGT with general domain VGGT [24], NeRF-based EndoSurf [30] and LRM-based EndoLRMGS [25] with Gaussian Splatting. 4.2 Effectiveness of DeGAT Mechanism and Visualization Table 1: Experiment results on different-level DeGAT strategies on SCARED dataset. “EndoVGGT-base” refers to the EndoVGGT model without DeGAT. Methods PSNR ↑ SSIM ↑ LPIPS ↓ (A) Baseline EndoSurf [30] 24.395 0.769 0.319 EndoLRMGS [25] 27.561 0.861 0.323 EndoVGGT-base 32.927 0.918 0.285 (B) EndoVGGT + Token-Level DeGAT cls token+learnable bias [Sec 3.1] 34.019 0.936 0.246 cls token + FiLM module [18] 31.941 0.869 0.258 cls token + cross attention 32.016 0.871 0.257 (C) EndoVGGT + Attention-Level DeGAT Learnable Bias Table 32.506 0.921 0.280 Continuous MLP Bias 32.894 0.923 0.247 (D) EndoVGGT + Feature-Level DeGAT Post-Transformer DeGAT 31.738 0.925 0.240 -Transformer DeGAT 34.348 0.939 0.240 To investigate the effectiveness of incorporating geometric information, we conduct an extensive study on the SCARED dataset. As presented in Table 1, the feature-level DeGAT applied before the transformer block consistently yields the best reconstruction performance across all evaluated metrics. Compared with the baseline, this configuration yields a notable 15.8%15.8\% reduction in LPIPS (from 0.2850.285 to 0.2400.240), a 4.3%4.3\% increase in PSNR (from 32.92732.927 to 34.34834.348), and a 2.3%2.3\% increase in SSIM (from 0.9180.918 to 0.9390.939). Token- and attention-level variants yield only marginal gains or even degrade performance, suggesting that aggregating feature-space neighborhoods before global attention helps establish stable local geometry under tissue deformation and instrument–tissue discontinuities. In contrast, late-stage graph aggregation can interfere with higher-level semantics. Similar trends on EndoNeRF are reported in the Appendix. Qualitative results in Fig. 4 mirror these findings. While token-level enhancement sharpens organ boundaries, it suffers from fragmented instrument edges. The attention-level variant captures more tool boundaries but lacks surface fidelity. In contrast, feature-level DeGAT in Fig. 4(d) achieves superior structural continuity, yielding more coherent instrument regions and sharper delineation than Fig. 4(b) and (c). This confirms that feature-space aggregation before global attention effectively preserves complex topological structures under surgical deformation. Fig. 1(a–b) further illustrates robustness to dynamic structural changes: the green query (★ ) bridges the surgical instrument to retrieve relevant neighbors across it, indicating that DeGAT leverages learned context rather than Euclidean proximity to handle tool-induced topological breaks. As shown in Fig. 1(c–d), our method yields smoother non-rigid surfaces and sharper boundaries in the highlighted region, whereas removing DeGAT leads to blurred details and artifacts. Figure 3: Experiment results on EndoNeRF and SCARED dataset. “Average” denotes the mean performance across all evaluated subsets. Figure 4: Visualization of DeGAT at different levels. The red boxes highlight complex instrument-tissue boundaries. Feature-level DeGAT (d) preserves sharper continuity. 4.3 Quantitative Comparisons of EndoVGGT Compared to the zero-shot VGGT baseline, fine-tuning on surgical data yields substantial improvements across all metrics in Fig. 4. Additionally, the bars show that integrating DeGAT consistently outperforms the variant without it. Specifically, on the SCARED dataset, EndoVGGT improves PSNR by 144% to 34.348 and SSIM by 3.7× to 0.939. On EndoNeRF, LPIPS drops by 75% on the “cutting” subset. This demonstrates that while general-domain models trained on rigid objects struggle in complex surgical environments, EndoVGGT effectively recovers high-fidelity geometric and photometric details in deformable scenes. Table 2: Cross-dataset evaluation. The models are zero-shot evaluated on the unseen SCARED and EndoNeRF datasets. The best results are highlighted in bold. Dataset EndoSurf EndoVGGT (Ours) PSNR ↑ SSIM ↑ LPIPS ↓ PSNR ↑ SSIM ↑ LPIPS ↓ SCARED-d1k1 10.672 0.631 0.425 27.945 0.828 0.396 SCARED-d2k1 10.846 0.725 0.322 31.741 0.876 0.229 SCARED-d3k1 10.307 0.535 0.498 20.899 0.854 0.345 EndoNeRF-pulling 16.035 0.770 0.346 22.013 0.907 0.154 EndoNeRF-cutting 15.513 0.733 0.387 23.584 0.915 0.242 4.4 Zero-Shot Cross-Dataset Generalization of EndoVGGT Unlike conventional NeRF and Gaussian Splatting methods that rely on per-scene optimization, EndoVGGT demonstrates exceptional zero-shot generalization. We trained the models on the EndoSLAM dataset and evaluated them on unseen SCARED and EndoNeRF datasets without test-time fine-tuning. As shown in Table 2, our method outperforms the EndoSurf baseline, achieving PSNRs near 30 across SCARED subsets. Notably, EndoVGGT achieves high SSIMs of 0.907 and 0.915 even under extreme soft-tissue deformations in EndoNeRF pulling and cutting scenes. These results suggest our DeGAT module captures intrinsic, domain-agnostic geometric priors rather than overfitting to scene-specific textures. 5 Conclusion We present EndoVGGT, a generalizable framework for 3D reconstruction in minimally invasive surgery that models tissue deformation and dynamic occlusion. We bridge the gap between rigid-scene assumptions and the non-rigid surgical environment via an LRM. Our key module, DeGAT, dynamically builds semantic graphs in feature space to restore instrument-induced topological breaks and enforce depth consistency. Experiments on SCARED demonstrate robustness with future extensions toward temporal consistency and robotic navigation. References [1] M. Allan, J. Mcleod, C. Wang, J. C. Rosenthal, Z. Hu, N. Gard, P. Eisert, K. X. Fu, T. Zeffiro, W. Xia, et al. (2021) Stereo correspondence and reconstruction of endoscopic data challenge. arXiv preprint arXiv:2101.01133. Cited by: §11.7, §4.1. [2] R. A. Bartholomew, H. Zhou, M. Boreel, K. Suresh, S. Gupta, M. B. Mitchell, C. Hong, S. E. Lee, T. R. Smith, J. P. Guenette, et al. (2024) Surgical navigation in the anterior skull base using 3-dimensional endoscopy and surface reconstruction. JAMA Otolaryngology–Head & Neck Surgery 150 (4), p. 318–326. Cited by: §2. [3] S. F. Bhat, R. Birkl, D. Wofk, P. Wonka, and M. Müller (2023) Zoedepth: zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288. Cited by: §2. [4] D. Clevert, T. Unterthiner, and S. Hochreiter (2015) Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 4 (5), p. 11. Cited by: §3.1. [5] B. Cui, M. Islam, L. Bai, and H. Ren (2024) Surgical-dino: adapter learning of foundation models for depth estimation in endoscopic surgery. International Journal of Computer Assisted Radiology and Surgery 19 (6), p. 1013–1020. Cited by: §1, §1. [6] I. Funke, S. T. Mees, J. Weitz, and S. Speidel (2019) Video-based surgical skill assessment using 3d convolutional neural networks. International journal of computer assisted radiology and surgery 14 (7), p. 1217–1225. Cited by: §2. [7] B. Göbel, J. Huurdeman, A. Reiterer, and K. Möller (2025) Robot-based procedure for 3d reconstruction of abdominal organs using the iterative closest point and pose graph algorithms. Journal of imaging 11 (2), p. 44. Cited by: §2. [8] K. Han, Y. Wang, J. Guo, Y. Tang, and E. Wu (2022) Vision gnn: an image is worth graph of nodes. Advances in neural information processing systems 35, p. 8291–8303. Cited by: §12.1. [9] Z. He and T. Wang (2023) Openlrm: open-source large reconstruction models. Cited by: §2. [10] Y. Hirohata, M. Sogabe, T. Miyazaki, T. Kawase, and K. Kawashima (2023) Confidence-aware self-supervised learning for dense monocular depth estimation in dynamic laparoscopic scene. Scientific Reports 13 (1), p. 15380. Cited by: §1. [11] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis (2023) 3D gaussian splatting for real-time radiance field rendering.. ACM Trans. Graph. 42 (4), p. 139–1. Cited by: §2. [12] Y. Liu, C. Li, C. Yang, and Y. Yuan (2024) Endogaussian: real-time gaussian splatting for dynamic endoscopic scene reconstruction. arXiv preprint arXiv:2401.12561. Cited by: §2. [13] Y. Lu, R. Wei, B. Li, W. Chen, J. Zhou, Q. Dou, D. Sun, and Y. Liu (2023) Autonomous intelligent navigation for flexible endoscopy using monocular depth guidance and 3-d shape planning. arXiv preprint arXiv:2302.13219. Cited by: §1. [14] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng (2021) Nerf: representing scenes as neural radiance fields for view synthesis. Communications of the ACM 65 (1), p. 99–106. Cited by: §2. [15] P. Mountney and G. Yang (2009) Dynamic view expansion for minimally invasive surgery using simultaneous localization and mapping. In 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, p. 1184–1187. Cited by: §2. [16] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. (2023) Dinov2: learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. Cited by: Figure 2, Figure 2. [17] K. B. Ozyoruk, G. I. Gokceler, T. L. Bobrow, G. Coskun, K. Incetan, Y. Almalioglu, F. Mahmood, E. Curto, L. Perdigoto, M. Oliveira, et al. (2021) EndoSLAM dataset and an unsupervised monocular visual odometry and depth estimation approach for endoscopic videos. Medical image analysis 71, p. 102058. Cited by: §11.7, §4.1. [18] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville (2018) Film: visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32. Cited by: §11.5, §3.1, Table 1. [19] R. Ranftl, A. Bochkovskiy, and V. Koltun (2021) Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference on computer vision, p. 12179–12188. Cited by: §2, Figure 2, Figure 2. [20] J. L. Schonberger and J. Frahm (2016) Structure-from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 4104–4113. Cited by: §2. [21] C. Sun, M. Sun, and H. Chen (2022) Direct voxel grid optimization: super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 5459–5469. Cited by: §2. [22] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio (2017) Graph attention networks. arXiv preprint arXiv:1710.10903. Cited by: §3.1. [23] A. Wagner and J. W. Rozenblit (2017) Augmented reality visual guidance for spatial perception in the computer assisted surgical trainer. In Proceedings of the Symposium on Modeling and Simulation in Medicine, p. 1–12. Cited by: §1. [24] J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, and D. Novotny (2025) Vggt: visual geometry grounded transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 5294–5306. Cited by: §1, §2, §3.2, §4.1. [25] X. Wang, S. Zhang, B. Huang, D. Stoyanov, and E. B. Mazomenos (2025) EndoLRMGS: complete endoscopic scene reconstruction combining large reconstruction modelling and gaussian splatting. arXiv preprint arXiv:2503.22437. Cited by: §2, §4.1, Table 1. [26] Y. Wang, Y. Long, S. H. Fan, and Q. Dou (2022) Neural rendering for stereo 3d reconstruction of deformable tissues in robotic surgery. In International conference on medical image computing and computer-assisted intervention, p. 431–441. Cited by: §11.7, §2, §4.1, §4.1. [27] R. Wei, J. Guo, Y. Lu, F. Zhong, Y. Liu, D. Sun, and Q. Dou (2024) Scale-aware monocular reconstruction via robot kinematics and visual data in neural radiance fields. Artificial Intelligence Surgery 4 (3), p. 187–198. Cited by: §1. [28] M. Xu, Z. Guo, A. Wang, L. Bai, and H. Ren (2024) A review of 3d reconstruction techniques for deformable tissues in robotic surgery. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 157–167. Cited by: §1. [29] Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan (2018) Mvsnet: depth inference for unstructured multi-view stereo. In Proceedings of the European conference on computer vision (ECCV), p. 767–783. Cited by: §2. [30] R. Zha, X. Cheng, H. Li, M. Harandi, and Z. Ge (2023) Endosurf: neural surface reconstruction of deformable tissues with stereo endoscope videos. In International conference on medical image computing and computer-assisted intervention, p. 13–23. Cited by: §4.1, Table 1. 6 Appendix 7 Theoretical Properties of DeGAT This appendix formalizes several properties of DeGAT that support its use as a geometry-consistent feature refinement operator under deformation and occlusion. 7.1 Row-stochastic aggregation and convexity Lemma 1(Row-stochasticity of DeGAT attention) For each node i, the attention coefficients αijj∈(i)\ _ij\_j (i) defined in Eq. (3) satisfy: (i) αij≥0 _ij≥ 0 for all j∈(i)j (i); and (i) ∑j∈(i)αij=1 _j (i) _ij=1. Equivalently, the induced sparse matrix ~ A in Eq. (4) is row-stochastic on the neighbor support. Proof Fix any node i. Since we select Top-K neighbors with K≥1K≥ 1, the neighbor set (i)N(i) is non-empty. Define the normalizer (partition function) Zi=∑m∈(i)exp(ℓim).Z_i\;=\; _m (i) ( _im). Because exp(⋅)>0 (·)>0 for any real input and (i)≠∅N(i)≠ , we have Zi>0Z_i>0. By Eq. (3), αij=exp(ℓij)Zi,j∈(i). _ij= ( _ij)Z_i, j (i). Hence αij≥0 _ij≥ 0 since both numerator and denominator are positive. Summing over j∈(i)j (i) yields ∑j∈(i)αij=1Zi∑j∈(i)exp(ℓij)=ZiZi=1. _j (i) _ij= 1Z_i _j (i) ( _ij)= Z_iZ_i=1. For the matrix statement, recall ~ij=αij A_ij= _ij if j∈(i)j (i) and 0 otherwise. Therefore ∑j=1L~ij=∑j∈(i)αij=1, _j=1^L A_ij= _j (i) _ij=1, and ~ij≥0 A_ij≥ 0, i.e., ~ A is row-stochastic on the neighbor support. Corollary 1(Convex-hull property of the aggregated message) Let j=valt,jv_j=W_valx_t,j and define the pre-activation message i=∑j∈(i)αijjm_i= _j (i) _ijv_j. Then im_i lies in the convex hull of jj∈(i)\v_j\_j (i). Proof By definition, the convex hull of jj∈(i)\v_j\_j (i) is the set conv(j)=∑j∈(i)wjj|wj≥0,∑j∈(i)wj=1.conv (\v_j\ )= \ _j (i)w_jv_j\; |\;w_j≥ 0,\ _j (i)w_j=1 \. Lemma 1 shows αij≥0 _ij≥ 0 and ∑j∈(i)αij=1 _j (i) _ij=1. Thus i=∑j∈(i)αijjm_i= _j (i) _ijv_j is exactly a convex combination of neighbor vectors, i.e., i∈conv(j)m_i (\v_j\). Corollary 2(Coordinate-wise bounds (min–max property)) For any coordinate (channel) index c, the c-th component of the message satisfies minj∈(i)(j)c≤(i)c≤maxj∈(i)(j)c. _j (i)(v_j)_c\;≤\;(m_i)_c\;≤\; _j (i)(v_j)_c. Proof For any channel c, (i)c=∑j∈(i)αij(j)c(m_i)_c= _j (i) _ij(v_j)_c is a weighted average of (j)c\(v_j)_c\ with nonnegative weights summing to one (Lemma 1). A weighted average must lie between the minimum and maximum of the values being averaged; hence the claim. 7.2 Stability bounds (why message passing will not explode) Proposition 1(Norm bound of one-hop DeGAT aggregation) Let t,iagg=ELU(i)x^agg_t,i=ELU(m_i) be the post-activation aggregated message used in Eq. (3). Then for each node i, ‖t,iagg‖2≤‖i‖2≤maxj∈(i)‖valt,j‖2.\|x^agg_t,i\|_2≤\|m_i\|_2≤ _j (i)\|W_valx_t,j\|_2. (6) Consequently, the residual update satisfies ‖t,iout‖2≤‖t,i‖2+maxj∈(i)‖valt,j‖2.\|x^out_t,i\|_2≤\|x_t,i\|_2+ _j (i)\|W_valx_t,j\|_2. (7) Proof Step 1 (ELU is non-expansive around zero). Consider the scalar ELU function: ELU(z)=zELU(z)=z if z≥0z≥ 0, and ELU(z)=ez−1ELU(z)=e^z-1 if z<0z<0. For z≥0z≥ 0, |ELU(z)|=|z||ELU(z)|=|z|. For z<0z<0, let a=−z>0a=-z>0. Then |ELU(z)|=|ez−1|=1−e−a≤a=|z||ELU(z)|=|e^z-1|=1-e^-a≤ a=|z| (since 1−e−a≤a1-e^-a≤ a for all a>0a>0). Hence for any scalar z, |ELU(z)|≤|z||ELU(z)|≤|z|. Applying this element-wise gives ‖ELU(i)‖22=∑c|ELU((i)c)|2≤∑c|(i)c|2=‖i‖22,\|ELU(m_i)\|_2^2= _c|ELU((m_i)_c)|^2≤ _c|(m_i)_c|^2=\|m_i\|_2^2, i.e., ‖t,iagg‖2≤‖i‖2\|x^agg_t,i\|_2≤\|m_i\|_2. Step 2 (bounding the pre-activation message). By definition, i=∑j∈(i)αijvalt,j.m_i= _j (i) _ij\,W_valx_t,j. Using the triangle inequality and Lemma 1, ‖i‖2≤∑j∈(i)αij‖valt,j‖2≤(maxj∈(i)‖valt,j‖2)∑j∈(i)αij=maxj∈(i)‖valt,j‖2.\|m_i\|_2≤ _j (i) _ij\,\|W_valx_t,j\|_2≤ ( _j (i)\|W_valx_t,j\|_2 ) _j (i) _ij= _j (i)\|W_valx_t,j\|_2. Step 3 (residual bound). Finally, Eq. (3) gives t,iout=t,i+t,iaggx^out_t,i=x_t,i+x^agg_t,i, hence ‖t,iout‖2≤‖t,i‖2+‖t,iagg‖2≤‖t,i‖2+maxj∈(i)‖valt,j‖2.\|x^out_t,i\|_2≤\|x_t,i\|_2+\|x^agg_t,i\|_2≤\|x_t,i\|_2+ _j (i)\|W_valx_t,j\|_2. Remark. Lemma 1 and Corollary 1 show that DeGAT forms a controlled convex mixture of neighbor information (before the value projection and activation), and Proposition 1 provides an explicit stability bound for one-hop message passing. 7.3 Permutation equivariance (token indexing should not matter) Proposition 2(Permutation equivariance of one-hop DeGAT) Let π be a permutation of token indices and P the corresponding permutation matrix. If token features and coordinates are permuted consistently, ′=X =PX and π(i)′=ip _π(i)=p_i, then the DeGAT output permutes in the same way: DeGAT(′,′)=DeGAT(,),DeGAT(X ,p )=P\,DeGAT(X,p), (8) assuming deterministic tie-breaking in the Top-K operator. Proof Let S denote the cosine-similarity matrix with entries sijs_ij computed from feature pairs. Under permutation, ′=X =PX simply reorders tokens, hence the similarity matrix is permuted as S′=S⊤,i.e.,sπ(i)π(j)′=sij.S =PSP , .e., s _π(i)\,π(j)=s_ij. With deterministic tie-breaking, applying Top-K to each row of S′S yields a permuted neighbor mapping: ′(π(i))=π(j)∣j∈(i).N (π(i))=\π(j) j (i)\. Next, the attention logit ℓij _ij (Eq. (2)) is computed by the same shared parameters on each ordered pair of tokens. Therefore, logits permute consistently: ℓπ(i)π(j)′=ℓij _π(i)\,π(j)= _ij for j∈(i)j (i), and consequently απ(i)π(j)′=αijα _π(i)\,π(j)= _ij by the softmax definition in Eq. (3). Finally, the DeGAT aggregation is a sum over neighbors: t,π(i)out′=t,π(i)′+ELU(∑j′∈′(π(i))απ(i)j′valt,j′).x^out _t,π(i)=x _t,π(i)+ELU\! ( _j (π(i))α _π(i)j W_valx _t,j ). Substituting t,π(i)′=t,ix _t,π(i)=x_t,i, j′=π(j)j =π(j), ′(π(i))=π(j)N (π(i))=\π(j)\, and απ(i)π(j)′=αijα _π(i)π(j)= _ij shows that the right-hand side equals t,ioutx^out_t,i up to the same permutation. Thus, DeGAT(′,′)=DeGAT(,)DeGAT(X ,p )=P\,DeGAT(X,p). 8 Implementation Details of DeGAT For each frame t, DeGAT constructs a directed K-N graph by computing the normalized dot-product cosine similarity across token features, applying a TopK operation while excluding self-matching. To maintain a linear message-passing complexity (LK)O(LK) (where L is the token count and K is the neighborhood size), the attention coefficients αij _ij are evaluated exclusively for j∈(i)j (i) and conceptually stored as a sparse matrix ~ A. To prevent numerical underflow in the attention-level variant (Eq. (LABEL:eq:attn_level)), we clamp αij _ij to a minimal positive value before applying the logarithmic transformation Bij=log(αij)B_ij= ( _ij). Operationally, a single-hop DeGAT layer executes the following sequence: (1) token feature normalization; (2) cosine similarity computation and Top-K neighbor retrieval; (3) calculation of attention logits ℓij _ij and weights αij _ij; (4) feature aggregation across the dynamic neighborhood; and (5) residual addition to produce the updated features toutX^out_t. 9 DeGAT Integration Variants This section elaborates on the three DeGAT integration levels introduced in Sec. 3.1. For token-level conditioning, we summarize the DeGAT-refined features into a pooled geometric prior t=1L∑i=1Lt,ioutg_t= 1L _i=1^Lx^out_t,i, which is then mapped via a lightweight, fixed-architecture MLP to additively modulate the camera token (t′=t+MLP(t)c _t=c_t+MLP(g_t)). For attention-level bias injection, we construct a sparse bias matrix B from the DeGAT affinities, where Bij=MLP(αij)B_ij=MLP( _ij) for j∈(i)j (i) and 0 otherwise. This matrix is directly added to the transformer attention logits (Eq. (LABEL:eq:attn_level)) and broadcasted across all attention heads, ensuring a parameter-free integration. Finally, for feature-level refinement, DeGAT functions as a residual graph operator on the patch tokens (t←toutX_t ^out_t) immediately preceding the transformer blocks, thereby enriching the representations with geometry-aware context before global multi-view reasoning. 10 Derivation of Uncertainty-Weighted Regression This appendix provides a self-contained justification for the confidence term ℒuncL_unc in Eq. (5). In particular, the confidence map C C is learned implicitly and does not require ground-truth supervision. Proposition 3(Closed-form optimal confidence for a fixed residual) For a single pixel, let r2=‖D^()−D()‖22r^2=\| D(u)-D(u)\|_2^2 be the squared depth residual and assume r2>0r^2>0. Consider the confidence objective (C^)=γr2C^−αlog(C^),with C^>0.J( C)=γ\,r^2\, C-α\, ( C), C>0. Then J is strictly convex in C C and achieves its unique minimum at C^⋆=αγr2. C = αγ\,r^2. Proof Step 1 (existence of a minimizer). For r2>0r^2>0 and C^>0 C>0, we have γr2C^→+∞γ r^2 C→+∞ as C^→+∞ C→+∞. Meanwhile, −log(C^)→+∞- ( C)→+∞ as C^→0+ C→ 0^+. Therefore, (C^)→+∞J( C)→+∞ at both boundaries of (0,∞)(0,∞), implying J attains a minimum at some finite C^⋆∈(0,∞) C ∈(0,∞). Step 2 (stationary point). Compute the derivative: ddC^=γr2−αC^. dJd C=γ r^2- α C. Setting it to zero yields the unique stationary point C^⋆=α/(γr2) C =α/(γ r^2). Step 3 (strict convexity and uniqueness). The second derivative is d2dC^2=αC^2>0for all C^>0, d^2Jd C^2= α C^2>0 all C>0, hence J is strictly convex, and the stationary point is the unique global minimizer. Corollary 3(Equivalent marginal penalty after eliminating confidence) For r2>0r^2>0, substituting C^⋆=α/(γr2) C =α/(γ r^2) into J yields minC^>0(C^)=α−αlog(αγr2)=αlog(γr2)+const, _ C>0J( C)=α-α \! ( αγ r^2 )=α (γ r^2)+const, where “const” is independent of the model outputs. Proof Direct substitution gives (C^⋆)=γr2⋅α/(γr2)−αlog(α/(γr2))J( C )=γ r^2·α/(γ r^2)-α (α/(γ r^2)). Rearranging yields the stated form. Remark. The optimal confidence is inversely proportional to the squared residual: pixels with larger errors are automatically down-weighted. The log term acts as a barrier preventing the trivial solution C^→0 C→ 0. In the degenerate case r2=0r^2=0, the objective favors increasing confidence, which aligns with the interpretation that perfectly predicted pixels should be assigned high confidence; in practice, network outputs are bounded and numerically stabilized during training. 11 Experiment Details 11.1 Feature-Level DeGAT Implementation Details Input. • Input feature tensor ∈ℝB×N×CX ^B× N× C from the VGGT Aggregator. • Number of neighbors K (e.g., K=10K=10). • Trainable parameters: – Projection matrix proj∈ℝ2C×C′W_proj ^2C× C , – Attention vector ∈ℝC′a ^C , – Value transformation matrix val∈ℝC×CW_val ^C× C. Output. Refined feature tensor out∈ℝB×N×CX_out ^B× N× C. Procedure. Given a batch index b∈1,…,Bb∈\1,…,B\, the refinement process is defined as follows. The set of trainable parameters is denoted as Θ=proj,,val =\W_proj,a,W_val\. 1. Initialization: out←.X_out 0. 2. Graph Construction: • Compute the pairwise Euclidean distance matrix: ij(b)=‖i−j‖2,(b)∈ℝN×N.D^(b)_ij= \|x_i-x_j \|_2, ^(b) ^N× N. • Identify the K nearest neighbors for each patch: (i)=topK(−i,:(b)).N(i)=topK (-D^(b)_i,: ). The graph topology remains fixed within each forward pass, but evolves dynamically across training iterations as the feature representations change. 3. Attention-Based Feature Aggregation: For each patch i∈1,…,Ni∈\1,…,N\: (a) Feature Mixing: For each neighbor j∈(i)j (i), concatenate the center and neighbor features: ij=[i∥j]∈ℝ2C.h_ij= [x_i\,\|\,x_j ] ^2C. (b) Attention Computation: ij=LeakyReLU(projij),e_ij=LeakyReLU (W_projh_ij ), sij=⊤ij,s_ij=a e_ij, αij=exp(sij)∑m∈(i)exp(sim). _ij= (s_ij)Σ _m (i) (s_im). (c) Weighted Aggregation: j=valj,v_j=W_valx_j, agg,i=ELU(∑j∈(i)αijj).x_agg,i=ELU ( _j (i) _ijv_j ). 4. Residual Update: out[b,i]=[b,i]+agg,i.X_out[b,i]=X[b,i]+x_agg,i. 5. Backward Propagation. Given the gradient from the task loss ℒL, denoted as ∇agg,i=∂ℒ∂agg,i _x_agg,i= _agg,i, the gradients for the learnable parameters are computed via the chain rule: • Update Value Matrix valW_val: The gradient flows through the weighted sum and the linear projection: ∇val←∑b,i∑j∈(i)(∇agg,i⋅αij)⊗j. _W_val← _b,i _j (i) ( _x_agg,i· _ij ) _j. • Update Attention Parameters ,proj\a,W_proj\: Let δij=∇agg,i⊤j _ij= _x_agg,i v_j be the gradient w.r.t. the attention weight. The gradients propagate through the Softmax and LeakyReLU functions: ∇←∑b,i,j∂αij∂sijδij⋅ij, _a← _b,i,j ∂ _ij∂ s_ij _ij·e_ij, ∇proj←∑b,i,j(∂αij∂sijδij⋅⊤⊙σ′(ij))⊗ij, _W_proj← _b,i,j ( ∂ _ij∂ s_ij _ij·a σ (e_ij) ) _ij, where σ′σ denotes the derivative of LeakyReLU and ⊗ represents the outer product. 11.2 Attention-Level DeGAT Implementation Details: MLP Bias Reference. Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5). Input. • Semantic features: ∈ℝB×N×CF ^B× N× C, patch tokens extracted from a frozen DINOv2 backbone. • Attention components: Query Q and Key ∈ℝB×H×N×dheadK ^B× H× N× d_head from the VGGT frame attention layers. • Hyperparameter: Number of quantization buckets K=8K=8. • Trainable parameters: Bias embedding table ∈ℝK×HE ^K× H. Output. Spatially modulated attention logits. Procedure. Given a batch index b∈1,…,Bb∈\1,…,B\, the semantic attention bias is computed as follows. The set of learnable parameters is the embedding table ∈ℝK×HE ^K× H. 1. Semantic Distance Calculation. We construct a fully connected semantic graph in which each node corresponds to a patch token, and the edge weights represent semantic dissimilarity. The pairwise Euclidean distance matrix ∈ℝB×N×ND ^B× N× N is computed as di,j=‖i−j‖2=∑c=1C(i,c−j,c)2.d_i,j= \|f_i-f_j \|_2= _c=1^C (f_i,c-f_j,c )^2. Smaller values of di,jd_i,j indicate higher semantic similarity, e.g., corresponding to similar tissue types. 2. Logarithmic Transformation. To reduce the influence of outliers while increasing sensitivity to highly similar patches, a logarithmic transformation is applied: d~i,j=log(di,j+1). d_i,j= (d_i,j+1 ). 3. Linear Mapping and Quantization. The transformed distances are normalized relative to the maximum semantic distance within the current view to ensure scale invariance: ratioi,j=d~i,jmax(d~)+ϵ.ratio_i,j= d_i,j ( d)+ε. The normalized distances are then discretized into K integer buckets: i,j=Clamp(⌊ratioi,j⋅K⌋,0,K−1),Idx_i,j=Clamp ( _i,j· K ,0,K-1 ), yielding ∈0,…,K−1B×N×NIdx∈\0,…,K-1\^B× N× N. 4. Bias Lookup. For each attention head h, a learnable scalar bias is retrieved from the embedding table: bi,j(h)=[i,j]h.b_i,j^(h)=b\! [Idx_i,j ]_h. This mechanism allows the model to learn distinct attention bonuses or penalties for different levels of semantic similarity. 5. Injection into Attention Mechanism. The semantic bias is added directly to the attention logits in the VGGT frame attention blocks: Attentioni,j(h)=Softmax(i⋅j⊤dhead+bi,j(h)).Attention_i,j^(h)=Softmax\! ( q_i·k_j d_head+b_i,j^(h) ). 6. Backward Propagation. Let ℒL be the total loss. The gradient of the loss with respect to the bias term is denoted as δi,j(h)=∂ℒ∂bi,j(h) _i,j^(h)= ∂ b_i,j^(h). Since the bias is added directly to the logits, this gradient is derived from the standard Softmax backward pass. The learnable embedding table b is updated by aggregating gradients from all patch pairs within the same bucket. For a specific bucket index k∈0,…,K−1k∈\0,…,K-1\ and head h, the gradient is computed as: ∇k,h=∑b=1B∑i=1N∑j=1N(i,j(b)=k)⋅δi,j(h,b), _b_k,h= _b=1^B _i=1^N _j=1^NI (Idx_i,j^(b)=k )· _i,j^(h,b), where (⋅)I(·) is the indicator function, which is 1 if the condition is met and 0 otherwise. 11.3 Attention-Level DeGAT Implementation Details: Bias Table Reference. Liu et al., Swin Transformer V2: Scaling Up Capacity and Resolution, adapted to semantic feature space. Input. • Feature map: ∈ℝB×N×CF ^B× N× C, flattened patch features from the current VGGT layer. • Attention components: Query Q and Key ∈ℝB×H×N×dheadK ^B× H× N× d_head. • Hyperparameters: maximum distance prior τ (e.g., C C), and MLP hidden dimension M=512M=512. • Trainable parameters: Bias MLP Ψ:ℝ→ℝH :R ^H, implemented as a two-layer network. Output. Content-adaptive attention bias matrix. Procedure. Given a batch index b∈1,…,Bb∈\1,…,B\, the continuous semantic bias is computed as follows. 1. Semantic Euclidean Distance. We abandon spatial coordinates in favor of feature-space representations. The pairwise Euclidean distance matrix ∈ℝB×N×ND ^B× N× N is computed between all patch tokens: di,j=‖i−j‖2.d_i,j= \|f_i-f_j \|_2. This distance measures the raw semantic discrepancy between two patch tokens. 2. Log-Space Normalization. To handle the heavy-tailed distribution of feature distances while preserving high resolution for semantically similar tokens, the distances are transformed and normalized in log space. First, a logarithmic transformation is applied: d^i,j=log(di,j+1). d_i,j= (d_i,j+1). The transformed distances are then normalized using the maximum distance within the current view: Δi,j=d^i,jlog(dmax+1). _i,j= d_i,j (d_ +1). Finally, the normalized values are scaled and shifted to obtain a continuous coordinate in [−1,1][-1,1]: i,j=2⋅Clamp(Δi,j,0,1)−1.x_i,j=2·Clamp( _i,j,0,1)-1. This yields a normalized tensor ∈[−1,1]B×N×N×1X∈[-1,1]^B× N× N× 1. 3. Continuous Bias Generation. Instead of a discrete embedding lookup, a lightweight MLP Ψ is employed to map the continuous distance coordinate to a head-specific attention bias: i,j=Ψ(i,j)=2(ReLU(1i,j+1))+2.B_i,j= (x_i,j)=W_2 (ReLU(W_1x_i,j+b_1) )+b_2. Here, 1∈ℝM×1W_1 ^M× 1, 2∈ℝH×MW_2 ^H× M, and 1b_1, 2b_2 are learnable bias terms. The MLP enables approximation of arbitrary continuous functions, thereby allowing nonlinear penalties or bonuses based on semantic dissimilarity. 4. Injection into Attention Mechanism. The generated continuous bias is added directly to the attention logits of the self-attention operation: Attentioni,j(h)=Softmax(i⋅j⊤dhead+i,j(h)).Attention_i,j^(h)=Softmax ( q_i·k_j d_head+B_i,j^(h) ). 5. Backward Propagation. Unlike quantization-based methods, the MLP projection enables end-to-end gradient flow. Let δi,j(h)=∂ℒ∂i,j(h) _i,j^(h)= _i,j^(h) be the gradient of the loss with respect to the generated bias. • Update MLP Parameters ΘΨ _ : Standard backpropagation is applied to update the MLP weights. Let i,j=ReLU(1i,j+1)z_i,j=ReLU(W_1x_i,j+b_1) be the hidden activation. The gradients for the second linear layer are: ∇2←∑b,i,ji,j⊗i,j,∇2←∑b,i,ji,j. _W_2← _b,i,j δ_i,j _i,j, _b_2← _b,i,j δ_i,j. Gradients are then backpropagated through the ReLU to update 1W_1 and 1b_1. 11.4 Additive Image-conditioned Camera Token Initialization We evaluate a simpler image-conditioned camera-token initialization strategy, which we use as a variant in our experiments. This design conditions the camera token via an additive bias predicted from global image features, which is the cls token + bias in Table 3 Global Image Feature Extraction. Given the patch tokens extracted from each input frame, we compute a global image representation by mean pooling over all patch tokens. This pooled feature captures frame-level semantic information and serves as the conditioning signal. Additive Conditioning via MLP. The global image feature is passed through a lightweight multi-layer perceptron (MLP) comprising two linear layers with GELU activation. The MLP outputs a feature vector with the same dimensionality as the camera token, which represents an image-conditioned offset: Δ=fMLP(global), =f_MLP(f_global), (9) where globalf_global denotes the pooled image feature and Δ is the predicted token offset. Camera Token Update. A learnable base camera token is expanded to match the batch and temporal dimensions. The image-conditioned offset is then added to the base camera token: cam=base+Δ.t_cam=t_base+ . (10) This additive formulation shifts the camera token in the embedding space based on image content, without introducing feature-wise scaling. Initialization and Stability. The base camera token is initialized with a small standard deviation, ensuring the overall magnitude of the conditioned token remains under control during early training. Compared with FiLM-based affine modulation, this additive conditioning provides a lightweight and stable alternative with reduced expressiveness; we include it as an ablation to study the effects of different conditioning mechanisms. 11.5 Implementation Details of FiLM-based Camera Token Modulation We adopt a Feature-wise Linear Modulation (FiLM) mechanism [18] to condition the camera token on image content. This design allows the camera token to adapt dynamically to the input frames while preserving training stability, as indicated by the cls + FiLM modulation in Table 3. Global Conditioning Signal. Given the patch tokens extracted from each input frame, we first compute a global image representation by mean pooling over all patch tokens. This pooled feature serves as the conditioning input for FiLM modulation, capturing frame-level semantic information. FiLM Parameter Prediction. The global image feature is passed through a lightweight modulation network comprising two linear layers with a nonlinear activation in between. The modulator outputs a vector twice the embedding dimension, split into a feature-wise scale parameter, γ, and a shift parameter, β. Camera Token Modulation. A learnable camera token is expanded to match the batch and temporal dimensions. FiLM modulation is then applied to the camera token as cam=base⊙(1+γ)+β,t_cam=t_base (1+γ)+β, (11) where baset_base denotes the static camera token, and γ and β are applied in a feature-wise manner. This formulation enables both amplitude scaling and feature shifting conditioned on the input frame. Initialization and Training Stability. To ensure stable optimization, the final linear layer of the modulation network is initialized with zero weights and biases. As a result, the FiLM parameters are initially γ=γ=0 and β=β=0, such that the modulated camera token is identical to the base token at the start of training. This initialization preserves the original token distribution and prevents instability during early training. Overall, this FiLM-based modulation provides an efficient and effective mechanism for incorporating global image context into the camera token without introducing additional attention operations or significant computational overhead. 11.6 Cross-Attention-based Camera Token Conditioning In this design, the camera token is treated as a learnable query that attends to all patch tokens extracted from the input image. Instead of conditioning the camera token through an MLP, global image context is incorporated through a Transformer-style cross-attention mechanism. This is the cls + cross attention in Table 3. Query, Key, and Value Construction. For each input frame, a learnable base camera token is expanded to match the batch and temporal dimensions and used as the query. All patch tokens corresponding to the same frame are used as keys and values: =cam,==pp=1P,q=t_cam, =v=\t_p\_p=1^P, (12) where cam∈ℝ1×Ct_cam ^1× C denotes the camera token and p\t_p\ are the patch tokens. Cross-Attention Update. A standard multi-head cross-attention layer is applied, followed by a residual connection: cam′=cam+Attn(,,),t_cam =t_cam+Attn(q,k,v), (13) where Attn(⋅)Attn(·) denotes multi-head attention. This operation allows the camera token to selectively aggregate information from spatially and semantically relevant patches. Feed-Forward Refinement. The attention-updated camera token is further processed by a feed-forward network (FFN) with a residual connection: camout=cam′+FFN(cam′),t_cam^out=t_cam +FFN(t_cam ), (14) which increases representational capacity and stabilizes optimization. Initialization and Stability. To ensure stable training, the cross-attention layer’s output projection and the final linear layer of the FFN are initialized to zero. As a result, the cross-attention module initially behaves as an identity mapping, preserving the original camera token at the beginning of training and allowing the model to gradually learn attention-based conditioning. Overall, this cross-attention-based formulation enables explicit and interpretable aggregation of global image information into the camera token, serving as a complementary alternative to MLP-based conditioning mechanisms. 11.7 Experiment Datasets We evaluate our method on three publicly available endoscopic reconstruction datasets. EndoSLAM [17] provides ex vivo and synthetic endoscopic sequences with accurate 6-DoF camera poses and dense ground-truth point-cloud maps of porcine gastrointestinal organs. SCARED [1] is a stereo-endoscopy dataset captured with the da Vinci Xi surgical system and offers ground-truth depth, enabling rigorous evaluation of depth estimation in realistic surgical settings. EndoNeRF [26] introduces photorealistic synthetic endoscopic videos rendered via physically based simulation. We use two subsets: “cutting”, which depicts tissue excision with topological changes, and “pulling”, which captures elastic deformation induced by traction instruments. This dataset provides perfectly aligned depth, pose, and segmentation labels, serving as a valuable benchmark for reconstruction under controlled anatomical conditions. 11.8 Formulas for PSNR, SSIM, and LPIPS To quantitatively evaluate the reconstruction quality, we employ three standard metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS). This subsection provides definitions of the evaluation metrics; readers familiar with them may skip it. 11.8.1 PSNR PSNR measures the pixel-level fidelity between the ground truth image I and the reconstructed image I I. It is defined based on the Mean Squared Error (MSE): MSE=1MN∑i=1M∑j=1N(I(i,j)−I^(i,j))2,MSE= 1MN _i=1^M _j=1^N (I(i,j)- I(i,j) )^2, (15) where M and N denote the height and width of the image, respectively. The PSNR is then calculated as: PSNR=10⋅log10(MAXI2MSE),PSNR=10· _10 ( MAX_I^2MSE ), (16) where MAXIMAX_I is the maximum possible pixel value of the image (e.g., 1.01.0 for floating-point images or 255255 for 8-bit integers). A higher PSNR indicates better reconstruction quality in terms of signal fidelity. 11.8.2 SSIM Unlike PSNR, SSIM evaluates the perceived quality by considering changes in structural information, luminance, and contrast. For two image patches x and y, SSIM is defined as: SSIM(x,y)=(2μxμy+C1)(2σxy+C2)(μx2+μy2+C1)(σx2+σy2+C2),SSIM(x,y)= (2 _x _y+C_1)(2 _xy+C_2)( _x^2+ _y^2+C_1)( _x^2+ _y^2+C_2), (17) where: • μx _x and μy _y are the average intensities of x and y. • σx2 _x^2 and σy2 _y^2 are the variances of x and y. • σxy _xy is the covariance between x and y. • C1=(k1L)2C_1=(k_1L)^2 and C2=(k2L)2C_2=(k_2L)^2 are constants to stabilize the division with a weak denominator, where L is the dynamic range of pixel values, k1=0.01k_1=0.01, and k2=0.03k_2=0.03. The final SSIM score is typically computed as the mean SSIM over all sliding windows in the image. 11.8.3 LPIPS LPIPS measures the perceptual distance between two images using deep features extracted from a pre-trained network (e.g., VGG or AlexNet). Let ϕφ be the feature extractor. The distance d between the ground truth x and the reconstruction x0x_0 is computed as: LPIPS(x,x0)=∑l1HlWl∑h,w‖wl⊙(y^hwl−y^0,hwl)‖22,LPIPS(x,x_0)= _l 1H_lW_l _h,w \|w_l ( y^l_hw- y^l_0,hw ) \|_2^2, (18) where: • y^l y^l and y^0l y^l_0 denote the feature maps extracted at layer l for image x and x0x_0, respectively. • The features are unit-normalized in the channel dimension. • wlw_l represents the learned scaling weights for layer l. • ⊙ denotes the element-wise product. A lower LPIPS score indicates better perceptual similarity to the ground truth. 12 Additional Experiments Table 3: Experiment results on EndoNeRF and SCARED dataset using PSNR, SSIM, and LPIPS metrics. For PSNR and SSIM, the higher ↑ the better. For LPIPS, the lower ↓ the better. The best results are highlighted in green, and the second best results are underlined. Dataset Method PSNR ↑ SSIM ↑ LPIPS ↓ EndoNeRF-pulling VGGT 23.349 0.659 0.396 EndoSurf 34.093 0.938 0.163 EndoVGGT w/o DeGAT 34.516 0.918 0.108 EndoVGGT w/ DeGAT 34.642 0.918 0.100 EndoNeRF-cutting VGGT 21.540 0.872 0.291 EndoSurf 30.606 0.909 0.168 EndoVGGT w/o DeGAT 31.859 0.870 0.159 EndoVGGT w/ DeGAT 32.227 0.888 0.156 SCARED-d1k1 VGGT 14.061 0.251 0.379 EndoSurf 23.401 0.669 0.505 EndoVGGT w/o DeGAT 33.309 0.929 0.249 EndoVGGT w/ DeGAT 34.348 0.939 0.240 SCARED-d2k1 VGGT 19.386 0.283 0.366 EndoSurf 24.894 0.856 0.254 EndoVGGT w/o DeGAT 36.411 0.856 0.247 EndoVGGT w/ DeGAT 36.634 0.871 0.221 SCARED-d3k1 VGGT 14.431 0.295 0.388 EndoSurf 21.357 0.641 0.470 EndoVGGT w/o DeGAT 25.300 0.930 0.190 EndoVGGT w/ DeGAT 26.906 0.930 0.167 Average VGGT 18.553 0.472 0.364 EndoSurf 26.870 0.803 0.312 EndoVGGT w/o DeGAT 32.279 0.901 0.190 EndoVGGT w/ DeGAT 32.951 0.909 0.177 12.1 Ablation Study on Number of Neighbors K Figure 5: Ablation Study on the number of neighbors K on SCARED dataset. To investigate the impact of graph connectivity density on feature aggregation, we conducted an ablation study in which we varied the number of neighbors (K) from 2 to 18 on the SCARED dataset. As illustrated in Figure 5, the model performance exhibits a clear convex trend, peaking at K=9K=9. Specifically, the setting with K=9K=9 achieves the best results across all metrics, yielding a PSNR of 34.3534.35, SSIM of 0.9390.939, and LPIPS of 0.2400.240. We observe that with a smaller K, the sparse connectivity limits the receptive field, hindering the formation of robust local cycles necessary for effective bridge characterization. Conversely, increasing K beyond 9 results in noticeable performance degradation, with PSNR dropping to 33.38 at K=18K=18. This suggests that excessive connectivity introduces irrelevant long-range noise and results in feature over-smoothing. Consequently, we adopt K=9K=9, following [8], as the optimal hyperparameter for our dynamic graph attention. Table 4: Ablation studies on the EndoNeRF-cutting dataset. We report PSNR, SSIM, and LPIPS for different deformation modeling strategies and GAT enhancement variants. Methods PSNR ↑ SSIM ↑ LPIPS ↓ (A) Baseline EndoVGGT(Ours) 31.8594 0.8700 0.1588 (B) EndoVGGT + Token-Level cls Enhancement cls token + bias 31.9641 0.8748 0.1570 cls token + FiLM modulation 31.9405 0.8693 0.1576 cls token + cross attention 32.0157 0.8711 0.1565 (C) EndoVGGT + Attention-Level DeGAT Learnable Bias Table 31.9592 0.8721 0.1571 Continuous MLP Bias 31.9700 0.8742 0.1574 (D) EndoVGGT + Feature-Level DeGAT Post-Transformer GAT 31.5710 0.8686 0.1689 Pre-Transformer GAT 32.2269 0.8693 0.1558