Paper deep dive
OmniCam: Omni-Camera Trajectory Generation via Geometry-Grounded Pose Token Learning
Zhenyang Liu, Chenjie Cao, Yisu Zhang, Xuhui Zuo, Xiangyang Xue, Yanwei Fu, Tengfei Wang, Chunchao Guo
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/11/2026, 2:38:30 AM
Summary
The paper introduces OmniCam, an autoregressive model for generating camera trajectory sequences from panoramic images and textual descriptions. It utilizes geometry-grounded pose token learning, combining a panoramic point-cloud encoder with hybrid absolute-rotation and relative-translation tokenization. The authors also present OmniCaT, a dataset of 267,700 trajectories across four camera behaviors, demonstrating that OmniCam significantly reduces trajectory errors and collision rates compared to baselines like GenDoP.
Entities (10)
Relation Signals (7)
OmniCaT → contains → 267,700 trajectories
confidence 98% · OmniCaT, containing 267,700 trajectories across four camera behaviors.
OmniCam → trainedon → OmniCaT
confidence 96% · We also construct OmniCaT... On the reported OmniCaT evaluation, OmniCam reduces trajectory errors...
OmniCam → uses → Panoramic Point Cloud
confidence 95% · Its geometry-grounded pose token learning combines... a panoramic point-cloud encoder for omnidirectional geometric context
OmniCam → outperforms → GenDoP
confidence 94% · OmniCam reduces trajectory errors by 28--47% and collision rate by 65.8% relative to GenDoP retrained on OmniCaT.
OmniCam → uses → Quaternion
confidence 92% · hybrid absolute-rotation and relative-translation tokenization with temporally consistent quaternion signs
OmniCam → enables → Camera-Controlled Video Generation
confidence 90% · downstream experiments examine camera-controlled video generation and robotic active perception.
OmniCam → enables → Robotic Active Perception
confidence 90% · downstream experiments examine camera-controlled video generation and robotic active perception.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Camera trajectories control viewpoint changes in video generation, scene reconstruction, and robotic perception. Generating them from language requires both scene geometry and target-aware framing. We introduce OmniCam, an autoregressive model that generates camera pose sequences from a single panorama and textual trajectory descriptions. Its geometry-grounded pose token learning combines three components: a panoramic point-cloud encoder for omnidirectional geometric context; hybrid absolute-rotation and relative-translation tokenization with temporally consistent quaternion signs; and separate geometric and semantic conditioning streams with an explicit 3D target anchor. We also construct OmniCaT, containing 267,700 trajectories across four camera behaviors. On the reported OmniCaT evaluation, OmniCam reduces trajectory errors by 28--47% and collision rate by 65.8% relative to GenDoP retrained on OmniCaT. Against the best baseline for each metric, the ATE and collision reductions are 43.0% and 62.3%, respectively. Component ablations support the use of geometric and target-aware conditioning, while downstream experiments examine camera-controlled video generation and robotic active perception.
Tags
Links
- Source: https://arxiv.org/abs/2610.09513v1
- Canonical: https://arxiv.org/abs/2610.09513v1
Trouble viewing inline? Open PDF directly →
Full Text
72,722 characters extracted from source content.
Expand or collapse full text
OmniCam: Omni-Camera Trajectory Generation via Geometry-Grounded Pose Token Learning Zhenyang Liu 1,2,3 Chenjie Cao 3 Yisu Zhang 4,3 Xuhui Zuo 3 Xiangyang Xue 1,‡ Yanwei Fu 1,2,‡ Tengfei Wang 3,† Chunchao Guo 3 1 Fudan University 2 Shanghai Innovation Institute 3 Tencent Hunyuan 4 Zhejiang University Project Page: ZhenyangLiu.github.io/OmniCam-Website OmniCaTDataset 267.7K Trajectories TargetSurround ReconstructWander ●PANORAMIC INPUT ●TrajectoryCaption <TASK: TARGET> <TARGET_OBJ: wooden fence> <TARGET_DIR: Right> The camera push in narrows the field of view, drawing attention steadily toward the wooden fence positioned at the Right. This is to focus on the wooden fence through a push in... OmniCamTrajectoryGenerationDownstreamApplication ● Camera-Controlled VideoGeneration ●Camera-Controlled 3D Reconstruction ●Camera-Guided Active Perception in Robotics Figure 1. Overview of OmniCam. Given a single panorama and textual trajectory descriptions, OmniCam generates spatially aware camera trajectories (SE(3) pose sequences). OmniCam supports four camera behaviors that enable various downstream applications such as camera-controlled video generation, 3D reconstruction, and active perception in robotics. Abstract Camera trajectories control viewpoint changes in video gen- eration, scene reconstruction, and robotic perception. Gen- erating them from language requires both scene geometry and target-aware framing. We introduce OmniCam, an au- toregressive model that generates camera pose sequences from a single panorama and textual trajectory descriptions. Its geometry-grounded pose token learning combines three components: a panoramic point-cloud encoder for omnidi- rectional geometric context; hybrid absolute-rotation and relative-translation tokenization with temporally consistent quaternion signs; and separate geometric and semantic con- ditioning streams with an explicit 3D target anchor. We also construct OmniCaT, containing 267,700 trajectories across four camera behaviors. On the reported OmniCaT evaluation, OmniCam reduces trajectory errors by 28–47% and collision rate by 65.8% relative to GenDoP retrained on OmniCaT. Against the best baseline for each metric, the † Project Lead. ‡ Corresponding authors. ATE and collision reductions are 43.0% and 62.3%, respec- tively. Component ablations support the use of geometric and target-aware conditioning, while downstream exper- iments examine camera-controlled video generation and robotic active perception. 1. Introduction Camera trajectories serve as a control interface for spatial- intelligence systems, including world models [1], video gen- eration [13,31], and robotics [2,16,21]. Specifying a tra- jectory requires deciding where to move, how to orient, and what to attend to. A prescribed orbit or linear sweep may fail when it intersects scene geometry or loses the intended target. We study generating scene-aware camera trajectories from a panoramic observation and a textual instruction (Figure 1). Recent trajectory generators [7,18,23,34] motivate three design questions. (i) Scene coverage: perspective RGBD inputs [34] provide limited angular coverage, while geometry-based planners require explicit spatial representa- 1 arXiv:2610.09513v1 [cs.CV] 7 Oct 2026 tions [29,30]. A panorama provides observations in all direc- tions, although occluded surfaces remain unknown. (i) Pose serialization: quantization trades precision against token vocabulary size. Quaternion sign ambiguity (q ≡ −q), co- ordinate scale, and the accumulation of relative increments can affect the resulting learning problem. (i) Conditioning: geometric free-space information and language-referenced targets serve different roles in planning. Providing dedi- cated pathways for these cues is a useful architectural choice whose value should be tested empirically. We use geometry- grounded pose token learning to describe a design that con- nects geometric perception, pose serialization, and target- aware conditioning. OmniCam implements this design with a panoramic point-cloud encoder, hybrid pose tokenization, and a dual- branch conditioning architecture. The point cloud, recon- structed from estimated panoramic depth, provides omnidi- rectional but partial geometry. Rotations are encoded abso- lutely with sequential quaternion sign alignment; translations are encoded as increments in a scene-normalized coordinate system. Sign alignment selects consistent representatives of the quaternion double-cover, while quantization remains lossy. Absolute rotations avoid composition of rotation- quantization residuals; relative translations trade a smaller per-step range against accumulated displacement error. We analyze these distinctions in Section I and compare the en- codings empirically. Separate geometric and semantic pathways supply the decoder with structural and target-related features. An attention-derived 3D target centroid is injected through gated cross-attention to provide a spatial anchor for object-directed trajectories. The geometry pathway does not directly receive text; this routing does not imply statistical independence or guarantee the absence of interference after fusion. At infer- ence, a format constraint restricts termination to complete pose-token groups. Training requires trajectories paired with scene observa- tions and instructions. Existing collections [24,33,34,37] cover recorded object, scene, and cinematic motions. We construct OmniCaT by recovering geometry and seman- tic targets from panoramas and synthesizing four behaviors: Target, Surrounding, Reconstruction, and Wandering. The dataset contains 267,700 trajectories with hierarchical de- scriptions of task, target, and motion. Because supervision is generated on estimated geometry, the learned model may inherit reconstruction errors and planner preferences. Our contributions are: • Panorama-conditioned trajectory generation. Omni- Cam combines explicit point-cloud features, dedicated geometric and semantic pathways, and a 3D target anchor to generate camera trajectories from panoramic imagery and language. •Hybrid pose serialization. We combine absolute rota- tions, relative translations, and consistent quaternion signs; provide qualified error and representation analyses; and compare the choices through ablations. • OmniCaT and empirical evaluation. We introduce a 267,700-trajectory dataset spanning four behaviors. The reported OmniCaT results improve on retrained baselines, with component ablations and downstream video and robotic-perception evaluations. 2. Related Work Camera Trajectory Generation. Early approaches rely on optimization-based planning [3,11,25,28] or learning- based control [4,10,15,17]. Recent methods integrate camera movement with scene dynamics: CCD [18] employs a diffusion model with text and keyframe conditioning in character-centric coordinates; E.T. [7] incorporates character trajectories in global coordinates; Director3D [23] adopts a DiT-based framework on multi-view data; Hou et al. [14] predict next-frame motion autoregressively for aerial videog- raphy; GenDoP [34] combines text with RGBD inputs in an autoregressive framework. These methods use different task-specific inputs and trajectory representations. Omni- Cam studies panoramic point-cloud conditioning and explicit target anchoring, together with the effect of absolute and rel- ative pose serialization in an autoregressive generator. Camera Trajectory Datasets. Existing trajectory datasets remain limited in spatial diversity.MVImgNet [33], RealEstate10K [37], and DL3DV-10K [24] provide cali- brated trajectories but focus on simple motions around sta- tionary objects or within static environments. CCD [18] and E.T. [7] offer character-centric tracking data that does not generalize to unconstrained spatial navigation. Data- DoP [34] pairs trajectories with depth maps and text for cinematic applications, yet relies on perspective-view depth covering only a limited field of view. None of these resources simultaneously provides omnidirectional scene geometry, di- verse spatial behaviors, and hierarchical textual guidance. OmniCaT fills this gap by grounding trajectory generation in panoramic point-cloud geometry and pairing each trajectory with structured language annotations across four distinct navigation behaviors. Pose Representation Learning.Rotation representa- tion [19,22] has a well-studied impact on learning stability. Zhou et al. [38] show that quaternions and Euler angles suf- fer from discontinuities under the topology ofSO(3)and propose continuous 6D representations for single-frame ro- tation regression. Subsequent work considers sequential settings such as human motion prediction [5,8] and robotic manipulation [21,26]. OmniCam uses sequential quater- nion sign alignment to reduce inconsistent discrete labels for equivalent rotations, combined with relative translation. This convention does not provide a globally continuous rotation parameterization or an injective finite quantizer. 2 Phase 1: Geometric and Semantic Grounding Panorama Input Geometry DepthEstimation andAlignment Hybrid Filtering Strategy Recast Navigation Panoramic Point Cloud Navigation Mesh Semantics Semantic Parsing Semantic Mask Generation 3D Semantic Target Generation Semantic Masks 3DSemanticTargets Phase 2: SpatialTrajectorySynthesis Phase 3: Rendering and Multimodal Annotation Navigation Mesh 3DSemantic Targets Target Approach landmarks fromoptimal viewpoints Surrounding Circle objects for complete visual coverage Reconstruct Fill under- observed regionsfor 3D reconstruction Wandering Navigate to farthest points for maximum coverage Camera Trajectories Panoramic Point Cloud Hierarchical Annotation Kinematic Motion Primitives ParallelizedPoint- based Rendering VLM Synthesis and Reasoning Detailed Descriptions Text-TrajectoryPairs Concise Commands (b) Task Types 36.5% 33.6% 16.2% 13.7% SURROUNDTARGET RECONSTRUCTEXPLORATION 267.7K Trajectories (c) Target Objects (Top 15) 2,498 2,688 2,689 2,846 3,064 3,251 3,402 3,600 3,822 4,157 8,856 11,123 13,714 14,267 glass door curtains lamp wooden door wooden pillar bed pillow sofa painting potted plant cabinet curtain door window (d) Camera Motions (Top 15) 1,450 1,452 1,510 2,291 5,165 5,296 7,222 7,887 10,980 11,225 26,527 30,446 49,642 49,710 50,007 orbit R + move FL orbit L + move FR pull out stationary move front-left move front-right orbit R→L orbit L→R orbit R orbit L push in + orbit R→L push in + orbit L→R push in + orbit R push in + orbit L push in (e) Target Directions 43,481 37,005 36,607 33,219 32,343 28,940 28,463 27,642 0 10,000 20,000 30,000 40,000 50,000 (a) OmniCaT Construction Pipeline. Figure 2. OmniCaT Dataset Overview. Left: (a) Construction pipeline comprising (1) Geometric and Semantic Grounding to extract nav-meshes and 3D targets, (2) Spatial Trajectory Synthesis to generate diverse camera behaviors, and (3) Rendering and Multimodal Annotation to produce text-trajectory pairs. Right: Dataset statistics showing (b) task distribution, (c) top object categories, (d) motion primitive diversity, and (e) target direction distribution. 3. The OmniCaT Dataset Collecting camera trajectories with controlled behavior and target coverage is costly. OmniCaT synthesizes trajectories on geometry reconstructed from panoramic images, enabling controlled variation of spatial behaviors. Feasibility is as- sessed against the reconstructed geometry and does not by itself guarantee collision avoidance in the physical scene. 3.1. Dataset Construction As illustrated in Figure 2, the pipeline consists of three phases. (1) Geometric and semantic grounding: panoramic depth maps are aligned into a global point cloudP pan [32], filtered for outliers, and augmented with semantic landmarks and 3D volumetric masks using language-assisted annotation and segmentation models [6,20,27]; a navigation mesh is then constructed to define the traversable space. (2) Spatial trajectory synthesis: four categories of camera behaviors (Target, Surrounding, Reconstruction, and Wandering) are generated on the navigation mesh, with waypoints and orien- tations refined via spline and spherical linear interpolation. (3) Rendering and multimodal annotation: a point-based ren- derer produces video sequences from the planned poses, and a hierarchical annotation scheme derives kinematic motion primitives from camera extrinsics before a vision-language model synthesizes motion-aware textual instructions. Full implementation details, including depth alignment optimiza- tion, NavMesh engineering, and trajectory planning heuris- tics, are provided in Section A. 3.2. Dataset Statistics Scale and Annotations. As summarized in Figure 2, Om- niCaT comprises 267,700 trajectories across 9,998 scenes (21.7M rendered frames), categorized into four spatial behav- iors: Surrounding (97,702), Target (90,065), Reconstruction (43,316), and Wandering (36,617). Each trajectory is an- notated with 3D semantic targets, directional bearings, and hierarchical textual descriptions, enabling joint geometric- semantic conditioning. Full annotation details are provided in Section A. Scope of Synthetic Supervision. The planner uses recon- structed geometry, navigation meshes, and semantic targets to generate feasible examples. OmniCam learns to predict trajectories from a panorama and text without construct- ing a navigation mesh or running graph search at inference. Downstream evaluation tests the usefulness of this learned mapping. A gap to the planner does not establish freedom from planner bias; evaluation on independently measured ge- ometry and independently specified tasks is needed to assess that limitation. 4. Method 4.1. Hybrid Pose Tokenization We serialize continuous camera parameters using scene- aware calibration and a hybrid absolute-relative represen- tation. Quaternion sign alignment addresses equivalent pos- itive and negative representations; finite binning still intro- duces quantization error. Scene-Aware Calibration. Panoramic depth is unprojected into a filtered 3D point cloud. Scene statistics, including a 3 Camera TrajectoryCaption <TASK: Surround> <TARGET_OBJ: wooden Building> <TARGET_DIR: Left> The camera push in and orbits left, drawing attention toward the wooden building positioned at the Left. This is to focus... Panoramic Image Omnidirectional Point Cloud Inputs Te x t Encoder 풁 풕풙풕 Vision Encoder 푭 풔풆풎 Language-Guided Semantic Resampler Cross-Modal Guidance 풁 풔풆풎 풁 풕풙풕 푭 풔풆풎 SemanticBranch GeometricBranch Point Cloud Encoder 푭 품풆풐 Geometry Resampler 풁 품풆풐 Attention-Aggregated position encoding 푬 풕품풕 PointCoordinates 풁 풕풙풕 Concat Autoregressive Decoder Ta r g e tCentroid Injection De - tokenizer GeneratedTr a j e c t o r y SE(3)-Consistent Camera Tokenization QQQQT T TFF Absolute Quaternion Relative Tr a n s l a t i o n Intrinsics Scene-awareCanonicalization Constrained Autoregressive Decoding 3DAttentionHeatmap 풁 풔풆풎 (ퟎ) Ta r g e t C e n t r o i d E x t r a c t i o n Figure 3. OmniCam Architecture. Three parallel branches encode the caption, panoramic image, and point cloud. A Language-Guided Resampler produces target-aware semantic tokens while a Geometry Resampler yields geometric descriptors; an attention-aggregated 3D target centroid provides an explicit spatial anchor for the decoder. The autoregressive decoder generates pose tokens from the concatenated condition, which are mapped to continuous SE(3) trajectories by a de-tokenizer. Right bottom: SE(3) tokenization with 9 tokens for each pose frame. centroidcand a scales, calibrate camera translations and the initial frame. Normalization reduces variation in coordinate magnitude across scenes but does not eliminate errors in estimated metric scale. Hybrid Absolute-Relative Pose Encoding. Rotations are encoded as absolute unit quaternions with sequential sign alignment: each frame is aligned with the preceding aligned representative. The deterministic convention and its assump- tions are stated in Proposition 2; implementation-specific tie handling must agree with that convention. Translations after the initial frame use relative increments, reducing the per- step dynamic range while allowing quantization residuals to accumulate. These components, together with normalized intrinsics, are quantized via uniform binning into per-frame token groups of 9 tokens: 4 rotation, 3 translation, 2 in- trinsics (Figure 3, right bottom). The token sequenceSis augmented with boundary tokens (BOS, EOS, PAD) and projected into a latent space through a learnable codebook. Theoretical Motivation. The hybrid scheme balances dif- ferent reconstruction-error tradeoffs (Section I). If each de- coded rotation has angular quantization error at mostε, ab- solute encoding retains this per-frame bound, whereas in- tegrating relative rotations permits a worst-case bound of minπ, (T − 1)εwhen the initial rotation is exact (Proposi- tion 1). The value ofεdepends on the component quantizer and decoder normalization. Relative translations can use a smaller displacement range and hence finer local resolution, but their errors can also accumulate. Proposition 2 shows that a specified initial sign rule followed by alignment to the already aligned preceding quaternion removes dependence on the arbitrary quaternion sign when adjacent rotations dif- fer by less thanπ. This resolves double-cover ambiguity in the supervision; finite quantization remains lossy and many-to-one. 4.2. Decoupled Dual-Branch Conditioning OmniCam encodes estimated panoramic geometry and language-referenced appearance in separate pathways. The point cloud supplies omnidirectional, partial scene structure; the semantic pathway supplies target-related features. This architecture separates input routing, without assuming that the learned representations are statistically independent. Semantic Branch (Text & Vision Encoders). The seman- tic branch jointly processes the trajectory caption and the panoramic image to produce target-aware features. A pre- trained text encoder maps the captionT txt into textual fea- turesZ txt ∈R N txt ×C txt , which also serve as cross-modal guidance for the vision pathway. In parallel, a vision encoder produces a visual feature mapF sem ∈R H×W×C sem from I pano . Text and the target descriptorT obj guide the subse- quent semantic resampling stage. Structured caption dropout varies instruction composition during training (Section H). Geometric Branch (Point Cloud Encoder). The geomet- ric branch receives no direct text input. At inference time, a monocular depth estimator [32] reconstructs panoramic depth from the input RGB panorama; the estimated depth is then unprojected into a 3D point cloud, voxelized, and processed by a sparse-convolution encoder combined with a Transformer encoder to yield geometric descriptorsF geo ∈ 4 R M×C geo . Its features provide structural context for trajec- tory generation. Architectural details for both branches are provided in Section C. 4.3. Spatial Grounding via Resampling and Target Centroid Injection This section introduces two parallel resamplers that com- press heterogeneous encoder outputs (text, vision, and point cloud features) into compact prefix tokens, and an attention- aggregated 3D centroid that serves as an explicit spatial anchor. Language-Guided Semantic Resampler. Given the vi- sual features from the vision encoder, we first project them into the 3D point-cloud space to obtain 3D semantic fea- turesZ (0) sem . The semantic resampler combines self-attention, spatial attention, and textual attention. The interaction is summarized schematically at layer l: Z (l) sem = SelfAttn Z (l−1) sem | z Q,K,V + CrossAttn Z (l−1) sem |z Q , [P; F sem ] | z K,V + CrossAttn Z (l−1) sem | z Q , Z txt |z K,V . (1) where 3D semantic features serve as the common query; self- attention captures inter-query dependencies; spatial cross- attention grounds queries in physical scene geometry ag- gregated by 3D point coordinatesPconcatenated with vi- sual featuresF sem , and text-guided cross-attention aggre- gates textual guidance fromZ txt . The resultingZ sem ∈ R M sem ×C serves as the semantic latent code. Geometry Resampler. A text-agnostic geometry resampler compresses dense per-point descriptorsF geo into a fixed- length latent sequenceZ geo ∈R M geo ×C through a stack of Transformer decoder layers, providing a geometric condition without a direct text input. Target Centroid Injection. A target centroidc tgt supplies an explicit spatial anchor. Leta j denote the target-attention logit associated with pointp j and letτ > 0be the tem- perature. The described soft-argmax aggregation has the form α j = exp(a j /τ ) P N k=1 exp(a k /τ ) ,c tgt = N X j=1 α j p j .(2) This requires point-indexed weights; attention over text to- kens alone does not directly define them. The centroid is positionally encoded into an auxiliary target memory E tgt ∈R M tgt ×C . When caption dropout removes the target object tag, these tokens are masked. The final multimodal conditionZis formed by concate- nating the textual features with the resampled panoramic latents: Z = [Z txt ; Z sem ; Z geo ].(3) The target token bankE tgt serves as auxiliary memory ac- cessed by the decoder through a gated cross-attention mod- ule, as detailed in Section 4.4. 4.4. Format-Constrained Autoregressive Decoding An autoregressive Transformer decoderDgenerates discrete trajectory tokens conditioned on the multimodal prefixZ and the target token bank E tgt . Decoder Architecture. The conditionZis prepended to the input sequence before the BOS token. Each decoder block consists of causal self-attention, gated cross-attention over E tgt , and a feed-forward network. The gate controls the con- tribution of the target memory. Its effect on target-directed trajectories is evaluated through the grounding ablations; per-behavior gate strength is not inferred from the aggregate results. Format-Constrained Decoding. During inference, the EOS token is masked within each group of 9 tokens (rotation, translation, and intrinsics), so termination occurs only at a frame boundary. The initial frame is constrained by the calibrated origin. This is a sequence-format constraint: valid rotations require an appropriate quaternion decoding rule, and collision avoidance is assessed empirically rather than guaranteed by token masking. Top-ksampling controls the sampling support. Training Objective.The model is optimized with a weighted cross-entropy lossL ce over pose tokens, where rotation and translation channels receive higher weights to prioritize geometric accuracy. An auxiliary smoothℓ 1 lo- calization lossL loc supervises the estimated centroidc tgt against the annotated 3D target position:L =L ce +λ·L loc . 5. Experiments 5.1. Experimental Setup Datasets. OmniCaT experiments are reported on disjoint training and evaluation splits. Exact scene-level counts and split manifests require verification. DataDoP [34] results are reported for OmniCam trained on OmniCaT without Data- DoP fine-tuning. The construction of OmniCam inputs and the evaluation geometry on DataDoP must be specified be- fore interpreting these as a controlled zero-shot comparison. Baselines. Director3D [23] and GenDoP [34] are retrained on OmniCaT. CCD [18] and E.T. [7] use released check- points as out-of-domain references. Their character-centric conditioning differs from the available OmniCaT inputs, so these rows do not isolate model quality under matched con- ditioning. Metrics. We report trajectory errors (ATE, FDE, and RPE- R/T, using the EVO toolkit [12] where applicable), collision against reconstructed navigation meshes, target visibility, 5 Table 1. Comparison with representative trajectory generation baselines on OmniCaT and the public DataDoP benchmark. MethodDatasetATE↓FDE↓RPE-R↓RPE-T↓Coll. (%)↓Vis. (%)↑Angle ( ◦ )↓ Evaluation on the OmniCaT benchmark CCD [18]Pre-trained2.4833.3963.1120.08236.912.332.5 E.T. [7]Pre-trained1.9832.3682.1510.06946.221.340.2 Director3D [23] Pre-trained1.8282.9422.3090.05228.428.127.2 Director3DOmniCaT1.5222.6482.1870.04927.629.326.5 GenDoP [34]Pre-trained1.8652.9511.6810.04432.631.529.1 GenDoP OmniCaT1.6242.4831.5770.03830.433.627.6 OmniCam (Ours)OmniCaT0.8681.4621.1350.02710.452.010.9 Evaluation on the public DataDoP benchmark [34] CCD [18]Pre-trained2.6973.3383.2180.09344.218.734.2 E.T. [7] Pre-trained2.5683.1423.0860.08445.922.533.8 Director3D [23]Pre-trained1.9442.8872.9680.07739.630.834.3 Director3D DataDoP1.6882.7622.8190.06241.732.229.5 GenDoP [34]DataDoP1.4372.2551.6420.05828.536.629.8 OmniCam (Ours)OmniCaT0.9251.8341.1180.0468.964.212.7 CCDDirector3DGenDoPOmniCam (Ours)Scenes <TASK: TARGET> <TARGET_OBJ: door> <TARGET_DIR: Front> The intent is to draw attention to the Front door through a push in, and finally turnright. The sequence concludes by arriving at the final <END_OBJ: door> position in the <END_DIR: Front> sector. <TASK: SURROUND> <TARGET_OBJ: sculpture> <TARGET_DIR: Back> The ... executes an orbit left to gracefully circle around the central structure. The sequence concludes by arriving at the final <END_OBJ: sculpture> position in the <END_DIR: Back> sector. <TASK: TARGET> <TARGET_OBJ: leather chair> <TARGET_DIR: Front-Right> The intent ...Front-Right through a push in, and finally turnright. The sequence concludes by arriving at the final <END_OBJ: leather chair> position in the <END_DIR: Front-Right> sector. <TASK: RECONSTRUCT> <TARGET_OBJ: computer workstation> <TARGET_DIR: Back> Inspect the back of the workstation to examine its frame and lighting.The sequence ... <END_OBJ: computer workstation> position within the <END_DIR: Back> sector of the panoramic scene. Figure 4. Qualitative comparison on representative panoramic scenes. Each column shows the trajectory produced by a specific method. The target object is indicated by a red bounding box. 6 Panorama 3D Reconstruction w/o OmniCam Traj. 3D Reconstruction w/ OmniCam Traj. Camera-Controlled Video Generation with OmniCam Traj. Figure 5. Qualitative comparison on panoramic scenes. Each row displays an input panorama, generated video sequences, and 3D reconstruction results. The two rightmost columns compare reconstruction outputs without and with the OmniCam trajectory; no quantitative reconstruction metric is reported here. Retrieving an apple partially hidden behind a bowl Locating and reaching an apple hidden behind a box Figure 6. Real-robot demonstration of OmniCam-guided active perception. OmniCam plans a camera trajectory that actively repositions the observation camera to resolve target occlusion. and angular deviation. These reference-trajectory errors measure agreement with the generated supervision; multiple distinct trajectories can satisfy the same instruction. Implementation. OmniCam uses an OPT-style [35] decoder with FlashAttention-2 [9], trained in bf16 on 8×H20 GPUs for∼100 epochs. The base configuration (hidden dim 1024, 8 heads, 12 layers;∼500M trainable parameters) is used for the main comparisons; a larger variant (∼1B) is analyzed in Section E.2. Hyperparameters are provided in Section B. 5.2. Main Results Quantitative Comparison. On OmniCaT (Table 1), Om- niCam reduces ATE, FDE, RPE-R, and RPE-T by 46.6%, 41.1%, 28.0%, and 28.9% relative to retrained GenDoP, re- spectively. Collision decreases from 30.4% to 10.4% (65.8% relative reduction), and visibility rises from 33.6% to 52.0% (54.8% relative increase). Director3D is the best baseline for ATE and collision; relative to it, the reductions are 43.0% and 62.3%. The scene-bootstrap intervals for OmniCam and GenDoP are reported in Section M; component-specific effects are examined separately below. Zero-Shot Cross-Dataset Transfer. The reported DataDoP 7 Table 2. Ablation studies on OmniCaT. The dual-branch panoramic encoder, target injection components, and pose tokenization schemes are investigated. VariantATE↓FDE↓RPE-R↓RPE-T↓Coll. (%)↓Vis. (%)↑Angle ( ◦ )↓ Encoder w/o semantic branch 2.1863.2472.6940.07213.611.564.7 w/o geometric branch1.7743.0842.8810.04945.833.619.7 Geo. Input Panorama RGB only1.5362.6872.1430.04438.736.218.4 Panorama RGB-D1.2472.1181.6850.03622.642.814.7 Grounding w/o target cross-attention1.6732.2592.7740.04215.225.835.6 w/o target localization loss 1.5822.1482.6610.03814.832.428.7 w/o target cross-attn. & loss1.9242.5762.9030.05118.921.340.2 Tokenization All absolute (rot. + trans.)1.2781.8861.6250.05814.341.615.3 All relative (rot. + trans.)1.4491.9271.9840.06216.643.216.4 Hybrid w/o hemisphere consistency 1.0941.7561.8470.03312.246.113.8 FullOmniCam0.8681.4621.1350.02710.452.010.9 results favor OmniCam over GenDoP trained on DataDoP: ATE 0.925 vs. 1.437, collision 8.9% vs. 28.5%, visibility 64.2% vs. 36.6%, and angle 12.7 ◦ vs. 29.8 ◦ . OmniCam is reported to use no DataDoP fine-tuning. Establishing the extent of transfer requires the input adaptation, data-overlap checks, and metric protocol identified above; these aggregate values alone do not rule out planner bias or differences in available scene information. Comparison with the Heuristic Planner. With access to reconstructed geometry, navigation meshes, and semantic annotations, the planner has 5.2% collision and 58.7% visi- bility, compared with OmniCam’s 10.4% and 52.0%. This is a privileged-information reference, not a mathematical performance bound. The reported runtimes are 0.84s and approximately 16.5s (Section J); matched timing conditions require verification. Per-Behavior Analysis. Section D reports lower ATE and FDE for OmniCam in all four behavior categories. These results describe coverage of the evaluated behaviors without identifying which component causes each gain. Qualitative Comparison. Figure 4 compares trajectories across representative scenes. OmniCam trajectories remain aligned with navigable regions, follow the intended view- point of the referenced object, and exhibit smoother transi- tions in both position and orientation compared to all base- lines. Additional qualitative examples are provided in Sec- tion K. 5.3. Ablation Studies Dual-Branch Encoder. Removing the semantic branch re- duces visibility from 52.0% to 11.5%, while removing the geometric branch increases collision from 10.4% to 45.8%. This supports the usefulness of both pathways. Because branch removal also changes capacity and available informa- tion, a capacity-matched single-branch comparison is needed to isolate the effect of routing. Geometric Input Modality. RGB-only, RGB-D, and point- cloud variants yield collision rates of 38.7%, 22.6%, and 10.4%, respectively. The results favor explicit 3D processing in these configurations. Since the point cloud is unprojected from depth, this comparison concerns representation and processing, rather than additional geometric information created by unprojection. Target Centroid Injection. Removing target cross-attention changes visibility from 52.0% to 25.8% and angle from 10.9 ◦ to 35.6 ◦ . Removing the localization loss yields 32.4% and 28.7 ◦ ; removing both yields 21.3% and 40.2 ◦ , the worst values among these grounding variants. The comparison supports explicit target anchoring in the tested architecture (Section G). Hybrid Pose Encoding. Hybrid encoding obtains RPE- R 1.135 and RPE-T 0.027, compared with 1.625/0.058 for all-absolute and 1.984/0.062 for all-relative encoding. Re- moving sign alignment increases RPE-R to 1.847. The rep- resentation analysis in Section I concerns quantization and sign selection; it does not guarantee the accuracy of autore- gressive predictions. 5.4. Downstream Application Validation With WorldStereo [36], OmniCam trajectories improve the reported CLIP score by 14.1% and consecutive-frame SSIM by 15.4% relative to GenDoP. Figure 5 illustrates video and reconstruction outputs. Robotic active-perception results and a qualitative real-robot demonstration are presented in Section L and Fig. 6. The robot success-rate aggregation needs verification; the demonstration is evidence of feasibil- ity rather than a systematic sim-to-real evaluation. 6. Conclusion OmniCam generates camera trajectories from panoramic imagery and language using point-cloud conditioning, hy- brid pose serialization, and explicit 3D target anchoring. 8 OmniCaT supplies 267,700 examples across four camera behaviors. The reported comparisons and ablations sup- port these design choices within the evaluated settings, and downstream studies examine their practical use. The repre- sentation analysis distinguishes quaternion sign consistency from lossy quantization and identifies the error-accumulation tradeoff of relative encoding. Remaining limitations include estimated and incomplete geometry, planner-derived super- vision, static-scene observations, and the need for indepen- dently verified evaluation protocols. Future work includes dynamic scenes and closed-loop planning. References [1]Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in Atari. Advances in Neural Information Processing Systems, 37:58757–58791, 2024. [2] Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 15791–15801, 2025. [3]Jim Blinn.Where am i?what am i looking at?(cinematography). IEEE Computer Graphics and Applica- tions, 8(4):76–81, 2002. [4] Rogerio Bonatti, Wenshan Wang, Cherie Ho, Aayush Ahuja, Mirko Gschwindt, Efe Camci, Erdal Kayacan, Sanjiban Choudhury, and Sebastian Scherer. Autonomous aerial cine- matography in unstructured environments with learned artistic decision-making. Journal of Field Robotics, 37(4):606–641, 2020. [5]Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik. Long-term human motion prediction with scene context. In European Conference on Computer Vision, pages 387–404. Springer, 2020. [6]Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoub- hik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. SAM 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719, 2025. [7]Robin Courant, Nicolas Dufour, Xi Wang, Marc Christie, and Vicky Kalogeiton. ET the exceptional trajectories: Text-to- camera-trajectory generation with character awareness. In European Conference on Computer Vision, pages 464–480. Springer, 2024. [8]Qiongjie Cui, Huaijiang Sun, and Fei Yang. Learning dy- namic relationships for 3D human motion prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6519–6527, 2020. [9] Tri Dao.FlashAttention-2: Faster attention with bet- ter parallelism and work partitioning.arXiv preprint arXiv:2307.08691, 2023. [10]Steven M Drucker, Tinsley A Galyean, and David Zeltzer. Cinema: A system for procedural camera movements. In Pro- ceedings of the 1992 symposium on Interactive 3D graphics, pages 67–70, 1992. [11]Quentin Galvane, Marc Christie, Chrsitophe Lino, and Rémi Ronfard. Camera-on-rails: automated computation of con- strained camera paths. In Proceedings of the 8th ACM SIG- GRAPH Conference on Motion in Games, pages 151–157, 2015. [12]Michael Grupp. evo: Python package for the evaluation of odometry and SLAM.https://github.com/ MichaelGrupp/evo, 2017. [13]Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. CameraCtrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101, 2024. [14]Yunzhong Hou, Liang Zheng, and Philip Torr. Learning cam- era movement control from real-world drone videos. arXiv preprint arXiv:2412.09620, 2024. [15] Chong Huang, Chuan-En Lin, Zhenyu Yang, Yan Kong, Peng Chen, Xin Yang, and Kwang-Ting Cheng. Learning to film from professional human motion videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4244–4253, 2019. [16]Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al.π 0.5 : a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025. [17]Hongda Jiang, Bin Wang, Xi Wang, Marc Christie, and Bao- quan Chen. Example-driven virtual cinematography by learn- ing camera behaviors. ACM Trans. Graph., 39(4):45, 2020. [18]Hongda Jiang, Xi Wang, Marc Christie, Libin Liu, and Bao- quan Chen. Cinematographic camera diffusion model. Com- puter Graphics Forum, 43(2):e15055, 2024. [19]Michael Kazhdan, Thomas Funkhouser, and Szymon Rusinkiewicz. Rotation invariant spherical harmonic repre- sentation of 3 d shape descriptors. In Symposium on geometry processing, pages 156–164, 2003. [20] Beomyoung Kim, Chanyong Shin, Joonhyun Jeong, Hyungsik Jung, Se-Yun Lee, Sewhan Chun, Dong-Hyun Hwang, and Joonsang Yu. ZIM: Zero-shot image matting for anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23828–23838, 2025. [21]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. OpenVLA: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024. [22]Soohwan Kim and Minkyoung Kim. Rotation representations and their conversions. Ieee Access, 11:6682–6699, 2023. [23] Xinyang Li, Zhangyu Lai, Linning Xu, Yansong Qu, Liujuan Cao, Shengchuan Zhang, Bo Dai, and Rongrong Ji. Direc- tor3D: Real-world camera trajectory and 3D scene generation from text. Advances in neural information processing systems, 37:75125–75151, 2024. [24] Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. DL3DV-10K: A large-scale scene dataset for deep learning- based 3D vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22160– 22169, 2024. 9 [25]Christophe Lino and Marc Christie. Intuitive and efficient camera control with the toric space. ACM Transactions on Graphics (TOG), 34(4):1–12, 2015. [26]Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. RDT-1B: a diffusion foundation model for bimanual manipu- lation. arXiv preprint arXiv:2410.07864, 2024. [27]Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. In Euro- pean conference on computer vision, pages 38–55. Springer, 2024. [28] Xinyi Liu, Tianyi Zhang, Matthew Johnson-Roberson, and Weiming Zhi. SplaTraj: Camera trajectory generation with semantic gaussian splatting. arXiv preprint arXiv:2410.06014, 2024. [29] Sebastian Pütz, Thomas Wiemann, Jochen Sprickerhof, and Joachim Hertzberg. 3D navigation mesh generation for path planning in uneven terrain. Ifac-Papersonline, 49(15):212– 217, 2016. [30]Fabio Ruetz, Emili Hernández, Mark Pfeiffer, Helen Oleynikova, Mark Cox, Thomas Lowe, and Paulo Borges. OVPC mesh: 3D free-space representation for local ground vehicle navigation. In 2019 International Conference on Robotics and Automation (ICRA), pages 8648–8654. IEEE, 2019. [31] Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025. [32]Ruicheng Wang, Sicheng Xu, Yue Dong, Yu Deng, Jianfeng Xiang, Zelong Lv, Guangzhong Sun, Xin Tong, and Jiaolong Yang. MoGe-2: Accurate monocular geometry with metric scale and sharp details. arXiv preprint arXiv:2507.02546, 2025. [33]Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. MVImgNet: A large-scale dataset of multi-view images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9150–9161, 2023. [34]Mengchen Zhang, Tong Wu, Jing Tan, Ziwei Liu, Gordon Wetzstein, and Dahua Lin. GenDoP: Auto-regressive camera trajectory generation as a director of photography. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 18229–18239, 2025. [35] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained trans- former language models. arXiv preprint arXiv:2205.01068, 2022. [36]Yisu Zhang, Chenjie Cao, Tengfei Wang, Xuhui Zuo, Junta Wu, Jianke Zhu, and Chunchao Guo. WorldStereo: Bridging camera-guided video generation and scene reconstruction via 3D geometric memories. arXiv preprint arXiv:2603.02049, 2026. [37]Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely.Stereo magnification:Learning view synthesis using multiplane images. arXiv preprint arXiv:1805.09817, 2018. [38] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5745–5753, 2019. 10 A. Implementation Details of OmniCaT A.1. Geometric Initialization and Refinement Depth Alignment Optimization. During geometry initial- ization, the ERP space is subdivided into 42 perspective views (an increase from the default 12) to ensure dense coverage. Depth maps are aligned via a GPU-accelerated Least-Squares Minimal Residual (LSMR) solver. To miti- gate artifacts, a grounding pipeline is employed to mask sky regions and to remove depth discontinuities (edge floaters). NavMesh Engineering. The raw NavMesh generated by the Recast algorithm is configured with cell size 0.1 m, cell height 0.1 m, agent height 0.1 m, agent radius 0.05 m, maximum climb 0.05 m, and maximum slope 30 ◦ . Post- processing consists of: (1) height filtering to remove poly- gons above a roof threshold; (2) connectivity filtering to retain only the largest connected component; (3) surface snapping through dense ray-casting to correct misaligned vertices that lie below the actual mesh surface; and (4) bound- ary erosion using KD-Tree accelerated searches to maintain a safety buffer from mesh edges. The resulting navigation graph is constructed by sampling points on the NavMesh surface at 0.05 m spacing (up to 600K nodes), connecting neighbors within5×spacing with a height coherence con- straint (|∆y| ≤ 0.5m), and computing shortest paths via Dijkstra from the camera origin. A.2. Trajectory Planning Heuristics Scoring Function for Target Paths. For target trajectories, 72 candidate nodes are sampled on a circular boundary at radiusr = min(1.5× s obj ,d obj ), wheres obj is the object scale andd obj is the distance from the camera origin to the target. The optimal node is selected by maximizing a scoring function:S = 5000· R los − 100· ∆θ, whereR los is the Line-of-Sight ratio calculated via a 50-point projection test between the candidate position and the target center, and∆θ is the angular deviation from the ideal approach direction. Paths are computed via Dijkstra on the NavMesh graph, pruned of backtracking segments, and smoothed with B- spline interpolation (smoothing factor 0.5). All trajectories are resampled to 81 keyframes. Surround Behavior. For surround trajectories, the orbit radius is computed asr = min(d obj , 2× s obj ), clamped by a scene-dependent threshold (4×global median depth). 72 points are sampled on the orbit circle; only those reachable via the NavMesh graph are connected to form continuous surround paths. Exploration Behavior. For wander trajectories, 8 radial directions are explored from the origin along the NavMesh, each extending up to4×global median depth in path length. The top 10 trajectories are selected via Farthest Point Sam- pling on endpoint positions, with a minimum angular sepa- ration of 40 ◦ and maximum overlap ratio of 0.5. Reconstruction Behavior. Reconstruction targets are se- lected via the top-10 semantic objects ranked by segment area and viewpoint diversity (FPS in 3D space). For each tar- get, multi-view paths are planned with a reduced smoothing factor (0.2) to preserve sharp viewpoint transitions optimal for 3D reconstruction coverage. Path Smoothing and Orientation. All discrete waypoints are smoothed using B-splines and Gaussian filtering (σ = 8.0). Camera orientations are managed via Spherical Linear Interpolation (SLERP). For “Wander” and “Target” modes, orientations follow the path tangent and target centroid, re- spectively. B. Training and Inference Details The reported tokenizer usesB = 256bins and a normalized displacement limit∆t max = 0.025. These settings deter- mine precision and clipping tradeoffs; sign alignment does not remove quantization boundaries. OmniCam is trained with the Accelerate framework under bf16 mixed precision. The autoregressive backbone is an OPT-style decoder equipped with FlashAttention-2. The optimizer is AdamW withβ = (0.9, 0.95), a weight decay of0.01, a peak learning rate of3× 10 −5 , a linear warm- up over the first one percent of the schedule, and cosine annealing for the remaining steps. The gradient is clipped to a global norm of1.0and gradient checkpointing is enabled throughout the decoder. The discrete bin size isB = 256, and the maximum translation delta is∆t max = 0.025in the normalized coordinate system. The hidden dimension, number of heads, and number of layers are configured as the base variant (hidden dim 1024, 8 heads, 12 layers). Training is performed for approximately one hundred effective epochs on a single node with eight NVIDIA H20 GPUs, with a per- GPU batch size of thirty-two. The localization loss weight is set toλ = 0.5. Inference adopts the sample strategy with top-ktruncation atk = 10and with the format-constrained decoding procedure described in Section 4.4. Weighted Cross-Entropy Loss.The per-token cross- entropy loss is weighted by three multiplicative factors to em- phasize the most critical pose channels and learning phases: (1) channel importance weights: rotation channels receive weight 2.0, translation channels 1.5, and intrinsic channels 0.5, reflecting the relative importance of orientation accuracy over positional precision; (2) temporal decay: a linear sched- ule assigns weight 2.0 to the first frame and decreases to 1.0 at the final frame, prioritizing early trajectory establishment; and (3) curriculum learning mask: during the initial training phase, only early frames contribute to the loss, with later frames gradually enabled as training progresses, preventing the model from attempting to learn long-horizon dynamics before mastering short-horizon patterns. Caption Dropout Probabilities. The five-variant caption dropout strategy applies the following distribution: full cap- 11 tion (35%), prefix tags only (25%), body text only (25%), motion tag only (5%), and partial prefix (10%). This is the distribution used in the reported experiments. Computational Cost. Table 3 reports training and inference costs for GenDoP and two OmniCam configurations. The original timing description includes depth estimation, point- cloud construction, encoding, and autoregressive decoding. The prefix condition is reused across decoding steps. These reported timings do not establish real-time operation under an unspecified deployment budget. C. Encoder Architecture Details The dual-branch encoder routes appearance and text through a semantic pathway and point-cloud features through a ge- ometry pathway. The outputs are fused by the decoder. Semantic Branch. The panoramic imageI pano is processed by a frozen SAM 3 vision encoder with 1024-dimensional features. The target descriptorT obj and caption guide the Language-Guided Resampler after visual feature extraction. This pathway provides target-related information, but it is not excluded from contributing to geometric decisions after decoder fusion. Geometric Branch. Panoramic geometric priors are un- projected into a 3D point cloud, which is then structured into a voxel grid (0.02m resolution) to enable efficient feature extraction. A LitePT encoder uses positional and surface-normal features to extract per-voxel descriptors via sparse convolution, which are re-projected to yieldF geo ∈ R M×C geo . This branch has no direct text input. Removing it increases collision rate from 10.4% to 45.8% in Table 2; this supports the usefulness of the geometric pathway but does not isolate routing from capacity or input-information changes. D. Per-Behavior Decomposition Table 4 decomposes the OmniCaT results. Compared with retrained GenDoP, OmniCam has lower ATE for Wander (1.150 vs. 2.421), Target (0.677 vs. 1.088), Surround (1.258 vs. 1.932), and Reconstruct (0.385 vs. 1.055). The corre- sponding Target and Reconstruct reductions are 37.8% and 63.5%. This comparison supports performance across the four evaluated behaviors; it does not isolate caption dropout, loss weighting, or geometry as the cause of individual gains. E. Scaling Analysis We examine the reported dependence on training-data vol- ume and model size. These finite comparisons show trends within the evaluated range; they do not establish a scaling law or the absence of representational bottlenecks. E.1. Data Scaling Table 5 reports the base model at four nominal data scales, labeled 27K, 53K, 107K, and 267K in the original experi- ment record. Metrics improve across these settings: ATE de- creases from 1.483 to 0.868, collision from 19.8% to 10.4%, and visibility increases from 34.6% to 52.0%. Four data points do not determine a power law or establish gains be- yond the observed range. E.2. Model Scaling Three model sizes are compared: Small (hidden dimension 512, 8 heads, 8 layers; approximately 120M trainable param- eters), Base (1024, 8, 12; approximately 500M), and Large (1536, 16, 24; approximately 1B). The reported setup holds other hyperparameters constant while changing architecture dimensions. ATE decreases from 1.127 to 0.743, collision from 13.8% to 8.7%, and visibility increases from 44.6% to 56.3% (Table 6). These observations support gains at the tested sizes, without establishing which representational choices explain the trend. F. Failure Case Analysis Understanding when and why the framework fails is essen- tial for guiding future improvements. Five categories of failures are identified, each linked to a specific component or assumption of the pipeline: • Noisy panoramic geometry / thin structures (geometric branch limitation): in large open environments or near thin structures (fences, poles, railings), the monocular depth es- timator [32] produces noisy or incomplete reconstructions with floating artifacts. These propagate into the LitePT voxel features, causing the geometric branch to either miss collision boundaries or hallucinate obstacles, resulting in physically implausible paths. •Ambiguous target phrases (centroid injection limitation): when multiple semantically similar objects coexist (e.g., “the wooden door” in a hallway with several doors, or “the pillar” among clustered columns), the attention-aggregated centroid may ground to an incorrect instance. Spatial descriptors in the textual guidance partially mitigate this, but the failure persists when such cues are absent or when objects are visually indistinguishable from the panoramic viewpoint. • Dynamic and interactive scenes (static-scene assump- tion): the current framework assumes a static panoramic observation. In environments with moving objects (people, vehicles, opening doors), the generated trajectory cannot react to temporal changes, potentially producing paths that collide with dynamic obstacles or fail to track mov- ing targets. Extending OmniCam to video-conditioned or closed-loop settings is an important direction for future work. 12 Table 3. Reported computational costs for GenDoP and OmniCam Base/Large. MethodTraining TimeInference Latency#Params GenDoP∼18h (8×H20)0.42s / trajectory ∼380M OmniCam (Base)∼24h (8×H20)0.84s / trajectory ∼500M OmniCam (Large) ∼40h (8×H20)1.20s / trajectory ∼1B Table 4. Per-behavior decomposition of the main result on OmniCaT. Metrics are reported separately for Wander, Target, Surround, and Reconstruct trajectories. MethodDataset WanderTargetSurroundReconstruct ATE↓FDE↓ATE↓FDE↓ATE↓FDE↓ATE↓FDE↓ CCD [18]Pre-trained3.7184.9071.4852.2852.5243.4192.2062.972 E.T. [7]Pre-trained3.1514.0831.7351.9451.5801.6401.4671.804 Director3D [23] Pre-trained2.9694.7791.4442.2071.6202.6881.2792.094 Director3DOmniCaT2.5624.2501.1302.0891.3512.3041.0451.950 GenDoP [34]Pre-trained2.6674.4081.3622.1312.1763.2481.2552.016 GenDoPOmniCaT2.4213.6521.0881.9331.9322.7731.0551.574 OmniCam (Ours)OmniCaT1.1502.3010.6770.9621.2581.8360.3850.749 •Reflective/transparent surfaces (semantic branch limita- tion): glass walls and mirrors produce ambiguous visual features in the vision encoder, leading to unreliable target- aware conditioning and erratic trajectory orientation near such surfaces. •Occluded targets (centroid injection limitation): when the target object is heavily occluded, the cross-attention weights spread to visible but irrelevant regions, causing the estimated centroid to shift away from the true target location. These cases identify limitations in perception and target grounding. Decoder errors, quantization, and exposure to planner-generated supervision can also contribute; the exam- ples do not isolate their relative importance. Broader Impacts. Scene-aware camera planning may assist content creation and robotic perception. Camera deployment can also raise privacy or surveillance concerns, and gener- ated video may be used misleadingly. In robotics, errors in inferred geometry or open-loop trajectories can produce unsafe camera motions; the reported collision metrics are not deployment guarantees. Dataset release requires source- specific permission and license review, and physical deploy- ment requires validation against the actual environment. G. Attention Visualization and Centroid Analy- sis The target centroid supplies an explicit spatial anchor (Sec- tion 4.3). The following visualizations illustrate the attention patterns used to derive this anchor. Figure 7 shows the input panorama, reconstructed point cloud, and attention projected onto scene points. The ex- amples show concentration near referenced regions. These visualizations complement the grounding ablation, but do not quantify localization accuracy or establish that attention alone causes improved visibility. Point-level aggregation is specified conceptually in Equation (2); its implementation details require verification. H. Caption Dropout Ablation We compare no caption dropout, single-variant dropout (dropping the entire caption with probability 0.1), and the five-variant distribution in Section B. The reported values are means over three training seeds, with ATE standard deviation below 0.02. The five-variant scheme has higher visibility than no dropout (52.0% vs. 38.7%) and lower angular devia- tion (10.9 ◦ vs. 18.4 ◦ ), while no dropout has slightly lower ATE (0.856 vs. 0.868) and collision (10.1% vs. 10.4%). This is a tradeoff between the reported metrics. Attribution to instruction-format overfitting requires separate evaluation on complete, partial, and rephrased instructions. I. Theoretical Analysis of Hybrid Pose Tok- enization We analyze reconstruction of a known trajectory after quan- tization and dequantization. These results concern repre- sentation error; they do not bound autoregressive prediction error, depth-estimation error, or errors in scene-scale calibra- tion. Letddenote the rotation-angle geodesic distance on SO(3), measured in radians, with range[0,π]. This distance 13 Table 5. Reported effect of nominal data scale using the base architecture. Exact training-subset counts and the fixed evaluation split require verification. #TrajectoriesATE↓FDE↓RPE-R↓RPE-T↓Coll. (%)↓Vis. (%)↑ 27K (10%)1.4832.5861.8360.05419.834.6 53K (20%)1.2062.0801.5280.04215.842.6 107K (40%) 1.0031.7091.2970.03312.748.2 267K (100%)0.8681.4621.1350.02710.452.0 Table 6. Reported effect of model scale. The common training subset and parameter-count scope require verification against training records. Size#ParamsATE↓FDE↓RPE-R↓RPE-T↓Coll. (%)↓Vis. (%)↑ Small∼120M1.1271.8931.4280.03613.844.6 Base ∼500M0.8681.4621.1350.02710.452.0 Large∼1B0.7431.2480.9820.0228.756.3 Door Door Pathway StreetLight MountainBuilding Figure 7. Attention visualization of the Language-Guided Semantic Resampler. Each group shows (from left to right) the input panorama, the reconstructed point cloud, and the 3D attention map. The maps visualize target-related attention on reconstructed geometry for the displayed examples; they are qualitative illustrations rather than a quantitative localization evaluation. Table 7. Caption-dropout ablation. The five-variant scheme favors visibility and angle, while no dropout has slightly lower geometric errors. Dropout StrategyATE↓ FDE↓RPE-R↓ RPE-T↓Coll. (%)↓Vis. (%)↑ Angle ( ◦ )↓ No dropout0.8561.4381.1180.02610.138.718.4 Single-variant0.8621.4511.1260.02710.345.814.2 Full (5-variant)0.8681.4621.1350.02710.452.010.9 14 is invariant under left and right multiplication by a rotation. Proposition 1 (Deterministic bounds on rotation reconstruc- tion error). LetR 1 ,...,R T ∈ SO(3)and∆R t = R −1 t−1 R t fort ≥ 2. Suppose decoded relative rotations satisfy d( d ∆R t , ∆R t ) ≤ ε t , and the decoded initial rotation sat- isfiesd( b R 1 ,R 1 )≤ e 1 . Reconstruct relative orientations by b R t = b R t−1 d ∆R t . Then d( b R t ,R t )≤ min π, e 1 + t X j=2 ε j .(4) If b R 1 = R 1 and everyε j ≤ ε, this bound becomes minπ, (t− 1)ε. In contrast, if absolute orientations are decoded with per-frame reconstruction error at mostε, then d( b R abs t ,R t )≤ εat every frame, without integrating earlier quantization residuals. Proof. The triangle inequality and bi-invariance give d( b R t ,R t )≤ d( b R t−1 d ∆R t , b R t−1 ∆R t ) + d( b R t−1 ∆R t ,R t−1 ∆R t )(5) = d( d ∆R t , ∆R t ) + d( b R t−1 ,R t−1 ).(6) Iterating this inequality proves the sum bound. Every rotation-angle distance is at mostπ, giving the stated mini- mum. The absolute-encoding bound follows directly from its assumed per-frame reconstruction guarantee. This is a worst-case upper bound, not a claim that relative errors necessarily grow at each frame or obey a particular expected growth rate. Residuals can cancel. Absolute en- coding avoids integration of rotation quantization residuals, but predicted absolute tokens can still depend on previous generated tokens and therefore need not have independent prediction errors. Connecting component quantization to angular error. The boundεdepends on the quantizer and the rotation decoder;Bcomponent bins do not by themselves imply ε = π/B. For a conditional example, letq ∈S 3 be a unit quaternion, and suppose component dequantization produces vwith∥v− q∥ 2 ≤ η < 1. If the decoder normalizesvto bq = v/∥v∥ 2 , then d(R(bq),R(q))≤ 2 arcsinη.(7) Indeed,q ⊤ v > 0, so the angleφbetweenqandvlies in [0,π/2). Decomposing the error alongbqgives∥v− q∥ 2 2 = (∥v∥ 2 − cosφ) 2 + sin 2 φ , henceφ ≤ arcsinη. The corre- sponding rotation angle is2φ. For uniform bins of width h, center decoding without clipping gives at mosth/2error per quaternion component and thusη ≤ h. In particular, B > 2equal bins on[−1, 1]give the conditional bound 2 arcsin(2/B). Applying such a numerical bound requires these decoder and range assumptions to hold. Proposition 2 (Representative-independent quaternion sign selection). LetR 1 ,...,R T satisfyd(R t−1 ,R t ) < πfor everyt ≥ 2, and choose arbitrary unit-quaternion repre- sentativesq t ofR t . Forq = (w,x,y,z), letc(q)select the sign for which the first nonzero component in the order (w,x,y,z) is positive. Define eq 1 = c(q 1 ), eq t = ( q t , ⟨eq t−1 ,q t ⟩ > 0, −q t , ⟨eq t−1 ,q t ⟩ < 0, t≥ 2. (8) This produces a unique aligned quaternion sequence deter- mined by the rotation sequence, independently of the initial choices betweenq t and−q t . Consequently, any determinis- tic component quantizer with fixed bin-boundary rules pro- duces the same token sequence from all such choices of input representatives. The tokenization remains many-to-one. Proof.A unit quaternion has a nonzero component, and c(q) = c(−q), so the first representative is uniquely fixed, including whenw 1 = 0. Suppose the aligned representative att− 1is fixed. The quaternion formula for rotation distance gives |⟨eq t−1 ,q t ⟩| = cos d(R t−1 ,R t ) 2 > 0. Exactly one ofq t and−q t therefore has positive inner prod- uct with the already aligned previous quaternion. Replacing the inputq t by−q t reverses the inner-product sign and leaves the selected representative unchanged. Induction proves rep- resentative independence of the entire sequence, and deter- ministic quantization preserves this property. It does not preserve injectivity: a finite token vocabulary cannot distin- guish every continuous rotation sequence. The proposition specifies sufficient sign and boundary conventions; a concrete implementation must document these choices. At an exactlyπinter-frame rotation, both inner products are zero and an additional deterministic tie rule is required, such as selectingc(q t ). No shortest-arc pref- erence exists at that tie. This sequence-level sign selection removes double-cover ambiguity under the stated conven- tions; it does not construct a globally continuous quaternion parameterization ofSO(3), remove quantization boundaries, or make lossy quantization invertible. Translation: local resolution versus accumulated error. For additive displacements in a common coordinate frame, relative translation trades a smaller encoding range for in- tegration error. If normalized absolute coordinates lie in [−L,L]and displacements lie in[−D,D]per component, uniform center decoding withBbins and no clipping bounds 15 Table 8. Privileged-information planner comparison. The planner receives reconstructed geometry, navigation meshes, and semantic annotations. OmniCam estimates geometry from the input panorama and text. MethodNavMesh/annotation accessRuntimeColl. (%)↓Vis. (%)↑Angle ( ◦ )↓ Heuristic PlannerYes (privileged)∼16.5s5.258.79.4 GenDoP [34]No0.42s30.433.627.6 OmniCam (Ours)No0.84s10.452.010.9 the per-frame absolute position error by √ 3L/Band each displacement error by √ 3D/B. Integrating the latter gives ∥ b p t − p t ∥ 2 ≤∥ b p 1 − p 1 ∥ 2 + (t− 1) √ 3D B .(9) ThusD ≪ Limproves local displacement resolution, while long sequences can accumulate translation error. If displace- ments are expressed in rotating local frames, orientation errors must also be included. The hybrid representation is therefore a design tradeoff whose trajectory-level bene- fit requires empirical evaluation, rather than a guarantee of universally minimal pose error. J. Heuristic Planner Oracle Comparison The heuristic planner is evaluated with reconstructed point clouds, navigation meshes, semantic masks, and target an- notations. This is a privileged-information comparison on estimated geometry, rather than an oracle with complete physical ground truth. It attains 5.2% collision and 58.7% visibility, versus OmniCam’s 10.4% and 52.0%. OmniCam estimates geometry internally from the panorama and does not run navmesh construction or graph search. The reported runtimes (approximately 16.5s vs. 0.84s) differ by a factor of 19.6; the timing assumptions must be matched before attributing the difference to a particular component. K. More Qualitative Results Figure 8 provides additional qualitative trajectory compar- isons. Each row shows one scene, and the columns show the outputs of different methods. These selected examples illustrate differences in trajectory placement and orientation; they do not establish dataset-wide collision or smoothness guarantees. Figure 9 presents additional downstream examples from indoor, outdoor, and stylized environments. Each row con- tains a panorama, selected generated video frames, and re- construction results with and without the OmniCam trajec- tory. The figure illustrates qualitative differences; reconstruc- tion accuracy and completeness require a separate quantita- tive evaluation. L. Downstream Application Validation Camera-Controlled Video Generation. OmniCam and GenDoP trajectories serve as camera control signals for WorldStereo [36]. The reported evaluation uses 484 tra- jectories across indoor, outdoor, and stylized scenes. For each prompt, the resulting poses drive point-cloud rendering and subsequent video synthesis. Generated videos are evaluated with five reported mea- sures. CLIP-Score averages the cosine similarity between CLIP ViT-B/32 image and scene-description embeddings over 10 sampled frames. Consecutive-frame SSIM measures adjacent-frame similarity and is sensitive to motion magni- tude; it is not a motion-compensated temporal-consistency metric. Aesthetic proxy combines CLIP embedding mag- nitude and inter-frame embedding coherence on a reported 0–10 scale. Its formula and normalization require verifica- tion. Known-region Fidelity compares generated frames with point-cloud renderings in observed regions using SSIM, so it uses a rendering reference. Generation Coverage is the mean unknown-region mask ratio, a descriptor of synthesis difficulty rather than an accuracy metric. Relative to GenDoP, the reported OmniCam results im- prove CLIP score by 14.1%, consecutive-frame SSIM by 15.4%, the aesthetic proxy by 18.0%, and known-region fi- delity by 9.2%. The unknown-region mask ratio is 38.5% vs. 35.8%. These results describe outputs of the trajectory- plus-video pipeline; they do not isolate framing, path length, speed, or geometric consistency as the cause of improvement. Quantitative reconstruction accuracy and completeness are not evaluated in this table. Robotic Manipulation (Active Perception). OmniCam is evaluated as an active-perception module forπ 0.5 [16], generating camera trajectories to improve target observations. A trial succeeds when the designated object is grasped within the time limit. The original record describes 200 episodes across 50 tabletop scenes, with reported success rates of 72.5% for OmniCam, 48.0% for GenDoP, and 56.3% for a fixed camera. The fixed-camera percentage is incompatible with a single pooled denominator of 200 binary trials; its aggregation remains unresolved. Figure 6 presents qualitative physical-robot grasping ex- amples with initially occluded targets. The demonstration illustrates feasibility in the shown cases. A systematic sim- 16 CCDDirector3DGenDoPOmniCam (Ours)Scenes Figure 8. Additional qualitative comparison on panoramic scenes. Extending the comparison in Figure 4, each row shows a different scene spanning indoor, outdoor, and stylized environments, and each column shows the trajectory produced by a specific method. The examples illustrate differences in target-directed trajectories across scene categories. The trajectory captions used for each scene are listed in Table 9. 17 Table 9. Trajectory captions for Figure 8. Each row corresponds to a scene in the qualitative comparison figure, listing the scene type, target object, and the hierarchical textual instruction provided to OmniCam. RowTypeTargetCaption 1Targetdoor<TASK: TARGET> <TARGET_OBJ: door> <TARGET_DIR: Back>Approach the back door to bring it into focus. The sequence concludes by arriving at the final <END_OBJ: door> position within the <END_DIR: Back> sector of the panoramic scene. 2Targetlight fixture<TASK: TARGET> <TARGET_OBJ: light fixture> <TARGET_DIR: Front> Approach the front light fixture to bring it into focus. The sequence concludes by arriving at the final <END_OBJ: light fixture> position within the <END_DIR: Front> sector of the panoramic scene. 3Wanderfree space<TASK: WANDER> <TARGET_OBJ: free space> <TARGET_DIR: Front> Approach the front workstation to explore the layout of the industrial corridor. The sequence concludes by arriving at the final <END_OBJ: free space> position within the <END_DIR: Front> sector of the panoramic scene. 4Targetdoor<TASK: TARGET> <TARGET_OBJ: door> <TARGET_DIR: Back>Approach the back door to bring it into focus. The sequence concludes by arriving at the final <END_OBJ: door> position within the <END_DIR: Back> sector of the panoramic scene. 5Targetice tunnel<TASK: TARGET> <TARGET_OBJ: ice tunnel> <TARGET_DIR: Front> Approach the ice tunnel to bring it into focus. The sequence concludes by arriving at the final <END_OBJ: ice tunnel> position within the <END_DIR: Front> sector of the panoramic scene. 6Wanderfree space<TASK: WANDER> <TARGET_OBJ: free space> <TARGET_DIR: Front> Approach the distant peaks to explore the mountain valley. The sequence concludes by arriving at the final<END_OBJ: free space> position within the <END_DIR: Front> sector of the panoramic scene. 7Surroundclouds<TASK: SURROUND> <TARGET_OBJ: clouds> <TARGET_DIR: Front> Capture the clouds from multiple angles to highlight their form and color against the twilight sky. The sequence concludes by arriving at the final <END_OBJ: clouds> position within the <END_DIR: Front> sector of the panoramic scene. 8Surroundneon sign<TASK: SURROUND> <TARGET_OBJ: neon sign> <TARGET_DIR: Right> Capture the right neon sign from multiple angles to highlight its glowing details and reflections. The sequence concludes by arriving at the final <END_OBJ: neon sign> position within the <END_DIR: Right> sector of the panoramic scene. 9Wanderfree space<TASK: WANDER> <TARGET_OBJ: free space> <TARGET_DIR: Right> Approach the right to explore the surreal landscape beyond the pagoda. The sequence concludes by arriving at the final <END_OBJ: free space>position within the<END_DIR: Right>sector of the panoramic scene. 10Surroundsand dune<TASK: SURROUND> <TARGET_OBJ: sand dune> <TARGET_DIR: Front> Capture the sand dune from multiple angles to reveal its textured ridges and surrounding lanterns. The sequence concludes by arriving at the final <END_OBJ: sand dune> position within the <END_DIR: Front> sector of the panoramic scene. Table 10. Downstream evaluation: camera-controlled video generation with WorldStereo [36]. Different trajectory sources are compared using the same video generation model. MethodCLIP-S↑Adj.-SSIM↑Aest. proxy↑Fidelity↑Coverage GenDoP [34] + WorldStereo0.26440.42487.6470.692435.8% OmniCam (Ours) + WorldStereo0.30180.49049.0210.756038.5% 18 Panorama 3D Reconstruction w/o OmniCam Traj. 3D Reconstruction w/ OmniCam Traj. Camera-Controlled Video Generation with OmniCam Traj. Figure 9. Qualitative downstream results across diverse panoramic scenes. Each row shows the input panorama (left), four selected frames from camera-controlled video generation using an OmniCam trajectory (middle), and 3D reconstruction results without and with the OmniCam trajectory (two rightmost columns). The examples include indoor, outdoor, and stylized environments. These visual comparisons illustrate the pipeline outputs; they do not quantify reconstruction accuracy or guarantee collision-free motion. 19 Table 11. Downstream evaluation: robotic manipulation withπ 0.5 . Reported grasp success (SR) and execution visibility (Vis.). The stated 200-episode setup and fixed-camera SR aggregation require reconciliation with the episode records. Trajectory SourceGrasp SR (%)↑Exec. Vis. (%)↑ π 0.5 [16] (fixed camera)56.347.2 GenDoP [34] 48.052.6 OmniCam (Ours)72.571.8 to-real claim would require repeated physical trials, matched fixed-camera controls, and documented camera motions and failure counts. M. Statistical Confidence Analysis Table 12 reports 95% confidence intervals from 1,000 boot- strap resamples of OmniCaT evaluation scenes for Omni- Cam and retrained GenDoP. The listed intervals do not over- lap. These estimates characterize variation across evaluation scenes, not variation across training seeds, and do not cover all other experiments. GenDoP is the comparison in this table, but is not the best baseline for every main-table metric. 20 Table 12. Bootstrap 95% confidence intervals on OmniCaT evaluation scenes (1,000 resamples). Intervals describe evaluation-scene variability for the two listed methods. MethodATE↓FDE↓RPE-R↓RPE-T↓Coll. (%)↓Vis. (%)↑Angle ( ◦ )↓ GenDoP (OmniCaT)1.624±0.0872.483±0.1241.577±0.0680.038±0.00330.4±1.833.6±2.127.6±1.4 OmniCam (Ours)0.868±0.0421.462±0.0711.135±0.0530.027±0.00210.4±1.152.0±1.910.9±0.8 21