Paper deep dive

PhotoAgent: A Robotic Photographer with Spatial and Aesthetic Understanding

Lirong Che, Zhenfeng Gan, Yanbo Chen, Junbo Tan, Xueqian Wang

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 42

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 3/26/2026, 1:39:12 AM

Summary

PhotoAgent is an embodied robotic photography system that bridges the semantic gap between high-level language commands and geometric control. It utilizes a two-stage pipeline: an LMM-driven intention parser that translates aesthetic goals into geometric constraints, and a closed-loop reflective reasoning module using 3D Gaussian Splatting (3DGS) for photorealistic internal simulation. This approach enables the agent to perform 'mental simulation' to refine camera poses, achieving superior spatial reasoning and image quality without costly physical trial-and-error.

Entities (4)

3D Gaussian Splatting · technology · 100%PhotoAgent · robotic-system · 100%Large Multimodal Models · ai-model · 98%Chain-of-Thought · reasoning-framework · 95%

Relation Signals (3)

PhotoAgent → employs → Large Multimodal Models

confidence 100% · PhotoAgent first translates subjective aesthetic goals into solvable geometric constraints via LMM-driven, chain-of-thought (CoT) reasoning

PhotoAgent → utilizes → 3D Gaussian Splatting

confidence 100% · This initial pose is then iteratively refined through visual reflection within a photorealistic internal world model built with 3D Gaussian Splatting (3DGS).

Large Multimodal Models → implements → Chain-of-Thought

confidence 95% · PhotoAgent first translates subjective aesthetic goals into solvable geometric constraints via LMM-driven, chain-of-thought (CoT) reasoning

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Embodied agents for creative tasks like photography must bridge the semantic gap between high-level language commands and geometric control. We introduce PhotoAgent, an agent that achieves this by integrating Large Multimodal Models (LMMs) reasoning with a novel control paradigm. PhotoAgent first translates subjective aesthetic goals into solvable geometric constraints via LMM-driven, chain-of-thought (CoT) reasoning, allowing an analytical solver to compute a high-quality initial viewpoint. This initial pose is then iteratively refined through visual reflection within a photorealistic internal world model built with 3D Gaussian Splatting (3DGS). This ``mental simulation'' replaces costly and slow physical trial-and-error, enabling rapid convergence to aesthetically superior results. Evaluations confirm that PhotoAgent excels in spatial reasoning and achieves superior final image quality.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

42,010 characters extracted from source content.

Expand or collapse full text

PhotoAgent: A Robotic Photographer with Spatial and Aesthetic Understanding Lirong Che1∗, Zhenfeng Gan1∗, Yanbo Chen1, Junbo Tan1†, Xueqian Wang1 This work was supported by the Natural Science Foundation of Shenzhen (No. JCYJ20230807111604008, No. JCYJ20240813112007010), the Natural Science Foundation of Guangdong Province (No. 2024A1515010003) and Cross-disciplinary Fund for Research and Innovation (No. JC2024002) of Tsinghua SIGS.*indicates equal contribution.1Center for Artificial Intelligence and Robotics, Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China, clr24@mails., gzf24@mails., cyb23@mails., tjblql@sz., wang.xq@sz.tsinghua.edu.cn†Corresponding author: Junbo Tan Abstract Embodied agents for creative tasks like photography must bridge the semantic gap between high-level language commands and geometric control. We introduce PhotoAgent, an agent that achieves this by integrating Large Multimodal Models (LMMs) reasoning with a novel control paradigm. PhotoAgent first translates subjective aesthetic goals into solvable geometric constraints via LMM-driven, chain-of-thought (CoT) reasoning, allowing an analytical solver to compute a high-quality initial viewpoint. This initial pose is then iteratively refined through visual reflection within a photorealistic internal world model built with 3D Gaussian Splatting (3DGS). This “mental simulation” replaces costly and slow physical trial-and-error, enabling rapid convergence to aesthetically superior results. Evaluations confirm that PhotoAgent excels in spatial reasoning and achieves superior final image quality. I Introduction Endowing embodied agents with the ability to seamlessly collaborate with humans on creative tasks is a long-standing pursuit in robotics and artificial intelligence. Among creative domains, photography presents an ideal yet challenging testbed, as it deeply intertwines technical execution with subjective aesthetics. A successful photographer must comprehend not only the geometric properties of the world, such as occlusion and perspective, but also higher-level abstract intentions, like capturing “a dramatic photo”. Early “robot photographers” hard-coded the rule-of-thirds yet falter outside curated scenes [3, 17]. Later methods split into two brittle camps. Reinforcement-learning treats viewpoint choice as black-box search [2, 14], but fusing geometric constraints and aesthetic semantics into one reward demands costly, environment-specific interaction data. Imitation systems such as PhotoBot simply retrieve and copy reference photos [19], amounting to template matching that cannot span the combinatorial diversity of novel scenes. Neither line bridges the fundamental semantic-to-geometric gap. How can we unlock genuine creativity in robotic photography? Recent studies indicate that pretrained Large Multimodal Models (LMMs) already encode human-aligned aesthetic preferences [10, 13] and can be further tuned with only modest data [18]. Yet these models are not natively trained to translate language into camera motion; directly prompting an off-the-shelf LMM for a 6-DoF pose produced numerically erratic results. To harness the LMM’s semantic power while restoring geometric soundness, we introduce PhotoAgent—an embodied photography agent whose entire decision loop is steered by the LMM. PhotoAgent carries out reflective reasoning in continuous geometric space: every “thought” emitted by the LMM is instantiated as a physically feasible pose, and every “reflection” is grounded in a view rendered on-the-fly by a real-time 3D Gaussian-splat world model. By internally simulating the visual consequences of candidate motions, the agent cuts real-world trial-and-error and converges rapidly to high-quality decisions (see Figure LABEL:fig:overview for an overview of the system and its key capabilities). Our main contributions are: • An aesthetics‑driven reasoning scheme. We introduce an anchor-point hypothesis and craft an explicit chain‑of‑thought that maps subjective aesthetic goals into solvable spatial‑geometric constraints. • An inverse viewpoint-solving paradigm. Continuing the same chain‑of‑thought, we reformulate the resulting geometric constraints as explicit mathematical problems. • A closed-loop architecture based on 3DGS. Leveraging 3DGS for real‑time, photorealistic rendering, the agent performs visual reflection and iteratively refines its decisions. PhotoAgent achieves state-of-the-art performance across two novel fronts, evaluated in both simulation and the real world through: (1) a language‑conditioned spatial task that seeks the best viewpoint, and (2) a complete pipeline check of image aesthetics and instruction fidelity. I Related Work I-A Robotic Photography Early event-photography systems demonstrated end-to-end autonomy in crowds using composition heuristics and social interaction to capture portraits [3, 33]. Subsequent rule/score-based pipelines encoded guidelines or analytic scoring for repositioning and face-aware composition [17, 5], showing feasibility but limited adaptability in cluttered scenes. Learning-based approaches broadened this space, including template imitation with deep RL and direct optimization of learned aesthetic estimators on mobile platforms [14, 2]. Beyond fixed rules and generic scores, instruction-conditioned pipelines incorporate user intent by retrieving a reference layout and imitating its pose [25, 19]; notably, a recent system (PhotoBot) introduces an LLM to reason over the user query and gallery captions before retrieval, then maps the query to the selected reference and solves a PnP-style pose to mimic composition [19]. Overall, prior systems either encode rules/scores or retrieve-and-mimic exemplars, whereas our method grounds free-form language directly in scene geometry to plan executable, novel viewpoints without dependence on a finite database. I-B Reasoning Architectures for Embodied Agents Recent advances in large language models (LLMs) have driven a shift from reactive policies to reasoning-driven agents. Chain-of-Thought (CoT) prompting [31] enables step-by-step reasoning, but lacks grounding in real-world feedback. The ReAct framework [32] addresses this by interleaving thoughts and actions in a “thought-action-observation” loop, enabling reasoning to influence actions and vice versa. Building on ReAct, Reflexion [32, 30] introduces a self-improvement layer, where the agent summarizes and critiques its own past behaviors using language. This “verbal reinforcement” loop enables iterative skill refinement through feedback-driven reflection. Despite their power, these reasoning agents are typically deployed in symbolic domains (e.g., text games, API calls). How to ground such reflection into real-world visual consequence remains an open problem—particularly for tasks like photography that demand geometric precision. I-C World Models for Visual Foresight Bridging the gap between symbolic reasoning and physical execution requires internal world models. Approaches like World Models [6] and Dreamer-style latent planners [7, 8, 9] learn compact latent dynamics to imagine action outcomes. While efficient, they often distort geometry, which is critical for image composition. Explicit 3D models address this. NeRF yields accurate appearance but is often too slow for online control, even with acceleration like Instant-NGP [23, 24]. 3D Gaussian Splatting (3DGS) attains real-time photorealistic rendering with explicit structure [16], and its uptake in robotics indicates promise for closed-loop use [22, 15, 34]. This combination makes 3DGS a practical backbone to couple language-level reasoning with controllable, view-accurate visual imagination in diverse scenes. I Method Despite their powerful generalization capabilities, LMMs are limited in embodied control due to three factors: (1) an LMM lacks a dedicated spatial-reasoning mechanism and struggles to generalize in cluttered visual environments. (2) as token-based generators, even state-of-the-art models exhibit numerically ill-conditioned behavior when directly asked to output SE(3) poses and tend to conflate camera egomotion with object motion, likely due to limited egomotion-supervised pretraining; (3) Its inference is slow, especially for visual inputs, and each call must be followed by physical motion and reobservation, making naive closed-loop control impractically sluggish. I-A System Overview PhotoAgent couples high-level language reasoning with a geometry-faithful world model through a two-stage cognitive pipeline (Figure 2). At its core, PhotoAgent is powered by an LMM that functions as its central reasoning engine. This LMM-driven agent augments its CoT with a lightweight toolset and memory to bridge perception, reasoning, and actuation. The agent first fuses its multi-modal inputs O into a metric 3D representation G using real-time 3DGS [12], giving planning a geometry-faithful, photorealistic substrate. With G in place, an LMM launches an internal counterfactual loop to determine the optimal action: Intent parsing. The LMM decomposes the instruction ℒL and recent observations into compositional targets. Pose proposal. It analytically solves candidate poses xi\x_i\ and renders predicted views (xi,)W(x_i,G). Reflective critique. Acting as a visual critic, the LMM scores and verbalizes how geometric changes affect aesthetics, then refines its hypothesis. This iterative process is inspired by the “reason-act” and “self-reflection” paradigms from recent work on language agents [32, 30]. Figure 2: Overall cognitive architecture of PhotoAgent. I-B Intention Parsing To address the first critical gap of LMMs—their tendency to be overwhelmed by cluttered scenes—the ‘Intention Parsing‘ module simplifies the problem space through our Anchor-Point Hypothesis. Instead of attempting to reason about all scene elements simultaneously, this strategy directs the LMM to emulate human cognition by selecting a single principal subject to serve as a compositional anchor point. This cognitive simplification is crucial: it reframes an ill-posed global optimization problem into a well-defined and tractable one of relative positioning: how should the agent move relative to this anchor point to achieve the desired photographic outcome for the entire frame? To reason meaningfully about this anchor point and its spatial context, the LMM must be grounded in the physical world. We achieve this via a structured input representation Z derived from raw robot observations O. Since LMMs are not natively trained for 3D geometric understanding, we explicitly supply the relevant cues through a modular tool-use paradigm [27, 29]. The structured perceptual inputs include camera intrinsics, and for each detected object: its semantic label, 2D bounding box (and its center (u,v)(u,v)), and 3D world coordinates. The modular design allows for task-specific extensions, such as including facial orientation in portrait photography, which supports reasoning over composition rules like “looking room.” Having simplified the perceptual problem using an anchor point, we now address the second LMM limitation: its inability to reliably generate stable SE(3) poses for physical execution. Instead of tasking the LMM with direct pose regression, we guide it to produce a set of well-defined geometric constraints through a structured CoT reasoning process [31]. This process mirrors a human photographer’s workflow, breaking down the decision into iterative workflow: • Intent–Scene Alignment and Aesthetic Diagnosis. The LMM maps the user’s goal onto specific scene elements, selects a single subject to serve as the anchor point, identifies occlusions, distractions, or layout flaws, and verbally proposes an aesthetic correction (e.g., “the subject should move slightly to the left”). • 2D Image-Plane Constraints. These determine where and how large the subject appears within the frame: – (u∗,v∗)(u^*,v^*): the target coordinates of the anchor point in the image plane, specifying horizontal and vertical layout. – s: the ratio between desired and current subject scale, controlling visual size. • 3D Viewpoint Constraints. These define the camera’s ideal spatial configuration relative to the anchor point: – θ: azimuth angle, determining the orbital direction around the subject. – φ : elevation angle, controlling vertical camera height. – ρ: camera-to-subject distance derived analytically from s, naturally coupling 2D visual scale with 3D spatial positioning without entanglement. The output of this reasoning process is a structured vector of geometric constraints: =(u∗,v∗,s,θ,φ).g=(u^*,v^*,s,θ, ). Figure 3 shows the subsequent reasoning-to-pose pipeline (using the input scene and user command already introduced in Figure LABEL:fig:overview). Figure 3: Intention Parsing workflow demonstration. “Take a photo of the toys with visual tension, like the reference.” Although camera roll around the optical axis remains mathematically free with a single anchor point, we freeze it at 0∘0 to avoid unstable horizon tilt—small rolls can enhance aesthetics, but large rolls often ruin the frame. We deliberately adopt this spherical parameterization instead of directly regressing a 6-DoF pose. This design decouples distance control (via s) from directional control (via θ, φ ), avoiding logical entanglement where a similar visual outcome could be produced by either translating or rotating the camera. Such disentanglement aligns with principles in language-conditioned robotics [1], enabling more stable and interpretable inference. I-C Geometric Solving Given the geometric constraint vector g, we recover a valid 6-DoF pose ∈SE(3)T (3) in two closed-form steps. This analytic mapping keeps every geometric term explicit, cleanly separates distance from direction, and remains numerically well-conditioned. Moreover, the explicit structure of this pipeline exposes interpretable intermediate steps, facilitating pattern discovery and causal reasoning by the LMM in the reflective optimization loop. We begin by modeling the subject as an upright cylinder. This assumption ensures consistent projections: the subject appears rectangular regardless of azimuth, with a fixed aspect ratio and distance-dependent size. Let h0h_0 be the subject’s height in pixels, ρ0 _0 its original depth and H its true physical height. Given the focal length f and the desired-to-current scale ratio s, we solve for the new camera-to-subject distance ρ: H H =h0ρ0f,h∗=sh0, = h_0\, _0f, h^*=s\,h_0, (1a) ρ ρ =fHh∗=fHsh0=ρ0s. = fHh^*= fHs\,h_0= _0s. (1b) Intuitively, s>1s>1 (a larger on-screen subject) implies moving closer (ρ decreases), while s<1s<1 implies moving back. We then compute the camera’s 3D position cp_c in the subject-centric coordinate frame using the predicted global azimuth θ and elevation φ : c=[ρcos⁡φsin⁡θρsin⁡φρcos⁡φcos⁡θ].p_c= bmatrixρ θ\\ ρ \\ ρ θ bmatrix. (2) Next, we determine the initial 6-DoF camera pose 0T_0 using a look-at function, which orients the camera from position cp_c to face the subject’s location subjectp_subject: 0=look-at(c,subject).T_0= look-at(p_c,p_subject). (3) We adopt a z-up world frame and keep the camera’s roll angle fixed at ψ=0∘ψ=0 to avoid horizon tilt. The look-at function constructs the rotation matrix R from the viewing direction and the world-up vector (0,0,1)(0,0,1), yielding the initial pose 0=[|]T_0=[R\,|\,t]. This initial pose is then refined using a visual servoing loop. We first project the subject’s center under 0T_0 and compute the pixel error vector: =[u−u∗,v−v∗]⊤.e= bmatrixu-u^*,\;v-v^* bmatrix . (4) Under small-angle assumptions, horizontal and vertical errors in the image plane are corrected by adjusting the camera’s local yaw (θyaw _yaw) and pitch (φpitch _pitch) angles, respectively. This yields a near-diagonal image Jacobian: ≈[∂u∂θyaw00∂v∂φpitch].J≈ bmatrix ∂ u∂ _yaw&0\\ 0& ∂ v∂ _pitch bmatrix. (5) This approximation holds for small angular errors (e.g., |Δθyaw|,|Δφpitch|≲5∘| _yaw|,| _pitch| 5 ) around the optical axis. We set the gain λ so that each update induces at most 5%5\% of the image width in pixel shift, preventing overshoot and oscillation. Using the classic image-based visual servoing framework [11, 4], the required angular correction is calculated as: [ΔθyawΔφpitch]=−λ−1. bmatrix _yaw\\ _pitch bmatrix=-λ\,J^-1e. (6) Since J is diagonal, this results in two decoupled first-order control loops. Applying the correction (Δθyaw,Δφpitch)( _yaw, _pitch) to the initial pose 0T_0 yields the final, refined pose ∗T^*. I-D Reflective Reasoning In complex or multi-subject scenes (with distractors or subtle compositional requirements), the single-anchor prior can be brittle; hence the analytical solution may require further refinement. To address this, we introduce a reflective optimization module that iteratively improves the camera pose by performing visual reflection within a geometrically faithful internal simulator. Inspired by the “reflexion” loop from language agents [32, 30], we adapt this paradigm from symbolic text space into 3D visual reasoning, leveraging a photorealistic and geometry-accurate world model based on 3D Gaussian Splatting [12]. By probing small viewpoint changes, the LMM exploits motion-induced regularities, enabling robust composition in multi-object scenes, see Sec. IV-A. Figure 4: Closed-loop reflective reasoning. Starting from an internal 3DGS “imagined world”, the agent renders candidate views, critiques them via the LMM, and issues optimized motion commands. Unlike latent imagination methods prone to hallucinations [7], our 3DGS model offers precise control over rendered views, ensuring that each “reflection” corresponds to a true visual consequence. As illustrated in Figure 4, the agent closes the perception–action loop entirely inside the 3DGS world model before a single physical motion command is issued. Algorithm 1 concisely summarizes this Observe–Think–Act reflective loop. Algorithm 1 Reflective Reasoning 1.Natural‐language instruction ℒL, observations O (including the current RGB image ℐ0I_0), 3DGS world model G 2.Final camera pose ∗∈SE(3)T^* (3) 3.(ℐ0,)←ExtractInputs()(I_0,Z)← ExtractInputs(O) 4.=(u∗,v∗,s,θ,φ)←IntentParsing(ℒ,ℐ0,)g=(u^*,v^*,s,θ, )← IntentParsing(L,I_0,Z) 5.0←GeometricSolve()T_0← GeometricSolve(g) 6.∗←PoseToSpherical(0)x^*← PoseToSpherical(T_0) 7.for t=0t=0 to K−1K-1 do 8. t←∗⊕±δρ,∗⊕±δθ,∗⊕±δφC_t←\x^*\! \!±δρ,\ x^*\! \!±δθ,\ x^*\! \!±δ \ 9. for all i∈¯tx_i∈ C_t do 10. ℐ~i←(i,) I_i (x_i,G) 11. ai←A(ℐ~i,ℒ)a_i← A( I_i,L) 12. end for 13. ′←arg⁡maxi∈¯t⁡aix ← _x_i∈ C_ta_i 14. if a(′)−a(∗)<ϵa(x )-a(x^*)<ε then 15. break 16. end if 17. ∗←′x^* 18.end for 19.∗←SphericalToPose(∗)T^*← SphericalToPose(x^*) 20.return ∗T^* At each iteration t, we begin from the current best pose. To enable fine-grained and interpretable optimization, we operate not on the full SE(3)SE(3) pose T directly, but on its spherical coordinate parameterization, t∗=(ρt,θt,φt)x_t^*=( _t, _t, _t). This three-parameter vector defines the camera’s position, while its orientation is implicitly determined by two constraints: the camera always points towards the subject, and its roll angle is fixed at zero (as detailed in Section I-C). This representation decomposes the complex 6-DoF exploration problem into independent adjustments of distance, azimuth, and elevation. A set of candidate poses is generated by applying a single-axis perturbation to the current best parameters t∗x_t^*: t _t =t∗⊕[±δρ,0,0],t∗⊕[0,±δθ,0], = \x_t^*\! \![±δρ,0,0],\ x_t^*\! \![0,±δθ,0], t∗⊕[0,0,±δφ], \ x_t^*\! \![0,0,±δ ] \, (7a) ¯t C_t =t∪t∗,|¯t|=7. =C_t∪\x_t^*\, | C_t|=7. (7b) To evaluate each candidate i∈¯tx_i∈ C_t, it is first converted from its spherical parameterization back into a full SE(3)SE(3) pose, which is then rendered into an image ℐ~i I_i via the world model G. This ensures both simplicity in optimization and fidelity in physical representation. The LMM serves as a vision-language critic A(⋅,ℒ)A(·,L), assigning a 5-point scalar score ai=A(ℐ~i,ℒ)a_i=A( I_i,L) to each image. After identifying the highest-scoring candidate, the model performs causal reasoning to explain the success (e.g., “Increasing azimuth θ improved composition by creating more looking room for the subject”). The single-axis sampling strategy is critical to isolating such causal effects. We update the best pose t+1∗x_t+1^* based on the scoring results. The LMM’s causal explanation from the previous step informs the next sampling direction. The process repeats until either the score gain falls below a threshold ϵε or the iteration count reaches a maximum K. The final pose ∗x^* is selected globally from the union of all candidates: ∗=arg⁡maxi∈⋃t=0K−1¯t⁡A((i,),ℒ).x^*= _x_i∈ _t=0^K-1 C_tA(W(x_i,G),L). (8) This “propose-simulate-critique-learn” loop transforms a blind exploration problem into a guided optimization process. Enabled by the high-quality initial solution and millisecond-level 3DGS rendering speeds [16], our reflective loop typically converges within few iterations, achieving real-time performance. We use K=3K=3, ϵ=0.2ε=0.2, δθ=δφ=8∘δθ=δ =8 , and a radial step δρ=αρtδρ=α\, _t with α=0.1α=0.1 (10% of the current distance). Figure 5: Performances of our method and baselines on three tasks of different levels. IV Experiments IV-A Spatial Reasoning Evaluation We designed several scenarios to separately evaluate the agent’s spatial imagination and instruction-following capabilities, demonstrated through horizontal movements and pitch adjustments, as well as its spatial composition skills when capturing images involving multiple objects. IV-A1 Experimental Setup All experiments were conducted on a workstation equipped with an NVIDIA RTX 4090D GPU. Physics-based simulation and rendering were performed in Isaac Sim [26], with all scene assets sourced exclusively from its built-in content library to ensure consistency and reproducibility. All methods use the same multimodal model (GPT-4.1). We define one iteration (step) as predicting one camera pose, executing it by directly setting the camera in Isaac Sim, and rendering the next RGB observation as input for the subsequent step. As this process is fully simulated, it introduces no physical motion time and requires no 3D reconstruction overhead. IV-A2 Baseline We adopt two direct-pose baselines following the ReAct/Reflexion-style prompting paradigm [32, 30], while keeping the input interface consistent with our method. Direct-6-DoF. We use chain-of-thought (CoT) prompting [31] to directly predict a 6-DoF camera pose at each step. Direct-6-DoF w/ Reflection. This baseline follows the same direct 6-DoF execution loop, but after observing the outcome of the previous step, it performs an explicit reflection to analyze the consequence of the executed motion and decide how to adjust the next move, before predicting the subsequent 6-DoF pose. Both methods operate under the same three-step environment-interaction budget; the reflection stage is an additional reasoning call and does not increase the number of environment interactions. IV-A3 Comparative Experiments To evaluate the performance of different methods, we designed three gradient tasks with increasing difficulty, as visually illustrated in Figure 5. • Easy—Isolated Banana. Center-frame a single banana in a low-clutter scene to test the reliability of object localization and basic pan–tilt control. • Medium—Cabin Inspection. Capture a top-down image of a cleaning robot’s cabin to determine whether it is full. This task introduces moderate visual clutter, requires nontrivial viewpoint selection, and includes a simple semantic verification. • Hard—Multi-Object Composition. Reframe via camera motion to center the cup and place the can in the upper-right quadrant without altering the scene; this requires precise horizontal translation, pitch control, and multi-object spatial reasoning. IV-A4 Evaluation Metrics Evaluation is based on success rate under a uniform three-step interaction budget. A trial is counted as successful if the agent achieves the objective within this budget. Table I reports the mean number of interaction steps over successful trials, with success counts (out of three trials) in parentheses; failed trials are excluded. Lower step counts indicate higher efficiency. TABLE I: Performance Comparison on Gradient Tasks Method Easy Medium Hard Direct-6-DoF 3.00 (2/32/3) N/A (0/30/3) N/A (0/30/3) Direct-6-DoF w/ Reflection 3.00 (1/31/3) N/A (0/30/3) N/A (0/30/3) PhotoAgent (ours) 2.33 (3/33/3) 2.00 (3/33/3) 2.00 (2/32/3) IV-A5 Results and Analysis As shown in Table I, our method achieves higher success rates and requires fewer interaction steps across all task difficulties. Empirically, Direct-6-DoF and Direct-6-DoF w/ Reflection produce coherent actions on simple tasks but degrade on complex scenarios. The key failure mode is direct 6-DoF pose regression, which is highly sensitive to initial deviations: a large first step pushes inference outside a trust region, lacks contractivity, and compounds errors. In contrast, our method exploits spatial structure and adopts an azimuth-based incremental parameterization. This preserves spatial coherence, promotes contractive updates, reduces sensitivity to the first step, and curbs error accumulation, yielding superior stability, convergence, and success rates on complex tasks. IV-B Aesthetic and Instruction Adherence Evaluation We conducted a human-centered evaluation study to evaluate the aesthetic quality and instruction adherence of the photographs generated by PhotoAgent. Drawing on prior work in robotic photography user studies [3, 19], we aimed to assess: (1) aesthetic improvement, (2) instruction alignment, and (3) statistical significance of results. IV-B1 Experimental Setup We deployed PhotoAgent on a custom mobile manipulator composed of an Agilex RangeMini2 mobile base and a TechRobots TB6-R3 6-DoF arm. An Intel RealSense D435i was mounted as the end-effector camera. Onboard computation was handled by a Thunderobot mini PC with an NVIDIA RTX 4070 Laptop GPU (8GB). The system architecture followed our method design. We used GroundingDINO [20] for open-vocabulary detection. In portrait scenarios, we employed MediaPipe FaceMesh [21] to extract facial landmarks as prior cues. The 3D scene was constructed using AnySplat [12], aligned via VINS-Fusion odometry [28]. We capture 5-7 views around the subject and reconstruct a 3DGS scene via a single feed-forward AnySplat pass; reconstruction is seconds-level [12]. For each scenario, we compare a baseline photo and an optimized photo. The baseline is the initial (unoptimized) view, deterministically chosen as the first captured view on our predefined initialization trajectory, while the optimized photo is the final output produced by our full pipeline. A total of 100 volunteers participated in a two-phase online study. Phase 1 (Independent Rating). All 16 images (8 scenarios × 2 versions: baseline and ours) were shown in randomized order. Participants rated aesthetic appeal on a 5-point Likert scale. Phase 2 (Paired Comparison). Participants compared baseline and optimized photos side-by-side with the original instruction and selected which better fulfilled the goal. Evaluation Metrics. We used: • Mean Opinion Score (MOS): average human rating; • GoB (%): “Good-or-Better” rate (score ≥ 4); • Instruction Adherence Win Rate (IAWR) (%): instruction adherence preference in Phase 2. Our metric choices follow established practice in robotic photography user studies: prior systems evaluate image quality with 5-point human ratings and report MOS and distributional summaries [3, 25, 17, 5, 33]; and instruction-following or aesthetic preference is routinely measured via paired comparison with per-scene win rates (voting-based preference), as in AutoPhoto and PhotoBot [2, 19]. We treat participants as the independent unit. Per scenario (n=100n=100), MOS was tested with a paired Wilcoxon signed-rank test, and Phase-2 IAWR with a one-sided exact binomial test (H0:p=0.5H_0:\,p=0.5, H1:p>0.5H_1:\,p>0.5). Bonferroni correction was applied across 8 scenarios (α′=0.00625α =0.00625), following common practice in robot-photography user studies [25, 2, 19]. TABLE I: Experimental scenarios and user instructions. § denotes simulated scenes; ‡ indicates same subject under different instructions. ID Scene Name User Instruction (a) Girl_Portrait‡ Take a close-up from the front. (b) Girl_Reading‡ Capture her reading intently. (c) Man_Whiteboard Capture a thoughtful-looking expression. (d) Girl_Library‡ Take a beautiful photo. (e) Teddy_Lab Give the teddy bear a close-up shot. (f) Dolls_Confrontation Take a photo of the toys with visual tension, like the reference. (g) RoboDog_Factory§ Carefully photograph the contents of the box on the robotic dog. (h) Truck_Warehouse§ Capture a full shot of the truck. Figure 6: Qualitative examples from real (a) and simulated (b) environments. Each row shows: third-person view, baseline, and PhotoAgent’s output. IV-B2 Experimental Design We curated 8 scenarios (Table I) spanning portraits and still-life settings in both real and simulated environments. User instructions varied in abstraction—from direct composition commands (e.g., “close-up”) to affective intent (e.g., “thoughtful-looking expression”). IV-B3 Results and Analysis Figure 6 shows results from two representative scenarios. In Girl_Portrait, PhotoAgent produces a frontal close-up with subject-centered composition that fulfills the aesthetic intent. In RoboDog_Factory, it interprets spatially complex instructions, selects a novel viewpoint to reveal the box contents, and excludes visual clutter. TABLE I: Performance comparison between PhotoAgent (ours) and the baseline across categories. Category (N) Metric Baseline PhotoAgent (ours) Δ Portraits (4) MOS 2.86 3.82 +0.96 GoB 25.2% 68.5% +43.2 p IAWR — 89.5% — Still Life (4) MOS 2.88 3.94 +1.07 GoB 28.2% 71.2% +43.0 p IAWR — 96.2% — Overall (8) MOS 2.87 3.88 +1.01 GoB 26.8% 69.9% +43.1 p IAWR — 92.9% — Figure 7: Stage-2 instruction adherence. Per-scene win rates under paired comparison. Scene IDs (a–h) are defined in Table I. The quantitative data in Table I confirms our qualitative findings. PhotoAgent markedly improves aesthetic outcomes, boosting the overall MOS by 1.01 points and the GoB rate by 43.1 percentage points. This improvement is consistent across both portrait (+0.96 MOS) and still-life (+1.07 MOS) categories, demonstrating the robustness of our method while still achieving a 92.9% instruction adherence rate. Figure 7 reports Stage-2 instruction-adherence results by scene: win rates span 79–100%. Object-centric scenes average ∼96.2% \!96.2\%, slightly higher than portraits ∼89.5% \!89.5\%. The lowest case (∼79% \!79\%, Girl_Reading) reflects a more abstract, mid-level instruction alongside limited permissible camera motion, which constrains attainable improvement despite correct intent understanding. Across all 8 scenarios, MOS gains were significant (Wilcoxon; all raw p<.001p<.001, remaining significant under Bonferroni), with Cohen’s dzd_z ranging from 0.550.55 to 0.880.88. IAWR values also exceeded chance in all scenarios (one-sided binomial; all raw p<.001p<.001, remaining significant under Bonferroni). V Conclusion We introduced PhotoAgent, an embodied robotic-photography system powered by LMM-guided reasoning. By formulating camera control as an inverse viewpoint-solving problem, PhotoAgent interprets user instructions, solves geometric constraints analytically, and refines its decisions via visual reflection in a 3D Gaussian-splat world model. Experiments demonstrate superior spatial reasoning and aesthetic composition over baselines. Beyond photography, the viewpoint-solving paradigm holds promise for broader embodied AI tasks. Most current systems decouple navigation and manipulation, with the latter often relying on a static camera view—making occlusions or ambiguity critical failure points. Active viewpoint exploration of PhotoAgent, similar to human perspective shifts, can bridge this gap, enabling stronger embodied intelligence that is aware of perception. References [1] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al. (2022) Do as i can, not as i say: grounding language in robotic affordances. arXiv preprint arXiv:2204.01691. Cited by: §I-B. [2] H. AlZayer, H. Lin, and K. Bala (2021) Autophoto: aesthetic photo capture using reinforcement learning. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 944–951. Cited by: §I, §I-A, §IV-B1, §IV-B1. [3] Z. Byers, M. Dixon, W. D. Smart, and C. M. Grimm (2004) Say cheese! experiences with a robot photographer. AI magazine 25 (3), p. 37–37. Cited by: §I, §I-A, §IV-B1, §IV-B. [4] F. Chaumette (2021) Visual servoing. In Computer vision: a reference guide, p. 1367–1374. Cited by: §I-C. [5] R. Gadde and K. Karlapalem (2011) Aesthetic guideline driven photography by robots. In IJCAI Proceedings-International Joint Conference on Artificial Intelligence, Vol. 22, p. 2060. Cited by: §I-A, §IV-B1. [6] D. Ha and J. Schmidhuber (2018) World models. arXiv preprint arXiv:1803.10122 2 (3), p. 440. Cited by: §I-C. [7] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi (2019) Dream to control: learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603. Cited by: §I-C, §I-D. [8] D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba (2020) Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193. Cited by: §I-C. [9] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap (2025) Mastering diverse control tasks through world models. Nature 640 (8059), p. 647–653. Cited by: §I-C. [10] S. Hentschel, K. Kobs, and A. Hotho (2022) CLIP knows image aesthetics. Frontiers in Artificial Intelligence 5, p. 976235. Cited by: §I. [11] S. Hutchinson, G.D. Hager, and P.I. Corke (1996) A tutorial on visual servo control. IEEE Transactions on Robotics and Automation 12 (5), p. 651–670. Cited by: §I-C. [12] L. Jiang, Y. Mao, L. Xu, T. Lu, K. Ren, Y. Jin, X. Xu, M. Yu, J. Pang, F. Zhao, et al. (2025) Anysplat: feed-forward 3d gaussian splatting from unconstrained views. ACM Transactions on Graphics (TOG) 44 (6), p. 1–16. Cited by: §I-A, §I-D, §IV-B1. [13] R. Jiang and C. W. Chen (2025) Multimodal llms can reason about aesthetics in zero-shot. In Proceedings of the 33rd ACM International Conference on Multimedia, p. 6634–6643. Cited by: §I. [14] H. Kang, J. Zhang, H. Li, Z. Lin, T. Rhodes, and B. Benes (2019) LeRoP: a learning-based modular robot photography framework. arXiv preprint arXiv:1911.12470. Cited by: §I, §I-A. [15] N. Keetha, J. Karhade, K. M. Jatavallabhula, G. Yang, S. Scherer, D. Ramanan, and J. Luiten (2024) Splatam: splat track & map 3d gaussians for dense rgb-d slam. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 21357–21366. Cited by: §I-C. [16] B. Kerbl, G. Kopanas, T. Leimkühler, G. Drettakis, et al. (2023) 3d gaussian splatting for real-time radiance field rendering.. ACM Trans. Graph. 42 (4), p. 139–1. Cited by: §I-C, §I-D. [17] K. Lan and K. Sekiyama (2019) Autonomous robot photographer with kl divergence optimization of image composition and human facial direction. Robotics and Autonomous Systems 111, p. 132–144. Cited by: §I, §I-A, §IV-B1. [18] Z. Liao, X. Liu, W. Qin, Q. Li, Q. Wang, P. Wan, D. Zhang, L. Zeng, and P. Feng (2025) Humanaesexpert: advancing a multi-modality foundation model for human image aesthetic assessment. arXiv preprint arXiv:2503.23907. Cited by: §I. [19] O. Limoyo, J. Li, D. Rivkin, J. Kelly, and G. Dudek (2024) PhotoBot: reference-guided interactive photography via natural language. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 2479–2486. Cited by: §I, §I-A, §IV-B1, §IV-B1, §IV-B. [20] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, et al. (2024) Grounding dino: marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, p. 38–55. Cited by: §IV-B1. [21] C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C. Chang, M. G. Yong, J. Lee, et al. (2019) Mediapipe: a framework for building perception pipelines. arXiv preprint arXiv:1906.08172. Cited by: §IV-B1. [22] H. Matsuki, R. Murai, P. H. Kelly, and A. J. Davison (2024) Gaussian splatting slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 18039–18048. Cited by: §I-C. [23] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng (2021) Nerf: representing scenes as neural radiance fields for view synthesis. Communications of the ACM 65 (1), p. 99–106. Cited by: §I-C. [24] T. Müller, A. Evans, C. Schied, and A. Keller (2022) Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG) 41 (4), p. 1–15. Cited by: §I-C. [25] R. Newbury, A. Cosgun, M. Koseoglu, and T. Drummond (2020) Learning to take good pictures of people with a robot photographer. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 11268–11275. Cited by: §I-A, §IV-B1, §IV-B1. [26] NVIDIA Corporation (2024) Isaac sim. Note: Online documentation External Links: Link Cited by: §IV-A1. [27] S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez (2024) Gorilla: large language model connected with massive apis. Advances in Neural Information Processing Systems 37, p. 126544–126565. Cited by: §I-B. [28] T. Qin, P. Li, and S. Shen (2018) Vins-mono: a robust and versatile monocular visual-inertial state estimator. IEEE transactions on robotics 34 (4), p. 1004–1020. Cited by: §IV-B1. [29] Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, et al. (2023) Toolllm: facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789. Cited by: §I-B. [30] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao (2023) Reflexion: language agents with verbal reinforcement learning. Advances in neural information processing systems 36, p. 8634–8652. Cited by: §I-B, §I-A, §I-D, §IV-A2. [31] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. (2022) Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, p. 24824–24837. Cited by: §I-B, §I-B, §IV-A2. [32] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao (2023) ReAct: synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, Cited by: §I-B, §I-B, §I-A, §I-D, §IV-A2. [33] M. Zabarauskas and S. Cameron (2014) Luke: an autonomous robot photographer. In IEEE International Conference on Robotics and Automation (ICRA), p. 1809–1815. Cited by: §I-A, §IV-B1. [34] Z. Zhu, S. Peng, V. Larsson, W. Xu, H. Bao, Z. Cui, M. R. Oswald, and M. Pollefeys (2022) Nice-slam: neural implicit scalable encoding for slam. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 12786–12796. Cited by: §I-C.

Instant research discovery