Paper deep dive
Co-policy: Responsive Human-Robot Co-Creation for Musical Performances
Xuetao Li, Wenke Huang, Mang Ye, Zijian Liu, Jinhua Xie, Jifeng Xuan, Miao Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 6/21/2026, 4:46:55 AM
Summary
Co-policy is a framework for human-robot musical co-creation that enables a robot to generate complementary musical responses to human creative seeds. The system separates the process into three stages: semantic intent grounding using a fine-tuned Qwen-vl (F-Qwen) with a semantic anchor bank, constrained musical variation to ensure motif consistency and physical playability, and low-latency visuomotor execution via a Gaussian-Mixture Visuomotor Policy (GMP). Unlike standard playback or diffusion-based systems, Co-policy uses a single-pass conditional mixture-density policy to handle multimodal action distributions, ensuring real-time responsiveness and expressive performance in musical interactions.
Entities (6)
Relation Signals (5)
Co-policy → incorporates → F-Qwen
confidence 100% · Co-policy... separates semantic intent grounding, constrained musical variation, and visuomotor execution. To ground musical semantics, Co-policy uses... a fine-tuned Qwen-vl planner (F-Qwen).
Co-policy → incorporates → Gaussian-Mixture Visuomotor Policy
confidence 100% · Co-policy introduces a Gaussian-Mixture Visuomotor Policy (GMP)...
F-Qwen → isbasedon → Qwen-VL
confidence 100% · a fine-tuned Qwen-vl planner (F-Qwen)
F-Qwen → uses → Semantic Anchor Bank
confidence 100% · Co-policy uses pre-inference semantic anchors and a fine-tuned Qwen-vl planner (F-Qwen)
Gaussian-Mixture Visuomotor Policy → uses → Guided Self-Attention
confidence 90% · GMP: a plug-and-play, goal-conditioned controller that combines a Guided Self-Attention encoder with a conditional mixture-density action head.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Art has long stood as a pivotal expression of human creativity. Embodied artificial intelligence offers a route for generative models to participate in that creativity through physical action rather than disembodied digital content. In robotic music co-creation, it is challenging to connect semantic musical understanding with real-time and physically executable performance. We present Co-policy, a framework for human-robot musical co-creation that separates semantic intent grounding, constrained musical variation, and visuomotor execution. To ground musical semantics, Co-policy uses pre-inference semantic anchors and a fine-tuned Qwen-vl planner (F-Qwen) to transform speech, live musical seeds, and visual observations into structured co-creation plans. To support low-latency execution, Co-policy introduces a Gaussian-Mixture Visuomotor Policy (GMP), implemented as a conditional mixture-density policy that maps target notes and visual context to multimodal robot actions in a single forward pass. Unlike robotic playback systems that merely reproduce user-specified notes, Co-policy generates complementary musical responses under both musical and physical constraints. Real-robot chime experiments, ablations, and expert evaluation show improved intent alignment, execution accuracy, and response frequency over diffusion-policy and ablated baselines, supporting physically grounded action generation as a key requirement for embodied human-AI co-creation.
Tags
Links
- Source: https://arxiv.org/abs/2606.19914v1
- Canonical: https://arxiv.org/abs/2606.19914v1
Trouble viewing inline? Open PDF directly →
Full Text
62,789 characters extracted from source content.
Expand or collapse full text
Co-policy: Responsive Human-Robot Co-Creation for Musical Performances Xuetao Li School of Computer Science, Wuhan University, Wuhan, China Joint First Authors Wenke Huang College of Computing and Data Science, Nanyang Technological University, Singapore Joint First Authors Mang Ye School of Computer Science, Wuhan University, Wuhan, China Zijian Liu School of Automation, Wuhan University of Technology, Wuhan, China Jinhua Xie School of Geodesy and Geomatics, Wuhan University, Wuhan, China Jifeng Xuan School of Computer Science, Wuhan University, Wuhan, China Correspondence: jxuan@whu.edu.cn, miao.li@whu.edu.cn. Project webpage: https://xtli12.github.io/Co-policy/ Miao Li School of Computer Science, Wuhan University, Wuhan, China School of Robotics, Wuhan University, Wuhan, China Correspondence: jxuan@whu.edu.cn, miao.li@whu.edu.cn. Project webpage: https://xtli12.github.io/Co-policy/ Abstract Art has long stood as a pivotal expression of human creativity. Embodied artificial intelligence offers a route for generative models to participate in that creativity through physical action rather than disembodied digital content. In robotic music co-creation, it is challenging to connect semantic musical understanding with real-time and physically executable performance. We present Co-policy, a framework for human-robot musical co-creation that separates semantic intent grounding, constrained musical variation, and visuomotor execution. To ground musical semantics, Co-policy uses pre-inference semantic anchors and a fine-tuned Qwen-vl planner (F-Qwen) to transform speech, live musical seeds, and visual observations into structured co-creation plans. To support low-latency execution, Co-policy introduces a Gaussian-Mixture Visuomotor Policy (GMP), implemented as a conditional mixture-density policy that maps target notes and visual context to multimodal robot actions in a single forward pass. Unlike robotic playback systems that merely reproduce user-specified notes, Co-policy generates complementary musical responses under both musical and physical constraints. Real-robot chime experiments, ablations, and expert evaluation show improved intent alignment, execution accuracy, and response frequency over diffusion-policy and ablated baselines, supporting physically grounded action generation as a key requirement for embodied human-AI co-creation. 1 Introduction The intersection of artificial intelligence (AI) and the arts has witnessed significant advancements in recent years, particularly in domains such as music, visual arts, and creative writing [32, 27, 36]. Most generative AI systems, however, remain disembodied: they produce text, images, audio, or symbolic scores, but do not have to act through a physical body, negotiate instrument constraints, or respond at the time scale of human interaction. Embodied musical co-creation provides a compact but demanding testbed for this next step. A physically situated agent must understand a human’s abstract intent, generate a musically meaningful response, and realize that response through timed contact with an instrument. Despite this opportunity, robotic music systems are often closer to playback devices than co-creative agents: a human specifies notes, and the robot executes them. This leaves two scientific questions unresolved. I) How can a robot turn an incomplete human seed into a complementary musical response? I) How can that response be executed physically, accurately, and with interaction-level latency? Diffusion policies are highly expressive and have recently achieved very strong performance in many robotic manipulation tasks [4]. However, their iterative denoising process can introduce latency that is undesirable for tightly timed human-robot musical interaction [9, 2]. While accelerated diffusion schemes (e.g., consistency models or fast samplers) mitigate this, they often trade off mode coverage for speed. Meanwhile, standard imitation models trained with near-duplicate views and repeated poses often collapse toward mean actions, losing the alternative striking modes needed for expressive performance. To tackle these challenges, we present Co-policy, a lightweight embodied AI agent that fuses F-Qwen, a fine-tuned Qwen-vl vision-language model (VLM), a constrained musical variation planner, and a low-latency Gaussian-Mixture Visuomotor Policy (GMP). Co-policy converts the disembodied foundation model Qwen-vl into a responsive embodied agent: the VLM grounds intent, the planner creates motif-, harmony-, novelty-, and playability-constrained musical content, and GMP executes it through timed strikes. This modular vision-language-action (VLA) design avoids the deployment cost of monolithic end-to-end VLA models while retaining a physically grounded action loop. Rethinking “co-creation”. A central premise of our work is that genuine co-creation is not mechanical replay of a human-specified score. We define embodied musical co-creation as a closed-loop interaction in which: (i) the human provides an incomplete creative seed, such as a motif, style, rhythm, or verbal intent; (i) the AI agent interprets this seed under musical and physical constraints; (i) the robot contributes a complementary response rather than directly copying the input; and (iv) the response is physically instantiated through real-time embodied action, with timing, strike dynamics, and spatial adaptation constrained by the robot and instrument. This definition yields four measurable dimensions: intent alignment, creative contribution, embodied feasibility, and interaction responsiveness. To address I), we construct a semantic anchor bank for Qwen-vl [39] consisting of paired musical seeds, score images, style descriptors, technical annotations, robot-playability tags, and target structured plans. During inference, retrieved anchors are inserted into a fixed prompt template, and the VLM outputs a JSON-formatted co-creation plan rather than unconstrained prose. The downstream planner then generates a robot response that preserves the human motif while introducing harmonic, rhythmic, or accompaniment variation. This design balances the emergent semantic reasoning of the VLM with structural guardrails for physical execution. To address I), we introduce GMP as a single-pass conditional mixture-density policy. Given the egocentric visual observation and the target note plan, our GMP directly predicts mixture weights, means, and covariances over full robot action vectors (or short-horizon trajectory segments). The mixture components represent latent action modes–alternative striking postures, approach directions, and contact timings–rather than individual joints. This preserves multiple feasible physical responses while avoiding the sequential denoising steps of diffusion policies. We evaluate Co-policy on real-robot chime performance and use ManiSkill2 as a secondary generalization sanity check. The real-robot study focuses on whether the agent aligns with human intent, contributes new musical material, executes feasible actions, and responds at interactive speed. In summary, our contributions are threefold: • A formal embodied co-creation presentation. We formulate musical co-creation as a closed-loop interaction problem in which a robot turns an incomplete human creative seed into a complementary musical response that is semantically aligned and physically executable, distinguishing it from robotic playback or disembodied generation. • A modular embodied agent for constrained musical variation. Co-policy combines semantic intent grounding, constrained musical variation, and visuomotor execution. The VLM outputs structured plans from speech, live notes, and visual context, while the planning layer constrains generated responses by motif consistency, harmonic validity, novelty, and robot playability. • A low-latency Gaussian-Mixture Visuomotor Policy. We propose GMP, implemented as a conditional mixture-density policy, to predict multimodal action distributions over full robot action vectors in a single forward pass. This avoids iterative denoising while preserving alternative feasible action modes. We validate the system through real-robot experiments, ablation studies, latency analysis, and blinded expert evaluation. 5107 Figure 1: Overview of Co-policy. Our framework processes multimodal inputs (human speech, live musical seeds χa _a, and RGB observations) to support embodied human-robot co-creation. The VLM grounds the human’s incomplete creative seed into a structured intent plan; the constrained musical variation module generates a complementary robot response; and the Gaussian-Mixture Visuomotor Policy (GMP) maps the response and visual context to executable robot actions in a single forward pass. Compared to diffusion-policy control, the conditional mixture-density policy preserves alternative action modes without iterative denoising; representative video demonstrations are provided on the accompanying project webpage. 1.1 Background and Related Work 1.1.1 Human-robot Co-creation Framework Human-robot co-creation (HRC) merges robotics, AI, and the creative arts to enable rich, collaborative interaction. Early Human-Robot Interaction (HRI) research focused on predefined tasks [6], evolving towards dynamic systems that perceive and respond to human behavior in real-time [35, 15]. Creative robotics now spans music, visual art, and beyond, while collaborative-creativity frameworks employ deep and reinforcement learning to generate content autonomously yet coherently with human partners [18, 16]. Multi-modal interaction enables robots to interpret various forms of human expression. Affective computing has been integrated to recognize and respond to human emotions [8, 7]. Models like InstructBLIP [5] and LLaVA [26, 25] have improved image-text integration. However, their use in robotics applications faces considerable challenges due to real-world variability, platform heterogeneity, and the necessity for reliable action control [31, 33]. We use Qwen-vl [39] as a semantic grounding module. Robot-specific states and constraints, such as reachable bells, timing limits, and playability rules, are exposed through structured prompts and downstream planning rather than assumed to be directly modeled by the VLM. 1.1.2 Action Generation Framework Learning-based motion planning is central to advanced robotic manipulation [19, 24, 4]. Reinforcement learning achieves strong performance but depends on carefully crafted motion primitives and reward functions [21, 38]. Imitation learning reduces reward design by copying expert demonstrations, yet its generalization is bounded by demonstration quality and coverage [14, 3, 30]. Diffusion models have recently been adopted as generative action planners that produce trajectories through iterative score-based optimization [17, 23, 37]. Although expressive, their repeated denoising steps can limit inference rate in tightly timed human-robot interaction [9, 10]. To address this bottleneck, we introduce GMP: a plug-and-play, goal-conditioned controller that combines a Guided Self-Attention encoder with a conditional mixture-density action head. GMP operates on RGB inputs and target note plans, producing multimodal joint commands in a single forward pass without denoising. 2 Results We evaluate Co-policy from three complementary perspectives: whether the agent generates semantically aligned and musically complementary responses, whether the generated note plans can be executed accurately by a real robot, and whether the visuomotor policy retains interactive response speed under matched baselines. The results show that semantic anchoring improves co-creation quality, while the Gaussian-mixture action head improves real-robot execution without diffusion-style repeated denoising. Figure 2: Illustration of latent action modes in GMP. Each Gaussian component represents one feasible action mode over the full H-step six-DoF action segment, such as a top-down strike, side swing, or gentle tap. Components do not correspond to individual joints. At inference time, GMP predicts mixture parameters in a single forward pass and selects or aggregates the executable action without EM fitting, gradient update, or iterative refinement. Figure 3: Detailed structure of our GSA model. Unlike vanilla self-attention, our GSA couples global interaction modeling with locally salient features across distinct maps: local features (red in F1F_1, processed by encoder ϕe _e) denote key regions in the observation, while global features (red in F2F_2, via Swin Transformer ϕs _s) capture long-range regional connections. Integrating AWCA (Adaptive Weighting Channel Attention) and PSNL (Patch-level Second-order Non-Loc) module for channel-wise and global feature alignment, the GSA enables accurate localization of the target chime bell (red in F3F_3). 2.1 Problem Formulation and Overview Real-time human–robot musical co-creation demands that a robot comprehend what to play (semantic and musical reasoning over speech and live notes), what to add (a complementary response rather than a copy), and how to play it (geometric and contact reasoning over visual cues). At each instant t, the robot consumes a multimodal observation (ℐt,Ntu,t)(I_t,N^u_t,O_t): the spoken instruction ℐtI_t, the human-provided seed notes NtuN^u_t extracted from the audio stream χa,t _a,t, and the egocentric RGB frame t∈ℝH×W×3O_t\!∈\!R^H× W× 3. The agent outputs a robot response NtrN^r_t and a physically executable action segment τt=[at,…,at+H]∈ℝH×6 _t=[a_t,…,a_t+H] ^H× 6. What constitutes embodied musical co-creation? We define co-creation as a four-part closed loop: (1) the human provides an incomplete creative seed NtuN^u_t or verbal intent ℐtI_t; (2) the agent interprets the seed under semantic, musical, and physical constraints; (3) the robot generates a complementary response NtrN^r_t with measurable novelty relative to NtuN^u_t; and (4) the response is physically realized through real-time embodied action τt _t. This formulation separates co-creation from two simpler settings: robotic playback, where Ntr=NtuN^r_t=N^u_t, and disembodied music generation, where the output is not constrained by a physical robot or instrument. 2.1.1 Why Guided Self-Attention Motivation. In robotic tasks, establishing an end-to-end mapping between the robotic end-effectors and targets is crucial for efficient action generation. Prevalent DP-based iterative methods suffer from slow inference, while traditional end-to-end transformers prioritize global dependencies but neglect local pixel-wise features that are essential for fine-grained motion planning. To address this, we propose an end-to-end visuomotor model, Guided Self-Attention (GSA), which first decouples global and local features from the same observation tO_t, and then applies guided attention to couple global reasoning with the relevant local features, mitigating over-reliance on long-range dependencies while establishing the end-to-end mapping mechanism. Design rationale of guided self-attention. We do not claim a formal guarantee that attention must perfectly localize physical contacts. Instead, GSA is designed to bias global contextual reasoning toward locally salient contact regions. The Swin stream provides a full-scene representation of the instrument layout, while the DenseNet stream preserves contact-scale local descriptors. By taking queries from the global stream and keys/values from the local stream, G(Fs,Fe)=softmax(QSKE⊤dk+B)VE,G(F_s,F_e)=softmax\! ( Q_SK_E^\! d_k+B )V_E, (1) GSA lets the policy ask “which part of the instrument is the target?” using global context, but answer through local evidence that corresponds to reachable bells and contact regions. We empirically verify this design through attention visualization, ablation of the global/local streams, and failure analysis rather than relying on an over-strong theorem. 2.1.2 Why GMP uses a mixture-density action head Motivation. In robotic manipulation datasets, near-duplicate viewpoints and similar target articulations create clustering artifacts that drive networks toward mean actions and erode motion diversity. We counteract this collapse by modeling the conditional distribution over the full robot action vector or short-horizon trajectory segment, rather than predicting a single deterministic mean. Given visual context tO_t, a robot note plan NtrN^r_t, and physical constraints ctc_t, the policy predicts: pθ(τt∣t,Ntr,ct)=∑k=1Kπk(τt;μk,Σk),p_θ( _t _t,N^r_t,c_t)= _k=1^K _kN\! ( _t; _k, _k ), (2) where τt∈ℝH×6 _t ^H× 6 can be reduced to a single-step action when H=1H=1. The mixture components do not correspond to individual joints. Each component represents a latent action mode, such as an alternative striking posture, approach direction, or contact timing pattern that can realize the same note under different visual configurations. In the current implementation, we use K=6K=6 as a practical capacity hyperparameter rather than as a proxy for the robot’s six degrees of freedom; systematic selection of K is left to future work. This formulation preserves multiple feasible embodied responses while remaining a single-forward-pass policy, offering an analytical alternative to accelerated diffusion models that often sacrifice mode coverage for inference speed. 2.2 Co-policy Framework We present Co-policy, a framework for real-time human-robot musical co-creation that integrates multimodal understanding (music-language-visual) with precise motion generation. Our approach leverages egocentric RGB images and real-time music streams to enable accurate 6D robotic control for expressive instrument performance. The whole pseudocode for our pipeline is presented in Algorithm 1. The following sections formalize Co-policy and describe its training procedure. Input: Human instruction ℐtI_t, human seed notes NtuN_t^u, RGB observation tO_t, semantic anchor bank S, demonstrations D Output: Robot note response NtrN_t^r and executable action segment τt∗ _t^* // Training Collect =(i,Nir,ci,τigt)i=1MD=\(O_i,N_i^r,c_i, _i^gt)\_i=1^M, where τigt∈ℝH×6 _i^gt ^H× 6; for i=1,…,Mi=1,…,M do hi←GSA(i)h_i (O_i); πk,μk,Σkk=1K←fθ(hi,Nir,ci)\ _k, _k, _k\_k=1^K← f_θ(h_i,N_i^r,c_i); ℒMDN←−log∑k=1Kπk(τigt;μk,Σk)L_MDN←- _k=1^K _kN( _i^gt; _k, _k); ℒ←ℒMDN+λ‖∑k=1Kπkμk−τigt‖22L _MDN+λ \| _k=1^K _k _k- _i^gt \|_2^2; Update θ by gradient descent on ℒL; // Inference for each interaction step t do t←ParseVLM(ℐt,Ntu,t,)Z_t _VLM(I_t,N_t^u,O_t,S); ct←EstimateConstraints(t)c_t (O_t); Ntr←fplan(Ntu,t,ct)N_t^r← f_plan(N_t^u,Z_t,c_t); ht←GSA(t)h_t (O_t); πk,μk,Σkk=1K←fθ(ht,Ntr,ct)\ _k, _k, _k\_k=1^K← f_θ(h_t,N_t^r,c_t); τt∗←μargmaxkπk _t^*← _ _k _k or τt∗←∑k=1Kπkμk _t^*← _k=1^K _k _k; Execute τt∗ _t^* on the robot; Algorithm 1 Co-policy training and inference 2.2.1 Embodied semantic grounding and planning At the highest level of the system, the VLM is not treated as the sole source of creativity. Instead, it is one component in an embodied planning pipeline that separates semantic intent grounding, constrained musical variation, and physical execution. We employ Qwen-vl [1] with a semantic anchor bank whose entries contain a score image, symbolic notes, style descriptor, meter/tempo annotation, robot-playability tags, and an expected structured plan. At inference, retrieved anchors enter a fixed prompt. 1) Semantic intent parsing. The VLM converts speech, live seed notes, and visual context into a structured representation: t=ParseVLM(ℐt,Ntu,t,),Z_t=Parse_VLM(I_t,N^u_t,O_t,S), where tZ_t contains fields such as style, tempo, meter, key, human seed, robot role, reachable bells, and maximum response latency. We require the VLM to output JSON rather than free-form text, for example: intent: "create", style: "energetic", tempo: 120, human_seed: [3,5,6,5], robot_role: "accompaniment", available_notes: [1,2,3,5,6], max_latency_ms: 500 . 2) Constrained musical variation. The robot response is then generated as a complementary note plan Ntr=fplan(Ntu,ζt,ct),N^r_t=f_plan(N^u_t, _t,c_t), where ζt _t is the style/intent latent extracted from tZ_t, and ctc_t represents physical constraints estimated from the observation, such as reachable bells and tempo limits. We constrain NtrN^r_t by four criteria: motif consistency (retaining recognizable structure from the human seed), harmonic validity (avoiding tonal conflict), novelty (preventing direct copying), and embodied playability (ensuring the plan can be executed within workspace and latency limits). 3) Embodied execution. Finally, the generated note plan is mapped to robot motion by the visuomotor policy: τt∼πθ(τ∣t,Ntr,ct), _t _θ(τ _t,N^r_t,c_t), where τt _t is a joint action segment. This separation is important: semantic anchoring improves intent grounding, while the creative planner and policy determine what the robot contributes and how it physically realizes that contribution. The formal representation of the planning module is refined as follows: t=plan(ℐt,Ntu,t|,),P_t=plan (I_t,N^u_t,O_t |C,S ), (3) where =s1,s2,…,snS=\s_1,s_2,…,s_n\ denotes the semantic anchor bank and C contains the fixed schema and playability rules. This framework makes the planning interface reproducible and inspectable through the implementation protocol described in the Methods. Constrained creative contribution beyond replay. Crucially, the human collaborator never specifies the full output sequence. As distilled in the prompt template of Figure 4, the human provides only a semantic anchor (a target mood) together with a short seed motif and the live tempo, while the VLM is asked to complete rather than copy. Conditioned additionally on the visually observed instrument state (which bells are reachable and currently free of residual vibration), the model fills in harmonically consistent responses, adds an accompaniment line, and emits per-note dynamics that track the human’s beat. Figure 4 contrasts two regimes for the identical seed motif C4 E4 G4: a non-anchored baseline that mechanically echoes the three input notes, versus our anchored agent that, given the anchor “joyful, allegro”, extends the motif into a C--G--Am--Em response with a syncopated accompaniment and stronger down-beat strikes while avoiding the occluded F bell. This constrained variation, which is absent from the seed and filtered by physical playability, is what we operationalize as the robot’s creative contribution. Co-creation prompt template (distilled) [System] You are a musical co-creator, not a transcriber. [Anchor S] mood=joyful; style=allegro, 4/4 [Motif NaN_a] seed_notes = C4 E4 G4 ; tempo = 120 BPM [Vision O] free_bells = C,D,E,G,A; occluded = F [Task] Complete & accompany the motif; output notes, [Task] right/left-hand assignment, and strike dynamics. Baseline (replay): C4 E4 G4 (echoes input). Ours (co-creative): response C--G--Am--Em with syncopated accompaniment and accented down-beats, avoiding the occluded F bell. Figure 4: Distilled co-creation prompt and constrained creative response. The human supplies only a mood anchor and a short motif; conditioned on the visually observed instrument state, the anchored VLM completes, not replays. 2.2.2 Gaussian-Mixture Visuomotor Policy Our Gaussian-Mixture Visuomotor Policy (GMP) bridges visual perception, generated note plans, and robotic action. GMP is implemented as a single-pass conditional mixture-density policy: given the egocentric RGB observation tO_t, the robot-generated note plan NtrN^r_t, and physical constraints ctc_t, the policy predicts a multimodal distribution over full robot actions (or short-horizon trajectory segments). This lets the controller preserve multiple feasible striking modes while meeting the low-latency requirements of musical interaction. Guided Self-Attention model. Robotic tasks demand precise spatial reasoning from visual inputs, requiring association of scene regions with end-effector positions for motion planning. However, conventional transformer self-attention [34] overemphasizes global dependencies while overlooking local pixel-wise features critical for fine-grained motion. This creates a critical yet challenging global-local imbalance in motion prediction. To resolve this, we propose the GSA model, a dual-stream architecture with two key innovations: 1) Decoupled Feature Extraction and 2) Cross-Stream Coupling Guided Attention. First, decoupled feature extraction explicitly separates global and local feature learning: a Swin Transformer stream (ϕs _s) captures long-range spatial dependencies via shifted windows [28] (e.g., scene-level structure), while a parallel DenseNet stream extracts pixel-wise independent local features (e.g., subtle motion cues). This design retains both global context and fine-grained details, resolving the trade-off in conventional transformers. Second, cross-stream coupling guided attention ensures global interactions are grounded in local relevance. Unlike vanilla self-attention (where queries, keys, and values derive from the same feature map) GSA pairs query vectors (QSQ_S) from the global Swin stream with key/value vectors (KE,VEK_E,V_E) from the local DenseNet stream. This coupling directs global reasoning toward task-relevant local features, mitigating over-reliance on long-range connections. The formulation of the guided self-attention is as follows: G=softmax(QSKETdk+B)VE,G=softmax ( Q_SK_E^T d_k+B )V_E, (4) where dkd_k indicates the dimension of queries and keys, and B is a relative position bias [29]. Figure 5: Experiment in real-world scenarios. (a) and (b): the ability of co-creation for semantic music and concertos; (c): we randomly adjust the position of the chime, and the robot successfully strikes the designated bell without disrupting its sound. In addition to exploiting local and global features, we apply a channel-wise attention module, namely AWCA [22], to reallocate channel-wise feature responses by integrating correlations between channels. To improve feature compatibility between Streams 1 and 2, we apply the PSNL [22] module (Figure 3), a convolution-based attention mechanism that captures long-range dependencies, analogous to the shift window scheme of Swin Transformer. GSA supplies visual features to the action head: ht=GSA(t)=Pool(Con[G⊙Ps(Aw(F1));Avg(F1)]),h_t=GSA(O_t)=Pool\! (Con[G P_s(A_w(F_1));Avg(F_1)] ), (5) where hth_t is the visual feature supplied to the mixture-density action head, Con[⋅;⋅]Con[·;·] indicates channel-wise concatenation, ⊙ represents Hadamard product, Avg(⋅)Avg(·) is average pooling, and AwA_w and PsP_s denote AWCA and PSNL, respectively. The action distribution is then predicted by: πk,μk,Σkk=1K=fθ(ht,Ntr,ct).\ _k, _k, _k\_k=1^K=f_θ(h_t,N_t^r,c_t). (6) Training uses the mixture negative log-likelihood and auxiliary stabilization loss in Eq. 9; the encoder is optimized jointly with the mixture-density action head. Conditional mixture-density action generation. In robotic musical performance, different trajectories can often realize the same target note: the robot may approach from the left, strike from above, or use a lighter contact depending on the visual layout and recent motion. A deterministic MSE policy tends to average these distinct modes and produce physically ambiguous actions. GMP instead directly predicts a conditional mixture distribution: Semantic Concerto Methods Int. Nov. Coh. AVG ↑ Comp. Nov. Coh. AVG ↑ ManiSkill2-1st Qwen-vl 49.1 45.3 56.1 50.2 38.0 42.5 41.1 40.5 F-Qwen 59.5 65.1 66.0 63.5 48.5 50.1 51.3 50.0 Diffusion Policy Qwen-vl 51.5 57.3 55.0 54.6 32.5 51.1 42.0 41.9 F-Qwen 59.3 64.1 61.5 61.6 42.1 56.3 52.5 50.3 π0.5 _0.5 Qwen-vl 56.5 55.5 63.1 58.4 51.5 47.1 57.5 52.0 F-Qwen 61.5 62.1 53.5 59.0 41.5 53.1 43.2 45.9 GR00T Qwen-vl 56.1 51.5 47.1 51.6 38.3 47.5 38.5 41.4 F-Qwen 63.3 65.1 70.5 66.3 54.3 52.0 61.1 55.8 our GMP Qwen-vl 66.1 61.5 57.1 61.6 48.3 57.5 48.5 51.4 F-Qwen 75.1 71.0 78.3 74.8 58.3 65.5 62.3 62.0 Table 1: Co-creation performance comparison. F-Qwen denotes the fine-tuned Qwen-vl with semantic anchoring. Intent, Novelty, Coher., and Compl. report intent alignment, creative contribution/novelty, musical coherence, and human-robot complementarity, respectively. AVG is the average score over all retained expert ratings, with no trimming of high-variance artistic responses. Figure 6: Four manipulation tasks in ManiSkill2. We validated the effectiveness of GMP in each of the four tasks on the virtual platform ManiSkill2. 11 Figure 7: Human demonstration collecting. For every RGB frame the corresponding action shows how the hand moves from its starting place to the goal spot with the chosen end-effector pose. For an action segment τt=[at,…,at+H]∈ℝH×6 _t=[a_t,…,a_t+H] ^H× 6, GMP predicts: pθ(τt∣t,Ntr,ct)=∑k=1Kπk(τt;μk,Σk),p_θ( _t _t,N^r_t,c_t)= _k=1^K _kN\! ( _t; _k, _k ), (7) where πk _k, μk _k, and Σk _k are outputs of the neural policy and depend on the current observation and target note plan. Each component represents a latent action mode rather than a joint. The full covariance Σk _k captures inter-joint coupling within that mode. We use diagonal or low-rank covariance in implementation for numerical stability and real-time inference, while retaining a multimodal distribution over action vectors. The policy is trained by negative log likelihood: ℒMDN=−log∑k=1Kπk(τgt;μk,Σk),L_MDN=- _k=1^K _kN\! (τ^gt; _k, _k ), (8) with an auxiliary stabilization term ℒ=ℒMDN+λ‖∑k=1Kπkμk−τgt‖22.L=L_MDN+λ \| _k=1^K _k _k-τ^gt \|_2^2. (9) At inference time, the trained policy computes the mixture parameters once and selects either the most likely mode or the mixture expectation: τ∗=μk∗,k∗=argmaxkπkorτ∗=∑k=1Kπkμk.τ^*= _k^*, k^*= _k _k τ^*= _k=1^K _k _k. (10) Online computation is therefore one GSA pass plus one mixture-head evaluation; multimodality is represented directly in the output distribution. 2.3 Experimental Evaluation 2.3.1 Hardware Setup and Simulation Platform The hardware platform for real-world scenarios is presented in the form of a desktop robot. In particular, to mimic the muscle tissue of human fingers and the bending structure of joints, we designed a flexible dexterous hand, thereby making the robotic chime-striking sound more consistent with that of a human hand striking the chime. To verify the generalization of our method, we tested the algorithm in real-life scenarios and on the ManiSkill2 simulation platform (Figure 5 and 6). Figure 8: Co-creation evaluation across policies and planners. a, Policy-planner heatmaps for retained AVG scores in semantic music co-creation and concerto co-creation. b, Planner-side gain, computed as F-Qwen minus Qwen-vl under the same action policy; orange indicates improvement and green indicates decrease. MS2-1st denotes ManiSkill2-1st. c, Descriptive radar profiles comparing Qwen-vl and F-Qwen for Diffusion Policy, GR00T, and GMP across intent alignment, novelty, coherence, and complementarity. d, Task-wise planner shift for representative policies. Each row reports one co-creation setting under a fixed action policy; left and right endpoints denote Qwen-vl and F-Qwen retained AVG scores, and endpoint labels indicate the planner-side gain. Figure 9: Simulation generalization across ManiSkill2 tasks. Success rates on Pour, Fill, Hang, Excavate, and their average. MS2-1st denotes the first-place ManiSkill2-style policy used as the simulation reference. 2.3.2 Dataset and Evaluation Criteria To validate the effectiveness of Co-policy, 350 real-world trajectories were collected for training. Each trajectory includes an execution path and an RGB image O (captured pre-action) linked to a target action: di=(,),d_i=(J,O), (11) where J is the robot-arm joint space (Figure 7). Real-world evaluation uses two criteria. Co-creation: 10 professional chime players scored the generated clips along four aggregate axes–intent alignment, creative contribution/novelty, musical coherence, and response fluency/complementarity–with a maximum score of 100 (Table 1). These axes operationalize the core co-creation requirements of intent alignment, creative contribution, musical coherence, and physically responsive performance while retaining all expert ratings. Scoring protocol and statistics. Following reviewer guidance, we do not discard any extreme scores: in artistic evaluation an outlier may signal a genuinely “out-of-distribution” improvisation rather than noise, so every rating is retained. The listening study is blinded and randomized: each expert evaluates anonymized clips from all methods without access to method identities, and each condition contains the same number of clips. We report the retained aggregate scores in Table 1 and summarize the policy-planner trends in Figure 8. Generalization: action success rate AccaAcc_a (accurate strikes without vibration/sound interference) and tuning accuracy AcctAcc_t (computed by the Methods note-recognition pipeline): Acc=Acca×Acct,Acct=clip[0,1](1−|Fs−Ft||Fs−Fn|),Acc=Acc_a× Acc_t, Acc_t=clip_[0,1]\! (1- |F_s-F_t||F_s-F_n| ), (12) where FsF_s is the striking frequency, FtF_t is the target frequency, and FnF_n is the adjacent note frequency immediately before or after the striking note; the clipping keeps tuning accuracy within [0,1][0,1]. The real-robot objective metrics are action success, tuning accuracy, combined accuracy, and post-command action frequency (Table 2). For the simulation platform, we utilize ManiSkill2 [13] only as a generalization sanity check for the visuomotor policy across four soft-body manipulation tasks. 2.3.3 Performance Comparison and Ablation Study Baselines and configurations. For a fair comparison, all baselines consume the same egocentric RGB observation, use the same 350 demonstration trajectories, predict the same six-DoF joint command, and are evaluated under the same action horizon. ManiSkill2-1st [12] is the first-place soft-body manipulation entry re-trained on our chime demonstrations. Diffusion Policy [4] uses the public CNN-based U-Net backbone with a DDPM schedule and T=50T=50–100100 denoising steps at inference. Because RH20T [11] and DROID [20] are datasets rather than single algorithms, we instantiate them as behavior-cloning controls rather than claiming to reproduce those datasets: the RH20T-style BC uses a ResNet visual encoder and MSE action regression, while the DROID-adapted BC uses a transformer-style visual-token encoder and the same MSE action loss. Neither baseline uses additional RH20T or DROID trajectories. In the co-creation study (Table 1), each control policy is paired with either the off-the-shelf Qwen-vl or our fine-tuned F-Qwen planner so that the contribution of semantic anchoring can be isolated. Latency measurement. The response-speed claim is evaluated under identical hardware, batch size one, and the same action horizon. Table 2 reports post-command action frequency, which captures policy-side response speed after the robot receives a command. This controlled comparison focuses on the visuomotor execution loop: GMP inference is included because it requires one encoder pass and one mixture-head evaluation, whereas diffusion-policy baselines require repeated denoising steps under the same action horizon. VLM planning is not included in this post-command frequency; robot communication and actuation delay are not separately isolated; and full speech-to-action interaction latency is not claimed by Table 2. Ablations for reproducibility. To avoid overclaiming, we use ablations as component diagnostics rather than a full factorial sweep. Semantic anchoring is assessed through the Qwen-vl/F-Qwen comparisons, while the action-side ablations remove ST, DN, GSA, or the GMP head from the deployed pipeline. These diagnostics are intended to show, in this study, that the reported co-creation quality does not simply arise from the base VLM and that multimodal action prediction contributes beyond a deterministic MSE policy. To evaluate the performance of GMP, we conducted comparative experiments in real-world settings and simulation platforms, benchmarking against the first-place entry [12] in the ManiSkill2 challenge and the SOTA models. As shown in Figure 8, in co-creation experiments, Co-policy outperforms the diffusion policy by 8.5% (74.8 vs. 66.3) and 6.2% (62.0 vs. 55.8). After applying pre-inference semantic anchoring to Qwen-vl, the model achieves a 7.4%–13.3% improvement in semantic music co-creation and a 3.5%–9.6% improvement in concerto co-creation. The negative concerto gain for π0.5 _0.5 suggests that semantic anchoring alone cannot compensate for policy-side execution mismatch. For real-robot chime striking (Table 2), we observe a 15% performance gain and a higher response frequency than the diffusion-policy baseline under the same action-horizon setting. In the ManiSkill2 simulation sanity check (Figure 9), GMP is treated as a visuomotor generalization sanity check rather than the central claim. Furthermore, ablation studies (Table 3) reveal the contributions of key GMP components: the Swin Transformer stage (32%), DenseNet block in the encoder (27%), GSA (21%), and mixture-density action head (16%). The main evidence is therefore the real-robot co-creation and chime-striking evaluation. Figure 10: Encoder structure for visuomotor grounding. This encoder diagram clarifies how global Swin-Transformer features and local DenseNet features are coupled before guided self-attention. AWCA reweights channel responses and PSNL improves long-range compatibility in the local stream, allowing the GMP head to condition action-mode prediction on both instrument-level layout and contact-level visual evidence. 3 Discussion This work reframes robotic musical performance as an embodied co-creation problem rather than a playback problem. The robot is not evaluated only by whether it strikes a requested bell, but by whether it can transform an incomplete human creative seed into a complementary musical response that is semantically aligned, musically coherent, physically executable, and responsive at interaction speed. This distinction is central for embodied AI: musical meaning is generated through the coupling of symbolic intent, acoustic context, visual grounding, and contact-rich action. The results suggest that physically grounded action generation is a necessary component of human-AI co-creation. A purely symbolic music model can generate plausible notes, but it does not decide whether a physical robot can reach the instrument, avoid occlusion, satisfy timing constraints, or produce the intended sound after contact. Conversely, a low-level controller can strike bells accurately but cannot determine whether the response contributes musically to the human seed. Co-policy therefore separates semantic grounding, constrained musical variation, and visuomotor execution while preserving information flow between them. This separation makes the system interpretable: semantic failures, perceptual localization failures, and execution failures can be inspected independently rather than being collapsed into a single end-to-end error. Methods AccaAcc_a ↑ AcctAcc_t ↑ Acc ↑ Freq (Hz) ↑ ManiSkill2-1st 0.60 0.73 0.44 15.2 BC (RH20T-style) 0.65 0.78 0.51 10.1 Diffusion Policy 0.68 0.83 0.56 1.01 DROID-adapted BC 0.62 0.75 0.47 14.2 Co-policy (ours) 0.78 0.88 0.69 18.6 Table 2: Evaluation of chime striking. AccaAcc_a, AcctAcc_t, Acc, and Freq denote action success, tuning accuracy, combined accuracy, and post-command frequency. The framework also clarifies the boundary of robotic creativity. The generated response is not a free-form hallucination from the VLM, nor is it a simple lookup from a fixed anchor bank. Instead, the VLM supplies broad semantic generalization over style, intent, and visual context, while the semantic anchors and playability filters define the domain in which variation is musically and physically meaningful. Operationally, the prompt acts as a constraint interface: user intent, seed notes, playable-note limits, timing, and hand-state information are passed through semantic-anchor guidance, and the VLM is required to return a structured response containing notes with beats, robot role, timing, and hand assignment. Creativity in this setting is therefore guided variation: the robot may alter rhythm, accompaniment role, or melodic contour, but it must remain within the constraints imposed by the instrument, the human seed, and the executable action space. This interpretation is conservative but important for embodied AI, where unconstrained novelty can easily become an unplayable or unsafe physical command. The comparison with diffusion-policy baselines should be understood in the same embodied context. Co-policy does not claim that mixture-density policies universally dominate diffusion models. Recent accelerated diffusion variants can reduce sampling cost, and diffusion models remain powerful for complex trajectory distributions. The contribution of GMP is narrower: for short-horizon chime striking, a conditional mixture-density head preserves multiple feasible latent action modes while requiring only one policy evaluation after the command has been received. This is useful when the interaction loop is constrained by real-time musical timing, where repeated denoising can be expensive and deterministic behavior cloning may collapse to an averaged, weak-contact action. ST DN GSA GMP head AccaAcc_a ↑ AcctAcc_t ↑ Acc ↑ Δ ↓ - ✓ ✓ ✓ 0.57 0.65 0.37 0.32 ✓ - ✓ ✓ 0.62 0.68 0.42 0.27 ✓ ✓ - ✓ 0.68 0.70 0.48 0.21 ✓ ✓ ✓ - 0.71 0.75 0.53 0.16 ✓ ✓ ✓ ✓ 0.78 0.88 0.69 0.00 Table 3: Ablation study of core GMP modules. We test the Swin Transformer stage (ST), DenseNet block (DN), Guided Self-Attention mechanism (GSA), and GMP mixture-density action head on 30 randomly positioned chime samples. Δ is the absolute Acc drop from the full model. (a) Attention map (b) Compliant hand (c) Robot perspective Figure 11: Discussion-oriented diagnostics. Attention visualization helps inspect whether the visual encoder grounds actions around target bells and contact-adjacent regions. The compliant hand illustrates the soft-contact design used to approximate human chime striking, while the robot-perspective view shows why small visual differences can still induce multiple feasible action modes. Together, these diagnostics clarify why Co-policy treats creativity, grounding, and execution as coupled but separately inspectable components. This design choice is linked to the encoder diagnostic summarized in Figure 10. In a musical robot, local texture around a bell edge and global layout of the chime frame are both relevant: the former affects contact localization, while the latter affects target selection and reachability. The encoder should therefore be read as a practical mechanism for improving feature compatibility before action-mode prediction, not as an independent claim about musical intelligence. Its role is to reduce perceptual ambiguity before the mixture-density policy decides among alternative feasible strikes. Several limitations remain. First, the current system is validated primarily on chime performance, so the musical space is bounded by the available notes, the anchor bank, and the visual geometry of the instrument. Second, acoustic note extraction can be affected by servo noise, hammer impact, and chime reverberation. The present system treats audio as a high-level symbolic cue for planning rather than as a dense feedback signal for contact control. It does not yet include closed-loop tactile or force sensing after impact, so mis-strikes are detected through pitch and action-success metrics rather than corrected during contact. Third, the evaluation focuses on short interaction windows; long-term improvisation would require memory, adaptation to a specific human partner, and higher-level musical form planning. These limits are also reflected in the diagnostic views in Figure 11: attention must remain concentrated near playable contact regions, the hand design can only approximate human compliance, and small visual changes from the robot perspective can still correspond to different feasible striking trajectories. These limitations point to clear future directions. Source-separated acoustic perception could distinguish human notes, robot impacts, and reverberant tails more reliably. Contact sensing and force feedback could enable online recovery from weak strikes or unintended collisions. Larger anchor banks and preference models could support more diverse musical styles while preserving playability constraints. Finally, multi-turn co-creation with memory would allow the robot to develop motifs, respond to repeated human themes, and participate in ensemble-level structure rather than isolated call-and-response phrases. 4 Methods 4.1 Problem Formulation and Overview At interaction step t, Co-policy receives a human instruction ℐtI_t, symbolic seed notes NtuN^u_t, and an egocentric RGB observation tO_t, from which it estimates physical constraints ctc_t. The system first plans a robot musical response NtrN^r_t and then predicts an executable action segment τt=[at,…,at+H] _t=[a_t,…,a_t+H]: Ntr=fplan(ℐt,Ntu,t,ct;),τt=fθ(t,Ntr,ct),N^r_t=f_plan(I_t,N^u_t,O_t,c_t;A), _t=f_θ(O_t,N^r_t,c_t), (13) where A is the semantic anchor bank and fθf_θ is the Gaussian-Mixture Visuomotor Policy. This factorization allows the semantic response, musical validity, and physical execution to be evaluated separately. 4.2 Semantic Grounding and Constrained Musical Planning Each anchor aj∈a_j contains a musical seed, score image, style descriptor, tempo or meter cue, playability tag, and expected structured plan. Given the query qt=(ℐt,Ntu,t)q_t=(I_t,N^u_t,O_t), the planner retrieves: t=TopMaj∈cos(ϕ(qt),ϕ(aj)),A_t=TopM_a_j \! (φ(q_t),φ(a_j) ), (14) where ϕ(⋅)φ(·) denotes the multimodal embedding. The retrieved anchors condition Qwen-vl, which must output a structured plan rather than free-form prose. Candidate note responses are selected by a constrained objective: Ntr=argmaxN∈(Ntu,ct) N^r_t= _N (N^u_t,c_t) αSsem(N)+βSmus(N,Ntu) \ α S_sem(N)+β S_mus(N,N^u_t) (15) +γSnov(N,Ntu)−ηCphy(N,ct), +γ S_nov(N,N^u_t)-η C_phy(N,c_t), where C denotes reachable and timing-valid candidates, SsemS_sem measures intent alignment, SmusS_mus musical coherence, SnovS_nov creative deviation from the seed, and CphyC_phy physical cost. Live audio is converted to NtuN^u_t through onset detection, pitch tracking, and rhythm quantization; this audio processing is distinct from the post-command policy frequency reported in Table 2. 4.3 Gaussian-Mixture Visuomotor Policy The visual feature is computed by coupling global scene context and local contact evidence: ht=GSA(gSwin(t),gDense(t)),h_t=GSA\! (g_Swin(O_t),g_Dense(O_t) ), (16) where AWCA reweights channels and PSNL improves long-range feature compatibility before guided attention. GMP then predicts a conditional mixture over complete short-horizon action segments: pθ(τt∣ht,Ntr,ct)=∑k=1Kπk(ht,Ntr,ct)(τt;μk,Σk).p_θ( _t h_t,N^r_t,c_t)= _k=1^K _k(h_t,N^r_t,c_t)N\! ( _t; _k, _k ). (17) Each component is a latent action mode, not a robot joint. K=6K=6 is used as a practical capacity hyperparameter, and Σk _k is constrained to positive diagonal or low-rank form for stable real-time inference. 4.4 Training, Evaluation, and Diagnostics Demonstrations are tuples di=(i,Nir,ci,τigt)d_i=(O_i,N^r_i,c_i, _i^gt). The policy is trained by mixture negative log likelihood plus stabilization: ℒGMP= _GMP= −∑ilog∑k=1Kπik(τigt;μik,Σik) - _i _k=1^K _ikN\! ( _i^gt; _ik, _ik ) (18) +λ∑i‖∑k=1Kπikμik−τigt‖22. +λ _i \| _k=1^K _ik _ik- _i^gt \|_2^2. At inference, the executable segment is τt∗=μk∗ _t^*= _k^*, k∗=argmaxkπk^*= _k _k, or the mixture expectation ∑kπkμk _k _k _k. Post-command frequency measures only this policy-side loop after an executable command has been received. The real-robot platform uses an egocentric camera, microphone, two six-degree-of-freedom arms, and compliant dexterous hands. We collect 350 real-world demonstrations and adapt all baselines to the same observation space, action horizon, and six-DoF command interface. Co-creation is evaluated by blinded expert ratings on intent alignment, creative contribution, musical coherence, and complementarity. Objective striking metrics are: Acc Acc =AccaAcct, =Acc_aAcc_t, (19) Acct Acc_t =clip[0,1](1−|Fs−Ft||Fs−Fn|), =clip_[0,1]\! (1- |F_s-F_t||F_s-F_n| ), et e_t ∈Esem,Evis,Eexec, ∈\E_sem,E_vis,E_exec\, where FsF_s, FtF_t, and FnF_n are the struck, target, and adjacent-note frequencies, and AcctAcc_t is clipped to [0,1][0,1]. The error label ete_t separates semantic-planning, visual-localization, and execution-contact failures. ManiSkill2 results are used only as a secondary visuomotor generalization check. 5 Data Availability The project webpage, source code, demonstration videos, and supplementary materials are publicly available at https://xtli12.github.io/Co-policy/docs/. Processed robot demonstrations, generated note plans, anonymized expert ratings, evaluation scripts, prompt templates, semantic-anchor schema, and trained-model configuration files are provided through the public project repository where release permissions allow. Raw videos that may contain identifiable participants are shared only in anonymized, cropped, or consent-permitted form. The semantic-anchor schema and JSON output format are described in the Methods to support reproduction independent of the full raw video release. 6 Code Availability The source code, webpage implementation, prompt templates, and configuration files associated with Co-policy are publicly available through the project repository linked from https://xtli12.github.io/Co-policy/docs/. 7 Acknowledgements The authors thank the participating musicians and annotators for assisting with real-robot evaluation and perceptual scoring. 8 Author Contributions X.L. designed and implemented Co-policy, conducted experiments, and drafted the manuscript. W.H. and M.Y. contributed to system design, experimental analysis, and manuscript revision. Z.L. and J.Xie contributed to robotic platform construction and evaluation. J.Xuan and M.L. supervised the project, provided guidance, and revised the manuscript. All authors reviewed and approved the manuscript. 9 Competing Interests The authors declare no competing interests. References [1] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou (2023) Qwen-vl: a frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966. Cited by: §2.2.1. [2] C. Chen, F. Deng, K. Kawaguchi, C. Gulcehre, and S. Ahn (2024) Simple hierarchical planning with diffusion. arXiv preprint arXiv:2401.02644. Cited by: §1. [3] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch (2021) Decision transformer: reinforcement learning via sequence modeling. Advances in neural information processing systems 34, p. 15084–15097. Cited by: §1.1.2. [4] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song (2023) Diffusion policy: visuomotor policy learning via action diffusion. In Robotics: Science and Systems (RSS), Cited by: §1.1.2, §1, §2.3.3. [5] W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi (2023) InstructBLIP: towards general-purpose vision-language models with instruction tuning. External Links: 2305.06500 Cited by: §1.1.1. [6] K. Dautenhahn (2005) Socially intelligent robots: dimensions of human–robot interaction. Philosophical Transactions of the Royal Society B: Biological Sciences 360 (1458), p. 679–704. Cited by: §1.1.1. [7] A. de Rooij, S. v. d. Broek, M. Bouw, and J. de Wit (2024) Co-creating with a robot facilitator: robot expressions cause mood contagion enhancing collaboration, satisfaction, and performance. International Journal of Social Robotics, p. 1–20. Cited by: §1.1.1. [8] A. De Rooij, S. Van Den Broek, M. Bouw, and J. De Wit (2023) Co-designing with a social robot facilitator: effects of robot mood expression on human group dynamics. In Proceedings of the 11th International Conference on Human-Agent Interaction, p. 22–29. Cited by: §1.1.1. [9] Z. Dong, J. Hao, Y. Yuan, F. Ni, Y. Wang, P. Li, and Y. Zheng (2024) DiffuserLite: towards real-time diffusion planning. arXiv preprint arXiv:2401.15443. Cited by: §1.1.2, §1. [10] Z. Evans, C. Carr, J. Taylor, S. H. Hawley, and J. Pons (2024) Fast timing-conditioned latent audio diffusion. In Forty-first International Conference on Machine Learning, Cited by: §1.1.2. [11] H. Fang, H. Fang, Z. Tang, J. Liu, C. Wang, J. Wang, H. Zhu, and C. Lu (2023) Rh20t: a comprehensive robotic dataset for learning diverse skills in one-shot. arXiv preprint arXiv:2307.00595. Cited by: §2.3.3. [12] F. Gao, X. Li, J. Yu, and F. Shaung (2023) A two-stage fine-tuning strategy for generalizable manipulation skill of embodied ai. arXiv preprint arXiv:2307.11343. Cited by: §2.3.3, §2.3.3. [13] J. Gu, F. Xiang, X. Li, Z. Ling, X. Liu, T. Mu, Y. Tang, S. Tao, X. Wei, Y. Yao, et al. (2023) Maniskill2: a unified benchmark for generalizable manipulation skills. arXiv preprint arXiv:2302.04659. Cited by: §2.3.2. [14] S. Haldar, J. Pari, A. Rai, and L. Pinto (2023-07) Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea. External Links: Document Cited by: §1.1.2. [15] B. Q. Hashmi (2023) Artificial intelligence and its role in information and communication technologies (ict): application areas of artificial intelligence. In AI and its convergence with communication technologies, p. 1–18. Cited by: §1.1.1. [16] P. Hsieh, D. Benros, and T. Dogan (2022) Conversational co-creativity with deep reinforcement learning agent in kitchen layout. In Design computing and cognition’20, p. 399–409. Cited by: §1.1.1. [17] S. Huang, Z. Wang, P. Li, B. Jia, T. Liu, Y. Zhu, W. Liang, and S. Zhu (2023) Diffusion-based generation, optimization, and planning in 3d scenes. In Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1.1.2. [18] P. Karimi, J. Rezwana, S. Siddiqui, M. L. Maher, and N. Dehbozorgi (2020) Creative sketching partner: an analysis of human-ai co-creativity. In Proceedings of the 25th international conference on intelligent user interfaces, p. 221–230. Cited by: §1.1.1. [19] G. Khandate, S. Shang, E. T. Chang, T. L. Saidi, J. Adams, and M. Ciocarlie (2023-07) Sampling-based Exploration for Reinforcement Learning of Dexterous Manipulation. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea. External Links: Document Cited by: §1.1.2. [20] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, et al. (2024) Droid: a large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945. Cited by: §2.3.3. [21] M. Kim, J. Han, J. Kim, and B. Kim (2023) Pre-and post-contact policy decomposition for non-prehensile manipulation with zero-shot sim-to-real transfer. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 10644–10651. Cited by: §1.1.2. [22] J. Li, C. Wu, R. Song, Y. Li, and F. Liu (2020) Adaptive weighted attention network with camera spectral sensitivity prior for spectral reconstruction from rgb images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, p. 462–463. Cited by: §2.2.2. [23] R. Li, R. Li, S. Guo, and L. Zhang (2024) Source prompt disentangled inversion for boosting image editability with diffusion models. arXiv preprint arXiv:2403.11105. Cited by: §1.1.2. [24] Z. Li, X. B. Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath (2023-07) Robust and Versatile Bipedal Jumping Control through Reinforcement Learning. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea. External Links: Document Cited by: §1.1.2. [25] H. Liu, C. Li, Y. Li, and Y. J. Lee (2023) Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744. Cited by: §1.1.1. [26] H. Liu, C. Li, Q. Wu, and Y. J. Lee (2024) Visual instruction tuning. Advances in neural information processing systems 36. Cited by: §1.1.1. [27] Y. Liu, K. Zhang, Y. Li, Z. Yan, C. Gao, R. Chen, Z. Yuan, Y. Huang, H. Sun, J. Gao, et al. (2024) Sora: a review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17177. Cited by: §1. [28] Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Z. Zhang, L. Dong, et al. (2022) Swin transformer v2: scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 12009–12019. Cited by: §2.2.2. [29] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo (2021) Swin transformer: hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, p. 10012–10022. Cited by: §2.2.2. [30] N. M. Shafiullah, Z. Cui, A. A. Altanzaya, and L. Pinto (2022) Behavior transformers: cloning k modes with one stone. Advances in neural information processing systems 35, p. 22955–22968. Cited by: §1.1.2. [31] M. Shridhar, L. Manuelli, and D. Fox (2023) Perceiver-actor: a multi-task transformer for robotic manipulation. In Conference on Robot Learning, p. 785–799. Cited by: §1.1.1. [32] B. Sturm, K. Déguernel, R. S. Huang, A. Kaila, P. Jäskeläinen, E. Kanhov, L. Cros Vila, D. Dalmazzo, L. Casini, O. Bown, et al. (2024) AI music studies: preparing for the coming flood. In AI Music Creativity, Cited by: §1. [33] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. (2024) Octo: an open-source generalist robot policy. arXiv preprint arXiv:2405.12213. Cited by: §1.1.1. [34] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. Advances in neural information processing systems 30. Cited by: §2.2.2. [35] S. F. Wamba, M. M. Queiroz, and L. Hamzi (2023) A bibliometric and multi-disciplinary quasi-systematic analysis of social robots: past, future, and insights of human-robot interaction. Technological Forecasting and Social Change 197, p. 122912. Cited by: §1.1.1. [36] T. Wang, J. Chen, Q. Jia, S. Wang, R. Fang, H. Wang, Z. Gao, C. Xie, C. Xu, J. Dai, et al. (2024) Weaver: foundation models for creative writing. arXiv preprint arXiv:2401.17268. Cited by: §1. [37] Z. Xian and N. Gkanatsios (2023) Chaineddiffuser: unifying trajectory diffusion and keypose prediction for robotic manipulation. In Conference on Robot Learning/Proceedings of Machine Learning Research, Cited by: §1.1.2. [38] K. Xu, H. Yu, Q. Lai, Y. Wang, and R. Xiong (2021) Efficient learning of goal-oriented push-grasping synergy in clutter. IEEE Robotics and Automation Letters 6 (4), p. 6337–6344. Cited by: §1.1.2. [39] A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, et al. (2024) Qwen2 technical report. arXiv preprint arXiv:2407.10671. Cited by: §1.1.1, §1.