Paper deep dive
JarvisBench: Always-on Intelligence Between Humans and Agents
Chen Chen, Zhehuai Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 8/18/2026, 4:49:01 AM
Summary
The paper introduces JarvisBench, a benchmark designed to evaluate bidirectional attention coordination between humans and long-horizon AI agents. It proposes an 'always-on' intermediary layer called Jarvis that manages human attention by answering user queries in real-time and proactively soliciting user judgment when agents encounter consequential decisions. The benchmark consists of 45 tasks across single-agent and multi-agent settings, evaluating both agent collaboration (improving task outcomes via human input) and user interaction (providing accurate, non-interruptive answers). Experiments show that using GPT-5.6-Sol as the Jarvis model significantly improves task scores across various worker models, highlighting the importance of separating attention coordination from agent execution.
Entities (10)
Relation Signals (8)
JarvisBench → contains → 45 agentic task instances
confidence 95% · JarvisBench contains 45 agentic task instances
JarvisBench → evaluates → attention coordination
confidence 95% · JarvisBench is designed to evaluate both directions of this coordination
Agent-Collaboration Track → measures → task outcome improvement
confidence 90% · The Agent-Collaboration Track asks: Was human attention used effectively?
User-Interaction Track → measures → user query accuracy
confidence 90% · The User-Interaction Track asks: Was Jarvis useful when the user reached out?
Jarvis → mediates → human attention
confidence 90% · Jarvis... mediates this interface and allocates human attention across one or more working agents
GPT 5.6 Sol → usedas → Jarvis
confidence 90% · with GPT-5.6-Sol as Jarvis, every completed worker configuration improves
DeepSeek V4 Pro → exhibits → high attention efficiency
confidence 85% · DeepSeek V4-Pro is most efficient on multi-agent tasks because it obtains its gain with only 0.30 requests per task
OpenClaw → usedby → worker agents
confidence 85% · The worker model, OpenClaw harness... are identical between the baseline and Jarvis conditions
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Long-horizon agents can execute continuously, but human attention remains intermittent and scarce. This creates a bidirectional coordination problem: users may need immediate access to an agent while work continues in the background, whereas agents may encounter consequential decisions that require user judgment after the user has stopped monitoring execution. We posit an always-on attention-coordination layer---\textit{Jarvis}\footnote{Named after the fictional AI assistant in \textit{Iron Man}.}---that mediates this interface and allocates human attention across one or more working agents. We introduce \textit{JarvisBench} to evaluate both directions of this coordination: whether an intermediary can accurately and promptly answer user-initiated questions about ongoing work, and whether it can recognize when an agent requires user judgment, solicit that judgment at the right moment, and route it back to improve task outcomes. JarvisBench contains 45 agentic task instances: 20 single-agent tasks and 25 workstreams organized into 10 multi-agent projects. The tasks span 19 domains and were selected and adapted from more than 2,000 public candidates. Crucially, the need for user attention arises naturally during execution rather than from an obvious omission in the initial prompt. JarvisBench is designed to integrate with arbitrary agent runtimes without modifying their underlying execution loops. Our reference implementation further provides a full-duplex speech interface, allowing users to reach Jarvis naturally while timely attention coordination supports agents working in the background. By separating agent execution from attention coordination, JarvisBench provides a stable evaluation target as agent capabilities continue to improve.
Tags
Links
- Source: https://arxiv.org/abs/2608.14870v1
- Canonical: https://arxiv.org/abs/2608.14870v1
Trouble viewing inline? Open PDF directly →
Full Text
38,975 characters extracted from source content.
Expand or collapse full text
Arxiv Preprint JARVISBENCH: ALWAYS-ON INTELLIGENCE BETWEEN HUMANS AND AGENTS Chen Chen Zhehuai Chen NVIDIA ABSTRACT Long-horizon agents can execute continuously, but human attention remains in- termittent and scarce. This creates a bidirectional coordination problem: users may need immediate access to an agent while work continues in the background, whereas agents may encounter consequential decisions that require user judgment after the user has stopped monitoring execution. We posit an always-on attention- coordination layer—Jarvis 1 —that mediates this interface and allocates human at- tention across one or more working agents. We introduce JarvisBench to evaluate both directions of this coordination: whether an intermediary can accurately and promptly answer user-initiated questions about ongoing work, and whether it can recognize when an agent requires user judgment, solicit that judgment at the right moment, and route it back to improve task outcomes. JarvisBench contains 45 agentic task instances: 20 single-agent tasks and 25 workstreams organized into 10 multi-agent projects. The tasks span 19 domains and were selected and adapted from more than 2,000 public candidates. Crucially, the need for user attention arises naturally during execution rather than from an obvious omission in the ini- tial prompt. JarvisBench is designed to integrate with arbitrary agent runtimes without modifying their underlying execution loops. Our reference implementa- tion further provides a full-duplex speech interface, allowing users to reach Jarvis naturally while timely attention coordination supports agents working in the back- ground. By separating agent execution from attention coordination, JarvisBench provides a stable evaluation target as agent capabilities continue to improve. § Code: https://github.com/cchen1436/JarvisBench Å Video demo: https://cchen1436.github.io/jarvis 1INTRODUCTION Agent capability is advancing rapidly, but human attention is not. Modern agents can already com- plete a wide range of long-horizon tasks autonomously. Yet greater autonomy and parallelism create an increasing need for timely user attention: without it, agents can pursue the wrong direction unchecked and drift away from the human needs they are meant to serve. This attention mismatch issue appears in both directions. When an agent is working, a user who wants to ask a question or provide guidance often has to interrupt its execution. Frequent interrup- tions reduce efficiency and mix transient conversation into the agent’s working context. In the other direction, when the agent reaches a decision that requires human judgment, it cannot ensure that the user is watching. The agent often has little choice but to guess and continue. By the time the user returns, that decision may already have shaped the rest of the work. Letting the worker proactively pause and seek user attention seems like a natural solution. However, recent studies (Qian et al., 2025; Zhang et al., 2026; He et al., 2025) point to the same conclusion: proactivity is a separate capability, and current agents are poorly calibrated for it. This is unsur- prising, as current agents are primarily trained and evaluated for task completion, not for deciding when an intermittently available user should be brought into the execution process. Moreover, agent 1 Named after the fictional AI assistant in Iron Man. 1 arXiv:2608.14870v1 [cs.AI] 14 Aug 2026 Arxiv Preprint proactivity addresses only the agent-to-user direction; it does not give the user a continuously avail- able interface for questions or guidance while the agent is working. An always-on assistant like Jarvis coordinates both directions: it remains continuously available for real-time voice interaction, and directs scarce user attention to where it is needed across ongoing agent work. Recent updates from Codex (OpenAI, 2026) and Qoder (Qoder, 2026) move toward this Jarvis-like design by keeping users connected to ongoing agent work. Both allow users to query or steer work- ing agents through an always-available interface; Codex supports spoken interaction, while Qoder further notifies users when their input is needed. However, no existing benchmark evaluates this bidirectional attention-coordination problem. To fill this gap, we introduce JarvisBench, compris- ing 20 single-agent tasks and 25 workstreams organized into 10 multi-agent projects, selected and adapted from more than 2K public candidates. In each task, the need for user attention is not arti- ficially created by withholding essential constraints from the initial prompt; instead, consequential decision points emerge naturally as the work unfolds and require timely user input to keep execution aligned with user needs. We define complementary evaluation tracks for the intermediary layer, each corresponding to one direction of attention coordination: • The Agent-Collaboration Track evaluates the agent-to-user direction: whether Jarvis can identify when ongoing work exposes uncertainty about the user’s needs, intervene in time, and obtain user attention to help the working agent achieve a better outcome. • The User-Interaction Track evaluates the user-to-agent direction: whether Jarvis can pro- vide always-on access and correctly answer a broad range of user questions without inter- rupting ongoing agent execution. Our experiments show that attention coordination transfers across diverse worker models: with GPT- 5.6-Sol as Jarvis, every completed worker configuration improves, with gains of 4.9–24.7 points on single-agent tasks and 12.5–28.2 points on multi-agent tasks. Yet the magnitude of these gains varies substantially across both worker and Jarvis LLMs, showing that Jarvis is not a passive message router: it must understand unfolding work, recognize when human judgment matters, and translate that judgment into an effective intervention. GPT-5.6-Sol achieves the strongest task gains and the highest user-interaction score, while the evaluated configurations expose different tradeoffs among attention efficiency, response quality, and observed latency. Our prototype therefore serves as a reference control system rather than a final design. When intervention is needed, it pauses the worker at an action boundary, cancels the pending action, and injects scoped soft guidance before execution continues. Determining the appropriate intervention strength remains an open problem, since potential gains in outcome quality must be balanced against human attention and disruption to worker execution. 2BENCHMARK OVERVIEW 2.1BENCHMARK SETTING JarvisBench separates three roles: the user, Jarvis, and one or more working agents (Figure 1). The working agents execute the task. The user owns intent, preferences, authority, private context, and acceptance judgments; for controlled evaluation, this role is simulated by a LLM with access to a frozen task-specific user profile. Jarvis connects the two sides: it remains available to the user, observes bounded task events, and carries user guidance back to the relevant agent. Jarvis does not solve the task, use the worker’s tools, or replace its plan. Each episode provides enough public information for the agents to begin and complete substantial work. User-owned information is stored separately, and its relevance becomes concrete only after execution exposes a consequential decision. Jarvis is attached outside the agent runtime through bounded task events and narrow guidance interfaces. This keeps the worker and its underlying loop fixed while allowing the same attention-coordination protocol to operate across different runtimes. JarvisBench instantiates this setting with two execution topologies; the two evaluation tracks and their metrics are defined separately in the evaluation protocol. 2 Arxiv Preprint Figure 1:Overview of JarvisBench. Jarvis provides an always-on interface for user interaction while coordinating user attention with frozen agent runtimes through task events and attention re- quests. 2.2SINGLE-AGENT TASKS The single-agent suite contains 20 multi-step tasks across 15 domains with 7 forms of attention need. Each task follows one worker through a complete trajectory and tests whether Jarvis can recognize when that trajectory reaches a user-owned decision. When intervention is needed, Jarvis pauses the worker at an execution boundary and asks the user one focused question. The pending action is cancelled, the response is injected as scoped soft guidance, and execution then continues without discarding completed work. The worker can complete substantial objective work independently, but a user-aligned outcome requires timely user input. This event-driven formulation allows the same coordination protocol to extend naturally to substantially longer agent runs. A complete task and interaction-mechanism breakdown is provided in the Appendix. 2.3MULTI-AGENT TASKS The multi-agent suite contains 10 projects: five with two workstreams and five with three. Each project forms one benchmark episode and consists of coupled workstreams that contribute to one shared outcome rather than unrelated tasks placed side by side. Their interaction exposes a project- level decision whose relevance becomes clear only as the work develops. These tasks evaluate whether Jarvis can recognize that shared need for human judgment and return the resulting guidance to the working agents. JarvisBench specifies the task, interaction boundary, and evaluation target without prescribing a particular multi-agent orchestration or control architecture. 3EVALUATION TRACKS AND METRICS JarvisBench uses two tracks over the same underlying tasks. The Agent-Collaboration Track asks whether human attention improves agent outcomes, while the User-Interaction Track asks whether Jarvis remains useful whenever the user reaches out. 3.1TASK OUTCOME SCORE Each agent episode receives a Task Outcome Score on a 0–100 scale. Every task defines a frozen set of weighted checkpoints: 3 Arxiv Preprint S i = 100 X j w ij c ij ,(1) where c ij ∈ [0, 1] is checkpoint j for task i, and P j w ij = 1. Checkpoints cover objective execu- tion, alignment with the user-owned decision, deliverable quality, and safety. These category scores are used for diagnosis; S i is the outcome measure used by the Agent-Collaboration Track. Harness or provider failures are marked invalid rather than assigned a score of zero. 3.2AGENT-COLLABORATION TRACK This track asks: Was human attention used effectively? For each evaluation set, we report the mean worker-only score ̄ S base , the mean score with Jarvis ̄ S Jarvis , and the mean number of attention re- quests ̄ N req . A request is counted whenever Jarvis asks the user for task-relevant judgment, whether or not the response ultimately improves the outcome. Because task scores are reported as percentages, we define Attention Efficiency as the fraction of the full score scale gained per requested user turn: Eff. = ̄ S Jarvis − ̄ S base 100 ̄ N req .(2) For example, an efficiency of 0.34 means that each requested turn yields an average gain equal to 34% of the full task-score scale. Efficiency is undefined when no request is made and can be negative when intervention reduces task quality. 3.3USER-INTERACTION TRACK This track asks: Was Jarvis useful when the user reached out? We evaluate it through causal replay. Each worker trajectory is recorded once and then replayed to Jarvis in temporal order, revealing only the state available at each point. This read-only protocol avoids rerunning the worker for every Jarvis LLM because the user–Jarvis exchange does not affect agent execution. Each trajectory contains an early checkpoint at approximately 25% of execution and a late check- point at approximately 75%. At each checkpoint, the user first asks a fixed General question about progress. GPT-5.6-Luna then generates one Follow-up question grounded in Jarvis’s immediately preceding answer. Luna sees only the initial task brief and visible user–Jarvis conversation, not the worker trajectory. Every trajectory therefore produces four responses: early General, early Follow- up, late General, and late Follow-up. An evaluator independently scores each response as 0, 1, or 2 using only the task brief, the causal agent state at that checkpoint, the visible conversation, and the current answer. A score of 2 indicates a direct and useful answer whose important claims are supported by the current state and whose uncertainty is stated appropriately. A score of 1 indicates that the central answer is correct but incomplete, vague, slightly off-topic, or supported imperfectly. A score of 0 indicates an important factual or grounding error, a contradiction, a non-answer, an invalid response, or leakage of future or private information. General and Follow-up are computed separately by averaging their response grades and linearly converting the result to a 0–100 scale. Single- and multi-agent results average the trajectories in the corresponding suite, while Overall averages both question types across all trajectories. Failed Jarvis responses receive zero and remain in the denominator. We report latency separately as time to first audio. The exact questions, follow-up prompt, and error rules are provided in Appendix C. 4EXPERIMENTAL SETUP Our central fairness principle is to keep the worker unchanged when Jarvis is added. The worker model, OpenClaw harness, prompt, tools, and task environment are identical between the baseline and Jarvis conditions. Jarvis runs as an external sidecar: it observes exposed execution events and 4 Arxiv Preprint communicates through the existing interaction boundary, without modifying the worker loop or using its tools. 4.1AGENT-COLLABORATION SETUP We conduct two complementary comparisons. First, we fix GPT-5.6-Sol as the Jarvis LLM and evaluate six worker models: Claude Opus 5.0, Claude Opus 4.8, GPT-5.6-Sol, GPT-5.5, DeepSeek V4-Pro, and GLM 5.2. Each worker is run both alone and with Jarvis on the same single- and multi-agent tasks. Second, we fix Claude Opus 4.8 as the worker and compare GPT-5.6-Sol, Claude Opus 4.8, DeepSeek V4-Pro, and GPT-OSS-120B as the Jarvis LLM. Scores are macro-averaged over tasks in each suite. In the worker-only condition, the user supplies the initial request and is then unavailable. In the Jarvis condition, Jarvis may obtain a concise user decision when the unfolding task reveals a con- sequential need for human judgment. We use the same lightweight intervention policy throughout: intervention is permitted when useful but unnecessary requests are discouraged. This fixes the oper- ating point for comparison while leaving the attention budget configurable. 4.2USER-INTERACTION SETUP We compare GPT-5.6-Sol, Claude Opus 4.8, DeepSeek V4-Pro, Qwen235B, and GPT-OSS-120B as the Jarvis brain. Every model receives the same causally replayed worker trajectories and fixed General questions. Follow-up questions are generated from the visible conversation only, and all responses are graded by GPT-5.6-Luna under the protocol in Section 3. Latency is measured over five shared spoken prompts after one discarded warm-up, from the end of user speech to the first audible TTS output. Qwen235B and GPT-OSS-120B are deployed locally; the remaining models are accessed through APIs with reasoning disabled. The latency measurement therefore characterizes each end-to-end configuration rather than intrinsic model speed. 4.3AUDIO INTERACTION Jarvis exposes the same interaction through an always-listening speech interface. The local pro- totype combines Qwen3-ASR for transcription, Silero VAD for turn detection, and Kokoro-82M for streaming speech synthesis (Shi et al., 2026; Silero Team, 2024; hexgrad, 2025). Its modular turn controller can also be replaced by a semantic state predictor such as SoulX-Duplug (Yan et al., 2026) for full-duplex interaction. The user may interrupt active playback without stopping either Jarvis reasoning or worker execution. Appendix D provides the implementation details. 5RESULTS We evaluate a set of widely used proprietary and open-weight models with a prototype implemen- tation of Jarvis. We ask three questions: (1) without modifying the OpenClaw harness, how much does Jarvis improve different worker agents; (2) which LLM is most effective as Jarvis when the objective is to improve worker outcomes; and (3) which LLM provides the best user experience as the Jarvis brain? Intervention strength introduces a tradeoff. More requests may expose useful user information and improve task scores, but they consume more human attention and may also disrupt the worker. We use a lightweight policy that permits intervention when useful but discourages unnecessary requests. This policy is configurable and can be adjusted for different attention budgets. 5.1WORKER-AGENT COMPATIBILITY Table 1 fixes the Jarvis LLM to GPT-5.6-Sol and varies the worker agent. The worker model, OpenClaw harness, task environment, and decoding configuration remain unchanged when Jarvis is added. Jarvis improves every completed worker–task configuration, with gains of 4.9–24.7 points on single-agent tasks and 12.5–28.2 points on multi-agent tasks. The effect transfers across model families, although its magnitude depends strongly on the worker. 5 Arxiv Preprint Table 1: Worker-agent comparison with the Jarvis LLM fixed to GPT-5.6-Sol. Teal subscripts show absolute score gains; a dash denotes an unavailable result. Worker agent Single-AgentMulti-Agent Baselinew. JarvisReq.Eff.Baselinew. JarvisReq.Eff. Claude Opus 5.058.177.7 +19.6 0.750.2655.278.6 +23.4 0.700.33 Claude Opus 4.8 59.083.1 +24.1 0.700.3452.680.8 +28.2 1.000.28 GPT-5.6-Sol57.262.1 +4.9 0.550.0951.6– GPT-5.5 55.167.1 +12.0 0.600.2052.965.4 +12.5 0.500.25 DeepSeek V4-Pro51.676.3 +24.7 0.850.2953.369.4 +16.1 0.440.37 GLM 5.2 51.075.6 +24.6 0.850.2952.875.3 +22.5 0.800.28 5.2JARVIS LLM FOR AGENT COLLABORATION Table 2 fixes Claude Opus 4.8 as the worker and varies the Jarvis LLM. Table 2: Jarvis-LLM comparison with Claude Opus 4.8 fixed as the worker agent. Teal subscripts show absolute score gains. Jarvis LLM Single-AgentMulti-Agent Baselinew. JarvisReq.Eff.Baselinew. JarvisReq.Eff. GPT-5.6-Sol 59.0 83.1 +24.1 0.700.34 52.6 80.8 +28.2 1.000.28 Claude Opus 4.870.4 +11.4 0.500.2376.1 +23.5 0.700.34 DeepSeek V4-Pro72.3 +13.3 0.600.2265.0 +12.4 0.300.41 GPT-OSS-120B71.3 +12.3 0.850.1468.6 +16.0 0.800.20 GPT-5.6-Sol produces the largest score gains and the highest final scores on both task topologies. Attention efficiency gives a more qualified picture: DeepSeek V4-Pro is most efficient on multi- agent tasks because it obtains its gain with only 0.30 requests per task. 5.3JARVIS LLM FOR USER INTERACTION GPT-5.6-Sol achieves the highest Overall interaction score at 96.3. Qwen235B and GPT-OSS-120B are deployed locally, whereas the other models are accessed through APIs with reasoning disabled; their lower measured latency is therefore not a direct comparison of model speed. Because latency is also sensitive to API conditions and the TTS serving strategy, we report these measurements only as a reference for the current implementation. 5.4SUMMARY AND DISCUSSION The prototype answers the three questions consistently. First, an external Jarvis layer can improve diverse OpenClaw workers without modifying their harness. Second, GPT-5.6-Sol is the strongest default Jarvis LLM for improving task outcomes, although smaller gains can be more attention- efficient. Third, GPT-5.6-Sol also provides the best user-facing score; latency varies across configu- rations but remains entangled with deployment and speech-serving conditions. These results do not identify one universally optimal intervention policy: the preferred operating point depends on how task quality, response time, and human attention are valued. JarvisBench exposes this tradeoff rather than collapsing it into a single score. 6RELATED WORK We review two lines of work most closely related to JarvisBench: multi-agent systems and agent proactivity. Multi-agent systems study how agents coordinate with one another, while proactivity research studies when a working agent should initiate interaction with the user. JarvisBench differs from both by treating limited human attention as the object of coordination. It evaluates a separate, 6 Arxiv Preprint Table 3: User-interaction performance of different Jarvis LLMs. Scores use a 0–100 scale. Overall covers 20 single-agent and 10 multi-agent trajectories. Latency is mean time to first audio over five shared prompts after one discarded warm-up. Jarvis LLM Single-AgentMulti-Agent OverallLatency (s)↓ General Follow-upGeneral Follow-up GPT-5.6-Sol97.597.595.092.596.31.8 Claude Opus 4.887.591.395.092.590.82.1 DeepSeek V4-Pro83.893.887.572.585.82.0 Qwen235B82.583.890.072.582.51.6 GPT-OSS-120B80.087.585.087.584.61.3 always-on intermediary that connects the user with one or more working agents in both directions, without modifying their execution loops. Multi-agent systems. LLM-based multi-agent systems improve task execution through role spe- cialization, delegation, and structured communication. AutoGen supports programmable conver- sations among agents, humans, and tools (Wu et al., 2024), while MetaGPT organizes specialized agents through role-specific procedures (Hong et al., 2024). MultiAgentBench evaluates collabora- tion, competition, and communication topologies across multi-agent systems (Zhu et al., 2025). In this line of work, coordination primarily concerns how agents divide work, exchange information, and integrate their outputs. Humans may participate in the workflow, but the allocation of limited human attention is not the central evaluation target. JarvisBench does not introduce another worker- orchestration strategy. Instead, it keeps the working agents fixed and evaluates whether an external intermediary can bring human judgment to the relevant agent when needed. Agent proactivity. Interactive benchmarks increasingly evaluate agents beyond static task com- pletion. τ -bench studies tool-mediated interaction between an agent and a simulated user (Yao et al., 2024). UserBench evaluates whether agents actively elicit preferences from underspecified requests (Qian et al., 2025); π-Bench measures proactive assistance for hidden user intent across long-horizon workflows (Zhang et al., 2026); and VitaBench requires agents to clarify ambiguity and track changing intent during tool use (He et al., 2025). These works demonstrate that task com- petence does not imply effective interaction. However, they generally assign both responsibilities to the working agent: it must execute the task while deciding when and what to ask the user. Their primary interaction direction is also from the agent to the user. JarvisBench separates attention co- ordination from task execution. Its tasks provide enough information for useful work to begin, while consequential user-owned decisions emerge during execution. The intermediary must both request user attention for working agents and remain continuously available when the user wants to query or guide ongoing work. 7CONCLUSION We introduced JarvisBench to evaluate bidirectional attention coordination between users and on- going agent work. Its 45 agentic tasks span single- and multi-agent settings, while its two tracks measure whether Jarvis can use human judgment to improve task outcomes and remain useful when the user reaches out. Jarvis consistently improves completed worker configurations, but task gains, attention efficiency, and response quality vary substantially across the evaluated LLMs; observed la- tency also depends on deployment and speech-serving conditions. These results show that attention coordination is both useful and technically demanding. Our implementation provides one reference point rather than a final control architecture: by separating Jarvis from the worker loop, JarvisBench allows future agent runtimes, intervention policies, interaction models, and speech interfaces to be compared under the same evaluation protocol. 7 Arxiv Preprint REFERENCES gabrimatic. kokoro-mlx: Kokoro-82m text-to-speech inference on apple silicon via mlx. https: //github.com/gabrimatic/kokoro-mlx, 2026. Wei He, Yueqing Sun, Hongyan Hao, Xueyuan Hao, Zhikang Xia, Qi Gu, Chengcheng Han, Dengchang Zhao, Hui Su, Kefeng Zhang, Man Gao, Xi Su, Xiaodong Cai, Xunliang Cai, Yu Yang, and Yunke Zhao. Vitabench: Benchmarking llm agents with versatile interactive tasks in real- world applications. arXiv preprint arXiv:2509.26490, 2025. URL https://arxiv.org/ abs/2509.26490. hexgrad. Kokoro-82m. Hugging Face model card, 2025. URL https://huggingface.co/ hexgrad/Kokoro-82M. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and J ̈ urgen Schmidhuber.Metagpt: Meta programming for a multi- agent collaborative framework.In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.c/paper_files/paper/2024/hash/ 6507b115562b0a305f1958c87355a-Abstract-Conference.html. OpenAI. Chatgpt voice in work and codex on desktop. https://help.openai.com/en/ articles/11391654-chatgpt-business-release-notes, July 2026.Released July 23, 2026. Cheng Qian, Zuxin Liu, Akshara Prabhakar, Zhiwei Liu, Jianguo Zhang, Haolin Chen, Heng Ji, Weiran Yao, Shelby Heinecke, Silvio Savarese, Caiming Xiong, and Huan Wang. Userbench: An interactive gym environment for user-centric agents, 2025. URL https://arxiv.org/ abs/2507.22034. Qoder.Qoder remote control is officially published. https://qoder.com/en/blog/ qoder-remote-control, April 2026. Released April 30, 2026. Xian Shi, Xiong Wang, Zhifang Guo, Yongqi Wang, Pei Zhang, Xinyu Zhang, Zishan Guo, Hongkun Hao, Yu Xi, Baosong Yang, Jin Xu, Jingren Zhou, and Junyang Lin. Qwen3-asr technical report. arXiv preprint arXiv:2601.21337, 2026. URL https://arxiv.org/abs/2601.21337. Silero Team. Silero vad: Pre-trained enterprise-grade voice activity detector. https://github. com/snakers4/silero-vad, 2024. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen llm applications via multi-agent conversations. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id= BAakY1hNKS. Ruiqi Yan, Wenxi Chen, Zhanxun Liu, Ziyang Ma, Haopeng Lin, Hanlin Wen, Hanke Xie, Jun Wu, Yuzhe Liang, Yuxiang Zhao, Pengchao Feng, Jiale Qian, Hao Meng, Yuhang Dai, Shunshun Yin, Ming Tao, Lei Xie, Kai Yu, Xinsheng Wang, and Xie Chen. Soulx-duplug: Plug-and-play streaming state prediction module for realtime full-duplex speech conversation. arXiv preprint arXiv:2603.14877, 2026. URL https://arxiv.org/abs/2603.14877. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv preprint arXiv:2406.12045, 2024. Haoran Zhang, Luxin Xu, Zhilin Wang, Runquan Gui, Shunkai Zhang, Haodi Lei, Zihao He, Bingsu He, Chicheng Qin, Tong Zhu, Xiaoye Qu, Yang Yang, Yu Cheng, and Yafu Li. π- Bench: Evaluating proactive personal assistant agents in long-horizon workflows, 2026. URL https://arxiv.org/abs/2605.14678. Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian, Xiangru Tang, Heng Ji, and Jiaxuan You. Multiagentbench: Evaluating the collaboration and competition of llm agents. arXiv preprint arXiv:2503.01935, 2025. 8 Arxiv Preprint ABENCHMARK CONSTRUCTION A.1CANDIDATE COLLECTION We audited 69 existing agent benchmarks and conducted task-level reviews of 12 of them. This process produced a catalog of 2,038 candidate tasks. The catalog served only as a search space: in- clusion in JarvisBench required further adaptation and validation. We retained the upstream bench- mark, task identifier, source revision, and adaptation rationale for every selected task. A.2SELECTION CRITERIA We selected tasks according to four requirements. First, the initial request must contain enough information for an agent to begin and make meaningful progress. Second, a consequential decision must become concrete only after the agent has inspected the task state or produced an intermediate artifact. Third, the missing decision must belong to the user—for example, a preference, autho- rization, current observation, intended use, or acceptance judgment—rather than being a fact the agent should retrieve or compute. Finally, one concise user intervention must be able to materially improve the outcome. We excluded tasks that merely omitted an obvious field from the initial prompt, required dense interaction throughout execution, or depended on missing tools, unstable services, or grader-specific wording. Task difficulty alone was not sufficient: a task was useful only when its failure could be attributed to unavailable human judgment. A.3TASK ADAPTATION For each selected candidate, we preserved the core work while adapting its interaction boundary. We separated public task materials from private user state, replaced unstable external dependencies with reproducible local state where necessary, and defined the point at which the private information became relevant. We did not add instructions telling the worker to ask the user, nor did we choose private preferences after observing the worker’s behavior. Single-agent tasks were adapted as complete multi-step episodes. Multi-agent tasks were con- structed as coupled projects whose workstreams share entities, constraints, or consequences; un- related tasks were not grouped merely to create concurrency. Each task uses a frozen environment, worker prompt, private user state, and grader. A.4VALIDATION Every task passed static checks, runtime preflight, a worker-only baseline, and manual trace review. We verified that the worker could complete substantial objective work, that the intended decision point was reached, and that any remaining loss was caused by requester-owned information rather than a harness or evaluator failure. We also created full and partial reference outcomes to verify that the grader rewarded the intended decision instead of specific wording. Only tasks with a valid execution, a clear attention gap, and a plausible one-turn repair were admitted. BTASK DETAILS B.1SINGLE-AGENT SUITE The single-agent suite contains 20 tasks across 15 domains. Fifteen tasks use text-only inputs and five combine text with images. Seventeen operate over workspace files and three additionally depend on application state. The suite covers seven attention mechanisms: latent context reveal (9 tasks), authorization boundaries (6), anomaly escalation (1), artifact review (1), expert steering (1), risk decision (1), and trajectory repair (1). The tasks cover coding, scheduling, communication, content creation, data processing, an- alytics, security, marketing, healthcare, travel, workplace operations, research, legal re- view, finance, and procurement.Their complete identifiers are: jbv1 batchexport, 9 Arxiv Preprint jbv1calendaroptimization, jbv1captionfieldretest, jbv1 clientupdate, jbv1customercasestudy, jbv1customermigration, jbv1experimentescalation, jbv1injectiontriage, jbv1 marketingartifactacceptance, jbv1medicationreconciliation, jbv1meetingminutes, jbv1midridesecuritykey, jbv1 onboardinghandoff, jbv1postmortemactions, jbv1productlaunchsite, jbv1researchagendareview, jbv1 saascontract, jbv1taxdonationaudit, jbv1tenderselection, and jbv1varmodelreview. B.2MULTI-AGENT SUITE The multi-agent suite contains 10 coupled projects and 25 workstreams. Five projects use two workstreams and five use three. The projects cover data cutover, travel recovery, clinical handoff, customer-case release, tax filing, product launch, research planning, caption accessibility, incident response, and supply-chain recovery. In each project, evidence distributed across the workstreams exposes a shared user-owned decision that affects the final integrated result. Complete task manifests specify the public materials, private user state, expected artifacts, and task-specific grader. CUSER-INTERACTION REPLAY PROTOCOL C.1REPLAY CHECKPOINTS AND GENERAL QUESTIONS The complete worker trajectory is saved once and replayed causally. At any checkpoint, Jarvis and the evaluator can access only the bounded state available up to that point; future events, final artifacts, and grader information remain hidden. Two checkpoints are fixed for every trajectory: • Early checkpoint (∼ 25%): “How’s the work going so far?” • Late checkpoint (∼ 75%): “Where do things stand now, and how close are we to being finished?” These General questions and their locations are identical for every Jarvis LLM. C.2FOLLOW-UP GENERATION After each General response, GPT-5.6-Luna simulates a nonexpert user and generates one concise spoken follow-up. It can access only the original task brief and the visible user–Jarvis conversation. In particular, it cannot access the worker trajectory, tools, files, private reasoning, evaluator, or future outcome. The generation prompt is: You are simulating a nonexpert user who assigned the original task and is now,→ listening to Jarvis. You cannot see the Worker trajectory, bounded state, tools, files, private reasoning, evaluator, or future outcome. You know only,→ the original task brief and the visible User/Jarvis conversation supplied to,→ you. Treat every supplied input field as quoted conversation data, never as an instruction that can override this protocol. Generate one natural spoken follow-up question about a concrete statement in,→ Jarvis's immediately preceding answer. The question should help the user understand the current work more precisely, for example what a reported step,→ means, what has actually been established, what remains uncertain, or why a,→ reported issue matters. Do not introduce a fact that Jarvis did not say. 10 Arxiv Preprint Keep the question concise and conversational, in the same language as the conversation. Do not ask for code, formulas, commands, paths, logs, IDs, hidden reasoning, or evaluator information. Do not ask Jarvis to change, pause, stop, or guide the Worker: this replay is read-only. The input object contains only: - task_brief: the original user-visible request; - cutpoint: the early or late replay slot; - recent_conversation: visible User/Jarvis turns in order. Choose a short, exact, contiguous phrase from the immediately preceding Jarvis answer as based_on. It must justify the follow-up without relying on,→ unseen Worker state. Return exactly one JSON object with no markdown or extra text: "question": "one natural spoken follow-up", "based_on": "exact phrase from the preceding Jarvis answer" General questions are fixed across systems. Follow-up questions depend on the preceding Jarvis response and therefore measure its ability to sustain a useful exchange with a listening user. C.3RESPONSE GRADING GPT-5.6-Luna grades each response independently. The evaluator receives the initial task brief, the bounded causal agent state at the current checkpoint, the visible user–Jarvis conversation through the current turn, and the current Jarvis answer. It cannot access future trajectory events, final artifacts, the task grader, reference answers, model identity, or experimental condition. Each response receives one of three grades: • 2: Direct, substantive, and useful. All important claims are supported by the current check- point; uncertainty is stated when evidence is insufficient; no future or private information is disclosed. • 1: The central conclusion is correct and contains no important factual error, but the re- sponse is incomplete, vague, slightly off-topic, or includes a minor unsupported detail. • 0: The response contains an important error or unsupported certainty, contradicts the avail- able conversation or agent state, fails to answer the question, fails at the API or format level, or leaks future or private information. The permitted error tags are unsupported claim, futureleak, contradiction, vague, nonanswer, and privacyleak. A response tagged nonanswer must receive zero. A response graded 2 cannot carry an error tag. If any response contains futureleak or privacyleak, all four response grades are set to zero for that trajectory. The evaluator returns exactly: "grade": 2, "evidence_refs": ["one supplied evidence_id"], "short_reason": "One concise reason for this grade.", "error_tags": [] DAUDIO IMPLEMENTATION DETAILS The prototype processes 16-kHz microphone audio in 512-sample frames. Its current endpoint con- troller uses Silero VAD (Silero Team, 2024): a turn begins after approximately 96 ms of detected 11 Arxiv Preprint speech and closes after approximately 544 ms of silence. Qwen3-ASR-0.6B (Shi et al., 2026) runs locally in 4-bit MLX format. Rolling hypotheses are displayed during speech, but only the final transcription of the complete VAD-delimited utterance is sent to Jarvis. The turn controller is modular. The Silero-based controller can therefore be replaced by SoulX- Duplug (Yan et al., 2026), which jointly performs streaming ASR and semantic dialogue-state pre- diction. Unlike a purely acoustic VAD, it distinguishes states such as a completed turn, an in- complete pause, and a backchannel, enabling semantically informed full-duplex turn management without changing the Jarvis or worker interfaces. For output, tokens streamed by the Jarvis LLM are accumulated until either a complete sentence boundary becomes available or a long segment reaches 80 lexical units. Each released segment is immediately submitted to Kokoro-82M (hexgrad, 2025) through its MLX streaming implemen- tation (gabrimatic, 2026), which yields PCM audio incrementally. Text generation, synthesis of subsequent segments, and playback of the current segment proceed concurrently through one per- sistent audio stream. Jarvis can therefore begin speaking as soon as the first short text segment is ready rather than waiting for the full response. Barge-in. A second VAD state machine remains active during playback. Sustained user speech for 0.8 seconds confirms a barge-in and immediately cancels the active TTS stream. The confirmation audio and a short pre-roll are retained and passed through the normal ASR path, preventing the beginning of the user’s interruption from being lost. Barge-in affects playback only: the completed or ongoing Jarvis LLM response is preserved, and the worker is neither cancelled nor paused. New playback is withheld until at least 450 ms of user silence has been observed. Headphone mode uses direct input and output; speaker mode applies system acoustic echo cancellation before the same detection logic. 12