Paper deep dive
Second Thought: Reasoning in Parallel as LLM Agents Act and Observe
Zhensu Sun, Chengran Yang, Yunbo Lyu, Jieke Shi, David Lo
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/17/2026, 4:01:54 AM
Summary
The paper introduces 'Second Thought', a training-free inference framework for LLM agents that exploits the 'reasoning idle window' (the time between an agent's action and the environment's observation) to perform parallel auxiliary reasoning. By forking four auxiliary branches (Check, Recall, Rehearse, Alternative) during this idle period, the framework generates complementary thoughts that are merged into the main trajectory for future turns. This approach reduces main-thread sequential decoding and average turn counts across various benchmarks without increasing latency, improving accuracy in specific settings.
Entities (14)
Relation Signals (17)
Second Thought â evaluatedon â SWE-bench Pro
confidence 95% ¡ The evaluation is on three agentic benchmarks covering repository-level software engineering (SWE-Bench Pro)
Second Thought â evaluatedon â Ď3-bench
confidence 95% ¡ The evaluation is on three agentic benchmarks covering... multi-turn tool-calling dialogue (Ď3-bench)
Second Thought â evaluatedon â Terminal-Bench 2.1
confidence 95% ¡ The evaluation is on three agentic benchmarks covering... terminal operation (Terminal-Bench 2.1)
Second Thought â istrainingfree â true
confidence 95% ¡ we propose Second Thought, a training-free inference framework
Second Thought â testedwithmodels â DeepSeek-v4-Flash
confidence 95% ¡ with three reasoning LLMs from distinct model families (DeepSeek-V4-Flash, Qwen3.6-Plus, and MiniMax-M3).
Second Thought â testedwithmodels â Qwen3.6-Plus
confidence 95% ¡ with three reasoning LLMs from distinct model families (DeepSeek-V4-Flash, Qwen3.6-Plus, and MiniMax-M3).
Second Thought â testedwithmodels â MiniMax-M3
confidence 95% ¡ with three reasoning LLMs from distinct model families (DeepSeek-V4-Flash, Qwen3.6-Plus, and MiniMax-M3).
Second Thought â utilizes â
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:LLM agents in the ReAct paradigm alternate between reasoning, acting, and observing, but deliberate reasoning is confined to the Thought phase: while the agent serializes an action and waits for the environment, its reasoning is frozen. We identify this recurring interval for Action and Observation as a reasoning idle window and ask whether it can host additional reasoning in parallel that serves future turns. Therefore, we propose Second Thought, a training-free inference framework that forks four auxiliary branches the instant each Thought phase concludes, decodes them concurrently with the main loop, and merges the generated thoughts back when the environment observation arrives. In this way, Second Thought relocates the added reasoning off the main thread's sequential decoding path. Across three agentic benchmarks and three reasoning LLMs, Second Thought lowers the average turn count in all nine (model,benchmark) pairs and reduces main thread decoding in six of them by up to 43% (roughly 20% on average among those settings), while leaving it essentially unchanged in a seventh; Pass@1 shows no significant change in seven of nine pairs and the two significant differences are +12.4 and +10.2 points. Against a compute-matched control that forces an equivalent budget onto the main thread's own reasoning, it attains strictly higher Pass@1 with 1.3 to 3.2 less sequential decoding in all four settings where the control applies.
Tags
Links
- Source: https://arxiv.org/abs/2608.13667v1
- Canonical: https://arxiv.org/abs/2608.13667v1
Trouble viewing inline? Open PDF directly â
Full Text
45,769 characters extracted from source content.
Expand or collapse full text
Second Thought: Reasoning in Parallel as LLM Agents Act and Observe Zhensu Sun, Chengran Yang, Yunbo Lyu, Jieke Shi, David Lo Singapore Management University 80 Stamford Road, Singapore 178902 zssun, cryang, yunbolyu, jiekeshi, davidlo@smu.edu.sg Abstract LLM agents in the ReAct paradigm alternate between reason- ing, acting, and observing, but deliberate reasoning is confined to the Thought phase: while the agent serializes an action and waits for the environment, its reasoning is frozen. We identify this recurring ActionâObservation interval as a reasoning idle window and ask whether it can host additional reasoning in parallel that serves future turns. Therefore, we propose Sec- ond Thought, a training-free inference framework that forks four auxiliary branches the instant each Thought phase con- cludes, decodes them concurrently with the main loop, and merges the generated thoughts back when the environment observation arrives. In this way, Second Thought relocates the added reasoning off the main threadâs sequential decoding path. Across three agentic benchmarks and three reasoning LLMs, Second Thought lowers the average turn count in all nine modelâbenchmark pairs and reduces main-thread decod- ing in six of them by up to 43% (roughly 20% on average among those settings), while leaving it essentially unchanged in a seventh; Pass@1 shows no significant change in seven of nine pairs and the two significant differences are +12.4 and +10.2 points. A paired wall-clock replay confirms that these reductions can translate into 10.9% lower median per- task latency. Against a compute-matched control that forces an equivalent budget onto the main threadâs own reasoning, it attains strictly higher Pass@1 with 1.3Ă to 3.2Ă less sequential decoding in all four settings where the control applies. Code â https://anonymous.4open.science/r/2nd-thought Introduction Large Language Model (LLM)-based agents have emerged as a dominant paradigm for solving complex, multi-step tasks ranging from software engineering to interactive tool use (Lyu et al. 2026). Central to their success is the capacity for autonomous reasoning, often structured via text-based thought trajectories that plan, verify, and adapt to a chang- ing environment. Recent advances demonstrate that scaling inference-time compute yields substantial improvements in problem-solving accuracy. However, this performance gain comes at a cost: every additional reasoning token is generated on the critical path, so accuracy is purchased with a propor- tional increase in wall-clock latency. This prolonged waiting time renders long-reasoning agents slow for interactive use. CopyrightŠ 2027, Association for the Advancement of Artificial Intelligence (w.aaai.org). All rights reserved. Yet an agent is not reasoning throughout its workflow. A typical ReAct agent (Yao et al. 2023b) operates in a se- quential Thought â Action â Observation loop, and only the Thought phase produces substantive reasoning. Once a Thought concludes, the agentâs thought is naturally frozen: the Action phase materializes the formed plan into a tool invocation, and the Observation phase passively awaits the environmentâs response. This creates a recurring reasoning idle window on every turn, a period during which additional thinking is known to improve agent performance, yet no thinking takes place at all. This motivates our question: can this idle window be used for additional parallel reasoning that complements the reasoning chain generated so far? Exploiting this idle window introduces a fundamen- tally distinct dimension of parallel reasoning that is or- thogonal to existing paradigms. Prior techniques, from Self-Consistency (Wang et al. 2022) and Tree-of-Thought search (Yao et al. 2023a) to recent work on population-based selection (Zhou et al. 2026) and cross-branch information sharing (Wang et al. 2026), parallelize within the Thought phase by sampling multiple candidate reasoning chains hori- zontally. While effective at expanding the search space, they cannot be directly applied to the reasoning idle window due to its unique nature. First, because the Thought phase has already concluded and its action is being executed during the idle window, auxiliary reasoning launched here can no longer alter the current turnâs decision. Consequently, rather than competing as alternative branches, it must condition on the established trajectory to serve future turns. This elimi- nates the need for complex voting or aggregation steps, al- lowing auxiliary thoughts to integrate seamlessly via sim- ple concatenation. Second, the idle window is governed by a hard external deadlineânamely, the arrival of the envi- ronment observation. This arrival abruptly terminates any ongoing reasoning, demanding that auxiliary thoughts be interruption-friendly: branches cut off mid-generation must still leave behind valid, usable partial results for subsequent turns. To realize this idea, we propose Second Thought, a training-free inference framework that enables agents to ex- ploit reasoning idle windows for additional reasoning. Upon the completion of each Thought phase, Second Thought forks four auxiliary reasoning branches that run concurrently with the main loop until the observation returns. Each branch rea- arXiv:2608.13667v1 [cs.AI] 13 Aug 2026 Act 1: grep -rn"def parse_config" src/ Thought 1: The bug is in config loading. I need to locate the function parse_configand inspect the override logic. Obs1: (no matches) ReAct-styleAgent Task: The CLI crashes when users override timeoutvia environment variable. Fix the bug. Constraint: the public API of load_config()must remain backward compatible. Act 2: ls -R src/ Thought 2: Strange, the file may be elsewhere. Let me list the source tree. Obs2: src/cli.pysrc/cfg/loader.pysrc/cfg/env.py... Act 3: cat src/cfg/loader.py Thought 3: Maybe it is in loader.py. Let me open it. Obs3: ... def load_config(path, *, strict=True)delegates to_parse_cfg(...) Act 4: edit: def load_config(path, env, *, strict=True) Thought 4: Found it âthe override logic is in _parse_cfg. I will add a new required argument env to load_configto pass overrides explicitly. Obs4: edit applied Act 5: python -m pytesttests/ Thought 5: Now run the tests. Obs5: 27 failed âTypeError: load_config() missing required positional argument: 'envâ Thought 6: The signature change broke existing callers. I must revert and keep the API compatible. .... Act 1: grep -rn"def parse_config" src/ Thought 1: The bug is in config loading. I need to locate the function parse_configand inspect the override logic. Obs1: (no matches) Second Thought 1:If the grep returns empty, search by symbol fragment: grep -rnâparse_cfgâ src/(rehearse).Could instead reproduce the crash first (set env var, run CLI) to get a stack trace(alternative)... Act 2: grep -rn"parse_cfg" src/ Thought 2: The grep is empty âas anticipated, the name is likely an alias. Searching by fragment instead. Obs2: src/cfg/loader.py:41: def _parse_cfg(raw, env_overrides) ... Second Thought 2:Task constraint: load_config() public signature must stay backward compatible(recall).I assumed the bug lives in _parse_cfg; the coercion could also happen in env.py(check). ... Act 3: edit: coerce env override types in _parse_cfg Thought 3: Found _parse_cfgâthe env override is dropped w hen the value is numeric. Per the recalled constraint, I will fix it inside _parse_cfg without touching load_config'ssignature. Obs3: edit applied Second Thought 3:If the edit is correct, run the full suite (pytesttests/), not only the config module(rehearse).Did the coercion preserve string- valued overrides? (check) Act 4: python -m pytesttests/ Thought 4: As rehearsed, validating with the full test suite. Obs4: allpassed ReAct-styleAgent+ Second Thought Figure 1: An illustrative comparison on a bug-fixing task. Left: a vanilla ReAct-style agent wastes turns on a failed search and breaks backward compatibility (red), requiring a costly revert. Right: with Second Thought, auxiliary reasoning generated during idle windows (green) rehearses fallback searches, recalls the task constraint, and checks assumptions, steering the agent to a correct fix in fewer turns. sons along a distinct dimension: verifying the assumptions underlying the current plan, rehearsing likely next steps, re- calling relevant context from earlier in the trajectory, and drafting contingency plans in case the current one fails. Un- like horizontal branches, these four branches are not com- peting solution candidates but complementary perspectives on the same trajectory, so collecting their outputs is mere concatenation. Figure 1 illustrates the effect: on a bug-fixing task, a vanilla ReAct agent breaks a backward-compatibility constraint and must revert its edit, whereas the recalled con- straint and rehearsed fallback produced during idle windows steer the augmented agent directly to a correct fix. Crucially, every branch is instructed to emit its response as a stream of atomic thoughts: each atomic thought encapsulates a sin- gle self-contained concept and depends on no other, so that interrupting a branch mid-generation invalidates at most the thought currently being produced, while all previously com- pleted atomic thoughts remain available. When the observa- tion arrives for the main loop, the atomic thoughts generated so far are collected and appended to the end of the tool mes- sage, allowing the agent to build upon this additional thinking in the next round of reasoning. In our experiments, we demonstrate the feasibility and value of utilizing the reasoning idle window with Second Thought. The evaluation is on three agentic benchmarks covering repository-level software engineering (SWE-Bench Pro), terminal operation (Terminal-Bench 2.1), and multi- turn tool-calling dialogue (Ď 3 -bench), with three reasoning LLMs from distinct model families (DeepSeek-V4-Flash, Qwen3.6-Plus, and MiniMax-M3). The results show that the additional reasoning from the idle window offers a better accuracyâlatency trade-off than both the unmodified agent and a compute-matched control that places the same or even more reasoning budget on main reasoning. Further analysis suggests two distinct sources of benefit: harvested thoughts pre-compute deliberation that would otherwise occupy the next turnâs sequential decoding, while the complementary reasoning dimensions surface considerations the main tra- jectory does not produce on its own. Our contributions can be summarized as follows: ⢠We identify and formalize the reasoning idle window in ReAct-style agent loops, i.e., the recurring Actionâ Observation interval in which no reasoning is generated, and characterize it as unused parallel capacity. ⢠We propose Second Thought, a training-free infer- ence framework that utilizes the reasoning idle window through four auxiliary branches (Check, Recall, Re- hearse, and Alternative), whose interruption-friendly atomic thoughts are harvested for the next turn, adding reasoning to the trajectory without adding sequential compute to the main thread. ⢠Across three benchmarks and three reasoning LLMs, Sec- ond Thought lowers the average turn count in all nine modelâbenchmark pairs and main-thread sequential de- coding in seven, with significant Pass@1 gains in two pairs (+12.4 and +10.2, both on Terminal-Bench 2.1) and no significant degradation in other pairs. Related Work Parallel Reasoning Parallel reasoning overcomes the sequential nature of au- toregressive decoding by generating multiple reasoning units concurrently. The dominant paradigm parallelizes horizon- tally: multiple solution paths are explored independently and later aggregated. Early instantiations sample complete trajec- tories and merge them by voting (Wang et al. 2022) or tree search (Yao et al. 2023a), while recent work makes branching a native model capability, allowing the model to spawn and join child threads for decomposed subtasks (Schroeder et al. 2024; Pan et al. 2025a; Biju et al. 2025; Lian et al. 2025; Yang et al. 2025; Wu et al. 2025). Another line relaxes the isola- tion between branches through shared attention caches (Hsu et al. 2025; Rodionov et al. 2025) or a learned organizerâ worker protocol (Chi et al. 2025). Across this diversity, the branches remain competing or partitioned solution attempts whose outcomes must be reconciled by an explicit merge step, and parallelism serves to shorten the critical path of a single deliberation phase. Moreover, parallel reasoning has to wait for the slowest branches, introducing additional latency. In contrast, the auxiliary branches of Second Thought are not solution candidates: they extend a single live trajectory from its current frontier along complementary dimensions, so reintegration is mere concatenation, with no voting, rank- ing, or reconciliation. Closest to our motivation is sleep-time compute (Lin et al. 2025), which pre-computes reasoning over a static context before any query arrives, so that test- time latency is reduced for the queries that follow. Second Thought shares the intuition of relocating deliberation off the critical path, but operates on a window that is intrinsic to the agent loop rather than externally scheduled. Asynchronous and Speculative Agent Execution To accelerate the agent loop, recent works propose execut- ing components of the workflow out of order. Asynchronous execution decouples the model from external latency, over- lapping token decoding with tool execution (Gim, seob Lee, and Zhong 2024; Feng et al. 2026; Abhyankar et al. 2024; Xu et al. 2024; Sun et al. 2026) or reasoning streams with response generation (Yakushev et al. 2025). Speculative exe- cution anticipates future states, committing predicted actions or reasoning steps only after verification (Ye et al. 2025; Pan et al. 2025b; Huang et al. 2025; Zhong et al. 2026). Real-time agent frameworks combine both paradigms, al- lowing the main reason-and-act thread to continuously is- ThoughtActionObservation fork SecondThought Alternative MainThread CheckRehearseRecall Thought merge Async.Branch Time Anagentturn Reasoningidlewindow Figure 2: Workflow of Second Thought. sue correctable speculative tool calls while awaiting exter- nal updates (Hooper et al. 2026). Crucially, in all these ap- proaches, the overlapped computation is either the main tra- jectoryâs own continuationâtokens that would be generated anyway, simply rescheduledâor a speculative prediction of it, guarded by correction machinery. Consequently, the total reasoning content along the trajectory remains unchanged, and task accuracy is typically preserved (Hooper et al. 2026). Second Thought inverts this paradigm: rather than overlap- ping computation merely to hide tool latency, it leverages this idle time to inject additional reasoning that does not lengthen the critical path, unlike test-time scaling, which places ev- ery additional token in sequence These auxiliary advisory thoughtsâwhich would not otherwise existâare consumed in the subsequent round rather than committed as actions, thereby requiring no verification or rollback mechanisms. Method Preliminaries: The Reasoning Idle Window We consider a standard ReAct-style agent (Yao et al. 2023b) that solves a task over a sequence of turns. At turn t, con- ditioned on the trajectory history H t , the agent proceeds through three phases: ⢠Thought: the LLM generates explicit reasoning R t that assesses the current state and plans the next move. ⢠Action: the model serializes an action A t âa structured tool invocation, optionally accompanied by a user-facing descriptionâthat instantiates the plan in R t . ⢠Observation: the environment executes A t and returns feedback O t , which is appended to the history: H t+1 = H t âŞR t ,A t ,O t . LetĎ think t ,Ď act t , andĎ obs t denote the wall-clock durations of the three phases; the total turn latency is Ď t = Ď think t +Ď act t +Ď obs t . Crucially, the ReAct format confines deliberate reasoning to the Thought phase. The Action phase merely renders the plan already articulated inR t into executable form, resolving at most low-level details such as argument literals, while the Observation phase involves no decoding at allâneither pro- duces reasoning directed at subsequent turns. We therefore refer to this contiguous interval, W t = Ď act t + Ď obs t , as the reasoning idle window, which recurs on every turn. Second Thought is designed to capitalize on W t by overlapping aux- iliary reasoning with the main loopâs Action and Observation phases. Overview Figure 2 illustrates the overall workflow of Second Thought. While the main thread advances through the standard ThoughtâActionâObservation loop, Second Thought runs alongside it on asynchronous auxiliary branches, injecting additional reasoning compute without delaying the crit- ical path. Specifically, the instant the main thread fin- ishes its Thought phase, i.e., at the onset of W t âSecond Thought triggers a fork operation, spawning four indepen- dent branches that each explore a complementary reasoning dimension: Check, Recall, Rehearse, and Alternative. These branches decode concurrently with the main threadâs action serialization and tool execution, occupying the entire reasoning idle window; since the windowâs duration is un- predictable, each branch formats its output as a stream of atomic thoughts, self-contained units that remain valid under interruption at any point. Once the observation O t arrives, a merge operation takes place: all active branches are immedi- ately terminated, and the atomic thoughts completed within the window are harvested and appended to the context. Con- sequently, at turnt+1, the main thread enters its next Thought phase already in possession of these insights. By construc- tion, Second Thought adds no reasoning tokens to the critical path: branches decode only while the main thread serializes the action and awaits the observation, and the merge reduces to truncating and concatenating text buffers. Forking via Trajectory Continuation Rather than instantiating an external critic to review the tra- jectory, Second Thought obtains auxiliary reasoning by let- ting the same model continue its own state. At the fork instant, each branch receives an identical conversation snapshotâthe complete history H t together with the newly generated R t â followed by a short branch-specific instruction. This design carries two practical benefits: the branches inherit the full task context verbatim, requiring no summarization or hand- off protocol; and because all branches share the prompt prefix up to R t with the main thread, the prefix KV cache is reused across all decoding streams. Finally, branch decoding dis- ables any model-native thinking to emit explicit thoughts as fast as possible before the window closes. Auxiliary Reasoning Protocol The auxiliary branches must satisfy two critical require- ments: (i) since their execution is strictly bounded by the unpredictable duration of W t , their outputs must remain us- able even if interrupted at an arbitrary token; and (i) to maximize the utility of the reasoning window, their content should complement, rather than duplicate, the main trajec- toryâs thought process. We address these challenges through Atomic Thoughts and Complementary Reasoning Dimen- sions, respectively. Atomic Thoughts To fulfill the first requirement, Sec- ond Thought enforces a unified output contract: every auxiliary branch streams its generation as a sequence of atomic thoughtsâstructurally independent reasoning units, each capturing a single self-contained insight. To ensure that mid-generation truncation yields valid and usable par- tial outputs, we enforce two structural constraints. Firstly, each atomic thought is enclosed in explicit XML tags (<thought> . . . </thought>), providing unambiguous boundary cues for parsing. Secondly, each thought focuses on a single localized point (⤠25 words) and avoids for- ward or backward references to other units in the stream. Under this contract, halting a branch mid-generation invali- dates at most the single unit currently in flight; all previously closed units remain structurally intact and semantically inter- pretable. Consequently, the main thread can terminate aux- iliary branches unilaterally upon receiving an observation, requiring zero graceful-exit protocols or branch-side coordi- nation. Reasoning Dimensions Second Thought is inherently modular: the number and semantics of auxiliary branches can be dynamically tailored to specific task characteristics, com- pute budgets, or domain requirements. Our reference instan- tiation for general agentic workflows targets four recurring failure modes of LLM agents, structured along two orthog- onal axes: temporal direction (retrospective vs. prospective) and scope (current turn vs. overall history). Crossing these axes yields four distinct branches, each mitigating a specific vulnerability: ⢠Check counters unverified assumptions, where agents prematurely commit to reasoning that silently presup- poses unconfirmed environmental states (Deshpande et al. 2025). It audits the newly finalized R t for fragile as- sumptions that the incoming observation might invalidate (e.g., âassumed the test framework is pytest without check- ing the configurationâ), flagging potential vulnerabilities without attempting immediate resolution. ⢠Recall counters context attrition, where constraints in- troduced early in a long trajectory lose efficacy as they recede into the context window (Liu et al. 2024). It ac- tively resurfaces critical historical context from H t that remains relevant but may have faded from active attention. ⢠Rehearse counters unprepared outcomes, where agents waste extra turns re-planning from scratch when a tool output deviates from expectations (Sun et al. 2023). It pre-computes conditional next steps for plausible out- comes (e.g., âif grep returns no matches, fall back to symbol-based lookupâ), enabling turnt+1 to execute pre- formulated reactions instantly. ⢠Alternative counters premature commitment, where agents stubbornly persist with a failing strategy rather than pivoting (Yao et al. 2023a). It proactively generates alternative candidate strategies for the active goal along with their trigger conditions, maintaining viable fallback avenues. For specialized deployments, this set can be seamlessly pruned, expanded, or customized (e.g., incorporating a domain-rule verification branch for legal or medical agents). Thought Harvesting When the observation O t arrives, Second Thought imme- diately cancels all in-flight branch generations and harvests Table 1: Overall performance of Second Thought across three benchmarks. #OUT main and #Turns respectively in- dicate the number of output tokens and turns of the main thread. Cells marked âââ denote settings where the s1 control is not applicable. We use BenjaminiâHochberg cor- rected McNemar test for Pass@1 and Wilcoxon signed-rank for rest metrics. Bench Model Setting Pass@1 #OUT main #Turns SWE-Pro DS-V4 base 48.7 23,841 56.2 s149.3 54,463* 58.5 ours 52.0 20,255* 52.8* Qwen3.6 base 52.0 36,519 57.1 s148.7 65,634* 55.7 ours 51.3 20,798* 50.6* M-M3 base 55.3 15,400 79.6 s1â ours 55.3 14,579 71.9* TB2 DS-V4 base 50.6 48,202 40.2 s150.6 68,003* 33.4* ours 52.8 32,892* 35.8* Qwen3.6 base 39.3 25,158 25.5 s146.1 40,642* 22.1 ours 51.7* 31,396 24.0 M-M3 base 49.4 36,686 44.6 s1â ours 59.6* 36,705 43.1 Ď 3 -bench DS-V4 base 24.0 16,260 27.5 ours 24.0 15,203 26.3 Qwen3.6 base 16.7 16,755 26.1 ours 19.8 13,764* 25.9 M-M3 base 17.7 10,004 24.0 ours 20.8 10,367 20.0* â p < 0.05 their output buffers as they stand. Each buffer is truncated at its last closed </thought> tag, discarding any incomplete unit, and the surviving thoughts are capped at 5 per dimen- sion, bounding per-turn context growth to a small constant. A branch that has not completed a single unit is simply omitted from the harvest; in the limiting case where no branch yields output, the agent degrades to the baseline ReAct loop. The merged block of atomic thoughts is appended to the end of the tool-observation message. On turn t+1, the model thus reads its updated history sequentially, i.e., prior reasoningR t , action A t , observation O t , and the attached second thoughts, and conditions its next Thought phase on all of them. Experiments Experimental Setup Benchmarks. We evaluate on three agentic benchmarks chosen to cover different tasks. SWE-Bench-Pro (Deng et al. 2025) is repository-level software engineering: each instance places the agent in a per-instance Docker container, where it must localize and patch a real bug, graded by held-out fail-to-pass tests. We use a randomly sampled subset of 150 instances with a 100-turn step limit. Terminal-Bench 2.1 (Merrill et al. 2026) comprises 89 containerized terminal- operation tasks, such as system administration, data process- ing, and software builds, graded by task-specific verifica- tion scripts. Ď 3 -bench (Barres et al. 2025) (banking domain, 97 tasks) is multi-turn customer-service dialogue grounded in knowledge retrieval: the agent converses with an LLM- simulated user while issuing function calls, and must ground its responses in policy documents retrieved from an unstruc- tured corpus. Agent. Each agent setup pairs a reasoning LLM with a benchmark-specific harness. We evaluate three reason- ing LLMs from distinct model families: DeepSeek-V4- Flash (et al 2026), Qwen3.6-Plus (Qwen Team 2026), and MiniMax-M3 (MiniMax 2026), all accessed via streaming APIs with their native reasoning/thinking modes enabled. For the execution harnesses, SWE-Bench-Pro employs mini- SWE-agent (Yang et al. 2024), Terminal-Bench 2.1 uses its official execution harness, and Ď 3 -bench uses its standard function-calling dialogue loop paired with the benchmarkâs user simulator. Baselines. We compare against two baselines. base is the unmodified agent. s1 adapts budget forcing (Muennighoff et al. 2025) to serve as a compute-matched control: it spends the same additional reasoning budget as Second Thought, but places it on the critical path as a continuation of the turnâs own thought. Concretely, whenever the main Thought would terminate before consuming the token budget that Second Thoughtâs branches produce in that turn, we suppress the end-of-thinking delimiter and force the model to continue reasoning until its thought length reaches at least that budget. This baseline isolates the effect of the amount of reasoning tokens. Notably, budget forcing requires suppressing the end- of-thinking delimiter and having the model continue its own thought from a prefilled prefix, but MiniMax-M3 does not honor an unclosed-thought prefix and emits a final answer instead of extending its deliberation, while on Ď 3 -bench pre- fix continuation cannot be combined with function-calling requests. Therefore, s1 is not available in these two settings. Main Results Table 1 presents the overall performance of Second Thought against the standard ReAct baseline and the compute- matched reasoning-extended baseline (s1) across nine benchmarkâmodel combinations. Second Thought reduces the average turn count in all nine pairs, and reduces main- thread decoding in six of them by up to 43% (from 36.5k to 20.8k tokens on SWE-Bench Pro with Qwen3.6-Plus) and by roughly 20% on average among those settings. Pass@1 is pre- served or improved in eight; the single decrease (from 52.0% to 51.3% on SWE-Bench Pro with Qwen3.6-Plus) amounts to one instance out of 150 and is not statistically significant, whereas the largest gain (+12.4 points on Terminal-Bench 2.1 with Qwen3.6-Plus) is. Notably, the only substantive increase 19,00020,00021,00022,00023,00024,00025,000 Output tokens 44 46 48 50 52 54 Pass@1 (%) baseline ours (all) only check only rehearse only recall only alternativew/o check w/o rehearse w/o recall w/o alternative baseline ours (all) only X w/o X Figure 3: Single-dimension (only-X) and leave-one-out (w/o- X) variants on SWE-Bench Pro with DeepSeek-V4-Flash; upper-left is better in main-thread decoding (+24.8%) occurs in exactly that set- ting, which suggests the harvested thoughts do not merely displace tokens the main thread would have produced, but can also open lines of reasoning it would not have pursued at all. Depending on how scaling reasoning length affects task accuracy, the overall trends fall into three distinct regimes. On tasks where simply extending the reasoning trace length (s1) hurts accuracy or inflates the main threadâs own decoding (e.g., SWE-Bench Pro with Qwen3.6-Plus, where s1 drives output tokens from 36,519 to 65,634 yet drops Pass@1 from 52.0% to 48.7%), Second Thought effectively suppresses redundant outputs while preserving accuracy. On Terminal-Bench 2.1 with DeepSeek-V4-Flash, it boosts Pass@1 to 52.8% (+2.2% over both base and s1) while re- ducing main-thread output tokens to 32,892, 32% below the baseline agent and 2.1Ă fewer than s1. On tasks where ex- tended reasoning improves accuracy (e.g., Terminal-Bench 2.1 with Qwen3.6-Plus), Second Thought delivers even su- perior Pass@1 gains (+12.4 points, +5.6 over s1) while decoding 1.3Ă fewer main-thread tokens than s1 (31,396 vs. 40,642) and taking fewer turns than the baseline agent (24.0 vs. 25.5). On Ď 3 -bench (banking), Second Thought achieves smaller Pass@1 gains and improves it by up to +3.1% (Qwen3.6-Plus and MiniMax-M3) at comparable or lower main-thread token counts and consistently fewer turns. We attribute this asymmetry to the nature of the domain: this domain has short idle window and banking failures stem largely from retrieval quality and policy adherence rather than from the planning errors that Check, Rehearse, and Alternative target, so the harvested thoughts shorten de- liberation without substantially changing which documents are retrieved. This points to a promising future direction: dynamically selecting branch types based on the domain. Ablation Study Figure 3 evaluates the trade-offs of single-dimension (only- X) and leave-one-out (w/o-X) variants using DeepSeek-V4- Flash on SWE-Bench Pro. Restricting to a single branch generally fails to match the full configurationâs performance, with most single-dimension variants falling near or below the baseline (48.7% Pass@1). Specifically, only-recall at- tains the fewest main-thread output tokens (19.7k) but drops Pass@1 to 46.0%, indicating that passive context retrieval without active validation or contingency planning is in- sufficient for accurate action selection. Conversely, leave- one-out experiments highlight the complementary roles of each component: removing Recall (w/o-recall) causes the sharpest accuracy drop (48.0% Pass@1), suggesting that resurfacing historical constraints helps prevent repeated er- rors, whereas removing Rehearse (w/o-rehearse) maintains Pass@1 (52.0%) but inflates main-thread output from 20.3k to 22.4k tokens (+10%), confirming that pre-computing con- ditional next steps primarily offloads deliberation the main thread would otherwise perform turn by turn. Overall, the full configuration is Pareto-optimal: no variant attains higher Pass@1, and the only variant that decodes fewer main-thread tokens (only-recall, 19.7k) sacrifices 6.0 points of accuracy for a 2.5% saving. In-depth Analysis Measured Timing Analysis. Raw wall-clock time from our main results is not comparable across our runs, which span different time periods with drifting API serving speed and fluctuating execution load on a shared server. We there- fore replay our experiment with a controlled setting to see if the reduced output tokens and turns can be translated to real reduction in latency. We replay 50 SWE-Bench Pro instances with DeepSeek-V4-Flash under a paired protocol: for each instance, base and ours run back-to-back within the same window against the same endpoint at a fixed concurrency of 4, repeated 3 times, and we report the median so that resid- ual drift affects both settings alike. Under this protocol, Sec- ond Thought reduces median per-task wall-clock time from 256.9 s to 229.0 s (â10.9%), decomposing along exactly the two quantities of Table 1: main-thread decoding time drops from 168.7 s to 146.1 s (â13.4%), broadly in proportion to the 15.0% fewer main-thread output tokens, while tool ex- ecution time drops from 71.6 s to 67.3 s (â6.0%), tracking the decrease from 56.2 to 52.8 turns at an unchanged 1.27 s per turn. The residual gap between the 15.0% token reduc- tion and the 13.4% time reduction is contention: running four branches lowers main-thread decoding throughput from 141.3 to 138.6 tokens/s (â1.9%), costing 2.8 s against the 27.9 s saved per task. This modest penalty can be attributed to branches decoding with native thinking disabled and shar- ing the prompt prefix KV cache with the main thread, so their serving load is dominated by cached prefill rather than sus- tained decoding. Overall, this demonstrates that the savings reported in Table 1 are not an accounting artifact of where tokens are attributed, but a genuine shortening of the critical path. Table 2: Controlled comparison on SWE-Bench Pro with DeepSeek-V4-Flash. w/o R t : branches see H t but not the just-finalized R t . fork onset: fork at turn start, widening the time window to cover the Thought phase. unbounded: the main thread stalls until branches close, so branch tokens fall on the critical path. MethodPass@1 #Output tokens #Turns #Atoms Main Branch baseline48.7 23.8k â56.20 s149.3 54.5k â58.50 ours52.0 20.3k 13.6k 52.8 376 - w/o R t 49.3 23.9k 21.4k 52.6 607 - fork onset 51.3 21.8k 48.6k 47.3 1,418 - unbounded 56.7 48.4k â43.6 846 Table 3: Per-task API cost of Second Thought on SWE- Bench Pro, computed over the same runs as Table 1 under each providerâs token prices on 1 Jul 2026. Setting #Tokens per task Cost per task Uncached Cached Generated DeepSeek-V4-Flash $0.140 / $0.0028 / $0.280 per M token base32.5k 1.27M 23.8k $0.015 ours (4 branches) 59.0k 2.45M 33.8k $0.025 (+66.4%) ours (1 branch) 40.9k 1.60M 25.0k $0.017 (+16.3%) Qwen3.6-Plus $0.325 / $0.325 / $1.950 per M token base34.0k 1.33M 36.5k $0.514 ours (4 branches) 92.3k 3.70M 32.4k $1.297 (+152.2%) ours (1 branch) 50.3k 1.82M 23.7k $0.653 (+27.0%) MiniMax-M3 $0.300 / $0.060 / $1.200 per M token base43.3k 2.13M 15.4k $0.159 ours (4 branches) 86.6k 6.49M 27.1k $0.448 (+181.5%) ours (1 branch) 52.5k 2.97M 18.0k $0.216 (+35.5%) Reasoning Substitution Effect. We measure the per-turn effect of harvested second thoughts with a replay experi- ment. We randomly sample 100 trajectories generated by DeepSeek-V4-Flash on SWE-Bench Pro. For each trajec- tory, we replay the turn that follows a harvest under two conditions: with the harvested thoughts in the context, or without them. The experiment is repeated three times. Re- moving these thoughts inflates next-turn reasoning volume from 196.2 to 316.5 tokens on average. This suggests that second thoughts substitute for reasoning that the agent would otherwise generate on the critical path of the next turn. Idle Window Utilization. We characterize the operational behavior of Second Thought across the three benchmarks and agents. Harvest yield directly scales with the duration of the reasoning idle window; when turns are binned by window length relative to their run median, the fraction yielding a non-empty harvest rises monotonically from 0.0% to 90.1%. One fixed configuration thus covers windows differing by orders of magnitude across three model families and three benchmarks, with no per-cell tuning. Coverage accordingly follows the idle time rather than the turn count: the 28.7% of turns that harvest hold 86.7% of it, and 96.5% of tasks receive second thoughts at least once. Disentangling Window Duration from Fork Condition- ing. Second Thought couples three choices that jointly de- termine what the branches produce: when they fork, what context they receive at the fork instant, and whether the ar- riving observation truncates them. Two of these vary the same underlying quantity â how much time the branches are given â by different means, and Table 2 separates it from the context they receive. Forking at turn onset buys a wider window by covering the Thought phase, but pays for it: the branches no longer see R t . Isolating that penalty at a fixed window, stripping R t costs 2.7 points (52.0% to 49.3%) even though it yields 1.6x the atoms (11.5 vs 7.1 per turn), since a branch writing without sight of the plan about to execute pro- duces more units of less use. Forking early incurs exactly this penalty yet still reaches 51.3%, within one instance of our default: the wider window is worth roughly +2.0 and nearly cancels it. Removing truncation isolates the window effect at fixed conditioning, and it achieves 56.7% Pass@1, +4.7 over our default. Window duration is therefore the dominant lever, and our default is window-starved, capturing 41% of the at- tainable gain over baseline (+3.3 of +8.0). The remainder is priced rather than unreachable: holding the main thread until the branches close moves their entire budget onto the critical path, raising sequential decoding from 20.3k to at most 48.4k tokens. Second Thought instead takes the endpoint at which that cost is zero, and the gap closes wherever the environment affords longer actionâobservation intervals. Cost Analysis. Table 3 summarizes per-task API costs on SWE-Bench Pro. Running four auxiliary branches increases the total API cost by 66.4% to 181.5% across models. This increase is almost entirely driven by input prompt process- ing rather than output generation. In fact, output token costs vary by less than $0.02 per task across all models. Instead, the cost overhead is dominated by cached prefix reads and tracks the providerâs cache discount heavily. Since all four branches share the main threadâs prefix KV cache, these reads incur near-zero marginal compute. For resource-constrained settings, keeping only the top-performing branch (ALTER- NATIVE) cuts the added cost down to 16.3%â35.5%. Conclusion We identify the reasoning idle window, i.e., the recurring actionâobservation interval within ReAct-style loops during which no reasoning is produced. We then propose Second Thought, a training-free framework that populates this win- dow with four auxiliary branches: Check, Recall, Rehearse, and Alternative. Interruption-safe atomic thoughts generated by these branches are collected and reused for the subsequent reasoning turn. Evaluated across three benchmarks and three reasoning LLMs, Second Thought lowers turn count in all nine configurations (significant in five) and main-thread de- coding in six (significant in four), while leaving Pass@1 statistically unchanged in seven of nine and significantly im- proved in two. References Abhyankar, R.; He, Z.; Srivatsa, V.; Zhang, H.; and Zhang, Y. 2024. InferCept: Efficient Intercept Support for Augmented Large Language Model Inference. Barres, V.; Dong, H.; Ray, S.; Si, X.; and Narasimhan, K. 2025. tau2-Bench: Evaluating Conversational Agents in a Dual-Control Environment.arXiv preprint arXiv:2506.07982. Biju, E.; Talaei, S.; Huang, Z.; Pourreza, M.; Mirhoseini, A.; and Saberi, A. 2025. SPRINT: Enabling Interleaved Planning and Parallelized Execution in Reasoning Models. ArXiv. Chi, Z.; Dong, L.; Dong, Q.; Hao, Y.; Wu, X.; Huang, S.; and Wei, F. 2025. The Era of Agentic Organization: Learning to Organize with Language Models. ArXiv. Deng, X.; Da, J.; Pan, E.; He, Y. Y.; Ide, C.; Garg, K.; Lauffer, N.; Park, A.; Pasari, N.; Rane, C.; et al. 2025. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks? arXiv preprint arXiv:2509.16941. Deshpande, D.; Gangal, V.; Mehta, H.; Krishnan, J.; Kan- nappan, A.; and Qian, R. 2025. TRAIL: Trace Reason- ing and Agentic Issue Localization. arXiv preprint arXiv: 2505.08638. et al, D.-A. 2026. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv preprint arXiv: 2606.19348. Feng, G.; Mao, H.; Dutta, P.; and Gonzalez, J. E. 2026. Concurrency without Model Changes: Future-based Asyn- chronous Function Calling for LLMs. Gim, I.; seob Lee, S.; and Zhong, L. 2024. Asynchronous LLM Function Calling. ArXiv. Hooper, C.; Kang, M.; Moon, S.; Lee, N.; Wen, E.; Wawrzynek, J.; Mahoney, M. W.; Shao, Y. S.; Gholami, A.; and Keutzer, K. 2026. Building Interactive Real-Time Agents with Asynchronous I/O and Speculative Tool Calling. Hsu, C.-J.; Buffelli, D.; McGowan, J.; Liao, F.; Chen, Y.-C.; Vakili, S.; and shan Shiu, D. 2025. Group Think: Multiple Concurrent Reasoning Agents Collaborating at Token Level Granularity. ArXiv. Huang, Z.; Zeng, W.; Fu, T.; Liu, T.; Sun, Y.; Hong, K.; Yang, X.; Liu, C.; Li, Y.; Zhang, Q.; Dai, G.; Zhu, Z.; and Wang, Y. 2025. Reducing Latency of LLM Search Agent via Speculation-based Algorithm-System Co-Design. ArXiv. Lian, L.; Wang, S.; Juefei-Xu, F.; Fu, T.-J.; Li, X.; Yala, A.; Darrell, T.; Suhr, A.; Tian, Y.; and Lin, X. 2025. Thread- Weaver: Adaptive Threading for Efficient Parallel Reasoning in Language Models. ArXiv. Lin, K.; Snell, C.; Wang, Y.; Packer, C.; Wooders, S.; Sto- ica, I.; and Gonzalez, J. E. 2025. Sleep-time Compute: Be- yond Inference Scaling at Test-time. arXiv preprint arXiv: 2504.13171. Liu, N. F.; Lin, K.; Hewitt, J.; Paranjape, A.; Bevilacqua, M.; Petroni, F.; and Liang, P. 2024. Lost in the middle: How language models use long contexts. Transactions of the association for computational linguistics, 12: 157â173. Lyu, Y.; Williams, D.; Shi, J.; Sun, Z.; Peng, C.; Yang, Z.; Sarro, F.; and Lo, D. 2026. How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study. arXiv preprint arXiv:2607.10856. Merrill, M. A.; Shaw, A. G.; Carlini, N.; Li, B.; Raj, H.; Bercovich, I.; Shi, L.; Shin, J. Y.; Walshe, T.; Buchanan, E. K.; et al. 2026. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868. MiniMax. 2026. MiniMax M3: Frontier Coding, 1M Con- text, Native Multimodality. https://w.minimax.io/blog/ minimax-m3. Muennighoff, N.; Yang, Z.; Shi, W.; Li, X. L.; Fei-Fei, L.; Hajishirzi, H.; Zettlemoyer, L.; Liang, P.; Candès, E.; and Hashimoto, T. B. 2025. s1: Simple test-time scaling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 20286â20332. Pan, J.; Li, X.; Lian, L.; Snell, C. B.; Zhou, Y.; Yala, A.; Dar- rell, T.; Keutzer, K.; and Suhr, A. 2025a. Learning Adaptive Parallel Reasoning with Language Models. ArXiv. Pan, R.; Dai, Y.; Zhang, Z.; Oliaro, G.; Jia, Z.; and Netravali, R. 2025b. SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoning. ArXiv. Qwen Team. 2026. Qwen3.6-Plus: Towards Real World Agents. Rodionov, G.; Garipov, R.; Shutova, A.; Yakushev, G.; Egiazarian, V.; Sinitsin, A.; Kuznedelev, D.; and Alistarh, D. 2025. Hogwild! Inference: Parallel LLM Generation via Concurrent Attention. ArXiv. Schroeder, P.; Morgan, N.; Luo, H.; and Glass, J. R. 2024. THREAD: Thinking Deeper with Recursive Spawning. Sun, H.; Zhuang, Y.; Kong, L.; Dai, B.; and Zhang, C. 2023. AdaPlanner: Adaptive Planning from Feedback with Lan- guage Models. arXiv preprint arXiv:2305.16653. Sun, Z.; Lin, Z.; Chen, Z.; Yang, C.; Zhou, M.; Li, L.; and Lo, D. 2026. Executing as You Generate: Hiding Execu- tion Latency in LLM Code Interpreters. arXiv preprint arXiv:2604.00491. Wang, X.; Lin, H.; Feng, S.; Yuan, P.; Li, Y.; Shi, J.; Zhang, Y.; Tan, C.; Zhang, J.; Pan, B.; et al. 2026. Share More, Search Less: Collaborative Parallel Thinking for Efficient Test-Time Scaling. arXiv preprint arXiv:2605.27030. Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E.; Narang, S.; Chowdhery, A.; and Zhou, D. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171. Wu, T.; Liu, Y.; Bai, J.; Jia, Z.; Zhang, S.; Lin, Z.; Wang, Y.; Zhu, S.-C.; and Zheng, Z. 2025. Native Parallel Reasoner: Reasoning in Parallelism via Self-Distilled Reinforcement Learning. ArXiv. Xu, Y.; Kong, X.; Chen, T.; and Zhuo, D. 2024. Conveyor: Efficient Tool-aware LLM Serving with Tool Partial Execu- tion. ArXiv. Yakushev, G.; Babina, N.; Dastgerdi, M. V.; Zhdanovskiy, V.; Shutova, A.; and Kuznedelev, D. 2025. Asynchronous Rea- soning: Training-Free Interactive Thinking LLMs. ArXiv. Yang, J.; Jimenez, C. E.; Wettig, A.; Lieret, K.; Yao, S.; Narasimhan, K. R.; and Press, O. 2024. SWE-agent: Agent- Computer Interfaces Enable Automated Software Engineer- ing. In The Thirty-eighth Annual Conference on Neural In- formation Processing Systems. Yang, X.; An, Y.; Liu, H.; Chen, T.; and Chen, B. 2025. Multiverse: Your Language Models Secretly Decide How to Parallelize and Merge Generation. ArXiv. Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T.; Cao, Y.; and Narasimhan, K. 2023a. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36: 11809â11822. Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y. 2023b. ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations (ICLR). Ye, N.; Ahuja, A.; Liargkovas, G.; Lu, Y.; Kaffes, K.; and Peng, T. 2025. Speculative Actions: A Lossless Framework for Faster Agentic Systems. ArXiv. Zhong, S.; Lu, B.; Chen, Q.; Liu, C.; Yang, F.; and Li, M. 2026. DualSpec: Accelerating Deep Research Agents via Dual-Process Action Speculation. ArXiv. Zhou, S.; Chai, W.; Liu, K.; Mao, H.; Mang, Q.; and Shang, J. 2026. OpenDeepThink: Parallel Reasoning via Bradley- Terry Aggregation. arXiv preprint arXiv:2605.15177.