Paper deep dive
StrategyBench: Evaluating Explicit Strategy Induction in Large Language Models
Jinghan Tan, Yuanzheng Wang, Lu Chen, Zijun Chen, Yuqian Wang, Maosong Sun
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As large language models are increasingly used in data-scarce and evolving task scenarios, few-shot in-context learning (ICL) has become a key paradigm for task adaptation. However, direct ICL often uses a small set of examples without explicitly abstracting task rules, making it sensitive to example construction. In contrast, human learners often reduce such sensitivity by first summarizing task rules from examples and then applying them to new instances. To evaluate this ability, we propose StrategyBench, which selects strategy-inducible tasks from BIG-Bench, constructs reference strategies, and defines evaluation metrics along two dimensions: strategy quality and downstream utility. We further analyze strategy induction from three perspectives: task variation, model configuration, and adaptation setting, covering category-wise differences, generator-executor choices, demonstration design, and SFT-based adaptation. Experiments show that explicit strategy utility differs substantially across task categories and depends on both strategy generation and execution conditions. The benchmark is released at: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.23475v1
- Canonical: https://arxiv.org/abs/2608.23475v1
Trouble viewing inline? Open PDF directly â
Full Text
69,562 characters extracted from source content.
Expand or collapse full text
StrategyBench: Evaluating Explicit Strategy Induction in Large Language Models Jinghan Tan, Yuanzheng Wang, Lu Chen, Zijun Chen, Yuqian Wang, Maosong Sun Abstract As large language models are increasingly used in data-scarce and evolving task scenarios, few- shot in-context learning (ICL) has become a key paradigm for task adaptation. However, direct ICL often uses a small set of examples without explicitly abstracting task rules, mak- ing it sensitive to example construction. In con- trast, human learners often reduce such sensitiv- ity by first summarizing task rules from exam- ples and then applying them to new instances. To evaluate this ability, we propose STRATEGY- BENCH, which selects strategy-inducible tasks from BIG-Bench, constructs reference strate- gies, and defines evaluation metrics along two dimensions: strategy quality and downstream utility. We further analyze strategy induction from three perspectives: task variation, model configuration, and adaptation setting, covering category-wise differences, generator-executor choices, demonstration design, and SFT-based adaptation. Experiments show that explicit strategy utility differs substantially across task categories and depends on both strategy gener- ation and execution conditions. 1 Introduction Large language models have shown strong task adaptation ability under in-context learning (ICL) (Brown et al., 2020; Wei et al., 2022; Wang et al., 2023). In standard few-shot ICL, which we refer to as direct ICL, models solve new tasks by conditioning on a small set of input-output exam- ples. However, this paradigm often exploits exam- ples only at a surface level and struggles to capture stable task rules. The limited context window re- stricts the number and diversity of examples, pre- venting models from observing sufficient example variations (Agarwal et al., 2024; Liu et al., 2021; Lu et al., 2022; Min et al., 2022). Moreover, even with more examples, models may rely on surface patterns or shortcuts rather than infer robust rules. Direct Few-shot ICL Strategy-based Solving Input Output Answer 7² is odd. Answer Strategy even : square is even. odd : square is odd. Unseen Q If 7 is odd, what is 7²? Example2 A: 2² is even. Example1 A: 3² is odd. step1 7² is odd. Example1 Unseen Q If 7 is odd, what is 7²? even : square is even. odd : square is odd. Strategy A: 2² is even. Example2 A: 3² is odd. step2 Q: If 2 is even, what is 2²? Q: If 3 is odd, what is 3²? Q: If 2 is even, what is 2²? Q: If 3 is odd, what is 3²? Input Output Figure 1: Comparison between direct few-shot ICL and strategy-based solving. In direct few-shot ICL, the model directly predicts the answer for a new query from the provided input-output examples. In strategy-based solving, the model first induces an explicit task-level strategy from the few-shot examples and then applies the induced strategy to solve the unseen query. Consequently, direct ICL remains sensitive to ex- ample construction and fragile under distribution shifts and complex reasoning scenarios (Liu et al., 2024; Mirzadeh et al., 2024). To address these limitations, recent studies draw inspiration from the human learning process of âsummarizing before applyingâ and abstract strategy-like representations, such as instructions, skills, or rules, from few-shot examples to guide subsequent problem solving (Honovich et al., 2023; Zheng et al., 2024; Chen et al., 2024). We refer to this paradigm as strategy-based ICL, where mod- els first induce explicit task strategies and then apply them to unseen questions. Unlike Direct ICL, strategy-based ICL makes the underlying task rules explicit and reduces dependence on exam- ple construction. Figure 1 illustrates how the two paradigms use example information differently. Despite the potential of intermediate representa- tions such as explicit strategies, systematic bench- marks for evaluating strategy induction in large lan- guage models remain limited. Existing evaluations mainly measure final-task performance in knowl- edge understanding, task solving, and mathemati- 1 arXiv:2608.23475v1 [cs.AI] 24 Aug 2026 cal reasoning (Hendrycks et al., 2021; Srivastava et al., 2023; Suzgun et al., 2023; Cobbe et al., 2021), offering only partial insight into intermediate rea- soning processes. Recent process-level evaluations have begun to address this gap, but they mostly assess instance-specific solution steps rather than general task-level strategies (Lightman et al., 2024; Zeng et al., 2023; Zheng et al., 2024). However, evaluating strategies is non-trivial, as it requires assessing not only whether a strategy is clear and abstract, but also whether it can be reliably applied by models and improve performance on unseen ex- amples.These requirements place higher demands on benchmark construction and metric design. To fill this gap, we propose STRATEGYBENCH, a benchmark for evaluating whether large language models can induce explicit task-level strategies from few-shot examples. STRATEGYBENCH se- lects strategy-inducible tasks from BIG-Bench and constructs high-quality reference strategies as train- ing data. Based on this dataset, we design a multi- dimensional framework that assesses generated strategies from two perspectives: strategy text qual- ity and downstream application utility. We further analyze how strategy induction and application are affected by model scale, few-shot example number, strategy format constraints, supervised fine-tuning, and strategy executors. The main contributions of this paper are summa- rized as follows: â˘Benchmark construction.We introduce STRATEGYBENCH, which is built from BIG- Bench to evaluate explicit strategy induction and supports both evaluation and training with reference strategies. ⢠Evaluation framework. We design a multi- dimensional evaluation framework, which measures both strategy quality and down- stream utility. â˘Experimental analysis. We analyze how model scale, few-shot settings, strategy for- mat constraints, supervised fine-tuning, and strategy executors affect strategy induction and application. Results show that readable strategies do not always yield better execution, and that strategy utility depends on the inter- action among the strategy generator, executor, and format constraints. 2 Preliminaries 2.1 Task and Episode Formulation This section formalizes the task and episode set- ting and introduces the direct ICL and strategy- based ICL paradigms adopted in this work. Let Tdenote the evaluation task set. For each task T â T, an example is represented as(q,a) âź T, whereqis the question andais the gold answer. Each task defines a class of problems that share the same inputâoutput format and are governed by common underlying rules. For example, in a Chinese-remainder-theorem task, all examples re- quire solving modular constraints. In each few-shot episode, the model observes a demonstration set D T few =(q i ,a i ) k i=1 ,(1) and is evaluated on an unseen example set D T new =(q new j ,a new j ) N j=1 .(2) Direct ICL. Direct ICL predicts each unseen answer directly from the demonstrations and the query: a j = h θ (D T few ,q new j ),(3) whereh θ is the task-solving model anda j is the prediction for q new j . Strategy-based ICL. Strategy-based ICL sepa- rates task solving into strategy induction and strat- egy application. A generator first induces a task- level strategy from the demonstrations: s T = f θ g (D T few ),(4) wheres T is the explicit strategy andf θ g is the strategy generator. An executor then predicts the answer using the strategy and the unseen query: a j = g θ e (q new j ,s T ),(5) where g θ e is the strategy executor. 3 Benchmark Construction We build STRATEGYBENCH from BIG-Bench, which covers diverse task types such as common- sense reasoning, mathematical computation, lan- guage understanding, instruction following, and symbolic manipulation. Because BIG-Bench tasks differ substantially in generation methods, evalua- tion protocols, and input-output formats, we first perform task filtering, dataset splitting, and task categorization, and then construct the reference strategy set. Figure 2 illustrates the overall data construction pipeline. 2 step4:reference strategy construction BIG-bench suitable tasks â test-ood test-id step 2:dataset splitting Tas k task type T Task Task strategy step 1:task filteringstep 3:task categorization t-test gain>0 test-ood categorized tasks Execution Figure 2: Overall data construction pipeline of STRATEGYBENCH. Starting from BIG-Bench, we first filter tasks with the assistance of LLM-based pre-screening and human verification to obtain a suitable task pool. The filtered tasks are then processed along two independent dimensions: dataset splitting, which produces training, ID test, and OOD test subsets, and task categorization, which assigns tasks to coarse-grained categories. Based on the resulting task types and few-shot examples, candidate strategies are generated and further filtered by downstream utility and statistical significance to form the effective strategy setS eff and the significant reference strategy setS sig . 3.1 Task Filtering & Data Normalization We follow the BBH-style procedure for task fil- tering and normalization. For filtering, we first remove tasks with dynamic evaluation, strong sub- jectivity, or overly long inputs. We then select strategy-inducible tasks through a âmodel-assisted pre-filtering + human verificationâ procedure along five dimensions: objective scorability, format stabil- ity, rule inductibility, low dependence on external knowledge, and strategy application potential. As shown in Table 1, the filtering process substantially reduces the original candidate pool while retaining a diverse set of strategy-inducible tasks. For normalization, we merge scattered task in- formation into a unified input field and convert original answers into a unified target field. De- tailed filtering rules and normalization procedures are provided in Appendix A. 3.2 Dataset Splitting After filtering, we split the data into training and test sets. The test set is further divided into in- distribution and out-of-distribution subsets to eval- uate strategy induction and generalization under Stage#Tasks#Subtasks#Examples Before filtering1085811,614,716 After filtering44356123,845 Table 1: Dataset statistics before and after task filtering. different distribution shifts. We also retain BBH as an additional challenging test set. Detailed split definitions and rules are provided in Appendix B. 3.3 Task Categorization After task filtering, we categorize the retained tasks to characterize their required abilities. We use an âunsupervised clustering + human verificationâ pro- cedure and group tasks into six categories: Numer- ical, Logic, Language, Spatial, Procedural, and Induction. Unlike the original BIG-Bench labels, our categorization reorganizes tasks by required abilities, input-output patterns, and underlying solv- ing rules. Details on semantic representation con- struction, clustering, and human verification are provided in Appendix C. 3 Strategy Set#Tasks#Subtasks#Strategies S eff 2224111,993 S sig 212217,801 Table 2: Statistics of the constructed reference strategy sets.S eff contains strategies that improve over Direct ICL, whileS sig further retains strategies with statisti- cally significant gains. 3.4 Reference Strategy Construction We construct reference strategy sets to evaluate explicit strategies and support subsequent SFT ex- periments. The construction process includes two stages: candidate strategy generation and utility- driven filtering. We first use gemini-3-flash-preview to generate candidate strategies for tasks in each category. We then compare their execution performance on un- seen examples against direct ICL and retain strate- gies that improve performance as the effective strat- egy setS eff . From this set, we further select strate- gies with statistically significant gains to form the significant reference strategy setS sig . Table 2 re- ports the statistics of both sets. Detailed generation settings, filtering rules, and additional statistics are provided in Appendix D. 4 Evaluation Metrics We build a multi-dimensional evaluation frame- work covering two perspectives: strategy applica- tion utility and strategy content quality. The for- mer measures whether a strategy can be effectively applied to improve performance, while the latter evaluates the readability of the generated strategy. 4.1 Strategy Utility For strategy application utility, we evaluate gen- erated strategies from three aspects: correctness, stability, and answer efficiency. These metrics char- acterize the practical utility of strategies on unseen examples. Correctness. Correctness measures whether a strategy helps the model answer unseen examples correctly. We use Corr as the main metric. Let a (m) j (s T ) denote the prediction in them-th sam- pling run under strategys T , withM = 3by default. We define: Corr j (s T ) = 1 3 3 X m=1 1a (m) j (s T ) = a new j .(6) Corr computes the fraction of predictions that match the gold answer across three sampling runs for the same query. A higher value indicates that the strategy more reliably guides the model to pro- duce correct answers under repeated sampling. Stability. Stability measures whether the model produces the same results when answering the same question multiple times (regardless of correctness). We evaluate it from two perspectives: sampling consistencySta samp , which measures consistency across repeated sampling runs, and robustness con- sistencySta rob , which measures consistency under perturbations of few-shot example order. For sampling consistency, leta (m) j (s T ) 3 m=1 denote the three sampled answers for thej-th un- seen example under strategys T , and letvdenote a candidate answer. We define: Sta j samp (s T ) = max v 1 3 3 X m=1 1a (m) j (s T ) = v.(7) This metric computes the fraction of the most fre- quent prediction across three sampling runs for the same query. A higher value indicates that the strat- egy more consistently guides the model to produce the same answer under repeated sampling. For robustness consistency, lets (m) T = f θ g (Ď m (D T few )) denote the strategy generated af- ter them-th permutation of the few-shot examples, and leta (m) j denote the corresponding prediction. We define: Sta j rob = max v 1 3 3 X m=1 1 a (m) j = v .(8) This metric computes the fraction of the most frequent prediction across three random example- order perturbations for the same query. A higher value indicates that the strategy more robustly guides the model to produce consistent answers under such perturbations. Efficiency.Answer efficiency measures whether a method can obtain correct answers with a lower token cost. Letc j (s T )â [0, 1]denote the correct- ness score of strategys T on thej-th query exam- ple. Letl in j (s T ) andl out j (s T ) denote the numbers of input and output tokens, respectively. Given the input length budgetL budget in and the output length budgetL budget out , we define input and output length efficiency as: LenEff j out (s T ) = c j (s T )¡ min L budget out l out j (s T ) , 1 ! ,(9) LenEff j in (s T ) = c j (s T )¡ min L budget in l in j (s T ) , 1 ! .(10) 4 If the prediction is incorrect, the efficiency score is 0. If the prediction is correct and the token cost does not exceed the budget, the method receives a high score. If the prediction is correct but requires a large token cost, the score is discounted according to the length ratio. 4.2 Strategy Content Quality Strategy content quality assesses the generated strategy text itself, independent of its effectiveness in specific tasks. Here, we evaluate two metrics: Conciseness and Format Compliance. Conciseness. Conciseness measures repetition and redundancy within a strategy. Lets T = u 1 ,...,u m denote a strategy, whereu i is thei-th sentence, and letsim(u i ,u j )denote the semantic similarity between two sentences. Form⼠2, we define Conc. as: Conc(s T ) = 1â 2 m(mâ 1) X i<j sim(u i , u j ).(11) Whenm < 2, we setConc(s T ) = 1. This formula computes the conciseness score by penalizing re- peated semantic content within a strategy. A higher value indicates less internal repetition and more concise strategy text. Format Compliance. Format compliance mea- sures whether the model output follows the prede- fined strategy format specified in the prompt. An output is assigned a score of1if it contains a com- plete strategy field that satisfies the required parsing format, and 0 otherwise: Fmat(s T ) = 1valid strategy format in s T .(12) where1[¡]is the indicator function. This metric is used to analyze whether the model can consis- tently generate strategy outputs that satisfy the for- mat constraints required for automatic parsing. A higher Fmat. indicates that the model produces more format-compliant strategy outputs. 5 Experimental Setup 5.1 Baseline We compare three few-shot ICL baselines. Di- rect ICL solves unseen questions directly based on few-shot examples (Brown et al., 2020). CoT- prompted ICL adds step-by-step reasoning in- structions during inference (Kojima et al., 2022). Least-to-Most Prompting decomposes a prob- lem into subproblems and solves them sequen- tially (Zhou et al., 2023). These methods represent three paradigms: example-based prediction, im- plicit reasoning-enhanced answer generation, and explicit intermediate reasoning. 5.2 Experimental Configuration We use Qwen3-1.7B, 4B, 8B, and 14B as backbone models. Unless otherwise specified, Qwen3-8B serves as the default strategy generator and Qwen3- 4B as the default executor. Prompting-based infer- ence usesk = 3few-shot examples. For stability evaluation, we sample each query3times and apply M = 3random perturbations to the few-shot exam- ple order. For fine-tuning, we construct SFT data from few-shot examples and reference strategies, and apply LoRA for parameter-efficient training. Detailed decoding parameters, generation and ap- plication settings, and training hyperparameters are provided in Appendix E. 6 Results and Analysis This section analyzes explicit strategy induction on STRATEGYBENCH through three research ques- tions: â˘RQ1: How do explicit strategies affect per- formance across task categories? â˘RQ2: What factors affect strategy genera- tion and application? â˘RQ3: How does SFT affect strategy gener- ation capability? 6.1 Main Result To answer RQ1, we compare our method with dif- ferent baselines across task categories. We merge the ID and OOD test examples and compute results by task category. We also evaluate all methods on BBH. Table 3 reports the results. Overall, CoT+Ours achieves the best results on most task categories and metrics, with clear gains on Numerical, Logic, and Language tasks. This suggests that explicit strategies and step-by-step reasoning prompts are complementary. Explicit strategies provide task-level rule constraints, while CoT strengthens instance-level reasoning. The strong results of CoT+Ours on Sta samp , Sta rob , LenEff in , and LenEff out further show that this com- bination improves both output stability and answer efficiency. Ours alone does not always outperform CoT or L2M, especially on Numerical and Procedural tasks. This suggests that explicit strategies serve better as high-level constraints and cannot fully replace instance-level reasoning or multi-step exe- cution. When combined with CoT, the model can 5 MetricMethod Task Category BBH NumericalLogicLanguageSpatialProceduralInduction Corr. L2M58.4342.3422.216.219.4727.5024.09 Direct ICL34.3111.184.400.690.760.5511.55 Ours15.9038.5822.113.590.9717.03 15.34 CoT57.3043.1219.434.7119.9128.9921.40 CoT+Ours69.5855.3236.098.6225.9540.9037.79 Sta samp L2M72.2564.9549.0536.9049.2349.8148.06 Direct ICL73.2739.8039.4734.6734.9535.6044.73 Ours60.1568.5352.1036.2454.8448.9051.90 CoT73.0060.9847.6336.3449.7650.5945.93 CoT+Ours82.9675.1859.4139.0756.3761.2960.01 Sta rob L2M77.8272.4160.8352.8460.6160.0359.89 Direct ICL78.1354.2754.0250.8751.7951.1857.66 Ours71.3975.5462.2851.5566.9162.4462.31 CoT78.6769.6959.6552.2361.4861.1458.54 CoT+Ours86.4080.1667.8953.9666.7070.4868.91 LenEff in L2M58.3341.7820.776.219.3727.45 23.46 Direct ICL34.2111.054.200.660.750.5211.53 Ours15.8938.2220.893.580.9717.0315.21 CoT57.2842.7718.534.7119.9128.9721.03 CoT+Ours69.4954.7832.808.6225.9540.8737.13 LenEff out L2M23.4718.509.092.513.8510.709.58 Direct ICL33.7711.164.380.630.710.5411.43 Ours15.8738.5722.103.580.9617.0215.32 CoT37.5533.8613.403.3013.4617.9314.01 CoT+Ours56.0449.0030.196.7622.6935.66 29.49 Table 3: Utility comparison across different task categories and BBH. Results are computed over all tasks within each category. Bold values indicate the best performance among the five methods for each metric within the same task category or BBH setting. use both task-level rules and step-by-step reason- ing, leading to more stable gains across most task categories. 6.2 Factors Affecting Strategy Generation and Application To answer RQ2, we further analyze factors that af- fect strategy-based ICL, including model scale, the number of few-shot examples, and strategy format constraints. 6.2.1 Effect of Generator and Executor Scale We use Qwen3-1.7B, 4B, 8B, and 14B as strategy generators, and use Qwen3-1.7B, 4B, and 8B as strategy executors. We evaluate different generator- executor combinations to analyze the effect of model scale. Table 4 reports the results. Overall, model scale affects strategy quality and utility differently. With a fixed executor, Qwen3- 14B produces the best text quality, whereas Qwen3- 8B often yields stronger execution performance. With a fixed generator, Qwen3-4B achieves the best execution results under most settings, while scaling the executor to Qwen3-8B does not consistently improve correctness. These results suggest that larger generators improve strategy readability and format quality, but effective strategy application depends on a proper generatorâexecutor match. 6.2.2 Effect of Few-shot Number To analyze the effect of the number of examples, we compare the strategy quality and utility of strategy- based ICL under1-,3-,5-,10-, and30-shot set- tings. Table 5 reports the results. This is because as the number of examples in- creases, the generated strategy becomes more de- tailed. Although such a strategy may be less con- cise and could even exceed length limits and be- come unusable, it helps the model answer questions more effectively. This implies that the model may be better at leveraging detailed rather than concise strategies. 6 Generator Strategy QualityStrategy Utility Conc.Fmat. Exec. 1.7BExec. 4BExec. 8B Corr.Sta samp Corr.Sta samp Corr.Sta samp 1.7B90.9834.8814.5048.3018.4854.9715.8654.49 4B90.0325.1213.2742.4419.4153.6614.9653.98 8B91.5722.0914.5745.0320.8656.8317.7454.30 14B92.9838.8413.2148.5120.8357.5116.7454.73 Table 4: Effect of generator and executor model scales. We compare four strategy generators and evaluate their generated strategies with three executors. Bold values indicate the best result under the same metric and executor setting. k Strategy QualityStrategy Utility Conc.Fmat.Corr.Sta samp 192.6731.3317.3446.51 391.5722.0920.8948.69 591.5421.8321.2648.06 1090.7715.4121.1948.41 3089.8314.5825.4951.29 Table 5: Effect of the number of few-shot examples on strategy generation and downstream application under k = 1, 3, 5, 10, 30. Bold values indicate the best perfor- mance across different few-shot settings. 6.2.3 Effect of Prompt Variant In addition to model scale and the number of exam- ples, prompt design also affects how models inter- pret the objective of strategy generation. We com- pare two prompt variants: a free-form prompt and a structured prompt. The free-form prompt uses flexible output constraints to elicit concise and exe- cutable strategies, whereas the structured prompt imposes explicit field and format constraints to im- prove output regularity and parsability. Their full prompt templates are provided in Appendix F, and Table 6 reports the comparison results. Prompt Type Strategy QualityStrategy Utility Conc.Fmat.Corr.Sta samp Structured Prompt91.6022.7920.8648.55 Free-form Prompt90.9157.3816.9645.02 Table 6: Comparison between structured and free-form prompts for strategy generation and downstream appli- cation. Bold values indicate the better result between the two prompt types under the same metric. The free-form prompt achieves higher Fmat., while the structured prompt performs better on Corr. and Sta samp . This suggests that the free- form prompt is more likely to produce outputs that satisfy the required format, whereas the structured prompt yields strategies with stronger downstream utility and sampling stability. Overall, prompt de- sign affects strategy generation in different ways: format compliance does not necessarily imply bet- ter downstream performance, and the utility of a strategy also depends on whether it captures task- relevant rules effectively. 6.3 Effect of SFT on Strategy Generation Capability To answer RQ3, we compare the original Ours with the supervised fine-tuned variant Ours+SFT on ID, OOD, and BBH test sets. Table 7 reports the results. Ours+SFT consistently improves Conc. and Fmat. over Ours on ID, OOD, and BBH. This indi- cates that SFT improves the conciseness and format compliance of generated strategies and generalizes across different test distributions. Ours+SFT also improves Sta samp and Sta rob on all three test sets, showing that SFT makes strategy generation more stable and robust. On the other hand, although stability has improved, it remains far from 100%, meaning the model retains a certain level of ex- ploratory capability. However, Ours+SFT does not improve Corr. and even leads to a slight decrease. This suggests that improving the form and stability of strategy gen- eration does not necessarily translate into higher single-run answer correctness. Strategy utility is also affected by task difficulty and the ability of the executor to apply the generated strategy. Therefore, although the SFT model does not become âsmarterâ, it achieves greater stability and controllability while maintaining some exploratory ability. This suggests that it could serve as a strong initialization for reinforcement learning (Ouyang et al., 2022), which remains a direction for future work. 7 Related Work Few-shot In-context Learning. Few-shot in- context learning (ICL) enables large language mod- els to adapt to new tasks using only a small number of examples without parameter updates (Brown 7 Method IDOODBBH QualityUtilityQualityUtilityQualityUtility Conc. Fmat. Corr. Sta samp Sta rob Conc. Fmat. Corr. Sta samp Sta rob Conc. Fmat. Corr. Sta samp Sta rob Ours92.136.6822.7159.6168.0891.4621.9425.4259.6768.1891.775.1515.3451.9062.32 Ours+SFT 94.1762.0621.3370.0877.7993.0361.9523.2962.8972.5893.4450.3514.2066.5176.41 Table 7: Comparison between Ours and Ours+SFT on ID, OOD, and BBH test sets. Each split reports two strategy quality metrics and three strategy utility metrics. Bold values indicate the best performance between the two variants under the same evaluation setting. et al., 2020). Prior studies show that ICL is highly sensitive to example selection, semantic relevance, and example order (Liu et al., 2021; Rubin et al., 2022; Lu et al., 2022). Increasing the number of examples does not fully eliminate the effects of context length and example organization (Agarwal et al., 2024). To improve the task-solving abil- ity of few-shot ICL, researchers have proposed several enhanced prompting paradigms, including chain-of-thought prompting (Wei et al., 2022), self- consistency (Wang et al., 2023), and least-to-most prompting (Zhou et al., 2023). Recent studies fur- ther explore explicit intermediate structures, such as Self-Discover, PAL, and PoT (Zhou et al., 2024; Gao et al., 2023; Chen et al., 2023). These methods improve performance in specific reasoning scenar- ios, but they usually serve individual queries or specific reasoning forms. In contrast, this paper focuses on whether models can induce transferable task-level strategies from few-shot examples. Benchmarks for In-context Learning.Existing benchmarks mainly evaluate final answer accuracy or task scores, such as MMLU, BIG-Bench, BBH, and GSM8K (Hendrycks et al., 2021; Srivastava et al., 2023; Suzgun et al., 2023; Cobbe et al., 2021). They provide limited evaluation of the reasoning process. Some recent studies begin to evaluate or supervise process-level reasoning, such as step verification for mathematical reasoning and reason- ing trace evaluation (Lightman et al., 2024; Lee and Hockenmaier, 2025). However, these eval- uations usually focus on the solution process of individual examples, rather than whether models can induce general strategies that transfer to similar examples. This paper takes explicit strategy induc- tion as the core evaluation target and further evalu- ates the executability of induced strategies through downstream task performance. Table 8 compares representative benchmarks and evaluations. Benchmark / Eval. Ans. Eval. Proc. Eval. Strategy Ind. Strategy Utility MMLU (Hendrycks et al., 2021)âĂ BBH (Suzgun et al., 2023)âĂ GSM8K (Cobbe et al., 2021)âĂ Step Verif. (Lightman et al., 2024)âĂ Trace Eval. (Lee and Hockenmaier, 2025)âĂ Oursâ Table 8: Comparison of representative benchmarks and process-level evaluations. Ans. Eval., Proc. Eval., Strategy Ind., and Strategy Utility denote answer-level evaluation, process-level evaluation, strategy induction, and strategy utility evaluation, respectively. Concurrent work. A concurrent anonymous study uses STRATEGYBENCH and an SFT model trained from our reference strategies to develop CoDI, a downstream method for OOD generaliza- tion (Anonymous, 2026). This work is comple- mentary to ours, which focuses on benchmark con- struction and evaluation; details are provided in Appendix G. 8 Conclusion This paper introduces STRATEGYBENCH, a bench- mark for evaluating explicit strategy induction in large language models, together with metrics cov- ering both strategy text quality and downstream application utility. Experiments show that explicit strategies can improve task-solving performance, but their utility depends on task type, example con- figuration, and the generatorâexecutor match. Su- pervised fine-tuning further improves the stability and generalization of strategy generation. Future work will extend STRATEGYBENCH to more task sources, including real-world, interac- tive, and tool-use scenarios. It is also important to distinguish strategy generation failures from execu- tion failures more precisely and to explore training methods that enhance the transferability and exe- cutability of generated strategies. 8 9 Limitations This paper has several limitations in methodology and experiments. Methodology. (1) STRATEGYBENCH is mainly built from BIG-Bench and BBH. Although the fil- tered tasks cover multiple reasoning categories, the data sources are still relatively limited. Future work can incorporate more independent benchmarks to evaluate the generalization of explicit strategy in- duction across broader task distributions. (2) This paper evaluates generated strategies from two per- spectives: strategy text quality and downstream application utility. However, it does not fully dis- tinguish different sources of failure. For example, an incorrect answer may result from an inaccurate induced strategy, or from the executorâs failure to understand and apply the strategy correctly. Future work should conduct a more fine-grained analysis of strategy generation errors and strategy execution errors. Experiments. (1) Due to computational re- source constraints, this paper mainly evaluates models from the Qwen3 family. This setting en- ables controlled scale comparisons within the same model family, but it does not cover more model families or closed-source models. (2) The fine- tuning experiments in this paper mainly focus on the effect of SFT on strategy generation capabil- ity. We have not further explored reinforcement learning, preference optimization, or multi-stage curriculum learning. These methods may further improve the executability and downstream utility of generated strategies. References Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Lorena Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, and Hugo Larochelle. 2024. Many-shot in- context learning. arXiv preprint arXiv:2404.11018. Anonymous. 2026.Cognitively decoupled meta- induction for out-of-distribution generalization. Un- der review. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems. Ping-Chen Chen, Szu-Lin Wei, Hen-Hsen Huang, and Hsin-Hsi Chen. 2024. Induct-learn: Short phrase prompting with instruction induction. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5204â5231. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023.Program of thoughts prompting: Disentangling computation from reason- ing for numerical reasoning tasks. Transactions on Machine Learning Research. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word prob- lems. arXiv preprint arXiv:2110.14168. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2023. Pal: Program-aided language models. In Proceedings of the 40th International Conference on Machine Learning. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language under- standing. In International Conference on Learning Representations. Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. 2023. Instruction induction: From few examples to natural language task descriptions. In Proceedings of the 61st Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1935â 1952. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199â22213. Jinu Lee and Julia Hockenmaier. 2025. Evaluating step- by-step reasoning traces: A survey. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 1789â1814. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri- son Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Letâs verify step by step. In International Conference on Learning Representations. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape, Michele Bevilacqua, Fabio Petroni, and Percy 9 Liang. 2024. Lost in the middle: How language mod- els use long contexts. Transactions of the Association for Computational Linguistics, 12:157â173. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048â11064. Association for Computational Linguistics. Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2024. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Proceedings of the 36th Interna- tional Conference on Neural Information Processing Systems, NIPS â22, Red Hook, NY, USA. Curran Associates Inc. Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Alex R. Brown, Adam Santoro, Aditya Gupta, Adri ` a Garriga-Alonso, Agnieszka Kluska, and 1 others. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research. Mirac Suzgun, Nathan Scales, Nathanael Sch Ě arli, Se- bastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. 2023. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Com- putational Linguistics: ACL. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowd- hery, and Denny Zhou. 2023. Self-consistency im- proves chain of thought reasoning in language mod- els. In International Conference on Learning Repre- sentations. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems. Zhiyuan Zeng, Peng Chen, Sheng Liu, Haoming Jiang, and Jia Jia. 2023. Mr-gsm8k: A meta-reasoning benchmark for large language model evaluation. arXiv preprint arXiv:2312.17080. Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2024. Take a step back: Evoking reasoning via abstraction in large language models. In Interna- tional Conference on Learning Representations. Denny Zhou, Nathanael Sch Ě arli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V. Le, and Ed H. Chi. 2023. Least-to-most prompting enables complex reasoning in large language models. In International Conference on Learning Representations. Pei Zhou, Jay Pujara, Xiang Ren, Xinyun Chen, Heng- Tze Cheng, Quoc V. Le, Ed H. Chi, Denny Zhou, Swaroop Mishra, and Huaixiu Steven Zheng. 2024. Self-discover: Large language models self-compose reasoning structures. In Advances in Neural Informa- tion Processing Systems. A Task Selection and Standardization Details This appendix provides additional details on the task selection and format standardization process of STRATEGYBENCH. Overall, we first follow the BBH-style procedure to preprocess and filter the original BIG-Bench tasks, removing tasks that are not suitable for unified few-shot strategy induction evaluation. We then adopt a âmodel-assisted pre- filtering + human verificationâ procedure to further select tasks suitable for strategy induction. Finally, we standardize the input and target formats of the retained examples. A.1 Task Filtering We first perform an initial filtering over the original BIG-Bench tasks and remove three types of tasks that are clearly unsuitable for static few-shot strat- egy induction evaluation. The first type includes tasks that rely on dynamic generation or external program execution for evaluation, as these tasks do not provide fixed input-output examples. The second type includes highly subjective or sensitive tasks, such as bias, toxicity, and ethical judgment tasks, which usually lack a single stable and objec- tive gold answer. The third type includes examples whose input length exceeds 2500 characters. We 10 remove these examples to reduce the interference of long contexts with few-shot example construc- tion, strategy generation, and strategy application, and to control inference cost. A.2 Model-assisted Task Selection After the initial filtering, we further determine whether each candidate subtask is suitable for ex- plicit strategy induction evaluation. Specifically, we use Qwen3-32B to automatically score each candidate subtask, and then conduct human veri- fication to finalize the selection. The scoring di- mensions include objective scorability, format sta- bility, rule inductibility, external knowledge risk, and strategy transfer potential. Each dimension is scored from 0 to 2. For all dimensions except exter- nal knowledge risk, a higher score indicates that the task is more suitable for strategy induction. Exter- nal knowledge risk is a negative indicator, where a higher score means that the task relies more heavily on external knowledge. Objective scorability.This dimension measures whether a task has clear and stable objective eval- uation criteria. If a task can be evaluated by exact match or clear multiple-choice answers, it receives a score of 2. If its answers are partially objective but contain some ambiguity, it receives a score of 1. If the task is open-ended or subjective and has unclear answer boundaries, it receives a score of 0. Format stability. This dimension measures whether the input-output format is stable within the same task or subtask. If the task format is fixed and the input-output structure is clear, it receives a score of 2. If the overall format is relatively stable but contains some noise or variants, it receives a score of 1. If the task format varies substantially and makes it difficult to extract stable patterns, it receives a score of 0. Rule inductibility. This dimension measures whether a task contains rules, steps, or algorithms that can be abstracted from a small number of ex- amples. If a task has clear inductive rules and can be written as stable operation steps, it receives a score of 2. If it contains certain patterns but the rules are incomplete or weakly generalizable, it receives a score of 1. If the task mainly relies on commonsense memory, linguistic intuition, or cul- tural background and is difficult to abstract into a general strategy, it receives a score of 0. External knowledge risk. This dimension mea- sures whether solving a task depends on the modelâs prior world knowledge, cultural knowl- edge, or domain-specific knowledge, rather than information that can be induced from few-shot ex- amples. This dimension is a negative indicator. If a task heavily depends on external knowledge, it receives a score of 2. If it requires some external knowledge but external knowledge is not the domi- nant factor, it receives a score of 1. If the task can mostly be solved through examples and induced rules, it receives a score of 0. Strategy utility potential. This dimension mea- sures whether an explicit strategy can potentially bring practical gains compared with the no-strategy setting. For example, BIG-Bench tasks such as moral permissibility and simple ethical questions usually depend on value judgments, contextual un- derstanding, or safety preferences, and are diffi- cult to formulate as stable objective rules. Ex- plicit strategies may oversimplify complex contexts into partial judgment rules and reduce the modelâs adaptability to specific examples. Therefore, these tasks have low strategy utility potential. If errors in a task can be substantially reduced by a clear strat- egy, the task receives a score of 2. If a strategy may provide some help but the improvement is limited or unstable across examples, it receives a score of 1. If a strategy is unlikely to improve performance, or the task mainly relies on memorization, common- sense knowledge, or surface matching, it receives a score of 0. A.3 Selection Rule Based on the five dimensions above, we define the suitability score of a candidate subtask as: Score = s obj +s format +s rule +s transfer âs know , (13) wheres obj ,s format ,s rule , ands transfer denote the scores of objective scorability, format stability, rule inductibility, and strategy transfer potential, respec- tively.s know denotes the score of external knowl- edge risk. Using only the total score may retain some tasks that are not suitable for strategy induction evalua- tion. For example, some tasks are format-stable and easy to score, but heavily rely on external knowl- edge or cultural background. Their performance changes may mainly come from the modelâs mem- orized knowledge rather than strategies induced from few-shot examples. To avoid this issue, we 11 use a filtering rule based on both the total score and necessary constraints. A candidate subtask must satisfy: Score⼠7,(14) and: s know ⤠1, s obj = 2, s transfer = 2.(15) Here,s know ⤠1controls the risk of external knowledge dependence,s obj = 2ensures stable objective evaluation criteria, ands transfer = 2en- sures clear strategy transfer potential. After model- assisted scoring, we further conduct human verifi- cation and remove tasks that clearly deviate from our research goal, such as multimodal tasks, highly sensitive tasks, and tasks that are clearly inconsis- tent with rule induction and strategy transfer. A.4 Input and Target Standardization After task filtering, we standardize the retained ex- amples so that different BIG-Bench tasks can be uniformly used for few-shot strategy induction and automatic evaluation. The standardization process mainly includes input field standardization and tar- get answer standardization. Input standardization. The input information in the original BIG-Bench examples may be scat- tered across multiple fields, including task prefix, example input prefix, input prefix, and example output prefix. We integrate these fields into a uni- fied input field. Specifically, task prefix usually represents task instructions, input represents the concrete question text, choice prefix provides the option prefix for multiple-choice tasks, and exam- ple output prefix specifies the expected output for- mat. For multiple-choice examples, if the original op- tions do not have explicit labels, we assign uni- fied option IDs such as A/B/C/D. This ensures that model outputs can be aligned with automatic eval- uation. This processing enables different tasks to be organized into a unified few-shot prompt format and supports subsequent exact-match evaluation. Target standardization. We convert the origi- nal answers into a unified target field. For binary classification tasks, such as yes/no questions, we di- rectly retain the answer word. For general multiple- choice tasks, we convert the answer into the cor- responding option ID, such as A, B, C, or D. For tasks with too many options or unstable option la- bels, we directly retain the original answer string. FieldExample Value task disambiguationqa subtask disambiguationqa inputIn the following sentences, explain the an- tecedent of the pronoun. Options: (A) . . . (B) . . . (C) Ambiguous. target ["A"] Table 9: Example of a standardized instance. For exact-match tasks, we retain the gold answer in string form. Standardized instance format. After standard- ization, each example is converted into a unified format. A simplified example is shown below: Here, task denotes the original BIG-Bench task name, subtask denotes the corresponding subtask, input denotes the normalized model input, and tar- get denotes the gold answer list used for automatic evaluation. B Dataset Splitting and Sampling Details This section provides additional details on the dataset splitting and sampling rules of STRATE- GYBENCH. Since the original BIG-Bench tasks vary substantially in sample size, subtask scale, input-output format, and evaluation difficulty, di- rectly using all examples may cause large-scale tasks to dominate training or evaluation and reduce comparability across tasks. Therefore, we follow two main principles when constructing the bench- mark. First, we avoid obvious information leakage between the training and test sets. Second, we aim to keep the number of examples relatively balanced across tasks and subtasks. Training and test split.We first split the prepro- cessed BIG-Bench data into training and test sets. The training set is mainly used to construct few- shot examples, generate reference strategy data, and support model development and debugging. The test set is used to evaluate the strategy induc- tion and strategy application abilities of models under different distribution conditions. To system- atically analyze model generalization, we further divide the test set into in-distribution and out-of- distribution subsets. In-distribution test set. The in-distribution test set is denoted as ID. Test examples in this set come from the same subtask set as the training examples, but the concrete examples do not overlap. This 12 setting evaluates whether models can induce stable and effective strategies from few-shot examples and apply them to new test examples when the task rules and input-output formats are relatively consistent. Out-of-distributiontestset. Theout-of- distribution test set is used to evaluate strategy induction and transfer ability beyond the training distribution. We assign examples to the OOD test set when they differ clearly from the training set in task source, input-output format, or required ability. We also avoid direct overlap with task rules and example templates used during training. This setting examines whether models can induce transferable explicit strategies from few-shot examples when facing unseen task structures or distribution shifts. BBH test set. In addition to the internal BIG- Bench split, we retain BIG-Bench Hard (BBH) as an additional challenging test set. BBH con- tains a set of tasks that are more challenging for large language models and can be used to exam- ine strategy induction ability in complex reasoning, symbolic manipulation, and multi-step problem- solving scenarios. We select tasks from BBH that are suitable for automatic evaluation and strategy induction, and ensure that they do not directly over- lap with tasks in the training set. The final retained BBH set serves as an additional challenging out-of- distribution evaluation set. Balanced sampling strategy. Since different tasks and subtasks vary greatly in sample size, we use ak-cut based balanced sampling method when constructing the training and evaluation sets. Given a set of data collectionsD 1 ,D 2 ,...,D m , where |D i |denotes the number of examples in thei-th collection, the sampling goal is to retain at mostk examples from each collection. If a collection con- tains fewer thankexamples, all examples in that collection are retained. To ensure that the overall sample size is not smaller than a predefined target sizeN, we solve for the smallest integerkthat satisfies: f(k) = m X i=1 min(k,|D i |)⼠N.(16) Here,f(k)denotes the total number of examples obtained when at mostkexamples are sampled from each collection. Sincef(k)is monotonically increasing with respect tok, the smallest feasible kcan be efficiently found by binary search. After obtainingk, we apply the following sampling rule to each collection: if|D i | ⤠k, all examples are retained; if|D i | > k, we randomly samplekex- amples from the collection. This method controls the overall sample size while preventing a small number of large-scale tasks from contributing too many examples, making the influence of different tasks more balanced during training and evaluation. Boundary cases. We handle boundary cases as follows. If the total number of examples across all candidate collections is still smaller than the target size, i.e., m X i=1 |D i |⤠N,(17) we do not perform truncation and directly retain all examples. If the number of candidate collections is already no smaller than the target size, i.e.,m⼠N , we setk = 1to ensure that each collection contributes at least one example and preserve task coverage as much as possible. In other cases, we solve for the smallestkthat satisfiesf(k) ⼠N and then perform sampling as described above. C Task Categorization To support category-aware strategy generation and task-type-specific result analysis, we further divide the retained tasks into six coarse-grained semantic categories. This categorization does not directly follow the original BIG-Bench task labels. Instead, it considers the core abilities required by each task, the input-output format, and the underlying solving process. Compared with the original task names, our categorization focuses more on the types of strategies that models need to induce and execute when solving the tasks. Therefore, it is more suit- able for analyzing how explicit strategies affect different task types. Category definitions. We define the six coarse- grained task categories as follows: ⢠Numerical. This category mainly involves numerical computation, arithmetic operations, counting, modular arithmetic, quantity com- parison, or other rules related to numerical manipulation. Models usually need to per- form deterministic mathematical or symbolic computations based on the numerical informa- tion in the question. 13 â˘Logic. This category mainly evaluates logical reasoning, rule constraints, symbolic relations, and consistency judgments. Models need to identify the conditions or constraints in the question and follow specific reasoning steps to obtain the answer. ⢠Language. This category focuses on natural language understanding, including lexical re- lations, syntactic patterns, textual entailment, paraphrase recognition, and judgments based on linguistic cues. The key ability is to ex- tract semantic relations from text and make predictions accordingly. ⢠Spatial. This category involves spatial posi- tions, directional relations, object movement, state changes, or transformations in structured spaces. Models usually need to maintain in- termediate spatial states and reason based on changes in positions or directions. â˘Procedural. This category requires models to follow given steps, operation sequences, or procedural rules. The main challenge is to correctly track intermediate states during multi-step operations and avoid state update errors in long-step reasoning. â˘Induction. This category requires models to induce hidden rules, mapping relations, or ab- stract patterns from examples and transfer the induced rules to new query examples. Since these tasks explicitly rely on the process of summarizing rules from examples and then applying them, they are highly related to the strategy induction ability studied in this paper. Categorization procedure. During categoriza- tion, we first construct a textual representation for each task. This representation consists of stan- dardized input examples, output formats, and task- related metadata. We then use a semantic embed- ding model to encode the task representations and perform unsupervised clustering based on the em- bedding vectors to obtain initial task groups. Since automatic clustering may be affected by surface textual forms and place tasks with different solv- ing mechanisms into the same group, we further conduct human semantic verification. Based on the main ability requirements and solving structures of the tasks, we merge the initial clusters into the six coarse-grained categories defined above. This process ensures that tasks within the same category have similar strategy requirements, while differ- ent categories reflect clear differences in reasoning abilities. D Reference Strategy Construction Details This section provides additional details on the con- struction of the reference strategy sets. We do not manually annotate solution processes for each test example. Instead, we automatically generate can- didate strategies that summarize task-solving rules from few-shot examples, and then filter effective reference strategies based on downstream execu- tion performance. The overall process consists of three steps: category-aware prompt design, candi- date strategy generation, and utility-driven strategy filtering. Category-aware prompt design.Different task types rely on different solving rules. For example, numerical tasks usually require step-by-step com- putation and boundary condition checking. Logic tasks rely more on constraint identification and stepwise reasoning. Procedural tasks require main- taining intermediate states, while induction tasks require extracting reusable hidden rules from ex- amples. Therefore, we do not use a single generic prompt to generate all candidate strategies. Instead, we design category-aware prompts for strategy gen- eration according to task categories. Specifically, we design strategy generation prompts for the six coarse-grained task categories. Each category has two generation templates. One template emphasizes step-by-step reasoning and executable operation procedures, while the other emphasizes concise summaries of core rules. The former is suitable for tasks that require explicit state tracking, constraint checking, or multi-step reasoning. The latter is suitable for tasks with short rules, stable patterns, or relatively direct solving processes. In total, we obtain 12 category-aware strategy generation prompts to cover the strategy expression needs of different task types. Candidate strategy generation. Given the few- shot example setD demo for a task or subtask, we use a strategy generatorf θ g to generate candidate strategies: s = f θ g (D demo ,p c ),(18) wherep c denotes the category-aware prompt cor- responding to the current task categoryc, ands 14 denotes the generated candidate strategy. A can- didate strategy may take the form of natural lan- guage rules, operation steps, input-output mapping relations, constraint checking methods, or error avoidance instructions. Its goal is not to restate the few-shot examples, but to extract general solving rules that can transfer to new query examples. For each few-shot setting used to construct strate- gies, we select the corresponding prompt according to the task category and feed the few-shot exam- ples into the strategy generator to obtain candidate strategies. The generated candidate strategies are not directly used as final reference strategies. In- stead, they are passed to the downstream utility filtering stage. Utility-based strategy filtering. Automatically generated candidate strategies may contain redun- dancy, errors, or information that is not helpful for downstream answering. Therefore, we further fil- ter candidate strategies based on their execution performance on unseen query examples. Specifi- cally, we use Direct ICL as the baseline and add the candidate strategy into the context to form strategy- based ICL. We then compare the accuracy of the two methods on the corresponding unseen query setD query . LetAcc direct denote the accuracy of Direct ICL, and letAcc strategy (s)denote the accuracy after us- ing the candidate strategys. If the candidate strat- egy satisfies: Acc strategy (s) > Acc direct ,(19) we consider the strategy to bring downstream per- formance gains and add it to the effective strategy setS eff . Based on the effective strategy set, we further select strategies with more stable gains and obtain the significant reference strategy setS sig : S sig âS eff .(20) Here,S eff retains all candidate strategies that im- prove performance over Direct ICL, whileS sig fur- ther emphasizes the stability and reliability of strat- egy gains. We useS sig as the main high-quality reference strategy set and useS eff for supplemen- tary statistics and analysis. Resulting strategy sets. Finally, we construct two levels of reference strategy data: the effective strategy setS eff and the significant reference strat- egy setS sig . The former characterizes the range of automatically generated strategies that bring practi- cal gains, while the latter provides a higher-quality and more stable source of reference strategies. The overall statistics of the two strategy sets are shown in Table 2 in the main paper. E Implementation Details This section provides additional details on strategy generation, strategy application, and supervised fine-tuning settings. The main paper only reports the major experimental settings, while the detailed implementation parameters are summarized in this section. Strategy generation settings. We use a local OpenAI-compatible inference interface for strategy generation. Unless otherwise specified, strategy generation uses prompt v1 as the prompt template, the default number of few-shot examples isk = 3, and each bucket generates one candidate strategy by default. The generation temperature is set to 0.2, and the maximum generation length is set to 700tokens. To improve request stability, we allow at most6retries and set a0.2-second interval be- tween adjacent requests. For stability and example perturbation experiments, we construct5strategy variants for each bucket, including3generations with the original example order and2generations with randomly permuted example orders. The ran- dom seed is set to 42. Strategy application settings. During strategy application, the executor model receives the in- duced strategy and a new query question, and is instructed to output only the final answer. Unless otherwise specified, the decoding temperature in the strategy application stage is set to0.7, the max- imum output length is set to256tokens, and at most3retries are allowed. The inference process supports concurrent requests, with the default con- currency set to8. For test example loading, the script prioritizes the test20 json file under the cor- responding task directory. If full testing is enabled, all test examples in the file are evaluated. For an- swer judgment, we first extract the final answer from the model output, apply lightweight normal- ization, and then compare it with the gold answer using exact match. Supervised fine-tuning settings.For SFT exper- iments, we construct supervised fine-tuning data from few-shot examples and their reference strate- gies, and use LoRA for parameter-efficient fine- 15 HyperparameterValue Base modelQwen3-8B Max length4096 Max steps2200 Batch size1 Gradient accumulation8 Learning rate1Ă 10 â4 Warmup ratio0.03 Eval / Save steps100 / 100 Logging steps10 LoRA rank16 LoRA alpha32 Target modulesAttention layers Precisionbf16 Table 10: Main hyperparameter settings for supervised fine-tuning. tuning. The training model is Qwen3-8B. The maximum sequence length is set to4096, and the maximum number of training steps is set to2200. The per-device batch size is1, and the gradient accumulation step is8. The learning rate is set to1Ă 10 â4 , and the warmup ratio is set to0.03. The model is evaluated and saved every100steps, and training logs are recorded every10steps. The LoRA rank is set to16, LoRA alpha is set to32, the target modules are attention layers, and training uses bf16 precision. SFT data format. Each SFT instance follows the previous JSONL message style. The input con- tains the strategy-generation instruction and the few-shot QA block, while the output is the corre- sponding reference strategy used as the supervised target. Figure 3 shows an example of the SFT data structure. Use of AI Writing Assistance. We used GPT- based writing assistance solely to polish the lan- guage of author-written text, including grammar correction, wording refinement, and clarity im- provement. The tool was not used to generate research ideas, experimental results, analyses, ci- tations, or conclusions. All content was reviewed, revised, and verified by the authors, who take full responsibility for the final manuscript. F Prompt Templates This section provides the prompt templates used for strategy generation. We consider two types of prompts: a structured prompt and a free-form prompt. The structured prompt specifies explicit output fields and format constraints, aiming to make the generated strategies more regular and eas- ier to parse. The free-form prompt imposes fewer structural constraints and instead encourages the model to summarize concise and executable task- level strategies from the demonstrations. In this paper, Prompt V1 refers to the structured prompt, and Prompt V2 refers to the free-form prompt. Un- less otherwise specified, Prompt V1 is used as the default strategy generation prompt in the main ex- periments. G Concurrent Work A concurrent anonymous study proposes CoDI for cognitively decoupled meta-induction and OOD generalization (Anonymous, 2026). It uses STRAT- EGYBENCH as a training and evaluation testbed and further leverages an SFT model trained from our reference strategies. The concurrent study de- velops a downstream method based on these re- sources, whereas this paper focuses on constructing the benchmark, reference strategy resources, and evaluation framework for explicit strategy induc- tion. Figure 6 illustrates the overall framework of CoDI. The method separates strategy induction from answer execution: a strategy generator first induces task-level representations from support ex- amples, and a frozen answer solver then applies the induced representation to answer new queries. This design is complementary to our benchmark- oriented contribution. 16 SFTDataStructureExample(JSONL) messages + meta "messages": [ "role":"system", "content":"You are a Strategy Induction Assistant. : a ...", , "role":"user", "content":"Inputs: Samples: <sample> <question> are 3 ho...", , "role":"assistant", "content":"<strategy> 1. Extract all fixed positions and relative co..." ], "meta": "task":"logic_grid_puzzle", "subtask":"logic_grid_puzzle", "qid":"007", "strategy_path":"positive_strategies/train_all/v1/gain_pos_sig/coarse_4...", "qa_json_path":"data/data_train_all/kmeans_6/bucketed_outputs_k6_split..." Figure: JSONL-formatted SFT training sample with system, user, and assistant messages. Figure 3: Example of the JSONL-style data structure used for strategy supervised fine-tuning. Each instance contains the strategy-generation instruction, the few-shot QA block, and the reference strategy as the supervised target. Prompt for Strategy Generation (V1) You are a Strategy Induction Assistant. Task: Given QA samples from the same task family, infer a concise, externally executable strategy that a small model can follow. Hard constraints (VERY IMPORTANT): - Derive the strategy ONLY from patterns explicitly supported by the QA samples. - Do NOT assume a specific task type beyond what samples demonstrate. - Do NOT introduce new entities unless clearly present in samples. - Do NOT rely on answer choices to guide intermediate reasoning. - Prefer deterministic checks and rule application over vague advice. - Focus on operations that remain valid under surface variation. Output requirements: - Output ONLY a single <strategy>...</strategy> block. - 3 to 5 numbered steps. - Each step MUST start with an action verb: Extract / Identify / Map / Transform / Apply / Compute / Check / Verify / Select / Output - Each step MUST be <= 12 words. - Steps MUST be concrete operations, not explanations. Inputs: QA Samples: QA_BLOCK Figure: Prompt template for strategy generation (V1). Figure 4: Prompt V1 used for strategy generation. 17 Prompt for Strategy Generation (V2) You are a Strategy Induction Assistant. Task: Given a set of QA samples from the same task, infer a general strategy that can help solve similar questions. The inferred task type is for internal use only and must NOT be output. The strategy should increase the likelihood of solving unseen questions from the same task. Core Requirements: - Do NOT output hidden chain-of-thought reasoning. - Do NOT describe the task itself. - Do NOT reference the specific QA samples. - Focus only on reusable reasoning actions. Expression Guidelines: - Write the strategy as 2â4 concise sentences. - Do NOT use numbered steps. - Do NOT use bullet points or lists. - Each sentence should describe a useful reasoning action or decision rule. - Prefer practical actions over generic advice. Output Structure: The output MUST contain ONLY the <strategy> block. <strategy> Describe the general strategy using a few concise sentences. </strategy> Inputs: QA Samples: QA_BLOCK Figure: Prompt template for strategy generation (V2). Figure 5: Prompt V2 used for strategy generation. Figure 6: Overview of the concurrent CoDI framework. CoDI uses STRATEGYBENCH as a training and evaluation testbed and leverages an SFT model trained from reference strategies. The framework separates task-level strategy induction from answer execution, serving as a downstream method built on the benchmark resources introduced in this paper. 18