Paper deep dive

Magic Words or Methodical Work? Challenging Conventional Wisdom in LLM-Based Political Text Annotation

Lorcan McLaren, James Cross, Zuzanna Krakowska, Robin Rauner, Martijn Schoonvelde

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 119

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 4/1/2026, 1:14:46 AM

Summary

This paper presents a controlled evaluation of LLM-based political text annotation, testing six open-weight models across four tasks. The authors find that interaction effects between pipeline choices (model choice, size, learning approach, and prompt style) dominate main effects, rendering 'best practices' unreliable. They introduce a validation-first framework to help researchers navigate these choices transparently.

Entities (5)

LLM · technology · 100%Lorcan McLaren · researcher · 100%Nvidia A100 · hardware · 100%CodeBook Studio · software · 95%Gemma 3 · model · 95%

Relation Signals (3)

Lorcan McLaren → authored → Magic Words or Methodical Work? Challenging Conventional Wisdom in LLM-Based Political Text Annotation

confidence 100% · Paper title and author list

Gemma 3 → evaluatedin → Magic Words or Methodical Work? Challenging Conventional Wisdom in LLM-Based Political Text Annotation

confidence 95% · The paper tests six open-weight models including Gemma 3

LLM → usedfor → Political Text Annotation

confidence 95% · Political scientists are rapidly adopting large language models (LLMs) for text annotation

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Political scientists are rapidly adopting large language models (LLMs) for text annotation, yet the sensitivity of annotation results to implementation choices remains poorly understood. Most evaluations test a single model or configuration; how model choice, model size, learning approach, and prompt style interact, and whether popular "best practices" survive controlled comparison, are largely unexplored. We present a controlled evaluation of these pipeline choices, testing six open-weight models across four political science annotation tasks under identical quantisation, hardware, and prompt-template conditions. Our central finding is methodological: interaction effects dominate main effects, so seemingly reasonable pipeline choices can become consequential researcher degrees of freedom. No single model, prompt style, or learning approach is uniformly superior, and the best-performing model varies across tasks. Two corollaries follow. First, model size is an unreliable guide both to cost and to performance: cross-family efficiency differences are so large that some larger models are less resource-intensive than much smaller alternatives, while within model families mid-range variants often match or exceed larger counterparts. Second, widely recommended prompt engineering techniques yield inconsistent and sometimes negative effects on annotation performance. We use these benchmark results to develop a validation-first framework - with a principled ordering of pipeline decisions, guidance on prompt freezing and held-out evaluation, reporting standards, and open-source tools - to help researchers navigate this decision space transparently.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

118,293 characters extracted from source content.

Expand or collapse full text

Magic Words or Methodical Work? Challenging Conventional Wisdom in LLM-Based Political Text Annotation Lorcan McLaren 1 , James P. Cross 1 , Zuzanna Krakowska 1 , Robin Rauner 1 , and Martijn Schoonvelde 2 1 University College Dublin 2 University of Groningen April 1, 2026 arXiv:2603.26898v2 [cs.CL] 31 Mar 2026 Abstract Political scientists are rapidly adopting large language models (LLMs) for text annotation, yet the sensitivity of annotation results to implementation choices re- mains poorly understood. Most evaluations test a single model or configuration; how model choice, model size, learning approach, and prompt style interact, and whether popular “best practices” survive controlled comparison, are largely unex- plored. We present a controlled evaluation of these pipeline choices, testing six open-weight models across four political science annotation tasks under identical quantisation, hardware, and prompt-template conditions. Our central finding is methodological: interaction effects dominate main effects, so seemingly reasonable pipeline choices can become consequential researcher degrees of freedom. No sin- gle model, prompt style, or learning approach is uniformly superior, and the best- performing model varies across tasks. Two corollaries follow. First, model size is an unreliable guide both to cost and to performance: cross-family efficiency differences are so large that some larger models are less resource-intensive than much smaller alternatives, while within model families mid-range variants often match or exceed larger counterparts. Second, widely recommended prompt engineering techniques yield inconsistent and sometimes negative effects on annotation performance. We use these benchmark results to develop a validation-first framework—with a prin- cipled ordering of pipeline decisions, guidance on prompt freezing and held-out evaluation, reporting standards, and open-source tools—to help researchers navi- gate this decision space transparently. 1 Introduction Large language models (LLMs) have been adopted as text annotation tools in political science faster than the evaluative tradition needed to distinguish principled guidance from annotation folklore. When researchers embraced dictionary methods and topic models, a substantial body of work evaluated how configuration choices including stopword lists, the number of topics, and seed words to name but a few, affected substantive conclusions (Grimmer and Stewart 2013; Grimmer, Roberts, and Stewart 2022). That evaluative tradition produced widely cited principles: that all models are wrong but some are use- ful, that no single method dominates, and that validation against human judgement is essential (Grimmer and Stewart 2013). No equivalent guidance yet exists for LLM-based annotation, even as these models are deployed to measure widely varying constructs, in- cluding policy positions (Mens and Gallego 2025), protest events (Halterman and Keith 2025), and political rhetoric (T ̈ornberg 2024b). The absence of systematic evaluation criteria matters because a researcher’s decision space is vast. Deploying an LLM as an annotator requires choosing a model family, a model size, a prompting strategy, and identifying whether or not to provide labelled examples. Each of these decisions interact with one another and with the properties of the annotation task itself. If these choices substantially affect results, then LLM-based annotation introduces a new form of researcher degrees of freedom: undisclosed analytical flexibility that can shift findings toward preferred conclusions, whether intentionally or not (Baumann et al. 2025). Recent work quantifying “LLM hacking” demonstrates that minor prompt paraphrases alone can render virtually any hypothesis statistically significant (Baumann et al. 2025). Yet most applied studies select pipeline configurations on the basis of convention, informal “best practice” guides (T ̈ornberg 2024a), or resource availability, without reporting how alternative choices would have affected their measurements. This paper contributes to a growing body of work evaluating LLM annotation pipeline choices (Alizadeh et al. 2024; Atreja et al. 2025; Halterman and Keith 2025; Baumann et al. 2025) by using a large controlled comparison to address a methodological problem in 1 political text analysis: how researchers should validate LLM annotation pipelines when multiple defensible implementation choices are available. We systematically vary four choices that an analyst must make when employing an LLM in an annotation workflow: model choice (six open-weight models), model size (three model families with 270M–72B parameter variants), zero-shot versus few-shot learning, and prompt style (standard, per- sona, chain-of-thought). We explore the effect of these choices on annotation performance across four annotation tasks spanning different text types, annotation formats, and levels of conceptual complexity. Two clarifications about scope are important. We do not argue that generative LLMs are the superior approach to these tasks: fine-tuned encoder-only models such as BERT still outperform generative models on many text annotation tasks in absolute terms, including some of ours. Nor do we seek to identify optimal settings for each model–task combination. Rather, we use benchmark variation across configu- rations to show the relative difference that implementation choices make, the at times unpredictable direction of these effects, and the implications this has for methodological practice. Our focus on open-weight models is deliberate and inspired by both open-science and en- vironmental concerns. Closed-source systems, while sometimes more performant, cannot guarantee reproducibility across time, preclude measurement of energy consumption, and may be modified or withdrawn without notice. This makes reproducibility hostage to the whims of private companies. By running all models locally, we can precisely quantify the computational cost of each design choice alongside its effect on annotation performance. Efficiency constraints, and energy use in particular, deserve legitimate attention: the in- ference costs of generative LLM pipelines can be orders of magnitude larger than those of earlier transformer models or other quantitative text analysis methods, and pipeline choices that appear inconsequential for performance can produce order-of-magnitude dif- ferences in energy consumption. All experiments use identical quantisation (Q4 KM), identical hardware (NVIDIA A100 GPUs), and standardised prompt templates, isolating the marginal effect of each design choice. The full experimental pipeline, from code- book definition to LLM benchmarking, is released as open-source software to facilitate 2 replication and extension (see Data and Code Availability). Our experiments are designed to adjudicate a set of concrete heuristics that already cir- culate in the field: that larger models produce better annotations, that larger models are necessarily more resource-intensive, that providing labelled examples (few-shot) is uniformly beneficial, and that common forms of prompt engineering reliably improve performance. Three headline findings emerge. First, interaction effects dominate main effects : no model, prompt technique, or learning approach is uniformly superior, and task- level variation routinely exceeds the average treatment effect of any single pipeline choice. Second, model size is an unreliable guide both to performance and to cost : cross-family ef- ficiency differences are so large that Gemma 3 at 27B parameters is less resource-intensive than the smallest tested variants of other families, while within model families annota- tion performance scales non-monotonically and mid-range variants sometimes outperform the largest available model. Third, popular prompt engineering techniques are unreliable: both persona and chain-of-thought prompting produce erratic performance effects, some- times improving and sometimes degrading performance, while potentially substantially increasing computational cost. Taken together, these results challenge not just the as- sumption that “bigger is better,” but also the assumption that bigger is necessarily more expensive to run. These findings demonstrate that LLM annotation pipeline choices are a consequential form of researcher degrees of freedom, comparable in importance to those documented for earlier text-as-data methods. We therefore make two linked contributions. First, we provide the empirical evidence base that has been lacking: a controlled compari- son showing how and why pipeline choices matter, and which existing “best practice” recommendations survive scrutiny. Second, we translate those results into a practical validation-first framework (Section 5) that specifies how researchers should move from codebook design to model evaluation: freeze the annotation instructions, benchmark al- ternative pipelines against human-coded samples, and treat prompt revision as a tuning decision that requires transparent reporting and, where iterative development occurs, a separate held-out test set. Echoing the principles articulated by Grimmer and Stew- 3 art (2013) for earlier text-as-data methods, we argue that any LLM annotation pipeline should be pilot-tested against human-coded samples before deployment at scale. 2 Related Literature 2.1 LLMs in the Text-as-Data Toolkit Large language models have demonstrated considerable promise for a variety of tasks in computational social science (e.g., Ziems et al. 2023; Bail 2024; Thapa et al. 2025; T ̈ornberg 2023, 2024b). Their versatility extends across multiple research applications, from text annotation to agent-based modeling to synthetic data generation. These recent efforts are part of a longer tradition of text-as-data studies, where not just measurement but also evaluation techniques play a central role in the scientific process. Grimmer and Stewart (2013) articulated four foundational principles: all quantitative models of language are wrong but some are useful; no single method dominates; the validation of results against human judgement is essential; and quantitative text analy- sis augments rather than replaces careful reading. These principles were developed for dictionary, scaling, and topic-model methods, but they apply with equal, and arguably greater force to LLM-based annotation, where the parameter space of implementation choices is orders of magnitude larger. Recent work has begun to extend this evaluative tradition to generative models. Halter- man and Keith (2025) propose a five-stage framework for “codebook LLM” measurement, demonstrating that current open-weight models (7–12B parameters) have significant lim- itations in following codebook instructions zero-shot but that supervised fine-tuning can substantially improve performance. Mens and Gallego (2025) show that instruction tuned LLMs can position political texts in ideological spaces with correlations exceeding 0.90 against expert benchmarks, while cautioning that empirical validation remains essential. Argyle et al. (2023) demonstrate that LLMs can simulate diverse human subpopulations in surveys, raising both opportunities and concerns about construct validity. In each 4 case, the sensitivity of results to model selection and prompting choices is acknowledged but not systematically evaluated. Most directly related to our study, Alizadeh et al. (2024) provide a practical guide to open- weight LLMs for text annotation, systematically varying model choice, temperature, zero- shot versus few-shot prompting, fine-tuning sample size, and model scale across political science tasks. Our study builds on this foundation by extending to more recent model families (released through mid-2025), adding prompt style variation (persona and chain- of-thought), measuring computational efficiency alongside performance, and holding all models to identical quantisation and hardware conditions to isolate pipeline effects from infrastructure confounds. Our contribution differs from these prior studies in that we evaluate the pipeline choices themselves as objects of inquiry under tightly controlled conditions. Rather than demon- strating that a particular LLM can perform a particular task, we ask how much the choice of model, size, prompting strategy, and learning approach matters, and whether the popular heuristics researchers use to navigate these choices are empirically justified. 2.2 Model Choice Most existing work on LLM-based annotation in political science has used closed-source models such as GPT-3.5 and GPT-4 from OpenAI (e.g., T ̈ornberg 2024b; Heseltine and Clemm von Hohenberg 2024). While these models offer advantages, lower barriers to en- try, reduced technical expertise requirements, minimal computational needs, high-quality documentation, and strong performance—their use raise significant concerns for repro- ducibility and transparency. Research exploring open-weight alternatives has typically focused on older or smaller models (e.g., Halterman and Keith 2025), creating unfair comparisons with state-of-the-art closed-source systems (e.g., Ziems et al. 2023; Atreja et al. 2024; Barrie, Palaiologou, and T ̈ornberg 2025; Barrie, Palmer, and Spirling 2024). While Halterman and Keith (2025) evaluate open-weight models in the 7–12B parame- ter range and Alizadeh et al. (2024) test models up to Llama-2 70B, our study extends 5 the comparison to models released through mid-2025 (including Qwen 3, Gemma 3, and GPT OSS), providing a more current picture of the open-weight landscape. Crucially, we hold quantisation level, hardware, and prompt structure constant across all compar- isons, isolating model choice from confounding implementation differences that complicate cross-study comparison. 2.3 Model Size Language model size (measured in parameters) significantly impacts performance, par- ticularly for in-context learning. In-context learning refers to the ability of large language models (LLMs) to learn and perform new tasks from examples or instructions provided within the prompt itself, without updating the model’s parameters (Dong et al. 2024). Larger models demonstrate superior ability to override semantic priors and learn con- tradictory input-label mappings, while smaller models rely more heavily on priors (Jerry Wei et al. 2023). Nevertheless, even small models can effectively learn from examples (Schick and Sch ̈utze 2021). Systematic investigation of the effect of model size on annotation performance is lacking in the literature, as researchers typically opt instead to compare different model choices. However, model size has significant implications for the energy use involved in inference (Samsi et al. 2023). Furthermore, much existing work involving open-weight models employs small variants (7-8B parameters), which do not provide a reasonable comparison to much larger closed-source models accessible via API (typically in the range of >100B parameters). The assumption that larger models perform better draws implicitly on neural scaling laws, which suggest predictable relationships between model size, training compute, and pre-training loss (Kaplan et al. 2020; Hoffmann et al. 2022). However, pre-training loss is a poor proxy for downstream annotation performance on domain-specific tasks (Halter- man and Keith 2025). Instruction tuning, task-model compatibility and model shrinking techniques such as pruning and distillation may introduce non-linearities that violate 6 the smooth scaling curves observed in pre-training. Our within-family size comparisons provide a direct test of whether scaling intuitions hold for political science annotation tasks. 2.4 Zero-Shot vs. Few-Shot Learning Zero-shot and few-shot learning represent the two dominant paradigms for deploying LLMs as text annotators without task-specific fine-tuning. In zero-shot settings, models rely solely on instructions and their pre-trained knowledge, while few-shot approaches additionally provide labelled examples within the prompt to guide annotation (Brown et al. 2020). Few-shot methods consistently outperform their zero-shot counterparts across political science applications. For instance, Laurer et al. (2024) demonstrate that BERT-NLI models trained on as few as 500 examples can match the performance of clas- sical supervised models trained on ten times as much data, while Burnham et al. (2024) show that domain-adapted models achieve state-of-the-art few-shot annotation of politi- cal texts with orders of magnitude greater computational efficiency than large commercial LLMs. However, performance gains from example provision vary systematically by task complexity. Binary and low-cardinality annotation tasks with semantically distinct cate- gories tend to benefit most, whereas fine-grained taxonomies with many categories show more modest improvements (Halterman and Radford 2021; Wen and Hauptmann 2023). Recent work also suggests that in-context learning consistently outperforms instruction tuning across computational social science tasks (Wang et al. 2024), and that the quality and representativeness of provided examples matters more than their quantity (Wang et al. 2024; Timoneda and Vera 2025). Despite these advances, most studies examining zero-shot and few-shot performance do so incidentally, evaluating one approach or the other, rather than systematically comparing the two under controlled conditions across multiple tasks. Furthermore, the computa- tional cost of few-shot prompting is rarely quantified: longer prompts increase inference time and energy consumption, yet these trade-offs are seldom reported alongside perfor- mance metrics. 7 2.5 Prompt Style LLMs have shown sensitivity to slight variations in prompt formatting (Sclar et al. 2024), with minor changes potentially causing substantial impacts on performance. However, larger models generally demonstrate greater robustness to these variations (He et al. 2024). Two widely adopted approaches to LLM prompting that are thought to improve model outputs are (1) persona prompting, and (2) chain-of-thought prompting. 2.5.1 Persona Prompting Persona prompting where the analyst provides an LLM with a character description to role play, represents a common technique in the field. This approach has been used to measure demographic stereotypes and bias embedded in LLMs (Cheng, Durmus, and Jurafsky 2023) and to simulate diverse samples of survey respondents (Argyle et al. 2023). Despite its widespread adoption among social scientists, evidence for its effectiveness in annotation tasks remains less compelling. While persona prompting may occasionally lead to performance gains, the effect of each persona can be largely random (Zheng et al. 2024). Other studies suggest that persona prompting offers modest yet significant improvements across some tasks, with the most pronounced benefits occurring in cases where annotators largely disagree but only by a small margin (Hu and Collier 2024). These findings suggest that the benefits of persona prompting for text annotation may be limited and task-dependent, motivating systematic empirical investigation of its effects under controlled conditions. 2.5.2 Chain-of-Thought Prompting Chain-of-thought (CoT) prompting (Jason Wei et al. 2022) encourages models to artic- ulate their reasoning process before providing final answers. While this approach is not necessarily a reliable path to interpretability (Turpin et al. 2023), it does appear to induce some form of reasoning. Despite this, CoT prompting does not always improve perfor- mance (Savelka et al. 2023), particularly for text annotation tasks where clear decision boundaries are more important than complex reasoning chains. 8 CoT implementations vary considerably, with some researchers simply encouraging mod- els to ”think step-by-step” (e.g., Kojima et al. 2022), while others require explicit expla- nations of reasoning either before or after annotation decisions, and provide reasoning steps alongside their example sets (Jason Wei et al. 2022). Recent research demonstrates that task-specific prompting significantly outperforms unsupervised prompt generation, emphasising the necessity of thoughtful human guidance in CoT prompting (Zhang et al. 2025). However, CoT prompting does not improve performance on language reasoning to the same extent as it does with arithmetic tasks (Wang et al. 2024). Despite its theo- retical appeal, the computational overheads of CoT prompting raise important questions about efficiency trade-offs when undertaking large-scale text annotation tasks. Most closely related to our work, Atreja et al. (2025) conduct a large-scale experiment testing how model selection and prompt design features (definition inclusion, output type, explanation requests, and prompt length) affect LLM annotation compliance and accuracy across four computational social science tasks. They find that compliance and accuracy are prompt-dependent and that minor changes can cause large shifts in label distributions. Our study extends this line of inquiry in two directions: we cross prompt style with model choice (testing all style–model combinations rather than evaluating each in isolation) and we systematically measure the efficiency costs of prompt engineering techniques, providing a performance-per-unit-cost perspective absent from prior work. 3 Methods This paper employs a controlled factorial design to measure the performance of LLMs on four distinct text annotation tasks from political science. By systematically varying four dimensions of the annotation pipeline (model choice, model size, learning approach, and prompt style), while holding hardware, quantisation, and prompt structure constant, the design isolates the marginal contribution of each factor and reveals interaction effects that single-dimension evaluations cannot detect. One of the four tasks is an original project involving one of the co-authors, while the remaining three are replications of 9 existing papers. The tasks are deliberately chosen to span a wide range of text types (parliamentary speeches, newspaper excerpts, transcriptions of committee deliberations), response formats (binary labels, multi-class categorisation, ordinal Likert scales), and units of analysis (sentences, article segments, debate interventions, full speeches). The conceptual complexity of the tasks also varies considerably: from economic sentiment, which requires a relatively straightforward assessment of tone, to manifesto topic annota- tion, which relies on a domain-specific coding scheme with definitions and examples that must be absorbed through in-context learning. Together, these tasks provide a robust testbed for evaluating LLM annotation performance under diverse conditions. 3.1 Task Definitions Each of the tasks is briefly introduced below, and Table 1 provides a summary overview. See the codebooks in Appendix A for a complete overview of the tasks, definitions for each dimension, and example sets. For the psychological distance task, the codebook reflects the exact coding instructions distributed to human annotators. For the three replication tasks, codebooks were reconstructed from the appendices and supplementary materials of the original papers, including the examples used to illustrate category bound- aries where these were available. For the psychological distance task, the examples are the same ones used in the human coding process: they were developed through iterative labelling rounds and intercoder-reliability checks to clarify recurrent disagreements, then frozen once the codebook was finalised. Codebooks for all four tasks were formalised using CodeBook Studio (McLaren 2026b), a web-based annotation application that sup- ports binary, categorical, Likert, and free-text annotation types and provides structured outputs. For the psychological distance task, CodeBook Studio was also used to collect the human annotations; for the remaining three tasks, ground-truth data were obtained from the original authors’ published materials. 10 3.1.1 Approval General approval (Wratil and Hobolt 2019) refers to the degree to which a speaker ex- presses approval of the proposal(s) being negotiated or the state of negotiations. The data consists of transcriptions of video footage of public deliberations of the Council of the EU’s ECOFIN configuration (2010–2015), with the debate participation as the unit of analysis. Coders first identify the major dimension of contestation within each de- bate, typically either the legislative proposal presented by the European Commission, the Council’s state of play on the proposal, or the state of negotiations with the Euro- pean Parliament, and then assess each speaker’s degree of approval on a 5-point Likert scale, where 1 indicates full approval and 5 indicates full disapproval. The ground truth comprises 1,110 debate participations by national delegations, the Commission, and other actors (e.g. the ECB) across 86 debates, coded by a single coder; a random 25% of the dataset was double-coded to ensure sufficient intercoder reliability (α = 0.82). The data is right-skewed: an approval score of 1 accounts for 37% of observations, with the pro- portion decreasing gradually as the scale increases, and a score of 5 accounting for just 5%. Prior research shows that asking LLMs to provide numerical scores, as in the case of a Likert scale, instead of labels reduces compliance and accuracy (Atreja et al. 2025). 3.1.2 Psychological Distance The psychological distance of climate change refers to the sense of proximity or distance from the impacts of climate change created by a speech or text (McLaren 2025). The data consists of European Parliament plenary speeches (2014–2024), with the sentence as the unit of analysis. The task is assessed in a three-step process: (1) determining whether references to environmental impacts are present [binary]; (2) assessing their specificity [categorical: specific; universal]; and (3) evaluating their implied proximity [categorical: proximate; distant]. The ground truth comprises 1,037 sentences, coded by three human annotators using the codebook in Appendix A; final annotations for each sentence were resolved through majority agreement. Because the three annotation dimensions are nested, each successive step applies to a smaller subset of the data. The 11 dataset is heavily imbalanced at the first step: 89.2% of sentences contain no reference to environmental impacts (Presence = No), leaving only 10.8% for subsequent annotation. Of those, roughly 60% are coded as Universal and 40% as Specific. Among the Specific subset, approximately two-thirds are coded as Proximate and one-third as Distant. 3.1.3 Economic Sentiment Economic sentiment refers to the tone of newspaper coverage of U.S. national economic performance (Barber ́a et al. 2021). The data consists of newspaper articles, with the article segment (first five sentences) as the unit of analysis. Each segment is annotated as either negative or positive [binary] based on the indication the text gives about how the U.S. economy is performing. The ground truth comprises 420 segments, each coded by a minimum of seven CrowdFlower coders on a 9-point ordinal scale (1 = very negative, 9 = very positive). The average tone across coders was computed for each segment and then dichotomised: scores of 1–4 were coded as negative (0) and scores of 6–9 as positive (1), while segments with an average score of 5 were discarded. The resulting dataset is relatively balanced, with 39% positive and 61% negative instances. 3.1.4 Manifesto Topic The manifesto topic task assigns political text to policy topic categories based on the Comparative Manifesto Project (CMP) coding scheme (Osnabr ̈ugge, Ash, and Morelli 2023). The data consists of speeches from the New Zealand parliament (1987–2002), with the speech as the unit of analysis. Following Osnabr ̈ugge, Ash, and Morelli (2023), we use a collapsed 8-topic specification rather than the full set of 44 Manifesto Project topics, as each topic is provided with its own definition and examples drawn from the original Manifesto Project codebook and including all 44 would excessively lengthen the prompt. Each speech is assigned to one of eight domain categories [categorical: No Domain, External Relations, Freedom And Democracy, Political System, Economy, Welfare And Quality Of Life, Fabric Of Society, and Social Groups]. The ground truth comprises 4,165 speeches, coded by a single coder from New Zealand trained by the Manifesto Project 12 team; the coder’s performance was validated by comparing their annotations with those of three other trained coders on a random sample of 250 speeches. The dataset exhibits significant class imbalance: Political System and Welfare and Quality of Life account for 26% and 19% of all speeches respectively, while No Topic comprises 4.6% and External Relations just 2.3%. Task Text Type Unit of Analysis Annotation Type(s)N No. Coders Aggregation StrategySource ApprovalCouncil of EU deliber- ations Debate participa- tion 5-pt Likert 1,110 1 (25% double- coded) Single coderWratil and Hobolt (2019) Psych. Distance EP plenary speeches SentenceBinary, categorical (2 classes) 1,0373Majority agreementOriginal Econ. Sentiment Newspaper articles Article segment (first 5 sent.) Binary420 ≥ 7Averaged 9-pt scale, dichotomised Barber ́a et al. (2021) Manifesto Topic NZ parl. speeches SpeechCategorical (8 classes) 4,165 1 (vali- dated against 3) Single coderOsnabr ̈ugge, Ash, and Morelli (2023) Table 1: Overview of the four annotation tasks. N = number of units in the ground truth dataset. 3.2 LLM Annotation Our research explores the performance of various approaches to using language models as annotators with minimal training data, validated against human performance. We also investigate the trade-offs involved in using larger models and longer prompts in terms of inference time, energy intensity, and response length. All models in our experiments are Q4 KM quantised and run on 2 x NVIDIA A100-PCIE- 40GB GPUs in the University College Dublin (UCD) Sonic HPC cluster. Quantisation techniques offer promising efficiency improvements, with 4-bit quantised models reducing energy consumption by approximately 3.5×, potentially cutting the overall energy foot- print of inference by up to 72% without sacrificing performance (Dettmers et al. 2023). 13 We use each model’s default sampling parameters as specified in its model card; where a model does not specify a parameter, Ollama’s defaults of temperature = 0.8, top k = 40, and topp = 0.9 apply. Table 3 reports the sampling parameters used for each model. Ex- periments are orchestrated by CodeBook Lab (McLaren 2026a), an open-source pipeline that takes the structured outputs of CodeBook Studio and benchmarks LLM performance across user-specified combinations of model, prompt style, and learning approach. The experimental grid is defined in a single configuration file, and inference is handled by Ollama. Energy consumption is tracked via CodeCarbon (Courty et al. 2024). Each experimental dimension is tested while holding the remaining dimensions constant. The model choice and model size experiments use zero-shot prompting with the stan- dard prompt style, isolating the effect of the model itself. The zero-shot versus few-shot comparison likewise uses the standard prompt style, varying only the provision of exam- ples. The prompt style experiments include few-shot examples in all conditions, so that observed differences can be attributed to the prompt style rather than being confounded with the presence or absence of examples. We structure prompts for different annotation types following standardised templates, as shown in Table 2. Prompt content (definitions, response options, and examples) is drawn directly from the codebooks in Appendix A. These task-specific materials were frozen prior to benchmarking: no examples were added, removed, or rewritten on the basis of model outputs. The few-shot examples are therefore the same examples embedded in the corresponding human codebooks, and the persona and chain-of-thought conditions are implemented as pre-specified template variants layered on top of those frozen task instructions. 14 Annotation Type Standard Prompt Template Binary<section name> <section instructions> <annotation name> <tooltip> Respond with 1 if “Yes” or 0 if “No”. Return your response in JSON format, with the key “response”. <example(s)> — Text: “<text to annotate>” Response: Categorical<section name> <section instructions> <annotation name> <tooltip> Respond with <option1>, or <option2>, or ... Return your response in JSON format, with the key “response”. <example(s)> — Text: “<text to annotate>” Response: Likert scale<section name> <section instructions> <annotation name> <tooltip> Respond with a whole number from <min value> to <max value> (inclusive), where <min value> means lowest and <max value> means highest. Return your response in JSON format, with the key “response”. <example(s)> — Text: “<text to annotate>” Response: Table 2: Standardised prompt templates by annotation type. Our experimental design explores the following dimensions: 3.2.1 Model Choice We evaluate six open-weight language models as text annotators: Qwen 3 (Yang et al. 2025), Qwen 2.5 (Qwen et al. 2025), Gemma 3 (Gemma Team et al. 2025), GPT 15 OSS (OpenAI et al. 2025), DeepSeek R1 (DeepSeek-AI et al. 2025), and Llama 3.1 (Grattafiori et al. 2024). Each model represents different architectural approaches and training methodologies. We assess their performance across all four annotation tasks to identify strengths, weaknesses, and efficiency trade-offs. Table 3 summarises the key characteristics of each model. ModelParams.ArchitectureRelease Reason. Temp. Top-k Top-p Qwen 332BDense transformerMay 2025Yes0.6200.95 Qwen 2.572BDense transformerSep. 2024No0.8*40*0.9* Gemma 327BDense transformerFeb. 2025No1.0640.95 GPT OSS20B (3.6B active) MoE transformerAug. 2025Yes1.040*0.9* DeepSeek R170BDense transformer † Jan. 2025Yes0.8*40*0.9* Llama 3.170BDense transformerJul. 2024No0.8*40*0.9* Table 3: Comparison of the six language models used in this study. All models are run in Q4KM quantisation. † Distilled from DeepSeek R1 (671B MoE); the 8B and 70B variants use Llama base models, while smaller variants use Qwen 2.5 bases. * = Ollama default (parameter not specified in model card). 3.2.2 Model Size To understand the relationship between model size and annotation performance, we test model variants with different numbers of parameters from the Gemma 3, DeepSeek R1, and Qwen 3 families. This analysis tests whether larger models are in fact better at anno- tation tasks, and how the performance–efficiency tradeoff evolves as model size increases within a family. 3.2.3 Zero-shot vs. Few-Shot Learning We compare zero-shot performance (using only instructions) against few-shot approaches (including instructions and examples) across our tasks. This comparison mirrors our manual annotation protocol, where human annotators received training examples. We measure whether and in what manner the provision of examples affects both performance and computational requirements. 16 3.2.4 Prompt Style We examine three distinct prompt styles to determine their impact on annotation per- formance and model efficiency: • Standard: This baseline approach consists of three components: (1) a definition that establishes the concept to be measured, (2) a task description that specifies what the model should do with the input text, and (3) a set of response options that constrain the output format to ensure consistency. • Persona: This approach adds a prefix to the standard prompt that describes specific attributes or expertise the model should embody while completing the task. We test whether framing the model with relevant expertise improves performance. • Chain-of-thought: This approach appends a suffix to the standard prompt that encourages the model to make explicit use of its reasoning abilities before provid- ing the final annotation. We investigate whether promoting reasoning improves performance, particularly for complex or ambiguous cases. Figure 1 illustrates the structure of each prompt style. We evaluate each approach to determine whether common prompt engineering techniques yield better performance on our annotation tasks, and at what computational cost. 17 Standard Prompt Conceptual Definition Task Description Response Options Persona Prompt “You are an expert political scientist and data annotator with extensive experience...” Conceptual Definition Task Description Response Options Chain-of-Thought Prompt Conceptual Definition Task Description Response Options “I’l think through this step by step: First, I’l identify key parts of the text relevant to this...” Figure 1: Structure of different prompt styles for LLM annotation 3.3 Evaluation Following the validation principles articulated by Grimmer and Stewart (2013), we treat human annotations as the ground truth against which LLM performance is evaluated. Our evaluation approach varies by annotation type to ensure appropriate assessment of model performance. In all cases, the human-annotated sample described in Table 1 serves 18 as validation data. For binary or categorical annotations, we use classic classification metrics including accu- racy, precision, recall, and F1 score. We also measure intercoder reliability between LLM annotations and ground truth through Cohen’s kappa and Krippendorff’s alpha, which allow direct comparison to human annotator agreement levels. For Likert scale annotations, we employ both classic classification metrics and measures specifically designed for ordinal data. Quadratic weighted kappa and Spearman’s corre- lation are particularly relevant for ordinal scales, as they penalise predictions that are further from the ground truth value more heavily than those that are closer. Researchers face real computational and budgetary constraints when deploying LLMs at scale: annotation projects involving tens of thousands of documents may require days of GPU time and significant energy expenditure. To quantify these practical constraints, we measure three efficiency metrics for each model configuration, each capturing a distinct researcher concern: • Energy consumption (kilowatt-hours), estimated using CodeCarbon (Courty et al. 2024), captures the sustainability cost of inference. • Total output characters captures the volume of text a model generates. Because API platforms charge per token, output length is a direct proxy for the financial cost of annotation at scale. • Inference time (average per query and total) captures the practical workflow con- straint, particularly relevant when annotating large corpora under deadline pres- sure. We report these metrics separately rather than collapsing them into a composite index because our results show they are heterogeneous: a model can be fast but verbose (high token cost, low time cost), or energy-efficient but slow. Collapsing them would obscure the tradeoffs researchers need to evaluate against their own constraints. A researcher on a tight budget may prioritise minimising token volume; one running on local hardware may 19 prioritise inference time; one concerned with environmental impact may prioritise energy. The normalised multi-panel figures we present let each reader weight these dimensions according to their own priorities. All experiments were conducted on the UCD Sonic HPC cluster in Ireland, using NVIDIA A100-PCIE-40GB GPUs. 1 These measurements allow us to examine tradeoffs between annotation performance and computational cost, drawing conclusions about the relative merits of using larger models or more complex prompts under realistic resource con- straints. 4 Results The results are organised by experimental dimension. Within each subsection, we present an overall performance-to-efficiency comparison, decomposed efficiency metrics, and a task-level performance breakdown that reveals the interaction effects central to our argu- ment. Together, these findings provide the empirical foundation for the validation-first framework we develop in Section 5. All metrics are macro-averaged, assigning equal weight to all classes for each response item. For tasks containing multiple response items of the same annotation type, we report the average metric across these items. 4.1 Model Choice Applied studies often select a model on the basis of benchmark rankings or community reputation, implicitly assuming that a model that performs well on general-purpose tasks will also perform well as a text annotator. We test this by comparing six open-weight models of varying size and architecture on four annotation tasks. Figure 2 plots the tradeoff between average annotation performance (F1) and energy consumption for each model. The ideal position is the lower-right corner: high perfor- mance at low energy cost. No model occupies this position convincingly, and no clear 1. The Sonic cluster contains a heterogeneous set of GPU nodes (including NVIDIA V100, A100, and H100 graphics cards). 20 relationship between model efficiency, size and/or performance is observed. GPT OSS and Qwen 3 are effectively tied as the strongest performers overall, even though GPT OSS is the smallest model by total parameter count (20B). By contrast, the weakest per- formers are LLaMA 3.1 and Qwen 2.5, both of which underperform GPT OSS while using substantially more parameters. DeepSeek R1, although the same size as Llama 3.1, con- sumes approximately twenty times more energy, indicating that efficiency is a function of more than just number of model parameters. These performance differences likely reflect variation in model architecture, instruction fine-tuning procedures, and model-prompt compatibility rather than parameter count alone. Gemma 3 is the most energy-efficient model we tested, and performs reasonably well across tasks. Figure 2: Model choice performance–energy trade-off. Points show average F1 score and energy consumption for each model across the four annotation tasks, with point size indicating model parameters. Further disaggregating efficiency metrics in Figure 3 reveals a more nuanced picture. While models that consume more energy also tend to have longer inference times, the correspondence between energy consumption and the other efficiency metrics is not per- fect, underscoring the importance of considering multiple dimensions of efficiency rather 21 than relying on any single measure. DeepSeek R1 takes the longest time to respond to a query by a considerable margin, consistent with its high energy consumption. GPT OSS produces the shortest responses on average but falls in the middle of the pack in terms of both inference time and energy consumption, despite being the smallest model tested. Observing total output characters, DeepSeek R1 and Qwen 3 appear least willing to obey response constraints held in the prompt (i.e. “Respond only with one of the follow- ing options: <option1>, or <option2>, or <option3>. Return your response in JSON format, with the key ‘response’”), as they generate considerably more characters of output compared to other models. Gemma 3 is the most energy-efficient model and has the shortest inference time, though it produces longer responses than GPT OSS and Qwen 2.5—illustrating that which model is “most efficient” depends on the metric considered. Though GPT OSS and Qwen 3 are the most performant models, Qwen 3’s strong performance comes with relatively high energy consumption, long responses, and the second longest inference time after DeepSeek R1. Figure 3: Model choice performance and efficiency metrics. Panels compare average F1 score, inference time, energy consumption, and output length for each model, aggregated across tasks. Figure 4 shows the performance spread for each model by task. Task leadership is dis- tributed across models: LLaMA 3.1 leads on approval, Qwen 3 on psychological distance, 22 DeepSeek R1 on economic sentiment, and GPT OSS on the manifesto topic task. No single model dominates across all tasks, reinforcing the centrality of task–model inter- action effects. Even the strongest overall performers show considerable variation: Qwen 3 achieves F1 scores in excess of 0.7 on the psychological distance and economic senti- ment tasks but performs considerably worse on the two remaining tasks. This indicates that no model can simply be used out-of-the-box without first establishing that it can approximate human performance on a given annotation task. Further sensitivity checks should also be conducted, as precision may be more relevant than recall in some cases, depending on application, or vice versa. Figure 4: Model choice performance by task. Task-level F1 scores are shown for each model across the four annotation tasks. Our initial determination around model choice is that, while certain models perform bet- ter than others on average, the variation is too task-dependent to arrive at any universal best practice. Furthermore, greater consideration should be paid to model efficiency, as some model choices will be more suitable to certain tasks based on the volume of data to be annotated and resource constraints. Scholars should reflect carefully on whether slightly increased performance for their task is worth greater requirements in terms of 23 time and energy use. 4.2 Model Size A persistent intuition in the field is that bigger models are better: more parameters should mean more capacity to follow complex annotation instructions. We test this by comparing multiple size variants within three model families. The Gemma 3 family exhibits relatively consistent performance gains for each increase in model size in Figure 5, though this seems to level off after a certain threshold. Curi- ously, the DeepSeek R1 family peaks in performance at 8B parameters, before dropping off sharply, only starting to recover again at the largest 70B parameter variant. The per- formance for the Qwen 3 family appears to alternate between increasing and decreasing with each increase in size, though performance trends upwards overall. Figure 5: Model size performance–energy trade-off within model families. Panels show average F1 score against model size for Gemma 3, DeepSeek R1, and Qwen 3 variants, with point size indicating energy consumption. Each of our efficiency metrics seems to track the others relatively well within model families but not across families, according to Figure 6. This suggests that size often covaries with efficiency within a family, but the relationship is not strictly monotonic and includes notable exceptions. Other factors remain more important overall, as the largest variant of the Gemma 3 family is less demanding than even the smallest variants of the 24 other families tested, despite having orders of magnitude more parameters. Notably, the smallest Gemma 3 variant (270M) and the largest variants of DeepSeek R1 (70B) and Qwen 3 (32B) exhibit weak compliance with the requested output format, generating substantially longer responses that drive up both energy consumption and inference time. In contrast to DeepSeek R1’s exponential growth in resource requirements, the increase for the Gemma 3 family appears quite modest as model size scales up. The 4B parameter version of Qwen 3 is significantly more verbose than other variants from the same family, and slower and more energy-intensive as a consequence. Figure 6: Model size efficiency metrics within model families. Panels compare average inference time, energy consumption, and output length across Gemma 3, DeepSeek R1, and Qwen 3 size variants. Breaking down performance by task in Figure 7, we see that the spike in performance for the 8B parameter variant of DeepSeek R1 (and subsequent dropoff in performance for larger variants) is driven by the economic sentiment and manifesto topic tasks, while this family’s performance increases relatively linearly with size for the other two tasks. Similarly, peaks and valleys in performance are observed for the Qwen 3 family depending on task. 25 Figure 7: Model size performance by task. Panels show task-level F1 scores for model variants within the Gemma 3, DeepSeek R1, and Qwen 3 families. Together, these findings disrupt both rules-of-thumb that bigger is better and that bigger is more resource-intensive. We find that large models may be considerably less demanding than small models of a different family. We also find that resource use does not increase with model size at the same rate between families, nor does it necessarily increase con- sistently within a given family. Even holding model family constant, increasing size may decrease performance and/or increase resource consumption. This further reinforces the idea that there are no universal best practices and that LLM decisions should be empirically justified based on the specific application. These non-monotonic patterns likely reflect, in part, differences in how model families produce smaller variants. Model developers typically reduce model size through knowledge distillation, in which a smaller “student” model is trained to reproduce the outputs of a larger “teacher” model, or through pruning, in which layers, neurons, or attention heads 26 are removed from a larger model, often with subsequent retraining to recover accuracy. In practice, the approach varies across and even within model families. The Gemma 3 and Qwen 3 families both use knowledge distillation to produce smaller variants from a common architecture (Gemma Team et al. 2025; Yang et al. 2025), which should in principle yield consistent behaviour across sizes. Gemma 3’s relatively monotonic scaling is consistent with this expectation. Qwen 3’s oscillating performance across sizes may reflect varying distillation fidelity at different compression ratios. The DeepSeek R1 family takes a different approach entirely: rather than distilling from a single architecture, it fine-tunes pre-existing, independently trained base models from other families—Qwen 2.5 at 1.5B, 7B, 14B, and 32B parameters, and Llama at 8B and 70B—using reasoning samples generated by the full 671B parameter teacher (DeepSeek- AI et al. 2025). The 8B performance spike thus compares a fundamentally different base architecture (Llama 3.1) to its nominal family members (Qwen 2.5). Notably, this archi- tectural inconsistency only became apparent after investigating the anomalous results; nothing in the Ollama naming convention signals that size variants use different base architectures. This illustrates a broader challenge: pipeline choices can interact with easily overlooked model properties in ways that are difficult to anticipate without both empirical validation and close reading of technical documentation. 4.3 Zero-Shot vs. Few-Shot Learning The effect of providing labelled examples, i.e. the distinction between zero-shot and few-shot prompting, is one of the most frequently cited recommendations in LLM an- notation guidelines (T ̈ornberg 2024a). Our results, however, reveal a substantially more nuanced picture than the conventional wisdom that few-shot learning uniformly improves performance. 27 Figure 8: Few-shot performance–energy trade-off. Points show the change in average F1 score and energy consumption when moving from zero-shot to few-shot prompting for each model, with point size indicating model parameters. Figure 8 plots the change in average F1 against the change in energy consumption when moving from zero-shot to few-shot conditions, revealing the tradeoff involved for each model. The direction of the effect is strikingly model-dependent. DeepSeek R1 is the clearest beneficiary, improving by approximately 0.05 F1 points under few-shot prompt- ing while also consuming less energy. LLaMA 3.1 and Qwen 2.5 show marginal gains, while Gemma 3 is essentially unchanged. Most notably, GPT OSS and Qwen 3—the two strongest zero-shot performers—degrade under few-shot conditions, with GPT OSS drop- ping by approximately 0.04 and Qwen 3 by approximately 0.03 F1 points. Qwen 3 also consumes significantly more energy under few-shot conditions, placing it in the worst-case quadrant of the tradeoff. This represents a cautionary finding for researchers who assume that providing examples is uniformly beneficial: the models that perform best without examples may actually be harmed by their inclusion. The efficiency implications of few-shot prompting are also more complex than commonly assumed. Figure 9 shows that DeepSeek R1 becomes substantially more energy-efficient 28 under few-shot conditions, while Qwen 3 becomes considerably less efficient. The remain- ing models show relatively modest changes. Gemma 3 remains the most energy-efficient model under both conditions. Figure 9: Few-shot performance and efficiency metrics. Panels compare average F1 score, inference time, energy consumption, and output length under zero-shot and few-shot prompting across models. Decomposing performance by task in Figure 10 reveals that the aggregate patterns mask considerable task-level heterogeneity. The psychological distance task shows the most widespread benefit from few-shot learning, with DeepSeek R1 exhibiting a particularly dramatic gain (from approximately 0.39 zero-shot F1 to 0.68 under few-shot conditions). This is consistent with the task’s nested, multi-step structure, which may benefit from worked examples that clarify the annotation logic. However, even here the effect is not universal: GPT OSS degrades under few-shot conditions on this task. By contrast, the approval task is largely insensitive to learning approach, with most models performing similarly under both conditions, suggesting that this relatively straightforward Likert- scale task is well-served by instructions alone. 29 The economic sentiment task shows most models maintaining roughly similar performance under both conditions. On the manifesto topic task, results are mixed: several models show modest declines, possibly because the eight-category scheme with accompanying definitions already provides substantial in-context information, and additional examples add noise or prompt-length overhead. Figure 10: Few-shot performance by task. Panels show task-level F1 scores for each model under zero-shot and few-shot prompting. Together, these findings challenge the blanket recommendation to use few-shot prompt- ing for annotation tasks. The decision to provide examples is itself a researcher degree of freedom with model- and task-dependent consequences. For some model–task com- binations, few-shot prompting delivers meaningful improvement; for others, it degrades performance or simply adds computational overhead. Researchers should empirically test both approaches on their specific model and task before committing to either. 30 4.4 Prompt Style Prompt engineering—iteratively modifying instructions to improve model output—is per- haps the most accessible lever available to researchers deploying LLMs as annotators, and the one most susceptible to undisclosed analytical flexibility. Without a held-out test set, each modification risks overfitting to the validation sample rather than improving gen- uine annotation ability. Our interest here is not only whether two prominent techniques help in themselves, but what they reveal about prompt engineering as a practice. We ex- amine persona prompting (framing the model as a domain expert) and chain-of-thought prompting (instructing the model to reason step-by-step) as case studies: both are widely recommended, theoretically motivated, and easy for researchers to adopt without formal validation. We do not claim to evaluate prompt engineering exhaustively; rather, we use these two well-motivated interventions to illustrate the risks posed by iterative prompt modification without a separate test set. Their unstable effects in our experiments suggest that prompt engineering should be treated as a consequential tuning decision requiring held-out evaluation, rather than as a source of portable best practices. Figure 11 plots the change in F1 against the change in energy consumption when moving from standard prompting to persona (left panel) and CoT (right panel) conditions. The ideal outcome—improved performance at low additional cost—would place a model in the lower-right quadrant. Instead, most models cluster in the upper half (energy increases) with F1 changes near zero or negative, and the aggregate picture is once again one of inconsistency. Neither persona nor CoT prompting reliably improves over the standard baseline. The most dramatic effect is negative: Llama 3.1 experiences a severe collapse under persona prompting, with average F1 dropping from approximately 0.40 to below 0.30. By contrast, GPT OSS improves modestly under both persona and CoT conditions at relatively low energy cost, owing to its small active parameter count under its mixture- of-experts architecture. Qwen 3 performance is largely invariant to prompt style but incurs a substantial energy penalty under persona conditions. DeepSeek R1 improves under persona prompting but occupies the worst-case position under CoT: increased energy consumption paired with decreased performance. Llama 3.1 shows a substantial 31 F1 gain under CoT but at the highest energy cost of any model, representing a high-cost, high-reward tradeoff. The net effect of either technique, averaged across models, is close to zero, with variance that far exceeds the mean. Figure 11: Prompt style performance–energy trade-off. The left panel shows the change in average F1 score and energy consumption from standard to persona prompting, and the right panel shows the same comparison for standard to chain-of-thought prompting. The decomposed efficiency metrics in Figure 12 reveal a more nuanced pattern than a simple “CoT costs more” narrative. GPT OSS spikes in response length under both persona and CoT conditions, with only marginal effects on performance. By contrast, response length for Qwen 3 and DeepSeek R1 plummets under persona and CoT condi- tions, suggesting that these reasoning models produce more focused outputs when given explicit reasoning instructions but not necessarily better ones. The energy picture does not track response length: Qwen 3 exhibits a dramatic spike in energy consumption under persona prompting despite producing shorter responses, while energy costs for the other models vary in both direction and magnitude across prompt styles. 32 Figure 12: Prompt style performance and efficiency metrics. Panels compare average F1 score, inference time, energy consumption, and output length across standard, persona, and chain-of-thought prompting. 33 Figure 13: Prompt style performance by task. Panels show task-level F1 scores for each model across standard, persona, and chain-of-thought prompting. The task-level decomposition in Figure 13 helps explain the aggregate null result. The approval task is largely insensitive to prompt style for most models, though GPT OSS no- tably degrades under CoT, suggesting that for simple tasks the additional reasoning can actively hurt performance. The psychological distance task shows more divergent effects: Llama 3.1 improves substantially under CoT (from 0.57 to 0.64 F1) but collapses under persona prompting (from 0.57 to 0.23 F1), illustrating how the same model can respond in opposite directions to different prompting interventions on the same task. Economic sentiment, despite being a relatively straightforward binary task, sees performance de- grade under CoT for DeepSeek R1 (from 0.71 to 0.47 F1), possibly because extended reasoning introduces second guessing on cases where the initial judgement was correct. The manifesto topic task shows minimal systematic differences across prompt styles. The practical implications of our findings are clear. Our evidence does not support the routine use of persona or chain-of-thought prompting for LLM-based text annotation. 34 The performance effects of both techniques are erratic, showing positive effects for some model–task combinations and negative for others. The efficiency costs of these tech- niques are similarly model-dependent: some models incur substantial energy increases while others show modest or even reduced costs, but these efficiency shifts do not reli- ably correspond to performance improvements. Researchers who adopt these techniques without sufficient attention to task-specific validation exercises risk both degrading their measurements and incurring unnecessary computational costs. 5 A Validation-First Framework for LLM Annota- tion The preceding results paint a consistent picture: the effect of any single pipeline choice on annotation performance depends on its interaction with the model, the task, and other pipeline choices. This pervasive task-dependence means that there are no universal “best practices” for LLM annotation, but this does not mean that anything goes. In this section, we distil our findings into a practical framework for researchers designing LLM annotation pipelines. 5.1 A Validation Workflow Any LLM annotation pipeline should be pilot-tested against a human-coded validation sample before being deployed at scale, echoing the third principle of Grimmer and Stewart (2013). Our results imply a specific order for this validation process: 1. Define efficiency constraints first. Cross-family efficiency differences dominate within-family differences: Gemma 3 at 27B parameters is less resource-intensive than the smallest DeepSeek R1 variant at 1.5B parameters (Figure 6). Shortlist two to three model families that fit your hardware, budget, and timeline before evaluating performance. 2. Run candidate models zero-shot with standard prompting on a human- 35 coded validation sample. Model choice is the highest-impact decision, with spreads exceeding 0.15 F1 points across models (Figure 2). Standard zero-shot prompting is the cheapest baseline to evaluate; a few hundred documents stratified across categories is typically going to be sufficient. 3. Select the best-performing model on your task. Do not rely on published benchmarks. As we demonstrate, task–model interactions dominate main effects. Task leadership is distributed across four different models, with no single model dominating (Figure 4). 4. Test few-shot on your selected model. The few-shot effect is model-dependent (Figure 8): strong zero-shot performers (GPT OSS, Qwen 3) may actually degrade, while weak zero-shot performers can improve dramatically (DeepSeek R1 on psycho- logical distance: 0.39 → 0.68 F1). If zero-shot performance is already satisfactory, skip this step. 5. Only test prompt styles if performance remains unsatisfactory. Prompt en- gineering is the lowest-impact decision, with erratic performance effects and model- dependent efficiency costs (Figure 11, Figure 12). This sequence of decisions is methodologically sensible, but it also creates a risk of over- fitting. Each pipeline decision—model selection, prompt wording, codebook design— functions as an adjustable hyperparameter. Researchers who iteratively modify prompts or codebook language to improve performance on a validation sample are, in effect, tun- ing the pipeline to a fixed set of documents rather than to the underlying task. If these modifications deviate from the instructions given to human annotators, the resulting comparison between human and LLM performance is no longer on equal footing. Our finding that performance varies unpredictably across tasks and configurations (Figure 4, Figure 13) reinforces this concern: gains observed on one sample may not generalise. The design reported here was constructed to minimise this risk: task definitions, response op- tions, and examples were frozen before benchmarking, and prompt-style manipulations were specified in advance as common template variants rather than tuned on the basis of 36 model performance. The standard remedy in machine learning is a three-way split into training, validation, and test sets, where the test set is withheld until all development decisions are finalised (see e.g. Goodfellow, Bengio, and Courville 2016). We recommend the same discipline for LLM annotation: researchers who undertake iterative prompt engineering or codebook revision should maintain a held-out test set and report final performance on it rather than on the validation sample used to guide design choices. Under this design, few-shot examples should be drawn from a training partition, pipeline variants compared on a validation partition, and the final workflow reported once on a held-out test partition. Alternatively, researchers can sidestep this problem by providing the model with the same codebook distributed to human annotators, without further modification. This preserves the two- way split, avoids the degrees of freedom introduced by prompt engineering, and ensures a direct comparison between human and LLM performance on identical instructions. 5.2 Open Questions: Task Type, Architecture, and Efficiency The preceding workflow treats all tasks alike, but our per-task decompositions suggest that task characteristics may predict which pipeline choices matter. For instance, the simple approval task is largely insensitive to both prompt style and few-shot learning, while the complex psychological distance task shows more widespread few-shot benefits. Whether these patterns generalise beyond our four tasks is an open question: our task set is too narrow to support confident heuristics about task-type effects, and we flag this as a priority for future work with a broader and more diverse set of annotation tasks. Two practical observations do emerge with more generality from our results. First, model architecture determines efficiency more than parameter count. When choosing models under resource constraints, researchers should compare efficiency across families rather than within them. Gemma 3 at 27B parameters is less resource-intensive than the smallest DeepSeek R1 variant (Figure 6). A model half the size from a different family can be more expensive to run. 37 Second, reasoning models may respond differently to prompting interventions. Models trained with reinforcement learning to produce intermediate reasoning steps (Qwen 3, GPT OSS, DeepSeek R1) might be expected to benefit less from chain-of-thought prompt- ing, since explicit CoT instructions are partially redundant with built-in reasoning ca- pabilities (see e.g. Sprague et al. 2025). Our results offer partial support: DeepSeek R1 degrades under CoT for some tasks, and Qwen 3 is largely invariant to prompt style (Figure 13). However, GPT OSS improves under CoT, and the few-shot results do not split cleanly along reasoning lines either. The reasoning distinction is therefore a use- ful heuristic—researchers should be cautious about adding CoT prompts to reasoning models—but it does not override the task–model interactions that dominate our findings. 5.3 Reporting Standards A recurring theme of our results is that pipeline choices interact in ways that are dif- ficult to predict from first principles. Model rankings shift across tasks (Figure 4), the effect of few-shot examples and prompt style varies by model and task (Figure 10, Fig- ure 13). These interactions mean that omitting pipeline details from a published study makes it impossible for readers to assess whether the reported performance would hold under alternative, equally defensible configurations. In adjacent fields, initiatives such as model cards (Mitchell et al. 2019) and datasheets for datasets (Gebru et al. 2021) have established norms around structured transparency; LLM-based annotation in the social sciences would benefit from analogous standards. We recommend that studies using LLM-based annotation report, at minimum, the following: 1. Model identity: the model name, version string, and parameter count. Our model choice experiments show performance spreads exceeding 0.15 macro F1 across mod- els of comparable size (Figure 2); model identity is therefore essential for interpret- ing any reported result. 2. Quantisation: the quantisation method and level (e.g., GGUF Q4 KM). All mod- els in our study were evaluated at identical quantisation; varying this parameter 38 introduces a further dimension of analytical flexibility that should be made explicit. 3. Prompt text: the full prompt or a representative template, including system instructions. Our prompt style results demonstrate that seemingly minor changes to prompt wording (e.g., adding a persona prefix or chain-of-thought instruction) can shift F1 by several points in either direction with unpredictable effects on computational cost (Figure 11, Figure 12). 4. Sampling hyperparameters: at minimum, temperature, top k, and topp. These settings govern the stochasticity of model output and vary across models (Table 3), but can also be modified by the researcher; omitting them hinders replication. 5. Learning approach: whether zero-shot, few-shot, or some other prompting was used, and if few-shot, the number and selection method of examples. Few-shot effects are strongly model- and task-dependent: some models improve substantially on certain tasks while others degrade (Figure 10). Without this information, neither replication nor comparison across studies is possible. 6. Hardware specification: the GPU model, memory, and any relevant infrastruc- ture details (e.g., inference framework, batch size). Our efficiency measurements are conditioned on specific hardware (NVIDIA A100 GPUs) and are likely to differ on other configurations. 7. Efficiency metrics: aggregate inference time and energy consumption alongside performance metrics. Our results show that pipeline choices with negligible effects on annotation performance can produce order-of-magnitude differences in compu- tational cost (Figure 11, Figure 12). Reporting only performance obscures these trade-offs. This information is necessary both for assessing the replicability of individual studies and for enabling the kind of cross-study comparisons that would allow the field to accumulate generalisable knowledge about LLM annotation pipelines. 39 6 Conclusion This paper uses a controlled benchmark of LLM annotation pipelines to make a method- ological argument about political text analysis: LLM annotation should be treated as a validation problem under researcher degrees of freedom. By varying model choice, model size, prompt style, and zero-shot versus few-shot learning across four annotation tasks, we show that implementation decisions are not mere engineering details. They are part of the measurement strategy itself. The central result is therefore not that one model family wins overall, but that interaction effects between model, task, and pipeline configuration dominate main effects throughout our results. Task-specificity is the rule, not the exception: the best-performing model varies by task, the value of few-shot examples depends on the model and task, and prompt engineering techniques such as persona and chain-of-thought prompting yield unstable effects. Two broader lessons follow. First, no portable set of “best practices” can substitute for task-specific empirical validation. Second, efficiency must be treated as part of methodological design rather than as a secondary implementation concern, because computational cost varies sharply across otherwise plausible pipeline choices. These findings motivate a validation-first framework in which researchers begin from a frozen codebook, benchmark candidate pipelines against human-coded data, and report the design choices that shaped the resulting measurements. Where the goal is direct com- parison between human and LLM coders, the cleanest design is often to give the model the same codebook, including examples, that was given to human annotators. Where prompt or codebook development is iterative, a stricter train/validation/test discipline is needed: few-shot examples should be drawn from a training partition, pipeline choices compared on a validation partition, and final performance reported on a held-out test set. We also propose reporting standards that make these choices transparent and re- lease CodeBook Studio and CodeBook Lab as open-source tools that operationalise this workflow. Three broader implications deserve emphasis. First, our results document relative dif- 40 ferences among pipeline configurations; they do not establish that LLM annotation is superior to fine-tuned encoder models or other established approaches. Whether an LLM pipeline is the right tool for a given task remains an empirical question that depends on the annotation scheme, the available training data, and the computational budget. Second, model-specific rankings will inevitably date as new architectures emerge, but the methodological lesson should not: pipeline heuristics require validation rather than un- critical adoption. Third, because prompt and pipeline decisions can alter measurements in substantively meaningful ways, making those decisions explicit is a prerequisite for cumulative research rather than a matter of descriptive completeness. Our study has several limitations. We evaluate only open-weight models; closed-source systems may exhibit different sensitivity profiles and represent an important benchmark for future work. Our four tasks, while spanning a range of text types and annotation formats, do not exhaust the diversity of political science annotation tasks. Tasks involv- ing longer documents, more granular coding schemes, or multilingual corpora may reveal different patterns, and these tasks deserve our attention. We also report point estimates from a single run of each experimental condition rather than bootstrapped confidence intervals or repeated runs. 2 This leaves residual uncertainty around some differences and should be addressed in future work. All models were evaluated at a single quantisa- tion level on the same hardware, so absolute efficiency estimates may differ under other quantisation settings or computational environments. Finally, we evaluate each pipeline dimension in relative isolation; future work should explore higher-order interaction effects (e.g., CoT prompting combined with few-shot examples, or the joint effect of model size and temperature), test a broader range of models including closed-source systems, and ex- tend the evaluation to tasks with richer annotation structures. Our model-specific results will inevitably date as new architectures emerge, but the broader lesson—that pipeline heuristics require task-specific validation rather than uncritical adoption—should not. As LLM-based annotation becomes routine in political science, the field will benefit from the 2. We considered both bootstrapped confidence intervals and multiple experimental runs to quantify sampling uncertainty and model stochasticity, but the size of the experimental grid and the associated computational cost required prioritising breadth of model and pipeline coverage. 41 same culture of systematic validation that has characterised the best work in computa- tional text analysis. Data and Code Availability Replication materials for this paper, including codebooks, ground-truth annotations, and experimental configuration files, are available at [replication materials URL]. The LLM annotation pipeline used to run all experiments is CodeBook Lab (McLaren 2026a), available at https://github.com/LorcanMcLaren/codebook-lab. Codebook definitions were created using CodeBook Studio (McLaren 2026b), available at https://github.com /LorcanMcLaren/codebook-studio (hosted instance: https://codebook.streamlit.app). A step-by-step tutorial covering the full workflow from codebook design through LLM benchmarking is available at https://lorcanmclaren.com/codebook-tutorial.html. 42 References Alizadeh, Meysam, Ma ̈el Kubli, Zeynab Samei, Shirin Dehghani, Mohammadmasiha Za- hedivafa, Juan D. Bermeo, Maria Korobeynikova, and Fabrizio Gilardi. 2024. “Open- source LLMs for text annotation: a practical guide for model setting and fine-tuning” [in en]. Journal of Computational Social Science 8, no. 1 (December): 17. issn: 2432- 2725, accessed March 27, 2025. https://doi.org/10.1007/s42001-024-00345-9. Argyle, Lisa P., Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. “Out of One, Many: Using Language Models to Simulate Human Samples” [in en]. Political Analysis 31, no. 3 (July): 337–351. issn: 1047- 1987, 1476-4989, accessed February 25, 2025. https://doi.org/10.1017/pan.2023.2. Atreja, Shubham, Joshua Ashkinaze, Lingyao Li, Julia Mendelsohn, and Libby Hemphill. 2024. Prompt Design Matters for Computational Social Science Tasks but in Unpre- dictable Ways [in en]. ArXiv:2406.11980 [cs], June. Accessed April 18, 2025. http: //arxiv.org/abs/2406.11980. Atreja, Shubham, Joshua Ashkinaze, Lingyao Li, Julia Mendelsohn, and Libby Hemphill. 2025. “What’s in a Prompt?: A Large-Scale Experiment to Assess the Impact of Prompt Design on the Compliance and Accuracy of LLM-Generated Text Annota- tions” [in en]. Proceedings of the International AAAI Conference on Web and So- cial Media 19 (June): 122–145. issn: 2334-0770, accessed October 20, 2025. https: //doi.org/10.1609/icwsm.v19i1.35807. Bail, Christopher A. 2024. “Can Generative AI improve social science?” Proceedings of the National Academy of Sciences 121, no. 21 (May): e2314021121. Accessed March 27, 2025. https://doi.org/10.1073/pnas.2314021121. Barber ́a, Pablo, Amber E. Boydstun, Suzanna Linn, Ryan McMahon, and Jonathan Na- gler. 2021. “Automated Text Classification of News Articles: A Practical Guide” [in en]. Political Analysis 29, no. 1 (January): 19–42. issn: 1047-1987, 1476-4989, accessed September 16, 2025. https://doi.org/10.1017/pan.2020.8. Barrie, Christopher, Elli Palaiologou, and Petter T ̈ornberg. 2025. Prompt Stability Scor- ing for Text Annotation with Large Language Models [in en]. ArXiv:2407.02039 [cs], February. Accessed April 3, 2025. http://arxiv.org/abs/2407.02039. Barrie, Christopher, Alexis Palmer, and Arthur Spirling. 2024. Replication for Language Models [in en], December. https://arthurspirling.org/documents/BarriePalmerSpirl ing TrustMeBro.pdf. Baumann, Joachim, Paul R ̈ottger, Aleksandra Urman, Albert Wendsj ̈o, Flor Miriam Plaza-del-Arco, Johannes B. Gruber, and Dirk Hovy. 2025. Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation. ArXiv:2509.08825 [cs], October. Accessed October 21, 2025. http://arxiv.org/abs/2509.08825. Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al. 2020. “Language Models are Few-Shot Learn- ers.” In Advances in Neural Information Processing Systems, 33:1877–1901. Curran Associates, Inc. Accessed April 28, 2024. https://proceedings.neurips.c/paper/202 0/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html. 43 Burnham, Michael, Kayla Kahn, Ryan Yank Wang, and Rachel X. Peng. 2024. Political DEBATE: Efficient Zero-shot and Few-shot Classifiers for Political Text. ArXiv:2409.02078 [cs] version: 1, September. Accessed September 16, 2024. http://arxiv.org/abs/2409 .02078. Cheng, Myra, Esin Durmus, and Dan Jurafsky. 2023. Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models. ArXiv:2305.18189 [cs], May. Accessed April 14, 2025. http://arxiv.org/abs/2305.18189. Courty, Benoit, Victor Schmidt, Sasha Luccioni, Goyal-Kamal, MarionCoutarel, Boris Feld, J ́er ́emy Lecourt, et al. 2024. mlco2/codecarbon: v2.4.1, May. https://doi.org/1 0.5281/zenodo.11171501. https://doi.org/10.5281/zenodo.11171501. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, et al. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. ArXiv:2501.12948 [cs], January. Accessed April 12, 2025. http://arxiv.org/abs/2501.12948. Dettmers, Tim, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. “QLoRA: Efficient Finetuning of Quantized LLMs” [in en]. Advances in Neural Information Processing Systems 36 (December): 10088–10115. Accessed April 18, 2025. https : / / proceedings . neurips . c / paper files / paper / 2023 / hash / 1feb87871436031bdc0f2 beaa62a049b-Abstract-Conference.html. Dong, Qingxiu, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, et al. 2024. A Survey on In-context Learning [in en]. ArXiv:2301.00234 [cs], October. Accessed April 18, 2025. http://arxiv.org/abs/2301.00234. Gebru, Timnit, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daum ́e I, and Kate Crawford. 2021. “Datasheets for datasets.” Com- mun. ACM 64, no. 12 (November): 86–92. issn: 0001-0782, accessed February 27, 2026. https://doi.org/10.1145/3458723. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, et al. 2025. Gemma 3 Technical Report. ArXiv:2503.19786 [cs], March. Accessed November 24, 2025. https://doi.org/10.48550/arXiv.2503.19786. http://arxiv.org/abs/2503.19786. Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning [in en]. Google-Books-ID: omivDQAAQBAJ. MIT Press, November. isbn: 978-0-262-33737- 3. Grattafiori, Aaron, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Ka- dian, Ahmad Al-Dahle, Aiesha Letman, et al. 2024. The Llama 3 Herd of Models. ArXiv:2407.21783 [cs], November. Accessed April 12, 2025. http://arxiv.org/abs/24 07.21783. Grimmer, Justin, Margaret E. Roberts, and Brandon M. Stewart. 2022. Text as Data: A New Framework for Machine Learning and the Social Sciences [in English]. Princeton (N.J.): Princeton University Press, March. isbn: 978-0-691-20755-1. 44 Grimmer, Justin, and Brandon M. Stewart. 2013. “Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts” [in en]. Political Analysis 21, no. 3 (July): 267–297. issn: 1047-1987, 1476-4989, accessed November 8, 2024. https://doi.org/10.1093/pan/mps028. Halterman, Andrew, and Katherine A. Keith. 2025. “Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Concepts” [in en]. Political Analysis (September): 1–17. issn: 1047-1987, 1476-4989, accessed October 10, 2025. https : //doi.org/10.1017/pan.2025.10017. Halterman, Andrew, and Benjamin J. Radford. 2021. “Few-Shot Upsampling for Protest Size Detection.” In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, edited by Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Nav- igli, 3713–3720. Online: Association for Computational Linguistics, August. Accessed February 27, 2026. https://doi.org/10.18653/v1/2021.findings-acl.325. He, Jia, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X. Wang, and Sadid Hasan. 2024. Does Prompt Formatting Have Any Impact on LLM Perfor- mance? [In en]. ArXiv:2411.10541 [cs], November. Accessed April 17, 2025. http : //arxiv.org/abs/2411.10541. Heseltine, Michael, and Bernhard Clemm von Hohenberg. 2024. “Large language models as a substitute for human experts in annotating political text” [in EN]. Research & Politics 11, no. 1 (January): 20531680241236239. issn: 2053-1680, accessed March 27, 2025. https://doi.org/10.1177/20531680241236239. Hoffmann, Jordan, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, et al. 2022. “Training compute-optimal large language models.” In Proceedings of the 36th International Conference on Neural In- formation Processing Systems, 30016–30030. NIPS ’22. Red Hook, NY, USA: Curran Associates Inc., November. isbn: 978-1-7138-7108-8, accessed February 27, 2026. Hu, Tiancheng, and Nigel Collier. 2024. Quantifying the Persona Effect in LLM Simula- tions. ArXiv:2402.10811 [cs], June. Accessed April 14, 2025. http://arxiv.org/abs/2 402.10811. Kaplan, Jared, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Re- won Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models [in en]. ArXiv:2001.08361 [cs], January. Accessed February 27, 2026. http://arxiv.org/abs/2001.08361. Kojima, Takeshi, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwa- sawa. 2022. “Large Language Models are Zero-Shot Reasoners” [in en]. Advances in Neural Information Processing Systems 35 (December): 22199–22213. Accessed April 17, 2025. https://proceedings.neurips.c/paper files/paper/2022/hash/8b0 d291acd4acf06ef112099c16f326-Abstract-Conference.html. Laurer, Moritz, Wouter van Atteveldt, Andreu Casas, and Kasper Welbers. 2024. “Less Annotating, More Classifying: Addressing the Data Scarcity Issue of Supervised Ma- chine Learning with Deep Transfer Learning and BERT-NLI” [in en]. Political Anal- ysis 32, no. 1 (January): 84–100. issn: 1047-1987, 1476-4989, accessed November 8, 2024. https://doi.org/10.1017/pan.2023.20. 45 McLaren, Lorcan. 2025. “Here and Now or There and Then? The Psychological Distance of Climate Change in Parliamentary Speech.” McLaren, Lorcan. 2026a. CodeBook Lab. V. v1.1.0, March. https : / / doi . org / 10 . 5281 /zenodo.19185921. https://github.com/LorcanMcLaren/codebook-lab. McLaren, Lorcan. 2026b. CodeBook Studio. V. v1.1.0, March. https://doi.org/10.5281 /zenodo.19185938. https://github.com/LorcanMcLaren/codebook-studio. Mens, Ga ̈el Le, and Aina Gallego. 2025. “Positioning Political Texts with Large Language Models by Asking and Averaging” [in en]. Political Analysis 33, no. 3 (July): 274– 282. issn: 1047-1987, 1476-4989, accessed October 10, 2025. https://doi.org/10.101 7/pan.2024.29. Mitchell, Margaret, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. “Model Cards for Model Reporting.” In Proceedings of the Conference on Fairness, Account- ability, and Transparency, 220–229. ArXiv:1810.03993 [cs]. January. Accessed Febru- ary 27, 2026. https://doi.org/10.1145/3287560.3287596. OpenAI, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Ed- win Arbus, et al. 2025. gpt-oss-120b & gpt-oss-20b Model Card [in en]. ArXiv:2508.10925 [cs], August. Accessed November 24, 2025. http://arxiv.org/abs/2508.10925. Osnabr ̈ugge, Moritz, Elliott Ash, and Massimo Morelli. 2023. “Cross-Domain Topic Clas- sification for Political Texts” [in en]. Political Analysis 31, no. 1 (January): 59–80. issn: 1047-1987, 1476-4989, accessed September 16, 2025. https://doi.org/10.1017 /pan.2021.37. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, et al. 2025. Qwen2.5 Technical Report. ArXiv:2412.15115 [cs], January. Accessed April 12, 2025. http://arxiv.org/abs/2412.15115. Samsi, Siddharth, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, and Vijay Gadepally. 2023. From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference [in en]. ArXiv:2310.03003 [cs], October. Accessed January 13, 2025. http: //arxiv.org/abs/2310.03003. Savelka, Jaromir, Kevin D. Ashley, Morgan A. Gray, Hannes Westermann, and Huihui Xu. 2023. “Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?” In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1, 117–123. ArXiv:2306.13906 [cs]. June. Accessed April 12, 2025. https://doi.org/10.1145/3587102.3588792. Schick, Timo, and Hinrich Sch ̈utze. 2021. It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners [in en]. ArXiv:2009.07118 [cs], April. Accessed April 12, 2025. http://arxiv.org/abs/2009.07118. Sclar, Melanie, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. ArXiv:2310.11324 [cs], July. Accessed April 4, 2025. http://arxiv.org/abs/2310.11324. 46 Sprague, Zayne, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. 2025. To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning. ArXiv:2409.12183 [cs], May. Accessed March 4, 2026. http://arxiv.org/abs/2409.12 183. Thapa, Surendrabikram, Shuvam Shiwakoti, Siddhant Bikram Shah, Surabhi Adhikari, Hariram Veeramani, Mehwish Nasim, and Usman Naseem. 2025. “Large language models (LLM) in computational social science: prospects, current state, and chal- lenges” [in en]. Social Network Analysis and Mining 15, no. 1 (March): 4. issn: 1869-5469, accessed April 18, 2025. https://doi.org/10.1007/s13278-025-01428-9. Timoneda, Joan C., and Sebasti ́an Vallejo Vera. 2025. Memory Is All You Need: Testing How Model Memory Affects LLM Performance in Annotation Tasks. ArXiv:2503.04874 [cs], March. Accessed February 27, 2026. http://arxiv.org/abs/2503.04874. T ̈ornberg, Petter. 2023. ChatGPT-4 Outperforms Experts and Crowd Workers in Anno- tating Political Twitter Messages with Zero-Shot Learning. ArXiv:2304.06588 [cs], April. Accessed March 27, 2025. http://arxiv.org/abs/2304.06588. T ̈ornberg, Petter. 2024a. “Best Practices for Text Annotation with Large Language Mod- els” [in en]. Number: 2, Sociologica 18, no. 2 (October): 67–85. issn: 1971-8853, accessed April 18, 2025. https://doi.org/10.6092/issn.1971-8853/19461. T ̈ornberg, Petter. 2024b. “Large Language Models Outperform Expert Coders and Su- pervised Classifiers at Annotating Political Social Media Messages” [in EN]. Social Science Computer Review (September): 08944393241286471. issn: 0894-4393, ac- cessed March 27, 2025. https://doi.org/10.1177/08944393241286471. Turpin, Miles, Julian Michael, Ethan Perez, and Samuel Bowman. 2023. “Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain- of-Thought Prompting” [in en]. Advances in Neural Information Processing Systems 36 (December): 74952–74965. Accessed April 4, 2025. https://proceedings.neurips. c / paper files / paper / 2023 / hash / ed3fea9033a80fea1376299fa7863f4a - Abstract - Conference.html. Wang, Taihang, Xiaoman Xu, Yimin Wang, and Ye Jiang. 2024. Instruction Tuning Vs. In-Context Learning: Revisiting Large Language Models in Few-Shot Computational Social Science. ArXiv:2409.14673 [cs], September. Accessed February 27, 2026. http: //arxiv.org/abs/2409.14673. Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. “Chain-of-Thought Prompting Elic- its Reasoning in Large Language Models” [in en]. Advances in Neural Information Processing Systems 35 (December): 24824–24837. Accessed April 4, 2025. https : //proceedings.neurips.c/paper files/paper/2022/hash/9d5609613524ecf4f15af0f7 b31abca4-Abstract-Conference.html?ref=https://githubhelp.com. Wei, Jerry, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, et al. 2023. Larger language models do in-context learning differently. ArXiv:2303.03846 [cs], March. Accessed April 14, 2025. http://arxiv.org/abs/2303.03846. 47 Wen, Haoyang, and Alexander Hauptmann. 2023. “Zero-Shot and Few-Shot Stance De- tection on Varied Topics via Conditional Generation.” In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), edited by Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, 1491– 1499. Toronto, Canada: Association for Computational Linguistics, July. Accessed February 27, 2026. https://doi.org/10.18653/v1/2023.acl-short.127. Wratil, Christopher, and Sara B Hobolt. 2019. “Public deliberations in the Council of the European Union: Introducing and validating DICEU” [in EN]. European Union Politics 20, no. 3 (September): 511–531. issn: 1465-1165, accessed April 12, 2025. https://doi.org/10.1177/1465116519839152. Yang, An, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, et al. 2025. Qwen3 Technical Report. ArXiv:2505.09388 [cs], May. Accessed Novem- ber 24, 2025. http://arxiv.org/abs/2505.09388. Zhang, Xiang, Juntai Cao, Jiaqi Wei, Chenyu You, and Dujian Ding. 2025. Why Does Your CoT Prompt (Not) Work? Theoretical Analysis of Prompt Space Complexity, its Interaction with Answer Space During CoT Reasoning with LLMs: A Recurrent Perspective. ArXiv:2503.10084 [cs], March. Accessed April 15, 2025. http://arxiv. org/abs/2503.10084. Zheng, Mingqian, Jiaxin Pei, Lajanugen Logeswaran, Moontae Lee, and David Jur- gens. 2024. “When ”A Helpful Assistant” Is Not Really Helpful: Personas in Sys- tem Prompts Do Not Improve Performances of Large Language Models.” In Find- ings of the Association for Computational Linguistics: EMNLP 2024, edited by Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, 15126–15154. Miami, Florida, USA: Association for Computational Linguistics, November. Accessed April 14, 2025. https://doi.org/10.18653/v1/2024.findings-emnlp.888. Ziems, Caleb, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can Large Language Models Transform Computational Social Science? [In en]. ArXiv:2305.03514 [cs], April. Accessed July 24, 2023. http://arxiv.org/abs/2305.03 514. 48 A Codebooks A.1 Approval General approval refers to the degree to which the speaker expresses approval of the proposal(s) being negotiated or the state of negotiations. In orientation debates, this dimension will typically be approval of the legislative proposal presented by the European Commission. Later on in the negotiation process, this dimension will typically be approval of the presidency’s suggestions. If the speaker presents a most preferred position and one on which they could compromise, always code the preferred position. Assess the degree of approval expressed in the speech below. • Question: What level of approval is expressed in the speech? • Options: Rate on the following scale: 1. The speaker expresses full approval. 2. The speaker expresses more approval than disapproval. 3. The speaker expresses a balance of approval and disapproval. 4. The speaker expresses more disapproval than approval. 5. The speaker expresses full disapproval. If the speaker conveys their position but the exact level of approval is hard to assess, choose 3. • Examples: – Code ‘1’ (Full approval): ∗ “Thank you Mr. Chairman, we congratulate the Presidency for the good compromise proposal just on the table, we fully support it and can accept it. Regarding the comitology procedure on Article 24 we’re part of the declaration together with the United Kingdom, Malta and Ireland. We’re ready and we accept the present proposal.” 1 – Code ‘2’ (More approval than disapproval): ∗ “Thank you Mr. President, just one word to say that we fully support the position presented by Malta 100%, thank you.” – Code ‘3’ (Balance of approval and disapproval): ∗ “Thank you for the floor. The Czech Republic is also very much interested in finding the good compromise. However we think we should maybe work one more month in the effort to try a full compromise, we share the opinion of the Dutch delegation and the Swedish delegation. Thank you.” – Code ‘4’ (More disapproval than approval): ∗ “I recognize the Presidency’s effort in finding compromise, but I am con- cerned about the unleveled playing field it introduces since some Member States will exchange a lot of information while other Member States will exchange only a little information. Therefore I agree with the Dutch pro- posal but at the same time we would accept the compromise if it were the only way to reach an agreement.” – Code ‘5’ (Full disapproval): ∗ “Thank you, we are all currently engaged in consolidating our national economies and cutting expenses on the national budgets and therefore it seems out of touch with the economic realities with Member States that the European Commission while recommending budget cuts in Member States at the same time asks for additional monetary resources for itself.” 2 A.2 Psychological Distance Respond to items 2-3 in this section only if you select ‘Yes’ for item 1 (Presence). Other- wise, leave these blank. You may also choose to leave item 1 (Presence) blank; this will be interpreted as a ‘No’ response. Respond to item 3 (Proximity) only if you select ‘Specific’ for item 2 (Specificity). A.2.1 Presence • Question: Does this text discuss the effects of climate change, pollution, or envi- ronmental destruction, defined as the adverse effects on natural ecosystems, human health, and the planet as a whole, caused by human activities? Environmental impacts are not equivalent to mitigation policy impacts (the effects – both posi- tive and negative – that arise from the implementation of laws, regulations, and initiatives aimed at managing and protecting the environment). • Options: Yes, No • Examples: – Positive examples (code ‘Yes’): ∗ “The threat that carbon emissions oppose to our way of society by in- ducing large scale climate change is a problem we must face up to very quickly.” ∗ “But we still have major problems: in many cities, air quality is badly polluted and people suffer from fine particulate emissions.” – Negative examples (code ‘No’): ∗ “The implementation of the Paris Agreement will also play an important role in terms of limiting the increase in global average temperature to below 2°C pre-industrial levels.” ∗ “On the one hand, the European fleet needs access to new fishing grounds, 3 particularly to be able to track the migration of certain fish species, no- tably tuna.” A.2.2 Specificity • Question: Are the environmental impacts described as having specific settings (impacting specifically mentioned individuals, groups, industries, or regions) or universal settings (impacting humanity as a whole or other lifeforms)? If in doubt or no clear explicit or implicit setting, code ‘Universal’. • Options: Specific, Universal • Examples: – Code ‘Specific’: ∗ “One hundred billion plastic bags are consumed in Europe every year, and their excessive use has a disastrous impact on the environment.” ∗ “Just tonight, in Romania, one of the most beautiful waterfalls, the Big ̆ar waterfall, collapsed under human action.” ∗ “In the first half of 2022 alone, 3,750 square kilometres of rainforest have been destroyed – gone forever.” – Code ‘Universal’: ∗ “However, it is a group of substances that are suspected of impairing liver and thyroid function and causing cancer.” ∗ “In the near future, this scenario will be common, with fires recurring, increasing in scale, intensity and frequency.” A.2.3 Proximity • Question: Are the environmental impacts portrayed as proximate or distant? Proximate means impacting EU nationals, EU member states, or specific com- 4 munities within the bloc, currently or in the near future. Distant means impacting non-EU states or communities, or future generations. • Options: Proximate, Distant • Examples: – Code ‘Proximate’: ∗ “According to a Commission impact assessment, the health costs caused by air pollution across Europe amount to around EUR 940 million per year.” ∗ “Indeed, 84% of Europeans have expressed their fears regarding the impact of chemicals on their health.” – Code ‘Distant’: ∗ “In the first half of 2022 alone, 3,750 square kilometres of rainforest have been destroyed – gone forever.” ∗ “This illegal form of fishing has emerged, particularly since the civil war, and has an adverse impact not only from an environmental point of view, but also financially damages Liberian fishermen and the state budget.” 5 A.3 Economic Sentiment Read the newspaper article excerpt and assess what indication it gives about how the US economy is performing. Rate the sentiment as either negative or positive only. If the text contains typos or is corrupted, do your best to interpret the intended message. A.3.1 Positivity • Question: Based on the indication the article gives about how the US economy is performing, rate whether the indication is negative or positive. Consider the overall tone and content regarding US economic performance mentioned in the excerpt. • Options: Negative, Positive • Examples: – Code ‘Negative’: ∗ “STAGNANT WAGES POSE ADDED RISKS TO WEAK ECONOMY: CONSUMERS FEEL A PINCH Bigger Deductions for Medical Coverage and Less Overtime Put Dent in Paychecks Stagnant Wages Add to Risk for Economy. Although the recession has ended, the wages of more than 100 million workers are still stagnant, endangering the consumer spending that sustains the fragile recovery [. . . ]” ∗ “Trade Deficit Widened by Oil Imports: Trade Deficit Widened by Oil Imports. WASHINGTON, March 20 — The trade deficit widened to $9.3 billion in January as oil imports surged to refill stocks depleted during December’s record cold [. . . ]” – Code ‘Positive’: ∗ “Market Undergoes Mild Consolidation As Economy Rises: THE WEEK IN FINANCE. ALMOST all the background news last week was decidedly constructive, but the stock market failed to take heart from it and instead endured a mild period of consolidation [. . . ]” 6 ∗ “Investors Regain Optimism as Crude Oil Prices Decline: Key Rates. Stocks climbed yesterday as a drop in crude oil prices from their peak eased investors’ worries that high fuel costs would crimp consumer spending and hurt company profits [. . . ]” 7 A.4 Manifesto Topic Classify each text into one of the domains below. A.4.1 Domain • Question: Select the domain that best matches the content of the text. • Options: – No Domain – External Relations: Foreign Special Relationships, Anti-Imperialism, Mili- tary, Peace, Internationalism, European Community/Union – Freedom And Democracy: Freedom And Human Rights, Democracy, Con- stitutionalism – Political System: Decentralization, Governmental And Administrative Effi- ciency, Political Corruption, Political Authority – Economy: Free Market Economy, Incentives, Market Regulation, Economic Planning, Corporatism, Protectionism, Economic Goals, Keynesian Demand Management, Economic Growth, Technology And Infrastructure, Controlled Economy, Nationalization, Economic Orthodoxy, Marxist Analysis, Anti-Growth Economy – Welfare And Quality Of Life: Environmental Protection, Culture, Equal- ity, Welfare State Expansion, Welfare State Limitation, Education Expansion, Education Limitation – Fabric Of Society: National Way Of Life, Traditional Morality, Law And Order, Civic Mindedness, Multiculturalism – Social Groups: Labour Groups, Agriculture And Farmers, Middle Class And Professional Groups, Underprivileged Minority Groups, Non-economic Demographic Groups 8 • Examples: – Code ‘External Relations’: ∗ “The United States government’s policy is that there is one China, as reflected in the three communiqu ́es and the Taiwan Relations Act.” ∗ “and Iraqis allowed their self-determination.” ∗ “Roaming Charges within the EU: Fine Gael strongly supports the prin- ciple that roaming charges should be harmonised across the EU and we will work in Europe to speed up progress in this regard.” – Code ‘Freedom And Democracy’: ∗ “The rights of citizenship do not stop at the ballot box.” ∗ “These goals require that we build a democratic developmental state ca- pable of mobilising all sectors and boldly intervening in the economy in favour of workers and the poor.” ∗ “You can vote to retain the old First Past the Post system, or you can vote for the Mixed Member Proportional system - MMP.” – Code ‘Political System’: ∗ “allow no major reorganisation of local government in our first term in office.” ∗ “The restructured Regional Development Australia network will navigate federal government funding programs, according to our fair share commit- ment, including a restructured Better Regions Program.” ∗ “Unlike Labour, I’m not prepared to mislead the New Zealand public about the situation this country finds itself in.” – Code ‘Economy’: ∗ “who maintain their homes well and have lower maintenance costs as a 9 result.” ∗ “Develop a national plan to address the impacts of climate change on rural communities and regional industries.” ∗ “Moreover, the inflation tax is regressive.” ∗ “The measures we propose as alternatives to austerity will, by halting and reversing the cuts to public services, restore lost jobs and create new ones.” ∗ “Our plan can be entirely implemented within the ten-year transport bud- get set out in the draft 2015-2025 Government Policy Statement on Trans- port Funding.” – Code ‘Welfare And Quality Of Life’: ∗ “Meanwhile, to help the poorest students now, we will immediately restore maintenance grants.” ∗ “At present New Zealanders are amongst the highest per capita emitters in the world.” ∗ “This new funding will not impact those students hoping to study in the humanities.” – Code ‘Fabric Of Society’: ∗ “and give Welsh speakers the right to use their language at all levels and for its status to be internationally recognised.” ∗ “Every percentage drop represents an assault on the ties that bind us together.” ∗ “Since then we have seen our nation build on a diverse heritage of cultures that has contributed to our proud nation.” ∗ “The referendum on marriage equality was a historic victory for the rights 10 of gay and lesbian people.” – Code ‘Social Groups’: ∗ “It also means providing support that has been proven to work, like work experience placements that help them get a first foot on the career ladder.” ∗ “We will reduce the tax burden for those on low and middle incomes Income tax and USC.” ∗ “Women in Politics: Fine Gael recognises that there needs to be a sub- stantial increase in the number of women in politics.” 11 B Performance Metrics Tables B.1 Model Choice Table B.1: Task-level performance metrics for the model choice comparison, reporting F1, accuracy, precision, recall, Cohen’s κ, and Krippendorff’s α for each model. ModelTaskF1 Accuracy Precision Recallκα DeepSeek-R1 approval0.3570.4140.4760.355 0.225 0.538 DeepSeek-R1 psych-dist 0.3930.9480.4150.376 0.643 0.656 DeepSeek-R1 sentiment0.7350.7430.7320.741 0.471 0.470 DeepSeek-R1 topic-80.2340.4960.2820.248 0.378 0.385 GPT-OSSapproval0.4270.4770.4810.418 0.285 0.571 GPT-OSSpsych-dist 0.6990.9350.7260.681 0.547 0.564 GPT-OSSsentiment0.7000.7190.7030.698 0.400 0.401 GPT-OSStopic-80.3590.4960.4160.391 0.385 0.447 Gemma 3approval0.3780.4130.5110.377 0.234 0.562 Gemma 3psych-dist 0.6780.8930.6380.769 0.500 0.499 Gemma 3sentiment0.7140.7290.7140.714 0.427 0.428 Gemma 3topic-80.1690.4480.2250.177 0.327 0.322 LLaMA 3.1approval0.4340.4730.5050.422 0.280 0.606 LLaMA 3.1psych-dist 0.4900.8480.4540.629 0.434 0.333 LLaMA 3.1sentiment0.4880.7400.4870.491 0.466 0.467 LLaMA 3.1topic-80.1460.4310.1880.157 0.321 0.344 Qwen 2.5approval0.3800.4030.4990.382 0.209 0.576 Qwen 2.5psych-dist 0.6870.9410.8120.626 0.516 0.528 Qwen 2.5sentiment0.3650.7330.3660.368 0.461 0.460 Qwen 2.5topic-80.1750.4130.2230.174 0.313 0.392 Qwen 3approval0.4190.4610.4750.408 0.268 0.560 Qwen 3psych-dist 0.7200.9250.6930.758 0.566 0.576 Qwen 3sentiment0.7210.7260.7220.733 0.447 0.443 Qwen 3topic-80.3040.5070.3470.322 0.394 0.439 B.2 Model Size Table B.2: Task-level performance metrics for the model size comparison, reporting F1, accuracy, precision, recall, Cohen’s κ, and Krippendorff’s α for each model variant. ModelTaskF1 Accuracy Precision Recallκα Deepseek-R1 1.5B approval0.1770.2190.2170.2200.024 -0.008 Deepseek-R1 1.5B psych-dist 0.0480.7790.0500.0470.0660.031 Deepseek-R1 1.5B sentiment0.0150.4860.0180.0130.183 -0.056 Deepseek-R1 1.5B topic-80.0020.1530.0040.0030.0710.025 12 Deepseek-R1 8Bapproval0.3350.4160.4350.3380.2160.503 Deepseek-R1 8Bpsych-dist 0.3050.9080.3220.3010.4370.411 Deepseek-R1 8Bsentiment0.6790.7000.6820.6770.3580.358 Deepseek-R1 8Btopic-80.2920.4810.3280.3120.3690.362 Deepseek-R1 14Bapproval0.3740.4140.4930.3610.2160.500 Deepseek-R1 14Bpsych-dist 0.2370.9270.2590.2280.5180.522 Deepseek-R1 14Bsentiment0.0690.6860.0700.0680.3770.048 Deepseek-R1 14Btopic-80.0450.4550.0530.0470.3470.528 Deepseek-R1 32Bapproval0.3960.4500.4530.3850.2560.587 Deepseek-R1 32Bpsych-dist 0.2900.9290.2990.2830.5580.525 Deepseek-R1 32Bsentiment0.1790.7240.1800.1790.4340.254 Deepseek-R1 32Btopic-80.0490.4840.0570.0520.3770.553 Deepseek-R1 70Bapproval0.3570.4140.4760.3550.2250.538 Deepseek-R1 70Bpsych-dist 0.3930.9480.4150.3760.6430.656 Deepseek-R1 70Bsentiment0.7350.7430.7320.7410.4710.470 Deepseek-R1 70Btopic-80.2340.4960.2820.2480.3780.385 Gemma3 270Mapproval0.1520.2590.1640.203 -0.0080.023 Gemma3 270Mpsych-dist 0.0630.5280.0640.0640.001 -0.152 Gemma3 270Msentiment0.0540.2760.1290.0420.0370.078 Gemma3 270Mtopic-80.0000.0030.0000.0000.000 -0.003 Gemma3 1Bapproval0.1180.1690.1580.2140.005 -0.265 Gemma3 1Bpsych-dist 0.2130.6150.2050.2220.000 -0.097 Gemma3 1Bsentiment0.2820.5240.3370.2460.1940.318 Gemma3 1Btopic-80.0160.0880.0470.0220.017 -0.038 Gemma3 4Bapproval0.2970.3690.3640.3080.1620.380 Gemma3 4Bpsych-dist 0.5680.8440.5350.7090.3810.395 Gemma3 4Bsentiment0.6890.7100.6930.6870.3780.379 Gemma3 4Btopic-80.0650.3510.1100.0820.1820.287 Gemma3 12Bapproval0.3860.4530.4860.3750.2630.588 Gemma3 12Bpsych-dist 0.6550.8860.6170.7410.4660.466 Gemma3 12Bsentiment0.7090.7360.7240.7030.4220.419 Gemma3 12Btopic-80.2590.4550.3510.2910.3370.345 Gemma3 27Bapproval0.3780.4130.5110.3770.2340.562 Gemma3 27Bpsych-dist 0.6780.8930.6380.7690.5000.499 Gemma3 27Bsentiment0.7140.7290.7140.7140.4270.428 Gemma3 27Btopic-80.1690.4480.2250.1770.3270.322 Qwen3 600Mapproval0.1330.2150.1920.2290.039 -0.021 Qwen3 600Mpsych-dist 0.1250.7070.1310.1530.1560.098 Qwen3 600Msentiment0.4130.6190.4530.4440.2940.239 Qwen3 600Mtopic-80.0580.2510.0920.0800.1390.117 Qwen3 1.7Bapproval0.2150.2930.2690.2790.0930.172 Qwen3 1.7Bpsych-dist 0.5940.9020.5820.6210.4740.459 Qwen3 1.7Bsentiment0.6880.6900.6960.7070.3880.377 Qwen3 1.7Btopic-80.1690.3080.2300.1740.1980.134 Qwen3 4Bapproval0.3820.4380.4180.3770.2330.406 Qwen3 4Bpsych-dist 0.6520.9360.7010.6200.5300.522 Qwen3 4Bsentiment0.2920.7380.2930.2900.4540.437 Qwen3 4Btopic-80.1470.5050.1660.1580.3960.411 13 Qwen3 8Bapproval0.3490.4400.4620.3460.2340.533 Qwen3 8Bpsych-dist 0.5990.9300.6540.5680.4930.492 Qwen3 8Bsentiment0.7010.7100.6990.7070.4030.402 Qwen3 8Btopic-80.3010.4500.3610.3250.3410.368 Qwen3 14Bapproval0.4350.4660.5210.4140.2750.560 Qwen3 14Bpsych-dist 0.4230.9370.4700.3970.5230.480 Qwen3 14Bsentiment0.7170.7330.7180.7160.4350.435 Qwen3 14Btopic-80.2990.4690.3580.3020.3630.446 Qwen3 32Bapproval0.4190.4610.4750.4080.2680.560 Qwen3 32Bpsych-dist 0.7200.9250.6930.7580.5660.576 Qwen3 32Bsentiment0.7210.7260.7220.7330.4470.443 Qwen3 32Btopic-80.3040.5070.3470.3220.3940.439 B.3 Zero-Shot vs. Few-Shot Table B.3: Task-level performance metrics for the zero-shot and few-shot comparison, reporting F1, accuracy, precision, recall, Cohen’s κ, and Krippendorff’s α by model and learning approach. Model Learning Approach TaskF1 Accuracy Precision Recallκα DeepSeek-R1 Few-Shotapproval0.3790.3910.4770.375 0.216 0.532 DeepSeek-R1 Few-Shotpsych-dist 0.6770.9370.6460.719 0.640 0.649 DeepSeek-R1 Few-Shotsentiment0.7130.7290.7130.712 0.426 0.427 DeepSeek-R1 Few-Shottopic-80.1630.4650.2120.181 0.334 0.366 DeepSeek-R1 Zero-Shot approval0.3570.4140.4760.355 0.225 0.538 DeepSeek-R1 Zero-Shot psych-dist 0.3930.9480.4150.376 0.643 0.656 DeepSeek-R1 Zero-Shot sentiment0.7350.7430.7320.741 0.471 0.470 DeepSeek-R1 Zero-Shot topic-80.2340.4960.2820.248 0.378 0.385 GPT-OSSFew-Shotapproval0.4240.4530.4530.417 0.267 0.582 GPT-OSSFew-Shotpsych-dist 0.5510.9140.5220.595 0.532 0.474 GPT-OSSFew-Shotsentiment0.7120.7290.7130.710 0.423 0.424 GPT-OSSFew-Shottopic-80.3570.4820.4410.387 0.367 0.416 GPT-OSSZero-Shot approval0.4270.4770.4810.418 0.285 0.571 GPT-OSSZero-Shot psych-dist 0.6990.9350.7260.681 0.547 0.564 GPT-OSSZero-Shot sentiment0.7000.7190.7030.698 0.400 0.401 GPT-OSSZero-Shot topic-80.3590.4960.4160.391 0.385 0.447 Gemma 3Few-Shotapproval0.4220.4120.5040.420 0.244 0.571 Gemma 3Few-Shotpsych-dist 0.6170.8350.5740.807 0.394 0.395 Gemma 3Few-Shotsentiment0.7030.7190.7030.702 0.406 0.406 Gemma 3Few-Shottopic-80.1780.3780.3010.207 0.253 0.288 Gemma 3Zero-Shot approval0.3780.4130.5110.377 0.234 0.562 Gemma 3Zero-Shot psych-dist 0.6780.8930.6380.769 0.500 0.499 Gemma 3Zero-Shot sentiment0.7140.7290.7140.714 0.427 0.428 Gemma 3Zero-Shot topic-80.1690.4480.2250.177 0.327 0.322 14 LLaMA 3.1Few-Shotapproval0.4610.4840.4980.459 0.309 0.652 LLaMA 3.1Few-Shotpsych-dist 0.5720.7870.5430.809 0.335 0.327 LLaMA 3.1Few-Shotsentiment0.4740.7170.4750.475 0.421 0.427 LLaMA 3.1Few-Shottopic-80.0980.4150.1240.111 0.313 0.335 LLaMA 3.1Zero-Shot approval0.4340.4730.5050.422 0.280 0.606 LLaMA 3.1Zero-Shot psych-dist 0.4900.8480.4540.629 0.434 0.333 LLaMA 3.1Zero-Shot sentiment0.4880.7400.4870.491 0.466 0.467 LLaMA 3.1Zero-Shot topic-80.1460.4310.1880.157 0.321 0.344 Qwen 2.5Few-Shotapproval0.4240.4000.5480.425 0.222 0.592 Qwen 2.5Few-Shotpsych-dist 0.7360.9260.6990.788 0.588 0.596 Qwen 2.5Few-Shotsentiment0.2860.7100.2910.281 0.422 0.417 Qwen 2.5Few-Shottopic-80.2000.3860.2760.205 0.290 0.425 Qwen 2.5Zero-Shot approval0.3800.4030.4990.382 0.209 0.576 Qwen 2.5Zero-Shot psych-dist 0.6870.9410.8120.626 0.516 0.528 Qwen 2.5Zero-Shot sentiment0.3650.7330.3660.368 0.461 0.460 Qwen 2.5Zero-Shot topic-80.1750.4130.2230.174 0.313 0.392 Qwen 3Few-Shotapproval0.4390.4560.4560.433 0.277 0.612 Qwen 3Few-Shotpsych-dist 0.7320.9350.7500.717 0.576 0.577 Qwen 3Few-Shotsentiment0.7230.7360.7220.725 0.447 0.447 Qwen 3Few-Shottopic-80.1600.4880.1910.173 0.370 0.394 Qwen 3Zero-Shot approval0.4190.4610.4750.408 0.268 0.560 Qwen 3Zero-Shot psych-dist 0.7200.9250.6930.758 0.566 0.576 Qwen 3Zero-Shot sentiment0.7210.7260.7220.733 0.447 0.443 Qwen 3Zero-Shot topic-80.3040.5070.3470.322 0.394 0.439 B.4 Prompt Style Table B.4: Task-level performance metrics for the prompt style comparison, reporting F1, accuracy, precision, recall, Cohen’s κ, and Krippendorff’s α by model and prompt style. Model Prompt StyleTaskF1 Accuracy Precision Recallκα DeepSeek-R1 CoTapproval0.4290.4330.5200.418 0.261 0.563 DeepSeek-R1 CoTpsych-dist 0.6260.9050.5940.678 0.516 0.499 DeepSeek-R1 CoTsentiment0.4740.7190.4790.469 0.418 0.401 DeepSeek-R1 CoTtopic-80.1660.4680.2150.181 0.337 0.465 DeepSeek-R1 personaapproval0.3890.3940.4620.384 0.215 0.525 DeepSeek-R1 personapsych-dist 0.7440.9350.7160.783 0.609 0.634 DeepSeek-R1 personasentiment0.7170.7310.7160.718 0.434 0.435 DeepSeek-R1 personatopic-80.1490.4590.2010.162 0.327 0.460 DeepSeek-R1 standard approval0.3790.3910.4770.375 0.216 0.532 DeepSeek-R1 standard psych-dist 0.6770.9370.6460.719 0.640 0.649 DeepSeek-R1 standard sentiment0.7130.7290.7130.712 0.426 0.427 DeepSeek-R1 standard topic-80.1630.4650.2120.181 0.334 0.366 15 GPT-OSSCoTapproval0.4470.4560.4850.430 0.269 0.597 GPT-OSSCoTpsych-dist 0.6160.9230.5940.649 0.548 0.559 GPT-OSSCoTsentiment0.7180.7360.7210.716 0.436 0.437 GPT-OSSCoTtopic-80.3630.4970.4290.391 0.381 0.419 GPT-OSSpersonaapproval0.4480.4680.4640.447 0.286 0.597 GPT-OSSpersonapsych-dist 0.6370.9320.6220.666 0.583 0.591 GPT-OSSpersonasentiment0.6890.7100.6930.687 0.378 0.379 GPT-OSSpersonatopic-80.3570.4780.4390.385 0.361 0.409 GPT-OSSstandard approval0.4240.4530.4530.417 0.267 0.582 GPT-OSSstandard psych-dist 0.5510.9140.5220.595 0.532 0.474 GPT-OSSstandard sentiment0.7120.7290.7130.710 0.423 0.424 GPT-OSSstandard topic-80.3570.4820.4410.387 0.367 0.416 Gemma 3CoTapproval0.4240.4290.4470.426 0.257 0.578 Gemma 3CoTpsych-dist 0.5670.7860.5390.773 0.316 0.301 Gemma 3CoTsentiment0.6920.7120.6950.690 0.384 0.385 Gemma 3CoTtopic-80.2140.4220.3090.239 0.298 0.359 Gemma 3personaapproval0.4180.4130.4960.418 0.244 0.573 Gemma 3personapsych-dist 0.6330.8460.5830.816 0.417 0.411 Gemma 3personasentiment0.7100.7240.7090.711 0.420 0.420 Gemma 3personatopic-80.2060.3900.3420.237 0.251 0.256 Gemma 3standard approval0.4220.4120.5040.420 0.244 0.571 Gemma 3standard psych-dist 0.6170.8350.5740.807 0.394 0.395 Gemma 3standard sentiment0.7030.7190.7030.702 0.406 0.406 Gemma 3standard topic-80.1780.3780.3010.207 0.253 0.288 LLaMA 3.1CoTapproval0.3950.4630.4500.382 0.275 0.599 LLaMA 3.1CoTpsych-dist 0.6440.8630.5940.797 0.437 0.446 LLaMA 3.1CoTsentiment0.7260.7330.7240.735 0.455 0.453 LLaMA 3.1CoTtopic-80.2470.4860.3020.265 0.367 0.403 LLaMA 3.1personaapproval0.4710.4970.5040.470 0.325 0.633 LLaMA 3.1personapsych-dist 0.2310.7060.2360.336 0.246 0.185 LLaMA 3.1personasentiment0.3630.7310.3640.362 0.448 0.461 LLaMA 3.1personatopic-80.1210.4080.1640.140 0.297 0.364 LLaMA 3.1standard approval0.4610.4840.4980.459 0.309 0.652 LLaMA 3.1standard psych-dist 0.5720.7870.5430.809 0.335 0.327 LLaMA 3.1standard sentiment0.4740.7170.4750.475 0.421 0.427 LLaMA 3.1standard topic-80.0980.4150.1240.111 0.313 0.335 Qwen 2.5CoTapproval0.4360.4170.5210.437 0.241 0.598 Qwen 2.5CoTpsych-dist 0.7490.9320.7290.774 0.606 0.610 Qwen 2.5CoTsentiment0.3650.7240.3700.361 0.452 0.477 Qwen 2.5CoTtopic-80.2040.4440.2600.212 0.328 0.368 Qwen 2.5personaapproval0.4290.4150.5230.430 0.240 0.619 Qwen 2.5personapsych-dist 0.7410.9280.7050.790 0.598 0.604 Qwen 2.5personasentiment0.3640.7190.3740.355 0.446 0.488 Qwen 2.5personatopic-80.1780.4140.2430.182 0.305 0.390 Qwen 2.5standard approval0.4240.4000.5480.425 0.222 0.592 Qwen 2.5standard psych-dist 0.7360.9260.6990.788 0.588 0.596 Qwen 2.5standard sentiment0.2860.7100.2910.281 0.422 0.417 Qwen 2.5standard topic-80.2000.3860.2760.205 0.290 0.425 16 Qwen 3CoTapproval0.4800.4960.5030.465 0.322 0.638 Qwen 3CoTpsych-dist 0.7350.9290.7180.756 0.576 0.573 Qwen 3CoTsentiment0.7410.7520.7390.743 0.482 0.483 Qwen 3CoTtopic-80.1520.4840.1790.163 0.361 0.446 Qwen 3personaapproval0.4610.4760.4700.457 0.302 0.637 Qwen 3personapsych-dist 0.7240.9290.7160.734 0.558 0.553 Qwen 3personasentiment0.7460.7550.7430.751 0.492 0.492 Qwen 3personatopic-80.1620.4840.2000.172 0.363 0.458 Qwen 3standard approval0.4390.4560.4560.433 0.277 0.612 Qwen 3standard psych-dist 0.7320.9350.7500.717 0.576 0.577 Qwen 3standard sentiment0.7230.7360.7220.725 0.447 0.447 Qwen 3standard topic-80.1600.4880.1910.173 0.370 0.394 17

Instant research discovery