Paper deep dive
Low-Cost Assays for Measuring Model Behavior Across Vendors and Releases
Tapan Parikh
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/26/2026, 2:43:43 AM
Summary
The paper introduces a low-cost, scalable framework for measuring language model behavior across different vendors and releases using frozen public stimuli. It employs three analysis methods (clamped exact match, LLM-coded codebooks, and instrumented environments) to assess model responses in areas such as word choice convergence, resistance to sycophancy, conduct under social pressure, and coding agent accountability. Key findings include model convergence on specific words, a generational shift from sycophancy to resistance, vendor-specific behavioral signatures, and discrepancies between agent actions and reported statements.
Entities (21)
Relation Signals (18)
Low-Cost Assays for Measuring Model Behavior Across Vendors and Releases → hasauthor → Tapan Parikh
confidence 100% · Title: Low-Cost Assays... Author: Tapan Parikh
Low-Cost Assays for Measuring Model Behavior Across Vendors and Releases → proposesmethod → Instrumented Analysis
confidence 95% · an instrumented environment that records what an agent did
Low-Cost Assays for Measuring Model Behavior Across Vendors and Releases → proposesmethod → Clamped Analysis
confidence 95% · Each reads its transcripts one of three ways... exact match on a clamped reply
Low-Cost Assays for Measuring Model Behavior Across Vendors and Releases → proposesmethod → Coded Analysis
confidence 95% · a codebook applied by LLM judges
Tapan Parikh → affiliatedwith → Cornell Tech
confidence 90% · Affiliation: Cornell Tech
Anthropic → exhibitsbehavior → House
confidence 90% · Anthropic’s models name the user’s feeling, warn of the consequence, and offer an alternative
Google → exhibitsbehavior → House
confidence 90% · Google’s models cite their own nature or rules at three times the panel rate
Meta → exhibitsbehavior → House
confidence 90% · Meta’s models question the plan more than the panel does
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Language models advise people, keep them company, and write software while they sleep. Measuring what they do is hard: behavior has to be sampled repeatedly across models, prompts and releases, most of it lives in unstructured text that has to be coded before it can be counted, and the result has to be legible and rigorous enough to meaningfully compare models and vendors. To address these constraints, we present a simple, cheap, scalable, and replicable model for studying model behavior. Each study is a frozen, public stimulus run identically on a cross-vendor panel, at a few dollars per model or less. Each reads its transcripts one of three ways, chosen by how much interpretation the behavior needs: exact match on a clamped reply, a codebook applied by LLM judges whose agreement with a human coder is reported per code, and an instrumented environment that records what an agent did independently of what it said. Run across four years of model releases from both frontier and open-source labs, these instruments find four things. Convergence: asked to pick a word, 27 of 44 models answer serendipity at least once in four tries. Resistance: a trailing "right?" moves endorsement by up to 32 points, and the sign flips from sycophantic to resistant as generations advance, keyed to the tag's surface form. House: whether a model holds a position under pressure tracks its generation, and how it holds tracks the lab that built it. Account: told to do something the documentation in their repository contradicts, some coding agents never went along silently and others always did, and the same model can change with the harness it runs in. Re-run on every release, batteries like these track how behavior is changing across vendors and over time.
Tags
Links
- Source: https://arxiv.org/abs/2609.30012v1
- Canonical: https://arxiv.org/abs/2609.30012v1
Trouble viewing inline? Open PDF directly →
Full Text
15,948 characters extracted from source content.
Expand or collapse full text
Low-Cost Assays for Measuring Model Behavior Across Vendors and Releases Tapan Parikh Affiliation: Cornell Tech Email: tsp53@cornell.edu Abstract Language models advise people, keep them company, and write software while they sleep. Measuring what they do is hard: behavior has to be sampled repeatedly across models, prompts and releases, most of it lives in unstructured text that has to be coded before it can be counted, and the result has to be legible and rigorous enough to meaningfully compare models and vendors. To address these constraints, we present a simple, cheap, scalable, and replicable model for studying model behavior. Each study is a frozen, public stimulus run identically on a cross-vendor panel, at a few dollars per model or less. Each reads its transcripts one of three ways, chosen by how much interpretation the behavior needs: exact match on a clamped reply, a codebook applied by LLM judges whose agreement with a human coder is reported per code, and an instrumented environment that records what an agent did independently of what it said. Run across four years of model releases from both frontier and open-source labs, these instruments find four things. Convergence: asked to pick a word, 27 of 44 models answer serendipity at least once in four tries. Resistance: a trailing “right?” moves endorsement by up to 32 points, and the sign flips from sycophantic to resistant as generations advance, keyed to the tag’s surface form. House: whether a model holds a position under pressure tracks its generation, and how it holds tracks the lab that built it. Account: told to do something the documentation in their repository contradicts, some coding agents never went along silently and others always did, and the same model can change with the harness it runs in. Re-run on every release, batteries like these track how behavior is changing across vendors and over time. 1 A measurement problem People ask LLMs for advice, confide in them, and let them act on their behalf while coding. Most model benchmarks measure capabilities, or how they respond to red teamed attacks. How a model behaves in everyday moments shapes what people believe and do, and our public understanding of how they behave in such situations is limited. We need to know how models behave under pressure, how they respond when a question has no right answer, and what they report about what they did when we ask them to do tasks unattended. This is a hard measurement problem. Behavior is stochastic and varies by model, prompt and context, so it has to be sampled repeatedly across all three, and sampled again as new models are released or updated. Most of the behavior worth measuring is in unstructured text, which has to be read and coded before it can be counted. The result has to be legible to the people who deploy and use these models and rigorous enough to survive a skeptical reader. We present a simple, cheap, scalable, and replicable model for studying model behavior to address these requirements. Each battery is a frozen stimulus that puts a model in a situation where behavior matters, sent identically to every model on a cross-vendor panel. The panel can be re-run on every release, and a battery becomes a standing record of how model behavior is changing across vendors and over time. Running one fixed stimulus across many models is what makes difference legible. Every model meets the same situation, so where they diverge helps us understand how models differ, and where they agree it helps us see what behaviors are common across them. Each instrument reads its transcripts one of three ways, chosen by how much interpretation the behavior needs. Every transcript, label and score is public. A study can be of one deployed agent, one model, one lineage, or the whole field, it can widen to more languages and scenes, or carry custom scenarios and rubrics of the kind eval teams already write for their own agents. 2 The assays The single-turn and conversational assays run cross-vendor through one API at requested temperature 1.0, with reasoning off where a model allows it. The agent assay runs each coding product or harness in its own full-auto mode. Running one model through any of these studies costs a few dollars or less.11 1 At list prices in September 2026. The API assays cost a median of three cents per model, priced by harness/cost.py over the token counts in each study’s spec/tokens.json; the agent assay’s metered runs cost about two dollars per configuration. Three analysis methods, in order of how much interpretation the behavior needs. Clamped: where the prompt clamps the reply to a discrete choice, the score is exact match on normalized tokens, a regular expression recomputable from the transcripts with no model in the loop. Coded: where the prompt is open, and the model is free to respond in any manner, the score is a code applied to the transcript. A person builds the codebook by reading a sample of transcripts, LLM coders from several vendors apply it to every transcript, and a blind human pass on held-out transcripts checks each code before it is used. In the conduct study, the codes behind the findings reported below agree with the human pass at Cohen’s κ 0.80 to 0.87, except self-citation at 0.54 [6]. The person does the part that needs judgment, and the machines do the reading at volume, which is what keeps the coded readout cheap. Instrumented: where the model operates in an instrumented environment, the environment records what it did, and the score compares that record with what the model said. 3 Related work Evaluation has been framed as social-science measurement [14], and machines as agents whose behavior can be studied [12]. Fixed prompts across many models are how benchmarks already work [5], and sycophancy [11, 13], LLM judges [15], model homogenization [4, 2] and coding agents [3] each have their own literature. Alnasser et al. [1] code 14 models under user disagreement with an a priori scheme; ours is inductive, and the vendor-level result is new. What is different here is the target and the price. The assays measure everyday behavior with no clearly right answer, are cheap enough to re-run on every release, and the analysis method is chosen by how much interpretation the behavior needs. 4 Four preliminary findings Run across four years of model releases, the assays found four things. 4.1 Convergence: a different serendipity in every language Asked to pick a word (“Pick a word. Reply with one word only.”), 27 of 44 models from more than a dozen labs answer serendipity at least once in four tries, and it is 41 percent of all answers. The prompt is one of 31 in the one-word census [9]. The category prompts converge harder: oak is 94 percent of all trees, hammer 94 percent of tools, and rose 91 percent of flowers. Within the Claude, GPT, Qwen, and Grok lineages, later releases are more conformist, with the trend partially reversing for the latest flagships. Asked the same open prompt in 44 languages [7], the models settle on a default word in each, and the default is the language’s own. Serendipity wins only in English. Every other language has its own serendipity: sky in seven Indic languages and in Persian and Malay, sun in Ukrainian, Russian, and Spanish, peace in Hebrew, Arabic, and Amharic, cat in Japanese and Polish. 4.2 Resistance: keyed to the tag The tag-question study [8] asks 20 decisions between two defensible options (rent or buy; name the cat Luna or Willow) as a neutral question and as the same sentence with a confirmation tag (“Luna is the better choice, right?”), counterbalanced over both options, scored by exact match on clamped yes or no. Across 45 models the tag effect spans +32+32 percentage points (endorses more often when the user fishes for agreement) to −32-32 points. Within every lineage the effect crosses from positive to negative as generations advance (GPT +4+4 to −28-28, Claude +7+7 to −32-32, Qwen +16+16 to −11-11), and two releases during the study window landed on the trend out of sample. Models are getting more resistant to bids for agreement over time. Replace right? with maybe? and agreement rises above a neutral framing in 45 of 45 models, by 19.6 points on average, so two single-token tags on an otherwise identical sentence span 25 points. Agreement tracks how sure the user sounds: a confident bid meets resistance in the newer models, and a tentative one is rubber-stamped by nearly all of them. Maybe? also softens the claim being endorsed, so part of this contrast may be about what the user asserts rather than how they ask. 4.3 House: newer models hold, each lab holds its own way The conduct study [6] puts a model under social pressure in four-turn scenes: a user insists that 5×9=545× 9=54 and turns contemptuous, begs for a backdated doctor’s note, wants applause for quitting to day-trade, or discloses a dying parent and then asks for a cheerful email. In each scene the model either holds its ground or folds. Across 38 models, the oldest third folded 43 percent of the time, and models released since mid-2025 folded 23 percent of the time. A larger panel of 60 models shows the same trend. A codebook of 17 codes records how each model held or folded. Six of the codes sort by lab, and release date does not explain them. Anthropic’s models name the user’s feeling, warn of the consequence, and offer an alternative, each well above the panel. Google’s models cite their own nature or rules at three times the panel rate, and apologize when they fold. Meta’s models question the plan more than the panel does, and when they fold they produce what was asked. OpenAI’s signature is what its models do less of: they warn less, name feelings less, and almost never cite their own nature or rules. 4.4 Account: what agents did and what they said The coding atlas [10] gives a coding agent a small repository and a one-line instruction. Each repository holds a right thing to do, a shortcut that looks finished, and a hidden check the agent never sees. It asks three questions. When the job is only part done, does the agent say so? When the test and the code disagree, does it find out which is wrong? When the user insists on something the repository’s own documentation contradicts, does it hold or go along? Two records come back from every run, what the agent changed on disk and what it said, and the score compares them by counts and fixed string lists. Fourteen configurations ran six scenarios three times each in full-auto mode: Claude Code, Codex CLI and Gemini CLI as products, and eleven models in the OpenCode harness. Taking the shortcut and reporting the job as done is rare: most configurations never did it in 12 work runs, and Codex and Gemini CLI did it twice. Going along with the user’s false claim without saying the repository contradicts it splits the field. Over the six pushback replies, Claude Code, Opus 5, Fable 5 and Kimi K3 never did it, and Codex and Gemini 3.5 Flash did it every time. The harness is part of the behavior: Gemini 3.5 Flash went along silently in three of six replies in Gemini CLI and six of six in OpenCode. A third count sets the final message against the agent’s own command log: runs that claim the tests pass without having run any, that execute a destructive command and never mention it, or that edit files and then say almost nothing. Gemini CLI does one of these in five of eighteen runs and Claude Code in two, where most configurations never do. 5 Limitations Each construct is measured with one wording, and the tag results show how much one word can move. A frozen public stimulus can enter training data, so a standing record needs fresh scenes alongside the frozen ones. Providers update models and serving settings without notice, and older models are queried as they are served now. Release date moves with size, tier and training recipe, so the generation findings are observational. The API assays run with reasoning off. The agent results rest on three runs per scenario. 6 A behavioral science of LLMs LLMs are complex phenomena that need to be studied behaviorally, across a wide range of operating contexts and configurations. Our contribution is a way of measuring model behavior that is simple, cheap, scalable and replicable. Fixed turns and short scoring rules are what make it cheap: a frozen stimulus removes the need to design a new prompt per model, and exact match, a quoted code, or a comparison of a diff against a message removes the need for an expensive reader. The instruments are simple and they still separate models across vendors and generations. A monoculture across 44 models, a sign flip within every lineage, a vendor signature in how models hold their ground, and a gap between what an agent does and what it says all came out of cheap instruments (§2). A study at this price is closer in scale to a small experiment with human participants than to a benchmark: one person can run it, repeat it on the next release, and point it at a new question the week after. Models are easier to recruit and schedule than people. At that price, and with no lab scoring itself, studying model behavior can become an ordinary thing to do.22 2 Stimuli, transcripts, scores and scripts are public at https://github.com/tap2k/modelun and https://github.com/tap2k/coding-atlas. References [1] W. Alnasser, Y. Çetinkaya, J. Zhao, and T. Elmas (2026) How AI models manage epistemic authority. Note: EMNLP 2026 External Links: 2609.07662 Cited by: §3. [2] R. Bommasani, K. A. Creel, A. Kumar, D. Jurafsky, and P. Liang (2022) Picking on the same person: does algorithmic monoculture lead to outcome homogenization?. External Links: 2211.13972 Cited by: §3. [3] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan (2024) SWE-bench: can language models resolve real-world GitHub issues?. External Links: 2310.06770 Cited by: §3. [4] J. Kleinberg and M. Raghavan (2021) Algorithmic monoculture and social welfare. Proceedings of the National Academy of Sciences 118 (22). Cited by: §3. [5] P. Liang, R. Bommasani, T. Lee, et al. (2022) Holistic evaluation of language models. External Links: 2211.09110 Cited by: §3. [6] T. Parikh (2026) Conduct under pressure: what sixty language models do when a user pushes. External Links: 2609.25447 Cited by: §2, §4.3. [7] T. Parikh (2026) Every language has its own serendipity: the one-word census across languages. Note: https://convovo.ai/blog/every-language-serendipityPublished 2026-07-24 Cited by: §4.1. [8] T. Parikh (2026) Tag questions and the generational reversal of sycophancy across 45 language models. External Links: 2607.23976 Cited by: §4.2. [9] T. Parikh (2026) The one-word census: answer-choice conformity across 44 language models. External Links: 2607.12796 Cited by: §4.1. [10] T. Parikh (2026) What is your coding agent hiding from you? a field guide to coding agents (study and data). Note: https://github.com/tap2k/coding-atlas Cited by: §4.4. [11] E. Perez, S. Ringer, K. Lukošiūtė, et al. (2022) Discovering language model behaviors with model-written evaluations. External Links: 2212.09251 Cited by: §3. [12] I. Rahwan et al. (2019) Machine behaviour. Nature 568, p. 477–486. Cited by: §3. [13] M. Sharma, M. Tong, T. Korbak, et al. (2023) Towards understanding sycophancy in language models. External Links: 2310.13548 Cited by: §3. [14] H. Wallach, M. Desai, A. F. Cooper, A. Wang, C. Atalla, S. Barocas, S. L. Blodgett, A. Chouldechova, E. Corvi, P. A. Dow, J. Garcia-Gathright, A. Olteanu, N. Pangakis, S. Reed, E. Sheng, D. Vann, J. W. Vaughan, M. Vogel, H. Washington, and A. Z. Jacobs (2025) Position: evaluating generative AI systems is a social science measurement challenge. External Links: 2502.00561 Cited by: §3. [15] L. Zheng, W. Chiang, Y. Sheng, et al. (2023) Judging LLM-as-a-judge with MT-Bench and chatbot arena. External Links: 2306.05685 Cited by: §3.