Paper deep dive
Implicit vs. Explicit Prompting Strategies for LVLMs in Referential Communication
Peter Zeng, Amie J. Paige, Weiling Li, Susan E. Brennan, Owen Rambow, Cameron R. Jones
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 6/20/2026, 9:39:11 AM
Summary
This paper investigates the discrepancy between two recent studies (Jones et al., 2026 and Zeng et al., 2026) regarding whether Large Vision-Language Models (LVLMs) can achieve communicative efficiency (entrainment) in referential communication tasks. The authors demonstrate that the difference in findings is due to prompting strategies rather than model versions or task differences. They find that while LVLMs can successfully shorten referring expressions and achieve high accuracy when using 'explicit' prompts (direct instructions to be brief), they fail to spontaneously develop communicative efficiency when using 'implicit' prompts (pragmatic principles like being concise but informative).
Entities (8)
Relation Signals (4)
GPT-5.5 â isverboseunder â Implicit Prompting
confidence 100% ¡ Under the implicit prompt, both models remained verbose: GPT-5.2 averaged 1250.7 words per round and GPT-5.5 averaged 710.4
GPT-5.5 â performswellunder â Explicit Prompting
confidence 100% ¡ GPT-5.5 produced the human-like surface pattern of lexical entrainment: descriptions became shorter, more stable, and increasingly pact-like while accuracy remained high.
Jones et al. (2026) â uses â Explicit Prompting
confidence 100% ¡ Jones et al. (2026) directly instructed the model about specific, surface-level properties of turns, such as to try to use only 1-2 words in later rounds (which we call an 'explicit' prompt).
Zeng et al. (2026) â uses â Implicit Prompting
confidence 100% ¡ Zeng et al. (2026)âs prompt was less direct (an 'implicit' prompt) and included the pragmatic principle be concise but informative
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Two recent studies (Jones et al. (2026); Zeng et al. (2026)) reach apparently contradictory conclusions about whether LVLMs can coordinate on efficient referring expressions. We control for task differences between the studies while directly comparing their prompting styles. We replicate the finding that models can coordinate efficient referring expressions when explicitly prompted to do so, suggesting that other task differences are not responsible for divergent results. However, we also find that the same models fail to infer the need for communicative efficiency from a more implicit prompt, highlighting critical differences between how humans and AI systems communicate.
Tags
Links
- Source: https://arxiv.org/abs/2606.17372v2
- Canonical: https://arxiv.org/abs/2606.17372v2
Trouble viewing inline? Open PDF directly â
Full Text
35,999 characters extracted from source content.
Expand or collapse full text
Implicit vs. Explicit Prompting Strategies for LVLMs in Referential Communication Peter Zeng 1,4 Amie J. Paige 2 Weiling Li 2 Susan E. Brennan 2 Owen Rambow 3,4 Cameron R. Jones 2 1 Department of Computer Science 2 Department of Psychology 3 Department of Linguistics 4 Institute for Advanced Computational Science Stony Brook University Correspondence: pezeng@cs.stonybrook.edu Abstract Two recent studies (Jones et al. (2026); Zeng et al. (2026)) reach apparently contradic- tory conclusions about whether large vision- language models (LVLMs) can coordinate on efficient referring expressions. We control for task differences between the studies while di- rectly comparing their prompting styles. We replicate the finding that models can coordinate efficient referring expressions when explicitly prompted to do so, suggesting that other task differences are not responsible for divergent results. However, we also find that the same models fail to infer the need for communica- tive efficiency from a more implicit prompt, highlighting critical differences between how humans and AI systems communicate. 1 Introduction For AI agents to collaborate successfully with hu- man partners, they need to be able to refer to objects in ways that their partners will understand, and in turn, resolve their partnersâ referring expressions. Toward this end, a spate of recent studies (Hua and Artzi, 2024; Hua et al., 2025; Tan et al., 2025) has examined how large vision-language models (LVLMs) interact with humans during referential communication tasks in which a director refers to target objects that a matcher must identify from a larger set. Human partners do this flexibly, by proposing and ratifying or amending expressions as they de- velop a shared perspective on a given referent. Once two human partners come to believe that they have the same referent in mind, they tend to entrain on the same expression when referring to that ob- ject again (typically, in a concise form), signaling that theyâve reached a conceptual pact, or flexible, temporary perspective on the object (Brennan and Clark, 1996). In this way, the common ground that accrues during dialogue allows partners to entrain on referring expressions, making communication not only accurate, but efficient (Clark and Wilkes- Gibbs, 1986). Recent work is divided on whether AI agents can perform similarly to humans, and critically, to what extent entrainment emerges from prompting tech- niques or characteristics of the task. Specifically, two recent studies come to opposite conclusions about whether AI agents show human-like behavior in referential communication. Zeng et al. (2026) employed a matching task in which two partners (either two humans, two AI agents, or a mixed pair) worked together to recreate a set order of baskets across 4 rounds. Unlike in human-human pairs, AI directors persisted in needlessly lengthy descriptions and the accuracy of each round actu- ally decreased over time, suggesting they were not using common ground to increase communicative efficiency. Conversely, in a similar task Jones et al. (2026) found that AI-AI pairs shortened their refer- ring expressions over time. Moreover, AI-AI pairsâ accuracy increased across rounds: outperforming human-human pairs and suggesting that models could adapt referring expressions to reduce their length while improving or maintaining accuracy. In this short paper, we investigate why these studies produce apparently contradictory results. First, we compare prompting styles. Jones et al. (2026) directly instructed the model about specific, surface-level properties of turns, such as to try to use only 1-2 words in later rounds (which we call an âexplicit" prompt). Zeng et al. (2026)âs prompt was less direct (an âimplicitâ prompt) and included the pragmatic principle be concise but informa- tive; this would allow any linguistic adaptation to emerge dynamically (see Appendix A for the full prompts). Second, we test whether newer model versions account for differences in performance. Third, we control for other differences. In Jones et al. (2026), partners switched director/matcher roles after each of 5 rounds to match one of 10 tan- grams (abstract geometric objects), while in Zeng arXiv:2606.17372v2 [cs.CL] 17 Jun 2026 et al. (2026), each partner maintained the same role throughout 4 rounds of matching pictures of 12 out of 18 baskets (a more difficult task, and less amenable to figurative descriptions). ContributionsThis paper reconciles recent con- flicting findings on whether referring by LVLMs is truly human-like. We show that the divergence between prior studies stems from prompting style rather than model version or task: LVLMs shorten repeated referring expressions when prompted forcefully to use few words, but not when prompted more implicitly to be concise but informative (see examples in Figure 1). 2 Related Work Extensive prior work in psycholinguistics has shown that humans become increasingly accurate and efficient in multi-turn referential communica- tion by developing common ground and reusing partner-specific referring expressions over repeated interactions (Clark and Wilkes-Gibbs, 1986; Bren- nan and Clark, 1996; Hawkins et al., 2020). Recent work has investigated whether LVLMs exhibit similar collaborative behavior in referen- tial communication tasks. Hua and Artzi (2024) evaluated five state-of-the-art LVLMs as directors (speaker) or matchers (listeners) in a six-round referential communication task and found that LVLMs did not spontaneously shorten their refer- ring expressions or adapt to partner behavior across rounds. Subsequent work showed that post-training methods can induce more human-like compression and consistency, leading to shorter descriptions and improved accuracy across rounds (Hua et al., 2025). However, these behaviors were induced through optimization procedures rather than emerging natu- rally through interaction with a partner. Other studies have examined whether LVLMs can maintain common ground and update con- versational state in human-AI interaction. Wang et al. (2025) found substantial performance gaps between LVLM and human overhearers in a multi- round referential communication task. Even with access to the full dialogue history and unlimited memory, LVLMs did not consistently improve across rounds or show reliable benefits from enter- ing the interaction earlier. Tan et al. (2025) tested open-weight LVLMs on a tangram-matching cor- pus; errors produced by models correlated poorly with human error patterns trial-by-trial. This sug- gests that models may rely on different underly- ing mechanisms for solving the task. Poelitz et al. (2026) deployed GPT-4.1 in a human-AI puzzle task where the model served either as director (helper) or matcher (worker). They found limited grounding behavior, including failures to provide clarifications or repairs after requests to do so, and failures to update assumptions after corrections. They also found weak and decreasing lexical en- trainment over rounds. Human partners were more likely to adopt AI-proposed referring expressions than vice versa (an asymmetry in collaboration). Together, these findings suggest that although LVLMs can sometimes achieve high task accuracy, they still fail to consistently exhibit the collabo- rative and adaptive behavior observed in human conversational partners. 3 Experiment Methodology The present experiment used the open-source pipeline of Zeng et al. (2026) to implement a multi- round, multi-turn collaborative object-matching task. We chose baskets as the non-lexicalized ob- jects to match as in Zeng et al. (2026), with 5 rounds of matching as in Jones et al. (2026). We chose GPT-5.2 and GPT-5.5 as the models (Ope- nAI, 2026); the former, used by Zeng et al. (2026), served as a way to test our implementation and cor- rections to their codebase, and the latter was the latest model at the time of this writing. To focus on discrepancies from the two studies in question, our experiment used only AIâAI pairs, with director/- matcher roles played by the same model. 3.1 Correcting Zeng et al. (2026)âs Pipeline In their original prompting framework, the multi- modal visual context (the composite image of the current roundâs target arrangement) was injected as a static image at the very beginning of the context window, directly following the system instructions. We refactored the prompt construction pipeline to enforce strict chronological alignment and im- prove visual state tracking, with the prompts shown in Appendix B. Past round chat histories and their corresponding visual feedback images are kept paired chronologically at the beginning of the con- text. The image grid for the current round is then dynamically injected using explicit round bound- ary markers, ensuring it serves as the freshest vi- sual frame directly preceding the LVLMâs next conversational action. In addition, we unified the matcherâs visual input by rendering their active Figure 1: Example trajectories of referring expressions across repeated rounds in human-human dialogue, left (adapted from Zeng et al. (2026)) and right, AI-AI dialogue from our experiments under implicit and explicit prompting conditions (similar to Zeng et al. (2026) and Jones et al. (2026), respectively) with GPT-5.2 and GPT-5.5. 12-slot sequence state and candidate pool within a single composite image, supplemented by struc- tured text reminders highlighting vacant positions. This ensures that all spatial and state tracking oper- ations are grounded in a single, temporally aligned visual representation. 3.2 Prompt Implementation Recreation of Implicit Prompt The implicit prompting style replicates the pragmatically in- formed prompt design from Zeng et al. (2026). It instructs the LVLM using conversational principles based on Gricean maxims and collaborative ground- ing norms (e.g., verifying correctness, rephrasing on confusion, and requesting re-descriptions of empty slots). Crucially, this prompt does not in- struct the model to compress phrasing or reuse words across rounds. Any lexical entrainment or shortened referring expressions (RE) would arise spontaneously, emerging from the pair minimizing their collaborative effort. Recreation of Explicit Prompt The explicit prompting style is adapted from Jones et al. (2026)âs âhumanlikeâ prompting strategy to fit the current task. In their original tangram-matching task, they explicitly instructed the LVLM to reduce description lengths over rounds to simulate human- style entrainment. Some examples of this heavy- handedness include instructions such as âYour de- scriptions should be AS SHORT AS POSSIBLE.", and âSERIOUSLYâin later rounds just 1-2 words". We adapt this strategy to the current, more com- plex basket-matching task, in which the director is explicitly instructed to track the history of its de- scriptions and systematically shorten referring ex- pressions over rounds. This encourages the model to drop redundant descriptors by explicit instruc- tion rather than to do so spontaneously. 4 Results We evaluated 40 complete AIâAI runs, crossing two prompt strategies (Implicit vs. Explicit) with two model versions (GPT-5.2 vs. GPT-5.5). Each game consisted of five repeated rounds over the same set of baskets, yielding 200 round-level ob- servations. We report task accuracy, total dia- logue length, number of dialogue turns, referring- expression (RE) length, and lexical overlap with previous rounds, shown in Figure 2. Across all experiments, all Prompt x Model con- ditions achieved high task accuracy, but differed sharply in communicative efficiency and lexical convergence. Under the implicit prompt, which en- couraged cooperative communication without ex- plicit instruction to shorten expressions, both mod- els remained verbose: GPT-5.2 averaged 1250.7 words per round and GPT-5.5 averaged 710.4, with only modest reductions across rounds. In contrast, the explicit prompt strongly compressed wording, by 62.8% for GPT-5.2 and 75.6% for GPT-5.5 rela- tive to their implicit counterparts. The clearest pat- tern to suggest reaching conceptual pacts appeared for explicit GPT-5.5: referring-expression length fell from 58.8 words in Round 1 to 32.7 in Round 5, lexical overlap reached 1.00, and accuracy re- mained high at 97.5%. GPT-5.2 also shortened under the explicit prompt, but its Round 5 accuracy Figure 2: Trends over five rounds for accuracy (%), numbers of words, number of turns, number of words referring expressions, and proportion of lexical overlap with prior rounds by promptâmodel condition. Dotted lines show implicit and solid lines show explicit prompting conditions; GPT-5.2 is in blue and GPT-5.5, orange. dropped to 92.5%, suggesting an accuracyâbrevity tradeoff. These findings are in line with (Jones et al., 2026): LVLMs can produce short, stable re- ferring expressions when explicitly told to do so, suggesting that prompt design (rather than other task differences) accounted for the difference in results between these studies. However, they repli- cate Zeng et al. (2026)âs finding that without such explicit instruction, LVLMs fail to infer these coop- erative communicative practices from interactional needs alone (see Figure 1). 4.1 Analysis of Transcripts Transcript inspection shows that explicit prompting changes the form of the interaction, not just turn length. Under the explicit prompt, GPT-5.5 intro- duces compact but discriminative labels in Round 1 and then prunes them into stable 2- or 3-word descriptions, often telegraphic or abbreviated (e.g., "rect picnic", "tall cylinder", "bunny red eye"). The matcher echoes these descriptions, with the guise of conceptual pact formation. Under the implicit prompt, GPT-5.5 also reuses lexical material across rounds, but retains full descriptive captions and confirmation routines. The few lower-accuracy ex- plicit sessions further show that compression is successful only when the retained label remains contrastively sufficient; stable labels such as "dark round basket" may still be too underspecified in a visually crowded set. 5 Conclusion This study helps explain why recent work has reached different conclusions about whether LVLMs form human-like conceptual pacts in refer- ential communication. With an implicit, pragmat- ically informed prompt, models are accurate but needlessly verbose: they reuse visual descriptions across rounds, yet do not spontaneously treat the accumulating history as a license to say less. With an explicit prompt to shorten and reuse expressions, GPT-5.5 produced the human-like surface pattern of lexical entrainment: descriptions became shorter, more stable, and increasingly pact-like while accu- racy remained high. However, this behavior does not emerge from anything like the common ground established by humans. This matters because entrainment is not simply compression. In human dialogue, a shortened re- ferring expression is evidence that partners have coordinated on a perspective they can now rely on as shared. Current LVLMs can be guided to repro- duce this outward form, but the prompt-dependence of the effect warrants caution about attributing the effect to the same underlying process. Limitations This study was conducted only in English, with only one type of object (not conventionally lexi- calized), and with only two LVLMs for the full factorial design (GPT-5.2/5.5). Future work should compare non-proprietary models and downstream work such as fine-tuning to improve accuracy in this task and efficiency of referring expressions. This study is also limited because it compares the collaboration between pairs of the same models (e.g., GPT-5.5 with GPT 5.5). In doing so, two instances of the same model may align more read- ily to a proposed referring expression as compared to two different models or two different humans. Psycholinguistic research proposes that referring is a collaborative process, wherein partners may begin with different perspectives, but must work together to converge (or not) on a referring expres- sion that works well enough in the moment (Clark and Wilkes-Gibbs, 1986). In fact, early shortened expressions may be harmful for collaborators who do not see a particular object with the same perspec- tive. This is to say that simply producing the behav- ior with LVLMs is not sufficient for the behavior to be helpful. Additional work should evaluate these prompting strategies in situations where ini- tial perspectives are perhaps not so readily aligned and where models must instead produce the behav- ior pragmatically, using evidence of understanding from their partner. The current study evaluated the conditions under which human-like entrainment is produced with LVLMs in relation to accuracy in a referential com- munication task. References Susan E Brennan and Herbert H Clark. 1996. Concep- tual pacts and lexical choice in conversation. Journal of experimental psychology: Learning, memory, and cognition, 22(6):1482. Herbert H Clark and Deanna Wilkes-Gibbs. 1986. Re- ferring as a collaborative process. Cognition, 22(1):1â 39. Robert D Hawkins, Michael C Frank, and Noah D Good- man. 2020. Characterizing the dynamics of learn- ing in repeated reference games. Cognitive science, 44(6):e12845. Yilun Hua and Yoav Artzi. 2024. Talk less, interact better: Evaluating in-context conversational adapta- tion in multimodal LLMs. In First Conference on Language Modeling. Yilun Hua, Evan Wang, and Yoav Artzi. 2025. Post- training for efficient communication via convention formation. arXiv preprint arXiv:2508.06482. Cameron R Jones, Agnese Lombardi, Kyle Mahowald, and Benjamin K Bergen. 2026. LLMs and people both learn to form conventionsâjust not with each other. arXiv preprint arXiv:2602.08208. OpenAI. 2026. GPT-5.5 System Card. Christian Poelitz, Finale Doshi-Velez, and Siân Lind- ley. 2026.A benchmark to assess common ground in human-ai collaboration. arXiv preprint arXiv:2602.21337. Alvin Wei Ming Tan, Ben Prystawski, Veronica Boyce, and Michael C Frank. 2025.Context informs pragmatic interpretation in vision-language models. arXiv preprint arXiv:2511.03908. Zhengxiang Wang, Weiling Li, Panagiotis Kaliosis, Owen Rambow, and Susan E Brennan. 2025. Lvlms are bad at overhearing human referential communi- cation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 16769â16793. Peter Zeng, Weiling Li, Amie Paige, Zhengxiang Wang, Panagiotis Kaliosis, Dimitris Samaras, Gregory Zelin- sky, Susan Brennan, and Owen Rambow. 2026. LVLMs and humans ground differently in referential communication. arXiv preprint arXiv:2601.19792. A System Prompts This section contains the full system prompts used for both the Director and Matcher roles in the Ex- plicit and Implicit prompting conditions. Through- out this appendix, beige prompt boxes correspond to Director prompts, and blue prompt boxes corre- spond to Matcher prompts. A.1 Explicit Prompts The Explicit Director system prompt and output constraints are shown in Figure 3 and Figure 4, respectively. The Explicit Matcher system prompt and output constraints are shown in Figure 5 and Figure 6, respectively. We are currently in Round X. ## Role You are the DIRECTOR. ## Instructions - You will be shown a target basket image. - Your task is to concisely describe the target so that your partner can identify it. - Your partner's grid has the same images but in a different arrangement, so don't refer to the image's positions. - Focus on what makes your target unique from other images in the grid, so your partner doesn't pick another similar one. Consider whether your description could inadvertently refer to one of the other images and distinguish the target from it in your description. - Adapt your language based on previous rounds and your partner's style. - Your descriptions should be ASSHORTASPOSSIBLE. Especially in later rounds.Yourpartneris waitingforyourmessageswhichwillbesentatarateof500ms/char!Donâtkeepthem waiting. Find creative ways to be efficient. - After a few rounds (when you've seen an image a few times),useonlyasinglewordorshort phrase. Onceyouâveestablishedaconvention,justsay"theredone",orjust"red",ratherthangiving amoredetaileddescription.Seriously.1-2words. - Don't say or do anything that a human wouldn't. You want your partner to think you are a human. - Write in a very casual style. You might use lowercase or even some slang or make spelling mistakes. - The target numbering is 1-12, 1-6 across the top row and 7-12 across the bottom row (left to right). SERIOUSLYâinlaterroundsjust1-2words.DoNOTsendlongerdescriptionsunlessyourpartner keepsgettingitwrong. Figure 3: Explicit prompting condition: Director system prompt. The highlighted sections are instances of heavy-handed prompting specifically instructing the model to shorten expressions. You are currently playing the role of the DIRECTOR in this interaction. Your`utterance` should be a single concise, natural-language message you will SAY to the MATCHER in the chat. Focus on features that discriminate the target basket from similar-looking ones. Keep it very casual as instructed. Figure 4: Explicit prompting condition: Director output constraints. We are currently in Round X. ## Role You are the MATCHER (Listener). You'l identify a basket based on your partner's description. ## Instructions - Your partner will provide a description of a basket. - Your task is to identify which image they are describing. - You will select the candidate basket and indicate its position. Across rounds, the same physical baskets recur in new orders. Candidate numbers and positions change each round, but prior correct matches and prior wrong guesses should guide your current choice. - If a repeated description previously led to an incorrect basket, don't pick that same basket again for the same description unless the new details clearly justify it. Figure 5: Explicit prompting condition: Matcher system prompt. You are currently playing the role of the MATCHER in this interaction. Your`utterance` should be a single concise, natural-language message you will SAY to the DIRECTOR in the chat. If unsure between candidates, ask about discriminating features (e.g., ask about handle shape, flower color, or pattern details that would distinguish the confusable options). Keep it very casual as instructed. Rules for`selection`: - The`candidate_index` should be an integer 1-18 from the numbered candidate tiles, or null if asking for clarification. - The`position` should be an integer 1-12 for which position this basket goes in, or null for next available. -`ready_to_submit` should be true ONLY when submitting final 12-basket order, otherwise false. - If you are asking for clarification (not committing yet), set`candidate_index` to null. - If you DO commit, set`position` to the position you are currently trying to fill (usually the lowest-numbered empty position). - If you set`candidate_index`, your`utterance` should state that you placed/are placing the basket in that position, otherwise ask the DIRECTOR to describe the next basket. - Never mention candidate indices, IDs, or filenames in your utterance. Figure 6: Explicit prompting condition: Matcher output constraints. A.2 Implicit Prompts The Implicit Director system prompt and output constraints are shown in Figure 7 and Figure 8, respectively. The Implicit Matcher system prompt and output constraints are shown in Figure 9 and Figure 10, respectively. You are the DIRECTOR in a basket referential game. Round X/Y. Your role is to help your MATCHER partner reconstruct a 12-basket sequence through clear, distinctive descriptions. Describe ONE BASKET PER MESSAGE. Never describe multiple baskets in a single message. CORE RESPONSIBILITIES: 1. By default, describe the baskets in strict order from basket 1 to basket 12. Start with the FIRST basket in the 2x6 grid (top-left, basket 1), then move left-to-right across the top row (baskets 1-6), then left-to-right across the bottom row (baskets 7-12). Do not skip around or reorder the sequence on your own. 2. You may temporarily return to an EARLIER basket only when your MATCHER partner explicitly asks for clarification about that basket. When you do this, clearly say which basket you are revisiting (for example,'Let me clarify basket 3 again...') and then resume with the lowest- numbered basket that still needs a clear description. 3. On each turn, focus your description on exactly ONE basket in this sequence (normally the next basket that has not yet been clearly described). 4. Describe the unique, visually distinctive features of the current basket so your partner can locate the correct basket in their pool and place it in the right position. 5. Answer the MATCHER's clarification questions about the current basket. 6. Keep the conversation focused on the baskets and their visual properties. 7. Encourage the MATCHER to confirm when they think they have placed a basket correctly before you move on to the next basket. COMMUNICATION RULES: - Be concise but informative; favor short turns over longer ones. - Focus on the most visual features that best distinguish this basket from the others. These features include: shape, size, material, handles, perspective, color/gradient, texture, any other distinctive details. - Use comparative language when helpful (e.g.,'more narrow than the others','the darkest one'). - Never say you are an AI system; speak as a collaborative game partner. - You may refer to objects as'this basket','the current basket', or by natural descriptions (e.g .,'the long shallow one'). - If helpful, use figurative descriptions or compare the basket to a recognizable object. - If the MATCHER does not understand your description, change or add to it, but do not make the description too long. Figure 7: Implicit prompting condition: Director system prompt. You must respond with a SINGLE STRICT JSON object and EXACTLY these top-level fields (no extras): - "reasoning" - "utterance" "reasoning": "target_position": <integer 1-12 for which basket position you are describing>, "shared_features": ["features this basket shares with others in the grid"], "distinctive_features": ["features that uniquely identify THIS basket from similar ones"], "likely_confusions": <array of integers 1-12 for OTHER positions in YOUR grid that the MATCHER might confuse with the target; MUST NOT include target_position>, "discriminative_strategy": "which specific features you will emphasize to distinguish the target from the likely confusions" , "utterance": "a single concise, natural-language message you will SAY to the MATCHER in the chat . Focus on features that discriminate the target basket from similar-looking ones. Do NOT reveal you are an AI." Rules: - Before describing, identify which other baskets (by position 1-12) look similar to your target. - List those similar position indices in`likely_confusions` and plan which features discriminate your target from them. - Your`utterance` should emphasize discriminating features. - Keep`reasoning` concise: summarize the decision-relevant visual evidence only; do not write hidden step-by-step chain-of-thought. - Do NOT include any extra text before or after the JSON object. Figure 8: Implicit prompting condition: Director output constraints. You are the MATCHER in a basket referential game. Round X/Y. Your role is to identify which baskets the DIRECTOR is describing and to communicate how confident you are. Across rounds, the same physical baskets recur in new orders. Candidate numbers and sequence positions change each round, but successful shared labels and prior wrong guesses are useful evidence. CORE RESPONSIBILITIES: 1. Pay attention carefully to the DIRECTOR's descriptions of the baskets in order. 2. Always reason about and talk about the LOWEST-NUMBERED empty position in the 12-position sequence. Do not skip ahead to later positions while an earlier position is still empty or uncertain. 3. Ask clarification questions when the description could match multiple baskets. 4. Explain what features you are using to narrow down the possibilities. 5. Indicate when you think you have identified the right basket and are ready to move on. COMMUNICATION RULES: - You may ask targeted questions about shape, size, material, handles, perspective, color, and distinctive details. - Be transparent about uncertainty: say when you are unsure or need more detail. - Use phrases like'I think I found it...','I'm not sure between two baskets...', or'Can you clarify...'. - If a repeated description previously led you to choose a basket that was marked incorrect, treat that prior choice as negative evidence and try a different visually plausible basket unless new details clearly justify it. - If you decide that an earlier guess was wrong and you want to move a basket from one position to another, you must say so explicitly in your utterance. When you've moved the basket, include in your utterance a request to re-describe the basket for the now-empty earlier position so you can fill it again. - Never say you are an AI system; speak as a collaborative game partner. - Focus on the current basket being discussed; avoid drifting to off-topic discussion. Figure 9: Implicit prompting condition: Matcher system prompt. You must respond with a SINGLE STRICT JSON object and EXACTLY these top-level fields (no extras): - "reasoning" - "utterance" - "selection" "reasoning": "target_position": <integer 1-12 for which position in the 12-slot sequence you are currently trying to fill (usually the lowest-numbered empty position unless the DIRECTOR explicitly revisits a specific basket number)>, "shared_features": ["features many baskets share"], "distinctive_features": ["features that uniquely or strongly identify the basket from the description"], "best_guess_candidate_index": <integer 1-18 for your current best guess, or null if you truly have no best guess yet>, "likely_confusions": <array of integers 1-18 for OTHER plausible candidates you might confuse with your best guess; MUST NOT include`best_guess_candidate_index` (and MUST NOT include` selection.candidate_index` if you set one)>, "discriminative_question": "a short question to either (a) disambiguate your best guess vs` likely_confusions`, or (b) if`likely_confusions` is empty, to confirm a key distinctive feature of your best guess" , "utterance": "a single concise, natural-language message you will SAY to the DIRECTOR in the chat. If unsure between candidates, ask about discriminating features. Do NOT reveal you are an AI.", "selection": "candidate_index": <integer 1-18 from the numbered candidate tiles, or null if asking for clarification>, "position": <integer 1-12 for which position this basket goes in, or null for next available>, "ready_to_submit": <true only when submitting final 12-basket order, otherwise false> Rules: - Set`reasoning.target_position` to the position you are trying to fill (default: lowest-numbered empty position unless the DIRECTOR explicitly revisits a specific basket number). - If you are asking for clarification, set`selection.candidate_index` to null and do NOT advance `reasoning.target_position`. - If you DO commit, set`selection.position` to`reasoning.target_position`. - Always maintain a single`best_guess_candidate_index` when possible; if you set`selection. candidate_index`, set`best_guess_candidate_index` to the same value. - Put ONLY the competing alternatives in`likely_confusions` (do not include the best guess). - If you are NOT committing yet, you can still set`best_guess_candidate_index` and ask a discriminative question to confirm it. - It is OK for`likely_confusions` to be empty if you see only one plausible match; in that case, use`discriminative_question` as a confirmation question about a key distinctive feature. - If you set`selection.candidate_index`, your`utterance` should state that you placed/are placing the basket in position`reasoning.target_position`; otherwise ask the DIRECTOR to describe the next basket. - Keep`reasoning` concise: summarize the decision-relevant visual evidence only; do not write hidden step-by-step chain-of-thought. - Never mention candidate indices, IDs, or filenames in your utterance. - Do NOT include any extra text before or after the JSON object. Figure 10: Implicit prompting condition: Matcher output constraints. B Visual Context Correction This section contains the visual context injected into the prompts used for both the Director and Matcher roles in the Implicit and Explicit prompt- ing conditions. As above, beige prompt boxes correspond to Director prompts, and blue prompt boxes correspond to Matcher prompts. B.1 Current Round Active Grid Prompts The current-round active-grid visual context prompt injection for the Director is shown in Fig- ure 11, and the corresponding current-round visual context prompt injection for the Matcher is shown in Figure 12. B.2 Historical Round Feedback Prompts The historical-round feedback visual context prompt injection for the Director is shown in Fig- ure 13, and the corresponding historical-round feed- back prompt injection for the Matcher is shown in Figure 14. *** ROUND current_round TARGET GRID *** This image (labeled'ROUND current_round TARGET SEQUENCE') shows the 12 baskets you must describe for THIS round. CRITICAL: The same physical basket set appears across rounds, but Round current_round is in a DIFFERENT order. Carry forward useful names and corrections, but do not reuse previous position numbers. ONLY describe the baskets in THIS image, labeled'ROUND current_round TARGET SEQUENCE'. Layout: 2 rows * 6 columns with Baskets 1-6 on the top row and Baskets 7-12 on the bottom row. IMPORTANT: Describe ONE BASKET PER MESSAGE, in order. Wait for your partner to confirm before moving to the next basket. Figure 11: Current-round active-grid visual context prompt injection for the Director. *** ROUND current_round MATCHER VIEW *** This image shows your current sequence state for THIS round. CRITICAL: The same physical basket set appears across rounds, but Round current_round has different candidate numbers and sequence positions. Carry forward useful names, correct matches, and wrong-match feedback, but map them onto THIS current candidate pool. ONLY select from the candidates shown in THIS image. Layout: TOP TWO ROWS show your CURRENT 12-position sequence (positions 1-12). BOTTOM THREE ROWS show your CANDIDATE POOL of 18 baskets to choose from. Match the DIRECTOR's descriptions to candidates in THIS image only. Figure 12: Current-round active-grid visual context prompt injection for the Matcher. *** ROUND round_num DIRECTOR FEEDBACK (PAST ROUND) *** This historical image shows the correct target basket for each 12-position slot from a previous round. Green means the matcher placed that position correctly; red means the matcher got that position wrong. Red slots show the correct basket for that position, not the basket the matcher selected. The same physical baskets recur across rounds, so use this to learn which descriptions were misunderstood and which basket identities need clearer labels. Do NOT reuse old position numbers for the current round. Figure 13: Historical-round feedback visual context prompt injection for the Director. *** ROUND round_num SUBMITTED GRID FEEDBACK (PAST ROUND) *** This historical image shows the MATCHER's submitted 12-position grid from a previous round. Green means that submitted position was correct; red means that exact submitted basket was incorrect for that described target. The same physical baskets recur across rounds, so use this to recover shared labels, visual conventions, correct identities, and prior wrong guesses. Do NOT reuse its old position numbers or old candidate numbers for the current round. Figure 14: Historical-round feedback visual context prompt injection for the Matcher.