Paper deep dive
Minimal Local Simulation Foundations for LLM- and VLM-Driven Agents in 2D and 3D Environments
Ryuki Hyodo
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) and vision-language models (VLMs) are expanding the range of behaviors that can be represented in agent-based simulations, but many contemporary platforms are difficult to study, modify, or run on ordinary computers. We present two intentionally minimal simulation foundations for education and rapid prototyping. SD-AgentFoundry-2D provides a two-dimensional multi-agent environment in which locally hosted LLM agents move, communicate, respond to place occupancy, and encounter spatially localized fire events. SD-AgentFoundry-3D provides a three-dimensional digital-twin environment in which a locally hosted VLM receives first-person images and produces natural-language movement instructions. Both codebases are designed to run locally on macOS, Windows, and Linux and are deliberately left open to modification rather than developed as finished applications. Together, they offer accessible starting points for learning about generative social simulation and for building domain-specific extensions.
Tags
Links
- Source: https://arxiv.org/abs/2608.22833v1
- Canonical: https://arxiv.org/abs/2608.22833v1
Trouble viewing inline? Open PDF directly â
Full Text
42,721 characters extracted from source content.
Expand or collapse full text
nameyeardelim & Minimal Local Simulation Foundations for LLM- and VLM-Driven Agents in 2D and 3D Environments Ryuki Hyodo Affiliation: Email: ryuki.hyodo@spacedata.co.jp; hyodo@elsi.jp Affiliation: SpaceData Inc., 1-17-1 Toranomon, Minato-ku, Tokyo 105-6490, Japan Affiliation: Graduate School of Artificial Intelligence and Science, Rikkyo University, 3-34-1 Nishi-Ikebukuro, Toshima-ku, Tokyo 171-8501, Japan Affiliation: Earth-Life Science Institute, Institute of Science Tokyo, 2-12-1 Ookayama, Meguro-ku, Tokyo, 152-8550, Japan Affiliation: UniversitĂ© Paris CitĂ©, Institut de Physique du Globe de Paris, CNRS, F-75005 Paris, France Abstract Large language models (LLMs) and vision-language models (VLMs) are expanding the range of behaviors that can be represented in agent-based simulations, but many contemporary platforms are difficult to study, modify, or run on ordinary computers. We present two intentionally minimal simulation foundations for education and rapid prototyping. SD-AgentFoundry-2D provides a two-dimensional multi-agent environment in which locally hosted LLM agents move, communicate, respond to place occupancy, and encounter spatially localized fire events. SD-AgentFoundry-3D provides a three-dimensional digital-twin environment in which a locally hosted VLM receives first-person images and produces natural-language movement instructions. Both codebases are designed to run locally on macOS, Windows, and Linux and are deliberately left open to modification rather than developed as finished applications. Together, they offer accessible starting points for learning about generative social simulation and for building domain-specific extensions. SD-AgentFoundry-2D: https://github.com/ryukih/SD-AgentFoundry-2D SD-AgentFoundry-3D: https://github.com/ryukih/SD-AgentFoundry-3D 1 Introduction Agent-based simulation has long provided a bottom-up way to study how individual decisions and interactions can produce collective patterns (3). Recent progress in foundation models has widened this approach: instead of specifying every behavior as a fixed rule, a simulator can ask an LLM to interpret a situation, use natural-language memory, communicate with other agents, and choose an action (7). Social Simulacra demonstrated LLM-supported prototyping of populated social-computing systems (16), while Generative Agents combined memory, reflection, and planning in an interactive two-dimensional town (15). Concordia generalized language-mediated agent-based modeling through components that connect LLM calls, associative memory, and an environment-controlling game master (24). More recent systems have increased the scale and application range of this paradigm: AgentSociety and its successor provide environments for large-scale computational social experiments (18; 19), and CitySim models urban schedules and collective city dynamics (4). These developments suggest several possible uses, provided that simulated behavior is not mistaken for validated human prediction. LLM simulations of established human-subject studies can reproduce some findings while also exhibiting systematic distortions (1). Studies that seek individual-level prediction require agents grounded in human data and evaluation against human responses (17). LLM-based simulations may support exploratory studies of communication, evacuation, public policy, urban activity, and collective response to external events. They may also help organizations prototype services, examine hypothetical customer or worker interactions, and identify scenarios that deserve later evaluation with real participants. Such simulations are therefore best treated as instruments for generating hypotheses, comparing mechanisms, and rehearsing possibilities, rather than as substitutes for empirical evidence. Textual and two-dimensional environments capture communication and social organization efficiently, but they abstract away much of the spatial and perceptual structure of physical action. VLMs and vision-language-action (VLA) models provide a complementary direction by grounding decisions in images and language. PaLM-E demonstrated an embodied multimodal language model that integrates visual, textual, and continuous state inputs for embodied reasoning (6). OpenVLA illustrates how visual and linguistic representations can be connected to actions in an open model for robot control (11). SimWorld exposes LLM and VLM agents to multimodal physical and social environments (20), CrowdVLA applies VLA agents to context-aware crowd simulation in semantically structured three-dimensional scenes (9), and TravelAgent integrates generative agents into three-dimensional built environments for navigation and wayfinding (14). These embodied approaches may support exploratory work on navigation, facility use, human-robot interaction, crowd behavior, and digital twins. The growing capability of such systems also creates an educational access problem. Large platforms, remote model services, specialized simulators, and substantial compute requirements can obscure the basic loop connecting perception, reasoning, communication, action, and observation. We present SD-AgentFoundry, a pair of minimal local simulation foundations comprising SD-AgentFoundry-2D for LLM-driven multi-agent social simulation and SD-AgentFoundry-3D for VLM-driven embodied simulation. Our goal is not to provide another feature-complete simulator. Instead, we release two compact codebases that expose this loop directly and can serve as readable starting points. They use locally hosted models, include setup paths for major desktop operating systems, and retain deliberately simple mechanisms that users can replace. The intended contribution is an accessible foundation from which students, researchers, and practitioners can develop their own scenarios, agent designs, measurements, and interfaces. 2 System Architecture This section describes the architecture and implementation of SD-AgentFoundry-2D and SD-AgentFoundry-3D. 2.1 SD-AgentFoundry-2D Figure 1: Snapshot of an example SD-AgentFoundry-2D simulation. The two-dimensional grid contains 20 LLM-driven agents, two places (a cafe and a library), and two active fire events. Colored markers represent the agents, shaded disks show the fire-perception radii, and gray lines connect pairs eligible for local communication. Such configurable environments may support social-simulation studies and characterization of LLM-agent behavior. Figure 2: Snapshot of an example SD-AgentFoundry-3D simulation in the Kibo digital twin. The Kibo module is loaded as the USD environment, where a single VLM-driven agent, shown as a red-clad human avatar, attempts the assigned task. The lower-right inset is a same-viewpoint re-rendering matched to the VLM camera pose and settings. Users can replace Kibo with their own USD scenes to run VLM-driven embodied simulations in custom three-dimensional spaces. SD-AgentFoundry-2D is a discrete two-dimensional multi-agent simulation implemented in Python (Figure 1). The environment is a bounded integer grid containing configurable places, each defined by a name, type, center, spatial extent, and capacity. The default scenario places a cafe and a library on opposite sides of the field. Agents begin outside the places at randomly generated positions and receive a simple persona field representing gender. Their available physical actions are to stay or move by one grid cell in one of four cardinal directions. At every simulation step, each agent first identifies communicable neighbors. Communication requires both spatial proximity and a shared area: agents can communicate when they are outside all places together or when they occupy the same place, but not across place boundaries. Each agent then makes a message decision, the messages are delivered using the pre-movement neighborhood, each agent makes an action decision, and all selected movements are finally executed. Separating communication from movement gives every agent a consistent view of the current step. Recent memories and received messages are retained in bounded histories and included in later prompts. Such bounded histories are a deliberate simplification of a much larger design space for agent memory, which ranges from parametric internal state to retrieval-augmented external stores (8). The prompts provide numerical state rather than qualitative prescriptions. An agent inside a place receives its population, capacity, and occupancy rate, but it is not told that the place is comfortable or crowded. Configurable fire events begin at specified steps and have positions, intensities, and perception radii. Only agents inside a fireâs radius receive its numerical properties and their distance from it; agents outside the radius can learn about the event only through messages. This design leaves interpretations such as avoidance, warning, coordination, or inaction to the LLM rather than encoding them as simulator rules. Related spatial LLM-agent studies have used capacity constraints and resource hazards to examine emergent, model-dependent behavior (23; 13). The LLM is served locally through Ollama. One call determines a message and a second determines movement, memory, and reasoning, using JSON-shaped responses with conservative fallback parsing. Thus, each agent receives two distinct prompts per step: a communication prompt followed by an action-decision prompt. Detailed communication and action-decision prompts are shown in Appendix A.1. The current default uses a compact instruction-tuned Qwen model and disables model-side thinking so that the response budget is available for the requested JSON. The simulator writes memory and reasoning records to JSONL and creates a message log when communication occurs. A display-independent Matplotlib Agg renderer writes the initial state and selected post-step states as PNG files; it never opens a live window. The saved frames can be inspected directly, loaded in the separate browser viewer, or converted to video. Configuration is centralized in YAML, and setup scripts are supplied for macOS/Linux and Windows. 2.2 SD-AgentFoundry-3D SD-AgentFoundry-3D is a three-dimensional, single-agent simulation built around Universal Scene Description (USD) assets. The default demonstration loads a digital twin of the Japanese Experiment Module âKiboâ and gives the agent the task of locating the JAXA logo and approaching it (Figure 2). The USD stage supplies geometry, scale, and coordinate information; the implementation uses usd-core for scene access and VTK/PyVista for rendering rather than requiring a heavyweight game engine. Table 1: Configuration and recorded artifacts for an example SD-AgentFoundry-2D run. Setting Value LLM backend and model Ollama; qwen3:4b-instruct-2507-q4_K_M Inference settings Temperature 0.2; thinking disabled; maximum 512 response tokens Agents and inspected point 20 agents; state recorded after step 7 World Integer grid with both axes ranging from â25-25 to 2525 Places Cafe at (â15,0)(-15,0) and library at (15,0)(15,0), with capacities 12 and 10 Fire events fire1: step 3, intensity 0.8, radius 15; fire2: step 5, intensity 0.5, radius 8 Step 7 state Both fires active; 10 agents within at least one perception radius; 0 agents inside places Recorded through step 7 8 PNG frames, 140 memory/reasoning records, and 88 per-recipient message-delivery records Table 2: Configuration for an example SD-AgentFoundry-3D run. Setting Value VLM backend and model Ollama; qwen3.6:27b Scene assets/kibou/KIBOU.usd, excluding the distant Earth backdrop Task Find JAXA logo and get close to it Initial pose Position (4.0,20.1)(4.0,20.1) m; yaw 180â180 ; pitch 0â0 ; eye height 1.6 m Run length 10 VLM decisions for the representative short run Camera 512Ă384512Ă 384 pixels; 70â70 vertical field of view Output First-person frames, raw and parsed decisions, pose history, configuration snapshot, and replay frames At each decision point, the simulator renders the agentâs first-person RGB image and sends that image and a dynamically assembled text prompt to a VLM. The text contains the task, current pose, and a bounded recent history. The detailed VLM prompt is shown in Appendix A.2. The model returns unrestricted natural language rather than a required action schema. A deterministic interpreter scans the reply and resolves each movement category from its latest occurrence in the text, since models that reason aloud state their conclusion at the end. It converts supported distance units to meters and extracts translation bearing, body rotation, gaze pitch, or a stay command. Configured limits clamp excessive motion, and unreadable responses are recorded without inventing a replacement action. This interface is a VLM-driven, VLA-style perceptionâdecisionâaction loop; it is not presented as a trained VLA policy. The agent can translate, rotate its body, and change its gaze within configured bounds. A live view combines a semi-transparent overview of the scene, the agent and its trajectory, and an inset re-rendered from the same camera pose and settings as the first-person image supplied to the VLM. The default backend uses a local vision-capable model through Ollama, while local Transformers, OpenAI-compatible, and scripted backends provide replaceable interfaces. Inference wall-clock time is kept separate from simulation time so that model latency does not alter the simulated dynamics. For reproducibility and auditing, every decision record preserves the raw model response, parsed command, parser notes, pre- and post-action poses, boundary effects, inference time, and the path of the input image when frame saving is enabled. The effective configuration is saved with the run. A separate replay path reconstructs overview frames from the log rather than treating the live window as a video recording. Cross-platform setup scripts are included. 3 Example Runs & Beyond The following representative runs are intended as qualitative usage examples, not as controlled evaluations or evidence of predictive validity. Their tables document the configurations and recorded outputs. Exact repetition of the current 2D example additionally requires control of its randomly generated initial positions and personas, for which the present configuration does not expose a seed. 3.1 Representative 2D LLM Run This subsection presents an illustrative 2D simulation using SD-AgentFoundry-2D. Twenty LLM agents are placed in a grid world containing two accessible places, a cafe and a library (Figure 1). Agents can converse with nearby agents by exchanging natural-language messages. Two fires occur during the run, and each agent uses locally available information and received messages to decide how to move and what to communicate to others. The scenario therefore provides a compact setting in which to observe how LLM agents respond to a shared hazard, choose between places, and interact with one another. Holding the scenario fixed while replacing the underlying LLM can also reveal model-dependent differences in agent behavior. The purpose of this example is to demonstrate the simulation workflow rather than to analyze its outcome in depth. Figure 1 presents one snapshot of the run, showing the two fires and their perception radii, the spatial distribution of the 20 agents, and the pairs eligible for local communication. Table 1 summarizes the parameters and recorded outputs associated with this example. The snapshot confirms that the decision, communication, fire-perception, logging, and visualization components operate together, but it should not be interpreted as evidence of a causal behavioral pattern or as a validated model of human response because it represents only a single illustrative run rather than a dedicated behavioral study. Related work has shown that two-dimensional or otherwise spatially explicit LLM-agent simulations can support the prototyping of populated social systems, the study of memory- and persona-conditioned behavior, and the observation of emergent interactions. Social Simulacra used simulated populations to prototype social-computing systems, while Generative Agents demonstrated socially interacting agents in a two-dimensional town, and Concordia provided a more general framework for language-mediated agent-based modeling (16; 15; 24). Spatial extensions of the El Farol Bar problem have examined collective choice under capacity constraints (23), while Sugarscape-style environments have revealed model-dependent behavior under resource scarcity and hazards (13). Larger platforms such as AgentSociety, AgentSociety 2, and CitySim further illustrate how generative agents may be used to explore collective behavior and urban dynamics (18; 19; 4). Accordingly, a simple two-dimensional world can serve both as a foundation for social-simulation experiments and as a controlled environment for studying the behavioral characteristics of LLM agents. For such studies, researchers can, for example, vary persona distributions, initial layouts, place geometry and capacity, communication distance, memory length, inference settings, and the position, timing, intensity, and perception radius of hazards. More importantly, they can hold these conditions constant while changing the LLM and compare model-specific behavioral signatures, including movement preferences, action stability, sensitivity to hazards, communication frequency, warning propagation, clustering, and collective place choice. These measurements can help identify recurring tendencies or biases in how a particular LLM behaves when instantiated as an agent. They characterize behavior within the specified simulator and prompt design, however, rather than an intrinsic human-like personality or an empirically validated prediction of human populations. Rigorous comparison would additionally require controlled random seeds, repeated trials, and validation against appropriate human or observational data, as exemplified by work that evaluates human-grounded agents against participantsâ own responses (17). 3.2 Representative 3D VLM Run This subsection presents an illustrative 3D simulation using SD-AgentFoundry-3D. The Japanese Experiment Module âKiboâ is loaded as a USD scene, and a single VLM agent, represented by a human avatar in red clothing, is instructed to find the JAXA logo and approach it (Figure 2). At each decision, the agent receives a first-person RGB image together with the task prompt and recent action history. The VLM describes the scene and proposes a movement in natural language, which the simulator interprets as a predefined translation, rotation, gaze change, or stay action. The scenario therefore provides a compact example of visual grounding and embodied decision making within a three-dimensional digital twin. The purpose of this example is to demonstrate the simulation workflow rather than to analyze task performance in depth. Figure 2 presents one snapshot of the run: the overview shows the agent and its trajectory within the Kibo geometry, while the inset shows a same-viewpoint re-rendering matched to the VLM camera pose and settings. Table 2 summarizes the parameters and recorded outputs associated with this example. The snapshot confirms that USD-scene loading, first-person rendering, VLM inference, action interpretation, movement, logging, and visualization operate together, but it should not be interpreted as a performance benchmark or as evidence of reliable task completion because it represents only a single illustrative run rather than a dedicated evaluation study. Related work has shown that visually grounded agents in three-dimensional environments can support research on navigation, interaction, and behavior in physical and social spaces. Habitat established a configurable, photorealistic simulation platform for embodied-AI tasks such as navigation and instruction following (21). Vision-and-language navigation grounded natural-language navigation instructions in visual observations of previously unseen environments (2). More recent examples include SimWorld, which provides multimodal environments for autonomous agents in open-ended physical and social scenarios (20); TravelAgent, which studies navigation and wayfinding by generative agents in built environments (14); and CrowdVLA, which addresses context-aware agent navigation and continuous locomotion in semantically structured crowd simulations (9). EmbRACE-3K further provides a benchmark for first-person, multi-step embodied reasoning and reports substantial limitations in the zero-shot task success of contemporary VLMs (12). These studies illustrate how three-dimensional simulation can connect language-based reasoning with spatial perception and action. Accordingly, a configurable USD world can serve both as a foundation for digital-twin experiments and as a controlled environment for studying the embodied behavioral characteristics of vision-language agents. For such studies, researchers can, for example, replace the Kibo asset with another USD scene and vary the task instruction, initial pose, camera field of view, movement and gaze limits, recent-history length, model backend, and response interpreter. Facilities, streets, workplaces, or other digital twins could thereby be used to study visual grounding, object search, wayfinding, viewpoint dependence, and sensitivity to language or scene layout. More importantly, researchers can hold the scene, task, and action constraints constant while changing the VLM and compare model-specific behavioral signatures, including object-recognition errors, exploration strategies, movement efficiency, action consistency, recovery from incorrect decisions, and sensitivity to viewpoint. Perspective-taking is also an explicit dimension of comprehensive VLM spatial-reasoning benchmarks (10). These measurements characterize behavior within the specified simulator, prompt, and action interface rather than general visual intelligence or real-world embodied competence. Rigorous comparison would additionally require controlled initial conditions, repeated trials, task-specific success criteria, and validation in appropriate physical or observational settings. The present VLM-driven loop should not be confused with a trained VLA policy. Here, a VLM produces natural-language text, and a deterministic interpreter maps recognized phrases onto a predefined set of translations, rotations, gaze changes, and stay actions. In contrast, VLA models such as RT-2 and OpenVLA are trained with robot trajectories or demonstrations so that visual and linguistic inputs are connected directly to embodiment-specific actions and transferable visuomotor skills (5; 11). CrowdVLA similarly introduces learned motion skills to bridge symbolic decisions and continuous locomotion in crowd simulation (9). Consequently, SD-AgentFoundry-3D is suitable for interpretable studies of perception, high-level decision making, and bounded simulated navigation, but it cannot by itself learn a new motor skill, generate joint- or end-effector-level control, handle contact-rich manipulation, or close a high-frequency sensorimotor loop. Those capabilities require a VLA or another trained low-level control policy, an embodiment-specific interface, and separate physical validation. Generative Pretrained Controllers exemplify a distinct learned low-level policy that maps reusable motion representations to physics-based character control (22). 4 Summary LLM- and VLM-driven agents make it possible to explore social interaction and embodied decision making with mechanisms that are difficult to express as fixed behavioral rules. Such simulations may inform hypothesis generation for social challenges, urban and facility planning, emergency communication, service prototyping, and other business scenarios, but their outputs require empirical validation before they can support real-world claims. The two SD-AgentFoundry foundations expose complementary levels of abstraction: SD-AgentFoundry-2D emphasizes communication and collective behavior in a transparent two-dimensional world, whereas SD-AgentFoundry-3D emphasizes visual grounding and physical action in a three-dimensional digital twin. Both systems are intentionally basic, locally operated, and designed for use on macOS, Windows, and Linux. Their purpose is educational accessibility and extensibility rather than completeness. By publishing readable implementations of the perceptionâreasoningâaction loop, configurable scenarios, visualizations, and auditable logs, we hope to lower the barrier to studying these techniques and to encourage users to add their own environments, agent architectures, models, measurements, and application-specific safeguards. Acknowledgment R.H. acknowledges financial support from JSPS KAKENHI Grant Numbers 26K00756, 23K0253, 22K14091, 21H04512, 21H04514, and 20K0080. ChatGPT was used to assist with English-language proofreading of the manuscript. References Aher et al. (2023) G. V. Aher, R. I. Arriaga, and A. T. Kalai Using large language models to simulate multiple humans and replicate human subject studies. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 202, p. 337â371. External Links: Link Cited by: §1. Anderson et al. (2018) P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. SĂŒnderhauf, I. Reid, S. Gould, and A. van den Hengel Vision-and-language navigation: interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 3674â3683. External Links: Document, Link Cited by: §3.2. Bonabeau (2002) E. Bonabeau Agent-based modeling: methods and techniques for simulating human systems. Proceedings of the National Academy of Sciences 99 (suppl. 3), p. 7280â7287. External Links: Document, Link Cited by: §1. Bougie and Watanabe (2025) N. Bougie and N. Watanabe CitySim: modeling urban behaviors and city dynamics with large-scale LLM-driven agent simulation. arXiv preprint arXiv:2506.21805. External Links: 2506.21805, Document Cited by: §1, §3.1. Brohan et al. (2023) A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. Gonzalez Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T. E. Lee, S. Levine, Y. Lu, H. Michalewski, I. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, G. Salazar, P. Sanketi, P. Sermanet, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich RT-2: vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818. External Links: 2307.15818, Document Cited by: §3.2. Driess et al. (2023) D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence PaLM-E: an embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 202, p. 8469â8488. External Links: Link Cited by: §1. Guo et al. (2024) T. Guo, X. Chen, Y. Wang, R. Chang, S. Pei, N. V. Chawla, O. Wiest, and X. Zhang Large language model based multi-agents: a survey of progress and challenges. In Proceedings of IJCAI-24, p. 8048â8057. External Links: 2402.01680, Document Cited by: §1. Huang et al. (2026) W. Huang, W. Zhang, Y. Liang, Y. Bei, Y. Chen, T. Feng, X. Pan, Z. Tan, Y. Wang, T. Wei, S. Wu, R. Xu, L. Yang, R. Yang, W. Yang, C. Yeh, H. Zhang, H. Zhang, S. Zhu, H. P. Zou, W. Zhao, S. Wang, W. Xu, Z. Ke, Z. Hui, D. Li, Y. Wu, L. He, C. Wang, X. Xu, B. Huang, J. Tan, S. Heinecke, H. Wang, C. Xiong, A. A. Metwally, J. Yan, C. Lee, H. Zeng, Y. Xia, X. Wei, A. Payani, Y. Wang, H. Ma, W. Wang, C. Wang, Y. Zhang, X. E. Wang, Y. Zhang, J. You, H. Tong, X. Luo, X. Liu, Y. Sun, W. Wang, J. McAuley, J. Zou, J. Han, P. S. Yu, and K. Shu A survey of agent memory in the second half: towards self-evolving and long-horizon agents. Transactions on Machine Learning Research. External Links: 2602.06052, Document Cited by: §2.1. Hwang et al. (2026) J. Hwang, S. Hong, J. Kim, J. Seon, G. Nam, H. Jang, and H. Kang CrowdVLA: embodied vision-language-action agents for context-aware crowd simulation. arXiv preprint arXiv:2604.05525. External Links: 2604.05525, Document Cited by: §1, §3.2, §3.2. Jia et al. (2026) M. Jia, Z. Qi, S. Zhang, W. Zhang, X. Yu, J. He, H. Wang, and L. Yi OmniSpatial: towards comprehensive spatial reasoning benchmark for vision language models. In The Fourteenth International Conference on Learning Representations, External Links: 2506.03135, Document Cited by: §3.2. Kim et al. (2025) M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn OpenVLA: an open-source vision-language-action model. In Proceedings of The 8th Conference on Robot Learning, Proceedings of Machine Learning Research, Vol. 270, p. 2679â2713. External Links: Link Cited by: §1, §3.2. Lin et al. (2025) M. Lin, W. Huang, Y. Li, C. Jiang, K. Wu, F. Zhong, S. Qian, X. Wang, and X. Qi EmbRACE-3K: embodied reasoning and action in complex environments. arXiv preprint arXiv:2507.10548. External Links: 2507.10548, Document Cited by: §3.2. Masumori and Ikegami (2025) A. Masumori and T. Ikegami Do large language model agents exhibit a survival instinct? an empirical study in a Sugarscape-style simulation. arXiv preprint arXiv:2508.12920. External Links: 2508.12920, Document Cited by: §2.1, §3.1. Noyman et al. (2024) A. Noyman, K. Hu, and K. Larson TravelAgent: generative agents in the built environment. arXiv preprint arXiv:2412.18985. External Links: 2412.18985, Document Cited by: §1, §3.2. Park et al. (2023) J. S. Park, J. C. OâBrien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein Generative agents: interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, p. 1â22. External Links: Document, 2304.03442 Cited by: §1, §3.1. Park et al. (2022) J. S. Park, L. Popowski, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein Social simulacra: creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, p. 1â18. External Links: Document, 2208.04024 Cited by: §1, §3.1. Park et al. (2024) J. S. Park, C. Q. Zou, J. Kamphorst, N. Egan, A. Shaw, B. M. Hill, C. Cai, M. R. Morris, P. Liang, R. Willer, and M. S. Bernstein LLM agents grounded in self-reports enable general-purpose simulation of individuals. arXiv preprint arXiv:2411.10109. External Links: 2411.10109, Document Cited by: §1, §3.1. Piao et al. (2025) J. Piao, Y. Yan, J. Zhang, N. Li, J. Yan, X. Lan, Z. Lu, Z. Zheng, J. Y. Wang, D. Zhou, C. Gao, F. Xu, F. Zhang, K. Rong, J. Su, and Y. Li AgentSociety: large-scale simulation of LLM-driven generative agents advances understanding of human behaviors and society. arXiv preprint arXiv:2502.08691. External Links: 2502.08691, Document Cited by: §1, §3.1. Piao et al. (2026) J. Piao, J. Zhang, H. Huang, K. Zhang, J. Y. Wang, X. Zhao, S. Li, B. Sun, J. Chang, F. Xu, C. Wang, F. Zhang, K. Rong, J. Su, T. Meng, Y. Liu, Q. Meng, Y. Wang, and Y. Li AgentSociety 2: an integrated research environment for executable social science. arXiv preprint arXiv:2607.11895. External Links: 2607.11895, Document Cited by: §1, §3.1. Ren et al. (2025) J. Ren, Y. Zhuang, X. Ye, L. Mao, X. He, J. Shen, M. Dogra, Y. Liang, R. Zhang, T. Yue, Y. Yang, E. Liu, R. Wu, K. Benavente, R. M. Nagaraju, M. Faayez, X. Zhang, D. V. Sharma, X. Zhong, Z. Ma, T. Shu, Z. Hu, and L. Qin SimWorld: an open-ended realistic simulator for autonomous agents in physical and social worlds. arXiv preprint arXiv:2512.01078. External Links: 2512.01078, Document Cited by: §1, §3.2. Savva et al. (2019) M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra Habitat: a platform for embodied AI research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 9339â9347. External Links: Document, Link Cited by: §3.2. Shi et al. (2026) Y. Shi, Y. Jiang, C. Tessler, and X. B. Peng GPC: large-scale generative pretraining for transferable motor control. In SIGGRAPH Conference Papers â26, External Links: Document, 2606.29148 Cited by: §3.2. Takata et al. (2025) R. Takata, A. Masumori, and T. Ikegami Emergent social dynamics of LLM agents in the El Farol Bar problem. arXiv preprint arXiv:2509.04537. External Links: 2509.04537, Document Cited by: §2.1, §3.1. Vezhnevets et al. (2023) A. S. Vezhnevets, J. P. Agapiou, A. Aharon, R. Ziv, J. Matyas, E. A. Duéñez-GuzmĂĄn, W. A. Cunningham, S. Osindero, D. Karmon, and J. Z. Leibo Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia. arXiv preprint arXiv:2312.03664. External Links: 2312.03664, Document Cited by: §1, §3.1. Appendix A Prompts Used in the Simulations This appendix documents the text supplied to the local models. Angle-bracketed expressions (e.g., <x>) in the templates denote values inserted at runtime. Blocks explicitly marked as conditional are omitted when the corresponding information is unavailable. Line breaks inside the framed listings are part of the prompts, whereas automatic visual wrapping does not insert additional characters. A.1 SD-AgentFoundry-2D LLM Prompts Each agent makes two sequential LLM calls during a simulation step. The first, a communication prompt (Section A.1.1), determines what the agent says to locally communicable agents and deliberately excludes their coordinates. The selected messages are delivered before movement. The second, an action-decision prompt (Section A.1.2), includes spatial coordinates and the exchanged messages and requests a movement or stay action, memory, and reasoning. These two sections reproduce the fixed text in the implementation and identify every dynamic or conditional field. Section A.1.3 then provides a reconstructed example showing how representative runtime values populate the action-decision template. A.1.1 Communication prompt template ⏠You are Agent <agent-id> (<gender>) in a 2D world with multiple places (<comma-separated unique place types>). === YOUR CURRENT STATE === Gender: <gender> In place: <Yes or No> <Current place: place-name; omitted when outside> <CONDITIONAL WHEN INSIDE A PLACE> You are currently in the <place-type> (<place-name>). Number of agents here: <agents-in-place> Capacity: <capacity> Occupancy rate: <occupancy-rate rounded to two decimals> <CONDITIONAL WHEN ONE OR MORE FIRES ARE PERCEIVED> === FIRE EVENT === <REPEATED FOR EACH PERCEIVED FIRE> Fire "<fire-name>": Position: (<fire-x>, <fire-y>) Intensity: <intensity> (scale: 0.0 to 1.0) Radius: <radius> Your distance: <distance rounded to two decimals> === NEARBY AGENTS (you can communicate with these agents) === <Agent id (gender) is in place-name (place-type), Agent id (gender) is outside the places, or No nearby agents.> === PREVIOUS MEMORY === <- one line per retained memory, or No previous experiences.> === MESSAGES FROM OTHERS === <from Agent id: message, or No messages received.> === YOUR TASK === Decide what message you want to send to nearby agents. You can share your observations, experiences, or thoughts about the places and situation. === RESPOND IN JSON === "message": "message to nearby agents (max 200 words, optional if you donât want to send a message)", "reasoning": "brief explanation of why you want to send this message" Step: <step> A.1.2 Action-decision prompt template ⏠You are Agent <agent-id> (<gender>) in a 2D world with multiple places (<comma-separated unique place types>). === YOUR CURRENT STATE === Gender: <gender> Position: (<x>, <y>) In place: <Yes or No> <Current place: place-name; omitted when outside> <CONDITIONAL WHEN INSIDE A PLACE> You are currently in the <place-type> (<place-name>). Number of agents here: <agents-in-place> Capacity: <capacity> Occupancy rate: <occupancy-rate rounded to two decimals> <CONDITIONAL WHEN ONE OR MORE FIRES ARE PERCEIVED> === FIRE EVENT === <REPEATED FOR EACH PERCEIVED FIRE> Fire "<fire-name>": Position: (<fire-x>, <fire-y>) Intensity: <intensity> (scale: 0.0 to 1.0) Radius: <radius> Your distance: <distance rounded to two decimals> === PLACE LOCATIONS === <place-name> (<place-type>): center at (<center-x>, <center-y>), covers X from <minimum-x> to <maximum-x>, Y from <minimum-y> to <maximum-y> <one line for every configured place> === NEARBY AGENTS === <Agent id (gender) is at (x, y) and is in place-name (place-type), Agent id (gender) is at (x, y) and is outside the places, or No nearby agents.> === PREVIOUS MEMORY === <- one line per retained memory, or No previous experiences.> === MESSAGES FROM OTHERS === <from Agent id: message, or No messages received.> <CONDITIONAL WHEN A MESSAGE WAS CHOSEN> === MESSAGE YOU DECIDED TO SEND === <message selected by the communication call> === AVAILABLE ACTIONS === - "stay": remain at current position - "move" with direction: "up" (Y+1), "down" (Y-1), "left" (X-1), "right" (X+1) Field boundaries: X and Y from -<half-space-size> to +<half-space-size> === RESPOND IN JSON === "action": "move" or "stay", "direction": "up", "down", "left", or "right" (only if action is "move"), "memory": "what you want to remember for the next step (your thoughts, observations, intentions)", "reasoning": "brief explanation of your decision" Step: <step> A.1.3 Example action prompt The 2D implementation does not retain the exact prompts sent to Ollama. The following is therefore not a historical record or an observed result. It is a representative prompt generated with the inspected code and current cafe/library place definitions at step 7, after fire1 has activated. Female Agent 3 is at (14,2)(14,2) in the library with two other occupants, one retained memory, one received message, nearby Agent 1, and perceived fire event fire1 at a distance of 8.06 grid units. The order in which distinct place types appear in the opening sentence can vary because the implementation derives that list through set iteration. ⏠You are Agent 3 (female) in a 2D world with multiple places (cafe, library). === YOUR CURRENT STATE === Gender: female Position: (14, 2) In place: Yes Current place: library You are currently in the library (library). Number of agents here: 3 Capacity: 10 Occupancy rate: 0.30 === FIRE EVENT === Fire "fire1": Position: (15, 10) Intensity: 0.8 (scale: 0.0 to 1.0) Radius: 15 Your distance: 8.06 === PLACE LOCATIONS === cafe (cafe): center at (-15, 0), covers X from -20 to -10, Y from -5 to 5 library (library): center at (15, 0), covers X from 10 to 20, Y from -5 to 5 === NEARBY AGENTS === Agent 1 (male) is at (16, 1) and is in library (library) === PREVIOUS MEMORY === - I entered the library to inspect its occupancy. === MESSAGES FROM OTHERS === from Agent 1: I can also see fire1 from inside the library. === MESSAGE YOU DECIDED TO SEND === I can detect fire1 near the library; please stay alert. === AVAILABLE ACTIONS === - "stay": remain at current position - "move" with direction: "up" (Y+1), "down" (Y-1), "left" (X-1), "right" (X+1) Field boundaries: X and Y from -25 to +25 === RESPOND IN JSON === "action": "move" or "stay", "direction": "up", "down", "left", or "right" (only if action is "move"), "memory": "what you want to remember for the next step (your thoughts, observations, intentions)", "reasoning": "brief explanation of your decision" Step: 7 A.2 SD-AgentFoundry-3D VLM Prompt The VLM receives a dynamically assembled text prompt together with the current first-person RGB image. Section A.2.1 presents the text-prompt template. Its recent-history block is limited by the configured memory size; each entry contains the earlier raw reply flattened to one line and truncated to at most 160 characters, the interpreterâs outcome, and the resulting pose. The final nudge is included only when the latest three retained commands are all idle. Section A.2.2 provides an example assembled for decision step 0 of the current Kibo configuration, when no action history is yet available. A.2.1 Dynamic text-prompt template ⏠<task prompt from config.yaml> Movement conventions: distances are in metres unless you name another unit; angles are in degrees. Turning left is counter-clockwise. You can look up or down, but not straight up or straight down. Your current pose: position (<x>, <y>) m, heading <yaw> deg, gaze <signed-pitch> deg. <CONDITIONAL WHEN RECENT HISTORY EXISTS> What you did recently: - step <decision-index>: you said "<flattened reply>" -> <interpreted outcome>. You were then at <pose summary>. <one line per retained history entry> <CONDITIONAL AFTER THREE IDLE DECISIONS> You have not moved for several steps. Try something different. What is your next single movement? A.2.2 Example text prompt The following is the text assembled for decision step 0 when the current task prompt in config.yaml is combined with the logged initial pose (x,y,z)=(4.0,20.1,â0.2358)(x,y,z)=(4.0,20.1,-0.2358), yaw 180â180 , and pitch 0â0 . The prompt exposes only the two-dimensional foot position, heading, and gaze. Because no earlier decision exists at step 0, there is no history block. The VLM receives this text together with the corresponding first-person RGB image. ⏠You are an agent standing inside the Japanese Experiment Module "Kibo" of the ISS. The image is your own first-person view. Task: Find JAXA logo and get close to it. Briefly say what you see and why you are moving, then state your next single movement in one sentence, LAST. Examples of the movement sentence: "move forward 2 m", "turn right 30 degrees", "look up 30 degrees", "stay". Movement conventions: distances are in metres unless you name another unit; angles are in degrees. Turning left is counter-clockwise. You can look up or down, but not straight up or straight down. Your current pose: position (4.00, 20.10) m, heading 180 deg, gaze +0 deg. What is your next single movement?