Paper deep dive
CoGrid & the Multi-User Gymnasium: A Framework for Multi-Agent Experimentation
Chase McDonald, Cleotilde Gonzalez
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 4/18/2026, 1:42:04 AM
Summary
The paper introduces CoGrid, a multi-agent grid-based simulation library with dual NumPy/JAX backends, and Multi-User Gymnasium (MUG), a platform for deploying these simulations as interactive web-based experiments. Together, these tools facilitate human-AI interaction research by supporting hardware-accelerated training and low-latency, peer-to-peer multiplayer experiments.
Entities (5)
Relation Signals (3)
CoGrid â supports â PettingZoo
confidence 100% · By adopting the PettingZoo API (Terry et al., 2021), COGRID allows researchers to build cooperative or competitive environments.
CoGrid â uses â JAX
confidence 100% · it is also built with an optional JAX (Bradbury et al., 2018) backend, enabling hardware acceleration
MUG â deploys â CoGrid
confidence 90% · MUG translates such simulation environments directly into interactive web-based experiments.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The increasing integration of artificial intelligence (AI) in everyday life brings with it new challenges and questions for regarding how humans interact with autonomous agents. Multi-agent experiments, where humans and AI act together, can offer important opportunities to study social decision making, but there is a lack of accessible tooling available to researchers to run such experiments. We introduce two tools designed to reduce these barriers. The first, CoGrid, is a multi-agent grid-based simulation library with dual NumPy and JAX backends. The second, Multi-User Gymnasium (MUG), translates such simulation environments directly into interactive web-based experiments. MUG supports interactions with arbitrary numbers of humans and AI, utilizing either server-authoritative or peer-to-peer networking with rollback netcode to account for latency. Together, these tools can enable researchers to deploy studies of human-AI interaction, facilitating inquiry into core questions of psychology, cognition, and decision making and their relationship to human-AI interaction. Both tools are open source and available to the broader research community. Documentation and source code is available at {cogrid, multi-user-gymnasium}.this http URL. This paper details the functionality of these tools and presents several case studies to illustrate their utility in human-AI multi-agent experimentation.
Tags
Links
- Source: https://arxiv.org/abs/2604.15044v1
- Canonical: https://arxiv.org/abs/2604.15044v1
Trouble viewing inline? Open PDF directly â
Full Text
73,836 characters extracted from source content.
Expand or collapse full text
CoGrid & the Multi-User Gymnasium: A Framework for Multi-Agent Experimentation Chase McDonald Department of Social and Decision Sciences Carnegie Mellon University Cleotilde Gonzalez Department of Social and Decision Sciences Carnegie Mellon University coty@cmu.edu Abstract The increasing integration of artificial intelligence (AI) in everyday life brings with it new challenges and questions for regarding how humans interact with autonomous agents. Multi-agent experiments, where humans and AI act together, can offer important opportunities to study social decision making, but there is a lack of accessible tooling available to researchers to run such experiments. We introduce two tools designed to reduce these barriers. The first, COGRID, is a multi-agent grid-based simulation library with dual NumPy and JAX backends. The second, Multi-User Gymnasium (MUG), translates such simulation environments directly into interactive web-based experiments. MUG supports interactions with arbitrary numbers of humans and AI, utilizing either server-authoritative or peer-to-peer networking with rollback netcode to account for latency. Together, these tools can enable researchers to deploy studies of human-AI interaction, facilitating inquiry into core questions of psychology, cognition, and decision making and their relationship to human-AI interaction. Both tools are open source and available to the broader research community. Doc- umentation and source code is available atcogrid, multi-user-gymnasium.readthedocs.io. This paper details the functionality of these tools and presents several case studies to illustrate their utility in human-AI multi-agent experimentation. AcknowledgementsThis research was supported by the Defense Advanced Research Projects Agency and was accomplished under Grant Number W911NF-20-1-0006 and by the NSF AI Institute for Societal Decision Making (AI-SDM), Award No. 2229881. arXiv:2604.15044v1 [cs.HC] 16 Apr 2026 1 Introduction Artificial intelligence (AI) research has made undeniable progress in producing highly capable systems, yet most benchmarks and evaluations emphasize isolated performance rather than interaction with hu- mans. In real-world settings, AI must not only be competent but also interact effectively with humansâ its main beneficiariesâand make decisions that align with and complement human decision-making pro- cesses (Russell, 2019; Wilder et al., 2020). Much of the literature has focused on agents that outperform humans in games or on established bench- marks (e.g., (FAIR)â et al. (2022), Perolat et al. (2022), and Silver et al. (2021)), with far less attention to how such agents interact with or impact human collaborators. Even when human interaction is considered, state-of-the-art AI systems are not always incorporated. For example, research on human-AI teaming often relies on surveys, rule-based expert systems, or Wizard-of-Oz paradigms rather than fully autonomous, learning-based agents (e.g., Duan et al., 2024; Salikutluk et al., 2024; Schelble et al., 2022; Zhang et al., 2021). Although early work highlights the importance of understanding human perceptions, preferences, and coordination dynamics (Carroll et al., 2020; Schmutz et al., 2024; Strouse et al., 2021), progress remains constrained by the lack of accessible platforms for conducting controlled human-AI experiments. Studying human-AI interaction poses unique challenges compared to pure simulation or human-only stud- ies. In the context of reinforcement learning, designing custom settings for single agents in pure simulation can be cumbersome (Bamford et al., 2022). Adding human interaction further complicates such inquiries (Ouyang et al., 2022), requiring the implementation of necessary infrastructure, from networking to user in- terfaces and data collection pipelines that are often non-standard for complex tasks and rebuilt from scratch for each study. These are significant barriers to entry in empirical human-AI research. To address these challenges, we present a framework comprised of two complementary tools. The first, COGRID, is a multi-agent grid-based simulation library originally inspired byMinigrid(Chevalier-Boisvert et al., 2023). The library is designed to facilitate extensibility and customization through modularized com- ponents. By adopting thePettingZooAPI (Terry et al., 2021), COGRIDallows researchers to build cooper- ative or competitive environments that fit into standardized simulation infrastructure (e.g., reinforcement learning algorithm libraries). COGRIDis built to follow the PettingZoo API using a NumPy (Harris et al., 2020) backend; however, it is also built with an optional JAX (Bradbury et al., 2018) backend, enabling hardware acceleration for fast parallelized simulations. 2 The second tool, MUG, streamlines the deployment of simulation environments into interactive web-based experiments. MUG takes Python-based environments that follow the Gymnasium or PettingZoo APIs and makes them playable in the browser without the need for game engines or following the video game de- velopment cycle (Glazer & Madhav, 2015). Environments can run on the server or directly in participantsâ browsers via Pyodide (Python compiled to WebAssembly; Team, 2021). For multiplayer experiments, it enables peer-to-peer networking with GGPO-style (Good Game Peace Out; Cannon, 2019) rollback netcode for latency correction. MUG provides common experiment infrastructureâincluding landing pages, wait- ing rooms, matchmaking, data collection, AI inference, and surveysâso that researchers do not need to rebuild these components for each study. In this paper, we detail the functionality of these tools, which we provide to the research community atcogrid, multi-user-gymnasium.readthedocs.io. We also present several case studies to demonstrate the use of both tools for designing simulation experiments and conducting both human-AI and human-human interaction experiments. In summary, our contributions are as follows: 1. We release COGRID, a multi-agent grid-based simulation library with dual NumPy/JAX backends and the PettingZoo API, supporting both rapid prototyping and hardware-accelerated training. 2. We release MUG, a platform that deploys Gymnasium and PettingZoo environments as browser- based experiments, with support for client-side execution and peer-to-peer multiplayer. 3. We demonstrate the use of both COGRIDand MUG, illustrating how they can facilitate new lines of research in human-AI interaction. 2 Related Work In recent years, there has been extensive development of simulation environments and platforms for be- havioral research. In this section, we review the relevant prior work that motivates and complements both COGRIDand MUG. Minigrid has become a standard for reinforcement learning experiments. Despite the simplicity of grid- based environments, significant complexity can arise in task formulation and dynamics (Bamford et al., 2022; Chevalier-Boisvert et al., 2023), from social dilemmas (Agapiou et al., 2022) to difficult exploration and generalization tasks (Hafner, 2021). The current work on COGRIDis largely motivated by the accessibility and ease of use of Minigrid (Chevalier-Boisvert et al., 2023): COGRIDaims to provide an equally accessible 3 multi-agent library for developing simulation environments, with the additional requirement of supporting hardware-accelerated training through a dual backend. The primary goal is to provide a platform that can be easily manipulated, extended, and customized to ask novel questions, rather than serving as a platform for standardized benchmarks. MUG aims to bridge the gap between environments that are useful for training and evaluating AI in sim- ulation, and tasks that can be used in human behavioral experiments. While there are several libraries that can be used to create environments, including COGRID, few generalized platforms exist for translating a standard-API Python-based simulation environment into an interactive experiment (Aydın et al., 2025). MUG addresses this by allowing researchers to deploy simulation environments as browser-based experi- ments, with environments running either on the server or directly in participantsâ browsers. 2.1 Multi-Agent Environments and Experiments The standard interface for the agent-environment cycle in reinforcement learning is Gymnasium (Towers et al., 2023). It defines the API for how an agent can interact with the environment; however, it does not standardize or even allow for multi-agent interactions. The vast majority of multi-agent environments augment the Gymnasium API through minor extensions or alterations, maintaining most of the interface (i.e., Bamford et al., 2022; Liang et al., 2018; Terry et al., 2021). Whereas the development of single-agent environments has largely unified around the Gymnasium API, the adoption of a multi-agent API in the reinforcement learning research community has been less uniform, although several platforms have made strides toward a standard approach. In an extension of the Gymna- sium API, the PettingZoo interface (Terry et al., 2021) has seen significant adoption. PettingZoo provides a standardized API for the agent-environment cycle in multi-agent settings, alongside a number of existing benchmark environments, such as the multi-agent Atari suite. An API for interacting with an environment is only one component of a multi-agent simulation frame- work. The other critical component is the implementation of the environment itself, including the state representation and transition dynamics. In the single agent setting, a wide range of mature frameworks and platforms support the construction and extension of environments (e.g., Chevalier-Boisvert et al., 2023; Samvelyan et al., 2021; Sukhbaatar et al., 2015). In contrast, the multi-agent domain offers substantially fewer such resources. Existing platforms typically define their own environment-creating formats, which vary considerably in complexity, abstraction level, and degree of customizability. 4 The framework most closely related to the present work isGriddly(Bamford et al., 2022). Griddly is an open-source gridworld game engine that uses its own description language, Griddly Description YAML (GDY), to configure environments and the interactions within them. It is highly extensible and supports a variety of agent interactionsâincluding both single- and multi-agent. In the present work, we make simi- lar concessions as those made by Chevalier-Boisvert et al. (2023): Griddly provides increased functionality relative to both Minigrid and COGRID. However, this increases the barrier to entry for understanding the library and developing new environments. In addition to the GDY description language for configuration, Griddly relies on a C++ core game engine, trading off ease of use and readability for efficiency. In the ap- proach we take here, we focus on lowering the barrier to entry to allow researchers to develop environments without needing significant outside knowledge. Melting Pot(Agapiou et al., 2022) provides an additional framework for grid-based multi-agent environ- ments. Its goal is to provide a tool to help in the development and evaluation of agents in multi-agent environments and their ability to learn policies that generalize to novel partners. Melting Pot includes social dilemma environments utilized in previous work (Hughes et al., 2018; Jaques et al., 2019; Leibo et al., 2017), which were used to study learning dynamics and behavior with reinforcement learning agents in temporally extended variations of classic economic games (e.g., prisonerâs dilemma and tragedy of the commons). Melting Pot relies on a combination of the Lua programming language and Python for the de- velopment of environments, resulting in an increased barrier to entry and a higher level of complexity for developing custom environments. Aside from generalized platforms for building custom environments, there are a number of specialized frameworks and one-off implementations that provide examples of desired functionality. These range from relatively simple abstractions (e.g., Skrynnik et al., 2022) to complex 3D video games (e.g., Ellis et al., 2024; Nor Ì en, 2020). 1 A particularly relevant example is the Overcooked-AI environment developed by Carroll et al. (2020). In their work, they adapted a popular collaborative video game, Overcooked, into a reinforce- ment learning environment to construct and evaluate agents for human-AI collaboration. Their setting has inspired a significant amount of follow-up work, with reimplementations and extensions of the Over- cooked environment (for example, Agapiou et al., 2022). The ad hoc nature of these implementations also demonstrates the difficulty in designing general-use multi-agent environments, a difficulty that COGRIDis designed to alleviate. 1 A number of such examples can be found through the third-party environments linked in the PettingZoo docu- mentation: https://pettingzoo.farama.org/environments/thirdpartyenvs/. 5 In the same vein as COGRID, there are existing efforts to add multi-agent functionality to Minigrid (Fickinger, 2020; Ndousse, 2020). Despite their initial efforts to provide multi-agent environments, these projects remain unmaintained and undocumented, preventing wider adoption, extension, and customiza- tion. Our work provides a complete and documented extension of Minigrid with increased customizability through modularization, as described below. A separate line of work has addressed the computational limitations of standard simulation environments through hardware acceleration. NAVIX (Pignatelli et al., 2024) reimplements Minigrid entirely in JAX, achieving substantial speedups, but retains Minigridâs single-agent design. JaxMARL (Rutherford et al., 2024) extends this idea to the multi-agent setting, implementing multi-agent environments and algorithms entirely in JAX for hardware-accelerated training. However, both NAVIX and JaxMARL are JAX-only, which limits their applicability in settings that require web-compatible environments for browser-based execution. COGRIDtakes a different approach through its dual-backend architecture: environments are written against a shared array namespace that dispatches to either NumPy or JAX at runtime, enabling the same environment code to be used for both rapid prototyping and high-throughput training. This allows COGRIDto utilize the JAX backend for accelerated training, then deploy the exact same environment to run natively in browsers using the NumPy backend in MUG. Table 1 summarizes the key dimensions along which the aforementioned environment libraries differ. LanguageMulti-agentHardware AccelerationStandard APIWeb Compatible MinigridPythonâGymnasiumâ NAVIXPythonââGymnasiumâ GriddlyGDY + C++ââCustomâ Melting PotLua + PythonââCustomâ JaxMARLPythonâCustomâ COGRIDPythonâPettingZooâ Table 1: Comparison of RL environment libraries.Languageindicates the language(s) required to implement new environments.Multi-agentindicates native support for multi-agent settings.Hardware Accelerationin- dicates support for hardware-accelerated (GPU/TPU) training.Standard APIindicates compatibility with Gymnasium or PettingZoo.Web Compatibleindicates the ability to run in a web browser (e.g., via We- bAssembly compilation). 2.2 Human-AI Interaction and Interactive Platforms In addition to training and testing autonomous agents in a pure simulation environment, there have been several efforts to provide platforms that enable humans to interact with these agents. Each platform has been developed largely to fill a specific need in the research communityâfrom interactive versions of single 6 games (Carroll et al., 2020) to frameworks for decision-making research (Balietti, 2017)âand they highlight the growing amount of research into human-AI interaction. However, existing work has identified that there is a gap for generalized platforms that take complex tasks used in simulation and allow for human in- teraction (Aydın et al., 2025). We provide an overview of some of the existing approaches and the particular niche that they fill. A generalized framework for real-time games and experiments with human participants isnodeGame (Balietti, 2017). It provides researchers with a JavaScript framework to design games for individual hu- man participants, multiple humans, and human-bot populations. It provides a framework that streamlines the creation of experiment flow, participant interaction, data collection, and much more. However, to im- plement more complex real-time games, as are commonly used in multi-agent reinforcement learning, the onus remains on the developer to integrate the game logic into the nodeGame experiment flow. Indeed, nodeGame is primarily used in settings where participants interact by selecting between alternatives (e.g., Prisonerâs Dilemma) or making allocations (e.g., Dictator Game), among other paradigms common in be- havioral research. A particularly relevant development that inspired the setup and laid the groundwork for MUG is the in- teractiveOvercooked-AIof Carroll et al., 2020. In their work, they provide an interactive demonstration of their Overcooked environment, which can be used to play the game with any combination of human and AI agents. Their framework is designed specifically for their implementation of Overcooked-AI, rather than being extensible to different environments. This lack of generality is exactly what motivated the de- velopment of MUG: we enhance their approach to provide a mapping from a generic class of simulation environments to interactive tasks, embedded within the full experiment pipelineâincluding surveys, con- dition randomization, instructions, and more. A recent and closely related effort is SHARPIE (Aydın et al., 2025), a framework for conducting experiments involving humans and reinforcement learning agents. SHARPIE provides a generic interface that wraps ex- isting Gymnasium environments and supports several human-AI interaction paradigms. While SHARPIE and MUG share similar goals of reducing barriers to human-AI experimentation, they differ in emphasis and capabilities. SHARPIE provides breadth across interaction types (e.g., reward specification, preference elicitation) but does not address the challenges of real-time, low-latency multiplayer interaction, client-side execution, or a complete and configurable experiment flow. MUG is designed specifically to address these issues to provide an experience that is scalable and accessible. 7 Similarly, CrowdPlay (Gerstgrasser et al., 2021) is a web-based platform for crowdsourcing human demon- stration trajectories in Gymnasium environments. CrowdPlay supports multi-agent games; however, it uses a server-authoritative architecture in which the environment renders frames as JPEG images that are streamed to the browser. No environment logic runs client-side. CrowdPlay provides no latency compen- sation beyond frame dropping, and its experiment lifecycle is limited (instructions and payment tracking, but no structured scene flow or surveys). Furthermore, the CrowdPlay project was deprecated in 2025 and is no longer maintained. HIPPO-Gym (Bewley et al., 2021) provides a web-based platform for human-in-the-loop RL research, al- lowing humans to interact with Gymnasium environments through a browser interface. HIPPO-Gym is limited to single-agent settings and supports only specific interaction paradigms (e.g., humans teaching RL agents), without support for multi-human or multi-agent experiments. A common limitation across these platforms is their reliance on server-authoritative architectures, where the environment runs on the server and each frame requires a network round trip to exchange actions and state. For single-player settings, this introduces latency proportional to the participantâs connection quality. For multiplayer, the problem compounds: all playersâ inputs must reach the server before the environment can step, and the updated state must be sent back to all clients. At frame rates typical of real-time tasks (e.g., 30â60 frames per second), this can make interactions unusable for participants with moderate latency. MUG addresses this through client-side execution and, for multiplayer, peer-to-peer input exchange with rollback-based synchronization, as described in Section 4. Table 2 summarizes the capabilities of existing interactive experiment platforms for RL environments. API Compat.Multi-humanClient-sideLatency comp.Exp. lifecycle Overcooked-AICustom onlyââ HIPPO-GymGymnasiumâPartial CrowdPlayGymnasiumââPartial SHARPIEGymnasiumââPartial MUGGymnasium/PettingZooâPyodideGGPOâ Table 2: Comparison of platforms for interactive experiments with simulation environments.API Compat. indicates which environment APIs are natively supported.Multi-humanindicates simultaneous interac- tion between multiple human participants.Client-sideindicates whether the environment can run in the participantâs browser.Latency Comp.indicates whether the platform provides a mechanism (e.g., rollback netcode) to mask network latency during real-time interaction.Exp. lifecycleindicates built-in support for the full experiment flow (instructions, matchmaking, surveys, data collection). As summarized in Tables 1 and 2, existing environment libraries require either non-Python languages for custom environments or lack hardware acceleration, and existing experiment platforms do not jointly sup- 8 port standard simulation environment APIs, real-time multi-human interaction, and client-side execution. COGRIDand MUG are designed to address these gaps. 3COGRID All COGRIDenvironments are partially observable Markov decision processes (POMDPs), described by the tuple(X,A,O,T,R,âŠ). Here,Xis the state space,Athe action space,Othe observation space,T: X ĂA â Xthe transition function,R:X ĂA âRthe reward function, and⊠:X â Othe observation function. We first review the components of Minigrid that we have utilized in COGRID, then detail the improvements made and the motivations behind each. Minigrid Functionality.Minigrid (Chevalier-Boisvert et al., 2023) is a single-agent grid-world library built on the Gymnasium API (Towers et al., 2023). Environments are 2DnĂmgrids in which each cell is either empty or occupied by aWorldObj. The environment tracks the agentâs position, inventory, and direction separately from the grid, rather than representing the agent as aWorldObjâthis is the primary reason, at the implementation level, that Minigrid does not support multiple agents. Minigrid provides fixed default observation, action, and reward interfaces: a partial grid view with agent direction and a mission string; a discrete action space for rotation, movement, and object interaction; and sparse rewards upon mission completion. Because Minigrid has no hardware-accelerated backend, simulation speed can become a bottleneck during large-scale training runs. NAVIX (Pignatelli et al., 2024) addresses this by reim- plementing Minigrid in JAX for hardware-accelerated simulation, but it retains the single-agent design. COGRIDâs Dual Backend.All simulation code in COGRIDoperates through a backend-agnostic ar- ray namespace (cogrid.backend.xp) that dispatches to either NumPy or JAX. When the JAX back- end is active, environment functions are automatically just-in-time (JIT) compiled, andjax.vmap can be used for batched execution across many environment instances in parallel. With the NumPy backend, the same code runs without a JAX dependency, which is critical for settings that require Pyodide-compatible libraries (e.g., for client-side execution via WebAssembly in MUG). This single- implementation approach distinguishes COGRIDfrom libraries such as JaxMARL (Rutherford et al., 2024), which target GPU-accelerated training but cannot run in the browser. A concrete illustration of the benefits of GPU acceleration is provided in Section 5.1 where we demonstrate empirical results 9 in an example environment. In practice, users can swap backends simply by changing thebackend argument in environment construction:cogrid.make("MyEnvironment", backend="numpy")or cogrid.make("MyEnvironment", backend="jax"). COGRIDAgents.COGRIDdefines agents as their ownAgentobject, inheriting from the sameGridObj class that all other environment objects do. This allows the environment to track an arbitrary number of agents within theGrid. COGRIDObservations.Rather than setting defaults for agent observations, we have chosen to customize the observation space individually for each environment. Environments specify features by name in a con- figuration file. EachFeaturesubclass provides a pure function that maps the current state to an obser- vation vector. At initialization, the selected features are composed into a single observation function that concatenates outputs and composes them into a single observation for each agent in the environment. COGRIDActions.The action space in COGRIDenvironments is, by default, similar to that of Minigrid. The default action space is discrete and consists of"turn left","turn right","move forward", "pickup or drop","toggle", and"no-op". Picking up and dropping are consolidated into a single action, with the environment logic depending on which cell the agent is facing. The"no-op"action allows agents the option to do nothing. This has a number of use cases, such as an agent waiting for another. Be- yond this default action set, we also provide direct movement actions that eliminate rotations. This allows the"turn left","turn right", and"move forward"actions to be replaced with"move left", "move right","move up"and"move down". The agent rotation is then changed to correspond with the direction moved. The motivation for this change, along with the unification of picking up and dropping, was to make the controls more intuitive for a human player if they were controlling an agent in the envi- ronment or interacting with other agents. Actions can be added or removed as long as the corresponding environment logic is added in the environment loop. The active action set is selected via the environment configuration. COGRIDRewards.Similar to the observations in COGRID, rewards have been modularized into their own class to allow increased flexibility and easy customization. EachRewardsubclass implements a computemethod that receives both the previous and current state together with the actions taken, and re- turns a per-agent reward array. COGRIDallows for abstractions that will automatically build rewards that are compatible with JAXâs JIT-compilation and parallel execution, alleviating complexity for researchers. 10 For example, we provide theInteractionRewardbase class that allows users to specify a small set of arguments to define custom rewards around basic interactions with objects. Further detail is provided in Appendix A and the online documentation. COGRIDVisualization.COGRIDretains an identical visualization scheme as Minigrid. In particular, every object in the environment has an associatedrenderfunction that adds to an RGB tile rendered at that objectâs location. It is possible to make these tiles arbitrarily complex; however, most commonly, the images are based on graphical primitives defined through Minigridâs rendering utilities (e.g., drawing circles, squares, and lines). An example visualization of a simple team-based search and rescue task 2 is shown in Figure 1. (a) A visualization of the full map from the COGRID search_rescueenvironment.Agents are repre- sented by triangles that point in their current direc- tion. In this environment, two agents must work to- gether to rescue âvictimsâ in the environment (green, red, and yellow circles). There is rubble that must be cleared by picking up a pickaxe and doors that can only be opened by picking up a key. Yellow victims can only be rescued when an agent is holding the medical kit (red square with a white cross), and red victims require both agents to rescue them simultane- ously. class GridAgent(GridObj): [...] defrender(self, tile_img): tri_fn = point_in_triangle( (0.12, 0.19), (0.87, 0.50), (0.12, 0.81), ) # Rotate based on agent direction tri_fn = rotate_fn( tri_fn, cx=0.5, cy=0.5, theta=0.5 * math.pi * self.dir ) fill_coords( tile_img, tri_fn, self.color ) (b) Every object has arenderfunction, which can be arbitrarily complex. The rendering shown here uses the Minigrid primitives for drawing a triangle for an agent, rotated based on the agentâs direction. Figure 1: Visualization of an example environment and sample code to draw theGridAgentobject. The default visualization utilizes the rendering functions developed for Minigrid. 2 This task is included in COGRIDas an adaptation of the Minimap task developed by Nguyen and Gonzalez, 2023. 11 4 Multi-User Gymnasium (MUG) While COGRIDprovides a library for building grid-based multi-agent environments, MUG serves as a library for conducting interactive experiments with such multi-agent environments. Importantly, COGRID is nota prerequisite for MUG: it is designed to be compatible with any environment that follows the same standardized API and can be customized to work with those that do not. We constructed the interface to allow experiments to be designed as a sequence of âscenesâ (that is,Scene objects). All scenes represent configurations that a researcher can fill out to create a component of their experiment. Human interaction with simulation environments is achieved through a specific type of scene: aGymScene. Here, a researcher will specify all the information related to their simulation environment, how users interact with it, and how it is displayed to a participant in their study. This includes mapping keystrokes to actions, providing a mechanism to instantiate simulation environments, specifying AI poli- cies, among much more. Behind the scenes, MUG facilitates client-server communication in order to send relevant data to the client, where a game engineâi.e., Phaserâis running to display the current state of the simulation to the participant. Any required information is then transmitted back from the client to the server to be processed and stored for analysis. 3 An important specification that can be made iswherethe environment is executed. A typical approach to allow Python-based applications to interact with web-based clients would be to have a continuous com- munication loop where all Python code is executed on the server and updated information is passed to the client on each âtick.â This is the approach of both SHARPIE and HIPPO-Gym, and it works in cases where participants have very low latency connections to the server or the tick rate of the environment is low. In cases where this communication is frequent (e.g., a simulation environment updating many times a second), network latency can make this approach ineffective. The novel approach we have developed in MUG is to execute Python-based environments directly in client browsers using Pyodide, which eliminates the need for this communication and allows researchers to serve their experiments to participants who may have high-latency connections. The distinction and general construction ofGymScenes are shown in Figure 2. Client-side execution removes the latency problem for single-player experiments. However, multiplayer introduces a new challenge. If each client runs its own copy of the environment, their states must remain synchronized. The common approach in existing work, where each client waits for all playersâ inputs 3 Data storage may take any form defined by the user through our flexible API. Trajectories are currently automati- cally exported as sequences of states and actions on the server; however, researchers can define any data storage format or location (e.g., external database). 12 before stepping, reintroduces the same latency that client-side execution was designed to avoid. To address this, MUG implements a form of GGPO (Cannon, 2019) rollback netcode, a technique originally developed for latency-sensitive fighting games. Each client runs the environment locally with a shared random seed, applies its own input immediately, and predicts the other remote playersâ input (by default, repeating their last known action). When another playerâs input arrives that differs from the prediction, the client rolls back to the last confirmed state and replays the intervening frames with the correct inputs. This process is illustrated in Figure 3. Critically, to enable this functionality the environments must be deterministic given a random seed and must implement state retrieval and setting (get_state()andset_state()methods for MUG compatibility). Beyond the ability to define interaction with simulation environments in aGymScene, we have created a collection of alternative scenes that have distinct functionalities and customization. This enables researchers to modularize their experiments and integrate customized components or pages that do not require inter- action with simulation environments. For example, static informational pages can be created alongside surveys or alternative forms of interaction. When constructing an experiment, all scenes are collected into a Stagerthat defines their orderâand any manipulation (e.g., randomization)âand the flow of the experi- ment. This is illustrated in Figure 4. Data Collection.In MUG, data collection is handled through hooks that are called throughout the envi- ronment loop. Users can defineGameCallbacks that provide hooks at each key environment event, such as episodes starting and ending, each environment step, players joining and leaving, among others. This provides functionality for data to be recorded (e.g., actions and environment state at each environment step) and saved either locally or to an external database. By default, all information returned in theinfos dictionary from the environment is logged at every step and stored on the disk. If a use case requires log- ging customized data to an external database, for example, a user would write their explicit logic into a GameCallbackthat logs data to that source at a desired frequency. Additional Functionality.Beyond the core experiment loop, MUG provides several utilities common to online behavioral research. Completion codes can be generated automatically for integration with recruit- ment platforms such as Prolific or Amazon Mechanical Turk. Scene order can be randomized or coun- terbalanced at the participant level through theStager. Participants can also be filtered on the basis of browser compatibility, device type, or measured network latency before they enter the experiment. Simi- larly, for multi-player experiments, we provide an interface for creating custom matchmaking logic. This 13 ĂĄ Ă 3 Ă” Ă s Server Client Config.Python Env. Database AI Game Engine Client-Server Loop 3 Ă Ă” Ă ĂĄ s s t a t Client Execution Ă Ă” ĂĄ s 3 Ă Ă âšs 0 , a 0 , r 0 , ..., s T , a T , r T â© Figure 2: The two ways in which we run simulation environments inGymScenes: either through a continu- ous client-server communication loop or by executing the Python environment code directly in the clientâs browser. In the former, the server maintains the environment and interactions with itâas well as all AI agentsâand simply accepts data from the user in the form of actionsa t , while providing updated statess t at every step. In the second case, the server provides all data necessaryâthrough the MUG configurationâ for the client to execute the Python code in their browser. 14 s t Ës t+1 s t+2 s t+3 T a 1 t a 1 t+1 a 1 t+2 Ëa 2 t a 2 t:t+1 a 2 t+2 s t s t+1 s t+2 T(s t , a 1 t , a 2 t ) T(s t+1 , a 1 t+1 , a 2 t+1 ) ConfirmedPredictedRe-simulated Figure 3: Illustration of GGPO rollback netcode in a two-player environment. The main timeline shows the simulation state at each tick; each transitionT(s t , a 1 t , a 2 t )âs t+1 requires both playersâ actions. Client 1âs actions (a 1 ) arrive on time at every tick. Client 2âs action att+1is delayed, so Client 1 predicts it (Ëa 2 ), producing a speculative state. When the delayed input arrives as a bundlea 2 t:t+1 att+1(green), the client rolls back to the last confirmed states t and re-simulates with the correct actions (green timeline), merging the corrected state back into the main timeline. Rollback occurs without rendering re-simulated frames and it takes place between rendered ticks, causing minimal visual disruption. can be configured based on a number of attributes, from task performance to peer-to-peer latency. Full descriptions of these features and their usage are available in the online documentation. 5 Proof of Concept: Case Studies To illustrate the functionality of COGRIDand MUG, we provide several demonstrations of experimental use cases: pure simulation, human-AI interaction, and human-human interaction. Any extension or varia- tion is possible within these regimes, including multiple humans with any number of AI agents. These case studies are not intended as definitive psychological experiments, but as proof-of-concept demonstrations showing how COGRIDand MUG come together to offer (1) environment customization, (2) integration of trained agents into human-AI studies, and (3) end-to-end browser-based experiment deployment with Python environments and variable numbers of human participants. We provide methodological details to illustrate how researchers can adapt these pipelines for their own questions. For transparency, we also report the experimental design elements used in these demonstrations (e.g., re- cruitment, task structure, compensation). These details are provided not as empirical validation, but to illustrate the methodological templates that researchers can adapt for their own studies. All experiments conducted here use server-side execution for Python environments but are also compatible with browser- based execution. 4 4 See the online documentation at multi-user-gymnasium.readthedocs.io for examples of browser-based versions of these experiments. 15 StartScene GymScene SurveyScene EndScene Stager Figure 4: MUG experiment flow using aStagerand sequence ofScenes. TheStagerdefines the flow of the experiment and also has the capabilities to manipulate the order or assignment at the participant level for experiments with multiple conditions. 16 The first use case is solely in the COGRIDenvironment, demonstrating the ease of defining a new multi- agent environment and training a reinforcement learning agent in that environment. We recreated the popular Overcooked environment originally developed by Carroll et al. (2020). In it, we trained a rein- forcement learning policy with the Proximal Policy Optimization (PPO; Schulman et al., 2017) algorithm to complete the task. This process is detailed in Appendix A.2, and the efficacy of the agent and environment is demonstrated in the following experiments. Next, we demonstrate how MUG can be utilized for interactive human studies, both with and without integration with COGRID. For the former, we demonstrate how agents trained in the COGRIDOvercooked environment can be integrated into a human-AI experiment. For the latter, we use an external environment that is not grid-based, Slime Volleyball (Ha, 2020). Lastly, we conducted studies with humans alone. In addition to human-AI experimentation, MUG enables researchers to run experiments where all actors are controlled by human players. Use cases can include data collection for imitation learning or simply collecting human data for the analysis of human-human interactions. All studies with human participants were conducted with approval from Carnegie Mellonâs Institutional Review Board, and participants were sourced from Amazon Mechanical Turk. For approximately 15 min- utes of total experiment time, participants were compensated with a $1.50 base payment and an additional $1.50 possible bonus payment based on task performance: $0.05 per point scored in Slime Volleyball and $0.03 per dish delivered in Overcooked, capped at $1.50. 5.1COGRIDOvercooked The first case study illustrates how COGRIDsupports flexible grid-based environment design and cus- tomization, and how MUG then enables us to use such environments for human-human and human-AI web-based experiments. We first demonstrate the benefits of the dual-backend approach to COGRID. Hardware acceleration in JAX is a result of being able to parallelize operations, enabling running many instances of the environment at the same time. To illustrate the magnitude of the speedup, we show the efficiency gain when increasing the number of parallel instances from 1 to 1,024 in Figure 5. Hardware acceleration offers a drastic increase in throughput, and COGRIDdemonstrates competitive performance with JaxMARL. 17 Figure 5: Environment throughput in the CoGrid Overcooked environment, comparing to the original Overcooked-AI (Carroll et al., 2020) and JaxMARL (Rutherford et al., 2024) implementations. The former has a constant rate of roughly 3,400 steps per second, while the latter scales from roughly 4,300 with a single instance to 2.9 million with 1,024 parallel instances. COGRIDâs JAX backend is competitive: scaling from 4,500 steps per second with a single instance to 5.6 million with 1,024 instances, leading to a 1.9x throughput improvement. COGRIDâs NumPy backend is by far the slowest at roughly 450 steps per second; however, it offers a a mode that enables browser-based execution via WebAssembly that is entirely absent with JaxMARL. Hardware accelerated execution was run on a single NVIDIA GeForce RTX 3090. As an illustration of COGRID, we implemented a replica of the Cramped Room layout in the Overcooked-AI environment originally developed by Carroll et al. (2020). In this cooperative task, two players coordinate to prepare and deliver onion soup, which requires coordinated sequential actions to combine ingredients, cook, and plate. Figure 6a shows the layout rendered using the basic rendering functions. This case study demonstrates how COGRIDenables researchers to reproduce complex cooperative environments with min- imal effort. The full implementation and environment details are described in the online documentation. Training an Overcooked Agent.We then trained a reinforcement learning agent in the COGRIDOver- cooked environment using RLlib (Liang et al., 2018) with the Proximal Policy Optimization (PPO; Schulman et al., 2017) algorithm. Training used self-play, 5 and the trained agent achieved an average of 7.5 dishes de- livered per episode. The training that produced the policy in the following experimentation was run using 5 Self-play is a training paradigm in multi-agent reinforcement learning where one policy controls all agents and interacts with itself. 18 the NumPy backend of COGRID, rather than the JAX-accelerated variant. An illustration of training with the JAX backend, which takes a small fraction of the time, is provided in the documentation and source repository. Human Experiments via MUG.Finally, we deployed the environment in a browser-based study using MUG. MUG allows for the specification of all required settings for experimentation, including mapping keyboard buttons to actions, creating landing pages and surveys, and specifying the environment display format. The experience for participants in the study can be seen in Figure 6b. These human experiments demonstrate how MUG can transform a COGRIDenvironment into an interactive online experiment. We conducted Human-Human and Human-AI studies through separate solicitations. We recruited 29 par- ticipants for the Human-AI study and 40 participants (20 pairs) for the Human-Human study through Amazon Mechanical Turk. Each participant completed 20 episodes of the Overcooked task with either the same human partner or a fixed reinforcement learning partner policy, as previously described. Participants first completed a consent form, then proceeded to the MUG landing page with task instructions and a start button. Participants in the Human-Human study were sent to a waiting room (minimum duration of 5 seconds) un- til they were paired with another participant. Human-AI study participants saw a simulated waiting room (randomized duration of 5-25 seconds). After completing the 20 episodes, all participants were redirected to a post-experiment questionnaire. The questionnaire asked a series of questions about relative contribu- tions and behavior, including whether or not the participant thought their partner was a human or an AI. To ensure data quality, we applied exclusion criteria (participants with inactivity at or above 97.5% or those who left the webpage for 90% of an episode), leaving 23 participants in our Human-AI study and 36 (18 pairs) in the Human-Human study. Results.These results illustrate the types of analyses enabled by data collected through MUG. The per- formance results for the Human-Human and Human-AI pairs for each episode are shown in Figure 7. The relative performance of AI-AI pairs and Human-AI pairs is consistent with the original results of Car- roll et al. (2020), with AI-AI pairs (roughly 7.5 dishes per episode) substantially outperforming human-AI pairs. Human-AI pairs showed relatively stable performance across episodes, with no statistically signif- icant change in performance over time. In contrast, human-human pairs improved steadily. The stability of human-AI performance is consistent with the findings of Carroll et al. (2020): the reinforcement learn- 19 (a) The COGRIDOvercooked Cramped Room implementation.The two chefs are represented by triangles, three yellow circles represent a stack of onions, three white circles a stack of plates, the grey cir- cle a pot, and the green square the deliv- ery zone. (b) A view of the web page that participants saw after they started the task in their browser. The graphics used to render the game are taken from Carroll et al. (2020) and their original open-source implementation of Overcooked-AI. The instructions and icons are part of a customized HTML file that is passed to the MUG configuration. The score and time left display are part of a dynamic block, updated with a Python function that returns the values to populate the display. Figure 6: The Overcooked visualization from COGRIDâs native rendering and visualized in the browser with MUG. The latter uses the assets originally used by Carroll et al. (2020). 20 Figure 7: Performance across episodes in both Overcooked studies. The reinforcement learning agent used, when playing with itself with a frame-skip of five, delivered an average of 7.5 dishes per episode (not displayed in the figure). Shaded regions represent standard error. ing agent can complete most of the task independently, so although humans likely continue to learn, their improvement is masked by the agentâs high baseline capability. In contrast, human-human pairs must develop shared strategies from scratch, and the upward trend reflects this gradual coordination process. We also show the relative contributions of humans and AI in the Human-AI study, as shown in Figure 8. We measured the number of instances where either the human or AI delivered the dish to the delivery zone. Over time, humans adapt their behavior and begin contributing a larger share of deliveries. In interacting with AI, humans appear to be more inclined to complete the dish delivery, rather than taking on the role of putting onions in the pot, even though it results in no observable change in overall task performance. The results in Figure 8 also demonstrate the types of behavioral phenomena that can be captured through studies in human-AI interaction. This kind of behavioral trendâconstant performance but varying contributionsâexemplifies psychological or cognitive questions in human-AI interaction that can be pursued with MUG and COGRID. 21 Figure 8: The relative contributions in the Human-AI study, the number of dishes delivered by the human and AI separately. 5.2 Slime Volleyball Next, to complement the Overcooked case study, we demonstrate MUGâs compatibility with environments not based on COGRID. We integrated Slime Volleyball (Ha, 2020) as a recreation of a classic 2D video game. In the game, players move laterally and jump to keep a ball off the ground on their side of the court. A game is won when the ball lands on the court of the other player, similar to volleyball. We again trained a reinforcement learning agent using the PPO algorithm. The environment was also fully ported through MUG. Full training and implementation details are provided in Appendix B. Figure 9 shows the original game visualization, as well as the MUG participant view. The experimental procedure mirrored that of the Overcooked studyâthe flow from the consent form, to instructions, waiting room, gameplay, and then the follow-up questionnaire remains unchanged. Partici- pants completed 30 episodes of the task, each consisting of a single point. We recruited 21 participants for the Human-AI condition and 22 participants (11 pairs) for the Human-Human condition from Amazon Me- chanical Turk. As before, we excluded participants for data quality. Participants were excluded in instances 22 (a) The original Slime Volleyball visualization from Ha (2020). (b) Participantsâ view of Slime Volleyball during the study, as implemented in MUG. Figure 9: The original and MUG Slime Volleyball interfaces. 23 where they exceeded a maximum RTT, 6 navigated away from the web page for 90% of the experiment, or were inactive (97.5% no-op actions). This resulted in dropping one participant from the Human-AI study. Results.Results are shown in Figure 10. Because Slime Volleyball is a zero-sum game, cumulative score is not an informative measure; instead, we report the episode length as a proxy for player skill. Longer episodes indicate a greater ability to keep the ball in play. In these short sessions, we observed little-to-no improvement over time. HumanâAI pairs maintained longer rallies than HumanâHuman pairs, reflecting the trained agentâs higher baseline skill. Indeed, AIâAI matches often reached the maximum length of 3,000 steps. Figure 11 further shows that the AI maintained higher âpossession,â keeping the ball on its side for longer periods. The relatively flat episode lengths reflect the difficulty of the task: Slime Volleyball is a fast-paced motor task requiring precise timing with unintuitive physics, and 30 episodes provides limited practice for substantial skill improvement. Human-AI pairs sustain longer rallies because the trained agent reliably returns the ball, keeping it in the air for longer. Human-human pairsâ episode length does not benefit from the high skill of the AI. Although average performance remained relatively low, some par- ticipants substantially exceeded baseline, with the longest HumanâAI match lasting 1,169 steps and the longest HumanâHuman match 538 steps. The Slime Volleyball study demonstrates that MUG is not limited to grid-based environments created with COGRID. The support for fast-paced, physics-based tasks illustrates MUGâs flexibility for running studies with rich interaction dynamics and complexity, broadening the scope of potential investigation. 6 Discussion Our contributions aim to expand the ecosystem of platforms for multi-agent research. Existing environ- ments and libraries (e.g., Agapiou et al., 2022; Carroll et al., 2020; Terry et al., 2021) have provided stan- dardized testbeds for AI agents, but they arenât designed to prioritize customization or accessibility. Sim- ilarly, work in human-AI teaming has been carried out using non-standard or bespoke experimental plat- forms that have significant barriers to entry, preventing researchers from extending them or investigating research questions with modern AI. COGRIDand MUG help to solve these problems by enabling simple customization of multi-agent environments and uniquely providing a standardized method to translate simulation environments into web-based experiments, respectively. Our framework complements existing 6 The RTT is the âround trip timeâ of a message to go from the server to the participant and back. Given the fast- paced nature of the game, poor connections result in negative experiences. 24 Figure 10: The average episode length over time in the human-AI and human-human studies of Slime Volleyball. The average episode length is an effective proxy for skill because it measures how long the ball is kept in the air, which requires competency. Shaded regions represent standard error. platforms by allowing researchers to move beyond isolated simulations and move to empirical evaluations with human participants. Our work on COGRIDextends the popular Minigrid library to the multi-agent domain, allowing researchers to easily define their own environments with minimal overhead and barriers to entry. It modularizes the components of Minigrid to enable arbitrary manipulation of environments and customizable execution logic in grid-based environments. We illustrated COGRIDâs functionality through a case study of imple- menting Overcooked (Carroll et al., 2020) in COGRID, demonstrating the streamlined process by which new objects and interactions can be added to an environment. Multi-agent environments, like those that can be designed with COGRID, provide opportunities to investigate interactions between agentsâwhich is particularly important as AI is introduced into our social world. Where other libraries are primarily focused on benchmarks and standardized environments (e.g., Petting- Zoo (Terry et al., 2021), Melting Pot (Agapiou et al., 2022), Overcooked-AI (Carroll et al., 2020)), COGRID emphasizes and prioritizes modularity, accessibility, and customizability. This focus situates COGRIDas a library that will support researchers in asking questions about novel interactions not represented by current 25 Figure 11: The average number of timesteps per episode in the Human-AI study where the ball is in pos- session of the human or AI. Possession is defined by the ball being on the corresponding side of the middle post. The AI agents have higher possession compared to the human counterparts. Shaded regions repre- sent standard error. 26 benchmarks. The primary limitation of COGRIDis the grid-based nature of the environments. Although substantial complexity can arise in grid-based interactions, it excludes physics-based or continuous control settings that may be of interest in other domains. In addition to COGRID, MUG is a unique contribution that establishes a streamlined process to translate simulation environments to interactive experiments in the browser. Where environments would typically have to be reimplemented to run in an interactive experiment, MUG provides a tool for researchers to produce a multiplayer game directly from a Gymnasium or PettingZoo environment. Through demon- strations with Overcooked and Slime Volleyball, we showed how MUG facilitates both Human-AI and Human-Human experiments without requiring bespoke reimplementations. These libraries lower techni- cal barriers and allow researchers to study human-AI interaction in an efficient and scalable manner. A significant limitation is that MUG does not currently support explicit communication between partici- pants or between humans and AI. All interaction is implicit, mediated through actions and observations in the shared environment. There is no built-in text chat or structured messaging system, which limits applicability to studies that require communication. Future work will address this by incorporating com- munication capabilities into MUG in the form of text-based chat. Furthermore, MUG is currently aimed at interactions within simulation environments, but could be extended to support complex interactions that include communication, demonstration, preference elicitation, and other advanced features. Despite these limitations, there are significant opportunities enabled by this framework. With COGRID, researchers can prototype novel multi-agent scenarios or dilemmas to answer their particular questions rather than relying on pre-defined settings. With MUG, such environments can be brought online to study how humans interact with the AI that was trained in them to answer questions on perception, preferences, and behavior in human-AI interaction. Through the introduction of COGRIDand MUG, we have demonstrated a simple setting to develop multi- agent simulation environments and a framework that streamlines the process of developing interactive experiments in simulation environments. The goal of this work is to support research in multi-agent sys- tems, particularly in human-AI interaction and complementarity. By providing tools that support the rapid development and testing of ideas, we hope to contribute to progress in building autonomous agents that better serve the humans with whom they interact in the real world. 27 Open Practices Statement All source code, documentation, and tutorials for COGRIDand MUG are available at cogrid.readthedocs.io and multi-user-gymnasium.readthedocs.io, respectively. The scripts to recreate the experiments are avail- able in the respective code repositories. Declarations FundingThis research was supported by the Defense Advanced Research Projects Agency and was ac- complished under Grant Number W911NF-20-1-0006 and by the NSF AI Institute for Societal Decision Making (AI-SDM), Award No. 2229881. Conflicts of InterestThe authors have no competing interests to declare that are relevant to the content of this article. Ethics ApprovalEthics approval was received by the Institutional Review Board at Carnegie Mellon Uni- versity under the projectBuilding Human-Machine Shared Mental Models, studySTUDY2021 00000324. Consent to ParticipateInformed consent was obtained from all individual participants. Consent for PublicationThe participants gave their consent for the publication of their behavioral data. AvailabilityofDataandMaterialsDataareavailablethroughcogrid.readthedocs.ioand multi-user-gymnasium.readthedocs.io. Code AvailabilityAll source code and documentation are available through cogrid.readthedocs.io and multi-user-gymnasium.readthedocs.io. References Agapiou, J. P., Vezhnevets, A. S., Du Ì e Ì nez-Guzm Ì an, E. A., Matyas, J., Mao, Y., Sunehag, P., K Ì oster, R., Madhushani, U., Kopparapu, K., Comanescu, R., et al. (2022). Melting pot 2.0.arXiv preprint arXiv:2211.13746. 28 Aydın, H., Godin-Dubois, K., Braz, L. G., Hengst, F. d., Baraka, K., C ̧ elikok, M. M., Sauter, A., Wang, S., & Oliehoek, F. A. (2025). Sharpie: A modular framework for reinforcement learning and human-ai interaction experiments.arXiv preprint arXiv:2501.19245. Balietti, S. (2017). Nodegame: Real-time, synchronous, online experiments in the browser.Behavior research methods,49, 1696â1715. Bamford, C., Huang, S., & Lucas, S. (2022, July). Griddly: A platform for AI research in games [Number: arXiv:2011.06363 arXiv:2011.06363 [cs]]. https://doi.org/10.48550/arXiv.2011.06363 Bewley, T., Lawry, J., & Richards, A. (2021). Hippo gym: Facilitating human-in-the-loop reinforcement learning experiments.Workshop on Reinforcement Learning Tools and Environments, AAAI. Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., & Zhang, Q. (2018).JAX: Composable transformations of Python+NumPy programs(Version 0.3.13). http://github.com/jax-ml/jax Cannon, T. â. (2019). GGPO: Good game, peace out rollback network sdk. Carroll, M., Shah, R., Ho, M. K., Griffiths, T. L., Seshia, S. A., Abbeel, P., & Dragan, A. (2020). On the Utility of Learning about Humans for Human-AI Coordination [arXiv: 1910.05789].arXiv:1910.05789 [cs, stat]. Retrieved July 6, 2021, from http://arxiv.org/abs/1910.05789 Chevalier-Boisvert, M., Dai, B., Towers, M., de Lazcano, R., Willems, L., Lahlou, S., Pal, S., Castro, P. S., & Terry, J. (2023, June). Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks [Number: arXiv:2306.13831 arXiv:2306.13831 [cs]]. https : //doi.org/10.48550/arXiv.2306.13831 Duan, W., Zhou, S., Scalia, M. J., Yin, X., Weng, N., Zhang, R., Freeman, G., McNeese, N., Gorman, J., & Tolston, M. (2024). Understanding the evolvement of trust over time within human-ai teams.Pro- ceedings of the ACM on Human-Computer Interaction,8(CSCW2), 1â31. Ellis, B., Cook, J., Moalla, S., Samvelyan, M., Sun, M., Mahajan, A., Foerster, J., & Whiteson, S. (2024). Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning.Advances in Neural Information Processing Systems,36. (FAIR)â , M. F. A. R. D. T., Bakhtin, A., Brown, N., Dinan, E., Farina, G., Flaherty, C., Fried, D., Goff, A., Gray, J., Hu, H., et al. (2022). Human-level play in the game of diplomacy by combining language models with strategic reasoning.Science,378(6624), 1067â1074. Fickinger, A. (2020). Multi-agent gridworld environment for openai gym. 29 Gerstgrasser, M., Trivedi, R., & Parkes, D. C. (2021). CrowdPlay: Crowdsourcing human demonstration data for offline learning in Atari games. Retrieved February 20, 2022, from https : / / openreview. net/forum?id=qyTBxTztIpQ Glazer, J., & Madhav, S. (2015).Multiplayer game programming: Architecting networked games. Addison-Wesley Professional. Ha, D. (2020). Slime volleyball gym environment. Hafner, D. (2021). Benchmarking the spectrum of agent capabilities.arXiv preprint arXiv:2109.06780. Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M. H., Brett, M., Hal- dane, A., del R Ì Ä±o, J. F., Wiebe, M., Peterson, P., . . . Oliphant, T. E. (2020). Array programming with NumPy.Nature,585(7825), 357â362. https://doi.org/10.1038/s41586-020-2649-2 Hughes, E., Leibo, J. Z., Phillips, M., Tuyls, K., Due Ì nez-Guzman, E., Garc Ì Ä±a Casta Ì neda, A., Dunning, I., Zhu, T., McKee, K., Koster, R., et al. (2018). Inequity aversion improves cooperation in intertemporal social dilemmas.Advances in neural information processing systems,31. Jaques, N., Lazaridou, A., Hughes, E., Gulcehre, C., Ortega, P., Strouse, D., Leibo, J. Z., & De Freitas, N. (2019). Social influence as intrinsic motivation for multi-agent deep reinforcement learning.Inter- national conference on machine learning, 3040â3049. Leibo, J. Z., Zambaldi, V., Lanctot, M., Marecki, J., & Graepel, T. (2017). Multi-agent reinforcement learning in sequential social dilemmas.arXiv preprint arXiv:1702.03037. Liang, E., Liaw, R., Nishihara, R., Moritz, P., Fox, R., Goldberg, K., Gonzalez, J., Jordan, M., & Stoica, I. (2018). Rllib: Abstractions for distributed reinforcement learning.International conference on machine learning, 3053â3062. Ndousse, K. (2020). Marlgrid. Nguyen, T. N., & Gonzalez, C. (2023). Minimap: An interactive dynamic decision making game for search and rescue missions.Behavior Research Methods, 1â22. Nor Ì en, J. F. W. (2020). Derk gym environment. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022). Training language models to follow instructions with human feedback. Advances in neural information processing systems,35, 27730â27744. 30 Perolat, J., De Vylder, B., Hennes, D., Tarassov, E., Strub, F., de Boer, V., Muller, P., Connor, J. T., Burch, N., Anthony, T., et al. (2022). Mastering the game of stratego with model-free multiagent reinforcement learning.Science,378(6623), 990â996. Pignatelli, E., Liesen, J., Lange, R. T., Lu, C., Castro, P. S., & Toni, L. (2024). Navix: Scaling minigrid environ- ments with jax.arXiv preprint arXiv:2407.19396. Russell, S. (2019).Human compatible: Ai and the problem of control. Penguin Uk. Rutherford, A., Ellis, B., Gallici, M., Cook, J., Lupu, A., Ingvarsson, G., Willi, T., Hammond, R., Khan, A., de Witt, C. S., Souly, A., Bandyopadhyay, S., Samvelyan, M., Jiang, M., Lange, R. T., Whiteson, S., Lac- erda, B., Hawes, N., Rockt Ì aschel, T., . . . Foerster, J. N. (2024). Jaxmarl: Multi-agent rl environments and algorithms in jax.The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Salikutluk, V., Sch Ì opper, J., Herbert, F., Scheuermann, K., Frodl, E., Balfanz, D., J Ì akel, F., & Koert, D. (2024). An evaluation of situational autonomy for human-ai collaboration in a shared workspace setting. Proceedings of the CHI Conference on Human Factors in Computing Systems, 1â17. Samvelyan, M., Kirk, R., Kurin, V., Parker-Holder, J., Jiang, M., Hambro, E., Petroni, F., K Ì uttler, H., Grefen- stette, E., & Rockt Ì aschel, T. (2021). Minihack the planet: A sandbox for open-ended reinforcement learning research.arXiv preprint arXiv:2109.13202. Schelble, B. G., Flathmann, C., McNeese, N. J., Freeman, G., & Mallick, R. (2022). Letâs think together! assess- ing shared mental models, performance, and trust in human-agent teams.Proceedings of the ACM on Human-Computer Interaction,6(GROUP), 1â29. Schmutz, J. B., Outland, N., Kerstan, S., Georganta, E., & Ulfert, A.-S. (2024). Ai-teaming: Redefining collab- oration in the digital era.Current Opinion in Psychology, 101837. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Al- gorithms [arXiv: 1707.06347].arXiv:1707.06347 [cs]. Retrieved August 24, 2021, from http://arxiv. org/abs/1707.06347 Silver, D., Singh, S., Precup, D., & Sutton, R. S. (2021). Reward Is Enough.Artificial Intelligence, 103535. https://doi.org/10.1016/j.artint.2021.103535 Skrynnik, A., Andreychuk, A., Yakovlev, K., & Panov, A. I. (2022). Pogema: Partially observable grid envi- ronment for multiple agents. https://doi.org/10.48550/ARXIV.2206.10944 31 Strouse, D. J., McKee, K. R., Botvinick, M., Hughes, E., & Everett, R. (2021). Collaborating with Humans without Human Data [arXiv: 2110.08176].arXiv:2110.08176 [cs]. Retrieved November 29, 2021, from http://arxiv.org/abs/2110.08176 Sukhbaatar, S., Szlam, A., Synnaeve, G., Chintala, S., & Fergus, R. (2015). Mazebase: A sandbox for learning from games.arXiv preprint arXiv:1511.07401. Team, T. P. D. (2021, August).Pyodide/pyodide(Version 0.29.3). Zenodo. https://doi.org/10.5281/zenodo. 5156931 Terry, J., Black, B., Grammel, N., Jayakumar, M., Hari, A., Sullivan, R., Santos, L. S., Dieffendahl, C., Horsch, C., Perez-Vicente, R., et al. (2021). Pettingzoo: Gym for multi-agent reinforcement learning.Ad- vances in Neural Information Processing Systems,34, 15032â15043. Towers, M., Terry, J. K., Kwiatkowski, A., Balis, J. U., Cola, G. d., Deleu, T., Goul Ì ao, M., Kallinteris, A., KG, A., Krimmel, M., Perez-Vicente, R., Pierr Ì e, A., Schulhoff, S., Tai, J. J., Shen, A. T. J., & Younis, O. G. (2023, March). Gymnasium. https://doi.org/10.5281/zenodo.8127026 Wilder, B., Horvitz, E., & Kamar, E. (2020). Learning to complement humans.arXiv preprint arXiv:2005.00582. Zhang, R., McNeese, N. J., Freeman, G., & Musick, G. (2021). â an ideal humanâ expectations of ai teammates in human-ai teaming.Proceedings of the ACM on Human-Computer Interaction,4(CSCW3), 1â25. A Overcooked Environment & Task The full implementation of our adaptation of the Overcooked-AI environment (Carroll et al., 2020) is docu- mented at cogrid.readthedocs.io. In this section, we provide an overview of the design of the environment and interactive task. All code to use the COGRIDOvercooked environment with MUG is available as an example in the MUG repository, linked at: multi-user-gymnasium.readthedocs.io. A.1COGRIDOvercooked The Overcooked environment is implemented by subclassingCoGridEnvand defining the task-specific components through COGRIDâs registration system. Each componentâobjects, rewards, and featuresâis a Python class that is registered at initialization. The environment layout is specified via an ASCII grid encoding in the configuration file. Full source code and implementation details are available in the online documentation at cogrid.readthedocs.io. 32 Listing 1 shows theOnionobject, illustrating how environment objects are defined.The @register_object_typedecorator registers the class within the environment scope. Interaction ca- pabilities are declared as class attributes using thewhen()descriptorâhere,can_pickup = when()in- dicates that onions can always be picked up. Each object specifies a uniquecharattribute used for ASCII layout encoding. 1@register_object_type("onion", scope="overcooked") 2class Onion(GridObj): 3color = Colors.Yellow 4char = "o" 5can_pickup = when() Listing 1: TheOnionobject. The@register_object_typedecorator registers the class and thewhen() descriptor declares that onions can be picked up. Additional objects (Plate,OnionSoup,DeliveryZone, etc.) follow the same pattern. Rewards are defined using theInteractionRewardbase class. Instead of writing complex reward logic, subclasses declare the triggering conditions as class attributes: theactionthe agent must perform, the item it musthold, and the object it mustface. The base class handles all condition checking and reward broadcasting. Listing 2 shows the delivery reward, which triggers when an agent performs a pickup/- drop action while holding anOnionSoupand facing aDeliveryZone. For more complex conditions (e.g., checking pot capacity or cooking status) subclasses can override anextra_condition()method to specify additional conditions. 1class OnionSoupDeliveryReward(InteractionReward): 2action = "pickup_drop" 3holds = "onion_soup" 4faces = "delivery_zone" Listing 2: The soup delivery reward using the declarativeInteractionRewardbase. Only class attributes are needed: the action, what the agent holds, and what it faces. Features for the observations follow a similar pattern, where users can define custom features by subclass- ing theFeaturebase class, and generate a function that maps from the current state of the environment to specific view of the environment (e.g., a one-hot encoding of which item an agent is holding). The full details and implementations are shown in the online documentation. 33 1class OnionInPotReward(InteractionReward): 2action = "pickup_drop" 3holds = "onion" 4faces = "pot" 5 6defextra_condition(self, mask, prev_state, 7fwd_r, fwd_c, reward_config): 8# Check pot has capacity for another onion 9returncondition_satisfied Listing 3: A reward with an additional condition. Theextra_condition()override narrows the trigger mask to check pot capacity and ingredient compatibility. For theOvercookedenvironment, the observation features roughly correspond to those used by Carroll et al., 2020. For each agentj, we calculate the following features. The observation is then the concatenation of all agentsâ feature arrays: âą Agentjâs direction as a one-hot encoding. âą Agentjâs inventory as a one-hot encoding of the possible inventory objects. âą A multi-hot indicator of whether agentjis adjacent to a counter. âą Agentjâs distance to the closest of eachOnion,Plate,PlateStack,OnionStack,OnionSoup, andDeliveryZone. âą Agentjâs pot features, consisting of: 1. an indicator of whether the pot is reachable, 2. a one-hot representation of the pot status, which can be empty, cooking, or ready; 3. the number of onions in the pot, 4. the number of cooking timesteps remaining for the pot, 5. an array of the row and column distances to the pot, 6. the row and column location of the pot. âą Agentjâs distance to the other chef. âą Agentjâs row and column position in the grid. A.2 Training a Reinforcement Learning Agent We train a reinforcement learning agent in theOvercookedenvironment using RLlib (Liang et al., 2018) and the PPO algorithm. A dish delivery reward of 1.0 is given when a dish is delivered, and agents also 34 receive 0.1 reward when an onion is placed in a pot and 0.3 when a dish is plated. The full training script, including all hyperparameters, is available in the online documentation and source code. We also provide a complete example for training with full hardware acceleration (using the JAX backend). The trained policy delivers an average of roughly 7.5 dishes per episode when controlling both agents. A.3 Overcooked in MUG The full source code is available in the examples of the MUG repository, documented at: multi-user-gymnasium.readthedocs.io. The online documentation provides all implementation details and configurations to launch Overcooked in MUG. B Slime Volleyball Environment & Task The Slime Volleyball environment we use was originally developed by Ha, 2020. We adapted their im- plementation to conform to the Gymnasium API, but did not make any major alterations to the environ- ment dynamics. The code for the Gymnasium formatted environment is available at https://github.com/ chasemcd/slimevolleygym. The observationo t âOis a size 12 vector with the(x, y)-position and(x, y)-velocity for each player and the ball. We use a modified training regime for Slime Volleyball with a self-play curriculum. The agent trains against a fixed version of itself, which is updated every time the average reward against that agent exceeds 0.5â agents receive a reward of -1 for losing, 0 for ties, and 1 for winning. The full training script, including all hyperparameters, for Slime Volleyball are provided in the MUG documentation and source code. Over the course of training, the policy is able to achieve an average episode length of nearly the maximum of 3,000 steps in the Slime Volleyball environment. B.1 Slime Volleyball in MUG The full configuration and implementation details to launch Slime Volleyball in MUG are documented at multi-user-gymnasium.readthedocs.io. 35