Paper deep dive
When Does Opponent Modeling Pay Off? Spatial Inference Stabilises Reinforcement Learning under Partial Observability
Marcel Petran, Stefan Edelkamp
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/14/2026, 9:14:02 AM
Summary
This paper introduces a Spatial Opponent Model (Spatial OM) that utilizes a transformer to infer opponent subgoal heatmaps from partial observations in competitive foraging games. This heatmap is fused early into a dueling double DQN agent. The study demonstrates that this approach stabilizes training in partially observable stochastic games (POSG), prevents the collapse seen in plain DQN, and improves zero-shot transfer to unseen opponents. The benefits are governed by boundary conditions: visibility (fog thickness), opponent intent, and resource density.
Entities (8)
Relation Signals (6)
Marcel Petran → authored → When Does Opponent Modeling Pay Off? Spatial Inference Stabilises Reinforcement Learning under Partial Observability
confidence 99% · 2415 Marcel Petran Affiliation: Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic
Spatial Opponent Model → fusedinto → Dueling Double DQN
confidence 95% · outputs a subgoal heatmap from the opponent's observed trajectory, fused early into a dueling double DQN.
Spatial Opponent Model → uses → Transformer
confidence 95% · We study a spatial opponent model: a transformer trained with hindsight-relabelled opponent goals
Spatial Opponent Model → appliedto → Partially Observable Stochastic Game
confidence 92% · Here we adapt it to partial observability... In symmetric constant-sum foraging games with partial observability, the module stabilises training.
Spatial Opponent Model → stabilizes → Training
confidence 90% · the module stabilises training. Plain TD(0) DQN suffers degenerate seeds... whereas the opponent-model agent never collapses
Spatial Opponent Model → uses → Hindsight Target Generation
confidence 90% · we generate labels post-hoc with Hindsight Target Generation
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Deep reinforcement learning agents in partially observable competitive games must act on beliefs about an unseen opponent's intent. We study a spatial opponent model: a transformer trained with hindsight-relabelled opponent goals that outputs a subgoal heatmap from the opponent's observed trajectory, fused early into a dueling double DQN. In symmetric constant-sum foraging games with partial observability, the module stabilises training. Plain TD(0) DQN suffers degenerate seeds in three of four one-step configurations, whereas the opponent-model agent never collapses and, wherever training succeeds, recovers the majority of the symmetric game value. Transfers zero-shot to a foraging opponent unseen by the learner, with a positive goal advantage over the baseline, and loses this edge exactly where an information argument predicts: against a full-information tracking adversary whose intent is the learner itself, and on resource-dense maps where both agents saturate. A blindfold ablation, zeroing the heatmap at deployment, decomposes the advantage into two mechanisms: training-time stabilisation, which dominates where the credit assignment is hard and survives blinding on the dense map, and deployment-time inference, whose reliance shrinks as fog thins on the sparse map. We conclude that opponent-goal inference is a stability-and-transfer mechanism whose value is governed by measurable boundary conditions: visibility (fog thickness), opponent intent, and resource density.
Tags
Links
- Source: https://arxiv.org/abs/2610.18229v1
- Canonical: https://arxiv.org/abs/2610.18229v1
Trouble viewing inline? Open PDF directly →
Full Text
51,828 characters extracted from source content.
Expand or collapse full text
M. Baldoni, F. Fang, W. Yeoh, N. Yorke-Smith (eds.) Paper Track When Does Opponent Modeling Pay Off? Spatial Inference Stabilises Reinforcement Learning under Partial Observability Conference: Proc. of the 26th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2027); 3–7 May 2027; Hanoi, Vietnam2415 Marcel Petran Affiliation: Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic and Stefan Edelkamp Affiliation: Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic 2027 Abstract. Deep reinforcement learning agents in partially observable competitive games must act on beliefs about an unseen opponent’s intent. We study a spatial opponent model: a transformer trained with hindsight-relabelled opponent goals that outputs a subgoal heatmap from the opponent’s observed trajectory, fused early into a dueling double DQN. In symmetric constant-sum foraging games with partial observability, the module stabilises training. Plain TD(0) DQN suffers degenerate seeds in three of four one-step configurations, whereas the opponent-model agent never collapses and, wherever training succeeds, recovers the majority of the symmetric game value. Transfers zero-shot to a foraging opponent unseen by the learner, with a positive goal advantage over the baseline, and loses this edge exactly where an information argument predicts: against a full-information tracking adversary whose intent is the learner itself, and on resource-dense maps where both agents saturate. A blindfold ablation, zeroing the heatmap at deployment, decomposes the advantage into two mechanisms: training-time stabilisation, which dominates where the credit assignment is hard and survives blinding on the dense map, and deployment-time inference, whose reliance shrinks as fog thins on the sparse map. We conclude that opponent-goal inference is a stability-and-transfer mechanism whose value is governed by measurable boundary conditions: visibility (fog thickness), opponent intent, and resource density. Keywords: opponent modeling, partial observability, multi-agent reinforcement learning, goal recognition 1. Introduction Competitive foraging (Christianos et al., 2020) under fog (Figure 1) is a Partially Observable Stochastic Game (POSG) in which the pivotal latent is not where the opponent is, but where it is going. Classic DQN-style learners are hard to train in this regime: delayed sparse rewards combined with one-step temporal-difference learning (TD(0)) destabilise training. The ability to reason about others’ behaviour can often be a defining factor between success and failure (Rabinowitz et al., 2018; Nashed and Zilberstein, 2022; Al-Shedivat et al., 2018; He and Boyd-Graber, 2016; Papoudakis et al., 2021; Yu et al., 2024; Raileanu et al., 2018). Relying solely on reactive policies is often insufficient. Instead, a prominent approach is to model the adversary directly, actively inferring their underlying strategies, intentions, or goals to anticipate future states. This approach is generally known as Opponent Modeling (OM). OM is usually evaluated only as “beats the baseline,” so we ask when it helps. The spatial opponent model and its ESS formalism were introduced in prior work (Petran, 2026) for Fully Observable Stochastic Games (FOSG) (Littman, 1994). Here we adapt it to partial observability, replacing privileged inputs with the fogged observation history and its first-order motion encodings, and fusing the inferred heatmap alongside learned belief channels. Using this adapted module, we contribute a boundary-condition analysis of when and why opponent-goal inference pays off. Figure 1. The two foraging maps (11×1111×11). map3 (left): five densely placed central goals; map4 (right): seven goals dispersed behind sectioned walls. Agents and foods spawn at fixed positions and compete for the resources. Our contributions are: POSG instantiation of the spatial OM: The opponent model of prior work (Petran, 2026) re-instantiated for fogged observation histories, with motion-difference channels, hindsight supervision, and early spatial fusion in physical coordinates. Stability: Across all training runs on two maps, the opponent-model agent never collapses, whereas plain DQN has degenerate seeds in three of four one-step configurations (every one-step configuration of the sparse map, and thick fog on the dense map). In-distribution advantage: Where training succeeds, the opponent model recovers 95–99.9% of the symmetric game value against the training opponent (occasionally exceeding it by exploiting the suboptimal heuristic), versus 77–88% for the baseline on map4 and 52–107% per seed on map3, with degenerate TD(0) training runs finishing as low as 47%. Generalisation across forager types: The paired advantage transfers to a foraging opponent unseen by the learner (+0.31+0.31–0.520.52) and, in well-trained scenarios, vanishes against a tracking adversary. Boundary conditions: The advantage disappears against opponents whose intent is the learner itself and saturates when resources are dense. Mechanism: A blindfold ablation decomposes the advantage into training-time stabilisation and deployment-time inference, and shows inference reliance scales with fog. 2. Related Work Opponent modeling has a long history, from theory-of-mind-style agent modeling (Rabinowitz et al., 2018; Premack and Woodruff, 1978; Albrecht and Stone, 2018) to game-specific predictors (Bakkes et al., 2009; He and Boyd-Graber, 2016). Nashed and Zilberstein survey opponent-modeling methods and their design axes (Nashed and Zilberstein, 2022). Closest to us, Yu et al. infer opponent subgoals and integrate them into policy learning (Yu et al., 2024); their subgoal inference conditions on the full state, whereas we infer a spatial intent distribution from partial observations, and ask when the inference channel pays off rather than whether it improves a benchmark. Goal and plan recognition share our prediction target (Charniak and Goldman, 1993); our hindsight-relabelled transformer (Section 4) is a lightweight online variant of such methods, using realised opponent outcomes as supervision in the spirit of HER (Andrychowicz et al., 2017). Under partial observability, agent modeling has been tackled with recurrent opponent embeddings (Papoudakis et al., 2021), Bayes-adaptive meta-learners (Zintgraf et al., 2020), and policy-inference networks (Hong et al., 2017); these predict opponent policies or types, not their spatial intent. Our games are partially observable stochastic games (Hansen et al., 2004), and our contributions are orthogonal to MARL algorithmics: we modify the observation pipeline of a standard dueling DDQN (Mnih et al., 2015; van Hasselt et al., 2016; Wang et al., 2016) and measure the effect on training stability and transfer. 3. Problem Formulation Definition 3.1 (Partially Observable Stochastic Game). A Partially Observable Stochastic Game (POSG) is a tuple Γ=⟨,,,,Ri,,Zi,γ⟩ = ,S,A,T,R_i,O,Z_i,γ where: • N is the set of N agents. • S is the global state space. • =1×⋯×NA=A_1×·s×A_N is the joint action space. • :×→Δ()T:S×A→ (S) is the state transition function conditioned on joint actions, where Δ() (S) denotes the probability simplex over the state space. • Ri:×→ℝR_i:S×A , i∈i , are the per-agent reward functions. • O is the set of observation spaces =1×⋯×NO=O_1×·s×O_N. • Zi:→iZ_i:S _i, i∈i , are the per-agent observation functions mapping the global state to agent i’s observation space. • γ∈[0,1)γ∈[0,1) is the discount factor. Foraging as a POSG We model competitive foraging under partial observability as a two-player constant-sum POSG Γ=⟨,, = ,S, ,,A,T, Ri,,Zi,γ⟩R_i,O,Z_i,γ with agents =1,2N=\1,2\, S that represents the global state configurations that describe the positions of all agents, walls, foods, and the food’s state (present or eaten). The agents share the movement action set =North, South, East, WestA=\North, South, East, West\. T is deterministic, meaning that the movement actions update the agent’s coordinates to the adjacent cell, agents automatically collect items upon entering a cell containing food and if agent i∈i takes an action that would result in a collision with a wall or stepping outside the boundaries of the map, agent i stays at its current position. Each episode places both agents and a finite set of food goals on a grid. Agent i receives a reward ri=1r^i=1 if and only if it moves onto a cell containing food while the opponent is not present on that exact cell. If both agents enter the same cell containing food simultaneously, the food is split, yielding a reward of ri=0.5r^i=0.5 for each agent. In all other scenarios, including empty movements and wall collisions, the step reward is ri=0r^i=0. Agent i does not observe the true state st∈s_t : it receives a local observation ot(i)=Zi(st)o^(i)_t=Z_i(s_t) restricted to a vision radius v around its position, with cells outside the radius masked. Agents are also unable to see through walls. All agents additionally maintain belief channels btb_t (estimated food and opponent positions, updated on sightings), initialised from the environment’s opening state. The game is a partially observable multi-agent variant of the travelling-salesman problem, making the grids minimal instances of the problem class we ultimately target. Explicit Sequential Subgoal Games The ESS formalism and the spatial opponent model below were introduced in prior work (Petran, 2026); we summarise them for completeness and describe our partial-observability adaptation. We now characterise the class of games for which inferring a spatial subgoal is the right prediction target. Many tasks are hierarchical: before winning, an agent must complete a sequence of intermediate objectives, or subgoals. Predicting the opponent’s full future state is unnecessarily noisy: the state carries contextual information that constrains movement but does not define the objective. We therefore decompose the state space as a Cartesian product =×S=P×Q where P is the subgoal feature space, the state variables whose values complete the atomic objective (e.g. an agent’s coordinates), and Q is the context feature space, variables that constrain behaviour but do not define the objective (walls, free space). Let ρ:→ρ:S , ρ(s)=pρ(s)=p for s=(p,q)s=(p,q), be the projection onto P. Let +=s∈∣R(s)>0S^+=\s R(s)>0\ be the rewarding states. We define the set of explicit subgoals =ρ[+]⊆G=ρ[S^+] . In the foraging POSG, P is the pair of agent coordinates, Q collects walls and remaining food, and G is the finite set of food coordinates. Definition 3.2 (ESS game). A POSG is classified as an ESS game if and only if there exists a non-trivial decomposition of the state space such that =×S=P×Q, and the resulting set of distinct subgoals G is finite. 4. The Spatial Opponent Model under Partial Observability The Spatial Opponent Model (Spatial OM) was introduced for fully observable games in prior work (Petran, 2026). Relative to that instantiation, partial observability requires three changes: (i) inference over a sliding window of fogged observations rather than the full state, (i) motion-difference channels as the primary sighting signal, and (i) offline pre-training; in preliminary runs without it, the fusion channel is ignored by the Q-network and does not recover. The prediction target is unchanged: the adversary’s physical destination as a dense heatmap over the grid, with the ESS formalism restricting predictions to meaningful destinations. Figure 2. Overview of the Transformer-based Spatial Opponent Model. The Transformer Encoder processes the sequence of past observations, attending to the current observation to output a 2D heatmap representing the inferred opponent subgoals. The Feature Embedder is detailed in Figure 3. The diagram also shows expected tensor shapes where: B is the batch dimension, T is the trajectory dimension, F is the environment Feature dimension, H and W are the environment Height and Width dimensions, and dmodeld_model is the embedding output dimension. Inference Model The core of our module is the inference model p, parameterised by ϕφ (visualised in Figure 2). It approximates the opponent’s intended spatial destination at timestep t, given the learner’s observation history and current observation, as a raw logit map G^logits,t∈ℝH×W G_logits,t ^H× W over the grid. Let τt=(ot−T,…,ot−1) _t=(o_t-T,…,o_t-1) be the learner’s observation history over a fixed-length sliding window of maximum length T (zero-padded for t<Tt<T). The architecture processes this sequence as follows: (1) Feature Embedding: Each observation in τt _t and the current observation oto_t are processed through a shared feature embedding. As depicted in Figure 3, the spatial tensor (2F×H×W)(2F× H× W), formed by concatenating the observation with its first-order motion encoding o−oprevo-o_prev, passes through a Convolutional layer with ReLU activation to extract local spatial features. The resulting feature map is downsampled, flattened and projected via a fully connected layer into a continuous vector of dimension dmodeld_model. The motion channels are the key partial-observability signal: a negative difference marks a cell that just disappeared under fog or the opponent leaving the vision radius. (2) Transformer Encoder: A multi-layer Transformer Encoder with standard positional encodings processes the sequence; a boolean key-padding mask prevents self-attention from attending to unobserved history. (3) Spatial Head: To generate the prediction, the model extracts the “summary token” (the first output token, corresponding to the current observation oto_t) and passes it through a shallow MLP that projects the dmodeld_model vector back into the spatial dimensions (H×W)(H× W), yielding the heatmap logits G^logits,t∈ℝH×W G_logits,t ^H× W. Figure 3. Feature embedding process inside the Spatial Opponent Model. We define the probability of the opponent targeting a specific coordinate (i,j)∈H×W(i,j)∈ H× W as the output of the inference model pϕp_φ. The Spatial Head produces a raw logit map, G^logits,t∈ℝH×W G_logits,t ^H× W. To ensure the final output represents a valid spatial probability distribution, we apply a Softmax activation function. This produces the predicted spatial intent heatmap G^t∈[0,1]H×W G_t∈[0,1]^H× W, where each element represents the probability of the opponent targeting a specific coordinate (i,j)(i,j) conditioned on the learner’s current observation oto_t and observation history τt _t: (1) G^t(i,j)=pϕ(gt=(i,j)|ot,τt)=exp(G^logits,t(i,j))∑x=1H∑y=1Wexp(G^logits,t(x,y)) G_t^(i,j)=p_φ(g_t=(i,j) |o_t, _t)= ( G_logits,t^(i,j)) _x=1^H _y=1^W ( G_logits,t^(x,y)) where the resulting heatmap G^t G_t is constrained such that the total probability mass sums up to 1: (2) ∑i=1H∑j=1WG^t(i,j)=1 _i=1^H _j=1^W G_t^(i,j)=1 By treating the opponent’s observable past as a sequence of tokens, the Transformer leverages self-attention to capture long-range temporal dependencies between historical movements and current intent. Training the model, however, requires ground-truth labels that are not directly observable in a decentralised multi-agent setting. Hindsight Target Generation To optimise the Spatial OM, the network requires spatial ground-truth labels. Because the opponent’s internal decision-making is not always accessible to the learner during an episode, we generate labels post-hoc with Hindsight Target Generation (strongly related to Hindsight Experience Replay (Andrychowicz et al., 2017)). At the conclusion of an episode, the trajectory is analysed backwards to determine the opponent’s achieved subgoals. If the opponent successfully reaches a food at timestep t+kt+k, that coordinate (x,y)(x,y) becomes the ground-truth subgoal for the preceding sequence; if the opponent fails to reach a definitive objective before the episode ends, their final occupied position becomes the fallback target. To account for spatial uncertainty and avoid overly sparse gradients, we transform this single coordinate into a continuous target intent heatmap Gtarget,t∈[0,1]H×WG_target,t∈[0,1]^H× W by applying a 2D Gaussian smoothing kernel with variance σ2σ^2 centred at the target coordinate: (3) Gtarget,t(i,j)=exp(−(i−x)2+(j−y)22σ2)G_target,t^(i,j)= (- (i-x)^2+(j-y)^22σ^2 ) This smoothing provides a dense learning signal, penalising “near-misses” less severely than spatially distant errors. The systematic process for extracting these labels from raw trajectories is detailed in Algorithm 1. Algorithm 1 Hindsight Subgoal Labelling with Gaussian Smoothing 1: 2: Logged episode E=[(ot,rtopp,stg,st+1g)]t=1TepE=[(o_t,r^opp_t,s^g_t,s^g_t+1)]_t=1^T_ep: the learner’s observations oto_t, the opponent’s rewards rtoppr^opp_t, and the pre-step and post-step privileged global states stgs^g_t and st+1gs^g_t+1 (used for labelling only) 3: Grid dimensions H×WH× W, Smoothing variance σ2σ^2 4: gtrue←Noneg_true ⊳ Current true goal coordinate (x,y)(x,y) 5: Gnext←H×WG_next 0_H× W ⊳ Target map for the next state 6: ⊳ Fallback goal for unsuccessful or truncated episodes 7: if E is not empty then 8: (oTep,rTepopp,sTepg,sTep+1g)←E[Tep](o_T_ep,r^opp_T_ep,s^g_T_ep,s^g_T_ep+1)← E[T_ep] 9: if rTopp=0r^opp_T=0 then 10: gtrue←ExtractOpponentPosition(sTg)g_true (s^g_T) 11: ⊳ Backwards Relabelling Trajectory 12: for t=Tept=T_ep downto 11 do 13: (ot,rtopp,stg,st+1g)←E[t](o_t,r^opp_t,s^g_t,s^g_t+1)← E[t] 14: if rtopp>0r^opp_t>0 then ⊳ Opponent achieved an objective (e.g., reached food) 15: gtrue←ExtractOpponentPosition(st+1g)g_true (s^g_t+1) 16: Gtarget,t←H×WG_target,t 0_H× W 17: if gtrue≠Noneg_true then 18: Gtarget,t[gtrue]←1.0G_target,t[g_true]← 1.0 ⊳ Assign one-hot target 19: Gtarget,t←GaussianFilter(Gtarget,t,σ2)G_target,t (G_target,t,σ^2) ⊳ Generate dense soft-target map 20: Label Gtarget,tG_target,t to transition t as true_goal_map 21: Label GnextG_next to transition t as true_goal_map_next 22: Gnext←Gtarget,tG_next← G_target,t ⊳ Carry target map backward 23: return Labelled episode E Training Objective Because Gtarget,tG_target,t is a continuous Gaussian density, we normalise it to a valid probability distribution (summing to 11 across the grid) and optimise the network against it with the KL Divergence objective: (4) ℒOM(ϕ)=KL(Gtarget,t∥G^t)=∑i=1H∑j=1WGtarget,t(i,j)(logGtarget,t(i,j)−logG^t(i,j)) splitL_OM(φ)&=D_KL(G_target,t G_t)\\ &= _i=1^H _j=1^WG_target,t^(i,j) ( G_target,t^(i,j)- G_t^(i,j) ) split This encourages the Transformer to capture the regional spatial intent rather than memorising discrete coordinate points. Feature fusion The Q-function is conditioned on the inferred intent: Q(ot,a,G^t)Q(o_t,a, G_t). Figure 4. OM Dueling Q-Network. The observation and belief channels are concatenated with the inferred heatmap and processed by the CNN; Advantage and Value heads compute the final Q-values by Equation 6. A low-dimensional latent summary of the opponent’s intent is easily drowned out by the far larger flattened spatial embedding of the Q-network’s CNN, so the intent signal is kept in physical coordinates: in an ESS game the opponent’s intent is tied to the grid’s physical layout. The Spatial Head therefore projects intent back into a heatmap, and the resulting G^t G_t (Eq. 1) is concatenated as an additional channel of the network input (early spatial fusion) as depicted in Figure 4. The Q-network’s CNN thus sees the opponent’s likely destination as a heatmap overlaid on walls, food, and the agent’s own position, and learns their geometric relationships directly. Let ot∈ℝF×H×Wo_t ^F× H× W be the learner’s observation with F feature channels, and bt∈ℝM×H×Wb_t ^M× H× W with M belief feature channels. This belief map is shared by all compared architectures. The inferred heatmap is concatenated with both along the feature dimension: (5) xt=[ot;bt;G^t]∈ℝ(F+M+1)×H×Wx_t=[o_t;b_t; G_t] ^(F+M+1)× H× W The embedding cnn=fCNN(xt)h_cnn=f_CNN(x_t) is processed by the dueling heads (Wang et al., 2016), which separate state value (V(⋅)V(·)) from per-action advantages (A(⋅,⋅)A(·,·)). The final Q-value is computed as: (6) Q(ot,a,G^t)=V(cnn)+A(cnn,a)−1||∑a′A(cnn,a′)Q(o_t,a, G_t)=V(h_cnn)+A(h_cnn,a)- 1|A| _a A(h_cnn,a ) The Q-network is optimised with the double-DQN update (van Hasselt et al., 2016): the online network selects the greedy next action and the Polyak-averaged target network Qθ−Q_θ^- evaluates it, with target (7) yt=rt+γQθ−(ot+1,argmaxa′Qθ(ot+1,a′,G^t+1),G^t+1)y_t=r_t+γ\,Q_θ^- (o_t+1, *argmax_a Q_θ(o_t+1,a , G_t+1), G_t+1 ) and loss Huber(yt−Qθ(ot,at,G^t))Huber(y_t-Q_θ(o_t,a_t, G_t)) (Huber, 1964) (Algorithm 2). Pre-training the opponent model If the Spatial OM and the Q-network are trained concurrently from random initialisation, the early heatmaps are indistinguishable from uniform noise; the Q-network’s CNN quickly learns to assign near-zero weights to the heatmap channel, and once ignored the channel rarely recovers. We therefore pre-train pϕp_φ offline before joint RL training. A supervised dataset of trajectories is collected by simulating games between heuristic agents (various pairings of the simple and greedy profiles, Section 5) and labelled via hindsight labelling (Algorithm 1). This grants the Spatial OM a foundational understanding of the grid’s topology, agent movement, and the environment’s rules, so that when joint OM-DDQN training begins (Algorithm 2) the Q-network receives structurally meaningful intent predictions from the first episode and integrates the heatmap channel during value estimation from the outset. Algorithm 2 DDQN with the Spatial OM (OM-DDQN) using HER 1: Initialise: 2: Replay Buffer RLD_RL 3: Agent Network QθQ_θ, Target Network Qθ−Q_θ^- 4: Spatial Opponent Model pϕp_φ 5: Pre-train: 6: Train pϕp_φ on offline heuristic dataset labelled via HER. 7: Collect data: 8: for episode =1=1 to NepN_ep do 9: Initialise environment, observe o0o_0, set history τ0=∅ _0= , and local episode buffer 10: for step t=0t=0 to TepT_ep do 11: Inference: 12: Predict subgoal heatmap: G^t=pϕ(ot,τt) G_t=p_φ(o_t, _t) 13: Action: 14: Select ata_t using Boltzmann exploration on Qθ(ot,⋅,G^t)Q_θ(o_t,·, G_t) 15: Step: 16: Execute ata_t, observe reward rtr_t and next observation ot+1o_t+1 17: Update history: τt+1←append(τt,ot) _t+1 ( _t,o_t) ⊳ slide window of length T 18: Store step data (ot,at,rt,ot+1,τt,G^t)(o_t,a_t,r_t,o_t+1, _t, G_t) in local episode buffer 19: Hindsight Labelling: 20: Label the logged episode as described in Algorithm 1 21: Transfer local buffer to global Replay Buffer RLD_RL 22: 23: Training Phase: 24: Sample batch of transitions and sequences from RLD_RL 25: ⊳ Train OM Dueling Q-Network 26: Generate next step heatmaps G^t+1 G_t+1 using the Spatial OM inference pϕ(ot+1,τt+1)p_φ(o_t+1, _t+1) 27: Select next actions amax′=argmaxa′Qθ(ot+1,a′,G^t+1)a _max= *argmax_a Q_θ(o_t+1,a , G_t+1) 28: Calculate target y=rt+γQθ−(ot+1,amax′,G^t+1)y=r_t+γ Q_θ^-(o_t+1,a _max, G_t+1) 29: Compute Loss ℒDDQN=Huber(y−Qθ(ot,at,G^t))L_DDQN=Huber(y-Q_θ(o_t,a_t, G_t)) 30: Update θ via gradient descent 31: Update target network (Polyak): θ−←βθ+(1−β)θ−θ^-←β\,θ+(1-β)\,θ^- 32: ⊳ Train the Spatial OM 33: Compute ℒOM(ϕ)L_OM(φ) using KL Divergence against normalised soft targets Gtarget,tG_target,t 34: Update ϕφ via gradient descent Figure 5. Example of Spatial OM predictions on map4 (vision radius 2). Left: the learner’s observation under fog; the learner (green dot) sees only its surroundings, while the opponent’s true position (dark red when obscured by fog) and the food (green squares) are shown for reference. Middle: the OM’s predicted subgoal distribution fed to the Q-network. Right: the opponent’s true subgoal distribution, that is not accessible during match. The OM concentrates mass on the opponent’s actual target even when the opponent is outside the learner’s vision radius. 5. Experimental Setup Environment. Foraging under partial observability (Section 3) played on two symmetric 11×1111×11 grids (Figure 1): map3 (dense central food, 5 goals) and map4 (sectioned walls, dispersed food, 7 goals), with fixed spawn and goal positions and vision radius v∈2,5v∈\2,5\ (fog thickness). The symmetry and game rules ensure that each agent’s game value is half of the total food. Greedy vs greedy play shows this empirically (map3: 2.50/2.50; map4: 3.42/3.42). Opponents. All opponents share the learner’s action space and movement rules and act on their own partial observations (same vision radius). simple commits to a random believed food and replans when it is consumed or seen as collected. greedy targets its nearest believed food and switches away whenever it believes the learner is closer. oracle-stalker (also denoted as “tracker”) sees the true state regardless of fog and races to the food nearest the learner that it can still reach first, loitering there to deny collection. Training. Dueling DQN, n-step returns n∈1,3n∈\1,3\ (Sutton and Barto, 2018), belief-map prior p∈0,1p∈\0,1\ (belief map off/on), 60 epochs, 3 seeds per configuration, against the greedy opponent. Training used a single Tesla V100-SXM2-32GB GPU per run, taking about 7 hours plus per-epoch evaluation (classic and om combined). Zero-shot evaluations ran on an Apple M3 Pro (12-core CPU, 18 GB RAM). Table 1 lists the shared hyperparameters. Table 1. Shared training hyperparameters and network sizes (both learning agents). Learning rate Adam; 3⋅10−43·10^-4 (Q), 1⋅10−41·10^-4 (OM) Discount γ / soft update τsoft _soft 0.985 / 0.001 Batch size / train frequency 128 / every 4 steps Replay capacity 450,000 Episodes 30,000 (60 epochs×500 episodes), max 50 steps Boltzmann τ 1.5→0.051.5→ 0.05 over 650k steps Label smoothing σ2σ^2 2.0→0.52.0→ 0.5 over 150k steps OM pre-training 20 epochs heuristic self-play Q-net 3×3×Conv3×33×3 (Cin→32→64→64C_in→32→64→64), dueling heads 256 OM transformer dmodel=128d_model=128, 2 layers, 4 heads, dff=256d_f=256, dropout 0.05, history length 50 Zero-shot evaluation. Checkpoints are evaluated against greedy (the training opponent), simple, and oracle-stalker, with seat-swap probability 0.5 and 1000 episodes per seed. We report differences paired by training seed (Δ=om−classic = om- classic; the same seed index for both agents), split by n-step configuration (pooled only where stated, as in the belief-prior comparison), mean ± std over seeds. 6. Results Training stability Figure 6. Per-epoch evaluation return against the training opponent, per seed: classic (blue) vs. om (orange), belief prior off. Top row map4, bottom row map3. classic degenerates at n=1n=1: on map4 one v=2v=2 seed collapses outright, all three v=5v=5 seeds finish below 1.71.7 of 3.53.5, and on map3 all seeds in the v=2v=2 cell finish below 2.02.0 of 2.52.5 (thin fog recovers). om never collapses on either map. Plain DQN is unstable at n=1n=1 (Figure 6). On map4 one classic seed collapses outright in v=2v=2 and all three v=5v=5 seeds finish at 37–46% of game value (1.3–1.6 of 3.5); on map3 at thick fog all n=1n=1 seeds finish at 47–68% of game value (1.17–1.71 of 2.5). om never collapses and never finishes below 80% of game value on map3 (floor 2.03 of 2.5). Across all zero-shot batches on both maps, seed-level standard deviation is 1.4–3.6× smaller for om (map4: 0.16–0.31 vs. 0.50–0.79). In-distribution performance Figure 7. Zero-shot return as a share of the symmetric game value, by opponent and cell (dots: seeds; bar: mean; dashed: game value). Left: map4 (prior off; collapsed classic seed excluded). Right: map3 (prior on/off pooled, 6 seeds per cell). Both agents exceed 100% under thin fog (best-response exploitation of the suboptimal heuristic) and both saturate on the dense map wherever training succeeds. Against the training opponent (Figure 7), om recovers 95% (map4, v=2v=2) to 99.9% (v=5v=5) of the game value in well-trained (n=3n=3) cells; classic reaches 77–88%. On map3 both agents reach 96–101% at n=3n=3, since the game is saturated. Under thin fog both even exceed the game value on average (104% for om, 101% for classic; individual seeds up to 107%): a best response to the suboptimal training heuristic. Thick fog separates them: in the hard cell (v=2v=2, n=1n=1) classic degenerates to 52–76% per seed (64% on average), because exploiting an opponent you rarely see requires predicting it. Opponent spectrum Table 2. Paired zero-shot advantage Δ=om−classic = om- classic (mean ± std over seeds) by opponent, fog, and n-step horizon on map4 (prior off). n=3n=3: well-trained cells; n=1n=1: cells where classic training is unstable. Seen forager means that the learner has trained against the opponent; unseen forager means that the opponent is not in the learner’s training set. Dense-map (map3) deltas are reported in the text and Table 3. Opponent Fog n=3n=3 n=1n=1 greedy (seen, forager) v=2v=2 +0.64±0.20+0.64± 0.20 +0.71±0.08†+0.71± 0.08 greedy (seen, forager) v=5v=5 +0.41±0.07+0.41± 0.07 +1.50±0.26+1.50± 0.26 simple (unseen, forager) v=2v=2 +0.31±0.31+0.31± 0.31 +0.66±0.05†+0.66± 0.05 simple (unseen, forager) v=5v=5 +0.52±0.10+0.52± 0.10 +1.16±0.05+1.16± 0.05 oracle-stalker (tracker) v=2v=2 +0.12±0.39+0.12± 0.39 +0.52±0.13†+0.52± 0.13 oracle-stalker (tracker) v=5v=5 +0.02±0.09+0.02± 0.09 +1.13±0.27+1.13± 0.27 †Excluding one collapsed classic seed (see Limitations); the collapse is reported as a finding. The advantage transfers across forager types, seen or unseen, and in well-trained scenarios disappears against the tracker (Table 2). simple is unseen by the learner, but its profile appears in the OM’s offline pre-training corpus (Section 4), so this tests the learned policy’s transfer, not the OM’s. This fits an information argument. A forager’s intent is a fact about the environment: its trajectory reveals it, and the learner’s fogged observations do not contain it. A tracker’s intent is the learner itself, so predicting it is redundant with self-knowledge. Tracking is also costly for the tracker: total extraction drops from ∼7 7 goals (against foragers) to ∼3.5 3.5, as the tracker spends roughly 2.2 goals of its own foraging to remove about 1 from the learner. On the dense map, pooling prior on and off, the paired advantage concentrates in one hard cell (v=2v=2, n=1n=1): +0.57+0.57 vs. greedy, +0.52+0.52 vs. simple, +0.48+0.48 vs. the tracker. Wherever classic trains, the map saturates and the advantage falls to at most +0.13+0.13 against foragers and +0.23+0.23 against the tracker. The tracker at the hard cell is our mechanism probe: the advantage survives (+0.48+0.48), and blinding the heatmap leaves it nearly intact (+0.44+0.44, Table 3). Against a tracker, the gap is training-time stabilisation, not inference. Ablating the inference channel Table 3. Blindfold ablation: heatmap zeroed at deployment. Mean om return over seeds; Δ=om−classic = om- classic paired by training seed, zero-shot. map4: prior off, n-split; map3: n=1n=1, prior on/off pooled, 6 seeds; vs. the opponent shown. Map / opponent Cell om om blind Δ Δ blind map4, greedy v=2v=2, n=3n=3 3.33 2.23 +0.64+0.64 −0.47-0.47 map4, greedy v=2v=2, n=1n=1 2.86 2.43 +0.71†+0.71 +0.28†+0.28 map4, simple v=2v=2, n=3n=3 2.77 2.10 +0.31+0.31 −0.37-0.37 map4, simple v=2v=2, n=1n=1 2.55 2.31 +0.66†+0.66 +0.42†+0.42 map4, oracle-stalker v=2v=2, n=3n=3 2.09 1.71 +0.12+0.12 −0.26-0.26 map4, oracle-stalker v=2v=2, n=1n=1 2.10 2.10 +0.52†+0.52 +0.52†+0.52 map4, greedy v=5v=5, n=3n=3 3.50 3.02 +0.41+0.41 −0.07-0.07 map4, greedy v=5v=5, n=1n=1 3.17 2.50 +1.50+1.50 +0.84+0.84 map4, simple v=5v=5, n=3n=3 2.81 2.57 +0.52+0.52 +0.28+0.28 map4, simple v=5v=5, n=1n=1 2.54 2.31 +1.16+1.16 +0.92+0.92 map4, oracle-stalker v=5v=5, n=3n=3 2.29 2.17 +0.02+0.02 −0.10-0.10 map4, oracle-stalker v=5v=5, n=1n=1 2.50 2.24 +1.13+1.13 +0.87+0.87 map3, greedy v=2v=2 2.18 2.13 +0.57+0.57 +0.53+0.53 map3, simple v=2v=2 1.67 1.64 +0.52+0.52 +0.48+0.48 map3, oracle-stalker v=2v=2 1.71 1.68 +0.48+0.48 +0.44+0.44 map3, greedy v=5v=5 2.60 2.44 +0.07+0.07 −0.09-0.09 map3, simple v=5v=5 2.44 2.29 +0.12+0.12 −0.03-0.03 map3, oracle-stalker v=5v=5 2.04 2.03 +0.23+0.23 +0.21+0.21 †Row excludes one collapsed classic seed (see Limitations); the collapse is reported as a finding. Table 3 completes the grid: all three opponents, both fog levels, and both returns configurations, with the paired advantage computed within the same runs. The picture splits along training difficulty. In the well-trained cells (n=3n=3), blinding erases the entire advantage in five of six cells: against the training opponent at thick fog +0.64→−0.47+0.64→-0.47, against the unseen forager +0.31→−0.37+0.31→-0.37. The one exception keeps just over half (+0.52→+0.28+0.52→+0.28, unseen forager, thin fog). Where training succeeds, the advantage is inference. In the unstable cells (n=1n=1), an edge survives blinding against every opponent (+0.28+0.28 to +0.92+0.92 over classic, excluding the collapsed seed at thick fog): stabilisation that does not require the heatmap at deployment. Reliance on the channel is strongest in distribution and in fog. The largest drop is against the training opponent at thick fog (3.33→2.233.33→ 2.23, leaving om half a goal below classic), with a worst per-seed drop of 1.811.81. Across the well-trained cells the drop shrinks as fog thins (1.10→0.481.10→ 0.48 goals against the training opponent): the policy consults the heatmap in proportion to how much its own observations leave unknown. The unstable cells show no consistent fog scaling, so their stabilisation is fog-independent even where their inference is not. Blinding never collapses an om seed (per-seed returns stay above 1.51.5), so the drop mixes information loss with input corruption (see Limitations). Where training is hard, the opponent model contributes stabilisation; where training is easy, it contributes inference. The tracker rows tell the same story on the sparse map. In the unstable cells the gap survives blinding almost fully (+0.52→+0.52+0.52→+0.52 at thick fog, +0.87+0.87 of +1.13+1.13 at thin fog): stabilisation. In the well-trained cells the advantage is small, but its blindfold response still tracks fog: near zero either way at thin fog (+0.02→−0.10+0.02→-0.10), yet at thick fog the channel is still consulted (+0.12→−0.26+0.12→-0.26, a 0.370.37-goal drop for om itself); the heatmap there carries not the target (the learner knows its own position) but the tracker’s approach under fog. An opponent whose intent is about the learner rather than the environment yields stabilisation, not inference. On the dense map, blinding is nearly neutral at thick fog (at most 0.050.05 goals in every v=2v=2 cell), so the whole +0.5+0.5 advantage there is stabilisation. The largest drops come at thin fog (−0.16-0.16 against the training opponent at v=5v=5) and flip the paired advantage slightly negative (−0.09-0.09). With the game saturated there is little for inference to add. The fog scaling of inference therefore needs a cell that is not saturated, which the dense map lacks: itself a density effect. Resource density Table 4. Dense map3 (game value 2.5 by symmetry): epoch-60 eval return vs. the training opponent as % of game value, mean over 3 seeds per cell. Δ : paired om−-classic in goals. Fog n Prior classic om Δ v=2v=2 1 off 55.1 82.3 +0.68+0.68 v=2v=2 1 on 65.8 89.4 +0.59+0.59 v=2v=2 3 off 90.5 99.5 +0.23+0.23 v=2v=2 3 on 98.4 99.8 +0.04+0.04 v=5v=5 1 off 96.2 105.3 +0.23+0.23 v=5v=5 1 on 104.0 106.9 +0.07+0.07 v=5v=5 3 off 99.9 101.5 +0.04+0.04 v=5v=5 3 on 97.8 99.6 +0.05+0.05 Table 4 completes the picture. Wherever classic trains successfully (≥90%≥ 90\% of value), om also sits at the ceiling and the advantage is at most 0.230.23 goals. On a dense map a lost contest costs little because another goal is nearby, so predicting the opponent’s target is worth little: the advantage vanishes by saturation, not by failure. The stability effect replicates and depends on fog. At thick fog all six classic n=1n=1 seeds fall below 2.02.0 of 2.52.5 (minimum 1.171.17) while om never drops below 2.032.03; at thin fog even classic n=1n=1 recovers (minimum 2.082.08). Across both maps, the advantage shrinks with the difficulty reducers that stabilise classic training (denser map, longer n-step horizon), from +0.5+0.5–1.51.5 goals in the sparse map’s unstable cells to +0.04+0.04 on the dense map under thin fog. Fog is the exception on the sparse map: at n=1n=1 the gap widens as fog thins (+0.71→+1.50+0.71→+1.50 vs. the training opponent), because om improves with visibility (2.84→3.172.84→ 3.17) while classic does not (2.1→1.72.1→ 1.7 goals excluding the collapsed seed). Belief-map prior On map3 the prior is neutral for om: the largest cell-level return difference between p=1p=1 and p=0p=0 is 0.180.18 goals, and its effect on inference quality is small and inconsistent in sign (−0.28-0.28 to +0.26+0.26). It partially rescues classic where TD(0) fails: at the hard cell (v=2v=2, n=1n=1; zero-shot vs. greedy), classic rises from 53% to 75% of game value with the prior, while om remains highest (91%). Table 4 shows the same direction with smaller gaps (55.1% → 65.8%, om 82.3–89.4%). The map4 replication turns this into a boundary condition. At thin fog the prior lifts the baseline (classic vs. the training opponent: 68% → 84% of game value) while om gives up 0.30.3–0.60.6 goals. The paired advantage, pooled over the n-split for this comparison, thus shrinks from +0.90+0.90/+0.96+0.96 to +0.60+0.60/+0.04+0.04 goals (thick/thin fog) and nearly vanishes against the tracker (from +0.51+0.51/+0.58+0.58 to +0.01+0.01/+0.24+0.24). Part of the prior-free edge was an information-starved baseline: a positional belief substitutes for subgoal inference at thin fog, but not at thick fog, where inference keeps a +0.6+0.6 goal edge over the baseline with the prior. The prior is a safe default for the baseline; the opponent model buys the thick-fog remainder. 7. Limitations Three seeds per configuration. We report per-seed strip plots and never pool across n-step configurations (except where stated, as in the belief-prior comparison), and several cells have seed-level standard deviations comparable to their means and should be read as non-significant. One classic seed on map4 collapsed during training and is excluded from paired statistics there, with the training curve cited as justification. Collapse frequency is itself reported. All results are best responses to a heuristic. In preliminary experiments, NFSP-style self-play (Heinrich and Silver, 2016) did not converge in the FOSG domain, so we make no MARL equilibrium claims. Published opponent-modeling methods assume different observation spaces (full state, action histories, or population priors). None infers spatial opponent subgoals from partial observations in this setting, so we compare against classic, whose Q-network matches the om Q-network on observation and belief channels and on capacity, differing only in the heatmap input, although that heatmap summarises an observation history classic does not see (see Conclusion). We fix the learner to value-based DQN methods because the claim concerns TD(0) stability under partial observability. Policy-gradient learners would change the failure mode rather than test it. Zeroing the heatmap is an input distribution shift. The drop measures information loss plus input corruption, not information loss alone. And lastly, this work focuses on grid foraging only, scale is future work. 8. Conclusion and Future Work Opponent-goal inference under partial observability is not a uniform advantage: its value is set by visibility (fog thickness), by whether the opponent’s intent is about the environment or about the learner, and by resource density. Its most robust benefit is making DQN training stable where plain TD(0) fails. We attribute the stabilisation to the module as a whole. Whether it stems from subgoal inference or from history-derived structured input is not isolated here. The blindfold ablation offers partial evidence: zeroing the heatmap, the only input beyond the channels shared with classic, costs the most deployment performance in the cells where the information argument says inference matters most (in distribution and at thick fog on the sparse map), consistent with the inference carrying information rather than acting as generic structured input. Future work scales these mechanisms to roadmap graphs of real-world maps, where foraging under fog becomes a multi-agent travelling-salesman problem under partial observability, in both competitive and cooperative settings, and replaces self-play with policy-space or population methods for equilibrium approximation. 9. Acknowledgements I would like to thank my supervisor, Ing. Rostislav Horčík, Ph.D., for his invaluable guidance and support throughout this research. The authors acknowledge the support of the OP V “Research Center for Informatics”. The access to the computational infrastructure of the OP V “Research Center for Informatics” is also gratefully acknowledged. References Al-Shedivat et al. (2018) M. Al-Shedivat, T. Bansal, Y. Burda, I. Sutskever, I. Mordatch, and P. Abbeel Continuous adaptation via meta-learning in nonstationary and competitive environments. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings, External Links: Link Cited by: §1. Albrecht and Stone (2018) S. V. Albrecht and P. Stone Autonomous agents modelling other agents: A comprehensive survey and open problems. Artif. Intell. 258, p. 66–95. External Links: Link, Document Cited by: §2. Andrychowicz et al. (2017) M. Andrychowicz, D. Crow, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba Hindsight experience replay. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett (Eds.), p. 5048–5058. External Links: Link Cited by: §2, §4. Bakkes et al. (2009) S. C. J. Bakkes, P. H. M. Spronck, and H. J. van den Herik Opponent modelling for case-based adaptive game AI. Entertain. Comput. 1 (1), p. 27–37. External Links: Link, Document Cited by: §2. Charniak and Goldman (1993) E. Charniak and R. P. Goldman A bayesian model of plan recognition. Artif. Intell. 64 (1), p. 53–79. External Links: Link, Document Cited by: §2. Christianos et al. (2020) F. Christianos, L. Schäfer, and S. V. Albrecht Shared experience actor-critic for multi-agent reinforcement learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.), External Links: Link Cited by: §1. Hansen et al. (2004) E. A. Hansen, D. S. Bernstein, and S. Zilberstein Dynamic programming for partially observable stochastic games. In Proceedings of the Nineteenth National Conference on Artificial Intelligence, Sixteenth Conference on Innovative Applications of Artificial Intelligence, July 25-29, 2004, San Jose, California, USA, D. L. McGuinness and G. Ferguson (Eds.), p. 709–715. External Links: Link Cited by: §2. He and Boyd-Graber (2016) H. He and J. L. Boyd-Graber Opponent modeling in deep reinforcement learning. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, M. Balcan and K. Q. Weinberger (Eds.), JMLR Workshop and Conference Proceedings, Vol. 48, p. 1804–1813. External Links: Link Cited by: §1, §2. Heinrich and Silver (2016) J. Heinrich and D. Silver Deep reinforcement learning from self-play in imperfect-information games. CoRR abs/1603.01121. External Links: Link, 1603.01121 Cited by: §7. Hong et al. (2017) Z. Hong, S. Su, T. Shann, Y. Chang, and C. Lee A deep policy inference q-network for multi-agent systems. CoRR abs/1712.07893. External Links: Link, 1712.07893 Cited by: §2. Huber (1964) P. J. Huber Robust Estimation of a Location Parameter. The Annals of Mathematical Statistics 35 (1), p. 73 – 101. External Links: Document, Link Cited by: §4. Littman (1994) M. L. Littman Markov games as a framework for multi-agent reinforcement learning. In Machine Learning, Proceedings of the Eleventh International Conference, Rutgers University, New Brunswick, NJ, USA, July 10-13, 1994, W. W. Cohen and H. Hirsh (Eds.), p. 157–163. External Links: Link, Document Cited by: §1. Mnih et al. (2015) V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis Human-level control through deep reinforcement learning. Nat. 518 (7540), p. 529–533. External Links: Link, Document Cited by: §2. Nashed and Zilberstein (2022) S. B. Nashed and S. Zilberstein A survey of opponent modeling in adversarial domains. J. Artif. Intell. Res. 73, p. 277–327. External Links: Link, Document Cited by: §1, §2. Papoudakis et al. (2021) G. Papoudakis, F. Christianos, and S. V. Albrecht Agent modelling under partial observability for deep reinforcement learning. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan (Eds.), p. 19210–19222. External Links: Link Cited by: §1, §2. Petran (2026) M. Petran Opponent modeling in zero-sum games. Master’s Thesis, Faculty of Electrical Engineering, Czech Technical University in Prague, Prague, Czech Republic. Cited by: §1, §1, §3, §4. Premack and Woodruff (1978) D. Premack and G. Woodruff Does the chimpanzee have a theory of mind?. Behavioral and Brain Sciences 1 (4), p. 515–526. External Links: Document Cited by: §2. Rabinowitz et al. (2018) N. C. Rabinowitz, F. Perbet, H. F. Song, C. Zhang, S. M. A. Eslami, and M. M. Botvinick Machine theory of mind. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, J. G. Dy and A. Krause (Eds.), Proceedings of Machine Learning Research, Vol. 80, p. 4215–4224. External Links: Link Cited by: §1, §2. Raileanu et al. (2018) R. Raileanu, E. Denton, A. Szlam, and R. Fergus Modeling others using oneself in multi-agent reinforcement learning. CoRR abs/1802.09640. External Links: Link, 1802.09640 Cited by: §1. Sutton and Barto (2018) R. S. Sutton and A. G. Barto Reinforcement learning - an introduction, 2nd edition. MIT Press. External Links: Link Cited by: §5. van Hasselt et al. (2016) H. van Hasselt, A. Guez, and D. Silver Deep reinforcement learning with double q-learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, D. Schuurmans and M. P. Wellman (Eds.), p. 2094–2100. External Links: Link, Document Cited by: §2, §4. Wang et al. (2016) Z. Wang, T. Schaul, M. Hessel, H. van Hasselt, M. Lanctot, and N. de Freitas Dueling network architectures for deep reinforcement learning. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, M. Balcan and K. Q. Weinberger (Eds.), JMLR Workshop and Conference Proceedings, Vol. 48, p. 1995–2003. External Links: Link Cited by: §2, §4. Yu et al. (2024) X. Yu, J. Jiang, and Z. Lu Opponent modeling based on subgoal inference. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang (Eds.), External Links: Link Cited by: §1, §2. Zintgraf et al. (2020) L. M. Zintgraf, K. Shiarlis, M. Igl, S. Schulze, Y. Gal, K. Hofmann, and S. Whiteson VariBAD: A very good method for bayes-adaptive deep RL via meta-learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, External Links: Link Cited by: §2.