Paper deep dive
SIGMAS: Second-Order Interaction-based Grouping for Overlapping Multi-Agent Swarms
Minah Lee, Saibal Mukhopadhyay
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 86%
Last extracted: 7/20/2026, 7:55:50 PM
Summary
The paper introduces SIGMAS, a self-supervised framework for predicting group structures in overlapping multi-agent swarms. Unlike traditional models relying on first-order (pairwise) interactions, SIGMAS utilizes second-order interactions to measure the similarity of agents' interaction patterns, enabling robust identification of latent group memberships in dense, overlapping swarm scenarios without ground-truth supervision.
Entities (9)
Relation Signals (9)
SIGMAS → uses → Second-Order Interaction
confidence 95% · SIGMAS models group-level dynamics via second-order interaction modeling.
Swarm-Level Encoder → computes → Second-Order Interaction
confidence 92% · Swarm-Level Encoder captures social behavior similarity among agents by modeling second-order interactions.
SIGMAS → contains → Agent-Level Encoder
confidence 90% · SIGMAS consists of three main components: (1) an Agent-Level Encoder...
SIGMAS → contains → Swarm-Level Encoder
confidence 90% · SIGMAS consists of three main components: (2) a Swarm-Level Encoder...
SIGMAS → uses → First-Order Interaction
confidence 90% · SIGMAS features a novel swarm-level encoder that computes second-order interactions by comparing agents’ first-order attention patterns.
SIGMAS → contains → Balancing & Gating Module
confidence 85% · SIGMAS consists of three main components: (3) a Balancing & Gating Module...
Balancing & Gating Module → fuses → Agent-Level Encoder Output
confidence 85% · The Balancing & Gating Module that adaptively fuses individual and swarm-level representations.
SIGMAS → integrateswith → CVAE
confidence 80% · We integrate SIGMAS into a conditional variational autoencoder (CVAE)-based multi-agent trajectory prediction framework.
Agent-Level Encoder → basedon → AgentFormer
confidence 75% · We adopt the encoder architecture from AgentFormer [29]...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Swarming systems, such as drone fleets and robotic teams, exhibit complex dynamics driven by both individual behaviors and emergent group-level interactions. Unlike traditional multi-agent domains such as pedestrian crowds or traffic systems, swarms typically consist of a few large groups with inherent and persistent memberships, making group identification essential for understanding fine-grained behavior. We introduce the novel task of group prediction in overlapping multi-agent swarms, where latent group structures must be inferred directly from agent trajectories without ground-truth supervision. To address this challenge, we propose SIGMAS (Second-order Interaction-based Grouping for Multi-Agent Swarms), a self-supervised framework that goes beyond direct pairwise interactions and model second-order interaction across agents. By capturing how similarly agents interact with others, SIGMAS enables robust group inference and adaptively balances individual and collective dynamics through a learnable gating mechanism for joint reasoning. Experiments across diverse synthetic swarm scenarios demonstrate that SIGMAS accurately recovers latent group structures and remains robust under simultaneously overlapping swarm dynamics, establishing both a new benchmark task and a principled modeling framework for swarm understanding.
Tags
Links
- Source: https://arxiv.org/abs/2603.00120v1
- Canonical: https://arxiv.org/abs/2603.00120v1
Trouble viewing inline? Open PDF directly →
Full Text
49,825 characters extracted from source content.
Expand or collapse full text
SIGMAS: Second-Order Interaction-based Grouping for Overlapping Multi-Agent Swarms Minah Lee The University of Texas at Dallas Richardson, TX, USA minah.lee@utdallas.edu &Saibal Mukhopadhyay Georgia Institute of Technology Atlanta, GA, USA saibal.mukhopadhyay@ece.gatech.edu Abstract Swarming systems, such as drone fleets and robotic teams, exhibit complex dynamics driven by both individual behaviors and emergent group-level interactions. Unlike traditional multi-agent domains such as pedestrian crowds or traffic systems, swarms typically consist of a few large groups with inherent and persistent memberships, making group identification essential for understanding fine-grained behavior. We introduce the novel task of group prediction in overlapping multi-agent swarms, where latent group structures must be inferred directly from agent trajectories without ground-truth supervision. To address this challenge, we propose SIGMAS (Second-order Interaction-based Grouping for Multi-Agent Swarms), a self-supervised framework that goes beyond direct pairwise interactions and model second-order interaction across agents. By capturing how similarly agents interact with others, SIGMAS enables robust group inference and adaptively balances individual and collective dynamics through a learnable gating mechanism for joint reasoning. Experiments across diverse synthetic swarm scenarios demonstrate that SIGMAS accurately recovers latent group structures and remains robust under simultaneously overlapping swarm dynamics, establishing both a new benchmark task and a principled modeling framework for swarm understanding. Keywords Swarm Dynamics; Multi-Agent Systems; Group Identification; Self-Supervised Learning; Trajectory Prediction 1 Introduction Swarming systems, including drone fleets, robotic teams, and autonomous vehicles, are key enablers of scalable and robust autonomous operations [28, 2, 24, 21, 4]. Through simple local interactions, these systems exhibit emergent group-level behaviors that support coordination, collision avoidance, and efficient resource utilization in dynamic and uncertain environments [19, 13]. Compared to traditional multi-agent scenarios such as pedestrian motion, sports analytics, or traffic navigation [16, 27, 25], swarming systems differ in two fundamental aspects. First, traditional settings typically involve many small groups with sparse interactions, whereas swarms typically consist of a few large groups with dense intra-group coordination and complex inter-group dynamics, as illustrated in Figure 1(a). Second, the notion of a “group” also differs fundamentally. In traditional domains, groups are often defined by transient interactions arising from spatial proximity, while in swarming systems, group membership is an intrinsic and persistent property of each agent. Agents from different swarms may even occupy the same spatial region at the same time, yet maintain distinct coordination patterns. A key challenge in multi-agent swarms is understanding the latent group structure that governs agent coordination. This is critical for robust behavior modeling and fine-grained interpretation of intra-swarm dynamics. To this end, we introduce the novel task of group prediction in overlapping multi-agent swarms, where the goal is to infer latent group memberships directly from agent trajectories without any ground-truth supervision. This self-supervised formulation is motivated by real-world swarm scenarios, where group affiliations are inherent but unobservable and may evolve over time. Since manually annotated group labels are rarely available in practice, we aim to learn group structure directly from motion dynamics without relying on explicit labels. However, most existing trajectory prediction models are not well suited to this regime for two main reasons. First, many are designed for low-density scenes with sparse interactions, such as the ETH/UCY datasets [12, 17], which typically include only 1–6 agents within each observation and prediction window. Second, and more importantly, these models primarily rely on spatial proximity or pairwise motion similarity to model direct influence between agents, here referred to as first-order attention mechanisms (Figure 1(b)). While such mechanisms can be effective for short-term prediction, they are often insufficient to capture true group affiliations, especially in scenarios where multiple swarms overlap and occupy the same spatial region simultaneously, or where members of the same swarm are spatially dispersed. These limitations highlight the need for a specialized approach capable of understanding latent group structure in dense and dynamic swarm settings. Figure 1: (a) Overlapping swarms with distinct coordination patterns, making group identification difficult based on proximity alone. (b) First-order interactions capture direct motion correlations between agents. (c) Second-order interactions compare agents’ first-order attention patterns to measure how similarly they interact with others. This captures social behavior similarity and enables robust group identification, even in the absence of direct interactions. Therefore, we propose modeling social behavior through second-order interactions, where second-order interactions refer to the similarity between agents’ first-order attention patterns. In other words, instead of asking whether two agents directly interact, we examine how similarly they interact with others, as illustrated in Figure 1(c). While first-order attention captures localized dependencies and can be effective for short-term trajectory forecasting, it often conflates transient spatial proximity with true group structure. In contrast, second-order interactions compare each agent’s full interaction profile, revealing shared behavioral patterns and uncovering latent affiliations. This approach is particularly useful in dense, overlapping, or partially converged swarm formations, where direct interactions are no longer a reliable signal of group membership. To address these challenges, we introduce SIGMAS (Second-order Interaction-based Grouping for Multi-Agent Swarms), a self-supervised framework for latent group inference in swarming systems. SIGMAS learns directly from agent trajectories without requiring manual annotations. SIGMAS features a novel swarm-level encoder that computes second-order interactions by comparing agents’ first-order attention patterns. This enables SIGMAS to model interaction similarity and recover cohesive group structures even when agents are spatially entangled or dynamically dispersed. In addition, SIGMAS integrates a dynamic balancing mechanism to fuse individual and group-level dynamics, and applies spectral clustering for self-supervised group inference. These components jointly allow SIGMAS to capture both localized motion and global coordination patterns, enhancing group understanding in complex multi-agent environments. This paper makes the following unique contributions: • We formulate a novel task of group prediction in overlapping multi-agent swarms, where latent group memberships must be inferred from dense agent trajectories without ground-truth supervision. To the best of our knowledge, this is the first work to explicitly address this problem setting. • We present SIGMAS, a self-supervised framework that models group-level dynamics via second-order interaction modeling and adaptively fuses them with agent-level features through a learnable gating mechanism, enabling joint reasoning over local and global behaviors. • We empirically demonstrate that SIGMAS accurately recovers latent group structures and maintains robust performance across diverse and overlapping swarm scenarios. 2 Background 2.1 Swarm Behavior and Interaction Modeling Figure 2: Illustration of emergent swarm behaviors. Top: Intra-swarm convergence, where agents within the same swarm gradually align and form cohesive clusters. Bottom: Inter-swarm overlap, where distinct swarms simultaneously share spatial regions while maintaining separate coordination dynamics. Craig W. Reynolds [20] introduced three canonical rules for flocking behavior in agent-based systems: alignment, cohesion, and separation. Each agent adjusts its velocity based on its local neighbors, leading to emergent collective behaviors such as coordinated motion and dynamic formation. We simulate these behaviors using the agentpy framework [5], which provides fine-grained control over agent-level parameters and swarm configurations. In our setup, agents within the same swarm share identical alignment, cohesion, and separation strengths, reflecting the assumption that group membership dictates underlying motion dynamics. For agents in different swarms, only the separation rule applies to avoid collision. Figure 2 illustrates two key swarm phenomena that emerge from these local interaction rules: Intra-Swarm Convergence (Top Row). Agents within the same swarm gradually align their velocities, forming cohesive motion patterns over time. As interactions accumulate, local clusters emerge and evolve into tightly coordinated formations. The rate of convergence depends on intra-group interaction strength: higher alignment and cohesion lead to faster convergence and denser formations (green agents), while lower values yield slower and more diffuse structures (yellow agents). Inter-Swarm Overlap (Bottom Row). Distinct swarms may occupy overlapping spatial regions simultaneously while maintaining separate coordination dynamics. Depending on the relative strengths of intra- and inter-swarm interactions, agents may either preserve their original affiliations or partially mix across swarm boundaries. These examples illustrate the challenge of inferring latent group structures in swarming systems. Spatial proximity alone is often misleading, as agents from different swarms may overlap while following distinct coordination patterns. This motivates the design of SIGMAS, which infers group membership by modeling second-order interaction, offering a more robust foundation for group prediction in complex, overlapping swarm environments. 2.2 Trajectory-based Interaction Modeling in Multi-Agent Systems Table 1: Comparison of key capabilities for group prediction in overlapping multi-agent swarms. ’Traj. Pred.’ denotes trajectory prediction and ’Inter. Type’ refers to the modeled interaction type. △ that group modeling is used internally in the latent space to support trajectory prediction, but is not evaluated as a standalone group prediction objective. Models Method Traj. Inter. Group Group Pred. Type Modeling Inference SocialGAN [6] LSTM ✓ 1st ✗ ✗ Trajectron++ [22] LSTM ✓ 1st ✗ ✗ AgentFormer [29] Transformer ✓ 1st ✗ ✗ Leapfrog [14] Diffusion ✓ 1st ✗ ✗ GroupNet [26] GNN ✓ 1st △ ✗ SocialCircle [25] Transformer ✓ 1st △ ✗ SIGMAS (Ours) Transformer ✓ 1st & 2nd ✓ ✓ Trajectory prediction has long been a core problem in multi-agent systems, with applications ranging from pedestrian modeling to human-robot interaction and autonomous navigation [6, 22, 29]. A key challenge lies in capturing complex inter-agent dependencies, that influence motion, coordination, and intent understanding. Early approaches relied on handcrafted social rules [1] or pooled neighborhood encodings [10], while recent works adopt graph-based and attention-based neural architectures for more expressive interaction modeling. Most existing works model inter-agent interactions through first-order mechanisms, where interactions are captured directly at the node or edge level. Node-centric approaches, such as SocialGAN [6], AgentFormer [29], and Leapfrog [14], focus on how individual agents respond to their neighbors based on proximity, attention, or goal-conditioned latent representations. Edge-centric models, such as Neural Relational Inference (NRI) [9], infer latent pairwise relations between agents by predicting the presence and type of edges that best explain observed dynamics. These methods are effective in low-density environments with moderate agent counts and clearly separated groups, such as pedestrian crowds or physical particle simulations. To address the limitations of purely pairwise modeling, several methods incorporate group-level cues as auxiliary signals to improve prediction accuracy. For example, GroupNet [26] optimizes a social affinity matrix to identify latent groups, while SocialCircle [25] enforces temporal group consistency to enhance trajectory forecasting. Other methods leverage group priors or apply graph partitioning to better structure the interaction space [11, 3, 8]. However, these methods do not explicitly model group emergence as a central objective. In contrast, our work targets a fundamentally different regime: inferring latent group structure in dense, overlapping swarms. In such settings, group identity is not a transient outcome of spatial proximity but an intrinsic, persistent property of each agent, emerging from how similarly they interact with others. This requires new modeling methods that go beyond direct pairwise influence. To this end, we introduce second-order interactions, which measure the similarity between agents’ first-order attention patterns. This mechanism enables robust group inference in densely interacting, dynamically evolving swarms where direct interactions may be ambiguous or unreliable. As summarized in Table 1, SIGMAS is the only framework that simultaneously supports trajectory prediction, second-order interaction reasoning, and self-supervised group inference, all of which are essential for understanding complex group behavior in multi-agent swarms. 3 Problem Statement Since group labels and the number of groups are unavailable during training, we formulate overlapping swarm identification as a joint task of trajectory prediction and latent group inference in a multi-agent swarm. Intuitively, given only the past H-step trajectories of agents (as shown in Figure 1(a)), the model must jointly predict their future motion over T time steps and identify the underlying group structure, i.e., =yellows,greensG=\\yellows\,\greens\\. To formalize this task, let N agents be observed up to time t≤0t≤ 0. This joint state is denoted by t=(x1t,x2t,…,xNt)X^t=(x_1^t,x_2^t,…,x_N^t), where xnt∈ℝdx_n^t ^d is the position of agent n at time t. The observation history up to time 0 is =(−H,…,0)X=(X^-H,…,X^0), with temporal horizon H. For future time steps t>0t>0, the corresponding joint state is t=(y1t,y2t,…,yNt)Y^t=(y_1^t,y_2^t,…,y_N^t), where ynt∈ℝdy_n^t ^d denotes the future position of agent n. The full future trajectories over T time steps is =(1,…,T)Y=(Y^1,…,Y^T). In addition to predicting Y, the goal is to infer latent group memberships for all agents. Let =g1,…,gCG=\g_1,…,g_C\ denote the set of groups, where the number of groups C is unknown during training, and each gc⊆a1,…,aNg_c \a_1,…,a_N\ represents the subset of agents assigned to group c. The overall objective is to jointly estimate both the future trajectories Y and the latent group structure G from past observations X in a self-supervised manner: pθ(,∣)p_θ(Y,G ), where θ denotes the model parameters. This formulation enables joint modeling of motion dynamics and latent group organization without requiring ground-truth group labels during training. 4 SIGMAS Figure 3: SIGMAS architecture. Agent-level and swarm-level features are extracted via separate encoders, and fused using balancing and gating to form swarm-aware embeddings for trajectory prediction and group inference. We propose SIGMAS (Second-order Interaction-based Grouping for Multi-Agent Swarms), a self-supervised framework that learns swarm-aware trajectory embeddings by jointly modeling individual motion and collective coordination. As illustrated in Figure 3, SIGMAS extracts agent-level and swarm-level trajectory embedding separately, and then fuses them to create a unified, swarm-aware representation for each agent. This fused embedding is used to predict future trajectories. In addition, the swarm-level attention matrix from the Swarm-Level Encoder is used to infer latent group structures during testing. The framework consists of three main components: (1) an Agent-Level Encoder that models first-order interactions and individual motion patterns, (2) a Swarm-Level Encoder that captures second-order interaction similarity across agents, and (3) a Balancing & Gating Module that adaptively fuses individual and swarm-level representations. 4.1 Agent-Level Encoder The Agent-Level Encoder captures localized dynamics and direct pairwise interactions among agents. Given temporally encoded trajectories, it produces two types of representations: • Agent-Level Attention: captures pairwise behavioral correlations (first-order interactions) by computing attention scores based on trajectory similarity. • Agent-Level Trajectory Embedding: represents each agent’s individual motion context. We adopt the encoder architecture from AgentFormer [29], which employs an agent-aware spatiotemporal attention mechanism. This allows each agent to attend differently to its neighbors depending on motion context, enabling flexible modeling of time-varying inter-agent dependencies. The resulting attention matrix reflects first-order interaction patterns, as illustrated in Figure 1(b). 4.2 Swarm-Level Encoder Figure 4: Overview of second-order interaction modeling in the Swarm-Level Encoder. (a) First-order interactions encode directional influence between agents. (b) Second-order interactions compare first-order attention patterns across agents, revealing social behavior similarity. The resulting matrix highlights agents with similar interaction profiles, enabling group identification via clustering. In the attention matrix, rows correspond to attending agents and columns to agents being attended to. Darker cells indicate stronger outgoing attention. Swarm-Level Encoder captures social behavior similarity among agents by modeling second-order interactions. It is motivated by the observation that agents within the same swarm tend to interact with others in similar ways, even without direct influence. The architecture is illustrated in Figure 4. We first compute first-order interaction features that capture direct correlations between agents based on their past trajectories. Given the trajectory embeddings ,∈ℝN×dQ,K ^N× d, we compute the first-order attention matrix as (1)=Softmax(⊤d) ^(1)=Softmax ( QK d ) where ij(1)A^(1)_ij represents how strongly agent i attends to agent j. These attention scores reflect directional influence and localized behavioral dependencies among agents. For example, in Figure 4(a), the red agent attends strongly to the purple agent, while the green agent distributes its attention more broadly across the population. To understand group-level structure, we compute second-order interactions by comparing first-order attention patterns across agents: (2)=(1)(1)⊤ ^(2)=A^(1)A^(1) Each entry ij(2)A^(2)_ij measures the similarity between agents i and j based on how similarly they attend to others, thus quantifying their social behavior similarity. If two agents have similar attention distribution, they likely exhibit aligned social behaviors and therefore belong to the same swarm. In Figure 4(b), the green and blue agents both attend strongly to the red and purple agents, indicating similar social behavior and suggesting shared group affiliation. The resulting second-order interaction matrix encodes pairwise similarity in social behavior. During training, we regularize this matrix to be symmetric and positive semi-definite (Section 5.2), enforcing mutual consistency in social similarity. That is, if agent i views agent j as socially similar, the converse must also hold. During testing, (2)A^(2) is passed to the clustering module (Section 5.1), which infers soft group assignments and, for instance, successfully groups the red, yellow, green, and blue agents into the same swarm. 4.3 Agent vs. Swarm Balancing and Gating As shown in Figure 3, the agent-level encoder produces two outputs: a first-order attention map (1)∈ℝHN×HNA^(1) ^HN× HN, which captures pairwise interactions across H observed time steps, and an agent-level trajectory embedding nagent∈ℝHN×df_n^agent ^HN× d. The swarm-level encoder generates a second-order attention matrix (2)∈ℝHN×HNA^(2) ^HN× HN that encodes second-order social similarity between agents. Using this matrix, we derive the swarm-level trajectory embedding via attention-weighted aggregation: ngroup=(2)nagent∈ℝHN×df_n^group=A^(2)f_n^agent ^HN× d. The key challenge lies in how to combine nagentf_n^agent and ngroupf_n^group into a unified representation that balances individual motion cues with group-level coordination. The optimal balance can vary depending on the swarm state, for example, the level of intra-group convergence, overlap across groups, or directional alignment, making static fusion suboptimal. To address this, we introduce an adaptive gating mechanism that fuses the two embeddings in a context-aware manner. A soft gating coefficient αn∈[0,1] _n∈[0,1] is computed for each agent using a multi-layer perceptron (MLP) over its agent-level embedding nagentf_n^agent, swarm-level embedding ngroupf_n^group, and normalized group-assignment entropy ℋnH_n: αn=MLP([nagent‖ngroup‖ℋn]), _n=MLP ([f_n^agent\,\|\,f_n^group\,\|\,H_n] ), where ℋn=−∑j=1HNAnj(2)logAnj(2)/log(HN)H_n=- _j=1^HNA^(2)_nj A^(2)_nj/ (HN) is the normalized entropy of the n-th row of (2)A^(2). This entropy reflects the level of convergence within the swarm. High entropy indicates broad attention across peers (indicating a cohesive swarm), while low entropy suggests focused attention on a few agents (indicating pre-converged or fragmented structures). The final trajectory embedding is computed as: nfinal=(1−αn)⋅nagent+αn⋅ngroup.f_n^final=(1- _n)·f_n^agent+ _n·f_n^group. This dynamic fusion allows the model to adjust the contribution of individual versus group-level cues based on the agent’s social context. 5 SIGMAS-based Prediction Framework Figure 5: SIGMAS-based CVAE prediction framework. Dotted paths are used during training only. We integrate SIGMAS into a conditional variational autoencoder (CVAE)-based multi-agent trajectory prediction framework following [26] (Figure 5). During the encoder stage, both past and future trajectories are used to estimate the posterior distribution parameters (μq,σq)( _q, _q). SIGMAS is applied in this stage to produce swarm-aware trajectory embeddings that capture both individual and group-level behavior via first- and second-order interactions. In the decoder stage, a series of residual blocks take the past trajectory and a sampled latent embedding as input to generate both the predicted future trajectory and a reconstruction of the past. This residual design helps preserve input information while refining future predictions. 5.1 Group Inference via Spectral Clustering. The second-order attention matrix (2)∈ℝN×NA^(2) ^N× N encodes meaningful group-relevant information by capturing similarities in social behavior. At early stages of swarm formation, when agents from the same group are still spatially dispersed, (2)A^(2) may reflect multiple subgroup structures corresponding to partially converged clusters. In this case, a post-processing step during inference is necessary to merge these subgroups based on their dynamic similarity and given the number of groups k. To infer group membership from (2)A^(2), we treat it as a soft adjacency matrix and apply spectral clustering using the normalized graph Laplacian [15]. Specifically, we compute: =−1/2(2)−1/2,L=I-D^-1/2A^(2)D^-1/2, where D is the degree matrix with diagonal entries Dii=∑jAij(2)D_i= _jA^(2)_ij. We then extract the k smallest eigenvectors of L to obtain low-dimensional agent embeddings that capture the global interaction structure. k-means clustering is then applied to these embeddings to produce the predicted group assignments. This procedure is fully self-supervised and relies on the geometry of the learned attention matrix to uncover cohesive swarm structures, even under spatial overlap or incomplete convergence, without requiring ground-truth labels. 5.2 Training Objectives The training objective of SIGMAS consists of three components: a trajectory prediction loss, a swarm-level attention regularization loss, and a balancing factor regularization loss. These are jointly optimized to ensure accurate forecasting while promoting structured group-aware representations. Trajectory Prediction Loss We adopt a composite loss [26] that includes: (1) an evidence lower bound (ELBO) loss from the CVAE formulation, (2) a reconstruction loss on the past trajectory, and (3) a diversity-promoting variety loss following Social-GAN: ℒelbo=‖^+−+‖22+KL((μq,Diag(σq2))∥(0,λI)),L_elbo= \| X^+-X^+ \|_2^2+\,KL\! (N( _q,Diag( _q^2))\,\|\,N(0,λ I) ), ℒrec=‖^−‖22,L_rec= \| X-X \|_2^2, ℒvariety=mink‖^+(k)−+‖22,L_variety= _k \| X^+(k)-X^+ \|_2^2, ℒtraj=ℒelbo+ℒrec+ℒvariety,L_traj=L_elbo+L_rec+L_variety, where X is the reconstructed past trajectory, ^+ X^+ is the predicted future trajectory, +X^+ is the ground-truth future trajectory, and ^+(k) X^+(k) represents the k-th stochastic sample. ∥⋅∥2\|·\|_2 denotes the ℓ2 _2 norm, and KL(⋅∥⋅)KL(·\|·) denotes the Kullback–Leibler divergence. This loss encourages the model to generate accurate, diverse predictions while preserving past dynamics and maintaining a regularized latent representation. Swarm-Level Attention Regularization To ensure that the second-order interaction matrix (2)A^(2) captures meaningful group structure, we introduce a regularization loss on the swarm-level similarity matrix ∈ℝN×NS ^N× N, obtained by averaging (2)A^(2) across the observation horizon. We enforce two key properties to make S a valid affinity kernel: • Symmetry Loss. Social similarity should be mutual: if agent i is behaviorally similar to agent j, then the reverse should also hold. We eencourage symmetry by minimizing: ℒsym=‖−⊤‖F2,L_sym=\|S-S \|_F^2, which penalizes asymmetric affinity scores and promotes bidirectional consistency across agent pairs. • Positive Semi-Definite (PSD) Loss. To ensure S is a valid inner-product kernel for spectral clustering, we encourage it to be positive semi-definite. Specifically, we penalize negative eigenvalues: ℒpsd=∑i=1N[max(0,−λi)]2,L_psd= _i=1^N [ (0,- _i) ]^2, where λi _i is the i-th eigenvalue of S. This regularization ensures geometric coherence in the learned affinity space. The total swarm-level regularization is defined as: ℒswarm=ℒsym+ℒpsd.L_swarm=L_sym+L_psd. These constraints guide (2)A^(2) to form a symmetric, geometrically valid similarity matrix that supports interpretable and robust group identification during inference. Balancing Factor Regularization To enable adaptive fusion between agent-level and swarm-level dynamics, the model learns a gating factor αn∈[0,1] _n∈[0,1] for each agent n, as introduced in Section 4.3. Ideally, αn _n should consider both the degree of swarm convergence as well as agent’s local motion. To guide this behavior, we introduce a regularization loss that aligns the learned αn _n with a target value based on the entropy of the swarm-level attention. Let ℋnH_n denote the normalized entropy of the n-th row of the second-order attention matrix (2)A^(2). We define the target balancing factor as: α~n=1−ℋn. α_n=1-H_n. Low entropy indicates that the agent strongly attends to a small subset of peers, suggesting sub-convergence and a stronger need to rely on swarm-level information. We encourage alignment via the loss: ℒbalance=1N∑n=1N(αn−α~n)2.L_balance= 1N _n=1^N ( _n- α_n )^2. This term promotes dynamic adjustment of the agent’s attention balance based on the current level of social organization, improving adaptability across diverse swarm formations. Total Loss The total objective is a weighted sum of the three terms: ℒtotal=ℒtraj+ℒswarm+ℒbalance,L_total=L_traj+L_swarm+L_balance, 6 Simulation Results 6.1 Experimental Setup Datasets Due to the lack of public datasets for large, overlapping multi-agent swarms, we simulate swarm dynamics using a flocking-based agent simulator (AgentPy [5]) based on Reynolds’ rules [20]. To constrain agent motion, we apply a reflective boundary condition and keep agents bouncing back at the simulation edge. The dataset contains 120 sequences in total: 100 for training, 10 for validation, and 10 for testing. Each sequence consists of 24 agents divided into two groups of 12 agents each, moving over 200 time steps. Agents are initialized with random positions and velocities and gradually converge into coherent swarms through local interactions. Evaluation Metrics Let =(z1,…,zN)z=(z_1,…,z_N) denote the ground-truth group labels and ^=(z^1,…,z^N) z=( z_1,…, z_N) the predicted assignments. We evaluate clustering performance using three standard metrics: (1) Adjusted Rand Index (ARI) [7] : ARI measures pairwise agreement between predicted and true groupings, correcting for chance. Let nijn_ij be the number of agents in true cluster i assigned to predicted cluster j, ai=∑jnija_i= _jn_ij, bj=∑inijb_j= _in_ij, and N=∑i,jnijN= _i,jn_ij. Then: ARI=∑i,j(nij2)−∑i(ai2)∑j(bj2)(N2)12[∑i(ai2)+∑j(bj2)]−∑i(ai2)∑j(bj2)(N2).ARI= _i,j n_ij2- _i a_i2 _j b_j2 N2 12 [ _i a_i2+ _j b_j2 ]- _i a_i2 _j b_j2 N2. ARI ranges from −1-1 (anti-correlation) to 11 (perfect match), with 0 indicating random labeling. (2) Normalized Mutual Information (NMI) [23] : NMI measures the mutual information between ground-truth and predicted clusters, normalized by their entropies. Let pijp_ij, pip_i, and pjp_j denote the joint and marginal probabilities from the contingency table: I(;^)=∑i,jpijlog(pijpipj)I(C; C)= _i,jp_ij ( p_ijp_ip_j ) H()=−∑ipilogpi,H(^)=−∑jpjlogpjH(C)=- _ip_i p_i, H( C)=- _jp_j p_j . The normalized score is: NMI(,^)=2I(;^)H()+H(^).NMI(C, C)= 2I(C; C)H(C)+H( C). which lies in [0,1][0,1], with higher values indicating stronger agreement. (3) F-score [18] : The F-score combines pairwise precision and recall. For all agent pairs, define TPTP (true positives), FPFP (false positives), and FNFN (false negatives). Then: Precision=TPTP+FP,Recall=TPTP+FN,F=2⋅Precision⋅RecallPrecision+Recall.Precision= TPTP+FP, = TPTP+FN, F= 2·Precision·RecallPrecision+Recall. F-score balances both false groupings and missed groupings, with F∈[0,1]F∈[0,1]. 6.2 Validation on Group Inference We validate the group inference capability of SIGMAS in a simulated environment with two swarming groups exhibiting overlapping trajectories and distinct internal dynamics. While both groups follow Reynolds’ flocking rules, they differ in cohesion, alignment, and movement speed, resulting in heterogeneous intra-swarm behavior. No explicit inter-group interaction is applied, creating scenarios where agents from different swarms spatially interleave. Swarm A in Table 2 is used. Figure 6: Comparison of first-order and second-order attention for group inference. (a) Agent trajectories show two overlapping swarms. (b) Agent-level attention (1)A^(1) reflects direct interaction relevance for trajectory prediction. (c) Swarm-level attention (2)A^(2) captures second-order interaction similarity across agents, enabling accurate latent group discovery via spectral clustering. Table 2: Swarm configurations for different scenarios. Swarm Config Cohesion Separation Alignment Speed Swarm A [0.005, 0.007] [0.1, 0.2] [0.3, 0.1] [1.0, 1.2] Swarm B [0.005, 0.010] [0.1, 0.3] [0.1, 0.3] [1.0, 1.3] Swarm C [0.005, 0.010] [0.1, 0.4] [0.1, 0.3] [1.0, 1.5] Latent Clustering This experiment demonstrates how the proposed swarm-level encoder, based on second-order interactions, enables accurate group inference in partially converged, overlapping swarm scenarios. Figure 6(a) visualizes trajectories from the two swarms, where each swarm forms local sub-groups that have not yet fully converged. Effective group prediction in such scenarios requires identifying fine-grained sub-structures while preserving global swarm affiliation. Figure 6(b) presents the first-order attention matrix (1)A^(1) from the agent-level encoder (Section 4.1), which captures direct behavioral correlations. While this matrix is useful for trajectory prediction, it primarily reflect transient spatial dependencies. For example, agents 13, 18, 19 from the yellow swarm attend to nearby green agents to avoid collisions. Such interactions are helpful for motion forecasting but do not reliably indicate shared group membership, making (1)A^(1) less effective for clustering. In contrast, Figure 6(c) presents the second-order attention matrix (2)A^(2) (Section 4.2), which compares first-order attention patterns across agents to capture social behavior similarity. This matrix exhibits a clear block-diagonal structure with low inter-swarm similarity, effectively separating the green and yellow swarms. It also uncovers four latent sub-groups (S1–S4). Spectral clustering analyzes the eigenspace of the graph Laplacian and merges S1 and S2 into one swarm, and S3 and S4 into another, based on the known number of groups. This example demonstrates that second-order interaction modeling captures both intra-swarm coordination and inter-swarm separation, enabling robust group inference even under partial convergence. Individual vs Swarm Balancing Since SIGMAS is trained with trajectory supervision, the utility of latent group information depends on the swarm’s configuration. It is therefore crucial to adaptively balance agent-level and swarm-level embeddings to capture both individual motion and collective behavior. To achieve this, SIGMAS learns a dynamic gating factor α∈[0,1]α∈[0,1] that adjusts the contribution of swarm-level information relative to individual dynamics. Figure 7 shows how α evolves over time in a multi-swarm simulation, along with corresponding snapshots at selected timesteps. Early in the sequence (e.g., T=20T=20), agents are loosely coordinated, so trajectory prediction relies more on individual behavior, resulting in low α. As intra-swarm cohesion increases, agents begin to exhibit more structured group motion. However, when swarms remain well-separated (e.g., T=80T=80), group identity becomes trivial, and individual cues again dominate, lowering α. In contrast, during inter-swarm overlap (e.g., T=60T=60, T=105T=105), spatial proximity between agents from different swarms introduces ambiguity. In these cases, swarm-level context becomes essential for resolving latent affiliations, leading to higher α values. This adaptive gating mechanism enables SIGMAS to dynamically shift its reliance based on the swarm interactions. Figure 7: Adaptive balancing factor α over time in response to swarm dynamics. The model increases α during inter-swarm overlap and decreases it when individual motion cues are more predictive. α=0α=0 corresponds to relying solely on agent-level trajectory embeddings, while α=1α=1 reflects full reliance on swarm-level embeddings. Comparison with Prior Work Table 3: Comparison of group prediction performance across baselines. AgentFormer [29] SIGMAS_IG SIGMAS ARI↑ 0.3155 0.3306 0.4045 NMI↑ 0.3739 0.3860 0.4086 F-score↑ 0.6863 0.6887 0.6968 Figure 8: Qualitative results of group prediction. Red dotted ellipses indicate incorrect group assignments. Group labels are permutation-invariant, so different colors indicate distinct predicted clusters. Results videos are in Supplementary. We evaluate group prediction accuracy by comparing SIGMAS against relevant baselines. As group prediction in swarming systems is a novel task, few existing models are directly applicable. Furthermore, many multi-agent trajectory models are not designed to handle large-scale swarms. We consider the following baselines: • AgentFormer [29]: A transformer-based trajectory model that uses self-attention to capture pairwise interactions. For group prediction, we apply spectral clustering to its final attention matrix. • SIGMAS_IG: A variant of our model that uses both encoders during training but applies spectral clustering to the agent-level attention matrix (1)A^(1) instead of the second-order matrix. Our full model, SIGMAS, performs group inference using (2)A^(2) from the swarm-level encoder, which captures second-order social behavior similarity. Figure 8 presents qualitative results. AgentFormer and SIGMAS_IG frequently produce incorrect groupings in cases with spatial overlap or partial convergence, often grouping agents with similar motion direction rather than true swarm affiliation (see red ellipses). In contrast, SIGMAS successfully recovers the underlying group structure by leveraging second-order interactions, enabling robust inference even under ambiguous conditions. These observations are reflected in Table 3, where SIGMAS achieves better performance across all clustering metrics. 6.3 Case study: Intra-Swarm Heterogeneity Figure 9: Group prediction performance across three heterogeneous swarm configurations (Swarm A, B, C) with increasing intra-swarm disparity. Shaded regions indicate standard deviation across multiple runs. Results videos are in Supplementary. To evaluate the robustness of SIGMAS under varying swarm dynamics, we conduct experiments across three heterogeneous swarm settings (Swarm A, B, C in Table 2). In each case, the two swarms differ in cohesion, separation, alignment, and speed, simulating different levels of swarm heterogeneity. A single SIGMAS model trained on Swarm A is evaluated on all three configurations to assess its generalizability. Figure 9 shows the Adjusted Rand Index (ARI) over time for all three scenarios. As agents gradually converge into coherent swarms, ARI improves steadily. Importantly, SIGMAS achieves consistent performance across all settings despite the growing heterogeneity in swarm dynamics. This indicates that SIGMAS reliably infers group structure even under diverse intra-swarm dynamics. Similar trends are observed for NMI and F-score, confirming the model’s generalizability to a wide range of heterogeneous swarm behaviors. 7 Conclusions Swarming systems exhibit complex, emergent behaviors driven by dense local interactions. Unlike traditional multi-agent settings, multiple swarms often operate in overlapping regions while maintaining persistent group identities, making latent group structure inference both critical and underexplored. In this paper, we introduce the novel task of group prediction in overlapping multi-agent swarms, where the objective is to recover group memberships directly from trajectory data without ground-truth supervision. This problem setting requires modeling beyond local motion interactions to capture global coordination behavior inherent in swarm dynamics To address this challenge, we propose SIGMAS, a self-supervised framework that jointly captures individual and collective behavior through a novel second-order interaction mechanism. By measuring how similarly agents attend to others, SIGMAS quantifies social behavior similarity and supports robust group inference even in spatially entangled or partially converged swarm formations. An adaptive gating module fuses agent-level and swarm-level embeddings based on social context. Experiments demonstrate that SIGMAS outperforms prior baselines and generalizes well across diverse heterogeneous swarm scenarios. Acknowledgments This work is supported by the Defense Advanced Research Projects Agency (DARPA) under Grant Numbers GR00019153. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Department of Defence, DARPA, or the U.S. Government. References [1] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese (2016) Social lstm: human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 961–971. Cited by: §2.2. [2] F. A. Almalki and M. C. Angelides (2024) A synthesis of machine learning and internet of things in developing autonomous fleets of heterogeneous unmanned aerial vehicles for enhancing the regenerative farming cycle. Computing 106 (12), p. 4167–4192. Cited by: §1. [3] I. Bae, J. Park, and H. Jeon (2022) Learning pedestrian group representations for multi-modal trajectory prediction. In European Conference on Computer Vision, p. 270–289. Cited by: §2.2. [4] J. Boubin, C. Burley, P. Han, B. Li, B. Porter, and C. Stewart (2021) Programming and deployment of autonomous swarms using multi-agent reinforcement learning. arXiv preprint arXiv:2105.10605. Cited by: §1. [5] J. Foramitti (2021) AgentPy: a package for agent-based modeling in python. Journal of Open Source Software 6 (62), p. 3065. Cited by: §2.1, §6.1. [6] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi (2018) Social gan: socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 2255–2264. Cited by: §2.2, §2.2, Table 1. [7] L. Hubert and P. Arabie (1985) Comparing partitions. Journal of classification 2 (1), p. 193–218. Cited by: §6.1. [8] S. Kim, H. Chi, H. Lim, K. Ramani, J. Kim, and S. Kim (2024) Higher-order relational reasoning for pedestrian trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 15251–15260. Cited by: §2.2. [9] T. Kipf, E. Fetaya, K. Wang, M. Welling, and R. Zemel (2018) Neural relational inference for interacting systems. In International conference on machine learning, p. 2688–2697. Cited by: §2.2. [10] N. Lee, W. Choi, P. Vernaza, C. B. Choy, P. H. Torr, and M. Chandraker (2017) Desire: distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 336–345. Cited by: §2.2. [11] S. Lee, J. Lee, Y. Yu, T. Kim, and K. Lee (2024) Mart: multiscale relational transformer networks for multi-agent trajectory prediction. In European Conference on Computer Vision, p. 89–107. Cited by: §2.2. [12] A. Lerner, Y. Chrysanthou, and D. Lischinski (2007) Crowds by example. In Computer graphics forum, Vol. 26, p. 655–664. Cited by: §1. [13] S. MahmoudZadeh, A. Yazdani, A. Elmi, A. Abbasi, and P. Ghanooni (2022) Exploiting a fleet of uavs for monitoring and data acquisition of a distributed sensor network. Neural Computing and Applications, p. 1–14. Cited by: §1. [14] W. Mao, C. Xu, Q. Zhu, S. Chen, and Y. Wang (2023-06) Leapfrog diffusion model for stochastic trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 5517–5526. Cited by: §2.2, Table 1. [15] A. Ng, M. Jordan, and Y. Weiss (2001) On spectral clustering: analysis and an algorithm. Advances in neural information processing systems 14. Cited by: §5.1. [16] P. R. Norambuena, J. Bekios-Calfa, and J. M. Torres (2023) Study of machine learning techniques for pedestrian dynamics simulation models. In 2023 IEEE 13th International Conference on Pattern Recognition Systems (ICPRS), Vol. , p. 1–8. External Links: Document Cited by: §1. [17] S. Pellegrini, A. Ess, K. Schindler, and L. Van Gool (2009) You’l never walk alone: modeling social behavior for multi-target tracking. In 2009 IEEE 12th international conference on computer vision, p. 261–268. Cited by: §1. [18] D. M. Powers (2020) Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation. arXiv preprint arXiv:2010.16061. Cited by: §6.1. [19] A. Pretto, S. Aravecchia, W. Burgard, N. Chebrolu, C. Dornhege, T. Falck, F. Fleckenstein, A. Fontenla, M. Imperoli, R. Khanna, F. Liebisch, P. Lottes, A. Milioto, D. Nardi, S. Nardi, J. Pfeifer, M. Popović, C. Potena, C. Pradalier, E. Rothacker-Feder, I. Sa, A. Schaefer, R. Siegwart, C. Stachniss, A. Walter, W. Winterhalter, X. Wu, and J. Nieto (2021) Building an aerial–ground robotics system for precision farming: an adaptable solution. IEEE Robotics & Automation Magazine 28 (3), p. 29–49. External Links: Document Cited by: §1. [20] C. W. Reynolds (1987) Flocks, herds and schools: a distributed behavioral model. In Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques, New York, NY, USA, p. . Cited by: §2.1, §6.1. [21] A. Ribeiro and J. Conesa-Muñoz (2021) Multi-robot systems for precision agriculture. Innovation in Agricultural Robotics for Precision Agriculture: A Roadmap for Integrating Robots in Precision Agriculture, p. 151–175. Cited by: §1. [22] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone (2020) Trajectron++: dynamically-feasible trajectory forecasting with heterogeneous data. In European Conference on Computer Vision, p. 683–700. Cited by: §2.2, Table 1. [23] A. Strehl and J. Ghosh (2002) Cluster ensembles—a knowledge reuse framework for combining multiple partitions. Journal of machine learning research 3 (Dec), p. 583–617. Cited by: §6.1. [24] E. Vlasceanu, D. Popescu, and L. Ichim (2019) Aerial robotic team for complex monitoring in precision agriculture. In 2019 15th International Conference on Distributed Computing in Sensor Systems (DCOSS), Vol. , p. 167–169. Cited by: §1. [25] C. Wong, B. Xia, Z. Zou, Y. Wang, and X. You (2024-06) SocialCircle: learning the angle-based social interaction representation for pedestrian trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 19005–19015. Cited by: §1, §2.2, Table 1. [26] C. Xu, M. Li, Z. Ni, Y. Zhang, and S. Chen (2022) GroupNet: multiscale hypergraph neural networks for trajectory prediction with relational reasoning. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2.2, Table 1, §5.2, §5. [27] C. Xu, R. T. Tan, Y. Tan, S. Chen, X. Wang, and Y. Wang (2023-10) Auxiliary tasks benefit 3d skeleton-based human motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 9509–9520. Cited by: §1. [28] Ö. Yildirim, K. Diepold, and R. A. Vural (2019) Decision process of autonomous drones for environmental monitoring. In 2019 IEEE International Symposium on INnovations in Intelligent SysTems and Applications (INISTA), Vol. , p. 1–6. External Links: Document Cited by: §1. [29] Y. Yuan, X. Weng, Y. Ou, and K. M. Kitani (2021) Agentformer: agent-aware transformers for socio-temporal multi-agent forecasting. In Proceedings of the IEEE/CVF international conference on computer vision, Piscataway, NJ, USA, p. 9813–9823. Cited by: §2.2, §2.2, Table 1, §4.1, 1st item, Table 3.