Paper deep dive
End-to-End Cell Detection via Instance-aware Graph Modeling
Ruochen Liu, Yalin Zheng, Jingxin Liu, Jianfeng Zhang, Shoujun Huang, Dexing Kong, Haofeng Li, Wei Lou
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Accurate cell detection and classification are crucial for pathological analysis, directly affecting diagnostic accuracy and treatment planning. To capture complex cellular interactions beyond visual appearance within the tumor microenvironment, several approaches have employed graph neural networks to model spatial and relational patterns among cell nuclei, yielding promising results. However, these methods typically adopt a two-stage paradigm of visual extraction followed by relational modeling, which necessitates separate tuning for each stage, thereby increasing pipeline complexity and hindering end-to-end joint optimization. In this paper, we propose an end-to-end framework for cell detection and classification that jointly models patch-level visual representations and instance-level interactions, which incorporates a dynamic graph construction module and an instance-aware graph network. Specifically, the graph construction module dynamically builds the graph structure using learnable queries derived from patch-level features as cell instance representations, with adjacency defined by integrating feature similarity and spatial distances. The instance-aware graph network performs adaptive instance filtering and feature reorganization, aggregating them over the cell graph into a topological latent state for a selective state-space transition driven by visual cues, fusing appearance and relational evidence. When evaluated on multiple datasets with different staining protocols for cell and nucleus detection, our method significantly outperforms existing approaches in both detection and classification performance. The code will be released at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.15354v1
- Canonical: https://arxiv.org/abs/2609.15354v1
Trouble viewing inline? Open PDF directly →
Full Text
71,669 characters extracted from source content.
Expand or collapse full text
[orcid=0009-0008-5112-7489] [orcid=0000-0002-7873-0922] [orcid=0000-0001-6071-9197] [orcid=0000-0001-9120-9843] [orcid=0000-0002-2071-4081] End-to-End Cell Detection via Instance-aware Graph Modeling Ruochen Liu sgrliu18@liverpool.ac.uk organization=Faculty of Science and Engineering, University of Liverpool, city=Liverpool, postcode=L69 3GJ, country=U.K. Yalin Zheng yzheng@liverpool.ac.uk organization=Department of Eye and Vision Sciences, University of Liverpool, city=Liverpool, postcode=L7 8TX, country=U.K. Jingxin Liu Jingxin.Liu@xjtlu.edu.cn organization=School of AI and Advanced Computing, Xi’an Jiaotong-Liverpool University, city=Suzhou, postcode=215400, country=China Jianfeng Zhang jfzhang@zjnu.edu.cn organization=College of Mathematical Medicine, Zhejiang Normal University, city=Jinhua, postcode=321004, country=China Shoujun Huang sjhuang@zjnu.edu.cn Dexing Kong dkong@zju.edu.cn organization=School of Mathematical Sciences, Zhejiang University, city=Hangzhou, postcode=310027, country=China Haofeng Li lihf95@mail.sysu.edu.cn organization=School of Systems Science and Engineering, Sun Yat-sen University, city=Guangzhou, postcode=510275, country=China Wei Lou louwei@zjnu.edu.cn Abstract Accurate cell detection and classification are crucial for pathological analysis, directly affecting diagnostic accuracy and treatment planning. To capture complex cellular interactions beyond visual appearance within the tumor microenvironment, several approaches have employed graph neural networks to model spatial and relational patterns among cell nuclei, yielding promising results. However, these methods typically adopt a two-stage paradigm of visual extraction followed by relational modeling, which necessitates separate tuning for each stage, thereby increasing pipeline complexity and hindering end-to-end joint optimization. In this paper, we propose an end-to-end framework for cell detection and classification that jointly models patch-level visual representations and instance-level interactions, which incorporates a dynamic graph construction module and an instance-aware graph network. Specifically, the graph construction module dynamically builds the graph structure using learnable queries derived from patch-level features as cell instance representations, with adjacency defined by integrating feature similarity and spatial distances. The instance-aware graph network performs adaptive instance filtering and feature reorganization, aggregating them over the cell graph into a topological latent state for a selective state-space transition driven by visual cues, fusing appearance and relational evidence. When evaluated on multiple datasets with different staining protocols for cell and nucleus detection, our method significantly outperforms existing approaches in both detection and classification performance. The code will be released at https://github.com/RuochenLiu23/IGM. keywords Cell and Nucleus Detection ,Graph Learning ,End-to-End Learning ,Query-based Detection ,Selective State Space Model †corresponding: Corresponding author 1 Introduction Histopathological images are the gold standard for diagnosing and grading complex diseases such as cancer, playing a critical role in clinical diagnostics and public health [4]. Cell detection and classification, which involve identifying cell or nucleus locations and predicting their types, provide essential quantitative and qualitative support for diagnosis and prognosis [43]. However, manual analysis is labor-intensive and costly due to the thousands of cells typically present in whole slide images (WSIs) [3, 40]. Although computer-aided methods enable rapid and accurate cell identification [40], their performance is challenged by dense clustering, extensive overlap, and adhesion that make adjacent instances difficult to separate [41, 16, 50, 29], ambiguous boundaries caused by uneven staining and imaging noise [41, 26, 47, 29], and high cellular heterogeneity that manifests as intra-sample variability in morphology and staining intensity as well as differences across laboratory protocols [16, 54, 38, 36, 33]. Figure 1: Comparison of different cell detection and classification approaches incorporating graph models: (A) Graph construction is initialized using the output of a pre-trained detector. (B) Graph construction is initialized based on the feature map. (C) Ours: Dynamic graphs are initialized using learnable queries. The Instance-aware Graph Network (IGN) performs adaptive noise filtering and feature reorganization, enabling effective capture of local‑global relationships. Extensive research has been dedicated to this challenging task, with cell and nucleus detection approaches broadly categorized into three groups: point-based, instance segmentation, and instance detection methods [33]. Point-based methods [1, 52, 53, 51] predict only centroids and their categories to reduce computational complexity and annotation costs. However, they lack explicit regional information, making it challenging to recover cell shape or boundary details. This limits their ability to support downstream morphological analyses such as chromatin pattern characterization [61, 68] or mitotic figure quantification [2, 19]. In contrast, instance segmentation methods [41, 16, 50, 29, 54, 38, 36, 7, 23] offer pixel-level delineation of individual cells but demand extensive annotations and high computational resources [55]. Instance detection [19, 11, 67, 33] strikes a favorable balance: it provides explicit regional localization via bounding boxes, with lower annotation and computational overhead than segmentation, while offering richer spatial detail than point-based approaches, making it well-suited for clinical deployment. Accordingly, we aim to develop an accurate and efficient framework for cell detection and classification. Within the instance detection paradigm, query-based Transformer models [5, 62, 64, 44] excel by using learnable queries to represent objects. However, in cell detection, their reliance solely on visual and positional cues leads to ambiguity in dense, overlapping, or morphologically heterogeneous regions with high inter-class similarity. Pathology-specific methods [41, 16, 50, 7, 23] mitigate this by incorporating geometric priors (distance maps or contour fitting) but require pixel-level annotations and are incompatible with detection-only settings. With the rise of pathology foundation models, recent methods counter such ambiguity by enriching instance representation with pre-trained features, yet the objective gap between representation learning and coordinate regression degrades these features under joint optimization, necessitating decoupled multi-stage training [60]. In general, these approaches follow the conventional computer vision paradigm, modeling cells largely in isolation, prioritizing local appearance over inter-instance contextual relationships. This limits detection performance in complex tissue architectures where cellular identity is inherently relational. To address these challenges in pathology, previous studies have focused on intercellular interactions to provide additional cues, leading to the exploration of graph-based methods to model such contextual relationships in the tissue microenvironment. Two-stage approaches [28, 37, 20, 35, 13, 63] (Figure 1(A)) first detect or segment cells and then build graphs from cropped regions, but suffer from error propagation from imperfect proposals and increased training complexity due to separate per-stage optimization. In contrast, pixel- or superpixel-based methods [39, 14] (Figure 1(B)) construct graphs directly from feature maps with pixels as nodes, yet struggle to model meaningful inter-instance relationships while producing overly dense and noisy structures that obscure instance-level information. Inspired by these observations, we propose an end-to-end framework to enhance instance awareness in query-based cell detection and classification. As shown in Figure 1(C), our method treats learnable queries as graph nodes which represent candidate cell instances and construct a context-aware graph. The graph connectivity is determined by both spatial proximity and feature similarity, enabling dynamic adaptation to diverse cell distributions and morphologies. However, since the number of queries typically far exceeds the true cell count, the graph contains redundant and spurious nodes, introducing significant structural noise. This poses a major challenge for conventional GNNs [31], which rely solely on local message passing and thus struggle to capture long-range dependencies in such large, noisy graphs. Although Transformer-based GNNs [58, 30] can model global relationships, their dense attention mechanisms incur substantial computational overhead, especially when applied to noisy graphs with excessive nodes. Irrelevant nodes in such cases not only inflate computation cost but also distort attention, undermining inter-instance relationship modeling. To address this, we leverage the selective State Space Model (SSM) [18], which effectively suppresses noise while preserving global structural patterns, thereby enabling robust relational modeling at linear computational cost. We embed the graph topology into the SSM, jointly modeling inter-cellular relations and visual representations. In this paper, our framework comprises three key components: a query feature learning network, a dynamic graph construction module (DGC), and an instance-aware graph network (IGN). The query feature learning network fuses initial query features with tissue contextual and pathological features extracted from patches, producing enriched query embeddings that encode instance-level semantics, bridging patch-level features to instance-level representations. The DGC module takes the queries as graph nodes and dynamically constructs the graph structure based on both spatial distance and feature similarity among queries. This adaptive construction strategy allows the graph to better capture varying cell distributions and morphologies, enhancing the model’s robustness during training. The IGN integrates two synergistic components: an Instance-aware Graph Learning (IGL) module, in which Selective Feature Reorganization (SFR) gates out redundant queries and reorganizes the retained ones into a directionally aligned latent space, whereupon graph aggregation superposes neighbors into near-orthogonal subspaces, yielding a noise-resistant topological representation; and a Topology-structured State Space (TSS) layer, where the state is transitioned along the cell-graph topology and query’s visual feature drives state update, so that relational and visual evidence accumulate within a unified state. To account for the high semantic complexity of histopathological images, both IGL and TSS are augmented with a Context-Guided State Encoding module, which injects context-aware priors into dynamic state transitions. Finally, the refined node features from the TSS are decoded into cell instance predictions, including bounding box regression and class labels. Our main contributions are summarized as follows: 1. We propose a new end-to-end cell detection framework that leverages learnable queries for graph construction and relational modeling, aiming to enhance instance awareness and detection performance. 2. We propose an instance-aware graph network that recasts graph learning as a topology-structured state space model, in which selective reorganization and orthogonal aggregation yield a noise-resistant topological structural representation as the latent state driven by visual evidence, fusing relational and appearance cues within the state transition. 3. We propose a dynamic graph construction method that integrates both the spatial relationships and feature similarities of queries to build graphs and in turn adaptively adjusts their structure in response to dynamic updates in query content. 4. Experimental results demonstrate that our framework achieves state-of-the-art performance on multiple public benchmarks for cell and nucleus detection across different staining protocols. 2 Related Work 2.1 Instance Detection for Cells and Nuclei Cell detection and classification typically rely on instance segmentation or point detection. Segmentation methods [50, 16, 54, 7, 23, 56] like Hover-Net [16] and StarDist [50] provide detailed masks but require costly pixel-wise labels and are computationally expensive. Point-detection approaches [1, 52, 53, 51], while efficient, only predict centroids and lack geometric information needed for morphometric analysis. Bounding-box-based instance detection [19, 11, 67, 33] offers a balanced alternative, providing richer spatial localization than point detection while avoiding the overhead of full segmentation. Recent query-based detectors [5, 62, 64, 44, 66, 34, 69, 8, 46, 45] such as Cell-DETR [45] and DINO [62, 44] enable end-to-end, anchor-free instance detection with strong global context modeling. However, these models have seen limited exploration in the field of pathology, as they primarily involve the adaptation of general-purpose models, leaving a need for more pathology-specific approaches. These limitations underscore the necessity of a cell instance detection framework. 2.2 Graph-Based Learning in Pathology In computational pathology, graph-based cell modeling typically involves two stages: graph construction and graph representation learning. For graph construction, two strategies dominate: (1) Two-stage methods [28, 37, 20, 35, 13] first detect or segment cells to form semantically meaningful nodes and connect them via spatial proximity, but they suffer from error propagation and pipeline complexity. (2) End-to-end pixel- or patch-level approaches [14] build graphs directly from feature maps using k-nearest neighbors (KNN), but often produce redundant or biologically irrelevant edges. For graph representation learning, Graph Convolutional Networks (GCNs) [31, 15, 6, 10] and Graph Attention Networks (GATs) [58] are widely adopted. GCNs perform fixed-weight neighborhood aggregation but suffer from limited receptive fields and over-smoothing in deep architectures. GATs improve adaptivity by assigning attention-based weights to neighbors, but standard attention mechanisms incur quadratic computational complexity with respect to the number of nodes [57], which hinders scalability in dense cellular graphs containing thousands of instances. Recently, Mamba has been introduced into graph learning [12], offering global modeling at linear complexity. Motivated by these observations, in this work, we propose an efficient dynamic graph learning approach that reformulates the state transition of a selective SSM over the cell graph topology, thereby achieving global contextual modeling of relational and visual representations at low complexity. 3 Method In this section, we introduce the proposed end-to-end cell detection and classification framework, which is shown in Figure 2. The framework consists of three key components: (1) a query feature learning network that learns tissue and pathological features, extracts high-quality query representations from patches, achieving the mapping from patch-level to instance-level features; (2) a dynamic graph construction strategy that adaptively builds the cell graph by jointly leveraging spatial proximity and feature similarity among learnable queries; and (3) an instance-aware graph network that learns cell-level features by modeling local and global relational contexts with the visual evidence to enhance the discriminative ability of query features for both instance separation and class prediction. Figure 2: Overview of the whole framework. The proposed end-to-end learning framework comprises three key components: a query feature learning network for initializing query embeddings with patch-to-instance-level mapping, a dynamic graph construction (DGC) module for building the cell graph, and an instance-aware graph network (IGN) for effective graph modeling of inter-cell interactions, all jointly optimized during training. 3.1 Query Feature Learning Network Given a pathological image of size H×W×3H× W× 3, it is fed into a backbone network to obtain multi-scale visual features fmulti=f0,f1,…,flf_multi=\f_0,f_1,…,f_l\. These features are flattened and concatenated into an initial global feature f∈ℝNenc×Cf ^N_enc× C, where NencN_enc is the total number of spatial positions across all scales. A 2D sinusoidal positional encoding is generated for each spatial location, flattened, and added to f, resulting in fenc∈ℝNenc×Cf_enc ^N_enc× C. The features are then processed by a Transformer encoder using multi-scale deformable attention [66], which captures fine-grained pathological details and aggregates contextual information from the tissue microenvironment to produce enhanced visual representations fenc′∈ℝNenc×Cf _enc ^N_enc× C. From fenc′f _enc, we generate initial object proposals. Specifically, each spatial position corresponds to a candidate box =[cx,cy,w,h] a=[c_x,c_y,w,h], where (cx,cy)(c_x,c_y) denote the center coordinates and (w,h)(w,h) are randomly initialized dimensions. After applying class-agnostic non-maximum suppression (NMS), the top-N high-scoring proposals are selected. Their features are extracted from fenc′f _enc at (cx,cy)(c_x,c_y) and linearly transformed to form the anchor features as∈ℝN×Ca_s ^N× C. Similar to common detection pipelines [66, 62, 64], for each anchor, a lightweight prediction head takes its associated feature to compute a score and a bounding-box regression offset, facilitating the selection of high-quality detection proposals: we first apply class-agnostic non-maximum suppression (NMS) on their boxes to remove duplicates, then keep the top-N anchors by score (N≫N the actual object count). To improve the quality of query embeddings, the selected anchor features as∈ℝN×Ca_s ^N× C are added to a set of learnable query vectors q′∈ℝN×Cq ^N× C, yielding the final query features Qenc∈ℝN×CQ_enc ^N× C. These learnable query vectors serve as instance-level priors for potential objects (e.g., cells), which are randomly initialized and optimized during training but remain fixed during inference. The query features QencQ_enc, together with 4D reference points (cx,cy,w,h)(c_x,c_y,w,h) normalized to [0,1][0,1], are fed into the Transformer decoder. In the Transformer decoder, each decoder layer iteratively updates the query features. First, positional embeddings are generated from the current reference points using sine-based encoding followed by a MLP layer. Each query is updated through self-attention and multi-scale deformable cross-attention, enabling it to gather visual evidence from relevant regions across encoder feature scales. Owing to the dense sampling of initial queries, the receptive fields of queries overlap with each other. To further reduce redundancy and ensure the distinctness of queries for accelerating model convergence, between decoder layers, the model selects a subset of high-quality, non-overlapping queries using class-agnostic NMS with an IoU threshold of 0.8, based on the current 4D reference points and their classification scores. Following the design of [64], a subset of queries determined by the configuration is treated as dense and routed to the auxiliary head for refinement to ensure the integrity of feature representations, while they are not used for final predictions at inference. In addition, 100 denoising (DN) queries are added only during training as noisy copies of ground-truth boxes and labels to provide direct supervision and speed up convergence. Finally, the Transformer decoder outputs the updated query features Q and predicts the offset of the selected anchor positions. Combining these offsets with the original anchor allows us to calculate the updated reference points, as well as the extraction of the center coordinates P∈ℝN×2P ^N× 2, providing key spatial cues for subsequent cell graph construction and relationship modeling. 3.2 Dynamic Graph Construction Existing graph-based cell classification methods typically use K-Nearest Neighbor (KNN) based on spatial proximity to construct cell graphs [28, 37, 20, 14]. However, in our end-to-end framework, the 2D coordinates of reference points are not always reliable. Furthermore, solely spatial information has limited capacity to model complex intercellular distributions and interactions, rendering such constructions sensitive to noise and harmful to convergence. To address this, we propose a Dynamic Graph Construction (DGC) strategy that combines both query embeddings and positional information to define a more robust query distance function in KNN, enabling accurate relationship modeling. Notably, by constructing the graph over queries, whose features and reference points are dynamically updated during training, the cell graph structure is refined via gradient propagation through query features. This dynamic refinement enhances flexibility and avoids the limitations of fixed graph topologies. Based on the query embeddings Q and reference points P output by the query feature learning network, we design a query distance function s for KNN computation, which consists of two components: spatial distance sps^p and feature distance sfs^f. For spatial distance, the distance is defined via an exponential decay of the Euclidean distance between any two reference points pip_i and pjp_j: sijp=exp(−‖pi−pj‖22d¯)s_ij^p= (- \|p_i-p_j\|_22 d ) (1) where ‖pi−pj‖2\|p_i-p_j\|_2 is the Euclidean distance between pip_i and pjp_j, and d¯ d denotes the mean distance between all pairs of reference points. A large sps^p indicates stronger spatial correlation. For feature distance, we first perform L2 normalization on the query embeddings Q to eliminate the influence of feature scale. Subsequently, we measure feature distance sijfs_ij^f using cosine similarity between any two queries qiq_i and qjq_j. The formula is as follows: sijf=qi⋅qjT∥qi∥2∥qj∥2s_ij^f= q_i· q_j^T q_i _2\, q_j _2 (2) In the end, the query distance function can be described as: sij=α⋅(1−sijp)+(1−α)⋅(1−sijf)s_ij=α·(1-s_ij^p)+(1-α)·(1-s_ij^f) (3) where α is a hyper-parameter that controls the contribution weights of the two distances. For dynamic balance in this specific setting, α is set to 0.5. Then, we apply the KNN algorithm to establish node connectivity based on the query distance sijs_ij, resulting in an adjacency matrix M∈ℝN×NM ^N× N, where n is the number of queries. The constructed graph is defined as G=(V,E)G=(V,E), where each node vi∈Vv_i∈ V corresponds to a query embedding q, and edges E are determined by the KNN connections. 3.3 Instance-aware Graph Network We propose an Instance-aware Graph Network (IGN) for dynamic query-based cell detection, as illustrated in Figure 3. IGN comprises two tightly coupled components: the Instance-aware Graph Learning (IGL) and the Topology-structured State Space (TSS) layer. In IGL, Selective Feature Reorganization (SFR) applies a selective mechanism to suppress redundant query features and reorganizes the retained features into a directionally structured latent space. The resulting representations are then aggregated over the dynamic cell graph, where near-orthogonal directions suppress interference from semantically inconsistent neighbors, yielding a noise-resistant topological representation of semantic associations. Within TSS, the state is updated over this representation under the drive of instance-level query features, integrating contextual information with global relational dependencies. A Context-Guided State Encoding (CGSE) module abstracts all queries into an instance-level context that steers the state-transition parameters of both SFR and TSS, guiding adaptation to instance-specific feature distributions. Figure 3: The proposed Dynamic Graph Construction (DGC) strategy and Instance-aware Graph Network (IGN) are employed to construct and model the query-based graph for simulating inter-cell interactions. 3.3.1 Instance-aware Graph Learning Within this module, selective feature reorganization is coupled with graph aggregation. SFR first gates out redundant queries and lifts the retained ones into a directionally structured latent space, rendering dissimilar cells near-orthogonal. Aggregating these over the dynamic cell graph then consolidates the orthogonalized node representations with graph connectivity into a unified topological structural representation, wherein inconsistent neighbors reside in mutually orthogonal subspaces and remain non-interfering. Similarity-driven selective aggregation therefore arises from the state geometry itself, obviating explicit edge pruning and rendering the structure robust to redundant nodes. Selective Feature Reorganization. Given a set of nodes V, we flatten their node features into a sequence x∈ℝN×dx ^N× d and feed it into the Selective Feature Reorganization (SFR) module. Inspired by the selective state space model [18], we apply a selective projection to adaptively filter and reconstruct each node feature xi∈ℝdx_i ^d, where the projection matrices Δ∈ℝN×d ^N× d and ∈ℝN×nB ^N× n are dynamically determined by Context-Guided State Encoding, yielding the hidden state hi∈ℝd×nh_i ^d× n. hi=(⊙xi)⊗ih_i= ( x_i ) _i (4) where ⊗ denotes the outer product. Δi∈ℝd _i ^d serves as an element-wise amplitude gate that controls the amount of information written from each node feature into the state space. By implicitly encoding the node information through CGSE, Δ achieves content-adaptive modulation, enabling effective filtering and compression of noisy node features in a fully data-driven manner. i∈ℝnB_i ^n expands each scalar component of xi∈ℝdx_i ^d onto the direction it spans, yielding hi∈ℝd×nh_i ^d× n. Lifting node features transforms the entangled space into a directionally discriminative one, mapping similar node vectors hih_i to proximal directions and dissimilar ones to near-orthogonal directions, whereby the affinity between two states factorizes as cos(hi,hj)=cos(x~i,x~j)⋅cos(i,j) (h_i,h_j)= ( x_i, x_j)· (B_i,B_j) with x~i=i⊙xi x_i= _i x_i, sharpening the contrast between related and unrelated node pairs. Directional Graph Aggregation. Building on these near-orthogonal features, we aggregate them over the cell graph, injecting graph connectivity to form a topological representation. We remap all hi∈ℝd×nh_i ^d× n as node features h∈ℝN×(d⋅n)h ^N×(d· n) into a cell graph G, whose connectivity is established using the adjacency matrix M∈ℝN×NM ^N× N derived from the dynamic graph construction strategy. To balance node degree differences and enhance stability, we first perform symmetric normalization on it [31], followed by aggregating adjacent features based on structural information. The relevant formulas are as follows: h^=−1/2⋅(M+I)⋅−1/2⋅h h=D^-1/2·(M+I)·D^-1/2· h (5) where I denotes the identity matrix for adding self-loops, D represents the degree matrix, and the two −1/2D^-1/2 terms perform symmetric normalization on the target and neighbor nodes respectively. Node-wise, each node takes a degree-normalized weighted sum of its own state and those of its neighbors, whereby the graph connectivity determines which neighbors are included and the normalization determines how much each contributes. Since each node’s state is confined to the direction assigned by its own iB_i, this weighted sum acts as a superposition along distinct directions rather than a blending of values. Semantically inconsistent neighbors, whose states point in near-orthogonal directions, therefore enter the aggregated state without contaminating the target node’s own state and remain separable at readout. Selectivity thus resides in the geometry rather than in the edges, requiring no explicit pruning, and the aggregated h h constitutes a noise-resistant topological structural representation consolidating node features with graph connectivity. 3.3.2 Topology-structured State Space Due to the complexity and global nature of graph structures, we employ Mamba [18] to model global features. Specifically, we propose the Topology-structured State Space (TSS) layer, which adopts the topological structure as the latent state and drives its update with the visual cues of the queries, thereby constructing a topology-structured state transition that yields the output y: H=¯h^+¯x,y=H= A h+ Bx, y=CH (6) In contrast to a canonical state space update, driven by the state of the preceding token, here the cell topology h h obtained by graph aggregation serves as the incoming state, with the visual features x continually re-injected to drive each update. Instance-level appearance is thereby assimilated into the topological structure, unifying relational and appearance evidence within a single state H. Since Mamba is defined on the continuous time, Zero-Order Hold (ZOH) discretization is applied to adapt the continuous-time model to discrete node sequence inputs and facilitate training: ¯=exp(Δ),¯=(Δ)−1(exp(Δ)−)⋅Δ A= ( ), B=( )^-1 ( ( )-I )· (7) Herein, A and B are all parameter matrices, which are discretized based on the parameter Δ and their specific parameters are determined by CGSE. Unlike Transformer, which explicitly computes pairwise similarities at quadratic complexity to obtain the global state, Mamba implicitly encodes global instance and topological context into a compact hidden state H. Transformer attention weights, by contrast, tend to disperse over complex graph structures, leaving irrelevant node signals difficult to suppress. Within TSS, ¯ A adaptively determines, conditioned on the representation of the corresponding instance, how much of the accumulated topological structure survives each update, so that the contextual topology aggregated onto visually substantiated cells persists. ¯ B writes the visual feature of each query into the very state that carries the topology, along the direction spanned by iB_i and scaled to the salience of that instance, thereby driving the state forward at every step. C performs the inverse operation, projecting the fused state back onto content-dependent directions to retrieve for each node its relevant global context, which keeps the readout selective at linear complexity. Meanwhile, cells admit no canonical ordering within tissue and thus form an unordered set rather than a sequence. We process nodes in parallel and keep the topological propagation permutation-equivariant, thereby avoiding the ordering bias inherent to sequential formulations. To enhance feature propagation and enrich representational capacity, a learnable feedforward matrix D is introduced to form a residual-style connection, yielding the IGN output Qf=y+x0Q_f=y+Dx_0. The fused features QfQ_f are fed into the classification head to predict the object categories. Simultaneously, QfQ_f and its corresponding reference point are used by the regression head to predict the bounding box coordinates. Finally, predictions are matched to ground truths via the Hungarian algorithm, and the overall loss is computed accordingly. ℒ=λclsℒcls+λbboxℒbbox+λiouℒiouL= _clsL_cls+ _bboxL_bbox+ _iouL_iou (8) where ℒbboxL_bbox is the L1 Loss, ℒiouL_iou is the GIoU Loss [48], and ℒclsL_cls employs Focal Loss [32] to address the positive-negative sample imbalance in pathological images. 3.3.3 Context-Guided State Encoding. In SSM, A, B, C, and are the key matrices governing the state transition. We propose a Context-Guided State Encoding (CGSE) module to generate these matrices from instance features, enabling adaptive, knowledge-guided node control. The state transition matrix A is initialized via HiPPO-LegS [17], providing a stable multi-scale decay spectrum for task-relevant retention and noise suppression during hidden state propagation. As A encodes task-level global decay structure, it is shared globally across all nodes as a single learnable parameter. For matrices B and C [18, 22], we employ a linear transformation to learn knowledge xW_x from the cell instance information. Following the tensor generation mechanism used in self-attention [57], this cellular knowledge, encompassing the morphological appearance and spatial distribution of cell instances, is transformed through learned mappings into corresponding matrices, enabling adaptive processing of nodes guided by all cell instance features. The calculation process is as follows: =f(x,x)B,=f(x,x)CB=f(x,W_x)W_B, =f(x,W_x)W_C (9) For the Δ , we use an activation function Softplus [18] to adapt the characteristics of the modulation matrix: =log(1+exp(f(x,x)⋅δ⋅dt)) = (1+ (f(x,W_x)·W_δ·W_dt) ) (10) Due to the large output dimensionality of Δ , a low-rank structure δ⋅dtW_δ·W_dt with δ∈ℝd×rW_δ ^d× r and dt∈ℝr×dW_dt ^r× d (r≪dr d), is additionally employed to control the parameter count. 4 Experiment 4.1 Datasets We evaluated our method on three public histopathological datasets: CoNSeP [16], CytoDArk0 [56], and OCELOT [49], with annotations converted to bounding box labels. CoNSeP comprises 41 H&E-stained 1000×1000-pixel images of colorectal adenocarcinoma at 40× magnification with nucleus-level instance segmentation masks. CytoDArk0 includes Nissl-stained mammalian brain tissue images at 40× magnification with cell-level masks for neurons and glial cells. OCELOT is derived from H&E-stained WSIs in the TCGA database, with cell nuclei annotated. To balance efficiency and resolution, 128×128 patches are extracted from CoNSeP due to its denser cell distribution, while 256×256 patches are used for CytoDArk0 and OCELOT. Evaluations on nuclear detection, cell detection, and multi-type datasets with diverse staining methods confirm the model’s adaptability across pathological detection tasks. Table 1: Performance comparison of different methods on instance detection tasks across three datasets. Dataset Methods AR mAP mAP50 mAP75 CoNSeP Deformable-DETR (ICLR’21) 49.3 24.5 50.3 25.2 DAB-DETR (ICLR’22) 49.6 24.9 50.5 25.5 DINO (ICLR’23) 54.8 28.5 52.4 28.9 CO-DINO (ICCV’23) 53.3 29.8 54.6 29.2 DDQ-DETR (CVPR’23) 56.5 30.0 54.4 31.4 Relation-DETR (ECCV’24) 55.1 29.7 54.3 30.3 Mamba-YOLO (AAAI’24) 51.3 25.2 50.7 23.6 DECO (ICLR’25) 53.5 26.9 51.4 25.7 CellMamba (BMVC’25) 53.2 25.7 51.1 23.8 Ours 58.7 32.0 57.0 33.5 CytoDArk0 Deformable-DETR (ICLR’21) 67.5 52.1 80.1 59.3 DAB-DETR (ICLR’22) 67.8 52.6 80.5 59.4 DINO (ICLR’23) 69.9 58.2 84.7 67.1 CO-DINO (ICCV’23) 70.0 59.0 85.6 68.7 DDQ-DETR (CVPR’23) 70.3 59.6 86.3 69.5 Relation-DETR (ECCV’24) 70.1 58.8 85.9 68.2 Mamba-YOLO (AAAI’24) 67.4 52.3 81.2 56.6 DECO (ICLR’25) 68.9 54.6 81.9 63.1 CellMamba (BMVC’25) 68.2 53.3 83.5 59.8 Ours 71.7 61.7 88.5 71.8 OCELOT Deformable-DETR (ICLR’21) 54.5 31.5 68.4 24.8 DAB-DETR (ICLR’22) 54.7 31.2 68.9 24.4 DINO (ICLR’23) 55.2 33.3 71.0 27.3 CO-DINO (ICCV’23) 54.9 33.5 71.0 27.5 DDQ-DETR (CVPR’23) 55.5 33.5 71.1 27.2 Relation-DETR (ECCV’24) 55.1 32.8 70.6 26.0 Mamba-YOLO (AAAI’24) 54.7 30.9 68.2 23.6 DECO (ICLR’25) 55.0 32.1 68.8 25.9 CellMamba (BMVC’25) 54.3 31.7 69.0 25.4 Ours 56.8 34.9 72.3 29.3 4.2 Implementation Details All experiments were conducted on an NVIDIA A40 GPU. We adopt ResNet-50 as the backbone, ensuring that the performance advantage stems from the detection architecture design itself rather than from backbone capacity. For training, we employed the AdamW optimizer with an initial learning rate of 2e-4 and a weight decay coefficient of 0.05. For the hyperparameter K in KNN adopted in Dynamic Graph Construction, experimental validation demonstrates that setting K=8K=8 on CoNSeP, K=3K=3 on CytoDArk0, and K=4K=4 on OCELOT achieves the optimal performance. To ensure fairness in the comparison, we unified the number of queries to 900 for general DETR-based methods. 4.3 Evaluation Setup and Metrics To comprehensively evaluate our model, we compare against both detection and classification methods. The comparison with detection methods spans general-purpose and pathology-specific models: general-purpose detectors, though not designed for histopathology, have shown strong capability on pathological tasks, while pathology-specific models are largely adapted from such frameworks, and both are therefore included for a thorough validation. We then compare with classification methods on the multi-class dataset, allowing us to assess our classification performance against pathology-specific models. In terms of metrics, we adopt standard detection metrics including Average Precision (mAPmAP), mAP50mAP_50, mAP75mAP_75, and Average Recall (ARAR) to assess detection and classification performance for detection methods across confidence thresholds. For comparison with cell-specific classification methods, which are mostly supervised with masks or points and differ in output granularity, we follow [45] and adopt the F1-score, the standard metric for classification, as a consistent measure across methods. 4.4 Comparison with the Existing Methods 4.4.1 Comparison with Detection Methods We evaluate our framework on CoNSeP, CytoDArk0, and OCELOT against state-of-the-art detectors. Specifically, Deformable-DETR [66] enhances feature learning; DAB-DETR [34], DINO [62], and DDQ-DETR [64] improve query design; CO-DINO [69] introduces more comprehensive supervision; Relation-DETR [24] exploits attention-based query-feature interaction; while Mamba-YOLO [59], DECO [8], and CellMamba [33] prioritize computational efficiency by leveraging the strengths of Mamba and convolution. As reported in Table 1, our method achieves top performance across all three benchmarks. On CoNSeP, it surpasses DDQ-DETR [64] by 2.0% in mAPmAP and 2.2% in ARAR, the most notable improvement among the three benchmarks. CoNSeP is also the most challenging, with dense nuclei and high inter-class similarity hindering both instance separation and type discrimination, explicitly modeling inter-cell relations showing a clear advantage over appearance-driven detection here. On CytoDArk0, the margin over DDQ-DETR [64] is 2.1% in mAPmAP and 1.4% in ARAR. On OCELOT, consistent superiority is maintained across all metrics. All improvements are statistically significant (p<0.05p<0.05). These results demonstrate that modeling inter-cell relationships via our instance-aware graph network provides critical relational cues absent in general-purpose detectors, validating its efficacy for pathological cell instance detection. We also visualize detection results from DINO, CO-DINO, DDQ-DETR, and our model in Figure 4 across different staining protocols, where (A) and (B) are from Nissl staining, and (C)–(E) are from H&E staining. The results confirm that our method detects more cells while reducing false positives, particularly for densely packed and adherent cells. Figure 4: Visualization of cell detection results. 4.4.2 Comparison with Classification Methods Since most existing nucleus classification methods rely on segmentation masks or point annotations, we further compare our detection-based approach against representative end-to-end mask-based and point-based cell instance classification models. As shown in Table 2, we select a set of representative models for comparison, including DIST [41], Micro-Net [47], Hover-Net [16], Triple-UNet [65], MCSpatNet [1], TSFD-Net [27], Mask2Former [9], PGT [25], ACFormer [26], SMILE [42], CellViT [23], Cell-DETR [45] and PathContext [51]. Our method achieves state-of-the-art performance across all metrics, outperforming mask-based methods with richer pixel-level supervision and point-based methods with inherently lower task complexity. These results validate the effectiveness of our detection-based framework for accurate cell localization and classification, particularly under severe class imbalance in pathological images. Notably, FcmF_c^m varies most substantially across methods, with several failing to identify this class entirely, as its scarcity and atypical morphology afford few reliable appearance cues. Our method attains the highest FcmF_c^m, suggesting that contextual dependencies among neighboring cells yield discriminative evidence beyond individual appearance. Table 2: Performance comparison of nuclei classification methods and our approach on the CoNSeP dataset, reporting detection, mean and per-class F1 scores. † Mask-based methods. ‡ Point-based methods. ⋆ Bounding Box-based methods. Model FdF_d FavgF_avg FcmF_c^m FciF_c^i FceF_c^e FcsF_c^s Model FdF_d FavgF_avg FcmF_c^m FciF_c^i FceF_c^e FcsF_c^s DIST† (TMI’18) 0.71 0.42 0.00 0.53 0.62 0.51 PGT‡ (MICCAI’23) 0.74 - - 0.62 0.64 - Micro-Net† (MIA’19) 0.74 0.47 0.12 0.59 0.62 0.53 ACFormer‡ (ICCV’23) 0.74 - - 0.64 0.64 - Hover-Net† (MIA’19) 0.75 0.57 0.43 0.63 0.64 0.57 SMILE† (MIA’23) 0.76 0.56 0.38 0.62 0.67 0.58 Triple-UNet† (MIA’20) 0.66 0.38 0.10 0.57 0.42 0.44 CellViT† (MIA’24) 0.76 0.58 0.46 0.64 0.65 0.57 MCSpatNet‡ (ICCV’21) 0.73 0.51 0.40 0.54 0.58 0.54 Cell-DETR† (MIDL’24) 0.74 0.49 0.21 0.63 0.61 0.51 TSFD-Net† (N’22) 0.68 0.44 0.12 0.57 0.56 0.51 PathContext‡ (AAAI’26) 0.76 0.59 0.49 0.65 0.65 0.57 Mask2Former† (CVPR’22) 0.66 0.41 0.33 0.46 0.46 0.41 Ours⋆ 0.77 0.61 0.52 0.66 0.67 0.59 4.5 Ablation Study We conduct ablation studies on the CoNSeP dataset to validate the design choices of our framework, as reported in Table 3. We ablate two key components: the Dynamic Graph Construction and the Instance-aware Graph Network (IGN). First, we evaluate the importance of dynamic graph construction by ablating either feature similarity or positional similarity, using only one cue for graph building while keeping all other components fixed. Removing positional similarity reduces mAPmAP from 32.0% to 31.4% (-0.6%) and ARAR from 58.7% to 57.3% (-1.4%). Removing feature similarity causes a larger drop: mAPmAP falls to 30.6% (-1.4%) and ARAR to 56.6% (-2.1%), confirming that both cues are essential, where feature similarity contributes more substantially to overall performance. Second, we validate the effectiveness of each component within IGN. Starting from the baseline (mAPmAP: 30.0%, ARAR: 56.5%), integrating the Instance-aware Graph Learning (IGL) module improves mAPmAP by 0.8%, and further incorporating the Topology-structured State Space (TSS) yields an additional 1.2% gain. To further verify the overall effectiveness of IGN, we replace it with alternative graph learners: the convolution-based GCN [31], the Transformer-based GAT [58] and TokenGT [30], and the Mamba-based DMbaGCN [21]. Our IGN outperforms all variants by clear margins, demonstrating its superiority. Table 3: Ablation study on CoNSeP dataset. Model Variant AR mAP mAP50 mAP75 Ablation of Dynamic Graph Construction w/o Feat. Sim. 56.6 30.6 54.9 31.8 w/o Pos. Corr. 57.3 31.4 55.5 32.9 Ours 58.7 32.0 57.0 33.5 Ablation of Instance-aware Graph Network Baseline (Query Feature Learning Network) 56.5 30.0 54.4 31.4 + IGL 57.4 30.8 56.1 32.3 + TSS (Ours) 58.7 32.0 57.0 33.5 GCN (ICLR’17) 56.3 30.2 54.6 31.7 GAT (ICLR’19) 56.7 30.5 55.0 32.0 TokenGT (NeurIPS’22) 56.9 31.1 55.7 32.6 DMbaGCN (AAAI’26) 57.2 30.9 55.5 32.1 Table 4: Computational complexity comparison. † Mask-based methods. ‡ Point-based methods. ⋆ Bounding Box-based methods. Method Para. (M) FLOP (G) Hover-Net† [16] 34.76 1992 Mask2Former† [9] 298.21 896 CellViT† [23] 142.85 3567 DINO⋆ [62] 47.55 279 CO-DINO⋆ [69] 65.49 - DDQ-DETR⋆ [64] 48.32 270 PathContext‡ [51] 48.08 186 Ours⋆ 49.57 276 4.6 Efficiency analysis We compare the computational complexity of our method with representative models on the CoNSeP dataset, as summarized in Table 4. Our model (49.57M parameters, 276 GFLOPs) is substantially lighter than instance segmentation methods and comparable to point-based models, yet accomplishes a more complex task by providing instance-level representations approximating segmentation quality via bounding box detection. The marginal overhead over DDQ-DETR (+1.25M parameters, + 6 GFLOPs) confirms that our graph-based refinement module introduces minimal additional cost, achieving a favorable trade-off between efficiency and detection performance. 4.7 Investigation of Hyper-parameters 4.7.1 Investigation of α-values In constructing the query distance function, we incorporate both spatial distance and feature distance, balanced by a weighting factor α. Therefore, we perform a hyperparameter study on α to determine its optimal setting. As shown in Figure 5(a), the optimal performance is achieved at α=0.5α=0.5, indicating that spatial distance and feature distance contribute equally and complementarily to the query distance function. Accordingly, α=0.5α=0.5 is adopted in our model. 4.7.2 Investigation of Query Numbers The number of queries plays a critical role in query-based detectors, as it directly determines the upper bound on the number of detectable instances. Therefore, we conduct a dedicated hyperparameter study on query numbers. As shown in Figure 5(b), increasing N consistently improves performance. Raising N from 100 to 300 boosts ARAR by 2.3% and mAP50mAP_50 by 1.9%; further increasing to 900 yields additional gains of 3.8% and 2.3%, respectively, indicating better candidate coverage and feature querying. When the number of queries is further increased, while AR improves, precision metrics decrease. We thus adopt 900 as the number of queries for our model to achieve the optimal balance between recall and precision. Larger values were not tested due to GPU memory constraints. (a) Different α-values. (b) Different query numbers. Figure 5: Hyper-parameter studies on CoNSeP. 4.7.3 Investigation of K-values In handling complex graphs, the graph construction step is of critical importance: the graph topology directly governs the neighborhood aggregation, which in turn conditions the subsequent global modeling. Herein, we utilize KNN to establish node connectivity, and the setting of K value is particularly vital to graph construction. Therefore, we conduct additional experiments on this hyperparameter K. Based on empirical observations, the selection of K can be broadly categorized into two regimes: sparse graph connectivity (K∈[3,5]K∈[3,5]) and dense graph connectivity (K∈[7,9]K∈[7,9]). We first conduct pilot experiments with K=5K=5 and K=8K=8 as representative values of each regime to determine which connectivity pattern is more suitable for each dataset. Based on these results, we further evaluate adjacent K values within the identified regime to pinpoint the optimal setting. As summarized in Table 5, on CoNSeP where nuclei are densely packed and highly overlapping, the best performance is achieved at K=8K=8, outperforming sparser graphs (e.g., K=5K=5) by up to 2.5% in mAPmAP. In contrast, CytoDArk0 contains well-separated cells, and its peak performance occurs at K=3K=3, while OCELOT achieves the best results at K=4K=4. Table 5: Performance comparison of different K values on detection tasks across three datasets. Dataset K values AR (%) mAP (%) mAP50 (%) mAP75 (%) CoNSeP 5 54.7 29.5 54.5 30.4 7 56.5 31.3 56.7 32.3 8 58.7 32.0 57.0 33.5 9 55.8 30.9 55.8 32.0 CytoDArk0 3 71.7 61.7 88.5 71.8 4 71.3 61.1 88.1 70.8 5 70.4 60.8 87.7 70.7 8 68.5 57.6 86.7 67.9 OCELOT 3 56.5 34.8 72.1 29.1 4 56.8 34.9 72.3 29.3 5 56.4 34.1 71.8 28.3 8 56.6 33.5 70.9 27.1 5 Discussion The central insight of this work is that cellular identity in histopathology is inherently relational. Intercellular interactions and signaling constitute the biophysical basis of structural integration and functional coordination in multicellular systems, such that the state of a cell is shaped not merely by its own morphology but largely by its interactions within the surrounding microenvironment. Existing detectors, however, model cells largely in isolation and prioritize local appearance, an approach that falters in dense, overlapping, or morphologically heterogeneous regions where visual cues alone cannot resolve instance identity. Certain pathology-specific methods instead introduce relational context through graph structures, yet predominantly follow a two-stage paradigm that decouples graph construction from detection: the graph is built upon the outputs of an independently trained detector or segmentor, serving only as post-hoc refinement, propagating upstream errors and precluding optimization under a unified objective. Neither line of work treats relational structure as something to be learned jointly with detection, which motivates a framework in which relational reasoning is not appended to detection but intrinsic to it. To realize an end-to-end graph-based detection framework, we reconceive what constitutes a graph node. Rather than deriving nodes from the outputs of an upstream detector or from pixels on a feature map, we take learnable queries as candidate cell instances and construct the graph directly over them. Through decoding, each query aggregates multi-scale visual evidence from the tissue microenvironment, thereby transforming patch-level contextual representations into instance-level semantic descriptions upon which graph construction and relational reasoning operate. A single representation thus spans the entire path from patch-level feature extraction, through instantiation, to inter-instance relational modeling, allowing the graph to focus on instances themselves and all three stages to be optimized under a unified objective. Connectivity is determined jointly by spatial proximity and feature similarity, enabling the graph to encode organizational regularities that neither cue expresses alone. The resulting structure is genuinely dynamic: as query embeddings evolve through learning, the induced adjacency shifts accordingly; conversely, gradients propagated through graph learning continually refine the query features themselves, so that representation and topology shape each other over training and erroneous connections arising from unreliable reference points or immature features are not permanently entrenched. The topology is thereby shaped dynamically by the data rather than fixed a priori by spatial heuristics. We next turn to how such a graph ought to be processed. As the number of queries far exceeds that of true cells, the graph abounds in redundant nodes, over which dense attention merely disperses irrelevant signals at quadratic cost. The state transition of a selective state space model, by contrast, is inherently endowed with content-adaptive selectivity, suppressing irrelevant signals while preserving global structural patterns at a complexity that scales linearly with the number of nodes. Departing from the denoise-then-model paradigm, we let graph learning itself yield a structure that the state space model can carry: once selectively reorganized, node states unfold along their respective directions, whereby aggregation becomes a superposition across distinct directions and semantically inconsistent neighbors, being near-orthogonal, remain mutually non-interfering. Conventional denoising schemes must assess neighbor relevance through attention weights or edge pruning, whereas filtering here follows directly from directional relations: irrelevant neighbors require no separate identification or removal, their contribution having attenuated substantially over end-to-end training. Building upon this, we recast the state transition of the SSM, taking the resulting topological representation as the latent state while the visual features of the queries are re-injected to drive its evolution, so that relational and appearance evidence accumulate within a single state. CGSE further abstracts all queries into an instance-level context from which the transition parameters are generated, enabling the dynamics to adapt to the feature distribution of each individual instance rather than conforming to a uniformly imposed prior. Within the overall architecture, neither NMS nor KNN, despite being non-differentiable, impedes end-to-end training. Both act as parameter-free structural selectors conditioned on the current query states: NMS deterministically retains high-quality, non-redundant candidates, while KNN establishes connectivity from the prevailing query features. Neither introduces learnable parameters nor interrupts any gradient path, and the features of the retained queries participate fully in backpropagation. The dynamic character of the graph arises not from any adjustment within KNN itself, but from the continual learning of the query embeddings upon which it operates, thereby establishing an implicit feedback loop between feature learning and graph topology that constitutes the core mechanism of end-to-end joint optimization throughout the framework. 6 Conclusion In this paper, we have presented a novel end-to-end framework for cell detection and classification, designed to enhance the instance discrimination capability of query-based methods through structured reasoning. By treating learnable queries as graph nodes, our approach dynamically constructs a context-aware topology using both spatial and feature cues, enabling adaptive modeling of complex and varying cell distributions. To address noise and long-range dependency challenges in large query sets, we introduce an instance-aware graph network (IGN) that formulates a Topology-structured State Space (TSS) model with graph topology as the latent state, driving global modeling. Extensive experiments demonstrate the effectiveness of our method, achieving state-of-the-art performance across multiple benchmarks. This work highlights the potential of combining query-based detection with structured graph learning, offering a promising direction for precise and robust cell instance analysis in digital pathology. References [1] S. Abousamra, D. Belinsky, J. V. Arnam, F. Allard, E. Yee, R. Gupta, T. Kurc, D. Samaras, J. Saltz, and C. Chen (2021) Multi-class cell detection using spatial context representation. In IEEE/CVF International Conference on Computer Vision, p. 4005–4014. Cited by: §1, §2.1, §4.4.2. [2] M. Aubreville, N. Stathonikos, C. A. Bertram, and et al. (2023) Mitosis domain generalization in histopathology images—the midog challenge. Medical Image Analysis 84, p. 102699. Cited by: §1. [3] S. M. Ayyad, M. Shehata, A. Shalaby, M. A. El-Ghar, M. Ghazal, M. El-Melegy, N. B. Abdel-Hamid, L. M. Labib, H. A. Ali, and A. El-Baz (2021) Role of ai and histopathological images in detecting prostate cancer: a survey. Sensors 21, p. 2586. Cited by: §1. [4] V. Baxi, R. Edwards, M. Montalto, and S. Saha (2022) Digital pathology and artificial intelligence in translational medicine and clinical practice. Modern Pathology 35, p. 23–32. Cited by: §1. [5] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko (2020) End-to-end object detection with transformers. In European Conference on Computer Vision, p. 213–229. Cited by: §1, §2.1. [6] M. Chen, Z. Wei, Z. Huang, B. Ding, and Y. Li (2020) Simple and deep graph convolutional networks. In International conference on machine learning, p. 1725–1735. Cited by: §2.2. [7] S. Chen, C. Ding, M. Liu, J. Cheng, and D. Tao (2023) CPP-net: context-aware polygon proposal network for nucleus segmentation. IEEE Transactions on Image Processing 32, p. 980–994. Cited by: §1, §1, §2.1. [8] X. Chen, S. Li, Y. Yang, and Y. Wang (2025) DECO: unleashing the potential of convnets for query-based detection and segmentation. In International Conference on Learning Representations, Cited by: §2.1, §4.4.1. [9] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar (2021) Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §4.4.2, Table 4. [10] E. Chien, J. Peng, P. Li, and O. Milenkovic (2021) Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations, Cited by: §2.2. [11] Q. Da, X. Huang, Z. Li, and et al. (2022) DigestPath: a benchmark dataset with challenge review for the pathological detection and segmentation of digestive-system. Medical Image Analysis 80, p. 102485. Cited by: §1, §2.1. [12] M. Duan, Z. Yang, Y. Ma, M. Wang, and Z. Song (2025) Knowledge-guided multi-scale graph mamba for whole slide image classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention, Cited by: §2.2. [13] A. L. Frei, J. Garcia-Baroja, T. Rau, C. Neppl, A. Lugli, W. Solass, M. Wartenberg, A. Fischer, and I. Zlobec (2026) GrEp: graph-based epithelial cell classification refinement in histopathology h&e images. Pattern Recognition 171, p. 112197. Cited by: §1, §2.2. [14] Z. Gao, Z. Lu, J. Wang, S. Ying, and J. Shi (2022) A convolutional neural network and graph convolutional network based framework for classification of breast histopathological images. IEEE Journal of Biomedical and Health Informatics 26, p. 3163–3173. Cited by: §1, §2.2, §3.2. [15] J. Gasteiger, A. Bojchevski, and S. Günnemann (2019) Predict then propagate: graph neural networks meet personalized pagerank. In International Conference on Learning Representations, Cited by: §2.2. [16] S. Graham, Q. D. Vu, S. E. A. Raza, A. Azam, Y. W. Tsang, J. T. Kwak, and N. Rajpoot (2019) Hover-net: simultaneous segmentation and classification of nuclei in multi-tissue histology images. Medical Image Analysis 58, p. 101563. Cited by: §1, §1, §1, §2.1, §4.1, §4.4.2, Table 4. [17] A. Gu, T. Dao, S. Ermon, A. Rudra, and C. Ré (2020) HiPPO: recurrent memory with optimal polynomial projections. In Advances in Neural Information Processing Systems, p. 1474–1487. Cited by: §3.3.3. [18] A. Gu and T. Dao (2024) Mamba: linear-time sequence modeling with selective state spaces. In Conference On Language Modeling, Cited by: §1, §3.3.1, §3.3.2, §3.3.3, §3.3.3. [19] H. Gu, M. Haeri, S. Ni, C. K. Williams, N. Zarrin-Khameh, S. Magaki, and X. A. Chen (2022) Detecting mitoses with a convolutional neural network for midog 2022 challenge. In MICCAI Challenge on Mitosis Domain Generalization, Cited by: §1, §2.1. [20] T. Hassan, S. Javed, A. Mahmood, T. Qaiser, N. Werghi, and N. Rajpoot (2022) Nucleus classification in histology images using message passing network. Medical Image Analysis 79, p. 102480. Cited by: §1, §2.2, §3.2. [21] X. He, Y. Wang, Y. Dai, and X. Wang (2026) Dual mamba for node-specific representation learning: tackling over-smoothing with selective state space modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, Cited by: §4.5. [22] X. He, Y. Wang, W. Fan, X. Shen, X. Juan, R. Miao, and X. Wang (2025) Mamba-based graph convolutional networks: tackling over-smoothing with selective state space. In arXiv:2501.15461, Cited by: §3.3.3. [23] F. Hörst, M. Rempe, L. Heine, C. Seibold, J. Keyl, G. Baldini, S. Ugurel, J. Siveke, B. Grünwald, J. Egger, and J. Kleesiek (2024) CellViT: vision transformers for precise cell segmentation and classification. Medical Image Analysis 94, p. 103143. Cited by: §1, §1, §2.1, §4.4.2, Table 4. [24] X. Hou, M. Liu, S. Zhang, P. Wei, B. Chen, and X. Lan (2024) Relation detr: exploring explicit position relation prior for object detection. In European conference on computer vision, Cited by: §4.4.1. [25] J. Huang, H. Li, W. Sun, X. Wan, and G. Li (2023) Prompt-based grouping transformer for nucleus detection and classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention, Cited by: §4.4.2. [26] J. Huang, H. Li, X. Wan, and G. Li (2023) Affine-consistent transformer for multi-class cell nuclei detection. In IEEE/CVF International Conference on Computer Vision, p. 21384–21393. Cited by: §1, §4.4.2. [27] T. Ilyas, Z. I. Mannan, A. Khan, S. Azam, H. Kim, and F. D. Boer (2022) TSFD-net: tissue specific feature distillation network for nuclei segmentation and classification. Neural Networks 151, p. 1–15. Cited by: §4.4.2. [28] G. Jaume, P. Pati, B. Bozorgtabar, A. Foncubierta, A. M. Anniciello, F. Feroce, T. Rau, J. Thiran, M. Gabrani, and O. Goksel (2021) Quantifying explainers of graph neural networks in computational pathology. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 8106–8116. Cited by: §1, §2.2, §3.2. [29] H. Jiang, R. Zhang, Y. Zhou, Y. Wang, and H. Chen (2023) DoNet: deep de-overlapping network for cytology instance segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 15641–15650. Cited by: §1, §1. [30] J. Kim, D. Nguyen, S. Min, S. Cho, M. Lee, H. Lee, and S. Hong (2022) Pure transformers are powerful graph learners. In Advances in Neural Information Processing Systems, p. 14582–14595. Cited by: §1, §4.5. [31] T. N. Kipf and M. Welling (2017) Semi-supervised classification with graph convolutional networks. In International conference on learning representations, Cited by: §1, §2.2, §3.3.1, §4.5. [32] T. Lin, P. Goyal, R. Girshick, K. He, and P. Dollar (2017) Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, Cited by: §3.3.2. [33] R. Liu, Y. Tian, J. Wang, H. Liu, X. Hou, and J. Liu (2025) CellMamba: adaptive mamba for accurate and efficient cell detection. In arXiv:2512.21803, Cited by: §1, §1, §2.1, §4.4.1. [34] S. Liu, F. Li, H. Zhang, X. Yang, X. Qi, H. Su, J. Zhu, and L. Zhang (2022) Dab-detr: dynamic anchor boxes are better queries for detr. In International Conference on Learning Representations, Cited by: §2.1, §4.4.1. [35] W. Lou, G. Li, X. Wan, and H. Li (2024) Cell graph transformer for nuclei classification. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 3873–3881. Cited by: §1, §2.2. [36] W. Lou, H. Li, G. Li, X. Han, and X. Wan (2022) Which pixel to annotate: a label-efficient nuclei segmentation framework. IEEE Transactions on Medical Imaging 42, p. 947–958. Cited by: §1, §1. [37] W. Lou, X. Wan, G. Li, X. Lou, C. Li, F. Gao, and H. Li (2024) Structure embedded nucleus classification for histopathology images. IEEE Transactions on Medical Imaging 43, p. 3149–3160. Cited by: §1, §2.2, §3.2. [38] J. Ma, R. Xie, S. Ayyadhury, and et al. (2024) The multimodality cell segmentation challenge: toward universal solutions. Nature Methods 21, p. 1103–1113. Cited by: §1, §1. [39] Y. Meng, H. Zhang, Y. Zhao, X. Yang, Y. Qiao, I. J. C. MacCormick, X. Huang, and Y. Zheng (2021) Graph-based region and boundary aggregation for biomedical image segmentation. IEEE Transactions on Medical Imaging 41, p. 690–701. Cited by: §1. [40] D. Y. Z. Miao Cui (2021) Artificial intelligence and computational pathology. Laboratory Investigation 101, p. 412–422. Cited by: §1. [41] P. Naylor, M. Laé, F. Reyal, and T. Walter (2018) Segmentation of nuclei in histopathology images by deep regression of the distance map. IEEE Transactions on Medical Imaging 38, p. 448–459. Cited by: §1, §1, §1, §4.4.2. [42] X. Pan, J. Cheng, F. Hou, R. Lan, C. Lu, L. Li, Z. Feng, H. Wang, C. Liang, Z. Liu, X. Chen, C. Han, and Z. Liu (2023) SMILE: cost-sensitive multi-task learning for nuclear segmentation and classification with imbalanced annotations. Medical Image Analysis 88, p. 102867. Cited by: §4.4.2. [43] X. Pan, D. Yang, L. Li, Z. Liu, H. Yang, Z. Cao, Y. He, Z. Ma, and Y. Chen (2018) Cell detection in pathology and microscopy images with multi-scale fully convolutional neural networks. World Wide Web 21, p. 1721–1743. Cited by: §1. [44] M. Pang, T. K. Roy, X. Wu, and K. Tan (2025) CelloType: a unified model for segmentation and classification of tissue images. Nature Methods 22, p. 348–357. Cited by: §1, §2.1. [45] O. Pina, E. Dorca, and V. Vilaplana (2024) Cell-detr: efficient cell detection and classification in wsis with transformers. In Medical Imaging with Deep Learning, Cited by: §2.1, §4.3, §4.4.2. [46] T. Prangemeier, C. Reich, and H. Koeppl (2020) Attention-based transformers for instance segmentation of cells in microstructures. In IEEE International Conference on Bioinformatics and Biomedicine, p. 700–707. Cited by: §2.1. [47] S. E. A. Raza, L. Cheung, M. Shaban, S. Graham, D. Epstein, S. Pelengaris, M. Khan, and N. M. Rajpoot (2019) Micro-net: a unified model for segmentation of various objects in microscopy images. Medical Image Analysis 52, p. 160–173. Cited by: §1, §4.4.2. [48] H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese (2019) Generalized intersection over union: a metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §3.3.2. [49] J. Ryu, A. V. Puche, J. Shin, S. Park, B. Brattoli, J. Lee, W. Jung, S. I. Cho, K. Paeng, C. Ock, D. Yoo, and Sérgio (2023) Ocelot: overlapped cell on tissue dataset for histopathology. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 23902–23912. Cited by: §4.1. [50] U. Schmidt, M. Weigert, C. Broaddus, and G. Myers (2018) Cell detection with star-convex polygons. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 265–273. Cited by: §1, §1, §1, §2.1. [51] Z. Shui, R. Guo, H. Li, Y. Sun, Y. Zhang, C. Zhu, J. Cai, P. Chen, Y. Su, and L. Yang (2026) Towards effective and efficient context-aware nucleus detection in histopathology whole slide images. In Proceedings of the AAAI Conference on Artificial Intelligence, Cited by: §1, §2.1, §4.4.2, Table 4. [52] Z. Shui, S. Zhang, C. Zhu, B. Wang, P. Chen, S. Zheng, and L. Yang (2022) End-to-end cell recognition by point annotation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 109–118. Cited by: §1, §2.1. [53] Z. Shui, S. Zheng, C. Zhu, S. Zhang, X. Yu, H. Li, J. Li, P. Chen, and L. Yang (2024) DPA-p2pnet: deformable proposal-aware p2pnet for accurate point-based cell detection. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 4864–4872. Cited by: §1, §2.1. [54] C. Stringer, T. Wang, M. Michaelos, and M. Pachitariu (2021) Cellpose: a generalist algorithm for cellular segmentation. Nature Methods 18, p. 100–106. Cited by: §1, §1, §2.1. [55] N. Tajbakhsh, L. Jeyaseelan, Q. Li, J. N. Chiang, Z. Wu, and X. Ding (2020) Embracing imperfect datasets: a review of deep learning solutions for medical image segmentation. Medical Image Analysis 63, p. 101693. Cited by: §1. [56] V. Vadori, J. Graïc, A. Peruffo, G. Vadori, L. Finos, and E. Grisan (2025) Cisca and cytodark0: a cell instance segmentation and classification method for histo(patho)logical image analyses and a new, open, nissl-stained dataset for brain cytoarchitecture studies. Computers in Biology and Medicine 197, p. 111018. Cited by: §2.1, §4.1. [57] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Advances in Neural Information Processing Systems, Cited by: §2.2, §3.3.3. [58] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio (2018) Graph attention networks. In International conference on learning representations, Cited by: §1, §2.2, §4.5. [59] Z. Wang, C. Li, H. Xu, X. Zhu, and H. Li (2025) Mamba yolo: a simple baseline for object detection with state space model. In Proceedings of the AAAI Conference on Artificial Intelligence, p. 8205–8213. Cited by: §4.4.1. [60] Z. Yang, C. Kuang, and D. Fu (2026) DeNuC: decoupling nuclei detection and classification in histopathology. In arXiv:2603.04240, Cited by: §1. [61] S. Yoon, A. Chandra, and G. Vahedi (2022) Stripenn detects architectural stripes from chromatin conformation data using computer vision. Nature Communications 13, p. 1602. Cited by: §1. [62] H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. M. Ni, and H. Shum (2023) DINO: detr with improved denoising anchor boxes for end-to-end object detection. In The eleventh international conference on learning representations, Cited by: §1, §2.1, §3.1, §4.4.1, Table 4. [63] L. Zhang, Y. Zhang, L. Cai, X. Guan, K. Zhang, and Y. Zhang (2025) Relation-aware graph attention network for nuclei classification. In 2025 IEEE International Conference on Multimedia and Expo (ICME), Cited by: §1. [64] S. Zhang, X. Wang, J. Wang, J. Pang, C. Lyu, W. Zhang, P. Luo, and K. Chen (2023) Dense distinct query for end-to-end object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 7329–7338. Cited by: §1, §2.1, §3.1, §3.1, §4.4.1, Table 4. [65] B. Zhao, X. Chen, Z. Li, Z. Yu, S. Yao, L. Yan, Y. Wang, Z. Liu, C. Liang, and C. Han (2020) Triple u-net: hematoxylin-aware nuclei segmentation with progressive dense feature aggregation. Medical Image Analysis 65, p. 101786. Cited by: §4.4.2. [66] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai (2021) Deformable detr: deformable transformers for end-to-end object detection. In International Conference on Learning Representations, Cited by: §2.1, §3.1, §3.1, §4.4.1. [67] Y. Zhu, Q. Yang, and L. Xu (2024) Active learning enabled low-cost cell image segmentation using bounding box annotation. In arXiv:2405.01701, Cited by: §1, §2.1. [68] D. Zink, A. H. Fischer, and J. A. Nickerson (2004) Nuclear structure in cancer cells. Nature Reviews Cancer 4, p. 677–687. Cited by: §1. [69] Z. Zong, G. Song, and Y. Liu (2023) Detrs with collaborative hybrid assignments training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 6748–6758. Cited by: §2.1, §4.4.1, Table 4.