Paper deep dive
Graph Representation Learning of Longitudinal Medical Imaging Trajectories for Treatment Response Prediction
Johannes Kiechle, Richard Osuala, Daniel M. Lang, Stefan M. Fischer, Ivana Janíčková, Karim Lekadir, Julia A. Schnabel, Jan C. Peeken
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/7/2026, 3:58:49 PM
Summary
This paper introduces GNN-pCR, a novel 3D spatio-temporal graph neural network framework designed to predict pathological complete response (pCR) following neoadjuvant chemotherapy (NACT) for breast cancer. By modeling longitudinal DCE-MRI trajectories as directed acyclic graphs and employing GraphSAGE for message passing, the method captures dynamic tumor evolution. The framework is guided by a novel Asymmetric Treatment Response Loss that enforces population-level alignment, patient-level representation decorrelation, and temporal consistency. Evaluated on the ISPY-2 dataset with 585 patients, GNN-pCR significantly outperforms vision and self-supervised learning baselines across balanced accuracy, AUC, and MCC, establishing a new benchmark for early treatment response prediction.
Entities (10)
Relation Signals (10)
GNN-pCR → evaluatedon → ISPY-2 dataset
confidence 95% · Experiments across a cohort of 585 patients from the public ISPY-2 dataset demonstrate that our method substantially outperforms both vision and self-supervised learning baselines
GNN-pCR → predicts → Pathological Complete Response (pCR)
confidence 95% · we propose an imaging-based 3D spatio-temporal framework for treatment response prediction that integrates a state-of-the-art graph neural network
Breast Cancer → treatedwith → Neoadjuvant Chemotherapy (NACT)
confidence 95% · While commonly treated with neoadjuvant chemotherapy (NACT), effective treatment decision-making remains challenging
GNN-pCR → achievesmetric → Balanced Accuracy
confidence 90% · our GNN-pCR consistently achieves the best performance across all evaluation criteria... bACC from 0.6520 to 0.6844
GNN-pCR → employs → Asymmetric Treatment Response Loss
confidence 90% · The overall Asymmetric Treatment Response Loss applies the introduced objectives asymmetrically to responders and non-responders
Asymmetric Treatment Response Loss → includes → Patient-level Representation Decorrelation
confidence 90% · We propose a patient-level representation decorrelation objective, where the key idea is that embeddings from different timepoints of the same patient should not collapse to redundant representations
Asymmetric Treatment Response Loss → includes → Temporal Consistency Loss
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In patients with breast cancer, pathological complete response (pCR) has been established as a clinically meaningful surrogate marker for long-term outcomes. While commonly treated with neoadjuvant chemotherapy (NACT), effective treatment decision-making remains challenging, as therapeutic response can vary substantially across patients, calling for predictive models capable of accurately estimating individualized treatment response. To address this, we propose an imaging-based 3D spatio-temporal framework for treatment response prediction that integrates a state-of-the-art graph neural network with relational modeling of temporal interactions across timepoints alongside three novel complementary self-supervised treatment trajectory representation learning objectives. Experiments across a cohort of 585 patients from the public ISPY-2 dataset demonstrate that our method substantially outperforms both vision and self-supervised learning baselines across several classification metrics. Alongside establishing a breast cancer pCR prediction benchmark, we include a principled ablation of our method and further introduce and empirically assess the impact of the available number of DCE-MRI timepoints per patient trajectory and the inclusion of inter-scan time-differences. Overall, our study substantiates the utility of clinically meaningful longitudinal medical imagaging modeling for predicting NACT-induced pCR. We will publicly share our code repository and a user-friendly PyPI library for dataset curation upon publication, effectively promoting reproducible open-source research.
Tags
Links
- Source: https://arxiv.org/abs/2607.04912v1
- Canonical: https://arxiv.org/abs/2607.04912v1
Trouble viewing inline? Open PDF directly →
Full Text
33,883 characters extracted from source content.
Expand or collapse full text
11institutetext: 1 Technical University of Munich, 2 TUM University Hospital Rechts der Isar, 3 Helmholtz Munich, 4 Universitat de Barcelona, 5 Medical University of Vienna, 6 King’s College London, 7 Munich Center for Machine Learning Graph Representation Learning of Longitudinal Medical Imaging Trajectories for Treatment Response Prediction Johannes Kiechle1,2,3,7 Richard Osuala2,4 Daniel M. Lang3 Stefan M. Fischer1,2,3,7 Ivana Janíčková5 Karim Lekadir4 Julia A. Schnabel1,3,6,7,† and Jan C. Peeken2,† Abstract In patients with breast cancer, pathological complete response (pCR) has been established as a clinically meaningful surrogate marker for long-term outcomes. While commonly treated with neoadjuvant chemotherapy (NACT), effective treatment decision-making remains challenging, as therapeutic response can vary substantially across patients, calling for predictive models capable of accurately estimating individualized treatment response. To address this, we propose an imaging-based 3D spatio-temporal framework for treatment response prediction that integrates a state-of-the-art graph neural network with relational modeling of temporal interactions across timepoints alongside three novel complementary self-supervised treatment trajectory representation learning objectives. Experiments across a cohort of 585 patients from the public ISPY-2 dataset demonstrate that our method substantially outperforms both vision and self-supervised learning baselines across several classification metrics. Alongside establishing a breast cancer pCR prediction benchmark, we include a principled ablation of our method and further introduce and empirically assess the impact of the available number of DCE-MRI timepoints per patient trajectory and the inclusion of inter-scan time-differences. Overall, our study substantiates the utility of clinically meaningful longitudinal medical imagaging modeling for predicting NACT-induced pCR. We will publicly share our code repository and a user-friendly PyPI library for dataset curation upon publication, effectively promoting reproducible open-source research. 44footnotetext: Shared senior authorship. Figure 1: Method overview. Longitudinal image-derived latent features are represented as a directed acyclic graph and aggregated through a graph neural network (GNN) projection head to obtain a compact patient-level embedding. Representation learning is guided by an asymmetric response-aware objective that combines (a) population-level alignment, (b) patient-level representation decorrelation, and (c) temporal consistency losses. The overall objective encourages discriminative separation between responders and non-responders while preserving the temporal structure of disease evolution. 1 Introduction With an estimated 2.3 million women diagnosed with breast cancer annually [24], optimizing treatment remains a clinical priority where neoadjuvant chemotherapy (NACT) crucially enables early response assessment to guide therapy adaptation and surgical planning [14]. Pathologic complete response (pCR), i.e. the absence of residual invasive disease in the breast and axillary lymph nodes at surgery following NACT [14], is commonly used to estimate NACT efficacy while being associated with improved long-term outcomes and event-free survival [22]. Reliable early prediction of pCR holds the promise of mitigating treatment-related toxicity and crucially supports informed clinical decision-making, including therapeutic escalation or de-escalation strategies. Moreover, early identification of non-responders may enable timely treatment modification and optimization of surgical intervention. Dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) plays an essential role in screening and assessment of response to NACT [7]. Accordingly, DCE-MRI–based methods for pCR prediction have attracted substantial attention. Many existing studies, however, are limited by the use of single-modality pre-treatment data, absence of rigorous baseline examinations, or reliance on single train-test split, thereby constraining confidence in generalizability of reported performances [7, 14, 19, 12, 23, 9, 16, 13, 8]. While imaging data from a single time point (e.g. pre-treatment) insufficiently captures subtle therapy-induced alterations [9], the integration of longitudinal DCE-MRI acquired over the course of NACT [15, 18] allow pCR prediction models to capture dynamic tumor evolution, signaling individual treatment trajectories. Nonetheless, only a limited number of studies have utilized longitudinal DCE-MRI data [8], and existing approaches remain constrained in their ability to learn complex temporal representations. Several methods rely on hand-crafted features [23] or simple concatenation of multiple time points [13], which may inadequately capture longitudinal tumor dynamics. For example, [3] incorporated multiple DCE-MRI time points but employed a fixed CNN pretrained on natural images for feature extraction, without adapting representations to the temporal treatment setting. Similarly, [9] applied an LSTM to model inter-timepoint dependencies, however, training was restricted to a supervised binary pCR classification objective, without dedicated temporal representation learning. Likewise, Zang et al. [25] proposed a temporal fusion module integrating attention-based and sequence modeling mechanisms to aggregate longitudinal MRI features, yet optimization was again limited to supervised pCR classification. Recently, a purely imaging-based 2D longitudinal representation learning approach [8] demonstrated promising results but evaluated performance on a single dataset split, likely overestimating generalization performance. While pointing towards the needs of (i) deeper multi-split validation and (i) CNN-based feature extractor and self-supervised baseline comparisons [8, 9, 20], these previous studies signaled potential capabilities of temporal representation learning of disease trajectories. Although these studies suggest potential, its downstream impact, specifically for breast cancer pCR prediction, including varying numbers of DCE-MRI time points remains insufficiently explored [8, 10]. We aim to bridge this gap by introducing a time-aware graph neural network with novel temporal representation learning objectives, alongside a principled pCR prediction framework and benchmark evaluation. Our key contributions are as follows: 1. We propose a novel time-aware graph neural network that explicitly encodes structured treatment response trajectories within its graph topology. 2. We introduce three novel representation learning objectives that enforce trajectory and response-consistent latent representations. 3. We establish a comprehensive imaging-based benchmark for breast cancer pCR prediction that evaluates different methods, available imaging timepoints, and the inclusion of inter-scan temporal differences. 2 Methods and Materials 2.1 Network Architecture Visual Feature Extraction Given a 3D input volume, representing a patient i at timepoint t, the longitudinal MRI set is described as xitt=1T\x^t_i\_t=1^T, where xi∈ℝC×D×H×Wx_i ^C× D× H× W. Each input volume is independently processed by a ResNet [6] backbone to extract corresponding feature maps, which are subsequently passed through a global average pooling layer, to obtain a compact embedding vector representing an 3D input volume of patient i at timepoint t, denoted as zit∈ℝDz_i^t ^D. Graph Topology Construction The set of image embeddings zitt=1T\z^t_i\_t=1^T representing a patient i across T timepoints is subsequently translated into a directed graph, where each node corresponds to one embedding zit∈ℝDz_i^t ^D and edges encode pairwise relationships. Therein, nodes are indexed according to their temporal ordering. For every pair of nodes (u,v)(u,v) with u<vu<v, we add a directed edge from node u to node v, with no reverse edges, resulting in a set of edges ℰ=(u,v)| 0≤u<v<NE=\(u,v)\;|\;0≤ u<v<N\. This yields a directed acyclic graph, which ensures that each node can propagate information to all subsequent nodes while preserving temporal dependencies. The choice of this topology ensures pairwise node connectivity under a strict ordering constraint, enabling relational modeling while maintaining a well-defined directional flow of temporal information. Graph Feature Representation Learning Let =(,ℰ)G=(V,E) denote the aforementioned directed acyclic graph, where V refers the set of nodes which are connected by a set of edges ℰ⊆×E ×V. Given a feature vector zit∈ℝDz_i^t ^D attached to the corresponding node, following their temporal ordering, we perform graph feature representation learning over the nodes using GraphSAGE [5], which follows the scheme of message passing. Specifically, nodes are updated by neighborhood feature aggregation, following the underlying graph structure [4]. 2.2 Asymmetric Treatment Response Loss Population-level Alignment We propose a population-level alignment objective that structures the embedding space according to the response status of a patient (Figure 1, bottom left). Let zi∈ℝDz_i ^D denote the embedding of a patient i, our goal is twofold: first, encourage similarity among responder patients (i.e. attract embeddings) and second, discourage similarity between responders and non-responders (i.e. repel embeddings), based on Cosine similarity, outlined as ℒalign=1−1|ℛ|∑i∈ℛsim(zi,zπ(i))⏟ 1 attract+1||∑(i,j)∈(,ℛ)sim(zi,zj).⏟ 2 repelL_align= 1- 1|R| _i sim(z_i,z_π(i))_ to9.22pt to9.22pt -4.60942pt to0.0pt @beginscope @invoke pgfstrokecolorrgb0,0,0 @color@rgb@stroke000 @invoke @color@rgb@fill000 @invoke @setlinewidth @invoke to0.0pt @beginscope @invoke @beginscope @invoke @moveto4.40942pt0.0pt @curveto4.40942pt2.43529pt2.43529pt4.40942pt0.0pt4.40942pt @curveto-2.43529pt4.40942pt-4.40942pt2.43529pt-4.40942pt0.0pt @curveto-4.40942pt-2.43529pt-2.43529pt-4.40942pt0.0pt-4.40942pt @curveto2.43529pt-4.40942pt4.40942pt-2.43529pt4.40942pt0.0pt @closepath @moveto0.0pt0.0pt @stroke @invoke @beginscope @invoke @transformcm1.00.00.01.0-1.99306pt-2.25555pt @invoke pgfstrokecolorrgb0,0,0 @color@rgb@stroke000 @invoke @color@rgb@fill000 @invoke 1 @invoke @endscope @invoke @endscope @invoke @endscope @discardpath @invoke @endscope attract+ 1|N| _(i,j)∈(N,R)sim(z_i,z_j)._ to9.22pt to9.22pt -4.60942pt to0.0pt @beginscope @invoke pgfstrokecolorrgb0,0,0 @color@rgb@stroke000 @invoke @color@rgb@fill000 @invoke @setlinewidth @invoke to0.0pt @beginscope @invoke @beginscope @invoke @moveto4.40942pt0.0pt @curveto4.40942pt2.43529pt2.43529pt4.40942pt0.0pt4.40942pt @curveto-2.43529pt4.40942pt-4.40942pt2.43529pt-4.40942pt0.0pt @curveto-4.40942pt-2.43529pt-2.43529pt-4.40942pt0.0pt-4.40942pt @curveto2.43529pt-4.40942pt4.40942pt-2.43529pt4.40942pt0.0pt @closepath @moveto0.0pt0.0pt @stroke @invoke @beginscope @invoke @transformcm1.00.00.01.0-1.99306pt-2.25555pt @invoke pgfstrokecolorrgb0,0,0 @color@rgb@stroke000 @invoke @color@rgb@fill000 @invoke 2 @invoke @endscope @invoke @endscope @invoke @endscope @discardpath @invoke @endscope repel (1) 1 attract To promote a shared latent space among responder patients ℛR, we randomly permute the embeddings within the responder group and align each responder with another responder. Formally, let π be a random permutation over indices in ℛR, the responder attract term aims at increasing the pairwise similarity within the responder population, encouraging embeddings to occupy a compact region in latent space. Moreover, as the permutation π changes across iterations, the loss approximates a stochastic estimate of responder coherence rather than enforcing fixed pairwise matches within this subgroup. 2 repel To promote discriminative separation between responders ℛR and non-responders N, we penalize similarity across these groups. Minimizing the repel term, pushes embeddings of the two subpopulations apart, thereby increasing inter-class separation. Furthermore, we intentionally omit the attract term for non-responders, as their heterogeneous clinical outcomes such as stable disease, progressive disease, or partial response are associated with distinct underlying imaging patterns, making a shared latent representation ill-posed. Patient-level Representation Decorrelation We propose a patient-level representation decorrelation objective, where the key idea is that embeddings from different timepoints of the same patient should not collapse to redundant representations, but rather encode complementary aspects of disease evolution. To this end, we penalize similarity between adjacent timepoint embeddings of responder patients ℛR using Cosine similarity, formally defined as ℒdecorrelate=1|ℛ|∑i∈ℛ1T−1∑t=0T−2sim(zi(t),zi(t+1)).L_decorrelate= 1|R| _i 1T-1 _t=0^T-2sim\! (z_i^(t),z_i^(t+1) ). (2) Minimizing the objective above encourages embeddings from different timepoints of the same patient to become decorrelated. As a consequence, each temporal segment zi(t)z_i^(t) of a patient trajectory is promoted to capture distinct, non-redundant factors of variation relevant for treatment response. Patient-level Temporal Relationships We propose a patient-level temporal relationship loss, with the key idea of modeling how latent embeddings zitt=0T−1\z^t_i\_t=0^T-1 of patient i evolve over time. While the decorrelation term enforces complementary information across timepoints, we now constrain the latent trajectory to follow a consistent additive progression z~i z_i, formally defined as ℒtemporal=1|ℛ|∑i∈ℛ[1−sim(z~i,zi(T))]wherez~i=∑t=0T−2(zi(t+1)−zi(t)).L_temporal= 1|R| _i [1-sim\! ( z_i,z_i^(T) ) ]\;\;\;where z_i= _t=0^T-2 (z_i^(t+1)-z_i^(t) ). (3) Minimizing the objective above encourages temporal consistency. The representation at the final timepoint T must be predictable from the sequence of intermediate latent transitions. Intuitively, the embeddings are encouraged to follow a coherent temporal trajectory rather than independent snapshots, thereby regularizing the model to encode meaningful disease progression dynamics. Loss Function The overall Asymmetric Treatment Response Loss applies the introduced objectives asymmetrically to responders ℛR and non-responders N: ℒtotal(i)=i∈ℛ(ℒalign+ℒdecorrelate+ℒtemporal)+i∈ℒrepelL_total(i)=1_\i \ (L_align+L_decorrelate+L_temporal )+1_\i \L_repel 3 Experiments and Results Data and Preprocessing In this work we use the publicly available ISPY-2 dataset [15, 18] and select 585 patients (204 responders) with complete DCE-MRI acquisitions at four neoadjuvant chemotherapy (NACT) timepoints, comprising one pre-treatment scan and three consecutive examinations acquired during therapy prior to surgery. Data preprocessing follows Jing et al. [9], resulting in a longitudinal sequence of four images per patient, each represented as a tensor of shape (3,64,64,64)(3,64,64,64). The three channels encode early contrast enhancement (phase 1 minus phase 0), late contrast enhancement (phase 4 minus phase 0), and functional tumor volume (FTV), which captures metabolically active tumor regions defined via pharmacokinetic thresholds on DCE-MRI [17]. To ensure reproducibility and enable fair comparison, we release a library at the Python Package Index (PyPI) implementing the full data pipeline upon publication. Implementation Details All models are trained and evaluated using stratified 5-fold cross-validation (3/1/1 train/val/test split per fold). Optimization is performed with SGD (learning rate 10−210^-2, and batch size 16 with gradient accumulation over 4 batches) for 100 epochs using a cosine annealing scheduler. The model with the highest validation AUC is selected for testing. To ensure fair comparison, all methods share identical data splits, inputs, and the same ResNet18 feature encoder implementation [1], such that performance differences reflect only the respective spatio-temporal learning strategies. DINOv3 [21] is included as a strong training-free foundational model baseline, using the 2D axial volume slice containing the largest metabolically active tumor region (based on the FTV channel). All vision baselines employ the same three-layer MLP classification head, with the CNN+LSTM approach additionally incorporating an LSTM layer for sequential modeling [9]. Self-supervised methods are evaluated using a support vector classifier. Experiments are conducted on an NVIDIA RTX A6000 Ada GPU. Further details can be found in our publicly available code repository. Table 1: Quantitative comparison of GNN-pCR with vision and self-supervised learning (SSL) baselines, with an ablation study of its key components. Performance is reported as the mean value over 5-folds, using balanced accuracy (bACC), Area Under the Receiver Operating Characteristic Curve (AUC) and Matthews correlation coefficient (MCC). Best in bold, second best underlined and ↑ means higher is better. Wilcoxon signed-rank test: (∗)refers to p-values ≤ 0.05, (†)refers to p-values ≤ 0.0625. Method Backbone bACC ↑ AUC ↑ MCC ↑ Vision baselines CNN ResNet18 0.6466(†) ± 0.03 0.6951(†) ± 0.03 0.2864(†) ± 0.06 CNN + LSTM[9] ResNet18 0.6107(†) ± 0.05 0.6825 ± 0.06 0.2278(†) ± 0.10 DINOv3[21] ViT-L/16 0.5628(∗) ± 0.04 0.6020(∗) ± 0.03 0.1316(∗) ± 0.09 SSL baselines 3D-LART3D-L_ART[8] ResNet18 0.6520(†) ± 0.03 0.6857 ± 0.02 0.2922(∗) ± 0.06 SSL-TOPC[11] ResNet18 0.6224(∗) ± 0.02 0.6775 ± 0.04 0.2434(∗) ± 0.03 GNN-pCR (ours) ResNet18 0.6844 ± 0.01 0.7203 ± 0.02 0.3561 ± 0.02 Ablation study w/o alignment loss ResNet18 0.6162(∗) ± 0.03 0.6936 ± 0.04 0.2490(∗) ± 0.06 w/o temporal loss ResNet18 0.6589(∗) ± 0.03 0.7025 ± 0.06 0.3047(∗) ± 0.05 w/o decorrelation loss ResNet18 0.6350(∗) ± 0.01 0.6594(∗) ± 0.03 0.2684(∗) ± 0.03 w/o GNN ResNet18 0.6610 ± 0.02 0.6895 ± 0.04 0.3077(†) ± 0.04 3.1 Empirical Results and Experimental Analysis Table 1 presents a quantitative comparison between our proposed GNN-pCR framework and several vision and self-supervised learning (SSL) baselines for the task of binary breast cancer pCR prediction. We evaluated [8] on the reference 2D data using 5-fold cross-validation and, in line with the authors [2], observed substantial performance variability, achieving an AUC of 0.56100.5610. To ensure fair evauluation, we extended the method to 3D using our unified encoder, denoted as 3D-LART3D-L_ART. Overall, our GNN-pCR consistently achieves the best performance across all evaluation metrics. In particular, compared to the strongest respective baseline model, our approach improves bACC from 0.65200.6520 [8] to 0.68440.6844, AUC from 0.69510.6951 (best CNN baseline) to 0.72030.7203, and Matthews correlation coefficient (MCC) from 0.29220.2922 [8] to 0.35610.3561. These results suggest that structured modeling of longitudinal relationships as proposed in our GNN-pCR approach, successfully translates into consistent improvements across all evaluation criteria and models. The performance improvement is even larger when compared to the SSL approach of [11], the sequence modeling strategy based on CNN+LSTM [9] and the recent transformer vision foundation model DINOv3 [21]. Ablation Study We conduct an ablation study by systematically removing individual loss terms and the GNN module from the full model while keeping all other components untouched. Results can be found in Table 1. Removing the alignment loss leads to a consistent performance drop across all metrics, with a substantial decline in e.g., bACC from 0.68440.6844 to 0.61620.6162. This finding is consistent with the objective of enhancing class separability between responders and non-responders, indicating that the alignment loss constitutes a key factor in shaping discriminative pCR representations. Similarly, removing the temporal loss (0.6844→0.65890.6844→ 0.6589) or decorrelation loss (0.6844→0.63500.6844→ 0.6350) leads to considerable performance declines across all metrics. These findings suggest that both encouraging complementary features (via decorrelation loss) and enforcing temporal consistency (via temporal loss) are essential to capture relevant longitudinal changes in responder patients. Finally, replacing the GNN with a linear prediction head also reduces performance (0.6844→0.66100.6844→ 0.6610), underscoring that relational modeling of temporal interactions across timepoints provides benefits beyond those achievable with dense feature connections alone. Table 2: Quantitative comparison of methods for (i) early response prediction and (i) impact of inter-scan time differences. Performance is reported as the mean value over 5-folds, using balanced accuracy (bACC), for consecutive timepoints t0→t1t_0→ t_1, t0→t2t_0→ t_2 and t0→t3t_0→ t_3. Best in bold, second best underlined and ↑ means higher is better. Method Inter-Scan Time Diff. bACC ↑ t0→t1t_0→ t_1 t0→t2t_0→ t_2 t0→t3t_0→ t_3 CNN ✗ 0.5042 ± 0.03 0.6508 ± 0.03 0.6466 ± 0.03 ✓ 0.5059 ± 0.02 0.6479 ± 0.03 0.6447 ± 0.02 CNN + LSTM[9, 20] ✗ 0.5237 ± 0.05 0.5721 ± 0.03 0.6107 ± 0.05 ✓ 0.5163 ± 0.04 0.5969 ± 0.03 0.6513 ± 0.02 3D-LART3D-L_ART[8] ✗ 0.5586 ± 0.05 0.6631 ± 0.03 0.6520 ± 0.03 ✓ 0.5369 ± 0.06 0.6391 ± 0.05 0.6616 ± 0.02 ✗ 0.5820 ± 0.04 0.6823 ± 0.01 0.6844 ± 0.01 GNN-pCR (ours) ✓ 0.5937 ± 0.06 0.6689 ± 0.04 0.6853 ± 0.03 Early Response Prediction To assess the capability of the models to predict therapy response at early stages, we restrict the available longitudinal information to partial time series and evaluate performance for consecutive time intervals (t0→t1t_0→ t_1, t0→t2t_0→ t_2, and t0→t3t_0→ t_3). Results are summarized in Table 2. Across all temporal settings, our proposed GNN-pCR consistently achieves the strongest performance. Notably, the superiority of the full time-series model (t0→t3t_0→ t_3) translates to earlier prediction scenarios. When only two timepoints are available (t0→t1t_0→ t_1), GNN-pCR attains the highest bACC (0.5937), clearly outperforming CNN-based and recurrent baselines. The margin over the second-best SSL method (3D-LART3D-L_ART) is particularly pronounced in this most challenging early setting, highlighting the benefit of our novel learning objectives combined with explicit modeling of relational structure under limited temporal data. We further analyze the impact of explicitly incorporating intra-patient time differences across scans as an additional feature. The effect is method-dependent. The CNN+LSTM [20] benefits most at the full time-series (t0→t3t_0→ t_3), improving from 0.61070.6107 to 0.65130.6513, although starting from a comparatively low score. In contrast, CNN, 3D-LART3D-L_ART and GNN-pCR exhibit only moderate changes when adding time-gap information, which we attribute to the systematic design of the ISPY-2 study, which shows near homogeneous time difference distribution across patients and their respective scans [15, 18]. 4 Discussion and Conclusion We have introduced a novel self-supervised framework for learning imaging-based patient treatment trajectories by combining a time-aware graph neural network with relational temporal modeling alongside three complementary trajectory-level objectives. The underlying graph structure explicitly encodes dependencies across timepoints, enabling structured representation learning aligned with clinically meaningful disease progression. Our experimental evaluations demonstrate consistent improvements over both supervised vision baselines and existing self-supervised approaches across multiple classification metrics. Ablation studies confirm the contribution of each novel loss component and highlight the added value of relational modeling via the GNN. Notably, early response prediction results suggest that the learned representations capture clinically relevant longitudinal characteristics. Incorporating inter-scan time differences, although holding promise, yielded only moderate gains, likely due to the structured study design of ISPY-2 and the resulting less pronounced variability in acquisition intervals across patients and scans. Even though our evaluations are based on breast cancer pCR prediction, our proposed framework is designed as a general approach for longitudinal SSL. To this end, future work will evaluate the proposed framework across a broader range of longitudinal clinical prediction and monitoring tasks, potentially in combination with complementary clinical biomarkers. References [1] M. J. Cardoso, W. Li, and R. Nic Brown (2022) MONAI: an open-source framework for deep learning in healthcare. arXiv preprint arXiv:2211.02701. Cited by: §3. [2] cirmuw (2026) Temporal representation learning of phenotype trajectories for pCR prediction in breast cancer. Note: https://github.com/cirmuw/temporal-representation-learning/blob/a1c721872b6c2caa4dbe7c352cd878e283e3dc8/README.mdGitHub repository, commit a1c7218, accessed 25 Feb 2026 Cited by: §3.1. [3] M. C. Comes, A. Fanizzi, S. Bove, V. Didonna, S. Diotaiuti, D. La Forgia, A. Latorre, E. Martinelli, A. Mencattini, A. Nardone, et al. (2021) Early prediction of neoadjuvant chemotherapy response by exploiting a transfer learning approach on breast DCE-MRIs. Scientific Reports 11 (1), p. 14123. Cited by: §1. [4] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl (2017) Neural Message Passing for Quantum Chemistry. In International Conference on Machine Learning, p. 1263–1272. Cited by: §2.1. [5] W. Hamilton, Z. Ying, and J. Leskovec (2017) Inductive Representation Learning on Large Graphs. Advances in Neural Information Processing Systems. Cited by: §2.1. [6] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 770–778. Cited by: §2.1. [7] N.M. Hylton and et al. (2016) Locally advanced breast cancer: MR imaging for prediction of response to neoadjuvant chemotherapy-results from ACRIN 6657/ISPY trial. Radiology 279 (1), p. 44–55. External Links: Document Cited by: §1. [8] I. Janíčková, Y. Y. Tan, T. H. Helbich, K. Miloserdov, Z. Bago-Horvath, U. Heber, and G. Langs (2025) Temporal representation learning of phenotype trajectories for pCR prediction in breast cancer. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 606–615. Cited by: §1, §1, §1, §3.1, Table 1, Table 2. [9] B. Jing, K. Wang, E. Schmitz, S. Tang, Y. Li, Y. Zhang, and J. Wang (2024) Prediction of pathological complete response to chemotherapy for breast cancer using deep neural network with uncertainty quantification. Medical Physics 51 (12), p. 9385–9393. Cited by: §1, §1, §1, §3.1, Table 1, Table 2, §3, §3. [10] J. Kächele, D. Bounias, A. Ertl, and K. Maier-Hein (2025) On tackling domain shift in breast MRI using only publicly-available data: reproducible breast cancer segmentation and pCR prediction. In Deep Breast Workshop on AI and Imaging for Diagnostic and Treatment Challenges in Breast Care, p. 310–319. Cited by: §1. [11] E. Kaczmarek, J. Szeto, B. Nichyporuk, and T. Arbel (2025) SSL-AD: spatiotemporal self-supervised learning for generalizability and adaptability across alzheimer’s prediction tasks and datasets. arXiv preprint arXiv:2509.10453. Cited by: §3.1, Table 1. [12] N. Khan, R. Adam, P. Huang, T. Maldjian, and T. Q. Duong (2022) Deep learning prediction of pathologic complete response in breast cancer using MRI and other clinical data: a systematic review. Tomography 8 (6), p. 2784–2795. Cited by: §1. [13] E. Krasniqi, L. Filomeno, T. Arcuri, G. Ferretti, S. Gasparro, A. Fulvi, A. Roselli, L. D’Onofrio, L. Pizzuti, M. Barba, et al. (2025) Multimodal deep learning for predicting neoadjuvant treatment outcomes in breast cancer: a systematic review. Biology Direct 20 (1), p. 72. Cited by: §1, §1. [14] W. Li, D. C. Newitt, J. Gibbs, L. J. Wilmes, E. F. Jones, V. A. Arasu, F. Strand, N. Onishi, A. A. Nguyen, J. Kornak, L. J. Esserman, D. Berry, and N. M. Hylton (2020) Predicting breast cancer response to neoadjuvant treatment using multi-feature MRI: results from the I-SPY 2 trail. NPJ Breast Cancer 6 (1), p. 63. Cited by: §1, §1. [15] W. Li, D. Newitt, J. Gibbs, L. Wilmes, E. Jones, V. Arasu, F. Strand, N. Onishi, A. Nguyen, J. Kornak, et al. (2022) I-SPY 2 breast dynamic contrast enhanced MRI trial (ISPY2)(version 1). The Cancer Imaging Archive (TCIA). Cited by: §1, §3.1, §3. [16] W. Li, S. C. Partridge, D. C. Newitt, J. Steingrimsson, H. S. Marques, P. J. Bolan, M. Hirano, B. A. Bearce, J. Kalpathy-Cramer, M. A. Boss, et al. (2024) Breast multiparametric MRI for prediction of neoadjuvant chemotherapy response in breast cancer: the BMMR2 challenge. Radiology: Imaging Cancer 6 (1), p. e230033. Cited by: §1. [17] D. C. Newitt, S. O. Aliu, N. Witcomb, G. Sela, J. Kornak, L. Esserman, and N. M. Hylton (2014) Real-time measurement of functional tumor volume by MRI to assess treatment response in breast cancer neoadjuvant clinical trials: validation of the aegis SER software platform. Translational Oncology 7 (1), p. 94–100. Cited by: §3. [18] D. C. Newitt, S. C. Partridge, Z. Zhang, J. E. Gibbs, T. L. Chenevert, M. A. Rosen, P. J. Bolan, H. Marques, J. Romanoff, L. Cimino, and N. M. Hylton (2025) ACRIN 6698/I-SPY2 breast DWI dataset (2021). DOI: https://doi. org/10.7937/tcia. k02-6d95. Cited by: §1, §3.1, §3. [19] Y. Peng et al. (2022) Pretreatment DCE-MRI-based deep learning outperforms radiomics analysis in predicting pathologic complete response to neoadjuvant chemotherapy in breast cancer. Frontiers in Oncology 12, p. 846775. Cited by: §1. [20] R. Santeramo, S. Withey, and G. Montana (2018) Longitudinal detection of radiological abnormalities with time-modulated LSTM. In International Workshop on Deep Learning in Medical Image Analysis, p. 326–333. Cited by: §1, §3.1, Table 2. [21] O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, et al. (2025) DINOv3. arXiv preprint arXiv:2508.10104. Cited by: §3.1, Table 1, §3. [22] L. M. Spring, G. Fell, A. Arfe, C. Sharma, R. Greenup, K. L. Reynolds, B. L. Smith, B. Alexander, B. Moy, S. J. Isakoff, G. Parmigiani, L. Trippa, and A. Bardia (2020) Pathologic complete response after neoadjuvant chemotherapy and impact on breast cancer recurrence and survival: a comprehensive meta-analysis. Clinical Cancer Research 26 (12), p. 2838–2848. Cited by: §1. [23] A. Syed, R. Adam, T. Ren, J. Lu, T. Maldjian, and T. Q. Duong (2023) Machine learning with textural analysis of longitudinal multiparametric MRI and molecular subtypes accurately predicts pathologic complete response in patients with invasive breast cancer. PLOS One 18 (1), p. e0280320. Cited by: §1, §1. [24] World Health Organization (2026) Breast cancer: global patterns of incidence, mortality, and survival. Note: https://w.who.int/news-room/fact-sheets/detail/breast-cancerAccessed: 2026-02-11 Cited by: §1. [25] S. Zhang, S. Du, C. Sun, B. Li, L. Shao, L. Zhang, K. Wang, Z. Liu, and J. Tian (2024) M2Fusion: multi-time multimodal fusion for prediction of pathological complete response in breast cancer. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 458–468. Cited by: §1.