Paper deep dive
OSF: On Pre-training and Scaling of Sleep Foundation Models
Zitao Shuai, Zongzhe Xu, David Yang, Wei Wang, Yuzhe Yang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 8:10:26 AM
Summary
The paper introduces OSF (Open Sleep Foundation), a family of sleep foundation models designed to address heterogeneity in polysomnography (PSG) data. The authors curate SleepBench, a large-scale benchmark of 166,500 hours of sleep recordings from nine public sources. Through systematic evaluation, they identify that existing models fail to generalize to missing channels, channel-invariant feature learning is essential, and scaling sample size, model capacity, and multi-source data improves performance. OSF achieves state-of-the-art results across nine datasets on sleep staging, event detection, and disease prediction tasks.
Entities (10)
Relation Signals (6)
OSF → achievesstateofthearton → SleepBench
confidence 95% · OSF achieves state-of-the-art performance across nine datasets on diverse sleep and disease prediction tasks.
Existing Sleep FMs → failstogeneralizeto → Missing Channels
confidence 93% · existing FMs fail to generalize to missing channels at inference
OSF → usespretrainingrecipe → Channel-Invariant Feature Learning
confidence 92% · Guided by these insights, we propose Open Sleep Foundation Model (OSF)... trained with an improved pre-training and scaling recipe... explicitly encouraging the model to learn channel-invariant representations.
SleepBench → containsdatafrom → SHHS
confidence 90% · Our benchmark aggregates nine publicly available datasets... SHHS [29]... aggregated from nine publicly available datasets
Channel Masking → improves → downstream performance
confidence 88% · Channel masking consistently improves downstream performance... SimCLR pre-trained with channel masking consistently outperforms
SimCLR → isusedin → OSF
confidence 85% · In our implementation, we use DINO as the base method... We adapt four representative method families: Contrastive Learning: SimCLR
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Polysomnography (PSG) provides the gold standard for sleep assessment but suffers from substantial heterogeneity across recording devices and cohorts. There have been growing efforts to build general-purpose foundation models (FMs) for sleep physiology, but lack an in-depth understanding of the pre-training process and scaling patterns that lead to more generalizable sleep FMs. To fill this gap, we curate a massive corpus of 166,500 hours of sleep recordings from nine public sources and establish SleepBench, a comprehensive, fully open-source benchmark. Leveraging SleepBench, we systematically evaluate four families of self-supervised pre-training objectives and uncover three critical findings: (1) existing FMs fail to generalize to missing channels at inference; (2) channel-invariant feature learning is essential for pre-training; and (3) scaling sample size, model capacity, and multi-source data mixture consistently improves downstream this http URL an enhanced pre-training and scaling recipe, we introduce OSF, a family of sleep FMs that achieves state-of-the-art performance across nine datasets on diverse sleep and disease prediction tasks. Further analysis of OSF also reveals intriguing properties in sample efficiency, hierarchical aggregation, and cross-dataset scaling.
Tags
Links
- Source: https://arxiv.org/abs/2603.00190v2
- Canonical: https://arxiv.org/abs/2603.00190v2
Trouble viewing inline? Open PDF directly →
Full Text
88,446 characters extracted from source content.
Expand or collapse full text
2026-7-7 OSF: On Pre-training and Scaling of Sleep Foundation Models Zitao Shuai 1 , Zongzhe Xu 1 , David Yang 2 , Wei Wang 1 , Yuzhe Yang 1† 1 University of California, Los Angeles, 2 Emory University Polysomnography (PSG) provides the gold standard for sleep assessment but suffers from substantial heterogeneity across recording devices and cohorts. There have been growing efforts to build general- purpose foundation models (FMs) for sleep physiology, but lack an in-depth understanding of the pre-training process and scaling patterns that lead to more generalizable sleep FMs. To fill this gap, we curate a massive corpus of 166,500 hours of sleep recordings from nine public sources and establishSleepBench, a comprehensive, fully open-source benchmark. LeveragingSleepBench, we systematically evaluate four families of self-supervised pre-training objectives and uncover three critical findings: (1) existing FMs fail to generalize to missing channels at inference; (2) channel- invariant feature learning is essential for pre-training; and (3) scaling sample size, model capacity, and multi-source data mixture consistently improves downstream performance. With an enhanced pre-training and scaling recipe, we introduce OSF, a family of sleep FMs that achieves state-of-the-art performance across nine datasets on diverse sleep and disease prediction tasks. Further analysis of OSF also reveals intriguing properties in sample efficiency, hierarchical aggregation, and cross-dataset scaling. Codes are available at: https://github.com/yang-ai-lab/OSF-Open-Sleep-FM. Code: https://github.com/yang-ai-lab/OSF-Open-Sleep-FM Website: https://yang-ai-lab.github.io/osf 1. Introduction Disease Prediction Hypertension Diabetes Coronary Disease Central Apnea Oxy. Desaturation Hypopnea Arousal Linear Probing Few-Shot Sleep Staging Arousal Sleep Staging 77.7 94.5 83.9 97.3 92.8 81.5 97.3 68.1 62.3 66.3 85.9 72.3 96.4 90.3 75.5 80.9 96.7 66.5 62.1 62.5 Figure 1|Performance comparison across downstream tasks. OSF consistently achieves state-of-the-art on downstream tasks. Sleep is a fundamental physiological process for hu- man health [36,39,41]. Assessing sleep quality [28] and detecting sleep disorder events [34] often rely on polysomnography (PSG). PSG typically includes multiple complementary signals that capture brain activity, respiratory effort, muscle movement, and car- diac activity. In practice, these recordings differ across patient cohorts and devices [16,19], and the available channel set is often inconsistent due to differences in acquisition protocols and occasional channel dropouts. For example, home studies often lack brain signals [3,46], and sensors can become dislodged during the night. Together, these challenges limit the transfer- ability of sleep analysis models [45]. Recent foundation models (FMs) [36] have shown rapid progress in modeling sleep recordings. They learn generalizable representations via self-supervised † Correspondence to: yuzhey@ucla.edu. arXiv:2603.00190v2 [cs.LG] 5 Jul 2026 OSF : On Pre-training and Scaling of Sleep Foundation Models learning (SSL) on large-scale 30-second PSG epochs. These representations support efficient adap- tation to diverse epoch-level tasks (e.g., sleep staging, arousal detection) and can be aggregated over time to predict patient-level outcomes. Prior work [36] has primarily focused on contrastive approaches, where sleep signals are grouped into multiple modalities and their embeddings are aligned in a shared latent space. While contrastive learning has achieved strong results, the broader design space for pre-training sleep FMs remains largely under-explored. In this work, we move beyond reporting gains from a single pre-training recipe and ask what actually drives transfer in sleep FMs. Existing sleep FMs differ in their pre-training objectives, but it is unclear which of these choices reliably improve downstream performance across cohorts, devices, and channel sets. A central question arises: Which pre-training and scaling design choices truly improve the generalization of sleep FMs, especially under cohort shift and missing-channel inference? To answer this question with controlled evaluation, we conduct a systematic study that quantifies how different pre-training design choices affect downstream performance. We also constructSleepBench, the largest fully open, multi-source sleep benchmark, aggregated from nine publicly available datasets with diverse demographics, comprising 166,500 hours of recordings from over 21,000 sleep studies, and spanning a broad set of PSG channels. LeveragingSleepBench, we identify three main findings. First, existing sleep FMs are not reliable under inference-time input incompleteness, showing large performance drops under realistic missing- channel settings. Second, through a controlled, step-by-step study of pre-training design choices, we find that learning channel-invariant features is critical for invariance-driven methods to learn stronger representations. Third, we observe consistent scaling trends in sleep: increasing training data and model capacity, together with multi-source data mixing, improves downstream performance across tasks. Guided by these insights, we propose Open Sleep Foundation Model (OSF), a family of sleep foundation models that achieves state-of-the-art across diverse sleep-related tasks and datasets (Fig. 1). Our contribution. In summary, we present a systematic study of how to build generalizable sleep FMs. We curate a large corpus of 166,500 hours of sleep recordings from nine public sources and establish SleepBench, a comprehensive, fully open-source benchmark. LeveragingSleepBench, we evaluate four families of self-supervised pre-training objectives and identify three key findings:❶existing sleep FMs fail to generalize to missing channels at inference;❷channel-invariant feature learning is essential for strong pre-training; and❸scaling sample size, model capacity, and multi-source data mixture consistently improves downstream performance. Guided by these results, we introduce OSF, a family of sleep FMs trained with an improved pre-training and scaling recipe, achieving state-of-the- art performance across nine datasets on diverse sleep and disease prediction tasks. Further analysis of OSF reveals strong sample efficiency, effective hierarchical aggregation, and consistent scaling across datasets. 2. Fully Open Sleep Benchmarking at Scale Our benchmark aggregates nine publicly available datasets hosted by the National Sleep Research Resource (NSRR) [46]: SHHS [29], NCHSDB [18], CFS [32], CCSHS [33], WSC [42], MROS [4], MESA [7], CHAT [22], and SOF [35]. In total, it contains 166,500 hours of PSG recordings spanning more than 21,000 nights. We further partition the data into an in-domain cohort for pre-training and an external cohort for out-of-domain (OOD) evaluation. Each cohort is split into training, validation, and test sets. All samples in the external cohort are held out from pre-training and used only for downstream evaluation. For the in-domain cohort, we use the training split for pre-training and also 2 OSF : On Pre-training and Scaling of Sleep Foundation Models for downstream fine-tuning in the in-domain setting. Detailed statistics across cohorts, datasets, and splits are provided in Appendix B.1. 2.1% 1.5% 2.4% Figure 2|Distribution of our established SleepBench. We focus on a shared subset of channels that is available across most cohorts. Specifically, we uti- lize a standardized 12-channel montage grouped by physiological modality:❶Brain (EEG/EOG):C3-A2, C4-A1,E1-A2,E2-A1,❷Respiration:Abdominal, Thorax,Nasal Pressure,Snore,❸Cardiac:ECG, and❹Somatic:EMG-Chin,EMG-LLeg,EMG-RLeg. De- tailed settings and preprocessing steps are provided in Appendix B.3. We process each night’s recording with a standard- ized pipeline to reduce cross-dataset heterogeneity. We first perform manual quality control to trim pro- longed wake periods at the beginning and end of the night, removing extreme artifacts from sensor setup and removal. Next, we apply per-night z-score normalization to each channel. For respiratory channels, we additionally apply area-dependent z-score normalization to reduce amplitude drift and improve cross-night consistency. We follow prior work and segment each night into non-overlapping 30-second epochs, yielding roughly 20 million epochs for self-supervised pre-training. For segmented samples, we resample all channels to 64 Hz and zero-pad missing channels. Epoch-level statistics are provided in Table 17. To the best of our knowledge, this corpus constitutes the largest fully public benchmark for pre-training and evaluating sleep foundation models across diverse channel types. We compare our SleepBenchother published benchmarks in Appendix B.1. With large-scale sleep data spanning diverse cohorts,SleepBenchenables controlled benchmarking of different pre-training methods on sleep data, and supports systematic studies of how performance scales with pre-training sample size and how multi-source data blending affects generalization. We acknowledge ongoing community efforts to harmonize public sleep datasets. Rather than treating SleepBenchas a standalone contribution, we view it as a methodological infrastructure that enables controlled multi-source pre-training and evaluation under a unified setting. We will open-source the codebase and processing pipeline, and maintainSleepBenchas a living resource that can be expanded with additional PSG datasets and community contributions. 3. On Pre-training Sleep FM In this section, we study practical design choices for self-supervised pre-training of sleep foundation models. Following [36], we conduct pre-training and downstream evaluation on 30-second epochs segmented from full-night recordings. 3.1. Generalization Breaks Under Missing-Channel Inference We first assess whether existing sleep FMs can cope with real-world deployment challenges. Real- world sleep recordings exhibit heterogeneous channel availability across cohorts, devices, and study protocols, such that channels present in the pre-training corpus may be absent at test time. For instance, home-based recordings [10] sometimes do not collect EEG and EOG channels, whereas EEG-centric sleep architecture studies [11] may not include breathing signals. This input mismatch motivates a key research question: Do current sleep FMs generalize under missing-channel settings? 3 OSF : On Pre-training and Scaling of Sleep Foundation Models ∆= 9.7 ∆= 4.5 ∆= 7.8 ∆= 8.4 Figure 3|Inference with full versus miss- ing channels. Existing sleep FM fails to generalize to missing channel samples. Time-wise masking - motivated by device drop Channel Time Channel Time Channel masking - simulate missing channel Figure 4|Illustration of considered augmentations. We consider time-wise masking and channel masking strategies. To this end, we simulate realistic missing-channel cases by zero-masking a subset of channels at inference time. We group the 12 channels into three groups: (1) brain activity channels, (EEG and EOG), which provide key cues for sleep staging; (2) respiratory channels, including breathing effort and airflow waveforms that characterize respiratory physiology; and (3) all remaining channels (ECG and EMG). We focus on two realistic settings: in-home studies that do not provide brain signals, and micro-event studies where breathing channels are not collected. We evaluate current sleep FMs under full-channel input and the two missing-channel settings described above. As shown in Fig. 3, removing brain activity channels leads to a substantial drop in sleep staging performance. Similarly, when respiratory channels are removed, hypopnea detection performance degrades significantly. These results match clinical intuition: hypopnea is driven by respiration, and sleep staging depends primarily on brain signals. Finding 1: Existing sleep FMs fail to generalize under missing-channel inference, motivating pre-training designs that explicitly handle channel incompleteness. 3.2. Channel-Invariant Pre-training Improves Robustness and Transfer Table 1|Comparisons of different masking strategies. Channel masking consistently improves downstream per- formance, and combining time and channel masking yields the best results across pre-training methods. Model Mask Strategy Sleep StagingHypopnea Time Channel AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ SimCLR ✓✗81.456.858.453.1 ✗✓94.884.373.359.2 ✓96.789.075.660.9 DINO ✓✗93.681.472.759.5 ✗✓96.487.977.161.4 ✓97.390.477.762.2 The failures of existing sleep FMs under missing-channel inference suggest that pre- training should encourage the model to learn representations that are invariant to the available channel subset. To test this hypothesis, we perform a controlled, step-by-step study using Sim- CLR [6]while varying its augmentation strategy. We focus on SimCLR because it learns by aligning two augmented views of the same input, and its objective is largely shaped by the augmentation design. Specifically, following prior work on self- supervised learning for time series [20], we use time-wise block masking as the de- fault augmentation. We compare it against an enhanced strategy that additionally masks a randomly selected subset of input channels. The augmentation strategies are illustrated in Fig. 4. For block masking, given an inputx∈ℝ 퐶×푇 with퐶channels and sequence length푇, we apply masking independently to each channel. For each 4 OSF : On Pre-training and Scaling of Sleep Foundation Models channel, we sample a masking ratio푟 ∈ 0.3, 0.6and a start index푝∼풰0,...,⌊1−푟푇⌋, and set the contiguous segment푝, 푝 ⌊푟푇⌋to zero. For channel masking, we randomly drop 50% of the channels by setting all time steps in the selected channels to zero. As shown in Table 1, SimCLR pre-trained with channel masking consistently outperforms the default time-only masking by a large margin across multiple downstream tasks. To isolate the contribution of channel masking, we further train a SimCLR variant that uses only channel masking, while keeping all other hyperparameters and masking ratios the same. Table 1 confirms that this channel-only variant still significantly outperforms the default setting, suggesting that SimCLR learns more robust features when the two views differ in channel availability. To test whether this effect extends beyond contrastive learning, we conduct an analogous controlled study with DINO [26], a distillation-based method. As shown in Table 1, we observe a similar pattern: although DINO performs strongly with time masking alone, adding channel masking yields a clear improvement. Based on these results, we summarize our second finding as follows: Finding 2: Explicitly encouraging channel-invariant feature learning during pre-training improves robustness and downstream transfer, particularly for contrastive and distillation-based methods. 3.3. Scaling Laws Emerge in Sleep FM Pre-training Figure 5|Comparison of scaling behavior. Cur- rent sleep FM is less scalable. In contrast, OSF better utilizes training data. Recent sleep FMs are pre-trained on millions of 30-second epochs, yet it remains unclear whether current designs continue to improve as we scale up the amount of pre-training data. We therefore ask whether sleep FM performance improves consistently as the pre-training sample size increases. We start by pre-training the prior sleep FM [36] on our pre-training cohort, matching the origi- nal parameter budget. As shown in Fig. 5, sleep staging performance on SHHS improves when in- creasing the pre-training data from 1% to 10% of the corpus, but shows little or no gain when further scaling from 10% to 100%. We observe a similar saturation trend on MROS. These results suggest that the baseline pre-training design may not fully benefit from large-scale sleep data. We then repeat the same data-scaling study using the pre-training recipe identified in Sec. 3.2, while keeping the model size fixed. As shown in Fig. 5, performance improves consistently as we scale up the pre-training sample size, indicating that data scaling can hold for sleep FM pre-training with appropriate SSL designs. We further analyze scaling and multi-source data mixture in more detail. We find that (1) consistent gains can be observed when scaling both sample size and model size, and (2) increasing data diversity via multi-source blending improves generalization relative to training on a single source. Results are presented in Sec. 5. Taken together, these results motivate our final pre-training recipe: we scale model capacity and pre-train on large-scale, multi-source data, while explicitly encouraging the model to learn channel-invariant representations. Finding 3: Baseline sleep FMs can saturate as pre-training data grows; with channel-invariant SSL, performance scales more consistently with larger data size, larger models, and multi-source data mixtures. 5 OSF : On Pre-training and Scaling of Sleep Foundation Models Table 2|Sleep staging and sleep event detection. OSF achieves the best overall performance among all compared methods. We report macro-AUC and macro-AUPRC. Method Sleep StagingArousalHypopneaOx. Desat.Central Apnea AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ Supervised: ViT [9]96.989.988.483.081.366.281.881.697.370.5 Linear Probing: SleepFM [36]96.487.790.384.975.560.680.980.396.770.3 SimCLR [6]81.456.873.667.858.453.166.865.079.453.6 DINO [26]93.681.482.275.772.759.578.678.296.166.8 VQ-VAE [37]95.886.786.280.375.060.879.879.596.768.8 MAE [12]96.5 88.789.784.377.661.981.280.897.371.7 Autoregression [30] 96.087.687.881.972.158.979.078.595.265.7 OSF97.390.492.888.377.762.281.581.097.370.7 Full Fine-tuning: SleepFM [36]97.089.593.789.685.069.082.882.497.771.0 SimCLR [6]95.285.584.077.465.956.280.580.196.869.3 DINO [26]97.090.292.388.084.568.982.682.397.975.2 VQ-VAE [37]97.4 90.893.589.581.865.882.482.198.075.2 MAE [12]97.390.693.389.385.169.383.383.098.176.0 Autoregression [30] 97.190.093.088.984.668.982.882.497.975.6 OSF97.992.094.691.085.069.283.583.198.175.5 4. OSF: Open Sleep Foundation Models 4.1. Experimental Setup Pre-training Setup. To rigorously benchmark self-supervised pre-training methods on sleep data, in addition to SleepFM [36], we adapt four representative method families: ❶ Contrastive Learning: SimCLR [6];❷Reconstruction-based Method: MAE [12] and VQ-VAE [37];❸Autoregressive Modeling: Autoregression [30]; and❹Self-Distillation: DINO [26]. In our implementation, we use DINO as the base method and incorporate our pre-training recipes, yielding OSF. Unless otherwise specified, all methods reported in the main tables use a Transformer [9] encoder with 85M parameters. All models are pre-trained on the same split of the pre-training cohorts inSleepBenchfor up to 30 epochs, with early stopping upon convergence. We use AdamW [21] with linear warmup (10% of total steps) followed by cosine annealing, and we tune the learning rate and batch size for each method. Evaluation Setup. For main experiments, we evaluate pre-trained models across five epoch-level tasks and three patient-level disease prediction tasks. The epoch-level tasks include (1) 4-class sleep staging, and (2) four binary sleep-event detection tasks: Arousal, Hypopnea, Oxygen Desaturation (Ox. Desat.), and Central Apnea. We also evaluate three patient-level disease classifications: Coronary Disease, Diabetes, and Hypertension. We report AUROC and AUPRC due to label imbalance. For epoch-level tasks, we consider three transfer protocols: (1) linear probing, (2) full fine-tuning, and (3) few-shot adaptation. For disease classification, we segment each recording into 30-second epochs, extract an embedding for each epoch using the pre-trained encoder, and then aggregate epoch embeddings for patient-level prediction. We use MROS as the primary OOD evaluation cohort and SHHS as the primary in-domain cohort. Unless otherwise specified, we report results on MROS in the main text. All pre-training and evaluation configurations, as well as implementation details, are provided in Appendix A. 6 OSF : On Pre-training and Scaling of Sleep Foundation Models Table 3|Linear probing sleep staging across diverse cohorts. OSF achieves the best performance across all datasets. Method CCSHSCFSNCHSDBWSCCHATMESASOF AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ In-Domain DatasetsOut-of-Domain Datasets SleepFM 97.894.997.493.495.289.097.789.297.895.194.682.896.289.7 SimCLR84.565.483.764.580.356.584.256.3 87.970.273.846.679.556.4 DINO95.688.995.989.491.278.896.285.396.590.789.471.594.385.0 VQ-VAE97.493.697.192.593.885.297.288.197.693.893.079.095.086.3 MAE97.794.897.493.595.088.397.789.197.994.992.378.495.888.8 AR97.694.497.393.294.787.497.689.3 97.894.590.473.795.788.6 OSF98.696.797.994.995.890.198.190.698.496.295.985.797.091.8 Figure 6|Few-shot adaptation to downstream tasks. OSF achieves the best performance across all tasks at every shot count. 4.2. Main Results OSF achieves state-of-the-art on sleep analysis tasks. We evaluate linear-probing performance for sleep staging across all cohorts inSleepBench. As shown in Table 3, OSF achieves the best performance across all cohorts. We further evaluate models on four sleep event detection tasks. On MROS, Table 2 shows that OSF achieves state-of-the-art performance on both sleep staging and event detection under linear probing and fine-tuning. We observe similar trends on SHHS (Appendix C.1). OSFSleepFM N1/N2 Wake N3 REM N1/N2 Wake N3 REM Figure 7|UMAP visualization of learned repre- sentations. OSF produces more stage-separable clusters, indicating stronger alignment between the embedding geometry and sleep stage structure. OSF adapts better in few-shot settings. To measure adaptation under limited supervision, we evaluate all models in a few-shot setting on the out-of-domain MROS cohort. We consider 퐾 ∈1, 5, 50labeled examples per class, train a linear classifier on the퐾-shot training subset, and evaluate on the same held-out test set. As shown in Fig. 6, OSF consistently outperforms all other methods across three sleep analysis tasks for all shot budgets. Detailed results are provided in Appendix C.1. OSF outperforms physiological-signal-specific pre-training methods. We further compare OSF with five recent domain-specific baselines: TS2Vec [44], ST-MEM [24], LSM-2 [38], Ped- SleepMAE [27], and SleepGPT [14]. We pre- train PedSleepMAE onSleepBench, evaluate SleepGPT from its released checkpoint, and adapt the remaining methods to our benchmark. Table 4 verifies that OSF achieves the strongest performance across all downstream tasks on MROS in this comparison. More results are in the Appendix C.1. 7 OSF : On Pre-training and Scaling of Sleep Foundation Models Table 4|Comparison with domain-specific pre- training methods. OSF outperforms baselines tai- lored for physiological signals. Method Sleep StagingHypopnea AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ TS2Vec [44]86.967.361.954.5 ST-MEM [24]96.187.575.460.0 LSM-2 [38]95.886.774.660.1 PedSleepMAE [27] 84.059.059.453.4 SleepGPT [14]95.485.765.155.7 OSF97.390.477.762.2 OSF learns clinically useful features for dis- ease prediction. We evaluate patient-level dis- ease prediction on MROS using the same em- bedding aggregation for all pre-trained models. As shown in Table 5, representations learned by OSF transfer better to unseen data and consis- tently outperform SleepFM [36]. The embedding space of OSF shows a more structured pattern. We further study what properties of the learned representation space may contribute to the strong performance of OSF. We visualize embeddings from OSF and SleepFM [36] using UMAP [23] on the SHHS pre-training cohort. Embeddings from OSF form clearer clusters aligned with sleep stage labels, while embeddings from SleepFM [36] appear less structured. This suggests that OSF learns more separable representations, which may help downstream adaptation. Overall, OSF provides a practical recipe for building sleep FMs and achieves strong performance, improved sample efficiency, and effective hierarchical aggregation. 5. On Scaling Sleep FM Table 5|Disease prediction results. OSF achieves the best overall performance. Model Coronary Dis.DiabetesHypertension AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ End2End ViT 61.460.155.852.557.757.4 SleepFM66.565.562.154.962.561.4 SimCLR60.460.060.454.858.857.9 DINO58.057.360.154.358.256.1 VQ-VAE65.564.461.854.662.159.7 MAE65.465.657.653.564.463.9 AR66.565.359.554.464.663.2 OSF68.167.462.356.366.366.1 In this section, we study how multi-source data mixtures affect downstream performance, and we examine the scaling behavior of sleep FM pre-training with respect to sample size and model capacity. Does multi-source data mixture help pre- training? Prior work often pre-trains sleep FMs on multi-source corpora. However, cohorts differ in acquisition protocols and population characteristics, and mixing datasets introduces distribution shift. To examine whether multi- source data mixture helps pre-training, we train two variants of OSF: one pre-trained on SHHS only and the other pre-trained on the full multi-cohort corpus. As shown in Fig. 8, multi-source pre-training yields consistent improvements across all tasks on the out-of-domain MROS dataset. Moreover, when we vary the pre-training sample size and train both single-source and multi-source variants under identical settings, multi-source pre-training consistently outperforms single-source pre-training. Additional results are provided in Appendix D.2. Takeaway 1: Multi-source pre-training improves generalization by increasing data diversity. Does scaling model size help pre-training on sleep data? To answer this question, we pre-train OSF with ViT encoders of increasing capacity, ranging from 1M and 5M to 85M parameters, and evaluate them via linear probing on hypopnea detection. As shown in Fig. 9, larger models consistently achieve better performance. We observe a similar trend for sleep staging (Appendix D). These results suggest that, given the richness and diversity of epoch-level sleep signals, increasing model capacity does not 8 OSF : On Pre-training and Scaling of Sleep Foundation Models Figure 8|Single-source vs. multi-source pre- training. Multi-source pre-training yields con- sistently better downstream performance. Scaling by Data Size Scaling by Model Size Figure 9|Scaling behavior. Linear probing results on hypopnea detection show that OSF improves with both model capacity and pre-training sample size. lead to obvious overfitting in our setting and translates into stronger transferable representations. Additional details are provided in Appendix D.1. Table 6|Robustness of pre-training with fewer channels. We pre-train both models using only ECG and respiratory signals and evaluate down- stream performance. OSF makes better use of this limited channel set. Method HypopneaOx. Desat. AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ SleepFM 71.358.679.778.9 OSF77.662.480.880.1 Does scaling sample size help pre-training on sleep data? Beyond the results in Sec. 3, we fur- ther test data scaling for OSF on harder settings and larger model sizes. Specifically, we pre-train OSF with ViT-1M, 5M, and 85M encoders using 1%, 10%, and 100% of the multi-source cor- pus, and evaluate via linear probing on MROS. As shown in Fig. 9, hypopnea detection perfor- mance improves consistently as the pre-training sample size increases. This indicates that pre- training performance scales positively with data size in our setting. Results on additional tasks and cohorts are provided in Appendix D.1. Takeaway 2: Scaling laws emerge in sleep data; jointly scaling model and data size yields the strongest gains. 6. Discussion and Analysis 6.1. On Pre-training and Inference with Incomplete Channels Evaluation under missing-channel inference. We evaluate pre-trained models under realistic missing-channel settings. Motivated by real-world deployment scenes, we consider four common study configurations:❶head-band device only [2];❷sleep disorder studies [10];❸sleep micro- event studies [11]; and❹in-home studies [31] that provide breathing signals only. We report linear probing results on sleep analysis tasks on MROS. As shown in Table 7, OSF consistently outperforms SleepFM across these missing-channel settings. Specifically, (1) OSF makes better use of the available channels. With brain-activity channels only, it achieves stronger sleep staging and arousal detection, suggesting stronger brain-related representations. Similarly, with respiratory channels only, it achieves stronger performance on hypopnea and oxygen desaturation. (2) OSF is more robust when key modalities are missing. When respiratory signals are removed, both methods degrade on hypopnea and oxygen desaturation, but OSF remains consistently better. Conversely, when brain-related channels are unavailable, sleep staging becomes much harder for both models; nevertheless, OSF better uses the remaining channels and yields stronger performance. 9 OSF : On Pre-training and Scaling of Sleep Foundation Models Table 7|Linear probing results under realistic missing-channel settings. OSF is more robust to missing channels. Realistic Settings Ref.Brain Resp. ECG Method Sleep StagingArousalHypopneaOx. Desat. AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ Head-Band Data ✓ ✗ ✗ SleepFM 96.688.790.885.567.556.769.968.9 [2]OSF97.2 (+0.6) 90.0 (+1.3) 92.2 (+1.4) 87.6 (+2.1) 68.8 (+1.3) 57.1 (+0.4) 70.7 (+0.8) 69.9 (+1.0) Sleep Disorder Study ✗ ✓ ✓ SleepFM 86.764.985.879.875.760.880.980.1 [10]OSF86.8 (+0.1) 65.3 (+0.4) 86.4 (+0.6) 80.5 (+0.7) 77.6 (+1.9) 62.2 (+1.4) 81.1 (+0.2) 80.4 (+0.3) Sleep Micro-Event. Study ✓ ✗ ✓ SleepFM 96.287.289.583.767.756.772.571.8 [11]OSF97.1 (+0.9) 89.9 (+2.7) 92.3 (+2.8) 87.5 (+3.8) 68.7 (+1.0) 57.1 (+0.4) 71.6 (-0.9) 70.7 (-1.1) In-Home Study ✗ ✓ ✗ SleepFM 86.765.285.779.777.462.081.180.5 [31]OSF87.4 (+0.7) 65.9 (+0.7) 86.6 (+0.9) 80.8 (+1.1) 81.0 (+3.6) 65.1 (+3.1) 81.9 (+0.8) 81.3 (+0.8) Table 8|Inference with different types of chan- nel corruption. OSF demonstrates stronger robust- ness across practical channel corruption scenarios. SettingMethod Sleep StagingHypopnea AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ Random Noise SleepFM 96.187.173.159.7 OSF97.190.069.457.5 Temporal Detach SleepFM 96.187.069.157.5 OSF97.390.476.261.2 Evaluation under more types of channel cor- ruption. To further evaluate robustness beyond missing-channel settings, we consider two addi- tional corruption scenarios applied to non-brain- related channels: noise interference and tempo- ral channel detachment. We show linear prob- ing results on MROS dataset in Table 8, OSF achieves the best performance in most settings and tasks, suggesting that its robustness extends beyond channel removal to other practically rel- evant forms of channel corruption. The advantage is particularly clear under temporal detachment, which is consistent with our use of channel masking and time-wise masking during pre-training. Pre-training with fewer channels. Brain-related channels provide strong cues for many sleep analysis tasks, but they are inconvenient for patients to collect. We therefore study a practical pre-training setup that uses only ECG and respiratory channels [10]. We evaluate on hypopnea and oxygen desaturation, since these tasks are primarily driven by respiratory and cardiac physiology. As shown in Table 6, OSF makes better use of the available channels and consistently outperforms SleepFM by a large margin on both hypopnea and oxygen desaturation. Overall, OSF better addresses practical constraints in inference and pre-training with incomplete channels. However, a meaningful performance gap remains when task-critical channels are absent. We encourage future work to further improve robustness under channel missingness. 6.2. Ablation on Design Variations Table 9|Comparisons of different embedding ag- gregation methods. Top-푘selection consistently out- performs mean pooling for patient-level prediction. Aggregation Coronary Dis.DiabetesHypertension AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ Avg Pool65.464.660.055.464.264.1 LSTM63.563.059.955.961.860.9 MIL [15]66.165.558.653.762.361.4 Top-푘 Selection66.866.362.856.364.163.5 How to aggregate epoch-level embeddings for patient-level tasks? Patient-level dis- ease prediction requires aggregating epoch- level representations into a single patient- level feature. Given a full-night recording, we segment it into푁non-overlapping 30- second epochs, feed each epoch into the pre- trained encoder, and obtain a sequence of embeddingsz 푖 푁 푖=1 wherez 푖 ∈ℝ 퐷 . Prior work [36] typically uses simple pooling or a learnable temporal aggregator (e.g., an LSTM [13]) to summarize this sequence. However, downstream datasets often contain only a limited number of patient-level recordings, which can increase the risk of overfitting. This motivates 10 OSF : On Pre-training and Scaling of Sleep Foundation Models us to test whether a learnable aggregation module is necessary. We hypothesize that disease-related signals concentrate in specific sleep periods, and thus simple average pooling may be suboptimal for aggregating epoch embeddings. To test this idea, we evaluate multiple-instance learning (MIL) [15] and an LSTM-based aggregator. We also test a selection module trained with straight-through estimation to select and average the top-푘disease-relevant epoch embeddings. We apply each method to embeddings extracted from multiple pre-trained models and average performance across models. As shown in Table 9, top-푘selection outperforms mean pooling overall. Takeaway 3: Patient-level prediction benefits from better joint modeling of epoch-level embedding. Table 10|Comparison of augmentation strate- gies. We report linear-probing performance (AUC ↑ ). Standard vision augmentations offer lim- ited additional benefit beyond channel masking. Crop Time ChannelSleep Staging Arousal Hypopnea Ox. Desat. ✓ ✗95.485.777.280.6 ✗✓96.490.577.180.3 ✓ ✓96.689.377.381.0 ✗✓97.392.877.781.5 Vision-style augmentations transfer to sleep data, but are not necessary for strong perfor- mance. To test whether standard vision aug- mentations (e.g., cropping) can be adapted to sleep data, we implement temporal cropping by randomly selecting a contiguous segment of each input signal, where the crop ratio푟is sam- pled uniformly from0.25, 0.75. We pre-train models under the same configuration using (1) crop-only augmentation and (2) cropping com- bined with our channel-masking augmentation. As shown in Table 10, cropping alone yields reasonable performance but remains worse than channel masking alone. Moreover, adding cropping on top of our augmentation does not yield consistent gains. These results suggest that additional augmentations can help in some cases, but are not required to obtain strong performance in our setting. Table 11|Ablation on channel masking ratios. A moderate masking ratio works the best. Masking Ratio Sleep StagingHypopnea AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ 0.195.987.277.461.5 0.597.390.477.762.2 0.995.385.975.060.3 Ablation on channel masking ratio. We further ablate the channel masking ratio by pre-training two additional variants with masking ratios of 0.1 and 0.9, while keeping all other settings un- changed. As shown in Table 11, the default ratio of 0.5 achieves the best overall linear probing performance across tasks, suggesting that moder- ate masking better balances pretext-task difficulty and input preservation for downstream transfer. More results are in Appendix E. 7. Related Work Physiological Foundation Models. Foundation models (FMs) have emerged as a strong approach for analyzing physiological time-series data [14,39,43]. Building on this line of work, [36] propose SleepFM for sleep physiology, showing strong transfer performance on downstream tasks such as sleep staging and disease prediction. SleepFM partitions channels into modality groups and applies contrastive objectives to align their embeddings. Despite these advances, the sleep domain still lacks a clear understanding of which pre-training design choices drive strong representations, and it remains unclear whether scaling trends reported for other physiological FMs [25,47] reliably hold for sleep data. Our work fills this gap by establishing a fully open multi-source benchmarkSleepBenchand conducting a controlled study of pre-training objectives and scaling behavior for sleep foundation models. 11 OSF : On Pre-training and Scaling of Sleep Foundation Models Self-Supervised Learning. Self-supervised learning is widely used to build FMs across domains. Invariance-based methods, including contrastive learning [6,40] and self-distillation [26], have achieved strong results, while reconstruction-based methods such as MAE [12] and VQ-VAE [37] are also effective. In language and time-series modeling, autoregressive pre-training has been especially successful [1,8,30]. However, existing sleep-focused studies cover only a subset of this design space [27,36] and lack a unified framework for evaluating and comparing methods. As a result, the field still lacks a systematic analysis of how pre-training design choices affect downstream performance for sleep FMs. In contrast, we benchmark representative objectives from these SSL families under a unified protocol and derive practical design guidelines for scalable sleep FM pre-training. 8. Conclusion In this paper, we present a systematic study of key design choices for building sleep FMs through controlled experiments on a fully open, multi-source corpus. Through open benchmarking, we identify three main findings and establish best practices for pre-training sleep FMs. Guided by these insights, we scale both pre-training data and model capacity, explicitly encourage channel-invariant feature learning, and develop the OSF family of sleep FMs, which consistently outperforms existing models. Extensive evaluations across nine datasets and eight downstream tasks demonstrate the effectiveness, robustness, and intriguing behaviors of OSF. Acknowledgments We gratefully acknowledge the support by Amazon Science Hub and UCLA DataX. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the funders. References [1]Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series. arXiv preprint arXiv:2403.07815, 2024. [2]Pierrick J Arnal, Valentin Thorey, Eden Debellemaniere, Michael E Ballard, Albert Bou Hernandez, Antoine Guillot, Hugo Jourde, Mason Harris, Mathias Guillard, Pascal Van Beers, et al. The dreem headband compared to polysomnography for electroencephalographic signal acquisition and sleep staging. Sleep, 43(11):zsaa097, 2020. [3]Indu Ayappa, Robert G Norman, Vijay Seelall, and David M Rapoport. Validation of a self- applied unattended monitor for sleep disordered breathing. Journal of Clinical Sleep Medicine, 4(1):26–37, 2008. [4]Terri Blackwell, Kristine Yaffe, Sonia Ancoli-Israel, Susan Redline, Kristine E Ensrud, Marcia L Stefanick, Alison Laffan, Katie L Stone, and Osteoporotic Fractures in Men Study Group. Associations between sleep architecture and sleep-disordered breathing and cognition in older community-dwelling men: the osteoporotic fractures in men sleep study. Journal of the American Geriatrics Society, 59(12):2217–2225, 2011. [5] Jonathan F Carter and Lionel Tarassenko. wav2sleep: A unified multi-modal approach to sleep stage classification from physiological signals. In Machine Learning for Health (ML4H), pages 186–202. PMLR, 2025. 12 OSF : On Pre-training and Scaling of Sleep Foundation Models [6]Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PmLR, 2020. [7]Xiaoli Chen, Rui Wang, Phyllis Zee, Pamela L Lutsey, Sogol Javaheri, Carmela Alcántara, Chandra L Jackson, Michelle A Williams, and Susan Redline. Racial/ethnic differences in sleep disturbances: the multi-ethnic study of atherosclerosis (mesa). Sleep, 38(6):877–888, 2015. [8] Ben Cohen, Emaad Khwaja, Youssef Doubli, Salahidine Lemaachi, Chris Lettieri, Charles Masson, Hugo Miccinilli, Elise Ramé, Qiqi Ren, Afshin Rostamizadeh, et al. This time is different: An observability perspective on time series foundation models. arXiv preprint arXiv:2505.14766, 2025. [9]Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. [10] Kevin Gleason, Donghoon Shin, Michael Rueschman, Tanya Weinstock, Rui Wang, James H Ware, Murray A Mittleman, and Susan Redline. Challenges in recruitment to a randomized controlled study of cardiovascular disease reduction in sleep apnea: an analysis of alternative strategies. Sleep, 37(12):2035–2038, 2014. [11]Naihua N Gong, Aditya Mahat, Samya Ahmad, Daniel Glaze, Mirjana Maletic-Savatic, Matthew McGinley, Anne Marie Morse, Alcibiades J Rodriguez, Audrey Thurm, Susan Redline, et al. Leveraging clinical sleep data across multiple pediatric cohorts for insights into neurodevelop- ment: the retrospective analysis of sleep in pediatric (rasp) cohorts study. Sleep, page zsaf157, 2025. [12]Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022. [13]Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. [14]Weixuan Huang, Yan Wang, Hanrong Cheng, Wei Xu, Tingyue Li, Xiuwen Wu, Hui Xu, Pan Liao, Zaixu Cui, Qihong Zou, et al. A unified time-frequency foundation model for sleep decoding. Nature Communications, 2026. [15] Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In International conference on machine learning, pages 2127–2136. PMLR, 2018. [16]Ziyu Jia, Youfang Lin, Jing Wang, Xiaojun Ning, Yuanlai He, Ronghao Zhou, Yuhan Zhou, and Li-wei H Lehman. Multi-view spatial-temporal graph convolutional networks with domain gen- eralization for sleep stage classification. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 29:1977–1986, 2021. [17]Magnus Ruud Kjaer, Rahul Thapa, Gauri Ganjoo, Hyatt Moore IV, Poul Joergen Jennum, Bran- don M Westover, James Zou, Emmanuel Mignot, Bryan He, and Andreas Brink-Kjaer. Stanford sleep bench: Evaluating polysomnography pre-training methods for sleep foundation models. arXiv preprint arXiv:2512.09591, 2025. [18]Harlin Lee, Boyue Li, Shelly DeForte, Mark L Splaingard, Yungui Huang, Yuejie Chi, and Simon L Linwood. A large collection of real-world pediatric sleep studies. Scientific Data, 9(1):421, 2022. 13 OSF : On Pre-training and Scaling of Sleep Foundation Models [19]Sirui Li, Shuhan Xiao, Mihir Joshi, Ahmed Metwally, Daniel McDuff, Wei Wang, and Yuzhe Yang. Hearts: Benchmarking llm reasoning on health time series. arXiv preprint arXiv:2603.06638, 2026. [20]Ziyu Liu, Azadeh Alavi, Minyi Li, and Xiang Zhang. Self-supervised contrastive learning for medical time series: A systematic review. Sensors, 23(9):4221, 2023. [21]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. [22]Carole L Marcus, Reneé H Moore, Carol L Rosen, Bruno Giordani, Susan L Garetz, H Gerry Taylor, Ron B Mitchell, Raouf Amin, Eliot S Katz, Raanan Arens, et al. A randomized trial of adenotonsillectomy for childhood sleep apnea. New England Journal of Medicine, 368(25):2366– 2376, 2013. [23]Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018. [24] Yeongyeon Na, Minje Park, Yunwon Tae, and Sunghoon Joo. Guiding masked representa- tion learning to capture spatio-temporal relationship of electrocardiogram. arXiv preprint arXiv:2402.09450, 2024. [25]Girish Narayanswamy, Xin Liu, Kumar Ayush, Yuzhe Yang, Xuhai Xu, Shun Liao, Jake Garrison, Shyam Tailor, Jake Sunshine, Yun Liu, et al. Scaling wearable foundation models. arXiv preprint arXiv:2410.13638, 2024. [26]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. [27]Saurav Raj Pandey, Aaqib Saeed, and Harlin Lee. Pedsleepmae: Generative model for multimodal pediatric sleep signals. In 2024 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI), pages 1–8. IEEE, 2024. [28]Mathias Perslev, Sune Darkner, Lykke Kempfner, Miki Nikolic, Poul Jørgen Jennum, and Christian Igel. U-sleep: resilient high-frequency sleep staging. NPJ digital medicine, 4(1):72, 2021. [29]Stuart F Quan, Barbara V Howard, Conrad Iber, James P Kiley, F Javier Nieto, George T O’Connor, David M Rapoport, Susan Redline, John Robbins, Jonathan M Samet, et al. The sleep heart health study: design, rationale, and methods. Sleep, 20(12):1077–1085, 1997. [30]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [31]Susan Redline, Daniela Sotres-Alvarez, Jose Loredo, Martica Hall, Sanjay R Patel, Alberto Ramos, Neomi Shah, Andrew Ries, Raanan Arens, Janice Barnhart, et al. Sleep-disordered breathing in hispanic/latino individuals of diverse backgrounds. the hispanic community health study/study of latinos. American journal of respiratory and critical care medicine, 189(3):335–344, 2014. [32]Susan Redline, Peter V Tishler, Tor D Tosteson, John Williamson, Kenneth Kump, Ilene Browner, Veronica Ferrette, and Patrick Krejci. The familial aggregation of obstructive sleep apnea. American journal of respiratory and critical care medicine, 151(3):682–687, 1995. 14 OSF : On Pre-training and Scaling of Sleep Foundation Models [33]Carol L Rosen, Emma K Larkin, H Lester Kirchner, Judith L Emancipator, Sarah F Bivins, Susan A Surovec, Richard J Martin, and Susan Redline. Prevalence and risk factors for sleep-disordered breathing in 8-to 11-year-old children: association with race and prematurity. The Journal of pediatrics, 142(4):383–389, 2003. [34] Scott A Sands, Philip I Terrill, Bradley A Edwards, Luigi Taranto Montemurro, Ali Azarbarzin, Melania Marques, Camila M De Melo, Stephen H Loring, James P Butler, David P White, et al. Quantifying the arousal threshold using polysomnography in obstructive sleep apnea. Sleep, 41(1):zsx183, 2018. [35]Adam P Spira, Terri Blackwell, Katie L Stone, Susan Redline, Jane A Cauley, Sonia Ancoli-Israel, and Kristine Yaffe. Sleep-disordered breathing and cognition in older women. Journal of the American Geriatrics Society, 56(1):45–50, 2008. [36]Rahul Thapa, Magnus Ruud Kjaer, Bryan He, Ian Covert, Hyatt Moore IV, Umaer Hanif, Gauri Ganjoo, M Brandon Westover, Poul Jennum, Andreas Brink-Kjaer, et al. A multimodal sleep foundation model for disease prediction. Nature Medicine, pages 1–11, 2026. [37]Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017. [38]Maxwell A Xu, Girish Narayanswamy, Kumar Ayush, Dimitris Spathis, Shun Liao, Shyam A Tailor, Ahmed Metwally, A Ali Heydari, Yuwei Zhang, Jake Garrison, et al. Lsm-2: Learning from incomplete wearable sensor data. arXiv preprint arXiv:2506.05321, 2025. [39] Zongzhe Xu, Zitao Shuai, Eideen Mozaffari, Ravi S Aysola, Rajesh Kumar, and Yuzhe Yang. Sleeplm: Natural-language intelligence for human sleep. arXiv preprint arXiv:2602.23605, 2026. [40] Yuzhe Yang, Xin Liu, Jiang Wu, Silviu Borac, Dina Katabi, Ming-Zher Poh, and Daniel McDuff. Simper: Simple self-supervised learning of periodic targets. In The Eleventh International Conference on Learning Representations, 2023. [41] Yuzhe Yang, Yuan Yuan, Guo Zhang, Hao Wang, Ying-Cong Chen, Yingcheng Liu, Christopher G Tarolli, Daniel Crepeau, Jan Bukartyk, Mithri R Junna, et al. Artificial intelligence-enabled detection and assessment of parkinson’s disease using nocturnal breathing signals. Nature Medicine, 28(10):2207–2215, 2022. [42]Terry Young, Mari Palta, Jerome Dempsey, Paul E Peppard, F Javier Nieto, and K Mae Hla. Burden of sleep apnea: rationale, design, and major findings of the wisconsin sleep cohort study. WMJ: official publication of the State Medical Society of Wisconsin, 108(5):246, 2009. [43] Hang Yuan, Shing Chan, Andrew P Creagh, Catherine Tong, Aidan Acquah, David A Clifton, and Aiden Doherty. Self-supervised learning for human activity recognition using 700,000 person-days of wearable data. NPJ digital medicine, 7(1):91, 2024. [44] Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 8980–8987, 2022. [45]Bing Zhai, Greg J Elder, and Alan Godfrey. Challenges and opportunities of deep learning for wearable-based objective sleep assessment. npj Digital Medicine, 7(1):85, 2024. 15 OSF : On Pre-training and Scaling of Sleep Foundation Models [46]Guo-Qiang Zhang, Licong Cui, Remo Mueller, Shiqiang Tao, Matthew Kim, Michael Rueschman, Sara Mariani, Daniel Mobley, and Susan Redline. The national sleep research resource: towards a sleep data commons. Journal of the American Medical Informatics Association, 25(10):1351–1358, 2018. [47] Yuwei Zhang, Kumar Ayush, Siyuan Qiao, A Ali Heydari, Girish Narayanswamy, Maxwell A Xu, Ahmed A Metwally, Shawn Xu, Jake Garrison, Xuhai Xu, et al. Sensorlm: Learning the language of wearable sensors. arXiv preprint arXiv:2506.09108, 2025. 16 OSF : On Pre-training and Scaling of Sleep Foundation Models A. Implementation Details A.1. Detailed Pre-training Setups In this section, we provide implementation details of our pre-training configurations for both the bench- marked baselines and our method. For our explorations, we aim to directly adapt well-established self-supervised pre-training methods, and benchmark their performance on the sleep data. In addition to the existing pre-training method SleepFM [36], we have considered four representative families of SSL objectives: (1) Contrastive learning: SimCLR [6]; (2) Reconstruction-based methods: MAE [12] and VQ-VAE [37]; (3) Autoregressive modeling: Autoregression [30]; (4) Self-distillation: DINO [26]. For OSF model family, we use DINO as a specification and incorporate it with our identified pre- training recipes, and form OSF used in the main experiments. We have also incorporated OSF with SimCLR, whose performance in the main evaluation tasks is shown in Table 22. We consider them because these invariance-based methods are a natural testbed for applying our pre-training recipes. Specifically, contrastive learning and self-distillation-based methods are invariance-based methods, which aim to align two (or more) augmented views of the same input in the latent space. As a result, their downstream performance usually depends on the approach we use to augment the raw input data. Table 12|Encoder backbone configurations used in our experiments. Model Architecture Params (M) Width Depth Heads MLP dim ViT-1M128645121.7M ViT-5M1921237685.5M ViT-85M7681212307285.5M Backbone and training protocol. Unless other- wise specified, all methods reported in the main tables use the same ViT-85M Transformer back- bone [9]. We replace the patchify layer in the original ViT with a simple convolution layer to project the input signal into tokens, and the ar- chitecture details are in Table 12. We pre-train all models on the same split of the pre-training cohorts in ourSleepBench. Each run is trained for up to 30 epochs, with early stopping when we observe training converges, based on the loss curves. Optimization and scheduling. We use AdamW [21] for all methods. For the scheduler of learning rate, we follow a standard configuration of self-supervised pre-training, and use a linear warmup for the first 10% of total training steps, and then apply cosine annealing decay. We have tuned the learning rate and batch size for each method independently to ensure a fair comparison under its best-performing regime. We first search batch size, for the maximized usage of GPU memory; and then optimize learning rate choices based on the performance. Augmentations for OSF. For OSF, as discussed in Sec. 3, we adopt a two-stage masking augmentation. We first apply channel masking by randomly dropping 50% of input channels. We then apply block- wise temporal masking, where the temporal masking ratio is independently sampled for each training example from a uniform distribution over 0.3, 0.6. The complete pre-training configuration, including detailed hyper-parameters and method-specific setup, is summarized in Table 13. 17 OSF : On Pre-training and Scaling of Sleep Foundation Models Table 13| Pretraining configurations across different methods. ConfigurationSleepFM SimCLR DINOMAEVQ-VAEAROSF (SimCLR) OSF (DINO) Max Epoch30 Batch Size25603200102496003200320032001024 Base Learning Rate1.00E-04 1.00E-04 5.00E-05 3.00E-04 1.00E-04 1.00E-041.00E-045.00E-05 OptimizerAdamW Opt. momentum 훽 1 ,훽 2 0.9, 0.95 Weight Decay0.20.20.20.20.20.050.20.2 Gradient Clipping–3–3 Learning Rate SchedulerLinearWarmup & CosineDecay Token window length64 Table 14| Training hyperparameters for sleep analysis tasks and disease prediction tasks. Hyperparameter Sleep Analysis TasksDisease Prediction Linear Probing Finetuning Supervised Mean Pool MIL LSTM TopK End2End Base learning rate0.11e-41e-31e-25e-3 5e-3 5e-31e-3 Batch size800 per GPU256 per GPU128 per GPU16 per GPU Max step500– Max epoch–5505 Learning rate schedulerLinearWarmup & CosineDecay OptimizerAdamW A.2. Detailed Downstream Evaluation Setups Evaluation setup. We evaluate pre-trained models on three types of downstream tasks under three standard transfer learning protocols. For epoch-level evaluation, we consider a 4-class sleep staging task, and four sleep-event detection tasks. For sleep staging, we follow [5] and merge N1 and N2 into a single light sleep category, and predict four classes: awake, light sleep, deep sleep, and REM. For sleep-event detection, we perform 2-class classification for the four following event detection tasks: arousal, hypopnea, oxygen desaturation, and central apnea. We also consider three patient-level disease classification: coronary disease, diabetes, and hypertension. We report both AUC and AUPRC since all tasks are class-imbalanced. For epoch-level evaluation, we consider three transfer learning settings: linear probing, full fine-tuning, and few-shot adaptation. In linear probing, we freeze the encoder and train a linear classifier on the full training split of each target dataset, then evaluate on the held-out test set. In full fine-tuning, we adopt a similar setting on data usage, and in addition to training a linear classifier, we also jointly optimize the encoder and classifier. In few-shot adaptation, we sample퐾examples per class from the training split. In particular, we consider 퐾 ∈ 1, 5, 50 in our main experiments. We keep the encoder frozen, train a linear classifier, and evaluate on the same test split across methods. For patient-level prediction, we segment each recording into non-overlapping 30-second epochs and extract an embedding for each epoch using the pre-trained encoder. For each pre-trained checkpoint, we pre-extract epoch embeddings for all recordings before training and evaluating on the downstream disease prediction tasks. During embedding extraction, we pad the epoch sequence length to 1200, corresponding to 10 hours. During downstream training, we consider two approaches of aggregating the epoch embeddings into a single patient-level representation: (i) we learn a lightweight aggregation module on top of epoch embeddings of each patient, or (i) we use simple mean pooling over all epochs. We then feed the aggregated patient-level representation to a classifier for disease prediction. Detailed configurations of evaluation tasks can be seen in Table 14. 18 OSF : On Pre-training and Scaling of Sleep Foundation Models ECG EMG_Chin EMG_LLeg EMG_RLeg ABD THX NP SN EOG_E1_A2 EOG_E2_A1 EEG_C3_A2 EEG_C4_A1 Figure 10|Visualization of the pre-processed 12 channels of the sleep data used in our paper. We pick a 30-sec epoch for demonstration. B. Details of SleepBench B.1. Dataset Statistics Table 15|Comparison betweenSleepBenchand existing open-source sleep datasets. DatasetNum. Recordings Num. Hours SSC [17]17,467163,000 SleepBench21,482166,500 To enable open benchmarking and our con- trolled explorations, we curated a large-scale benchmarkSleepBench, comprising nine di- verse sleep datasets: SHHS [29], NCHSDB [18], WSC [42], CCSHS [33], CFS [32], MROS [4], MESA [7], CHAT [22], and SOF [35]. This consolidated corpus represents a significant va- riety of demographic profiles and sleep disor- der prevalences, totaling 20M epochs of sleep recording data. Table 16 and Table 17 report the statistics of patients and 30-second epochs, across different datasets and data splits. B.2. Downstream Task Labels Sleep Event Distribution. We report the prevalence of sleep events of datasets used in main experi- ments, in the Table 18. The datasets exhibit substantial heterogeneity in event distributions, which enables the evaluation of the generalization capability of pre-trained models. During dataset splitting, 19 OSF : On Pre-training and Scaling of Sleep Foundation Models Table 16|Distribution of sleep recordings across different datasets and splits. Models are pretrained on pretrain datasets and evaluated on both pretrain and out-of-domain (OOD) datasets. The test split of pretrain data and all OOD data are unseen during pre-training. Split Pretrain DatasetsO.O.D. Datasets Total SHHS NCHSDBWSC CCSHS CFS MROS MESA CHAT SOF Train6,6633,094 1,3974032693,1011,367531361 17,186 Valid811392182593438517466452,148 Test802403194532638118067422,148 Total8,2763,889 1,7735153293,8671,721664448 21,482 Table 17| Distribution of 30-second epochs across different datasets and splits. Split Internal Pretrain DatasetsExternal Eval. Datasets Total SHHS NCHSDBWSC CCSHSCFSMROSMESA CHATSOF Train 5,919k2,892k 1,270k432k 255k 3,015k 1,312k562k 341k 15,997k Valid724k367k164k64k32k370k164k69k42k1,996k Test711k376k175k57k24k372k172k70k38k1,995k Total7,354k3,634k 1,609k553k 311k 3,757k 1,648k701k 421k 19,989k Table 18|Distributions of sleep event detection tasks across datasets used for sleep event evaluation. For each sleep disorder event, we report its prevalence in each split of each used dataset. DatasetSplitEpochsArousalHypopneaOxygen Desat.Central Apnea SHHSTrain5,919,21717.85%29.13%23.46%0.71% SHHSValid724,05617.73%29.00%23.71%0.74% SHHSTest710,82117.60%29.00%23.32%0.71% MROSTrain3,014,91519.46%12.27%52.28%1.32% MROSValid370,37819.55%12.62%52.65%1.66% MROSTest371,87319.77%12.26%51.93%1.42% we perform stratified sampling at the patient level to prevent information leakage and to keep event distributions consistent across splits. Specifically, we first randomly assign patients to the training, validation, and test splits. We then assign each epoch to the split of its corresponding patient via patient ID. As shown in Table 18, the data was divided into train, validate, and test sets with an approximate global ratio of 80:10:10 strictly on the subject level. Disease Classification: For more clinically valuable downstream tasks, we perform patient-level disease classification on the MROS dataset. We focused on three diagnosis tasks: Coronary Disease, Diabetes, and Hypertension. These labels are formed from disease-related variables provided MROS dataset. We set the disease label as positive, if the patient has self-reported related medication usage of diagnosis history. As detailed in Table 19, the MROS dataset was split into training, validation, and testing sets. The three labels cover both class-balance and class-imbalance scenes, therefore more comprehensively evaluating the pre-trained model’s ability on disease prediction tasks. 20 OSF : On Pre-training and Scaling of Sleep Foundation Models Table 19|Distribution of subject-level disease labels in the MROS dataset. We filtered out patients with NaN values on the selected labels. Here we report the number and percentage of both negative and positive samples across splits. Disease CategorySplit NegativePositive Total Samples CountRatioCountRatio Coronary Disease Train1,62671.3%65428.7%2,280 Valid18867.1%9232.9%280 Test18364.2%10235.8%285 Diabetes Train1,98786.9%30013.1%2,287 Valid24788.2%3311.8%280 Test24887.0%3713.0%285 Hypertension Train1,15250.4%1,13449.6%2,286 Valid13849.3%14250.7%280 Test14249.8%14350.2%285 B.3. Dataset Pre-processing Details As mentioned in Sec. 2, we utilize a standardized 12-channel montage covering 4 physiological modality groups:❶Brain (EEG/EOG):C3-A2,C4-A1,E1-A2,E2-A1,❷Respiration:Abdominal, Thorax,Nasal Pressure,Snore,❸Cardiac:ECG, and❹Somatic:EMG-Chin,EMG-LLeg,EMG-RLeg. The detailed channel descriptions are in Table 20. For the pre-processing of the full-night recordings, we follow [5], and first resample each channel to a unified sampling frequency, as detailed in Table 20. Next, we conduct manual quality control by trimming prolonged wake periods at the beginning and end of the night, which also removes extreme artifacts introduced during sensor setup and removal. We then apply per-night z-score normalization to each channel, followed by clipping the normalized signals to the range−6, 6to suppress outliers. The clipping thresholds are chosen based on the empirical distribution of z-scored signals. For respiratory channels, we additionally apply area-dependent z-score normalization to reduce amplitude drift and improve cross-night consistency. We follow prior work and segment each night into non-overlapping 30-second epochs, yielding roughly 20 million epochs for self-supervised pre-training. We resample all channels in the segmented epochs to 64 Hz and zero-pad missing channels. The pre-processed data inSleepBenchis demonstrated in Fig. 10. C. Additional Results and Analysis C.1. Additional Results of Sleep Analysis Tasks OSF achieves state-of-the-art on sleep analysis tasks on in-domain evaluation set. We also conduct evaluation on sleep analysis tasks on in domain dataset SHHS. As shown in Table 21. OSF achieves the best performance under both linear probing and full fine-tuning. The same trend also holds on the out-of-domain cohorts reported in the main text, further supporting the superiority of our method. OSF achieves state-of-the-art on few-shot adaptation tasks. We conduct few-shot learning evalua- tions on the MROS dataset, on three sleep event detection tasks and the sleep staging task, across 21 OSF : On Pre-training and Scaling of Sleep Foundation Models Table 20|Channel configuration used for main experiments. During post-processing, they are resampled to 64 Hz. GroupChannel Full nameElectrode DetailPreprocessing Frequency CardiacECGElectrocardiographyECG left - ECG right128 Hz SomaticEMGElectromyography EMG_Chin64 Hz EMG from left leg64 Hz EMG from right leg64 Hz Resp. Effort ABDAbdominal respiratory effort8 Hz THXThoracic respiratory effort8 Hz NPNasal pressure8 Hz SNSnore32 Hz Brain Activity EOGElectrooculography EOG E1 - A264 Hz EOG E2 - A164 Hz EEGElectroencephalography EEG C3 - A264 Hz EEG C4 - A164 Hz 1-shot, 5-shot, and 50-shot settings. OSF achieves the overall best performance under 1-shot and 5-shot evaluation (Tables 23 and 24), and performs best on all tasks under 50-shot evaluation (Table 25). Table 27|Disease prediction results on SHHS. OSF outperforms the baseline method on most metrics. Method DiabetesHypertension AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ SleepFM 79.556.576.7 76.1 OSF79.8 58.1 76.874.7 OSF outperforms physiological-signal-specific pre- training methods. To assess whether methods tailored to physiological signals transfer effectively to sleep foundation modeling, we further compare OSF with five recent domain-specific baselines: TS2Vec [44], ST-MEM [24], LSM-2 [38], PedSleep- MAE [27], and SleepGPT [14]. We pre-train Ped- SleepMAE onSleepBench, evaluate SleepGPT using its released checkpoint, and adapt the remaining methods to our benchmark following their original settings when applicable. As shown in Table 26, OSF consistently achieves the strongest performance on all of the downstream tasks on MROS dataset. OSF improves in-domain disease prediction. We further evaluate in-domain disease prediction on SHHS using two additional outcomes, hypertension and diabetes. As shown in Table 27, OSF consistently outperforms the compared baselines, further supporting the effectiveness of our pre- training strategy for clinically relevant downstream prediction tasks. Table 29|Additional sleep staging evaluation metrics. OSF remains the best. Method Acc. (micro) F1 (weighted) Cohen Kappa SleepFM85.285.079.9 SimCLR63.259.525.2 DINO79.478.863.9 VQ-VAE83.683.474.6 MAE85.485.279.6 AR84.083.775.7 OSF87.086.982.8 Confidence intervals. We additionally report confidence intervals for the main results to en- able a more reliable comparison across meth- ods. As shown in Table 28, the confidence inter- vals show that the performance gains of OSF over competing baselines are consistent across tasks, further supporting the robustness of our improvements. Additional evaluation metrics for sleep stag- ing. Since several downstream tasks are label- imbalanced, we use AUROC and AUPRC as the primary threshold-independent metrics. For 22 OSF : On Pre-training and Scaling of Sleep Foundation Models Table 21|Sleep staging and sleep event detection results on SHHS. OSF achieves overall best performance among all compared methods. Eval Method Sleep StagingArousalHypopneaOx. Desat.Central Apnea AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ Supervised ViT 97.092.388.782.379.174.876.070.195.261.7 LP SleepFM 96.691.192.286.382.077.080.675.095.964.7 SimCLR86.167.471.964.264.960.762.958.574.651.3 DINO95.187.487.079.978.373.776.570.594.860.6 VQ-VAE96.791.389.483.080.275.478.072.295.763.6 MAE96.9 91.991.485.783.078.480.575.096.766.0 AR96.991.890.688.479.073.378.672.994.461.2 OSF97.593.294.089.382.878.181.375.696.966.5 FT SleepFM 97.092.095.191.086.783.382.577.197.168.6 SimCLR95.488.385.878.480.376.375.569.695.764.1 DINO97.192.493.889.386.283.081.876.197.469.0 VQ-VAE97.593.494.790.785.582.281.776.297.268.4 MAE97.493.094.690.687.1 84.283.277.997.768.5 AR97.493.194.290.086.483.382.477.197.268.2 OSF97.994.395.792.187.684.683.778.697.669.8 Table 22|Sleep staging and sleep event detection results on SHHS. SimCLR can also be upgraded by our pre-training recipts and achieve better performance. Eval Method Sleep StagingArousalHypopneaOx. Desat.Central Apnea AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ SimCLR86.167.471.964.264.960.762.958.574.651.3 OSF (SimCLR)97.392.992.987.482.277.280.474.796.364.8 completeness, we also report Accuracy and weighted F1 for sleep staging results shown in the main table, as shown in Table 29. The results are consistent with the main findings, with OSF remaining the strongest overall method. Demographic analysis. We further evaluate subgroup generalization within cohorts using age subgroups in CFS dataset and gender subgroups in MESA dataset. For each pretrained model, we fine-tune and evaluate the downstream classifier separately within each subgroup. As shown in Tables 30, OSF outperforms SleepFM in most subgroup-task settings and shows lower performance variation across demographic groups. These results suggest that the benefits of OSF extend beyond cohort-level averages to subgroup-level evaluation within heterogeneous sleep cohorts. C.2. Additional Analysis Studies Channel masking consistently improves downstream performance across cohorts and tasks. Here, we report additional experiments on more downstream tasks and data cohorts. In Table 32, we report results across 3 sleep disorder event detection tasks and sleep staging tasks, on both the out-of-domain dataset MROS and the in-domain dataset SHHS. Under these broader evaluation settings, we observe that channel masking remains a strong augmentation strategy, compared to crop or time-masking strategies. Other findings in Sec. 3.2 and Sec. 6.2 also hold. These results suggest that encouraging channel-invariant representations is important for improving downstream performance. 23 OSF : On Pre-training and Scaling of Sleep Foundation Models Table 23|1-shot few-shot results on MROS. OSF shows better adaptability to unseen out-of-domain data. Few-shot Method Sleep StagingArousalHypopneaOx. Desat. AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ 푘=1 SleepFM 59.632.356.653.849.349.855.254.1 SimCLR50.825.151.250.649.950.050.950.8 DINO67.041.652.751.245.848.760.258.9 MAE63.036.649.350.049.149.954.453.4 VQ-VAE62.633.756.053.350.8 50.355.854.7 AR67.643.448.049.447.449.449.349.4 OSF75.948.961.457.055.451.856.655.7 Table 24|5-shot few-shot results on MROS. OSF shows better adaptability to unseen out-of-domain data. Few-shot Method Sleep StagingArousalHypopneaOx. Desat. AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ 푘=5 SleepFM 71.845.355.553.049.349.859.8 58.6 SimCLR53.027.154.252.250.050.050.750.5 DINO82.855.856.853.347.749.257.856.1 MAE82.659.256.753.749.749.962.961.2 VQ-VAE78.151.852.251.151.150.458.957.6 AR81.960.157.154.050.950.455.153.9 OSF89.270.167.961.355.952.056.455.7 Table 33|Heart-rate prediction results on SHHS and MROS. Reconstruction- based SSL objectives help the regression task. Method MROSSHHS MAE ↓ RMSE ↓ MAE ↓ RMSE ↓ SleepFM 3.385.00 3.525.06 SimCLR8.11 10.09 8.14 10.06 DINO7.399.26 7.409.23 VQ-VAE4.035.79 3.505.22 MAE2.333.992.043.63 AR2.173.91 1.893.56 OSF3.224.94 3.424.94 Mean7.97 10.30 7.97 10.15 Our identified pre-training recipes also apply to other pre-training methods. To verify whether our pre-training recipes generalize to other self-supervised learning meth- ods, we adopt our identified pre-training recipes to the original SimCLR objective, yielding an upgraded vari- ant (OSF (SimCLR)). We evaluate both SimCLR and OSF (SimCLR) models on the in-domain SHHS cohort. As shown in Table 22, the upgraded SimCLR (OSF (Sim- CLR)) consistently outperforms the original SimCLR by a large margin across all tasks. Ablation on data normalization strategy. We further compare our per-night per-channel z-score normalization with min-max normalization, which is commonly used in prior physiological self-supervised learning studies. Using the same model architecture and training protocol, we pre-train a variant with min-max normalized inputs and evaluate it on SHHS dataset and MROS dataset for sleep staging and hypopnea detection. As shown in Table 34, per-night z-score normalization achieves better performance across both datasets and tasks, suggesting that it better handles cross-subject and cross-cohort variation in sleep signal scale. Additional UMAP visualizations on imbalanced downstream tasks. UMAP is computed from 2,000 randomly sampled epochs from the unseen SHHS test split. We visualize embeddings for hypopnea 24 OSF : On Pre-training and Scaling of Sleep Foundation Models Table 25|50-shot few-shot results on MROS. OSF shows better adaptability to unseen out-of-domain data on all tasks. Few-shot Method Sleep StagingArousalHypopneaOx. Desat. AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ 푘=50 SleepFM 85.964.072.365.856.352.361.760.3 SimCLR56.629.155.853.150.650.249.549.6 DINO87.467.072.164.557.252.763.561.3 MAE92.1 79.676.568.960.253.666.865.1 VQ-VAE85.363.368.662.351.550.565.463.7 AR90.875.472.064.858.853.263.061.4 OSF94.582.583.977.362.954.968.466.7 Table 26|Comparison with domain-specific pretraining methods. OSF outperforms baselines tailored for physiological signals. Method SHHSMROS Sleep StagingHypopneaSleep StagingHypopnea AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ TS2Vec [44]90.276.768.664.386.967.361.954.5 ST-MEM [24]97.092.281.376.396.187.575.460.0 LSM-2 [38]95.989.680.375.495.886.774.660.1 PedSleepMAE [27] 89.273.464.861.184.059.059.453.4 SleepGPT [14]96.290.071.366.495.485.765.155.7 OSF97.593.282.878.197.390.477.762.2 and arousal events, which are substantially more imbalanced than sleep staging. As shown in Fig. 11, although rare-event tasks show less clean separation, OSF generally exhibits clearer task-related structure than the baseline. Reconstruction objectives transfer better on regression tasks on sleep data. As shown in Table 2, reconstruction-based methods such as MAE, VQ-VAE, and AR can be adapted to sleep signals with minimal modification and achieve reasonable performance; MAE even outperforms SleepFM [36] in many cases. In contrast, vanilla SimCLR and DINO lag behind both reconstruction-based methods and SleepFM. This indicates that invariance-based objectives require domain-appropriate augmentations to work well on sleep data. We also designed a heart-rate prediction task to test each pre-trained model’s capability in regression, and found that MAE and AR achieve good performance, as shown in Table 33. D. Additional Results on Scaling Analysis D.1. Scaling across Pre-training Sample Scale and Model Capacity Scaling pre-training data leads to larger gains in few-shot adaptation. Besides training with the full training split of the downstream task, we also tested the sample efficiency brought by scaling the pre-training sample size. Particularly, we run 1-shot, 5-shot, and 50-shot adaptation experiments to test whether increasing the pre-training sample size improves few-shot transfer. We conduct these experiments on MROS. As shown in Fig. 12, in both the 1-shot and 50-shot settings, models pre-trained 25 OSF : On Pre-training and Scaling of Sleep Foundation Models Table 28|Confidence intervals for sleep staging and sleep event detection. OSF achieves the best overall performance among all compared methods with statistical significance. Method Sleep StagingArousalHypopneaOx. Desat. AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ SleepFM96.4± 0.04 87.7± 0.15 90.3± 0.1284.9± 0.1775.5± 0.20 60.6± 0.17 80.9± 0.14 80.3± 0.15 SimCLR81.4± 0.11 56.8± 0.21 73.6± 0.21 67.8± 0.21 58.4± 0.25 53.1± 0.10 66.8± 0.18 65.0± 0.17 DINO93.6± 0.06 81.4± 0.18 82.2± 0.17 75.7± 0.21 72.7± 0.22 59.5± 0.16 78.7± 0.14 78.2± 0.15 VQ-VAE95.8± 0.05 86.7± 0.16 86.2± 0.15 80.3± 0.20 75.1± 0.21 60.9± 0.17 79.9± 0.14 79.5± 0.15 MAE96.5± 0.04 88.7± 0.1489.7± 0.12 84.3± 0.17 77.6± 0.1961.9± 0.1781.2± 0.1380.8± 0.14 Autoregression 96.0± 0.04 87.6± 0.15 87.8± 0.13 81.9± 0.18 72.1± 0.22 58.9± 0.15 79.0± 0.14 78.5± 0.15 OSF97.3± 0.0490.4± 0.1492.8± 0.1088.3± 0.1577.7± 0.1962.2± 0.1881.5± 0.1381.0± 0.15 Table 30|Robustness analysis on demographic subgroups. OSF achieves better performance and deliver fairer outcome. Subgroup of CFS Subgroup Setting Sleep Staging Hypopnea Young SleepFM97.992.2 OSF98.191.0 Old SleepFM96.883.6 OSF97.685.3 Disparity SleepFM0.786.08 OSF0.354.03 Subgroup of MESA Subgroup Setting Sleep Staging Hypopnea Male SleepFM94.674.5 OSF95.577.4 Female SleepFM94.276.8 OSF96.177.9 Disparity SleepFM0.281.63 OSF0.420.35 with more data consistently outperform those pre-trained with less data. Additional results on the effect of sample size scaling. We pre-train OSF and SleepFM with a ViT-85M backbone using different fractions of the pre-training corpus. As shown in Tables 36 and 35, we conduct more comprehensive evaluations on sleep staging task and three sleep event detection tasks. The downstream performance of OSF improves consistently as we increase the percentage of pre-training data. We observe a similar scaling trend for SleepFM when using the same 85M backbone, as shown in Tables 38 and 37. The models pre-trained with larger parameter size demonstrate good scaling behaviors, which motivates us to jointly scale both model size and sample size. AUC Figure 13|Scaling on pretraining steps. OSF scales on training com- putation. Single-source pre-training also exhibits scaling with sam- ple size. To further verify the generalization of scalability on sample size, we pre-train models on with different percentage of the SHHS cohort. As shown in Tables 36, Tables 35, Ta- bles 38, and Tables 37, models pre-trained on the single-source SHHS cohort also show clear scaling behavior. When we in- crease the fraction of SHHS used for pre-training, downstream performance improves consistently. Comprehensive results on model-size scaling. We pre-train OSF and SleepFM on the full pre-training corpus, with different parameters of the backbone: 1M, 5M, 85M. We then evaluate these checkpoints on both the in-domain SHHS cohort and the out-of-domain MROS cohort, on three sleep event detection tasks and the sleep staging task. As shown in Table 40, the scaling trend holds across all tasks and both cohorts, therefore, increasing model size leads to improved downstream performance. 26 OSF : On Pre-training and Scaling of Sleep Foundation Models Table 31|Controlled study of masking strategies for self-supervised pretraining on sleep data. We report linear-probing results on sleep analysis tasks (AUC) on SHHS and MROS datasets. Channel masking as an augmentation consistently improves representation quality. Method Mask StrategySHHSMROS Time ChannelSleep Staging Arousal Hypopnea Ox. Desat. Sleep Staging Arousal Hypopnea Ox. Desat. SimCLR ✓✗86.171.964.962.981.473.658.466.8 ✗✓96.590.179.075.594.886.573.379.1 ✓97.392.982.280.496.790.875.681.1 DINO ✓✗95.187.078.376.593.682.272.778.6 ✗✓97.493.982.880.196.490.577.180.3 ✓97.594.082.881.397.392.877.781.5 Table 32|Ablation studies between augmentation from the vision domain (crop) and our masking- based strategies. We show linear probing results (AUC ↑ ) here. Vision domain’s augmentation works for sleep data, but not necessary to be added. Method CropMaskingSHHSMROS Time ChannelSleep Staging Arousal Hypopnea Ox. Desat. Sleep Staging Arousal Hypopnea Ox. Desat. SimCLR ✓ ✗95.286.779.277.194.783.875.080.2 ✗ ✗✓96.590.179.075.594.886.573.379.1 ✓ ✓97.191.983.480.496.689.779.381.5 ✗✓97.392.982.280.496.790.875.681.1 DINO ✓ ✗95.688.181.078.595.485.777.280.6 ✗ ✗✓97.493.982.880.196.490.577.180.3 ✓ ✓ 97.091.482.179.896.689.377.381.0 ✗✓97.594.082.881.397.392.877.781.5 D.2. Additional Results of Multi-Source Data Mixture Here we report detailed results comparing multi-source data mixture pre-training with single-source pre-training. We pre-train two groups of models on SHHS only, as well as on data sampled from the full pre-training corpus. For each group, we consider using 1%, 10%, 100% of the given pre-training cohorts. As shown in Tables 36, Tables 35, Tables 38, and Tables 37, across four sleep analysis tasks and three data scales, multi-source pre-training consistently outperforms single-source pre-training, over all data percentages. D.3. Jointly Scale Model and Sample Size is Needed We further examine how model capacity and pre-training sample size jointly influence downstream performance. We pre-train both OSF and the baseline SleepFM [36]with different model sizes, on different sample size, and evaluate them on four sleep analysis tasks. As shown in Fig. 15, we find that increasing model capacity can improve the utilization of additional pre-training data. In particular, while SleepFM with its original model capacity shows only marginal gains as the pre-training sample size grows, scaling its each backbone encoder to 85M parameters yields substantially larger improvements. Motivated by this observation, we adopt a ViT-85M backbone when benchmarking different pre-training strategies and pre-train on the full multi-source data. D.4. Scaling with Pre-training Computation We further study whether OSF benefits from increased pre-training computation. Specifically, we evaluate few-shot sleep staging performance across different numbers of pre-training steps using 27 OSF : On Pre-training and Scaling of Sleep Foundation Models Table 34|Ablation on normalization strategies. Per-night z-score normalization leads to better performance. Normalization SHHSMROS Sleep StagingHypopneaSleep StagingHypopnea AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ Min-Max96.390.480.676.095.886.477.162.1 Per-Night Z-Score 97.593.282.878.197.390.477.762.2 OSFSleepFM N1/N2 Wake N3 REM No ArousalArousal (a) Arousal OSFSleepFM N1/N2 Wake N3 REM No Hypopnea Hypopnea (b) Hypopnea Figure 11|UMAP visualization of learned representations. OSF produces more separable em- beddings, indicating stronger alignment between the embedding geometry and sleep event labels, including arousal and hypopnea events. models with 1M, 5M, and 85M parameters. As shown in Fig. 13, all models improve rapidly during the first10 4 steps. After that, smaller models tend to saturate, while the 85M model continues to improve with additional pre-training. These results suggest that OSF can benefit from increased pre-training computation when paired with sufficient model capacity. E. Additional Results on Handling Missing Channels OSF better handles missing channel inference problems on in-domain data corpus as well. To validate the robustness of the conclusion in Sec. 6.1, We report additional results for inference on the SHHS cohort on the four realistic missing channel settings. As shown in Table 41, across four sleep analysis tasks, OSF consistently outperforms SleepFM [36], indicating that OSF learns more robust representations under inference-time channel missingness. Additional results on pre-training with fewer channels. Here we report detailed results for pre- training only with breathing channels and ECG channel. We compare OSF against SleepFM under the same channel subset. As shown in Table 42, across both the in-domain SHHS cohort and the out-of-domain MROS cohort, on most sleep analysis tasks, OSF makes better use of the available channels and achieves higher downstream performance. F. Additional Ablation Results Additional results of ablation studies on channel masking ratio. We further ablate the channel masking ratio by pre-training two additional variants with masking ratios of 0.1 and 0.9, while 28 OSF : On Pre-training and Scaling of Sleep Foundation Models Figure 12|Scaling performance on few-shot adaptation. OSF scales on the pre-training sample size. Table 35|Detailed data scaling results (linear probing) for OSF evaluated on SHHS. All methods are trained with ViT-85M. Large model capacities or better pretraining strategies enable scalability. Pretrain Set Data Pct Sleep StagingArousalHypopneaOx. Desat. AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ single-src1% 91.879.874.967.270.065.268.562.7 single-src10% 96.390.490.284.179.474.877.671.6 single-src100% 97.593.394.189.583.178.481.475.9 multi-src1% 92.881.978.069.972.067.170.364.5 multi-src10% 96.891.691.986.380.275.478.873.1 multi-src100% 97.593.294.089.382.878.181.375.6 Scaling by Data Size Scaling by Model Size Figure 14|Scaling performance on linear probing sleep staging. OSF scales on both the model capacity and the pre-training sample size. Figure 15|Improved scaling performance of SleepFM. Increasing model capacity helps weaker pre-training methods utilize more training samples effectively. keeping all other settings unchanged. As shown in Table 39, the default ratio of 0.5 achieves the best overall linear probing performance across SHHS and MROS. This suggests that moderate channel masking provides a better balance between task difficulty and preserved input information: weak masking may make the pretext task less informative, whereas overly aggressive masking can remove too much training signal and hurt downstream transfer. 29 OSF : On Pre-training and Scaling of Sleep Foundation Models Table 36|Detailed data scaling results (linear probing) for OSF evaluated on MROS. All methods are trained with ViT-85M. Large model capacities or better pretraining strategies enable scalability. Pretrain Set Data Pct Sleep StagingArousalHypopneaOx. Desat. AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ single-src1% 88.971.375.468.763.555.374.573.5 single-src10% 94.884.584.778.371.358.778.177.6 single-src100% 97.089.592.187.475.960.980.880.4 multi-src1% 90.574.877.670.866.856.676.475.7 multi-src10% 96.187.888.482.873.960.179.879.5 multi-src100% 97.390.492.888.377.762.281.581.0 Table 37|Detailed data scaling results (linear probing) for SleepFM [36] evaluated on SHHS. All methods are trained with ViT-85M. Large model capacities or better pretraining strategies enable scalability. Pretrain Set Data Pct Sleep StagingArousalHypopneaOx. Desat. AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ single-src1% 93.784.084.276.076.371.274.468.7 single-src10% 95.187.588.181.179.174.277.671.7 single-src100% 96.290.191.885.681.576.380.074.3 multi-src1% 94.385.784.075.576.872.075.369.8 multi-src10% 95.187.788.781.978.773.877.371.3 multi-src100% 96.791.192.286.382.077.080.675.0 Table 38|Detailed data scaling results (linear probing) for SleepFM evaluated on MROS. All methods are trained with ViT-85M. Large model capacities or better pretraining strategies enable scalability. Pretrain Set Data Pct Sleep StagingArousalHypopneaOx. Desat. AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ single-src1% 91.277.280.773.969.157.978.377.9 single-src10% 92.980.582.676.570.458.378.477.9 single-src100% 96.086.889.283.474.260.080.079.6 multi-src1% 92.379.181.274.668.357.477.877.3 multi-src10% 94.482.985.078.670.958.578.378.0 multi-src100% 96.487.790.384.975.560.680.980.3 Table 39| Ablation on channel masking ratios. A moderate masking ratio works the best. Masking Ratio SHHSMROS Sleep StagingHypopneaSleep StagingHypopnea AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ 0.196.991.981.777.095.987.277.461.5 0.5 (main setting)97.593.282.878.197.390.477.762.2 0.995.488.779.574.595.385.975.060.3 30 OSF : On Pre-training and Scaling of Sleep Foundation Models Table 40|Model size scaling results (linear probing) on SHHS and MROS. Large models show better performances. Dataset ModelEnc Size Sleep StagingArousalHypopneaOx. Desat. AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ SHHS OSF ViT-1M96.290.289.282.778.172.976.770.5 ViT-5M96.991.891.886.280.275.178.572.5 ViT-85M 97.593.294.089.382.878.181.375.6 SleepFM ViT-1M93.282.685.978.078.072.776.169.7 ViT-5M93.984.489.482.679.173.177.971.8 ViT-85M 96.691.192.286.382.077.080.675.0 MROS OSF ViT-1M95.386.186.080.170.758.478.778.3 ViT-5M96.288.188.983.672.559.379.579.1 ViT-85M 97.390.492.888.377.762.281.581.0 SleepFM ViT-1M92.075.881.675.268.357.276.169.7 ViT-5M93.478.885.078.769.157.478.377.5 ViT-85M 96.487.790.384.975.560.680.980.3 Table 41|Linear probing results on SHHS under realistic missing-channel settings. OSF (DINO) is more robust to missing channels compared to existing sleep FM. Brain Resp. ECG Method Sleep StagingArousalHypopneaOx. Desat. AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ ✓✗ ✗ SleepFM 97.092.092.286.275.069.773.867.5 OSF97.593.493.788.975.970.575.369.0 ✗✓ ✓ SleepFM 86.769.187.380.382.377.580.174.5 OSF85.266.687.480.782.878.480.174.4 ✓✗ ✓ SleepFM 96.791.392.386.376.170.676.470.6 OSF97.593.394.089.276.470.976.770.8 ✗✓ ✗ SleepFM 86.668.787.079.882.878.280.174.3 OSF85.567.086.980.284.180.380.374.7 Table 42|Robustness of pre-training with fewer channels. We pre-train the two models and evaluate them using only ECG and respiratory signals, OSF better utilizes these channels. Dataset Method Sleep StagingArousalHypopneaOx. Desat. AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ AUC ↑ AUPRC ↑ SHHS SleepFM 82.161.184.477.179.574.578.172.6 OSF81.560.484.877.881.577.078.673.3 MROS SleepFM 83.259.082.776.871.358.679.778.9 OSF83.259.483.377.677.662.480.880.1 31