Paper deep dive
MIP Candy: A Modular PyTorch Framework for Medical Image Processing
Tianhao Fu, Yucheng Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 1:31:16 PM
Summary
MIP Candy (MIPCandy) is a modular, open-source PyTorch framework for medical image processing that bridges the gap between low-level component libraries and rigid monolithic pipelines. It features a deferred configuration mechanism called LayerT for runtime module substitution, a complete training pipeline with built-in cross-validation, deep supervision, and experiment tracking, and an extensible bundle ecosystem for pre-built models like U-Net and UNet++.
Entities (10)
Relation Signals (9)
MIP Candy → builton → PyTorch
confidence 95% · MIP Candy (MIPCandy), a freely available, PyTorch-based framework
MIP Candy → licensedunder → Apache 2.0
confidence 95% · MIPCandy is open-source under the Apache-2.0 license
MIP Candy → implements → LayerT
confidence 92% · Central to the design is LayerT, a deferred configuration mechanism
MIP Candy → includesbundle → U-Net
confidence 90% · MIPCandy ships with bundles for U-Net
MIP Candy → requiresversion → Python 3.12
confidence 90% · MIPCandy ... requires Python 3.12 or later.
MIP Candy → includesbundle → UNet
confidence 88% · MIPCandy ships with bundles for ... UNet++
MIP Candy → comparedto → nnU-Net
confidence 85% · MIP Candy is designed to combine the completeness of an end-to-end pipeline with the modularity of a component library. Like nnU-Net...
MIP Candy → comparedto → MONAI
confidence 85% · Like MONAI, every component is independently usable and replaceable.
MIP Candy → supportstrackingwith → Weights & Biases
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Medical image processing demands specialized software that handles high-dimensional volumetric data, heterogeneous file formats, and domain-specific training procedures. Existing frameworks either provide low-level components that require substantial integration effort or impose rigid, monolithic pipelines that resist modification. We present MIP Candy (MIPCandy), a freely available, PyTorch-based framework designed specifically for medical image processing. MIPCandy provides a complete, modular pipeline spanning data loading, training, inference, and evaluation, allowing researchers to obtain a fully functional process workflow by implementing a single method, $\texttt{build_network}$, while retaining fine-grained control over every component. Central to the design is $\texttt{LayerT}$, a deferred configuration mechanism that enables runtime substitution of convolution, normalization, and activation modules without subclassing. The framework further offers built-in $k$-fold cross-validation, dataset inspection with automatic region-of-interest detection, deep supervision, exponential moving average, multi-frontend experiment tracking (Weights & Biases, Notion, MLflow), training state recovery, and validation score prediction via quotient regression. An extensible bundle ecosystem provides pre-built model implementations that follow a consistent trainer--predictor pattern and integrate with the core framework without modification. MIPCandy is open-source under the Apache-2.0 license and requires Python~3.12 or later. Source code and documentation are available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2602.21033v1
- Canonical: https://arxiv.org/abs/2602.21033v1
Trouble viewing inline? Open PDF directly →
Full Text
46,466 characters extracted from source content.
Expand or collapse full text
MIP CANDY: A MODULAR PYTORCH FRAMEWORK FOR MEDICAL IMAGE PROCESSING TECHNICAL REPORT Tianhao Fu ∗ University of Toronto, Toronto, ON, Canada Vector Institute, Toronto, ON, Canada Project Neura, Toronto, ON, Canada UTMIST, Toronto, ON, Canada terry.fu@projectneura.org Yucheng Chen ∗ Project Neura, Toronto, ON, Canada Amplimit, Toronto, ON, Canada steven.chen@projectneura.org ABSTRACT Medical image processing demands specialized software that handles high-dimensional volumetric data, heterogeneous file formats, and domain-specific training procedures. Existing frameworks either provide low-level components that require substantial integration effort or impose rigid, monolithic pipelines that resist modification. We present MIP Candy (MIPCandy), a freely available, PyTorch- based framework designed specifically for medical image processing. MIPCandy provides a complete, modular pipeline spanning data loading, training, inference, and evaluation, allowing researchers to obtain a fully functional process workflow by implementing a single method—build_network— while retaining fine-grained control over every component. Central to the design isLayerT, a deferred configuration mechanism that enables runtime substitution of convolution, normalization, and activation modules without subclassing. The framework further offers built-ink-fold cross-validation, dataset inspection with automatic region-of-interest detection, deep supervision, exponential moving average, multi-frontend experiment tracking (Weights & Biases, Notion, MLflow), training state recovery, and validation score prediction via quotient regression. An extensible bundle ecosystem provides pre-built model implementations that follow a consistent trainer–predictor pattern and integrate with the core framework without modification. MIPCandy is open-source under the Apache- 2.0 license and requires Python 3.12 or later. Source code and documentation are available at https://github.com/ProjectNeura/MIPCandy. 1 Introduction Medical image segmentation—the task of assigning a semantic label to each voxel in a clinical scan—is a fundamental step in computer-aided diagnosis, treatment planning, and longitudinal disease monitoring. Unlike natural images, medical data are typically stored in domain-specific formats (NIfTI, DICOM, MHA) that encode acquisition metadata such as voxel spacing, orientation, and modality. Volumes are often three-dimensional, high-resolution, and acquired with anisotropic spacing, making both the data handling and the model training substantially more involved than in standard computer vision pipelines. Meanwhile, expert annotations are scarce and expensive, placing a premium on training strategies—cross-validation, data augmentation, deep supervision—that extract maximal information from limited labels. General-purpose deep learning frameworks such as PyTorch [Paszke et al., 2019] and TensorFlow [Abadi et al., 2016] provide the computational substrate but lack medical-imaging-specific functionality. Building a segmentation pipeline from scratch on top of these frameworks requires implementing format-aware data loading, geometry-preserving transforms, specialized loss functions, sliding window inference for large volumes, and reproducible experiment management—a significant engineering effort that is duplicated across research groups. ∗ Equal contribution. arXiv:2602.21033v1 [cs.CV] 24 Feb 2026 MIP CandyTECHNICAL REPORT Several domain-specific frameworks have been developed to address this gap, ranging from comprehensive component libraries such as MONAI [Cardoso et al., 2022] and TorchIO [Pérez-García et al., 2021] to fully automated pipelines such as nnU-Net [Isensee et al., 2021]. However, existing solutions tend toward one of two extremes: they either provide low-level components that require substantial assembly effort or impose monolithic pipelines that resist modification. We review these approaches in Section 2. We present MIP Candy (hereafter MIPCandy), a PyTorch-based framework designed to occupy the middle ground between these extremes. MIPCandy provides a complete pipeline—from data loading and dataset inspection through training and inference to evaluation—yet every component is independently usable and replaceable. A researcher can obtain a fully functional segmentation workflow by implementing a single abstract method,build_network, on top of the providedSegmentationTrainerpreset; alternatively, individual modules such as the metric functions, the visualization utilities, or the dataset classes can be adopted incrementally into an existing PyTorch codebase. The principal contributions of this work are as follows: 1.LayerT, a deferred module configuration mechanism that enables runtime substitution of convolution, normal- ization, and activation layers without subclassing (Section 4.2). 2.A hierarchical training framework with pre-configured segmentation presets, deep supervision, exponential moving average, training state recovery, and multi-frontend experiment tracking (Section 4.3). 3.A dataset inspection system that automatically computes foreground bounding boxes, class distributions, and intensity statistics, enabling region-of-interest-based patch sampling (Section 4.1). 4. Validation score prediction via quotient regression, which fits a rational function to the validation trajectory and estimates both the maximum achievable score and the optimal stopping epoch (Section 4.3). 5.An extensible bundle ecosystem that packages model architectures, trainers, and predictors into self-contained, reusable units (Section 5). MIPCandy requires Python 3.12 or later and makes deliberate use of modern language features—type aliases (PEP 613), pattern matching, theSelftype, and the@overridedecorator—to improve readability and catch errors at development time. The framework is released under the Apache-2.0 license athttps://github.com/ProjectNeura/MIPCandy. 2 Related Work Existing software for medical image segmentation can be broadly organized into three categories: general-purpose deep learning frameworks, domain-specific component libraries, and end-to-end segmentation pipelines. General-purpose frameworks. PyTorch [Paszke et al., 2019] and TensorFlow [Abadi et al., 2016] provide the foundational building blocks—automatic differentiation, GPU-accelerated tensor operations, and modular neural network layers—on which all contemporary medical imaging tools are built. Higher-level wrappers such as PyTorch Lightning [Falcon and The PyTorch Lightning team, 2019] reduce boilerplate by standardizing the training loop, checkpoint management, and distributed training. However, none of these frameworks are aware of the particularities of medical data: volumetric file formats, voxel spacing, anisotropic resolution, or the class-imbalance and small-dataset regimes that are characteristic of clinical annotations. Researchers building on these frameworks must therefore implement format-aware data loading, geometry-preserving transforms, specialized loss functions, and reproducible experiment management from scratch—an engineering effort that is duplicated across groups. Domain-specific component libraries.MONAI [Cardoso et al., 2022] is the most widely adopted medical imaging library for PyTorch. It provides a large collection of transforms (spatial, intensity, crop/pad, with both array and dictionary interfaces), network architectures, loss functions, and metrics, together with Ignite-based training engines. MONAI follows an opt-in, compositional design: individual components can be imported independently and composed with vanilla PyTorch code. This flexibility, however, comes at the cost of assembly effort. Constructing a complete training pipeline in MONAI requires the user to select and configure each component—data loaders, transform chains, network, optimizer, loss, metric, engine, and event handlers—and wire them together manually. There is no single entry point that produces a working segmentation workflow with researched defaults. TorchIO [Pérez-García et al., 2021] focuses on a narrower scope: efficient loading, preprocessing, augmentation, and patch-based sampling of medical images. It integrates well with PyTorch’sDataLoaderand supports queue-based patch extraction for large 3D volumes. TorchIO is complementary to, rather than competitive with, pipeline frameworks; it addresses data handling but does not provide training loops, experiment management, or evaluation utilities. 2 MIP CandyTECHNICAL REPORT Table 1: Feature comparison of active medical image segmentation frameworks. FeaturennU-NetMONAITorchIOMIPCandy Complete training pipeline✓–✓ One-method setup✓–✓ Modular / individually usable–✓ Custom architecture swapHardManualN/A build_network Deep supervision✓ManualN/AOne flag EMA support–ManualN/AOne flag Training state recovery✓ManualN/ABuilt-in Real-time metric visualization–Via handlersN/ABuilt-in Prediction previews–N/ABuilt-in Score prediction / ETC–✓ Multi-frontend trackingTensorBoardTensorBoardN/AWandB / Notion / MLflow Dataset inspection & ROIInternal– inspect() Patch-based sampling✓ k-fold cross-validation✓–✓ Bundle / model ecosystem–MONAI Bundles–✓ End-to-end segmentation pipelines.nnU-Net [Isensee et al., 2021] occupies the opposite end of the spectrum. Given a dataset in a prescribed format, it automatically determines the preprocessing strategy, network topology, training schedule, and post-processing, achieving state-of-the-art results on a wide range of benchmarks [Isensee et al., 2024]. This automation, however, comes at the cost of modularity. The pipeline’s components—data augmentation, architecture selection, loss function, and training loop—are tightly coupled and not designed to be used independently. Substituting a custom network architecture, loss function, or training strategy requires modifying nnU-Net’s internal code rather than composing external modules. Furthermore, the training process provides limited real-time visibility: intermediate predictions, per-epoch metric trajectories, and estimated time to completion are not surfaced to the user during training. MIST [Celaya et al., 2024] is a more recent end-to-end framework that similarly automates preprocessing and training for 3D medical image segmentation, though with a simpler, more configurable pipeline than nnU-Net. Earlier efforts. NiftyNet [Gibson et al., 2018], built on TensorFlow, was among the first open-source platforms dedicated to medical image analysis, providing configurable pipelines for segmentation, regression, and image gen- eration. DLTK [Pawlowski et al., 2017] offered reference deep learning implementations for medical imaging, and DeepNeuro [Beers et al., 2021] targeted neuroimaging workflows. All three projects are now largely unmaintained and incompatible with current versions of their underlying frameworks. Positioning. Table 1 summarizes the capabilities of the most relevant active frameworks. MIPCandy is designed to combine the completeness of an end-to-end pipeline with the modularity of a component library. Like nnU-Net, it provides a fully configured training workflow with researched defaults—a working segmentation pipeline can be obtained by implementing a single method (build_network). Like MONAI, every component is independently usable and replaceable. Unlike both, MIPCandy emphasizes training transparency: per-epoch metric curves, input–label– prediction previews, validation score prediction with estimated time to completion, and multi-frontend experiment tracking are built into the training loop rather than requiring external configuration. 3 Design Philosophy MIPCandy is guided by four design principles that together shape the framework’s API, implementation, and extension model. PyTorch-native. Every trainable component in MIPCandy is a standardnn.Module; every dataset is a standard torch.utils.data.Dataset. Loss functions, normalization layers, padding operators, and deep supervision wrap- pers are allnn.Modulesubclasses that compose with the rest of the PyTorch ecosystem without adaptation. As a consequence, any existing PyTorch utility—distributed data parallelism, automatic mixed precision,torch.compile— can be applied to MIPCandy components without modification. 3 MIP CandyTECHNICAL REPORT Opt-in and incremental. No module assumes that the rest of the framework is present. A researcher can adopt a single component—a loss function, a dataset class, a metric—into an existing codebase and later integrate additional modules as needed. Composition over inheritance.MIPCandy favors runtime configuration over class proliferation. TheLayerTmecha- nism (Section 4.2) stores a module type together with its constructor arguments and instantiates the module on demand, enabling users to swap convolution, normalization, or activation layers by passing differentLayerTinstances rather than defining new subclasses. The same compositional approach appears throughout:DeepSupervisionWrapper wraps any loss module,BinarizedDatasetwraps any supervised dataset, andTrainerToolboxbundles model, optimizer, scheduler, and criterion into a flat dataclass. Minimal API surface. The common case should require no configuration.SegmentationTrainerships with a pre-configured optimizer, scheduler, and loss that selects the appropriate variant based on the number of classes. A complete training run can be launched withtrainer.train(100); all optional keyword arguments have researched defaults. Conversely, every default is overridable via class attributes or method overrides. 4 System Architecture MIPCandy is organized into nine loosely coupled modules, summarized in Table 2. Each module can be imported and used independently; the training framework, for example, has no compile-time dependency on the evaluation module, and the metrics module depends only on PyTorch tensors. Table 2: MIPCandy module overview. ModuleResponsibility mipcandy.dataMulti-format I/O, dataset classes,k-fold cross-validation, trans- forms, dataset inspection, visualization mipcandy.layer LayerTconfiguration, device management, checkpoint I/O, WithNetwork and WithPaddingModule base classes mipcandy.training Trainer base class,TrainerToolboxdataclass, experiment management, validation score prediction mipcandy.presets SegmentationTrainerpreset with pre-configured loss, opti- mizer, scheduler, and deep supervision mipcandy.inference Predictorbase class,parse_predictantutility, file-level pre- diction and export mipcandy.evaluation Evaluatorclass,EvalResultcontainer with per-case and ag- gregate metrics mipcandy.metrics Dice-familymetrics:binary_dice, dice_similarity_coefficient, soft_dice mipcandy.frontendExperiment tracking frontends: Weights & Biases, Notion, MLflow, and hybrid combinations mipcandy.commonBuilding blocks: convolution blocks, loss functions, learning rate schedulers, quotient regression The remainder of this section describes each module in detail. 4.1 Data Pipeline Multi-format I/O. MIPCandy reads and writes medical images via SimpleITK [Yaniv et al., 2018], supporting NIfTI, MetaImage, and raster formats. Theload_image()function performs automatic format detection, optional isotropic resampling, and direct device placement. For intermediate storage,fast_save()andfast_load()use the safetensors format [Hugging Face, 2023], providing zero-copy deserialization. Dataset hierarchy.All datasets inherit from a generic base that extendstorch.utils.data.Datasetand provides device management,k-fold splitting, and a path-export interface. Key implementations includeNNUNetDataset (nnU-Net raw format with multimodal support),BinarizedDataset(multiclass-to-binary wrapper), and composition utilities for merging datasets. Every dataset exposes afold()method fork-fold cross-validation with configurable splitting strategies. 4 MIP CandyTECHNICAL REPORT Dataset inspection. The inspect() function scans a supervised dataset and records per-case foreground bounding boxes, class distributions, and intensity statistics. From these annotations the framework computes a statistical foreground shape and derives a region-of-interest (ROI) shape for patch-based training.RandomROIDatasetsamples random patches with configurable foreground oversampling (default: 33% of patches contain foreground). 4.2 LayerT Configuration System Neural network architectures are typically parameterized by the choice of convolution, normalization, and activation layers. The standard approaches to making these choices configurable are either to accept many constructor arguments or to require subclassing for each combination. Both scale poorly: a network that supports 2D and 3D convolutions, batch and group normalization, and multiple activations would need2× 2× ksubclasses under an inheritance-based approach. LayerTsolves this by storing a module type together with its constructor keyword arguments as a lightweight descriptor. The module is instantiated only whenassemble()is called, at which point positional and keyword arguments are merged with the stored defaults: from mipcandy.layer import LayerT from torch import n # Define layer configurations conv = LayerT(n.Conv2d) norm = LayerT(n.BatchNorm2d, num_features="in_ch") act = LayerT(n.ReLU, inplace=True) # Instantiate at build time conv_module = conv.assemble(64, 128, 3, padding=1) # n.Conv2d(64, 128, 3, padding=1) norm_module = norm.assemble(in_ch=128) # n.BatchNorm2d(128) act_module = act.assemble() # n.ReLU(inplace=True) The string"in_ch"acts as a deferred parameter: it is resolved to the integer value passed toassemble(), allowing a single descriptor to adapt to different channel counts. MIPCandy usesLayerTpervasively; for example,ConvBlock2d acceptsLayerTarguments for convolution, normalization, and activation, with pre-configured defaults that can be overridden at construction time: from mipcandy.common import ConvBlock2d from mipcandy.layer import LayerT from torch import n # Default: Conv2d + BatchNorm2d + ReLU block = ConvBlock2d(64, 128, 3, padding=1) # Custom: Conv2d + GroupNorm + GELU block = ConvBlock2d( 64, 128, 3, padding=1, norm=LayerT(n.GroupNorm, num_groups=8, num_channels="in_ch"), act=LayerT(n.GELU), ) 4.3 Training Framework Trainer and TrainerToolbox.TheTrainerbase class manages the training lifecycle. Training state is encapsulated in aTrainerToolboxdataclass that bundles the model, optimizer, scheduler, criterion, and an optional EMA [Polyak and Juditsky, 1992] model. The toolbox is constructed from builder methods (build_network,build_optimizer, etc.) that subclasses override to customize each component. Each training run produces a timestamped experiment folder containing checkpoints, per-epoch metrics (CSV), progress plots, log files, and worst-case prediction previews (see Section 6). Before the first epoch, a sanity check validates the output shape and reports MACs and parameter count. Training state is serialized every epoch, enabling seamless recovery after interruptions. SegmentationTrainer preset.SegmentationTrainerextendsTrainerwith pre-configured defaults: a combined Dice–cross-entropy loss [Sudre et al., 2017] that selects the binary or multiclass variant automatically, SGD with 5 MIP CandyTECHNICAL REPORT momentum 0.99 and Nesterov acceleration, a polynomial learning rate scheduler, and gradient clipping. When the deep_supervisionflag is set, the criterion is wrapped in aDeepSupervisionWrapper[Lee et al., 2015] with auto-computed weights w i = 2 −i . EMA via PyTorch’s AveragedModel can be enabled with a single flag. Validation score.MIPCandy defines the validation score as the negated combined loss:s =−L val . This convention maps every loss function to a unified “higher is better” scale, so that best-checkpoint selection, early stopping, and score prediction all use a single comparison direction (s new > s best ) regardless of the underlying criterion. The framework then fits a quotient regression model—a rational functionP(x)/Q(x)—to the validation score trajectory, estimating the maximum achievable score and the epoch at which it will be reached (ETC). Frontend integrations. Experiment tracking uses a pluggableFrontendprotocol; shipped implementations cover Weights & Biases [Biewald, 2020], Notion, and MLflow [Zaharia et al., 2018], with a factory for combining multiple frontends. The visual aspects of training transparency—console output, metric plots, prediction previews, and frontend screenshots— are presented in Section 6. 4.4 Inference and Evaluation ThePredictorclass mirrors the trainer’sWithNetworkinterface: the user implementsbuild_network()and the framework handles lazy checkpoint loading, device placement, and padding. A unifiedparse_predictant() function accepts file paths, directories, tensors, or datasets, normalizing them into a common format. Predictors support single-image, batch, and file-level output (.png for 2D, .mha for 3D). TheEvaluatorclass accepts arbitrary metric functions and produces anEvalResultcontainer with per-case and aggregate scores, supporting evaluation from datasets, raw tensors, or end-to-end predict-and-evaluate workflows. MIP- Candy provides Dice-family metrics—binary_dice,dice_similarity_coefficient, andsoft_dice—covering boolean, one-hot, and soft-probability formats. The same functions serve dual roles as both loss components and evaluation metrics. 5 Bundle Ecosystem While the core framework provides the infrastructure for training, inference, and evaluation, specific network archi- tectures and their associated configurations are distributed as bundles—self-contained packages that plug into the framework without modifying it. Bundle structure. Each bundle follows a consistent three-file pattern: •Model: annn.Modulesubclass implementing the architecture, plus builder functions (make_unet2d, make_unet3d, etc.) that construct common configurations. •Trainer: a class extendingSegmentationTrainerthat overridesbuild_network()(and optionally build_padding_module(), build_optimizer(), or backward()). • Predictor: a class extending Predictor that overrides build_network(). The only mandatory override isbuild_network(), which receives the shape of a single input tensor and returns annn.Module. All other training infrastructure—loss, optimizer, scheduler, checkpointing, metric tracking, deep supervision—is inherited from the preset. Integration.Bundles depend on the core framework through its public API and useLayerT, presets, and data pipeline classes directly. No monkey-patching or registration is required. Bundle-specific behavior (e.g., custom normalization selection based on batch size, architecture-specific deep supervision) is expressed through standard method overrides. Extensibility. The bundle mechanism is not limited to model architectures. Augmentation pipelines, loss functions, and task-specific workflows can all be packaged as bundles. At the time of writing, MIPCandy ships with bundles for U-Net [Ronneberger et al., 2015], UNet++ [Zhou et al., 2018], V-Net [Milletari et al., 2016], CMUNeXt [Tang et al., 2024], MedNeXt [Roy et al., 2023], and UNETR [Hatamizadeh et al., 2022], covering both 2D and 3D segmentation tasks. 6 MIP CandyTECHNICAL REPORT (a) Combined loss and validation score.(b) Validation score trajectory. Figure 1: Training progress plots automatically generated by MIPCandy during a U-Net training run on the PH2 dermoscopy dataset. The validation score is the negated combined loss (Section 4.3); higher values indicate better performance. 6 Training Transparency A recurring frustration in medical image segmentation research is the opacity of the training process. Many frameworks report only a final score after training completes, leaving the researcher with little insight into how the model evolved, which cases are problematic, or whether training should be stopped early. MIPCandy treats training visibility as a first-class design goal: every training run automatically produces a rich set of artifacts that allow the researcher to monitor, diagnose, and communicate results without additional code. Console output and metric reporting. During training, MIPCandy prints a structured summary after each epoch via the Rich library [McGugan, 2019], including the current epoch, all tracked losses, validation scores, learning rate, epoch duration, and—when available—the estimated time of completion (ETC). After each validation pass, a per-case metric table is displayed, highlighting the worst-performing case so that the researcher can immediately identify failure modes. Appendix A (Figure 5) shows a representative console screenshot when resuming a previously interrupted training run, illustrating both the recovery mechanism and the per-epoch metric reporting. Training progress visualization. At the end of each epoch, MIPCandy updates a set of metric curve plots saved to the experiment folder. These include combined loss and validation score on a single progress plot, as well as individual plots for each loss component (Dice, cross-entropy), per-class Dice scores, learning rate schedule, and epoch duration. Researchers can monitor these plots in real time via any file viewer or integrate them into slide decks and lab notebooks. Figure 1 shows the progress plot and validation score curve from a U-Net trained on PH2 for 90 epochs. Prediction previews and worst-case tracking. After each validation epoch, the framework identifies the worst- performing validation case (by validation score) and saves a set of preview images: the raw input, the ground-truth label, the model’s prediction, and two overlay composites—the expected overlay (ground truth superimposed on the input) and the actual overlay (prediction superimposed on the input). By always displaying the worst case rather than a random or cherry-picked example, this mechanism ensures that the researcher’s attention is directed to the most informative failure mode. Figure 2 shows these previews from a 2D skin lesion segmentation experiment. For 3D volumes,visualize3d()renders the label and prediction as interactive PyVista [Sullivan and Kaszynski, 2019] meshes with automatic downsampling, as shown in Figure 3. Validation score prediction. After a configurable warm-up period (default: 20 epochs), MIPCandy fits a quotient regression model to the validation score trajectory and extrapolates the maximum achievable score and the epoch at which it will be reached. From these estimates the framework computes an ETC (Estimated Time of Completion) that is displayed after each validation epoch. This allows researchers to make informed decisions about early stopping, hyperparameter adjustment, or resource allocation without waiting for the full training run to complete. A full console screenshot illustrating both the recovery mechanism and the per-epoch output is provided in Appendix A. 7 MIP CandyTECHNICAL REPORT (a) Input image.(b) Expected (GT overlay).(c) Actual (prediction overlay). Figure 2: Worst-case prediction previews automatically saved during training. The framework identifies the validation case with the lowest score and generates overlays comparing the ground truth (b) and model prediction (c) against the input image (a). This example is from a U-Net trained on PH2. (a) BraTS ground-truth label (4 classes). (b) PANTHER [Betancourt Tarifa et al., 2025] predicted seg- mentation. Figure 3: 3D volume previews rendered via PyVista. MIPCandy automatically generates 3D visualizations of labels and predictions for volumetric segmentation tasks. Frontend integrations.For team-level experiment management, MIPCandy integrates with external tracking services via a lightweightFrontendprotocol. Shipped frontends include Weights & Biases [Biewald, 2020], Notion, and MLflow [Zaharia et al., 2018], and thecreate_hybrid_frontend()factory allows simultaneous logging to multiple services. Figure 4 shows a Notion database populated by MIPCandy, providing a persistent, shareable experiment ledger. Training state recovery.Long-running 3D training jobs are frequently interrupted by hardware failures, preemption, or resource limits. MIPCandy serializes the full training state—optimizer, scheduler, criterion state dictionaries, and a state orb recording epoch, best score, and all training arguments—at every epoch. Training can be resumed via recover_from()followed bycontinue_training(), restoring the exact state and continuing from the interrupted epoch (Figure 5). 7 Case Studies This section demonstrates MIPCandy workflows on representative segmentation tasks, illustrating both the minimal code required and the artifacts produced by the framework. 8 MIP CandyTECHNICAL REPORT Figure 4: Notion frontend integration. MIPCandy automatically logs experiment metadata, progress, and scores to a Notion database. 7.1 2D Skin Lesion Segmentation The following script performs binary segmentation on the PH2 dermoscopy dataset [Mendonça et al., 2013] using a U-Net bundle. The complete pipeline—data loading,k-fold splitting, trainer configuration, and training—requires 8 lines of code: import torch from torch.utils.data import DataLoader from mipcandy.data import NNUNetDataset from mipcandy_bundles.unet import UNetTrainer device = "cuda" if torch.cuda.is_available() else "cpu" train, val = NNUNetDataset(folder="Dataset501_PH2", split="Tr").fold(fold=0) trainer = UNetTrainer( "experiments", DataLoader(train, batch_size=2, shuffle=True), DataLoader(val, batch_size=1), device=device, ) trainer.num_classes = 1 trainer.train(100) Without a bundle, the same workflow requires implementing a single method on SegmentationTrainer: from typing import override from torch import n from mipcandy.presets import SegmentationTrainer class MyTrainer(SegmentationTrainer): @override def build_network(self, example_shape: tuple[int, ...]) -> n.Module: from mipcandy_bundles.unet import make_unet2d return make_unet2d(example_shape[0], self.num_classes) Upon completion, the experiment folder contains model checkpoints, per-epoch metrics (CSV), all training curve plots shown in Section 6, and worst-case preview images. Evaluation on a held-out test set is equally concise: from mipcandy.evaluation import Evaluator from mipcandy.metrics import binary_dice from mipcandy_bundles.unet import UNetPredictor predictor = UNetPredictor("experiments/UNetTrainer/20240901-1234", example_shape=(3, 384, 384), device="cuda") 9 MIP CandyTECHNICAL REPORT evaluator = Evaluator(binary_dice) result = evaluator.predict_and_evaluate("test_images/", "test_labels/", predictor) print(result.mean_metrics) 7.2 3D Volumetric Segmentation For 3D tasks, MIPCandy’s dataset inspection system automates the determination of patch shapes and foreground sampling rates. The following script trains a multiclass 3D segmentation model on the BraTS 2021 brain tumor dataset [Baid et al., 2021] with deep supervision and ROI-based patch sampling: import torch from torch.utils.data import DataLoader from mipcandy.data import NNUNetDataset from mipcandy.data.inspection import inspect, RandomROIDataset from mipcandy_bundles.unet import UNetTrainer device = "cuda" if torch.cuda.is_available() else "cpu" dataset = NNUNetDataset(folder="Dataset320_BRaTS", split="Tr") train_full, val_full = dataset.fold(fold=0) # Inspect dataset to compute ROI shape and class distribution annotations = inspect(train_full) train = RandomROIDataset(annotations, batch_size=2) val = RandomROIDataset( inspect(val_full), batch_size=1, oversample_rate=0 ) trainer = UNetTrainer( "experiments", DataLoader(train, batch_size=2, shuffle=True), DataLoader(val, batch_size=1), device=device, ) trainer.num_dims = 3 trainer.num_classes = 4 trainer.deep_supervision = True trainer.train(200, early_stop_tolerance=20) Theinspect()call scans the training set to compute per-case foreground bounding boxes, class distributions, and intensity statistics.RandomROIDatasetuses these annotations to sample patches of a statistically determined shape, with 33% of patches forced to contain foreground voxels. Deep supervision is enabled by setting a single flag; the trainer automatically wraps the loss function, computes scale-dependent weights, and generates multi-resolution targets. The framework produces 3D preview renderings (Figure 3), per-class Dice curves, and all other transparency artifacts described in Section 6. 8 Conclusion We have presented MIPCandy, a modular, PyTorch-native framework for medical image segmentation that prioritizes four qualities: flexibility in swapping components, transparency during training, usability through minimal-code setup, and extensibility via a bundle ecosystem. The framework provides a complete pipeline from data loading through training and inference to evaluation. A functional segmentation workflow can be obtained by implementing a single method—build_network—while all infrastructure (loss selection, optimizer configuration, checkpointing, metric tracking, deep supervision, EMA, validation score prediction, and experiment tracking) is handled by the framework with researched defaults. At the same time, every component is independently usable and replaceable, allowing incremental adoption into existing PyTorch codebases. The key technical contributions—LayerTfor compositional module configuration, built-in training transparency with worst-case tracking and validation score prediction, dataset inspection with ROI-based patch sampling, and the bundle ecosystem—address practical pain points in medical image segmentation research. Unlike fully automated pipelines that treat training as a black box, MIPCandy ensures that the researcher retains full visibility into and control over every stage of the process. 10 MIP CandyTECHNICAL REPORT MIPCandy is open-source under the Apache-2.0 license and is actively developed. Future work includes expanding the metric library with surface-distance metrics (Hausdorff distance, average symmetric surface distance), adding sliding window inference for large volumes, supporting semi-supervised and self-supervised learning paradigms, and extending the bundle ecosystem with task-specific bundles for detection and registration. References Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. OSDI, 16: 265–283, 2016. M Jorge Cardoso, Wenqi Li, Richard Brown, Nic Ma, Eric Kerfoot, Yiheng Wang, Benjamin Murrey, Andriy Myronenko, Can Zhao, Dong Yang, et al. MONAI: An open-source framework for deep learning in healthcare. arXiv preprint arXiv:2211.02701, 2022. Fernando Pérez-García, Rachel Sparks, and Sébastien Ourselin. TorchIO: A Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning. Computer Methods and Programs in Biomedicine, 208:106236, 2021. Fabian Isensee, Paul F Jaeger, Simon A Kohl, Jens Petersen, and Klaus H Maier-Hein. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 18(2):203–211, 2021. William Falcon and The PyTorch Lightning team.PyTorch Lightning, 2019.URLhttps://github.com/ Lightning-AI/pytorch-lightning. Fabian Isensee, Tassilo Wald, Constantin Ulrich, Michael Baumgartner, Saikat Roy, Klaus H Maier-Hein, and Paul F Jaeger. nnU-Net revisited: A call for rigorous validation in 3D medical image segmentation. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2024, volume 15009 of Lecture Notes in Computer Science, pages 488–498. Springer, 2024. Adrian Celaya, Evan Lim, Rachel Glenn, Brayden Mi, Alex Balsells, Dawid Schellingerhout, Tucker Netherton, Caroline Chung, Beatrice Riviere, and David Fuentes. MIST: A simple and scalable end-to-end 3D medical imaging segmentation framework. arXiv preprint arXiv:2407.21343, 2024. Eli Gibson, Wenqi Li, Carole Sudre, Lucas Fidon, Dzhoshkun I Shakir, Guotai Wang, Zach Eaton-Rosen, Robert Gray, Tom Doel, Yipeng Hu, et al. NiftyNet: a deep-learning platform for medical imaging. Computer Methods and Programs in Biomedicine, 158:113–122, 2018. Nick Pawlowski, Sofia Ira Ktena, Matthew CH Lee, Bernhard Kainz, Daniel Rueckert, Ben Glocker, and Martin Rajchl. DLTK: State of the art reference implementations for deep learning on medical images. arXiv preprint arXiv:1711.06853, 2017. Andrew Beers, James Brown, Ken Chang, Katharina Hoebel, Elizabeth Gerstner, Bruce Rosen, and Jayashree Kalpathy- Cramer. DeepNeuro: an open-source deep learning toolbox for neuroimaging. Neuroinformatics, 19:127–140, 2021. Ziv Yaniv, Bradley C Lowekamp, Hans J Johnson, and Richard Beare. SimpleITK image-analysis notebooks: a collaborative environment for education and reproducible research. Journal of Digital Imaging, 31:290–303, 2018. Hugging Face. Safetensors: A simple, safe and fast file format for storing tensors, 2023. URLhttps://github.com/ huggingface/safetensors. Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838–855, 1992. Carole H Sudre, Wenqi Li, Tom Vercauteren, Sebastien Ourselin, and M Jorge Cardoso. Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pages 240–248. Springer, 2017. Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply-supervised nets. In Artificial Intelligence and Statistics, pages 562–570, 2015. Lukas Biewald. Experiment tracking with Weights and Biases, 2020. URL https://w.wandb.com/. Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, et al. Accelerating the machine learning lifecycle with MLflow. IEEE Data Engineering Bulletin, 41(4):39–45, 2018. 11 MIP CandyTECHNICAL REPORT Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmenta- tion. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 234–241. Springer, 2015. Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. UNet++: A nested U-Net architecture for medical image segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, volume 11045 of Lecture Notes in Computer Science, pages 3–11. Springer, 2018. doi:10.1007/978-3-030-00889-5_1. Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-Net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 Fourth International Conference on 3D Vision (3DV), pages 565–571. IEEE, 2016. doi:10.1109/3DV.2016.79. Fenghe Tang, Jianrui Ding, Quan Quan, Lingtao Wang, Chunping Ning, and S. Kevin Zhou. CMUNeXt: An efficient medical image segmentation network based on large kernel and skip fusion. In 2024 IEEE International Symposium on Biomedical Imaging (ISBI), pages 1–5. IEEE, 2024. doi:10.1109/ISBI56570.2024.10635609. Saikat Roy, Gregor Köhler, Constantin Ulrich, Michael Baumgartner, Jens Petersen, Fabian Isensee, Paul F. Jäger, and Klaus H. Maier-Hein. MedNeXt: Transformer-driven scaling of convnets for medical image segmentation. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2023, volume 14223 of Lecture Notes in Computer Science, pages 405–415. Springer, 2023. doi:10.1007/978-3-031-43901-8_39. Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R. Roth, and Daguang Xu. UNETR: Transformers for 3D medical image segmentation. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1748–1758. IEEE, 2022. doi:10.1109/WACV51458.2022.00181. Will McGugan. Rich: A Python library for rich text and beautiful formatting in the terminal.https://github.com/ Textualize/rich, 2019. C. Bane Sullivan and Alexander A. Kaszynski. PyVista: 3D plotting and mesh analysis through a streamlined interface for the Visualization Toolkit (VTK). Journal of Open Source Software, 4(37):1450, 2019. doi:10.21105/joss.01450. Amparo Soeli Betancourt Tarifa, Faisal Mahmood, Uffe Bernchou, and Peter Jan Koopmans. PANTHER challenge: Public training dataset, 2025. Teresa Mendonça, Pedro M Ferreira, Jorge S Marques, André RS Marcal, and Jorge Rozeira. PH2 – a dermoscopic image database for research and benchmarking. In International Conference of the IEEE Engineering in Medicine and Biology Society, pages 5437–5440, 2013. Ujjwal Baid, Satyam Ghodasara, Suyash Mohan, Michel Bilello, Evan Calabrese, Errol Colak, Keyvan Farahani, Jayashree Kalpathy-Cramer, Felipe C Kitamura, Sarthak Pati, et al. The RSNA-ASNR-MICCAI BraTS 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107.02314, 2021. A Console Output Figure 5 shows the full console output when resuming a previously interrupted training run viarecover_from() andcontinue_training(). The framework restores the optimizer, scheduler, and training tracker from a previous checkpoint and resumes from the interrupted epoch. Each epoch produces a structured metric summary, a per-case validation table with per-class label and prediction statistics, and aggregated validation metrics. The worst-performing validation case is highlighted, and the estimated time of completion (ETC) is displayed based on quotient regression of the validation trajectory. 12 MIP CandyTECHNICAL REPORT MIPCandy Training Recovery [2026-02-2314:30:00] Training progress recovered from 20260223-14-a3f2 from epoch 8 [2026-02-2314:30:00.100000] Set to manual seed 42 [2026-02-2314:30:00.600000] Example input shape: (3, 384, 384) [2026-02-2314:30:00.700000] Building a template model to run sanity check on... [2026-02-2314:30:02.700000] Model: UNet2d [2026-02-2314:30:02.800000] MACs: 54.3 G / Params: 34.5 M [2026-02-2314:30:02.900000] Example output shape: (1, 384, 384) [2026-02-2314:30:03] Building toolbox... Training epoch 9 (0.2987)━100%-:--:--⠋Training epoch 9 (0.2987)━ [2026-02-2314:30:04.500000] Training combined loss: 0.2987 @[0.1923, 0.4123](-0.0258) [2026-02-2314:30:04.520000] Training soft dice: 0.8612 @[0.8023, 0.9067](+0.0189) [2026-02-2314:30:04.540000] Training bce loss: 0.1378 @[0.0912, 0.1923](-0.0156) Epoch 9 Training ┏━┳━┳━┳━┓ ┃Metric ┃Mean Value┃Span ┃Diff ┃ ┡━╇━╇━╇━┩ │combined loss│0.2987 │[0.1923, 0.4123]│-0.0258│ │soft dice │0.8612 │[0.8023, 0.9067]│+0.0189│ │bce loss │0.1378 │[0.0912, 0.1923]│-0.0156│ └─┴─┴─┴─┘ [2026-02-2314:30:04.560000] Epoch 9 training completed in 31.8 seconds Validating epoch 9 case 9 (0.8945)━100%-:--:--⠋Validating epoch 9 case 9 (0.8945)━ [2026-02-2314:30:36.360000] Validation score: -0.2987(+0.0258) [2026-02-2314:30:36.460000] Maximum validation score -0.2067 predicted at epoch 63 [2026-02-2314:30:36.560000] Estimated time of completion in 4156.8 seconds at 02-2315:39:53 Epoch 9 Metrics per Case ┏━┳━┳━┳━┳━┳━┳━┳━┓ ┃Case ID┃soft dice┃bce loss┃dice ┃% label class 0┃% label class 1┃% output class 0┃% output class 1┃ ┡━╇━╇━╇━╇━╇━╇━╇━┩ │1 │0.9312 │0.0756 │0.9234│0.8234 │0.1766 │0.8289 │0.1711 │ │2 │0.8945 │0.1167 │0.8812│0.7856 │0.2144 │0.7912 │0.2088 │ │3 │0.9134 │0.0889 │0.9056│0.9012 │0.0988 │0.8945 │0.1055 │ │4 │0.8678 │0.1389 │0.8523│0.8567 │0.1433 │0.8601 │0.1399 │ │5 │0.7956 │0.2023 │0.7789│0.9234 │0.0766 │0.9101 │0.0899 │ │6 │0.9423 │0.0678 │0.9334│0.7623 │0.2377 │0.7689 │0.2311 │ │7 │0.9201 │0.0812 │0.9123│0.8423 │0.1577 │0.8489 │0.1511 │ │8 │0.8834 │0.1245 │0.8712│0.8134 │0.1866 │0.8212 │0.1788 │ │9 │0.9145 │0.0856 │0.9067│0.8756 │0.1244 │0.8801 │0.1199 │ │10 │0.9023 │0.1056 │0.8945│0.8423 │0.1577 │0.8490 │0.1510 │ └─┴─┴─┴─┴─┴─┴─┴─┘ [2026-02-2314:30:36.660000] Validation worst case: 5 [2026-02-2314:30:36.760000] Validation soft dice: 0.8965 @[0.7956, 0.9423](+0.0097) [2026-02-2314:30:36.780000] Validation bce loss: 0.1087 @[0.0678, 0.2023](-0.0068) [2026-02-2314:30:36.800000] Validation dice: 0.8860 @[0.7789, 0.9334](+0.0098) [2026-02-2314:30:36.820000] Validation % label class 0: 0.8475 @[0.7623, 0.9234](+0.0000) [2026-02-2314:30:36.840000] Validation % label class 1: 0.1525 @[0.0766, 0.2377](+0.0000) [2026-02-2314:30:36.860000] Validation % output class 0: 0.8453 @[0.7689, 0.9101](-0.0056) [2026-02-2314:30:36.880000] Validation % output class 1: 0.1547 @[0.0899, 0.2311](+0.0056) Epoch 9 Validation ┏━┳━┳━┳━┓ ┃Metric ┃Mean Value┃Span ┃Diff ┃ ┡━╇━╇━╇━┩ │soft dice │0.8965 │[0.7956, 0.9423]│+0.0097│ │bce loss │0.1087 │[0.0678, 0.2023]│-0.0068│ │dice │0.8860 │[0.7789, 0.9334]│+0.0098│ │% label class 0 │0.8475 │[0.7623, 0.9234]│+0.0000│ │% label class 1 │0.1525 │[0.0766, 0.2377]│+0.0000│ │% output class 0│0.8453 │[0.7689, 0.9101]│-0.0056│ │% output class 1│0.1547 │[0.0899, 0.2311]│+0.0056│ └─┴─┴─┴─┘ [2026-02-2314:30:36.900000] ======== Best checkpoint updated (-0.3245 -> -0.2987) ======== [2026-02-2314:30:37] Epoch 9 completed in 44.7 seconds [2026-02-2314:30:37.100000] =============== Best Validation Score -0.2987 =============== Figure 5: Console interface during training state recovery. The output shows a single epoch after recovery: sanity check, training metrics with a structured summary table, per-case validation metrics with per-class statistics, score prediction with ETC, and checkpoint management. 13