Paper deep dive
MarsRecon: Self-Supervised and Multimodal Surface Representations for Mars
Akshay Naik, Marius F. R. Juston, Jay Mahajan
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/22/2026, 4:45:00 AM
Summary
The paper introduces MarsRecon, a geospatially aware pipeline for learning self-supervised visual and multimodal representations from HiRISE orbital imagery of Olympus Mons. The system calibrates NASA Planetary Data System products, extracts valid georeferenced patches, and trains a masked autoencoder (Stage A) to reduce reconstruction loss. It then aligns frozen visual features with text, coordinates, and local-global context (Stage B), achieving strong image-to-text and text-to-image retrieval recall. The work establishes a reproducible foundation for Mars surface embedding and retrieval, highlighting challenges in crop geometry and geological evaluation.
Entities (10)
Relation Signals (7)
MarsRecon → focuseson → Olympus Mons
confidence 98% · The project begins with a data layer... For Olympus Mons, the project processes approximately 149,921 valid patches
MarsRecon → uses → HiRISE
confidence 98% · We present MarsRecon, a geospatially aware pipeline for learning visual and multimodal representations from HiRISE observations
MarsRecon → calibrates → NASA Planetary Data System
confidence 95% · The pipeline calibrates NASA Planetary Data System products
Stage A → implements → Masked Autoencoder (MAE)
confidence 95% · Stage A learns a Mars visual backbone without geological labels... trains a masked autoencoder on unlabeled imagery.
Stage B → aligns → image features
confidence 92% · Stage B freezes the Stage A encoder and learns a shared embedding space for image, text, coordinate, and local/global context.
Stage A → isbasedon → SatMAE
confidence 90% · The implemented Stage A path adapts a SatMAE-style MAE recipe to the Mars pipeline
Stage B → uses → T5
confidence 90% · The implementation uses frozen Stage A image features, frozen T5 text embeddings
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:High-resolution orbital imagery offers a rich record of the Martian surface, but sparse geological labels limit supervised representation learning. We present MarsRecon, a geospatially aware pipeline for learning visual and multimodal representations from HiRISE observations of Olympus Mons. The pipeline calibrates NASA Planetary Data System products, extracts valid georeferenced patches, and trains a masked autoencoder on unlabeled imagery. Increasing input resolution and filtering invalid tokens reduced held-out reconstruction loss from 0.1751 to 0.1342 in the principal Stage A model series. We then freeze the visual encoder and align its features with observation text, coordinates, and local--global image context. The strongest current local-primary model achieves image-to-text recall@10 of 0.3787, text-to-image recall@10 of 0.9161, and local-to-global recall@10 of 0.4350 on the held-out test split. These results establish a working Mars-specific pretraining and retrieval pipeline; further crop-overlap controls and downstream geological evaluations are needed to assess the broader utility of its embeddings.
Tags
Links
- Source: https://arxiv.org/abs/2609.22379v1
- Canonical: https://arxiv.org/abs/2609.22379v1
Trouble viewing inline? Open PDF directly →
Full Text
62,843 characters extracted from source content.
Expand or collapse full text
MarsRecon: Self-Supervised and Multimodal Surface Representations for Mars Marius F. R. Juston * Department of Industrial and Enterprise Systems Engineering The Grainger College of Engineering University of Illinois Urbana-Champaign Urbana, IL 61801-3080, USA mjuston2@illinois.edu Akshay Naik * Department of Electrical and Computer Engineering The Grainger College of Engineering University of Illinois Urbana-Champaign Urbana, IL 61801-3080, USA akshayn3@illinois.edu Jay Mahajan † Department of Computer Science University of Illinois Urbana-Champaign Urbana, IL 61801-3080, USA jaym2@illinois.edu * Equal contribution. † Contributed to manuscript revisions. Abstract High-resolution orbital imagery offers a rich record of the Martian surface, but sparse geological labels limit supervised representation learning. We present MarsRecon, a geospatially aware pipeline for learning visual and multimodal representations from HiRISE observations of Olympus Mons. The pipeline calibrates NASA Planetary Data System products, extracts valid georeferenced patches, and trains a masked autoencoder on unlabeled imagery. Increasing input resolution and filtering invalid tokens reduced held-out reconstruction loss from0.1751to0.1342in the principal Stage A model series. We then freeze the visual encoder and align its features with observation text, coordinates, and local–global image context. The strongest current local-primary model achieves image-to-text recall@10 of0.3787, text-to-image recall@10 of0.9161, and local-to-global recall@10 of0.4350on the held-out test split. These results establish a working Mars-specific pretraining and retrieval pipeline; further crop-overlap controls and downstream geological evaluations are needed to assess the broader utility of its embeddings. 1 Introduction and Motivation Mars exploration increasingly depends on interpreting orbital imagery at scale. HiRISE aboard the Mars Reconnaissance Orbiter provides some of the highest-resolution orbital views of another planet, resolving crater rims, dust mantling, lava-flow textures, aeolian forms, layered terrains, and other geomorphological structure [1,2]. However, learning from HiRISE is not a standard image-classification problem. Observations are long orbital strips with irregular support, inconsistent color availability, product-specific map projections, radiometric calibration terms, and nodata regions. MarsRecon frames this setting as a foundation-representation problem. Rather than depending on dense labels for every geological class, we construct a patch-level geospatial data pipeline and learn visual representations through masked autoencoding [3]. This follows geospatial foundation-model work such as SatMAE and Scale-MAE, where self-supervised transformer pretraining exploits large unlabeled remote- sensing corpora [4,5]. The long-term goal is a Mars surface embedding that supports retrieval, mapping, site analysis, and downstream probes. This Mars planetary data, due to the lack of rich environmental features on Mars compared to Earth, makes Mars more difficult to generate embeddings. While Earth could be represented as a feature rich environment with forests, desert, oceans, urban areas and more, Mars is 1 arXiv:2609.22379v1 [cs.CV] 17 Sep 2026 MarsReconPreprint essentially a planet of deserts and variations of crater, lava-flows, aeolian forms, and other geomorphological structures [1, 2]. 2 Related Work MarsRecon builds on three research threads. First, HiRISE and PDS provide the scientific data foundation for high-resolution Mars surface analysis [1,6]. Prior Mars machine-learning work has explored crater detection, landform mapping, and terrain analysis, but these supervised tasks are constrained by sparse labels and heterogeneous terrain [7,8,9,10]. Second, geospatial tooling such as TorchGeo supports reproducible spatial sampling and metadata-aware learning pipelines, but Mars requires custom coordinate, projection, and calibration handling [11]. Third, ViT, MAE, SatMAE, and Scale-MAE provide the technical basis for learning reusable features from unlabeled image patches [12,13,3,4,5]. Stage B also connects to CLIP-style contrastive learning and remote-sensing/location-language alignment models such as RemoteCLIP and GeoCLIP [14, 15, 16]. 3 Stage 0: HiRISE Data Infrastructure The project begins with a data layer that turns raw HiRISE products into valid geospatial training samples. Each observation is read with product-specific projection metadata and transformed into a common Mars IAU 2000 geographic coordinate system. Raw DN values are converted into calibrated I/F reflectance using per-product scaling and offset parameters, while nodata masks are preserved so fill regions do not become artificial signal. Patch sampling is strip-aware because HiRISE bounds often overstate usable support: orbital strips are narrow, angled, and partially empty. The pipeline tracks valid strip footprints and patch validity before producing patch records containing centroid coordinates, bounds, source metadata, valid-channel information, and nodata support. For Olympus Mons, the project processes approximately 149,921 valid patches across 422 strips, with train/validation/test splits designed to reduce geospatial leakage. 4 Methodology 4.1 Stage A: Masked Autoencoder Visual Pretraining Stage A learns a Mars visual backbone without geological labels. A calibrated HiRISE patch is split into image tokens, a subset of valid tokens is masked, and a ViT-based masked autoencoder reconstructs the missing content from visible context [3]. The encoder is the reusable component; the lightweight decoder is used only during pretraining. The implemented Stage A path adapts a SatMAE-style MAE recipe to the Mars pipeline [4]. Mars-specific handling is essential. The trainer uses valid-mask-aware token selection and reconstruction loss, filters or downweights nodata-heavy tokens, and refines validity masks from image tensors at runtime. This prevents the model from spending capacity on fill pixels and focuses the objective on real surface structure. The strongest main-line model uses256× 256image inputs, patch-token size8, mask ratio0.5, strict valid-token filtering, and a ViT-base MAE backbone. The256/p4ablation produced a lower reconstruction loss, but we do not treat it as the backbone result. Its smaller patch tokens make reconstruction easier; many masked targets can be close to constant local color 2 MarsReconPreprint or texture patches, so lower pixel loss is not by itself evidence of a more transferable geological representation. We therefore use the 256/p8 strict-validity line as the main Stage A result. 4.2 Stage B: Image–Text–Location Alignment Stage B freezes the Stage A encoder and learns a shared embedding space for image, text, coordinate, and local/global context. The implementation uses frozen Stage A image features, frozen T5 text embeddings, coordinate encoders, projection heads, and contrastive losses. A false-negative-aware loss is important because the Olympus split has only 244 unique rationale strings, causing many image patches to share the same text target. The Stage B experiments proceed from simpler controls to harder spatial alignment. B0 validates frozen- image and frozen-text caches, W&B/evaluation plumbing, and false-negative-aware retrieval. B1a-geo aligns image, text, and coordinate context. Paired local/global experiments compare an easy center-crop proxy against true expanded geographic crops. The strongest current formulation is local-primary expanded training: the local0.005 ◦ patch remains the main image/text target, while a larger crop is auxiliary global context. We frame Stage B as substantial progress rather than a closed result; the remaining question is which gains reflect learned cross-scale alignment rather than easier crop geometry. 5 Experimental Setup All experiments use the Olympus Mons HiRISE subset generated by the Stage 0 pipeline. Stage A uses held-out reconstruction loss over valid masked support as the primary quantitative metric, supported by qualitative reconstruction previews. Stage B is evaluated as retrieval: image-to-text and text-to-image recall@K, MRR-based alignment score, and local/global retrieval for paired-crop runs. 6 Results 6.1 Stage A Results Stage A moved from setup validation to a reproducible MAE training recipe. Table 1 summarizes the main ablation line: validation loss decreased from0.1751for the128/p8baseline to0.1479for256/p8with valid-mask-aware training, and then to 0.1342 with stricter token validity. Table 1: Stage A masked-autoencoder ablations. Within the mainp8line, higher resolution and stricter valid-token filtering improve held-out reconstruction loss. The256/p4row is shown as a caveated ablation because small patch tokens can make reconstruction easier without proving better downstream representation quality. RunImagePatch pxEpochsBest val lossInterpretation 128/p8 e201288200.1751Baseline p8 configuration 128/p8 e801288800.1510Longer training helped but remained above 256/p8 256/p8 validmask v12568200.1479Higher resolution plus valid-mask-aware training 256/p8 strict802568200.1342Best p8 metrics-of-record 256/p4 strict802564200.0964Lower loss but caveated by easier tiny-token reconstruction 3 MarsReconPreprint These results are best read as an engineering ablation rather than a leaderboard. The p8 improvements isolate two practical lessons: larger inputs expose more geomorphic context, and strict valid-token handling prevents nodata from becoming an easy reconstruction target. The p4 row is retained for transparency, but Stage B uses the p8 backbone to favor transferable context over tiny-patch interpolation. This choice matters because Stage B freezes the image tower. If the Stage A backbone over-optimizes a pixel task that is too local, the downstream alignment problem inherits a weaker visual representation even when reconstruction loss appears better. The selected p8 backbone is therefore a conservative research choice: it preserves enough spatial context for retrieval while still showing reliable masked reconstruction. Figure 1: Stage A reconstruction preview. Visible context, masked input, reconstruction, and composite preview show that the MAE learns coherent Martian surface structure from valid unlabeled patches. 6.2 Stage B Results Stage B results are promising but deliberately framed as in-progress research. The important result is not a single final number; it is that the complete alignment stack now trains, evaluates, and exposes the right failure modes. B0+ showed that frozen Stage A image features can align with text rationales, reaching alignment score0.5080on full validation. Table 2 shows the higher-signal B1a test runs. B1a-geo+ establishes meaningful image/text/coordinate alignment on the full 12,285-sample held-out test split. The center-crop proxy is strongest but intentionally easier; true expanded0.015 ◦ crops are harder; and local-primary0.010 ◦ is currently the strongest practical candidate. The main Stage B conclusion is not that the multimodal model is solved, but that the direction has narrowed. The system has moved from a proposed extension to a working pipeline with interpretable controls. Smaller global contexts improve local/global retrieval, but final controls are needed to separate learned cross-scale alignment from easier crop overlap. 7 Discussion The strongest completed contribution is the Stage 0 to Stage A path: calibrated geospatial data, valid patch sampling, and self-supervised visual pretraining now operate as one reproducible pipeline. Stage A demonstrates that MAE training is viable on Mars HiRISE patches and that Mars-specific valid-mask 4 MarsReconPreprint Table 2: Stage B held-out retrieval summary. Center-crop retrieval is strong but too easy. True expanded global con- text is harder. Local-primary training with smaller0.010 ◦ global context gives the most promising current local/global retrieval. RunAlignI2T@10T2I@10L2G@10Interpretation B1a geo+0.49180.36710.9068n/aImage/text/coordinatealignment baseline Center proxy0.50900.38670.92280.6189Strong but easier nested-crop con- trol True expanded 0.0150.44510.31290.90570.2402Roadmap-aligned but difficult Local-primary 0.015 lg0.250.49950.37440.89390.2121Recovers image/text quality but weak local/global retrieval Local-primary 0.010 lg0.25 final0.49810.37870.91610.4350Best practical candidate so far handling matters. The main limitation is that reconstruction loss measures the pretraining objective, not every downstream representation property; future work should add qualitative retrieval panels and downstream probes for crater, landform, or terrain tasks. Stage B extends the backbone toward a multimodal Mars embedding, but the results should be framed as active research. The alignment stack works, yet text supervision is weak because many patches share a small number of rationales, and local/global learning is sensitive to crop scale. The retrieval audit in Appendix 5 shows that the model can recover plausible local/global and text neighbors, but it also exposes generic Olympus Mons language as a recurring confound. Stage B has made considerable engineering and empirical progress, but the final research claim should remain careful until the last crop-geometry controls are complete. The most important Stage B lesson is therefore methodological. The center-crop proxy achieves strong retrieval because it preserves an easier visual relationship between local and global views; true expanded crops are harder because they ask the model to connect a local surface patch to a broader surrounding context. That drop should not be treated as failure. It exposes the central scientific confound in planetary embedding learning: a model can appear to improve either because it learns meaningful cross-scale structure or because the crop construction makes the retrieval task geometrically easier. The current0.010 ◦ local-primary results are promising because they recover much of the local/global signal while retaining a more realistic context relationship, but they should be interpreted as a hypothesis that still requires overlap-matched controls and qualitative geological retrieval panels. This interpretation also clarifies the role of Stage A in the overall project. Stage A is not merely a preprocessing step for Stage B; it is the reusable visual foundation that makes these controlled multimodal experiments possible. By freezing the p8 SatMAE backbone, the project can isolate whether improvements come from text supervision, coordinate encoding, local/global context, or crop geometry rather than from simultaneously changing the visual representation. This modularity is the strongest research position for MarsRecon: the final claim is not that a complete Mars CLIP model has been solved, but that the project has built a calibrated data pipeline, a validated self-supervised Mars backbone, and an auditable experimental framework for testing whether Mars surface embeddings become scientifically meaningful. 8 Limitations and Future Work The main limitation of the current study is scope. The experiments are centered on Olympus Mons, which is scientifically useful because it contains diverse volcanic, erosional, and aeolian terrain, but it is not a full-Mars 5 MarsReconPreprint benchmark. A stronger future evaluation should expand to additional geographic regions and enforce spatially disjoint splits at the observation or strip level. This would test whether the learned representation generalizes across terrain regimes rather than memorizing local imaging conditions, repeated HiRISE observation rationales, or region-specific textures. A second limitation is evaluation. Stage A reconstruction loss is appropriate for selecting a stable masked- autoencoder checkpoint, but it is an indirect proxy for geological usefulness. Stage B retrieval metrics are more task-oriented, but they still depend on coarse observation-level rationales and crop construction choices. The qualitative retrieval panel is a first audit of geological coherence; the stronger next step is downstream probing with crater, landform, or terrain classification labels when such labels are available. These tests would connect the learned embedding to scientific utility rather than only to self-supervised losses. Finally, the current system is strong enough to support a more ambitious next study. The infrastructure now supports calibrated HiRISE patch extraction, valid-mask-aware self-supervised pretraining, frozen- feature caching, multimodal alignment, logged ablations, and reproducible held-out evaluation. Future work can build on this foundation by adding viewing-geometry conditioning, stronger text supervision, larger geographic coverage, and comparisons against generic Earth-trained remote-sensing encoders. The central open question is no longer whether the pipeline can be built; it is which combination of Mars-specific data handling and multimodal supervision produces embeddings that planetary scientists would trust for search, mapping, and discovery. 9 Conclusion MarsRecon demonstrates a practical path toward reusable Mars surface representations from HiRISE imagery. The project first solves the geospatial data problem, then trains a self-supervised Stage A visual backbone on unlabeled Martian patches. The strongest Stage A result reduces p8 validation reconstruction loss to0.1342 through higher-resolution inputs and stricter valid-token filtering. The Stage B experiments further show that this backbone can support image–text–location alignment. Stage B remains in progress, but the current evidence is strong enough to define a focused next step: validate whether the0.010 ◦ local/global gains reflect real cross-scale learning or easier crop geometry. The broader contribution is the staged research design. MarsRecon separates the planetary representation- learning problem into auditable components: calibrated data construction, valid-mask-aware self-supervised visual learning, and multimodal alignment with explicit controls. This separation is important because Mars data introduces failure modes that are easy to hide in an end-to-end model, including nodata artifacts, repeated observation rationales, regional sampling bias, and crop-overlap shortcuts. By making those issues visible, the project establishes a stronger foundation for future work than a single opaque embedding model would provide. The evidence therefore supports three bounded claims. First, the Stage 0 pipeline produces enough valid, calibrated HiRISE patches to train modern self-supervised models reproducibly. Second, the Stage A p8 MAE backbone improves reconstruction under stricter Mars-specific validity handling and is a defensible frozen visual representation. Third, Stage B demonstrates measurable image–text–location alignment, but its strongest local/global result should be treated as an active hypothesis until crop-overlap controls and qualitative retrieval analysis confirm geological meaning. The immediate path forward is therefore clear: keep the Stage A backbone fixed as the visual foundation, complete the remaining Stage B crop-geometry controls, and add qualitative retrieval panels that show whether nearest-neighbor results are geologically meaningful. If those controls hold, MarsRecon becomes a credible starting point for downstream Mars tasks such as crater/landform probing [17,18], site search, 6 MarsReconPreprint and large-scale surface similarity mapping. Crater detection in particular was not performed due to the low availability of datasets. The human labeled and validated dataset from NASA only supports craters of size greater than 1 km; however, the HiRISE resolution is much smaller than that [17]. The other datasets are generated using machine learning [18, 19] and would need thorough validation of the datasets. References [1]Alfred S. McEwen, Eric M. Eliason, James W. Bergstrom, Nathan T. Bridges, Candice J. Hansen, W. Alan Delamere, John A. Grant, Virginia C. Gulick, Kenneth E. Herkenhoff, Laszlo Keszthelyi, et al. The high resolution imaging science experiment (HiRISE) during MRO’s primary science phase (PSP). Icarus, 205(1):2–37, 2010. [2]HiRISE Team. HiRISE — high resolution imaging science experiment.https://hirise.lpl. arizona.edu/, 2026. Accessed: 2026-02-25. [3] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoen- coders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. [4]Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David Lobell, and Stefano Ermon. SatMAE: Pre-training transformers for temporal and multi-spectral satellite imagery. In Advances in Neural Information Processing Systems, 2022. [5]Colorado J. Reed, Ritwik Gupta, Xinyu Li, Will Brockman, Christopher Funk, Jonah Clipp, Ross Candela, Mark Maloof, W. Ben Wang, Madhav Marathe, Shweta Patel, Dimitris Samaras, Lucas Joppa, Ross Girshick, and Jia Deng. Scale-MAE: A scale-aware masked autoencoder for multiscale geospatial representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023. [6]NASA PDS Imaging Node. Mars reconnaissance orbiter (MRO) volumes.https://pds-imaging. jpl.nasa.gov/volumes/mro.html, 2026. Accessed: 2026-02-25. [7] Daniel M. DeLatte, Sarah T. Crites, Nicholas Guttenberg, and Takehisa Yairi. Automated crater detection algorithms from a machine learning perspective in the convolutional neural network era. Advances in Space Research, 64(8):1615–1628, 2019. [8] Chulmin Lee. Automated crater detection on mars using deep learning. Planetary and Space Science, 170:16–28, 2019. [9]Gian Paolo Nodjoumi, Roberto Pozzobon, Francesco Sauro, and Angelo Pio Rossi. Deeplandforms: A deep learning computer vision toolset applied to a prime use case for mapping planetary skylights. Earth and Space Science, 10(1):e2022EA002278, 2023. [10]Rui Huang, Peng Ye, Yuting Xu, Peng Wang, Licheng Jiao, Yun Feng, and Xiaohua Tong. Scribble- to-map: Geomorphological mapping of multi-class martian landforms using scribble annotations and high-resolution dems. Planetary and Space Science, 270:106230, 2026. [11]Adam J. Stewart, Caleb Robinson, Isaac A. Corley, Anthony Ortiz, Juan M. Lavista Ferres, and Arindam Banerjee. TorchGeo: Deep learning with geospatial data. ACM Transactions on Spatial Algorithms and Systems, 2025. 7 MarsReconPreprint [12]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. [13] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017. [14]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021. [15]Fan Liu, Delong Chen, Zhangqingyun Guan, Xiaocong Zhou, Jialu Wu, Wenqi Zhang, Jue Wang, Peng Li, and Lizhe Zhang. RemoteCLIP: A vision language foundation model for remote sensing. IEEE Transactions on Geoscience and Remote Sensing, 62:1–16, 2024. [16]Vicente Vivanco Cepeda, Gaurav Kumar Nayak, and Mubarak Shah. GeoCLIP: CLIP-inspired alignment between locations and images for effective worldwide geo-localization. arXiv preprint arXiv:2309.16020, 2023. [17]Stuart J. Robbins and Brian M. Hynek. A new global database of Mars impact craters >1 km: 1. Database creation, properties, and parameters. Journal of Geophysical Research: Planets, 117(5):5004, 5 2012. [18] Miao Zhuo, Yiran Wang, Xiaoran Zhang, and Zeyu Chen. High-Accuracy Crater Detection for Complex Planetary Surfaces. IEEE Transactions on Geoscience and Remote Sensing, 63, 2025. [19]Gary Doran, Emily Dunkel, Steven Lu, and Kiri Wagstaff. Mars orbital image (HiRISE) labeled data set version 3.2. [20]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. [21]Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual Reasoning with a General Conditioning Layer. 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, pages 3942–3951, 9 2017. [22] Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy. Focal frequency loss for image reconstruction and synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. [23]Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxlcr, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron Courville. On the Spectral Bias of Neural Networks. 36th International Conference on Machine Learning, ICML 2019, 2019-June:9230–9239, 6 2018. [24] NASA PDS Imaging Node. Dsm map.https://hirise-pds.lpl.arizona.edu/PDS/ CATALOG/DSMAP.CAT, 2026. Accessed: 2026-02-25. 8 MarsReconPreprint [25]Sarah S. Sutton, Matthew Chojnacki, Alfred S. McEwen, Randolph L. Kirk, Colin M. Dundas, Ethan I. Schaefer, Susan J. Conway, Serina Diniega, Ganna Portyankina, Margaret E. Landis, Nicole F. Baugh, Rodney Heyd, Shane Byrne, Livio L. Tornabene, Lujendra Ojha, and Christopher W. Hamilton. Reveal- ing Active Mars with HiRISE Digital Terrain Models. Remote Sensing, 14(10):2403, 5 2022. [26] Topographic Map of Mars | USGS I Map 2782. A Custom Mars-Specific MAE Architecture Although the final Stage A backbone follows the SatMAE-style path, the project also implemented a Mars- specific MAE comparison model. This was useful for validating which parts of the problem were architectural and which parts were caused by Mars data handling. •Purpose. A Mars-specific model within the ViT/MAE family, used to compare standardized remote- sensing MAE design against Mars-oriented valid-mask handling. • Input. A multispectral HiRISE patch, a valid-pixel mask, and patch-level scale features. •Patch tokenization. Non-overlapping patches are embedded with a convolutional patch projection before the transformer blocks. • Encoder. The encoder processes visible valid tokens only. Small presets used low-dimensional encoders for smoke tests; larger presets followed a ViT-base-style configuration. •Decoder. A lighter decoder reconstructs masked patches after encoder latents are projected into decoder space. • Validity-aware masking. Patch-level valid-pixel fractions are computed from the HiRISE nodata mask. Invalid or nodata-heavy patches are excluded from visible-token selection and masked reconstruction loss. •Visible-token packing. Valid visible tokens are packed for efficient encoding and scattered back into spatial positions before reconstruction. •Scale conditioning. Tokens can receive sinusoidal scale encodings, following the motivation of scale- aware geospatial transformers [5]. •Loss. Reconstruction loss is computed only on masked pixels that are also spatially valid, reducing the influence of fill regions. B Stage A Additional Notes The256/p4run is intentionally not used as the main Stage A backbone despite its lower reconstruction loss. The smaller token patch can make the objective easier by turning many targets into locally smooth or near-constant reconstruction patches. A fair claim of superiority would require downstream retrieval, probing, or qualitative evidence beyond pixel loss. 9 MarsReconPreprint Figure 2: Caveated256/p4reconstruction preview. The lower pixel loss of this run should not be interpreted as a stronger backbone without additional representation-quality evidence, because smaller patch tokens can make the reconstruction target easier. 10 MarsReconPreprint Table 3: Stage B full metrics ledger. This table preserves the broader run history used to interpret the compact Stage B result table in the main text. Center-crop is a high-capacity/easy control, train-only augmentation is a negative result, and local-primary 0.010 ◦ runs are the strongest current practical candidates. RunNAlign I2T@10 T2I@10 GlobalText L2G@10 G2L@10 Overlap@10 B1a geo latlon12285 0.47360.34100.9289n/an/an/an/a B1a geo+ coords rff-siren12285 0.49180.36710.9068n/an/an/an/a center-crop proxy12285 0.50900.38670.9228n/a0.61890.6170n/a true expanded 0.01512285 0.44510.31290.9057n/a0.24020.2470n/a train-only augment12285 0.08200.06090.1310n/a0.24010.2447n/a dual text expanded12285 0.44630.25950.9036n/a0.22140.2109n/a local-primary 0.015 lg012285 0.48640.36130.92800.41110.19700.16960.4314 local-primary 0.015 lg0.1012285 0.49460.33740.91960.42510.20690.1853n/a local-primary 0.015 lg0.25 sched 12285 0.49950.37440.89390.43440.21210.19540.4519 local-primary 0.010 lg0.25 best12285 0.49610.36050.91410.48570.43430.43530.5394 local-primary 0.010 lg0.25 final12285 0.49810.37870.91610.49140.43500.43670.5403 local-primary 0.0125 lg0.25 best 12285 0.50410.33350.91330.45080.30580.29020.4475 local-primary 0.0125 lg0.25 final 12285 0.49820.37480.90450.46360.31130.30300.4564 local-primary 0.0125 lg0.1012285 0.49250.37220.90880.44930.30530.29300.4497 local-primary 0.010 lg0.10 best12285 0.50130.35520.90250.48080.42680.42550.5318 local-primary 0.010 lg0.10 final12285 0.49150.37510.89860.48930.42800.42730.5343 C Stage B Full Metrics Ledger D Stage B Design Context Stage B aims to turn the Stage A visual backbone into a richer Mars surface embedding. Three non-visual information sources are relevant. First, text rationales describe why a HiRISE observation was collected; this motivates frozen T5-style text embeddings [20] and image–text retrieval evaluation. Second, planetary coordinates provide location context, so Stage B includes latitude/longitude encoders rather than treating each patch as an isolated image. Third, orbital viewing geometry such as incidence, emission, and phase angle can affect observed appearance and may be incorporated through FiLM-style conditioning [21]. The Stage B experiments reported here focus on frozen image/text caches, coordinate encoders, false-negative-aware contrastive alignment, and local/global crop retrieval; viewing-geometry conditioning remains future work. E Pipeline Overview Figure 3 summarizes the MarsRecon pipeline: Stage 0 builds calibrated geospatial patches, Stage A learns a self-supervised visual backbone, and Stage B aligns the learned visual representation with non-visual context. The Stage B figure illustrates the broader research design, including components that remain under development. F Validation Details F.1 Stage A Validation Stage A evaluates whether the ViT-based masked autoencoder learns the distribution of Martian surface textures without labels. The primary metric is masked reconstruction loss over valid pixels, which is 11 MarsReconPreprint Phase 0: Data Infrastructure Stage A: Masked Autoencoder (MAE) Stage B: Contrastive Multi-Modal Alignment Independent Modality Encoders NASA PDS HiRISE RDR (Raw DNs, Metadata) COG Conversion & Geographic CRS Reproj Strip-Aware Spatial Sampler (Single Crop Iᵢ) Footprints Radiometric Calibration (I/F Reflectance, Nodata Mask) Patches Input Patch (224×224×3) Image Crop (Single Scale Iᵢ) Spectral Band Tokenization + Nodata Dropping MSE Reconstruction Loss (Masked, Valid Pixels Only) xᵢ (Target) Scale-Aware Positional Encoding S(ρ) using MAP_SCALE Random Masking (75%) ViT Encoder (Processes Visible Tokens Only) 25% Visible Tokens Insert Learnable Mask Tokens + Re-add Positional Encoding Latents (hᵢ) Vision Encoder (E V ) Pretrained MAE Encoder Transfer Pretrained Weights Lightweight ViT Decoder (Processes All Tokens) x̂ᵢ Viewing Parameters (θᵢ, θₑ, θₚ, t, Lₛ, φₛ) Planetary Location (Lat, Lon) Geometry Encoder (E G ) RFF + Periodic → MLP Scientific Rationale (LLM Expanded) Location Encoder (E L ) Spherical Harmonics → SirenNet Text Encoder (E T ) Frozen Sentence Transformer FiLM Layer Modulation via gᵢ gᵢ ∈ ℝᵈlᵢ ∈ ℝᵈtᵢ ∈ ℝᵈ vᵢ ∈ ℝᵈ Δγ, Δβ Cross-Attention Fusion zᵢ = LayerNorm(tᵢ + gate · (α Vₗ)) Keys / Values Queries / Residual Soft-Target InfoNCE KL( P(i,j) || softmax(vᵢ · zⱼ / τ) ) vᵢ Fused Context Target (zᵢ) zⱼ Figure 3: MarsRecon pipeline architecture. Stage 0 ingests NASA PDS HiRISE products, performs reprojection and radiometric calibration, and emits valid patch-level training samples. Stage A trains a masked autoencoder on valid Mars surface tokens. Stage B extends the frozen visual backbone into a multimodal embedding space using text, coordinate, and local/global context. appropriate for monitoring optimization but should not be treated as a complete representation-quality metric. Mean squared error can favor smooth reconstructions, so qualitative reconstruction previews and downstream probes remain important complements [22, 23]. F.2 Stage B Validation Stage B is evaluated as a retrieval problem in the learned embedding space. Image-to-text and text-to-image recall@Kmeasure cross-modal alignment, while MRR-based alignment score summarizes rank quality. For paired local/global experiments, local-to-global and global-to-local recall@Kmeasure whether the model retrieves the corresponding context crop. These metrics are intentionally stricter than training loss because they evaluate whether the embedding is useful for search and association rather than merely optimizing a contrastive objective. Figure 5 provides a qualitative audit of the final local-primary Stage B run. The panel was generated by exporting the final checkpoint embeddings on all 12,285 held-out test patches, computing nearest-neighbor rationale and local/global retrieval, and then selecting visually diverse examples with a mix of successes and confounds. This is important because the aggregate metrics alone hide the shape of the problem: image-to-text recall@10 is only0.3787, while text-to-image recall@10 is0.9161. That asymmetry is expected when 12,285 12 MarsReconPreprint 6. Runtime Data Retrieval & Processing 5. Valid Patch Pre-Computation 4. Sampler Splitting & Assignment 3. Verifcation, Indexing & Download Orchestration 2. Dataset Initialization 1. High-Level Execution Pipeline Triggers No Yes Yes & Exists No / Miss Yes Yes No No Triggers Yes No GeographicRandom K-FoldPercentage Triggers YesNo TriggersTriggers NoYes Any COLOR channel requested Only RED requested Color fle missing on disk YesNo Triggers Start main execution Parse Args: Target 'Olympus' vs Bounding Box, Length, Seed Initialize MarsHiRISE Dataset Initialize HiRISEGeoSampler Pass Sampler to PyTorch DataLoader Training Loop Requests Batches Dataset.plot: Apply Percentile Stretch & Save Figure MarsHiRISE.init Validate Channels: NIR, RED, BG Setup Image Normalizer via dataset_stats.json MarsHiRISEBase.init _verify: Trigger index validation RDRCUMINDEX LBL/TAB Exist? _download_index from PDS _load_index: Read Cumulative Table Apply Target text flter e.g., 'Olympus' Apply Bounding Box spatial flter _build_spatial_index Reuse Cache = True? _try_load_cache: Load GeoPackage Group _raw_index by obs_id _run_footprint_extraction: Parallel ThreadPool extracting Convex Hulls _geometry_from_footprint_result: Fallback logic: Hull BBox Corners Index Limits Build GeoDataFrame & Save to GeoPackage Download = True? _download_images _build_download_tasks: Map Missing JP2s/LBLs Asyncio / aiohttp Download Pool with Exponential Backoff Files Successfully Downloaded? _post_download_verify: Sample Check disk for valid JP2s HiRISEGeoSampler.init Validate Split Params & Resolve Size/Stride _get_split_assignments _split_cache_key: Hash Confg Seed, Method, Folds, Sizes, Overlap Cache Match? _load_cached_split: Load JSON Assignments Which Split Strategy? _extract_pair_coordinates: Reproject to Fat CRS Get Centroids Reproject to Degrees Sort Use Seeded RNG Permutation Array Cross-Validation? _kfold_split: Divide strips into K buckets. Assign Target Fold to Test, subsplit remainder to Train/Val _compute_split_assignments: Divide ordered strips strictly by fractions 80/10/10 _save_cached_split to JSON _build_valid_centers: Filter dataset down to ONLY strips assigned to current Split Loop through assigned strips Shrink strip Polygon footprint inwards by 5% to avoid edge artifacts Calculate Bounding Box of shrunken footprint Generate dense uniform grid of X/Y points across BBox based on Stride Loop through generated points Project a theoretical Patch polygon around the X/Y coordinate Shapely Intersection: Calculate area overlap between Patch and Strip Footprint Overlap > min_overlap threshold? Store X/Y/T coordinate in Valid _centers list Discard coordinate DataLoader requests Batch Sampler.iter: Randomly select N points from pre- computed _centers Convert X/Y center points into Bounding Box Slices Dataset.getitem: Receive Slices Query GeoDataFrame Spatial Index for Intersecting Strips Matches Found? Raise IndexError _load_tile: Loop over matched observations Resolve File Logic Use _COLOR.JP2 Use _RED.JP2 _load_from_jp2 Rasterio Check Overlap reproject_band: Transform from Equirectangular to Geographic Mars CRS Radiometric Calibration: I/F = DN * SCALING_FACTOR + OFFSET Clip values between 0.0 and 1.0, mask nodata with 0.0 Stack requested bands into Tensor merge_tiles: Mosaic overlapped observations using frstnonzerowins Normalize True? Apply Z-Score normalization, preserve 0.0 nodata pixels Skip Normalization Apply TorchGeo Transforms Yield Final Sample Dict: image, bounds, crs Figure 4: Dataset loader and sampler architecture. The data system connects PDS product discovery, Mars CRS/projection handling, radiometric calibration, valid-footprint estimation, and patch sampling into one trainable HiRISE dataset interface. 13 MarsReconPreprint patches share only 244 unique observation-level rationales. A text query often has many plausible image positives, but a single image can be close to several generic Olympus Mons rationales. The successful rows show that the embedding is not random: crater, lava-flow, caldera, and basal-scarp rationales often retrieve visually plausible HiRISE texture and the paired global crop is frequently ranked first. The failure rows are equally useful. They show that dominant regional words such as “Olympus Mons,” “lava,” “crater,” and “channel” can pull together semantically adjacent but geologically different samples. The takeaway is therefore bounded: Stage B has learned a useful coarse Mars search space, but not yet a patch-specific geological captioner. The next research step is to reduce this text bottleneck with stronger patch-level descriptions, harder negative mining among visually similar Olympus terrains, and overlap-matched local/global controls. G Dataset and Calibration Details The dataset layer is a major project contribution because HiRISE is not a ready-to-train machine-learning dataset. Each JP2 product has per-observation map projection metadata and must be interpreted in a Mars-specific coordinate system. We use a Mars IAU 2000 geographic CRS derived from the HiRISE RDR metadata. The global Mars ellipsoid uses equatorial radiusa = 3, 396, 190m and polar radiusb = 3, 376, 200 m [24]. Each observation records a local radius at the center latitude: R = ab q (b cos(LatP)) 2 + (a sin(LatP)) 2 . HiRISE imagery is provided as raw digital numbers rather than physical reflectance. Each product supplies scaling and offset metadata, allowing conversion to radiance factor: I/F = DN× SCALING_FACTOR + OFFSET. This calibration is applied before patch extraction so training sees physically meaningful image values rather than product-specific integer ranges. Table 4: HiRISE spectral bands used by the MarsRecon data pipeline. The RED channel provides the highest- resolution structural texture signal, while blue-green and near-infrared channels provide additional spectral context where color products are available [1, 25]. BandWavelength rangePrimary use Blue–Green400–600 nmFrost, ice, atmospheric scattering, and color context RED550–850 nmHigh-resolution geomorphology, structural texture, and topographic interpretation Near-Infrared800–1000 nmMineralogical differentiation and spectral contrast where available H Olympus Subset and Patch Validation The Olympus Mons subset was chosen to make the project computationally tractable while preserving geological diversity. Rather than downloading the full HiRISE archive, the project filters relevant products, builds valid patch records, and splits them into train/validation/test partitions. Figure 7 shows the global context of the selected observations. 14 MarsReconPreprint Local query PSP_003199_2020 Top global crop contains query 1. Impact crater and possible related lava tube collapse features 0.545 2. Possible Olympus Mons lava flow features 0.520 3. Lava fans on the flank of Olympus Mons 0.499 4. Fissure source region for channels to the south 0.464 5. Secondary vent on the northeast side of Olympus Mons 0.410 Local query PSP_003331_2005 Top global crop contains query 1. Lava fan on Olympus Mons 0.510 2. Fissure source region for channels to the south 0.478 3. Fractures east of Olympus Mons 0.457 4. Possible ice sheet cutout from old lava flow in Olympus Mons region 0.447 5. Exposed bedrock with steep slopes near 28-m diameter fresh crater 0.446 Local query PSP_003977_1940 Top global crop contains query 1. Cross-section through the base of southern Olympus Mons 0.497 2. Hummock on lower flank of Olympus Mons 0.490 3. Inner nested caldera wall at summit of Olympus Mons 0.471 4. Channels and eroded fissure in Tharsis region 0.455 5. Upper southeastern flank of Olympus Mons 0.435 Local query PSP_004966_1985 Top global crop different context 1. Olympus Mons caldera 0.482 2. Slump deposit in Olympus Mons caldera 0.451 3. Channels in Tharsis region 0.448 4. Cracked Tharsis plain and associated channels 0.435 5. Upper southeastern flank of Olympus Mons 0.434 Local query PSP_005955_1985 Top global crop contains query 1. Olympus Mons northeast flank-plains contact 0.539 2. Pangboche Crater - fresh 11-kilometer diameter crater on Olympus Mons 0.536 3. Knobby terrain east of Olympus Mons 0.429 4. Slump deposit in Olympus Mons caldera 0.403 5. Small volcanic complex in Tharsis region 0.387 Local query ESP_018824_1965 Top global crop different context 1. Channel cutting young lava flows 0.539 2. Contact between different lava types on Olympus Mons 0.467 3. High ridge in Olympus Mons aureole 0.436 4. Terrain east of Olympus Mons 0.416 5. Flows at the base of Olympus Mons 0.400 Figure 5: Authentic Stage B retrieval examples from the held-out test set. Each row shows a real local query patch, the model’s top retrieved global-context crop, and the top five retrieved rationale strings from the final local-primary 0.010 ◦ /0.025 ◦ Stage B run. Green text bars mark the true rationale when it appears in the top five; orange bars show confounding alternatives. Thumbnails are grayscale contrast-stretched for display, but the ranks and similarity scores are computed directly from exported model embeddings. 15 MarsReconPreprint Table 5: Radiometrically calibrated dataset statistics. Statistics are computed over the Olympus patch corpus after DN-to-I/F conversion. P2/P98 are the 2nd and 98th percentiles, CV is the coefficient of variation, and DRCR is (P 98 − P 2 )/μ. ChannelN pixels MeanStdMinMaxP2P98CV / DRCR Near-infrared 4.5949×10 10 0.19650.05690.00970.77000.09280.31250.290 / 1.118 RED4.9983×10 10 0.16680.03910.00970.36000.08790.24710.235 / 0.954 Blue-green4.6060×10 10 0.07230.02110.00970.31840.03890.12700.292 / 1.218 0.00.20.40.60.81.0 Calibrated I/F value 0.0 0.2 0.4 0.6 0.8 1.0 Pixel count 1e9 BLUE-GREEN | mean=0.0723 std=0.0211 (a) Blue-green 0.00.20.40.60.81.0 Calibrated I/F value 0.0 0.5 1.0 1.5 2.0 Pixel count 1e9 RED | mean=0.1668 std=0.0391 (b) RED 0.00.20.40.60.81.0 Calibrated I/F value 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Pixel count 1e8 NEAR-INFRARED | mean=0.1965 std=0.0569 (c) Near-infrared Figure 6: Radiometrically calibrated pixel distributions. The histograms summarize calibrated I/F values for the Olympus patch corpus and show why dataset-level normalization and percentile-aware visualization are useful for training and qualitative inspection. Table 6: Representative PDS metadata fields used by the data pipeline. These fields make each patch geospatially and physically interpretable rather than a detached image crop. ParameterExample valueRole in MarsRecon OBSERVATION_IDTRA_000873_2015Unique orbital capture identifier. PRODUCT_IDTRA_000873_COLORProduct-specific identifier, including color/RED distinction. OBSERVATION_START_TIME2006-10-03T12:51:33.015Timestamp for temporal and mission-phase context. MAP_PROJECTION_TYPEEQUIRECTANGULARPer-product map projection used before transformation into the common Mars geographic CRS. MAP_SCALE0.25 m/pixelSpatial resolution at the projected center latitude. CENTER_LATITUDE / LONGITUDE20.000 ◦ , 180.000 ◦ Planetocentric map center and coordinate context. INCIDENCE_ANGLE47.099670 ◦ Solar zenith angle; relevant for illumination and surface scattering. EMISSION_ANGLE0.282987 ◦ Instrument line-of-sight relative to local surface normal. PHASE_ANGLE47.047614 ◦ Sun-target-observer angle, useful for BRDF/viewing-condition analysis. LOCAL_TIME15.40208Local solar time of the observation. SOLAR_LONGITUDE115.346344 ◦ Areocentric solar longitude L s , encoding seasonal context. SCALING_FACTOR / OFFSET2.1639×10 −4 / 0.0461Linear coefficients for DN-to-I/F radiometric calibration. Patch validation is necessary because HiRISE strips are long, angled, and often surrounded by nodata regions inside their metadata bounding boxes. The pipeline therefore estimates valid data support and rejects patches that are dominated by fill pixels. I Rationale Text Analysis The HiRISE index contains aRATIONALE_DESCfield explaining why each observation was collected. The following analysis parses the official PDS fixed-width index, which enforces a 75 character limit, deduplicates repeated RED/COLOR products by observation ID, computes observation centers from latitude/longitude bounds, tokenizes the rationale text, removes non-scientific stopwords, preserves hyphenated geological terms, and assigns science themes through keyword sets tuned to HiRISE vocabulary. This analysis motivates 16 MarsReconPreprint Figure 7: HiRISE coverage and Olympus subset. Blue points show available HiRISE sample locations in the plotted range, while red points show the project subset selected around Olympus Mons. The base topographic map is from USGS Mars topography products [26]. Stage B: the text is scientifically meaningful, but it is coarse and repetitive enough that false-negative-aware contrastive learning is necessary. J Sampling and Preprocessing We use boundary-aware sampling because simple axis-aligned grids waste many candidate patches along angled HiRISE strips. Given a convex valid footprintP, an axis-aligned square patchS(c)of side lengthL, and a required footprint-overlap fraction p, valid patch centers are defined as R =c∈R 2 | Area(P ∩ S(c))≥ pL 2 . Within each horizontal strip of the valid center region, patch centers are spaced by strideσ = L(1− δ), whereδis the allowed patch-overlap fraction. For a continuous 1D strip segment of widthW, the maximum number of centers is N max = W σ + 1. The final Stage A/B experiments use patch records produced by this broader valid-support infrastructure. Training directly from JPEG2000 products is inefficient because small random windows can require decoding large internal codeblocks. The preprocessing path converts products into tiled Cloud-Optimized GeoTIFFs, enabling random patch reads that scale with the patch window rather than the full observation. 17 MarsReconPreprint 0.0300.0250.0200.0150.0100.005 Longitude (°E) 1.331e2 24.23 24.24 24.25 24.26 24.27 Latitude (°N) TRA_000838_1945 Valid: 20 | Rejected: 4 | Waste prevented: 17% Rejected (4) Valid (20) 0.0300.0250.0200.0150.0100.005 Longitude (°E) 1.331e2 24.23 24.24 24.25 24.26 24.27 Latitude (°N) TRA_000838_1945 20 sampled patches Single-strip detail grid validation Figure 8: Valid-support estimation for a HiRISE strip. The gray rectangle is the metadata bounding box, while the valid image footprint is narrower and angled. Valid-mask-aware sampling prevents the model from learning artificial nodata patterns. 18 MarsReconPreprint (-138.163°, 42.939°) nodata: 0% mean: 0.1341 max: 0.1856 (-131.718°, 20.971°) nodata: 0% mean: 0.1123 max: 0.1863 (-130.030°, 33.347°) nodata: 0% mean: 0.1744 max: 0.2868 (-131.788°, 34.554°) nodata: 0% mean: 0.1156 max: 0.2341 (-130.870°, 31.660°) nodata: 0% mean: 0.1555 max: 0.2603 (-133.094°, 17.493°) nodata: 0% mean: 0.0849 max: 0.1553 (-132.752°, 43.736°) nodata: 0% mean: 0.1336 max: 0.2256 (-131.205°, 24.685°) nodata: 0% mean: 0.1349 max: 0.2353 (-134.697°, 32.825°) nodata: 0% mean: 0.2002 max: 0.3170 (-133.649°, 24.044°) nodata: 0% mean: 0.1815 max: 0.3696 (-134.976°, 32.690°) nodata: 0% mean: 0.1847 max: 0.2799 (-137.942°, 40.385°) nodata: 0% mean: 0.1136 max: 0.1969 (-135.347°, 33.094°) nodata: 0% mean: 0.0291 max: 0.1200 (-129.397°, 30.684°) nodata: 0% mean: 0.1427 max: 0.2666 (-136.075°, 43.099°) nodata: 0% mean: 0.1797 max: 0.2891 (-129.422°, 30.859°) nodata: 0% mean: 0.1545 max: 0.2655 Sample patch thumbnails Figure 9: Example Olympus patches. Random examples from the selected subset illustrate the range of surface textures, illumination conditions, and strip geometries seen during training. 19 MarsReconPreprint What HiRISE looks at on Mars word cloud over 100,490 observation rationales (a) Full word cloud HiRISE targeting rationale Mars-disk word cloud (b) Mars-disk visual variant Figure 10: HiRISE targeting vocabulary. The word clouds summarize the most prominent terms in observation rationales after stopword filtering. They show that HiRISE text is strongly geologic rather than generic metadata, with repeated concepts such as craters, dunes, layers, polar processes, slopes, deposits, and channels. 050001000015000200002500030000 mentions crater polar impact terra planitia north south layered dunes deposits dune chasma gully slope northern gullies floor fossae planum vallis 30595 8457 6835 6390 5854 5806 5359 5169 4877 4794 3883 3702 3670 3662 3428 3208 3120 3007 2782 2506 Top single words 05001000150020002500300035004000 mentions north polar layered deposits south polar polar layered impact crater recent impact residual cap polar residual northern plains arabia terra hellas planitia crater floor dunes dubbed new impact crater wall utopia planitia terra cimmeria dune database usgs dune terra sirenum 4015 3539 3342 3218 2976 2075 2040 1775 1700 1616 1595 1524 1279 1209 1185 1122 993 909 904 884 Top two-word phrases Most-mentioned terms in HiRISE rationales (n=100,490) Figure 11: Top rationale words and phrases. The script counts filtered unigrams and adjacent token bigrams, making the text distribution auditable rather than relying only on the word-cloud visualization. 20 MarsReconPreprint 050001000015000200002500030000 rationales mentioning this feature type Crater Terra Planitia Chasma Fossae Planum Vallis Valles Mons Mensae Montes Patera Labyrinthus Colles Dorsa Sulci Cavi Tholus Mensa Rupes Scopulus Fossa Chasmata Fluctus 29485 6383 5849 3695 3005 2782 2497 2458 1941 1361 1259 588 517 470 384 357 305 258 256 225 62 49 39 12 Mars geographic feature types named in HiRISE rationales (a) Named geographic feature types 05000100001500020000250003000035000 observations matching this theme Impact craters Other Gullies & slopes Polar / ice / frost Layers & stratigraphy Aeolian (wind) Fluvial / channels Mineralogy / hydrated Light/dark-toned Volcanic Landing sites & rovers Glacial / periglacial Tectonic / structural Mass wasting & landslides Calibration / engineering 34460 22725 14424 12875 11710 10101 9983 3238 2724 2355 2138 1996 1122 822 495 Science themes inferred from HiRISE targeting rationales (b) Keyword-derived science themes Figure 12: From words to geological categories. Geographic-feature counts track named Mars feature types such as craters, chasmata, valles, fossae, and montes. Theme counts group rationales into science categories including impact craters, gullies/slopes, aeolian processes, polar/ice/frost, layers/stratigraphy, mineralogy, volcanic processes, tectonics, and landing-site/engineering observations. 15010050050100150 east longitude (°) 80 60 40 20 0 20 40 60 80 planetocentric latitude (°) HiRISE observations on Mars, coloured by primary science theme Calibration / engineering (239) Mass wasting & landslides (358) Tectonic / structural (568) Light/dark-toned (741) Landing sites & rovers (1,245) Glacial / periglacial (1,281) Mineralogy / hydrated (1,369) Volcanic (1,434) Layers & stratigraphy (3,527) Aeolian (wind) (5,544) Fluvial / channels (7,196) Polar / ice / frost (9,317) Gullies & slopes (10,486) Other (22,725) Impact craters (34,460) Figure 13: Observation map colored by primary science theme. Each observation is assigned a primary theme from its matched keyword list and plotted by longitude/latitude. This connects the Stage B text signal back to physical Mars geography. 21 MarsReconPreprint 50 0 50 Impact craters 34,460Other 22,725Gullies & slopes 10,486Polar / ice / frost 9,317Fluvial / channels 7,196 50 0 50 Aeolian (wind) 5,544Layers & stratigraphy 3,527Volcanic 1,434Mineralogy / hydrated 1,369Glacial / periglacial 1,281 15010050050100150 50 0 50 Landing sites & rovers 1,245 15010050050100150 Light/dark-toned 741 15010050050100150 Tectonic / structural 568 15010050050100150 Mass wasting & landslides 358 15010050050100150 Calibration / engineering 239 Where on Mars is each science theme studied? hex-density per theme Figure 14: Theme-specific Mars maps. Small multiples reduce overplotting by drawing one hex-density map per science theme, revealing where different rationale classes cluster spatially. 15010050050100150 east longitude (°) 80 60 40 20 0 20 40 60 80 planetocentric latitude (°) Dominant science theme per region of Mars (hue = theme that wins the hex; opacity = observation density) Impact craters (34,460) Other (22,725) Gullies & slopes (10,486) Polar / ice / frost (9,317) Fluvial / channels (7,196) Aeolian (wind) (5,544) Layers & stratigraphy (3,527) Volcanic (1,434) Mineralogy / hydrated (1,369) Glacial / periglacial (1,281) Landing sites & rovers (1,245) Light/dark-toned (741) Tectonic / structural (568) Mass wasting & landslides (358) Calibration / engineering (239) Figure 15: Dominant theme per Mars region. Each spatial bin is colored by the locally dominant primary theme, with opacity scaled by log observation count. This preserves a single-map view while reducing the overplotting problem of point-wise thematic maps. 22 MarsReconPreprint Impact craters 34460 Other 22725 Gullies & slopes 14424 Polar / ice / frost 12875 Layers & stratigraphy 11710 Aeolian (wind) 10101 Fluvial / channels 9983 Mineralogy / hydrated 3238 Light/dark-toned 2724 Volcanic 2355 Landing sites & rovers 2138 Glacial / periglacial 1996 Tectonic / structural 1122 Mass wasting & landslides 822 Treemap of HiRISE science themes area = number of observations (a) Theme proportions EquatorialN mid-latN polarS mid-latS polar 0 10000 20000 30000 40000 50000 observations Where on Mars does each theme cluster? themes per latitude band Impact craters Other Gullies & slopes Polar / ice / frost Layers & stratigraphy Aeolian (wind) Fluvial / channels Mineralogy / hydrated Light/dark-toned Volcanic Landing sites & rovers Glacial / periglacial Tectonic / structural Mass wasting & landslides Calibration / engineering (b) Themes by latitude band Figure 16: Theme imbalance and latitude dependence. The treemap shows the global class imbalance in rationale- derived science themes, while the stacked latitude-band plot shows that theme frequencies vary by equatorial, mid- latitude, and polar regions. This helps explain why Stage B should combine text with location rather than treating rationales as globally exchangeable labels. 23 MarsReconPreprint 024681012 2 0 2 4 6 8 Standard Rectangle Base Polygon Valid Region for Center Sample Square on Boundary 024681012 0 1 2 3 4 5 6 Chamfered Rect (Your Use Case) Base Polygon Valid Region for Center Sample Square on Boundary 23456789 0 1 2 3 4 5 6 Trapezoid Base Polygon Valid Region for Center Sample Square on Boundary 246810 0 1 2 3 4 5 6 7 8 Hexagon Base Polygon Valid Region for Center Sample Square on Boundary Valid Center Regions (Square Size: 4.0x4.0, Target Overlap: 50.0%) Figure 17: Valid-center region from overlap constraints. The green polygon shows where patch centers can be placed while maintaining the minimum required overlap with the valid strip footprint. This converts a nodata-prone bounding-box problem into a constrained sampling problem. 510152025 10 5 0 5 10 simple (stride = patch size) n = 47 | coverage = 88.2% Strip footprint Patch centres 510152025 10 5 0 5 10 optimal (patch_overlap = 0.00) n = 58 | coverage = 98.9% Strip footprint Valid-centre region Patch centres Simple vs optimal on a HiRISE-like strip (patch_overlap = 0.0) (a) 0% patch-overlap stride 510152025 10 5 0 5 10 simple (stride = patch size) n = 183 | coverage = 99.6% Strip footprint Patch centres 510152025 10 5 0 5 10 optimal (patch_overlap = 0.50) n = 204 | coverage = 100.0% Strip footprint Valid-centre region Patch centres Simple vs optimal on a HiRISE-like strip (patch_overlap = 0.5) (b) 50% patch-overlap stride Figure 18: Simple grid versus independent strip packing. The strip-aware method adapts each row to the valid center region, improving valid patch yield over rigid column-locked grids for angled orbital strips. 24 MarsReconPreprint 510152025 5 0 5 10 strip 0 ( = +23%) simple (47) optimal (58) 0.02.55.07.510.012.515.017.5 4 2 0 2 4 6 8 10 strip 1 ( = +20%) simple (41) optimal (49) 0510152025 0 5 strip 2 ( = +29%) simple (59) optimal (76) 51015202530 10 5 0 5 10 15 strip 3 ( = +25%) simple (76) optimal (95) 051015202530 5.0 2.5 0.0 2.5 5.0 7.5 strip 4 ( = +49%) simple (41) optimal (61) 05101520 0 2 4 6 strip 5 ( = +12%) simple (50) optimal (56) 2.55.07.510.012.515.017.520.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 strip 6 ( = +27%) simple (41) optimal (52) 0.02.55.07.510.012.515.017.520.0 2 0 2 4 6 strip 7 ( = +26%) simple (42) optimal (53) 510152025 10 5 0 5 10 strip 8 ( = +44%) simple (39) optimal (56) Simple (green dots) vs optimal (red crosses) across 9 synthetic strips Figure 19: Sampler robustness across strip geometries. Across synthetic angled strip footprints, the boundary-aware method consistently finds more valid patch centers than a simple grid. 25 MarsReconPreprint 510152025 10 5 0 5 10 (A) simple n = 81, coverage = 95.5% Strip footprint 510152025 10 5 0 5 10 (B) optimal (patch_overlap = 0.25) n = 100, coverage = 100.0% Strip footprint Valid-centre region 0.00.10.20.30.40.50.60.7 patch_overlap 100 200 300 400 500 600 mean patches / strip (C) Patch density vs patch_overlap optimal simple (equivalent stride) 101214161820222426 optimal vs simple gain (%) 0.0 0.5 1.0 1.5 2.0 2.5 3.0 # strips (D) Per-strip gain distribution median = 21% HiRISE sampler: simple vs optimal patch placement Figure 20: Boundary-aware patch sampling. The sampler increases valid patch density by adapting candidate centers to the valid strip footprint rather than relying on a rigid grid over the metadata bounding box. 0.10.20.30.40.50.60.70.80.9 min_overlap threshold 30 35 40 45 50 55 60 65 70 number of valid patches Sensitivity to the min_overlap threshold simple optimal (a) Minimum valid-overlap threshold 0.00.20.40.60.8 patch_overlap 0 200 400 600 800 1000 1200 number of patches Patch count vs patch_overlap optimal simple 0.00.20.40.60.8 patch_overlap 0 20 40 60 80 100 coverage of footprint (%) Coverage vs patch_overlap optimal simple How patch_overlap controls density (b) Patch-overlap stride Figure 21: Sampling sensitivity. The sweeps show how patch yield and coverage respond to stricter valid-overlap requirements and different patch-overlap strides. These curves justify treating patch sampling as an explicit design choice rather than a hidden preprocessing detail. 26 MarsReconPreprint 4. DTM to COG Conversion Pipeline3. JP2 to COG Conversion Pipeline 2. Dynamic Memory Optimization 1. CLI & Batch Orchestration No Yes No Yes NoYes Triggers YesNo Catch IOError YesNo Dispatches jp2_to_cog YesNo Dispatches img_to_cog Start CLI Entry Point Parse Args: root, workers, overwrite, skips Initialize Logging Confg convert_all: Orchestrator Skip JP2? rglob matching *.JP2 / *.jp2 Skip DTM? rglob matching *.IMG fltering for 'DTE*' Compile Task List Are there fles to process? Exit: Return 0 counts Trigger Dynamic Worker Allocation _safe_worker_count Check OS for Available System RAM Sort target fles by size & sample top 8 largest Calculate average ondisk size of sample Multiply by 5x inflation factor to estimate worst- case decompressed RAM usage Reserve 25% of Available RAM for OS/GDAL overhead Calculate max concurrent workers that ft in usable RAM Clamp fnal worker count between 1 and user workers request Spawn ProcessPoolExecutor using 'spawn' mp_context _worker_init: Bind SIGINT & attach Logger to Worker Submit Tasks to Pool Worker receives JP2 Task Overwrite = False & COG exists? Skip File Suppress Geotransform Warnings Pass 1 Open Source JP2 via Rasterio Strip compression tags to speed up intermediate write Write uncompressed bands to .tmp.tif scratch fle Build Overviews on .tmp.tif using Average Resampling Close Source & trigger Garbage Collection to release decompressed RAM footprint Pass 2 rasterio.shutil.copy Apply Predictor=2, Deflate Compression, & 512512 Tiling Embed Overviews effciently during copy to fnal .tif Delete .tmp.tif scratch fle Is Error 'Corrupt Bitstream'? Delete corrupted source JP2 fle Log Failure Worker receives IMG Task Overwrite = False & COG exists? Skip File Pass 1 Open Source PDS3 .IMG Force dtype to float32 & Set Nodata to -3.4028e+38 Write flat binary data to .tmp.tif scratch fle Build Overviews on .tmp.tif Close Source & trigger Garbage Collection Pass 2 rasterio.shutil.copy Apply Predictor=3 floatdifferencing, Deflate Compression, & 512512 Tiling Write fnal .tif and delete scratch fle Figure 22: Preprocessing workflow. JPEG2000 HiRISE products are transformed into tiled, metadata-preserving assets for efficient random patch access during training. 27