Paper deep dive
Sim2Act: Robust Simulation-to-Decision Learning via Adversarial Calibration and Group-Relative Perturbation
Hongyu Cao, Jinghan Zhang, Kunpeng Liu, Dongjie Wang, Feng Xia, Haifeng Chen, Xiaohua Hu, Yanjie Fu
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/13/2026, 12:58:08 AM
Summary
Sim2Act is a robust simulation-to-decision learning framework designed to address simulator bias and policy instability in mission-critical domains like supply chains. It introduces two key innovations: adversarial simulator calibration, which re-weights prediction errors in decision-critical regions to improve fidelity, and group-relative perturbation, which stabilizes policy learning under uncertainty without causing policy collapse.
Entities (6)
Relation Signals (3)
Sim2Act → utilizes → Adversarial Calibration
confidence 100% · We propose the Sim2Act framework to address the two issues... we introduce adversarial simulator calibration.
Sim2Act → utilizes → Group-Relative Perturbation
confidence 100% · we adopt group-relative decision-making perturbation.
Sim2Act → evaluatedon → DataCo
confidence 95% · Extensive experiments on the DataCo, GlobalStore, and OAS benchmarks show that Sim2Act consistently outperforms
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Simulation-to-decision learning enables safe policy training in digital environments without risking real-world deployment, and has become essential in mission-critical domains such as supply chains and industrial systems. However, simulators learned from noisy or biased real-world data often exhibit prediction errors in decision-critical regions, leading to unstable action ranking and unreliable policies. Existing approaches either focus on improving average simulation fidelity or adopt conservative regularization, which may cause policy collapse by discarding high-risk high-reward actions. We propose Sim2Act, a robust simulation-to-decision framework that addresses both simulator and policy robustness. First, we introduce an adversarial calibration mechanism that re-weights simulation errors in decision-critical state-action pairs to align surrogate fidelity with downstream decision impact. Second, we develop a group-relative perturbation strategy that stabilizes policy learning under simulator uncertainty without enforcing overly pessimistic constraints. Extensive experiments on multiple supply chain benchmarks demonstrate improved simulation robustness and more stable decision performance under structured and unstructured perturbations.
Tags
Links
- Source: https://arxiv.org/abs/2603.09053v1
- Canonical: https://arxiv.org/abs/2603.09053v1
Trouble viewing inline? Open PDF directly →
Full Text
47,246 characters extracted from source content.
Expand or collapse full text
Sim2Act: Robust Simulation-to-Decision Learning via Adversarial Calibration and Group-relative Perturbation Hongyu Cao1 Jinghan Zhang2 Kunpeng Liu2 Dongjie Wang3 Feng Xia4 Haifeng Chen5 Xiaohua Hu6 Yanjie Fu1 1Arizona State University 2Clemson University 3University of Kansas 4RMIT University 5NEC Laboratories America 6Drexel University hongyuca@asu.edu, jinghaz@clemson.edu, kunpenl@clemson.edu, wangdongjie100@gmail.com, f.xia@ieee.org, haifeng@nec-labs.com, xh29@drexel.edu, yanjie.fu@asu.edu Abstract In digital twins, simulation-to-decision enables safe decision learning in digital worlds without risking real-world deployments, thus, has become a cornerstone in mission-critical domains (e.g., supply chains and industrial systems). However, real-world system data and labels are often noisy, biased, and incomplete, thus, results into unstable decisions when simulators learned from such data are used for policy training. Existing methods often fall short: 1) surrogate simulation models tend to be biased particularly in decision-critical regions in which data labels are sparse or biased; 2) when interacting with biased surrogate simulators, policy learning is sensitive to perturbations. It is critical to investigate how to robustify both simulation and decisions. To this end, we propose a robust learning framework to advance 1) simulation fidelity by adversarial calibration, 2) policy robustness with group-relative perturbations. Our solution enables non-disruptive robustness that is stable under perturbation while preserving decision performance. We present extensive experiments on multiple supply chain benchmarks (DataCo, GlobalStore, and OAS) to demonstrate the simulation and decision robustness of our method in perturbation settings. 1 Introduction In many high-stakes applications such as supply chain management and complex industrial operations, decision-makers are increasingly trained using learned surrogate simulators to avoid costly, risky, or privacy-constrained interactions with real environments (Agrawal, 2025). This simulation-to-decision paradigm firstly develops a surrogate simulator of the environment: given an action and output next state and reward, then learns optimized policies by interacting with the simulator, among which the Sim2Dec pipeline (Bai et al., 2025) is a recent instantiation. Investigating robustness is critical because simulators are often learned from real world noisy or imperfect data . These data imperfections create an issue: while a simulator might look accurate (e.g., RMSE, MAE) on average in terms of reward prediction, it can exhibit significant biases on rewards over actions; that is, predicted rewards of common actions with large training data are accurate, yet predicted rewards of rare risky actions are less accurate. Such uneven reward predictive accuracies are dangerous for decision-making. Because policies choose actions by comparing predicted rewards of one action against another, small reward prediction errors on some actions can flip the ranking orders of all actions, leading to unstable or unsafe decisions. In this paper, we study the problem of how to robustify both simulation (i.e., surrogate environment modeling) and decision-making (i.e., policy learning). Exiting studies of robust decision making are two fold: 1) from the simulator perspective: how to improve simulation fidelity; 2) from the decision making perspective, how to regularize decision policies under uncertainty. Despite these advances, two fundamental challenges remain unresolved. On the simulator side, Sim2Dec and other simulation-based methods aim to enhance average predictive accuracy through optimizing simulation fidelity (Barykin et al., 2020), ensembles (Correia et al., 2023), physics-informed planning (Bai et al., 2025), or introducing uncertainty and probabilistic reward (Atanassov and Dimov, 2008). Most simulators reduce average errors in order to be mostly right across all scenarios. However, in certain state-action pairs, which we call decision-critical regions, small simulation errors can fully distort action ranking orders and significantly reduce policy performance (Fonteneau et al., 2013; Gregor et al., 2019; Zhao et al., 2021). Because the action rewards in simulation are not aligned with true rewards of actions in those decision-critical regions, a tiny 1%1\% error in reward prediction result in much more than a 1%1\% error in decision paths (e.g, a rank reversal). Issue 1 (Simulation-Action Unalignment): How can we reduce simulation errors in decision-critical regions while preserving average accuracies, so that small mispredictions can not flip the entire action ranking orders? On the decision-maker side, existing model-based and offline reinforcement learning methods (Huang, 2022; Liu et al., 2021) typically rely on adversarial perturbations (Pinto et al., 2017; Zhang et al., 2021, 2025; Liu et al., 2024) or conservative regularization schemes (Derman and Mannor, 2020; Yang et al., 2022; Li et al., 2025) to improve robustness. However, this often results in “policy collapse”, where the model becomes too afraid of potential errors that it abandons not just high-risk low-reward, but also high-risk high-reward opportunities entirely. Instead of treating every uncertainty as a threat, we need to distinguish between unacceptable risks and recoverable errors, maintaining robustness without sacrificing the pursuit of high rewards. Issue 2 (Seeing Every Uncertainty as Threat): How can we protect a policy against prediction errors without forcing it to become overly timid and discard all high-risk high-reward actions? Our Perspective: We propose the Sim2Act framework to address the two issues. To address Issue 1 (simulation-action unalignment), we introduce adversarial simulator calibration. Instead of optimizing average simulation accuracy, we identify and re-weight prediction errors of state-action pairs that have high impact on action ranking. Through adversarial reweighting, the simulator is calibrated to prioritize decision-critical regions and align predicted outcomes with real action utility instead of global prediction metrics (Liu et al., 2024; Guo et al., 2025). To address Issue 2 (seeing every uncertainty as treat), we adopt group-relative decision-making perturbation. Rather than enforcing pessimistic worst-case constraints, we optimize the policy by preserving relative action preferences across a group. This group-relative formulation stabilizes policy learning under simulator imperfection while maintaining nominal performance (Shao et al., 2024; Zhang and Zuo, 2025). These two technical components form a robust pipeline to robustify both simulator surrogate and policy learning. Our main contributions are as follows: • We introduce an adversarial simulator calibration method that reweighs surrogate outputs based on decision-critical errors, in order to align simulation fidelity with downstream action selection. • We propose a group-relative perturbation strategy to stabilize policy preferences and enable robust action selection under simulator uncertainty. • Extensive experiments on the DataCo, GlobalStore, and OAS benchmarks show that Sim2Act consistently outperforms existing robustness baselines under a range of structured and unstructured perturbations. (a) Overview (b) Module Details Figure 1: Sim2Act Framework. (a) The overview illustrates the two-stage structure of our framework: simulator calibration and perturbed decision-making. (b) Module details: The simulator, implemented as an LSTM-based encoder-decoder model with linear feature extractor, predicts outcomes from state-action pairs and applies adversarial correction to reduce systematic prediction errors. During decision-making, latent states are perturbed using Gaussian noise derived from the simulator’s estimated covariance Σ , producing a distribution of plausible states. The decision-maker θD_θ learns a robust policy by maximizing group-relative advantages across these perturbed states, enhancing generalization under latent uncertainty. 2 Problem Statement We consider a simulation-based decision-making learning process. Concretely, the process follows a two-stage pipeline. First, a simulation model is trained offline from historical data to approximate real-world environment dynamics and rewards. Second, a decision policy is optimized by interacting with the learned simulator, without access to risky real environment feedback. Although this approach can achieve high performance under the simulator, policies may behave unreliably at deployment time when simulator imperfections affect action-sensitive decisions. Therefore, our objective is not only to maximize expected return under the learned simulator, but also to ensure robust decision behavior that remains stable under realistic uncertainty. We now introduce the formal setting and notation. Simulation Model and Decision Policy. We model the learned simulator as a parameterized function θsS_ _s trained offline to approximate environment dynamics. Given a state–action pair (st,at)(s_t,a_t), the simulator outputs a predicted next state and reward: θs(st,at)=(s^t+1,r^t)S_ _s(s_t,a_t)=( s_t+1, r_t). A decision policy θdD_ _d, parameterized by θd _d, maps states to actions, where at∼θd(⋅∣st)a_t _ _d(· s_t). During policy training, the decision-maker θdD_ _d interacts with trained simulator θsS_ _s. Robustness of Decision Policies. We characterize robustness as the sensitivity of a policy’s performance to structured perturbations of simulator predictions. Intuitively, a robust policy should maintain stable behavior under locally constrained variations of simulator outputs. Formally, we consider perturbed states s′s generated via latent-space noise: s′∼s+(0,Σ)s s+N(0, ). The robustness of a policy θdD_ _d is defined as the expected degradation in predicted rewards as Equation (1): R(θd)=−s[θs(s,θd(s))−θs(s′,θd(s′))],R(D_ _d)=-E_s [S_ _s(s,D_ _d(s))-S_ _s(s ,D_ _d(s )) ], (1) where θs(⋅)S_ _s(·) denotes the reward prediction (i.e., the r r component of the simulator output). Learning Objective. Our goal is to learn a decision policy that both achieves high expected return under the simulator and maintains sufficient robustness. This leads to the following optimization problem as Equation (2): θd∗=argmaxθd[∑t=0Tθs(st,θd(st))],s.t. R(θd)>δ. _d^*= _ _dE [ _t=0^TS_ _s(s_t,D_ _d(s_t)) ], .t. R(D_ _d)>δ. (2) 3 The Sim2Act Approach 3.1 An Overview of Sim2Act Figure 1 shows Sim2Act includes robust simulator learning and robust policy learning: Step 1: Action-aligned Simulator Calibration is to reduce simulation errors in decision-critical regions while preserving average accuracies, so that small mispredictions can not flip the entire action ranking orders. We introduce an adversarial calibration method to re-weight surrogate outputs based on errors of predicting decision-critical state-action pairs, in order to strengthen the robustness. Step 2: Group-Relative Perturbation is to protect a policy against becoming overly timid due to prediction error penalization so that the policy will not see every uncertainty as threat and discard high-performing actions. We propose to train the policy with a group-relative perturbation strategy. In particular, we sample a coherent group of latent perturbations (e.g., from the simulator’s learned covariance) around each nominal state. We then compare action rewards across that group, and optimize the policy to preserve relative action rankings within the group. This group-wise comparison enables the policy to distinguish and consider high-risk high-reward actions over high-risk low-reward actions, thus, prevent policy from seeing all uncertainty as threat and to maintain robustness. 3.2 Step 1: Action-Aligned Simulator Calibration Why Action-aligned Simulator Calibration Matters? In Sim2Act, policy learning relies on a learned simulator to provide state transitions and rewards. Compared with traditional simulator training objectives (e.g., global MSE minimization), the simulator shouldn’t just be accurate on average. This is because not all mistakes of predicting action rewards are equal. The simulator must be most accurate on the specific actions that change the final decision. The Simulation Calibration Model Structure. The simulation calibration structure includes a simulator and a calibrator. 1) The simulator. The simulator (denoted by (s,a)→(s^′,r^)S(s,a)→( s , r)) is a latent-variable model that maps a state-action pair (s,a)(s,a) to a predicted next state and a reward, given by: (s,a)=(s^′,r^)=G(E(s,a)),S(s,a)=( s , r)=G(E(s,a)), (3) where E(⋅)E(·) encodes the input into a latent representation and G(⋅)G(·) decodes it to environment outcomes. 2) The calibrator. We develop a lightweight action-conditioned calibrator (denoted by b¯(s,a;w) b(s,a;w)) that mimics a policy to output a normalized importance weight for each state-action pair, given by: b¯(s,a;w)=exp(⟨s,wa⟩)/∑i=1Kexp(⟨s,wai⟩). b(s,a;w)= ( s,w_a )/ _i=1^K ( s,w_a_i ). (4) Here, waw_a denotes a learnable parameter associated with action a. The calibrator modulates simulator training by assigning higher weights to state-action pairs whose prediction errors are more consequential for action ranking, thereby enabling action-aligned calibration in decision-critical regions. 3) The interactions between simulator and calibrator. The simulator outputs predicted reward for each action along with corresponding errors (i.e., potential deviation of predicted reward compared with real reward). Based on the prediction errors of each action, the calibrator outputs a probability distribution of each action as weights (b¯(s,a;w) b(s,a;w))), which are used in the gradient descent optimization of simulator to re-weight reward prediction losses. The Objective Function of Calibration. The simulation calibration can be seen as a mini-max optimization problem, because the calibrator acts as an adversary that identifies the worst-case errors (the max part), forcing the simulator to prioritize fixing the mistakes that matter most for decision-making (the mini part). In particular, in the max part, the calibrator scans the data to find where the simulator is most wrong about action outcomes, then shine a spotlight on those errors by giving them the highest weights; in the mini part, the simulator then tries to minimize the errors that the calibrator has highlighted, ensuring it becomes accurate in the most critical areas rather than being average accurate. This minimax objective is given by: J(w,θs)=∑i=1K(x,y)∈D[b¯(s,ai;w)⋅‖y−(x;θs)‖],J(w, _s)= _i=1^KE_(x,y)∈ D [ b(s,a_i;w)· \|y-S(x; _s) \| ], (5) where x=(s,a)x=(s,a), y=(s′,r)y=(s ,r) are ground-truth transitions from the offline dataset D. Here, the calibrator w seeks to maximize this objective (to increase the weights of high-impact errors for actions), while the simulator θs _s seeks to minimize it (to correct those errors). Solving the Optimization Problem. We employ an alternating optimization strategy to stabilize simulator training. This process iteratively refines the focus of the calibrator and the accuracy of the simulator: First, we freeze the simulator parameters θs _s and update the calibrator w via gradient ascent during the optimization process as Equation (6). w(t)←w(t−1)+ηw∇wJ(w,θs(t−1)).w^(t)← w^(t-1)+ _w _wJ(w, _s^(t-1)). (6) This step identifies high-error regions in the current simulator by increasing the weights b¯(s,a;w) b(s,a;w) for samples with large residuals. We next freeze w and update the simulator parameters θs _s via gradient descent as Equation (7): θs(t)←θs(t−1)−ηs∇θsJ(w(t),θs). _s^(t)← _s^(t-1)- _s _ _sJ(w^(t), _s). (7) By minimizing the weighted loss, the simulator is forced to reduce errors in the regions highlighted by the calibrator. This alternating procedure continues until the validation error stabilizes, effectively balancing global accuracy with local fidelity in critical regions. Theoretical Insight for Step 1 Action-aligned simulator calibration prioritizes decision-critical regions, tightening the deployment-time performance gap by aligning simulated rewards with action ranking for decision making. Formal statements and proof are in Appendix B. 3.3 Step 2: Group-relative Perturbation Why Group-Relative Perturbation Matters? Perturbations matter because a learned simulator is different messy real world, and perturbations expose whether a policy is dangerously over-optimized to a simulator’s specific imperfections. Group-relative perturbations matter because instead of reacting to single noisy perturbed state, they train the policy to compare actions across a local group of perturbations. This can stabilize relative action preferences, reduce overreaction, and keep the policy robust without becoming overly conservative. By maximizing the compared rewards, the policy effectively learns to task risky action when reward is high. The Model Structure. 1) The decision-maker The decision-maker θdD_ _d is a stochastic policy that maps a state to a distribution over actions: θd(a∣s)D_ _d(a s). The policy outputs action distribution and get predictive rewards from calibrated simulator for policy learning and then update policy. 2) Perturbed states and perturbation groups. We generate perturbations in the simulator’s latent space. Specifically, given a state-action pair (s,a)(s,a), the simulator predicts a latent distribution characterized by a centroid z and a covariance matrix Σ(s,a) (s,a). We define a perturbation group as a set of M latent vectors sampled from this distribution: zii=1M\z_i\_i=1^M, where zi∼(z,Σ)z_i (z, ). These latent vectors are then decoded back into a set of perturbed states S~=s~1,…,s~M S=\ s_1,…, s_M\. 3) The interaction between decision-maker and perturbation groups. For every decision step, the policy is exposed to the perturbed states of a perturbation group, and generate appropriate action for each state. The policy then select the best action with highest reward from the actions and update its parameters with selected state-action pairs. The Objective Function of Perturbation. We design a composite loss function to balance robust exploration with utility maximization. It consists of a group-relative advantage term and a utility alignment term. We denote the reward prediction component of the simulator as r(s,a)∈ℝS^r(s,a) . To encourage the policy to favor actions that are robustly superior, we formulate a Group-Relative Advantage. Here, the performance of a specific action aia_i is evaluated not in absolute terms, but relative to the group average r¯ r of the perturbed neighborhood, which serves as a local baseline: r¯=1M∑jr(s~j,aj) r= 1M _jS^r( s_j,a_j). The robust advantage loss is defined as Equation (8): ℒgroup-adv=−1M∑i=1M(r(s~i,ai)−r¯)⏟Group Advantage⋅logθd(ai∣s~i).L_group-adv=- 1M _i=1^M (S^r( s_i,a_i)- r)_Group Advantage· _ _d(a_i s_i). (8) It encourages the policy to increase the probability of actions that perform better than the local average under uncertainty, while suppressing actions that fall below the group mean. Simultaneously, we define a target reference r∗r^* (e.g., an expert baseline or a maximization target) and minimize the “regret” gap (r∗−r(s,a))(r^*-S^r(s,a)). This forces the policy to not just be stable, but to push the absolute predicted reward r(s,a)S^r(s,a) towards the ideal outcome. The final objective combines these two goals as Equation (9): ℒdecision=η⋅ℒgroup-adv+(r∗−r(s,a∼θd))⏟Utility Gap,L_decision=η·L_group-adv+ (r^*-S^r(s,a _ _d))_Utility Gap, (9) where η>0η>0 balances the stability induced by the group advantage with the aggressiveness of utility maximization. Solving The Optimization Problem. The training process alternates between perturbed states generation and policy update. In each iteration, we first map a batch of states s to the simulator’s latent space to capture local uncertainty (covariance Σ ). By sampling M latent perturbations, we generate a group of perturbed contexts s~i s_i around the nominal state. Next, the policy acts on this entire group. We calculate the group mean r¯ r to determine the relative advantage of each action (ℒgroup-advL_group-adv) while simultaneously monitoring the absolute utility gap on the original state. Finally, θd _d is updated via gradient descent to minimize the joint objective, ensuring the policy is both robust to variance and high-performing on average. The policy parameters θd _d are updated via gradient descent to minimize the joint loss ℒdecisionL_decision. This procedure repeats until the policy converges to a solution that is both high-performing and robust to the modeled variance. Detailed algorithmic steps are provided in Appendix F. Theoretical Insights for Step 2. Group-relative learning acts as a control variate, reducing optimization variance while preventing over-conservative policy collapse. Formal statements and proofs are in Appendix B. Dataset DataCo (43, 165445) GlobalStore (27, 51290) OAS (22, 28136) Sim Acc Risk Time Status Overall Risk Time Status Overall Risk Time Status Overall Markov 0.4978 0.1487 0.5040 0.3835 0.4961 0.1355 0.4934 0.3750 0.5100 0.0011 0.5068 0.3393 Prediction 0.7019 0.3395 0.8161 0.6191 0.8440 0.6767 0.8430 0.7879 0.7157 0.3706 0.7510 0.6124 Generation 0.7024 0.3485 0.8156 0.6221 0.9366 0.8066 0.9355 0.8929 0.7149 0.3916 0.7503 0.6189 S2D 0.9508 0.8851 0.9695 0.9351 0.9743 0.9255 0.9756 0.9585 0.7215 0.3985 0.7574 0.6258 S2A(Ours) 0.9563 0.8875 0.9618 0.9352 0.9723 0.9744 0.9750 0.9650 0.7270 0.3937 0.7629 0.6279 Dec Rwd TTimely^Timely TProfit^Profit Diff Overall TTimely^Timely TProfit^Profit Diff Overall TTimely^Timely TProfit^Profit Diff Overall Real 0.5244 0.0364 0.4880 0.5608 0.3320 0.0848 0.2472 0.4168 0.4800 0.0000 0.4800 0.4800 LP 0.5162 0.5434 0.0272 1.0596 0.3552 0.6001 0.2449 0.9554 0.5037 0.1043 0.3994 0.6080 DQN 0.5276 0.2071 0.3205 0.7347 0.2827 0.9326 0.6499 1.2153 0.4817 0.0000 0.4817 0.4817 PPO 0.5343 0.0000 0.5343 0.5343 0.3476 0.0004 0.3472 0.3480 0.4865 0.0000 0.4865 0.4865 GPT3.5 0.5258 0.2459 0.2800 0.7717 0.3298 0.0439 0.2859 0.3736 0.4844 0.0000 0.4844 0.4844 S2D 0.5397 0.5637 0.0240 1.1034 0.3446 0.9278 0.5828 1.2724 0.4882 0.1611 0.3271 0.6493 S2A(Ours) 0.5447 0.5786 0.0339 1.1232 0.3446 0.9460 0.6014 1.2906 0.4830 0.1886 0.2944 0.6717 Table 1: Simulation and Decision Performance Comparison (Refer to Appendix J for standard deviations). 4 Experimental Results We conduct extensive experiments on various datasets to evaluate the performance of our method. Specifically, our experiments aim to answer: Q1: Can our method outperform baselines on the robustness under random or latent perturbations? Q2: Aside from robustness, can our method achieve comparable accuracies of in simulation and decision-making like other strong baselines? Q3: How can we understand and interpret the mechanism and influence of our algorithm? 4.1 Experimental Setup Datasets. We evaluate our approach on three open-source supply chain datasets: 1) DataCo (Constante et al., 2019), 2) Global-Store (G, 2023), and 3) OAS (Vinay34, 2024) that cover diverse logistics and shipping scenarios with order-level records, including products, shipment modes, and delivery outcomes. We use supply chains as experiment applications because they are a representative industrial system for simulation and decision science studies. Each dataset is split into training, validation, test sets (8:1:1), using the same protocol as Sim2Dec (Bai et al., 2025) to ensure a fair comparison. To prevent overfitting, we use early stopping and ℓ2 _2 weight penalties. Both simulator and decision models are trained on the training set and evaluated on the test set. Dataset statistics are reported in the sub-captions of Table 1 in the format (number of features, number of items). Evaluation Metrics. The performance of a simulator is evaluated based on three tasks of predicting delay risk, delivery time, and on-time status, with test-set accuracy averaged across these outputs. To assess simulator robustness, we use: (i) worst-case accuracy, defined by the minimum Overall score across runs with the same perturbation level; (i) variance, defined by the standard deviation of overall scores at the same perturbation level; and (i) drop rate, defined by the average decline from the unperturbed setting. The performance of a decision-maker is measured by the two metrics: average profit and on-time rate (both normalized to [0,1][0,1]). We report the (i) absolute difference (denoted by Diff) to capture imbalance, and (i) sum (denoted by Overall) of the average profit and on-time rate as total reward, ranging in [0,2][0,2]. To assess decision robustness evaluation, we examine how the Overall metric drops as the perturbation magnitude increases relative to the unperturbed baseline at ϵ=0ε=0, rather than relying on single-point measurements. To evaluate safety under the worst-case scenarios, we report Conditional Value at Risk (CVaR@5%), which measures the expected return of the worst 5% of trajectories. All robustness experiments are conducted under two controlled settings: Nominal (p=0p=0) and Perturbed (p=0.5p=0.5), In the Perturbed setting, we introduce latent Gaussian noise to represent severe, structured distribution shifts, while the perturbation sweep probes increasingly adverse conditions. Baseline Algorithms. We consider three paradigms for simulation: 1) Markov-based simulation (Gagniuc, 2017), which models transitions using predefined probabilities. 2) Prediction-based simulation (Caruana, 1997), which uses multi-task learning to separately predict status variables. 3) Generation-based simulation (Gu et al., 2018), a non-autoregressive model that jointly generates multiple order features in a single forward pass. For decision-making, we compare our method against standard and robust optimization strategies: 1) Linear Programming (LP) (Dantzig, 2002), a rule-based optimization approach serving as a deterministic baseline. 2) Deep Q-Network (DQN) (Mnih et al., 2015) and PPO (Schulman et al., 2017), representing standard value-based and policy-gradient reinforcement learning methods, respectively. 3) ChatGPT-3.5 (Brown et al., 2020), evaluated under a zero-shot setting to assess large language model capabilities in supply chain logic. 4) RARL (Pinto et al., 2017). 5) EPOpt (Rajeswaran et al., 2017), two representative robust RL methods that utilize adversarial training and ensemble-based generalization, respectively, serving as key benchmarks for robustness evaluation. 6) S2D (Sim2Dec) (Bai et al., 2025), the state-of-the-art simulation-to-decision framework and the backbone of our method, which we aim to improve upon. 7) S2D-l: a variant of S2D, perturbed under latent-structured perturbation), and 8) Ours-l: a variant of Sim2Act, perturbed under latent-structured perturbation. Implementation details, selection rationale, Hyper-parameters and environment are detailed in Appendix H. 4.2 Experimental Results Q1: A Study of Simulation and Decision Robustness Under Perturbations. (a) Decision-maker: DataCo (43, 165445) (b) Decision-maker: GlobalStore (c) Decision-maker: OAS Figure 2: Decision-maker robustness under increasing perturbation across three datasets. Curves show the degradation of overall decision reward as perturbation strength increases, with slopes indicating sensitivity to uncertainty. Sim2Act maintains flatter degradation curves and smaller slopes than baselines, demonstrating stable performance under both latent-structured and unstructured perturbations (Goal 2). Figure 3: CVaR@5 Robustness on DataCo, GlobalStore, and OAS datasets. Solid bars denote nominal performance (p=0p=0), while hatched bars denote performance under perturbation (p=0.5p=0.5). (a) Sim: DataCo (b) Sim: GlobalStore (c) Sim: OAS Figure 4: Simulator robustness under perturbation across three datasets. Radar plots summarize worst-case accuracy, variance, and drop rate, Sim2Act improves worst-case accuracy and reduces variability, demonstrating effective decision-critical calibration. A: Decision Robustness. To assess decision robustness, we conduct sensitivity tests using two complementary perturbation mechanisms that correspond to realistic deployment failure modes: latent-structured perturbation (denoted by the -l suffix), which samples Gaussian perturbations in the simulator latent space guided by the learned covariance, and random input perturbation (denoted by no suffix), which applies unstructured noise directly to inputs. Implementation details remain in Appendix K. These two mechanisms serve as representative probes of failure behavior (structured vs. unstructured) and are evaluated across a range of magnitudes so that we can characterize whether decision performance drop is gradual (graceful) or abrupt (catastrophic). We observe that Sim2Act does not rely on any specific noise distribution: calibration and group-relative mechanisms implemented by correcting decision-critical inconsistencies and stabilizing relative action preferences, rather than by tailoring to a single perturbation family. Average Performance Stability. Figure 2 highlights that our method maintains stable performance under both latent-structured and random perturbations. On DataCo, our approach shows almost no degradation (1.1232→1.12221.1232→ 1.1222) while S2D drops notably (1.1034→1.03421.1034→ 1.0342). On GlobalStore, Ours-l remains invariant around 1.291.29, whereas baselines degrade substantially. On OAS, our method preserves near-constant rewards (0.6717→0.67050.6717→ 0.6705), outperforming both S2D and LP. Risk Analysis (CVaR). Beyond average returns, we explicitly evaluate policy safety using CVaR@5(Figure 3). While S2D shows competitive nominal performance (p=0p=0), it becomes brittle under perturbation, suffering a 20.9% drop on DataCo (0.67→0.530.67→ 0.53) at p=0.5p=0.5. In contrast, Sim2Act demonstrates robust risk control, maintaining a high CVaR of 0.61, which validates that our group-relative mechanism effectively mitigates tail risks caused by simulation errors. These results confirm that Sim2Act achieves robust and non-disruptive performance across structured and unstructured perturbations. Full detailed degradation curves and distributional analyses, are in Appendix D. B: Simulation Robustness. Figure 4 shows that our method achieves consistently higher worst-case accuracy across all datasets, e.g., 0.6790.679 on DataCo compared to 0.2250.225 for Markov. Besides, it yields the lowest performance variance that highlights that calibration improves simulator reliability in decision-critical regions. Complete variance and drop-rate statistics are reported in Appendix J. Q2: Simulation and Decision Performance Comparison. A: Simulation Accuracy. As illustrated in Figure 1, we observe that S2A achieves not just comparable simulation accuracy to S2D but also improve downstream decision quality. On DataCo, the profit score improves from 0.56370.5637 to 0.5786, and the overall decision score increases from 1.10341.1034 to 1.1232. Similar trends are observed across other datasets, highlighting that localized surrogate corrections enhance fidelity in decision-critical regions. Detailed per-metric results are provided in Appendix J. B: Decision Reward. As illustrated in Figure 1, our method not just enhances decision robustness but also preserve decision reward. On GlobalStore, S2A boosts profit from 0.92780.9278 to 0.9460 while maintaining timeliness at 0.34460.3446. On OAS, the overall decision score rises from 0.64930.6493 to 0.6717, indicating improved robustness under perturbations. These results demonstrate that S2A fulfills our design goals by improving decision-critical calibration and enabling robust yet high-quality policy behavior in offline decision settings. (a) DataCo Ablation (b) GlobalStore Ablation (c) OAS Ablation (d) Calibration Heatmap Figure 5: Ablation study of Sim2Act components across three datasets. The first three panels report decision robustness under perturbation for different module combinations: None (S2D), +SimCal, +DecPert, and +Both. +SimCal improves robustness in decision-critical regions by reducing performance degradation, while +DecPert stabilizes decision behavior under uncertainty. The calibration heatmap (right) visualizes action-level reliability gains, showing that simulator calibration concentrates improvements on decision-critical actions. C: Ablation Study of Calibration and Perturbation. We assess the individual contributions of the simulator calibration (+SimCal) and decision perturbation (+DecPert), with neither component enabled (None) as the baseline (Figure 5). C.1: Ablation Study on Simulator. +SimCal meaningfully reduces vulnerability in decision-sensitive regions. For example, on DataCo the profit drop rate decreases from 6.9%6.9\% to 4.8%, indicating better alignment between simulated outcomes and decision rewards. (See Appendix A for full numeric tables and per-strength breakdown.) C.2: Ablation Study on Decision-maker +DecPert improves robustness under perturbation while keeping nominal performance stable; on OAS the timely-delivery drop is reduced from 0.37%0.37\% to 0.22%. The combined setup (+Both) yields the most consistent results across datasets. (Full ablation curves and exact numbers are in Appendix A .) Q3: Studying Calibration Improvement Heatmap Over Datasets and Actions. Figure 5(d) visualizes action-level calibration gains of accuracy after calibration and accuracy before calibration. We focus on high-frequency, decision-critical actions: e.g., on GlobalStore the most frequent action (Action 3, 9132 samples) improves from 0.94470.9447 to 0.9531; on OAS, Action 3 improves from 0.56940.5694 to 0.5718; on DataCo, Action 3 improves from 0.91500.9150 to 0.9155. These representative numbers support the claim that calibration concentrates gains on high-impact actions, more detailed analysis is in Appendix C. 5 Related Work Simulation-based Decision Making and Calibration. Simulation is critical for decision-making in high-stakes domains where real-world exploration is prohibitive, such as supply chains and industrial control (Agrawal, 2025; Barykin et al., 2020). Recently, Sim2Dec (Bai et al., 2025) is a significant step forward by using generative models (LSTM) to approximate real-world environment dynamics and serve as simulator surrogates. However, existing environment simulator surrogates suffer from issues caused noisy, drifted, or imperfect data (Bi et al., 2022). Although these methods can minimize global prediction errors (e.g., MSE) or maximizing likelihood (Correia et al., 2023; Atanassov and Dimov, 2008), they are limited by the inability to reduce simulation errors in decision-critical regions (i.e., state-action pairs) where small mispredictions can not flip the entire action ranking orders and impact downstream policy performance (Zhao et al., 2021; Fonteneau et al., 2013). Robustness in Model-Based Reinforcement Learning. Robustness is essential Model-Based Reinforcement Learning (MBRL), particularly for offline settings where RL agents cannot correct its knowledge via interactions with real environments. Prior studies are two fold: 1) adversarial robustness and 2) conservative regularization. Adversarial methods (Pinto et al., 2017; Zhang et al., 2021) train agents against a worst-case perturbation and are modeled as an opposing player. Such min-max formulation can lead to overly conservative policies that sacrifice average performance for worst-case safety (Zhang et al., 2025). Offline RL methods employ pessimistic regularization (Yang et al., 2022; Li et al., 2025) or constrain the policy to stay close to the policy distribution of previous step (Liu et al., 2021). These methods however often struggle with distribution shifts and sparse data regions (Kumar et al., 2020). Instead, our method adopts a group-relative perturbation strategy for policy robustness. Adversarial Learning and Group-Relative Optimization. Our method is inspired by the insights of adversarial learning and group-relative optimization. Beyond attack defense, adversarial training can be reformulated to enforce regularization, such as fairness or consistency (Liu et al., 2024; Guo et al., 2025). We adapt this paradigm to simulator calibration, treating the error re-weighting process as an adversarial game. Besides, the Group-Relative Proximal Optimization (GRPO) (Shao et al., 2024; Zhang and Zuo, 2025) has demonstrated remarkable success in large language models and reasoning by evaluating outputs relative to a group mean rather than a fixed baseline, thus, reduces variance and enhances generalization. We adapt this group-relative concept to perturbation as a tool to robustify policies. Moreover, there is no existing work that integrates the adversarial and group-relative concepts to jointly improve simulation surrogates and decision-making policy learning. 6 Conclusion Remarks Simulation-to-decision learning aims to learn a simulator as environment surrogate and decision policies by interacting with the surrogate, to avoid testing in real mission-critical systems. We study how to robustify both simulator and policy learning. Classic methods are limited by the inability to control prediction errors in decision-critical regions that flip entire action rankings; they often are overly regularized to see every uncertainty as threat and discard high-risk high-reward actions. We propose the Sim2Act that develops two concepts: action-aligned reweighing-based adversarial calibration and group-relative perturbation, to achieve robustness in both simulator and policy learning. Our results find that: 1) correcting high-impact state-action prediction errors is more effective than minimizing average reward loss; 2) group-relative perturbation can robustify policies without downgrading decision reward. Theoretically, our findings imply that probabilistic consistency is often superior to worst-case adversarial defense. Practically, our framework enables more reliable digital twins (simulation and decision-maker) deployment in real-world mission-critical domains (e.g., transportation, supply chains) without risking expensive infrastructures. Our future work is to study physics knowledge guided simulation surrogate and policy learning in complex scientific and engineering systems. References R. Agrawal (2025) Adaptive few-shot learning (afsl): tackling data scarcity with stability, robustness, and versatility. External Links: 2501.13479, Link Cited by: §1, §5. E. Atanassov and I. T. Dimov (2008) What monte carlo models can do and cannot do efficiently?. Applied Mathematical Modelling 32 (8), p. 1477–1500. Note: Special issue on numerical and computational issues related to applied mathematical modelling External Links: ISSN 0307-904X, Document, Link Cited by: §1, §5. H. Bai, H. Wang, N. Gong, X. Wang, W. Ying, H. Chen, and Y. Fu (2025) Supply chain optimization via generative simulation and iterative decision policies. External Links: 2507.07355, Link Cited by: §1, §1, §4.1, §4.1, §5. S. Y. Barykin, A. A. Bochkarev, O. V. Kalinina, and V. K. Yadykin (2020) Concept for a supply chain digital twin. International Journal of Mathematical, Engineering and Management Sciences 5 (6), p. 1498–1515. External Links: Document, Link Cited by: §1, §5. M. Bi, G. Chen, D. M. Tilbury, S. Shen, and K. Barton (2022) A model-based multi-agent framework to enable an agile response to supply chain disruptions. In 2022 IEEE 18th International Conference on Automation Science and Engineering (CASE), p. 235–241. External Links: Link, Document Cited by: §5. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020) Language models are few-shot learners. External Links: 2005.14165, Link Cited by: §4.1. R. Caruana (1997) Multitask learning. Machine Learning 28 (1), p. 41–75. External Links: Document, Link, ISSN 1573-0565 Cited by: §4.1. F. Constante, F. Silva, and A. Pereira (2019) DataCo SMART SUPPLY CHAIN FOR BIG DATA ANALYSIS. Mendeley Data. External Links: Document Cited by: §4.1. J. B. Correia, M. Abel, and K. Becker (2023) Data management in digital twins: a systematic literature review. Knowl. Inf. Syst. 65 (8), p. 3165–3196. External Links: Document, Link, ISSN 0219-1377 Cited by: §1, §5. G. B. Dantzig (2002) Linear programming. Operations Research 50 (1), p. 42–47. External Links: Document, Link, https://doi.org/10.1287/opre.50.1.42.17798 Cited by: §4.1. E. Derman and S. Mannor (2020) Distributional robustness and regularization in reinforcement learning. External Links: 2003.02894, Link Cited by: §1. R. Fonteneau, S. A. Murphy, L. Wehenkel, and D. Ernst (2013) Batch mode reinforcement learning based on the synthesis of artificial trajectories. Annals of Operations Research 208 (1), p. 383–416. External Links: ISSN 1572-9338, Document Cited by: §1, §5. A. G (2023) Global superstore data. Note: https://w.kaggle.com/datasets/anandaramg/global-superstore/dataAccessed: 2025-05-20 Cited by: §4.1. P. Gagniuc (2017) Markov chains: from theory to implementation and experimentation. External Links: ISBN 978-1-119-38755-8, Document Cited by: §4.1. K. Gregor, D. Jimenez Rezende, F. Besse, Y. Wu, H. Merzic, and A. van den Oord (2019) Shaping belief states with generative environment models for RL. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Cited by: §1. J. Gu, J. Bradbury, C. Xiong, V. O. K. Li, and R. Socher (2018) Non-autoregressive neural machine translation. External Links: 1711.02281, Link Cited by: §4.1. H. Guo, F. Li, J. Li, and H. Liu (2025) Offline reinforcement learning via conservative smoothing and dynamics controlling. In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vol. , p. 1–5. External Links: Document Cited by: §1, §5. M. Huang (2022) Sensitivity analysis in the generalization of experimental results. External Links: 2202.03408, Link Cited by: §1. A. Kumar, A. Zhou, G. Tucker, and S. Levine (2020) Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33, p. 1179–1191. External Links: Link Cited by: §5. X. Li, B. Wang, and X. Ling (2025) Conservative reward enhancement through the nearest neighbor integration in model-based offline policy optimization. Expert Systems with Applications 274, p. 126888. External Links: ISSN 0957-4174, Document, Link Cited by: §1, §5. Q. Liu, Y. Kuang, and J. Wang (2024) Robust deep reinforcement learning with adaptive adversarial perturbations in action space. External Links: 2405.11982, Link Cited by: §1, §1, §5. Z. Liu, H. Zhou, B. Chen, S. Zhong, M. Hebert, and D. Zhao (2021) Constrained model-based reinforcement learning with robust cross-entropy method. External Links: 2010.07968, Link Cited by: §1, §5. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis (2015) Human-level control through deep reinforcement learning. Nature 518 (7540), p. 529–533. External Links: Document, Link, ISSN 1476-4687 Cited by: §4.1. L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta (2017) Robust adversarial reinforcement learning. External Links: 1703.02702, Link Cited by: §1, §4.1, §5. A. Rajeswaran, S. Ghotra, B. Ravindran, and S. Levine (2017) EPOpt: learning robust neural network policies using model ensembles. External Links: 1610.01283, Link Cited by: §4.1. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. External Links: 1707.06347, Link Cited by: §4.1. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo (2024) DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300, Link Cited by: §1, §5. Vinay34 (2024) Supply chain analysis dataset. Data.World. External Links: Link Cited by: §4.1. R. Yang, C. Bai, X. Ma, Z. Wang, C. Zhang, and L. Han (2022) RORL: robust offline reinforcement learning via conservative smoothing. External Links: 2206.02829, Link Cited by: §1, §5. H. Zhang, H. Chen, C. Xiao, B. Li, M. Liu, D. Boning, and C. Hsieh (2021) Robust deep reinforcement learning against adversarial perturbations on state observations. External Links: 2003.08938, Link Cited by: §1, §5. J. Zhang and C. Zuo (2025) GRPO-lead: a difficulty-aware reinforcement learning approach for concise mathematical reasoning in language models. External Links: 2504.09696, Link Cited by: §1, §5. Z. Zhang, T. Duan, Z. Lin, D. Huang, Z. Fang, Z. Sun, L. Xiong, H. Liang, H. Cui, and Y. Cui (2025) State-aware perturbation optimization for robust deep reinforcement learning. External Links: 2503.20613, Link Cited by: §1, §5. S. Zhao, M. P. Kim, R. Sahoo, T. Ma, and S. Ermon (2021) Calibrating predictions to decisions: a novel approach to multi-class calibration. External Links: 2107.05719, Link Cited by: §1, §5.