Paper deep dive
Explainable Planning for Hybrid Systems
Mir Md Sajid Sarwar
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/20/2026, 12:58:28 PM
Summary
This thesis presents a comprehensive study on Explainable Artificial Intelligence Planning (XAIP) for hybrid systems. It introduces a contrastive plan explanation framework that generates hypothetical models to explain why a specific plan was chosen over alternatives. It also addresses the challenge of explaining unsolvability in hybrid planning problems through two methods: a model reconciliation framework using path-based analysis and an irreducible infeasible sets (IIS) approach, and a divide-and-conquer strategy that decomposes unsolvable tasks into sub-problems using longest common subsequence analysis. Additionally, the work includes a robotic software framework for autonomous navigation and a web-based tool for experimenting with these explanation algorithms.
Entities (13)
Relation Signals (10)
Mir Md Sajid Sarwar → authored → Explainable Planning for Hybrid Systems
confidence 100% · Thesis Submitted for the Degree of Doctor of Philosophy by Mir Md Sajid Sarwar
Mir Md Sajid Sarwar → supervisedby → Rajarshi Ray
confidence 100% · Supervisor: Dr. Rajarshi Ray
Mir Md Sajid Sarwar → collaboratedwith → Ansuman Banerjee
confidence 95% · collaborate closely with him... Dr. Ansuman Banerjee
Contrastive Plan Explanation Framework → usedfor → Explaining Plans
confidence 95% · The contrastive explanation framework provides explanations for the plans in Hybrid Systems.
Contrastive Explanation Tool → implements → Contrastive Plan Explanation Framework
confidence 90% · The tool incorporates the iterative re-modeling and re-planning algorithm
Divide-and-conquer strategy → usedfor → Decomposing Unsolvable Tasks
confidence 90% · decompose an unsolvable planning task into sub-problems
Model Reconciliation → usedfor → Explaining Unsolvability
confidence 90% · model reconciliation framework for explaining the unsolvability of planning problems
Divide-and-conquer strategy → uses → Longest Common Subsequence
confidence 90% · casting it to an instance of longest common subsequence problem
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The recent advancement in artificial intelligence (AI) technologies facilitates a paradigm shift toward automation. Autonomous systems are fully or partially replacing manually crafted ones. At the core of these systems is automated planning. With the advent of powerful planners, automated planning is now applied to many complex and safety-critical domains, including smart energy grids, self-driving cars, warehouse automation, urban and air traffic control, search and rescue operations, surveillance, robotics, and healthcare. There is a growing need to generate explanations of AI-based systems, which is one of the major challenges the planning community faces today. The thesis presents a comprehensive study on explainable artificial intelligence planning (XAIP) for hybrid systems that capture a representation of real-world problems closely.
Tags
Links
- Source: https://arxiv.org/abs/2604.09578v1
- Canonical: https://arxiv.org/abs/2604.09578v1
Trouble viewing inline? Open PDF directly →
Full Text
476,164 characters extracted from source content.
Expand or collapse full text
Explainable Planning for Hybrid Systems Thesis Submitted for the Degree of Doctor of Philosophy by Mir Md Sajid Sarwar Registration No. 2022 03 05 01 02 073 School of Mathematical & Computational Sciences Indian Association for the Cultivation of Science Kolkata, India June 2025 arXiv:2604.09578v1 [cs.AI] 24 Feb 2026 Indian Association for the Cultivation of Science KOLKATA-700032, INDIA 1. Title of the Thesis: Explainable Planning for Hybrid Systems 2. Name, Designation & Institution of the Supervisor/s: Dr. Rajarshi Ray Associate Professor, School of Mathematical & Computational Sciences, Indian Association for the Cultivation of science, Kolkata - 700 032, India 3. List of Publications: (A) Journal Publications (2): i. Sarwar, Mir Md Sajid; Ray, Rajarshi; and Banerjee, Ansuman. “Con- trastive Plan Explanation Framework for Hybrid System Models”, ACM Transactions on Embedded Computing Systems, Volume 22, Issue 2, Art- icle 22, Pages 1-51, Year (March 2023). [doi:10.1145/3561532]. i. Sarwar, Mir Md Sajid; and Ray, Rajarshi. “Exploring Inevitable Way- points for Unsolvability Explanation in Hybrid Planning Problems”, ACM Transactions on Embedded Computing Systems, volume 24, Issue 6, Article 163, Pages 1-20, Year (October 2025). [https://doi.org/10.1145/3767745]. (B) Conference Publications (4): i. Sarwar, Mir Md Sajid, Ray, Rajarshi; and Banerjee, Ansuman. “Con- trastive Plan Explanation Framework for Hybrid System Models.” In Proceedings of the 18th ACM-IEEE International Conference on Formal Methods and Models for System Design (MEMOCODE 2020). [doi:10.1109 /MEMOCODE51338.2020.9315040]. i. Sarwar, Mir Md Sajid, Ray, Rajarshi; and Banerjee, Ansuman. “Ex- plaining Unsolvability of Planning Problems in Hybrid Systems with Model Reconciliation.” In Proceedings of the 21st ACM-IEEE International Con- ference on Formal Methods and Models for System Design (MEMOCODE 2023). [doi:10.1145/3610579.3611082] i. Sarwar, Mir Md Sajid; Yadav, Rajeshwar; Samanta, Sudip; Ray, Ra- jarshi; Halder, Raju; Banda, Gourinath; Bhattacharya, Ansuman; and Thakur, Atul. “A Robotic Software Framework for Autonomous Navig- ation in Unknown Environment.” In Proceedings of the International Sym- posium of Asian Control Association on Intelligent Robotics and Industrial Automation (IRIA 2021). [doi:10.1109/IRIA53009.2021.9588693] iv. Dey, Devdan; Sarwar, Mir Md Sajid; Ray, Rajarshi; and Banerjee, Ansuman. “A Contrastive Explanation Tool for Plans in Hybrid Domains.” In Proceedings of the 17th Innovations in Software Engineering Conference (ISEC 2024). [doi:10.1145/3641399.3641424]. 4. List of Presentations in National / International / Conferences/ Work- shops/ Symposiums: • 18th ACM-IEEE International Conference on Formal Methods and Models for System Design (MEMOCODE 2020), Virtual Event, December 02-04, 2020, Jaipur, India. – Contrastive Plan Explanation Framework for Hybrid System Models. (Oral) • 21st ACM-IEEE International Conference on Formal Methods and Models for System Design (MEMOCODE 2023), September 21-22, 2023, Hamburg, Ger- many. – Explaining Unsolvability of Planning Problems in Hybrid Systems with Model Reconciliation. (Oral) • 17th Innovations in Software Engineering Conference (ISEC 2024), February 22-24, 2024, Bangalore, India. – A Contrastive Explanation Tool for Plans in Hybrid Domains. (Oral) • International Symposium of Asian Control Association on Intelligent Robotics and Industrial Automation (IRIA 2021), Virtual Event, September 20-21, 2021, Goa, India. – Robotic Software Framework for Autonomous Navigation in Unknown En- vironment. (Oral) • Formal Methods Update Meeting, July 4-5, 2022, IITDelhi, India. – Contrastive Plan Explanation Framework for Hybrid System Models. (Oral) • Research Highlights in Programming Languages, December 16-18, 2024, IIT Gandhinagar, India. – Explaining Unsolvability of Planning Problems in Hybrid Systems with Model Reconciliation. (Oral) – Exploring Inevitable Waypoints for Unsolvability Explanation in Hybrid Planning Problems. (Poster) 5. List of Additional Publications (Not Relevant to PhD Thesis): 4 (a) Iqbal, Sk Asif; Sarwar, Mir Md Sajid; and Ray, Rajarshi. “Explaining Un- solvability of Planning Problems in Cyber-Physical Systems” In Preceedings of the 17th Innovations in Software Engineering Conference (ISEC) 2025). ACM, Article No.: 14, Pages 1 - 11. [https://dl.acm.org/doi/10.1145/3717383.3717395] (b) Sarwar, Mir Md Sajid; Samanta, Sudip; and Ray, Rajarshi. “AUTONAV: A Tool for Autonomous Navigation of Robots”. arXiv:2504.12318. Year 2025. [https://doi.org/10.48550/arXiv.2504.12318] 5 6 i i iv Dedicated to My Parents, My Wife, My Daughter & My Elder Sister. Acknowledgements This thesis marks the culmination of my enriching Ph.D. journey at the Indian Association for the Cultivation of Science (IACS), which began in October 2020. I’m delighted to express my gratitude to everyone who contributed to making this period a successful and transformative experience. I extend my deepest gratitude to my supervisor, Dr. Rajarshi Ray, of the School of Mathematical & Computational Sciences, Indian Association for the Cultivation of Science, Kolkata. His exceptional guidance, continuous support, and profound insights have been pivotal in shaping my research and pushing the boundaries of my academic capabilities. He truly taught me how to conduct rigorous research and consistently inspired me with innovative ideas and unwavering encouragement. I’m equally grateful to Dr. Ansuman Banerjee of the Indian Statistical Institute, Kolkata. It was a privilege to collaborate closely with him, and his invaluable mentorship, constructive feedback, and collaborative spirit significantly enriched my academic experience and the writing of my dissertation. I’m immensely thankful for my parents’ unwavering support. Their enduring belief in me has been my anchor on this arduous journey, and my gratitude for them knows no bounds. I cannot imagine my research journey without the wonderful companionship and col- laborative spirit of my lab mates, juniors, and friends here at IACS. A heartfelt thank you goes to Atanu, Devdan, Asif, Suman, Payel, Prantika, Madhusudan, Samiran, Shrimon, Avishek, Anal, Atanu (math), Goirika, Maitree, Sudip, Sarmistha, and Tanushree. Your insightful discussions, invaluable suggestions, and the sheer joy you brought made this journey truly special. My sincere thanks go to all the faculty members, teaching and non-teaching staff, and the administration for their unwavering help and support throughout this endeavor. I am also deeply grateful for the financial and infrastructural support provided by the IMPRINT-2 project and the Institute Fellowship, generously funded by IACS and the Department of Science and Technology (DST), Government of India. This research would not have been possible without their collective contributions. Lastly, I thank all those whom I have missed out on the above list. vii Abstract The recent advancement in artificial intelligence (AI) technologies facilitates a paradigm shift toward automation. Autonomous systems are fully or partially replacing manually crafted ones. At the core of these systems is automated planning. With the advent of powerful planners, automated planning is now applied to many complex and safety- critical domains, including smart energy grids, self-driving cars, warehouse automation, urban and air traffic control, search and rescue operations, surveillance, robotics, and healthcare. There is a growing need to generate explanations of AI-based systems, which is one of the major challenges the planning community faces today. The thesis presents a comprehensive study on explainable artificial intelligence planning (XAIP) for hybrid systems that capture a representation of real-world problems closely. While XAIP is the main focus of the thesis, we start investigating motion planning problems in hybrid systems and acquaint ourselves with the literature in planning. We propose a motion-planning algorithm for a robot in an unknown environment based on iterative constraint-solving. An integrated software framework consisting of a simultaneous localization and mapping module, a motion planning module, and a plan execution module specially designed for a lizard-inspired quadruped robot has been designed and developed in collaboration with our colleagues. The techniques of motion planning and plan execution are the contributions of this thesis. When an AI agent generates a plan for a given task, that plan needs to be understand- able to its user. Contrastive approach has been a promising methodology from social sciences for explaining when there is a solution to a given planning problem. We propose a contrastive plan explanation framework for a hybrid system that builds a hypothetical model from the user’s questions on the plan given by a planner and generates a hypo- thetical plan to provide an explanation by highlighting its contrast with the original plan. We discuss a few classes of contrastive questions. The proposed framework, based on the re-model and re-plan idea, advocates the construction of a hypothetical model for each of these contrastive questions in such a way that any valid plan, if generated on this model, will imitate the contrastive question. In addition, we propose to prove the absence of a plan (no plan) in a hybrid domain using bounded reachability analysis. Since planning problems are undecidable for hybrid systems in general, when a planner fails to generate a valid plan for a given problem instance, it is not clear whether this is due to the underlying undecidability or because the problem instance does not admit a valid plan. With the objective of giving a user-friendly access to our explanation algorithms, we develop a tool with a web-based graphical user interface. The tool incorporates the iter- ix ative re-modeling and re-planning algorithm of our previous work to find a competitive contrastive plan with respect to the comparison metrics (e.g., plan makespan, plan length) that the underlying planner can provide. It provides provisions to experiment with two planners for hybrid domains and compare and contrast the explanations produced thereof. This work has been done in collaboration, where framework design and the algorithmic backbone of the tool are the contributions of this thesis. No plan situation, when there is no solution to a given planning problem, motivates us to look for an explanation of unsolvability. In this direction, model reconciliation problem (MRP) is a successful approach to explanation when there is a mismatch of knowledge between the AI agent and the human. The reconciliation process brings these knowledge bases closer by providing model updates. We present a model reconciliation framework for explaining the unsolvability of planning problems in hybrid domains, which facilitates explanation through a path-based reconciliation process. To this effect, we use a mix of graph traversal and path analysis, along with linear programming, to carry out the reconciliation process. In particular, we use the concept of irreducible infeasible sets (IIS) to generate explanations. The AI agent updates the human about the causes of the unsolvability of the planning problem through the explanation. The divide and conquer strategy has a significant impact on many computer science problems. This approach has been widely applied to plan generation and automated problem solving to decompose tasks into sub-problems that help progressively converge towards the goal. We propose to adopt the same philosophy of sub-problem identification as a mechanism for analyzing and explaining the unsolvability of planning problems in hybrid systems. We present a framework that decomposes an unsolvable planning task into sub-problems through a novel waypoint identification method by casting it to an instance of longest common subsequence problem, a widely popular approach in computer science, typically considered as an illustrative example for the dynamic programming paradigm. This thesis explores the area of explainable planning for hybrid systems under the broader field of XAIP. As hybrid systems closely represent real-world problems, by at- tempting to explain the behaviour of such systems, we address the issues that touch everyday lives. The contrastive explanation framework provides explanations for the plans in Hybrid Systems. The contrastive explanation tool provides a provision to experiment with different planning domains in hybrid systems and plug in different hybrid system plan- ners as a plan generation engine. The model reconciliation framework and sub-problem framework can draw the causes of unsolvability. We believe these frameworks can be useful to the hybrid systems planning community for explaining the behaviour of autonomous systems. Title of the Thesis Explainable Planning for Hybrid Systems xi Contents Acknowledgementsvii Abstractix Title of the Thesisxi List of Tablesxvii List of Figuresxviii 1 Introduction1 1.1 AI Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.1 Planning in Hybrid Systems . . . . . . . . . . . . . . . . . . . . . . . 3 1.1.2 A Case Study: Motion-planning Problem in Robotics from a Hybrid System Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.2 Explainable AI Planning (XAIP) . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2.1 XAIP Perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.2.2 Relevant XAIP concepts . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.3 XAIP in Hybrid Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.3.1 Explaining Plans in Hybrid Systems . . . . . . . . . . . . . . . . . . 10 1.3.2 Explaining Unsolvability in Hybrid Systems . . . . . . . . . . . . . . 10 1.4 Popular Approaches to Explanation Generation in XAIP . . . . . . . . . . . 11 1.4.1 Contrastive Explanation Approach . . . . . . . . . . . . . . . . . . . 11 1.4.2 Model Reconciliation Approach . . . . . . . . . . . . . . . . . . . . . 12 1.4.3 Logic-Based Approach . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.4.4 Divide and Conquer Strategy . . . . . . . . . . . . . . . . . . . . . . 14 1.5 Decidability of Planning Problems in Hybrid Systems . . . . . . . . . . . . 14 1.6 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.7 List of Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2 Hybrid System Planning19 2.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.2 Motion Planning in Robotics from a Hybrid System Perspective . . . . . . . 21 2.3 A Case Study for Autonomous Navigation in an Unknown Environment . . 22 2.3.1 Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 xiii CONTENTS 2.3.2 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.3.2.1 SLAM Module . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.3.3 Robotic Navigation Framework . . . . . . . . . . . . . . . . . . . . . 25 2.3.3.1 Planning and Control Module . . . . . . . . . . . . . . . . 27 2.3.3.2 Integrating SLAM, Planning and Control . . . . . . . . . . 33 2.3.4 Simulation and Experimental Evaluation . . . . . . . . . . . . . . . 35 2.3.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3 A Contrastive Plan Explanation Framework for Hybrid System Models 39 3.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.2 Explanation Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.2.1 Contrastive Explanation Framework . . . . . . . . . . . . . . . . . . 48 3.3 Contrastive Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 3.3.1 Replacing an Action by Another in the Plan . . . . . . . . . . . . . 51 3.3.2 Restricting an Action to Appear After a Certain Time . . . . . . . . 55 3.3.3 Restricting an Action to Appear Before a Certain Time . . . . . . . 58 3.3.4 Barring an Action from Appearing in the Plan . . . . . . . . . . . . 60 3.3.5 Restricting an Action to Occur Less Than a Certain Number of Times 62 3.3.6 Questioning the Optimality of Plan Duration . . . . . . . . . . . . . 64 3.3.7 Question on the Sequence of Actions in the Plan . . . . . . . . . . . 67 3.3.8 Questioning the Optimality of Plan Length . . . . . . . . . . . . . . 75 3.4 Proving the Absence of a Plan . . . . . . . . . . . . . . . . . . . . . . . . . 77 3.4.1 δ-Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 3.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 3.5.1 Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 3.5.2 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 3.5.3 Performance Evaluation of Proving Absence of Plan . . . . . . . . . 85 3.6 Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 3.7 Limitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 3.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 4 A Contrastive Explanation Tool for Plans in Hybrid Domains91 4.1 Tool Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 4.1.1 Module Description . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 4.2 Implementation and Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 4.3 Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 4.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 5 Explaining Unsolvability of Planning Problems in Hybrid Systems with Model Reconciliation101 5.1 Motivating Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 5.2 Problem Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 5.3 Our Reconciliation Methodology . . . . . . . . . . . . . . . . . . . . . . . . 108 5.3.1 Discrete Path Generation and Reconciliation . . . . . . . . . . . . . 108 xiv CONTENTS 5.3.2 Path Feasibility in the Continuous Dynamics . . . . . . . . . . . . . 110 5.3.3 Explanation and Human Model Update . . . . . . . . . . . . . . . . 115 5.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 5.5 Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 5.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 6 Exploring Inevitable Waypoints for Unsolvability Explanation in Hybrid Planning Problems121 6.1 Motivating Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 6.2 Problem Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 6.3 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 6.3.1 Decomposition into Sub-Problems . . . . . . . . . . . . . . . . . . . 128 6.3.1.1 Abstraction over Planning Problems . . . . . . . . . . . . . 129 6.3.2 Reduction to Longest Common Subsequence (LCS) Problem . . . . 130 6.3.3 Explanation Generation by Reachability Analysis . . . . . . . . . . . 132 6.3.4 Complexity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 6.4 Results and Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 6.4.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 6.4.2 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 6.4.3 Analysis of results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 6.4.4 Domain Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 6.5 Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 6.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 7 Conclusion and Future Work143 Bibliography146 8 Appendix167 8.1 Car Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 8.2 Generator Events Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 8.3 Planetary Lander Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 xv List of Tables 2.1 Clockwise and anti-clockwise rotations. θ 1 and θ 2 are angles of rotation of the joints in the first and second phases, respectively. . . . . . . . . . . . . . 30 2.2 Simulation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.1 The makespan of the plan for each re-plan iteration . . . . . . . . . . . . . . 66 3.2 A brief summary of the contrastive explanations generated for the user questions discussed above. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 3.3 BM represents the problem domains in PDDL+,and Ins represents the problem instances of a domain. PL reports the hybrid system planner chosen for solving the instances, where SM, EN, and UP represent SMT- Plan+, ENHSP, and UPMurphi, respectively. Dur and Len represent the original plan-duration and plan length for a problem instance, QN rep- resents the set of contrastive questions presented in Section 3.3, H-dur and H-len represent the plan-duration and the plan-length of the hypothetical plan HPlan. DC τ and DC l represent the plan-duration difference and the plan-length difference of the HPlan with the original plan, respectively; these are the contrastive explanation (CE) metrics illustrated in Def 3.2.6. CM reports the number of constraint modifications that are made in the domain (inclusive of constraint addition and deletion). Time and H-time represent the plan generation time whereas Mem and H-mem report the memory usages for the original plan and the HPlan, respectively. HQ rep- resents an evaluation on the quality of the HPlan against the original plan. 84 3.4 Bounded reachability analysis of the no-plan-models for a given δ perturb- ation of 0.01. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 4.1 Table depicting the mean response time of our tool to each of the seven contrastive questions in the car and the generator-events domain. Q. No. is the contrastive question from the set of questions in Section 4.1. The columns SMTPlan+ and ENHSP show the time taken by these planners to generate a plan. The column HModel Compilation Time shows the time taken to construct the Hmodel, and Best Contrastive Plan Time reports the time to find the best contrastive plan by iterative re-modeling with Bisection. NP denotes no plan produced within a given time-bound. NA denotes not applicable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 xvii LIST OF TABLES 5.1 Results on the warehouse automation and water-level monitoring systems. HM and AM represent Human Model and Agent Model, respectively. . . . . 116 5.2 Average explanation generation time. . . . . . . . . . . . . . . . . . . . . . . 117 6.1 Explanation generation on unsolvable planning problem instances. . . . . . 135 6.2 Performance analysis of our framework. PE indicates Path-exploration time, RA indicates Reachability analysis time, and AT indicates Accumu- lative Time = (a) + (b) + (c). . . . . . . . . . . . . . . . . . . . . . . . . . 136 xviii List of Figures 1.1 XAIP perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.2 The proposed organization to discuss the challenges and research directions in XAIP for hybrid systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.1 Stealth surveillance system for monitoring hostage scenario . . . . . . . . . 23 2.2 CoppeliaSim model of the quadruped robot . . . . . . . . . . . . . . . . . . 27 2.3 The robotic software architecture for SLAM, path-planning, and control. . . 28 2.4 Phases of Move-forward and Rotate-Clockwise maneuvering of the robot . . 30 2.5 Robot movement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.6 Robot’s trajectory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.7 Robot’s deviation from the path . . . . . . . . . . . . . . . . . . . . . . . . 33 2.8 Hybrid controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 2.9 Planning and Navigation: simulation results of plan generation and nav- igation in an office-like environment using CoppeliaSim and ROS: (a) is shown here. The initial plan is given in (b) when most of the environment was unknown, assuming the unknown as free spaces. (c) shows detection of obstacles on the path, (d) and (e) present two re-planning steps at different time points during navigation, and (f) depicts the robot’s actual traversed path while following the generated plans. . . . . . . . . . . . . . . . . . . . 37 3.1 The hybrid automaton model of the Car domain . . . . . . . . . . . . . . . 46 3.2 Example scenarios where makespan is useful. In the figure, t encodes time, and a and b are variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.3 The Contrastive Plan Explanation Framework. . . . . . . . . . . . . . . . . 49 3.4 The hybrid automaton for the corresponding HModel of the question is addressed in sect 4.1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.5 The hybrid automaton for the corresponding HModel of the question is addressed in sect. 4.2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 3.6 The hybrid automaton for the corresponding HModel of the question is addressed in sect. 4.3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 3.7 The hybrid automaton for the corresponding HModel of the question ad- dressed in sect 4.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 3.8 The hybrid automaton for the corresponding HModel of the question ad- dressed in sect. 4.5. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 xix LIST OF FIGURES 3.9 The hybrid automaton for the corresponding HModel of the question ad- dressed in sect. 4.7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 3.10 The hybrid automaton for the corresponding HModel for question 8. . . . . 76 3.11 Scatter graph comparing the HPlan generation time over the original plan. OP represents the original plan whereas QN1, QN2, QN3, QN4, QN5, QN7 and QN8 represent the HPlans generated against the compilations for the contrastive questions discussed in Sec. 3.3.1, 3.3.2, 3.3.3, 3.3.4, 3.3.5, 3.3.7 and 3.3.8, respectively. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 3.12 Scatter graph comparing the memory usages by the planner for the gen- eration of HPlans and the original plan. OP represents the original plan whereas QN1, QN2, QN3, QN4, QN5, QN7 and QN8 represent the HPlan corresponding to the contrastive questions addressed in Section 3.3.1, 3.3.2, 3.3.3, 3.3.4, 3.3.5, 3.3.7 and 3.3.8, respectively. . . . . . . . . . . . . . . . . 87 4.1 Tool Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 4.2 A snapshot of the tool’s GUI . . . . . . . . . . . . . . . . . . . . . . . . . . 95 4.3 Binding templatized question with instances. . . . . . . . . . . . . . . . . . 96 4.4 An HPlan generated by SMTPlan+ according to the contrastive question. 97 5.1 The human and the agent view of the warehouse environment is depicted. Each cell is associated with an identifier shown in the top-left corner of the cell. The Blue dotted line represents an expected plan by a human under its knowledge of the warehouse. Our explanation algorithm examines this plan in the agents’ view of the world and detects the presence of an obstacle in cell number 20. This knowledge is reported to the user as an explanation of the infeasibility of the plan. The Green dashed line represents a candidate plan in the human model when abstracting its continuous dynamics. This plan is refuted by our algorithm when the feasibility of the path is checked against the continuous dynamics of the human model. . . . . . . . . . . . . 103 5.2 Infeasibility of the plan explained by continuous dynamics, particularly the higher charge depletion rate at orange cells . . . . . . . . . . . . . . . . . . 106 5.3 The HA model for the warehouse automation problem, where locations rep- resent the corresponding cells shown in Figure 5.1. The automaton provided here is a part of the warehouse automation domain. . . . . . . . . . . . . . 107 5.4 The flow diagram of our model reconciliation framework . . . . . . . . . . . 109 6.1 The rover domain is depicted. Initially, the rover is at cell 11. Mountains and craters are impassable regions of the terrain, such as cells 7, 12, etc. The rover needs to collect soil samples from cell 1 and rock samples from cell 24. The base station is at cell 25. The up-slope areas in the terrain are shown in orange. The inevitable waypoints in the domain are cells w1 to w8 marked in blue. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 6.2 A hybrid automaton model of the planetary rover domain (partially shown). 125 6.3 Proposed Explanation Framework. . . . . . . . . . . . . . . . . . . . . . . . 127 x LIST OF FIGURES 6.4 All runs of valid plans from the initial set to the goal set must pass through the blue doorway. We envision the blue region as an inevitable waypoint to the planning problem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 6.5 A depiction of a graph of a H. Let S and D be the source and the goal locations in a planning problem Π. Every path from S to D visits locations A and B in sequence. Thus, S-A-B-D is the LCS of paths in PS(Π). Clearly, if a valid run of a plan exists in the H, the run must visit the invariants of S, A, B, and D sequentially. . . . . . . . . . . . . . . . . . . . . . . . . . . 131 6.6 Illustration of results in the example scenarios. . . . . . . . . . . . . . . . . 137 6.7 The city route network is depicted. Each node represents the important junctures of the city. The blue-colored routes represent both-way trans- portation between junctures. The green-colored routes represent one-way traffic. Initially, the car is at juncture A (shown as the green-colored node). The junctures I (yellow-colored), H (red-colored), and J (blue-colored) are the destinations of three different planning problems of the domain. The orange-colored nodes in the figure are the waypoints that appear in every source-to-destination path for a planning problem of A to H. . . . . . . . . 139 6.8 Warehouse automation domain. . . . . . . . . . . . . . . . . . . . . . . . . . 139 8.1 A hybrid automaton model of the generator-events domain for the planning problem instance 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 xxi Chapter 1 Introduction A I planning has been an active area of research for several decades. Autonomous sys- tems are envisioned to automate and replace manually crafted ones, where planning remains a core component of such systems. Classical AI planning is concerned with find- ing a sequence of feasible actions from an initial configuration of a system to a desired goal. Planning has been an active area of research for AI practitioners, leading to the development of planners for diverse planning domains, goal descriptions, and varied op- timization objectives. Techniques ranging from graph traversals to recent developments around constraint solvers for efficient and scalable planning have been widely explored. As AI techniques mature, the number of application areas in which humans and autonomous agents collaborate increases. In such scenarios, cooperative plans derived from mutual trust and understanding are important for achieving a desired objective. Indeed, such sys- tems’ safety, robustness, and trustworthiness are analyzed before deployment. However, the major challenge that the planning community is facing today is interfacing with its users; that is, any AI-based system must be able to explain its reasoning to humans in the loop (Gunning and Aha [2019]). With automated planning being applied in safety-critical systems, the need for explanation and trust in the agent’s behaviour has become ever more important. The ability to explain the rationale behind a decision of an autonomous agent is widely regarded as one of the precursors needed for humans to engage in trustworthy collaborations with autonomous agents. Further, the autonomous agent often generates a plan based on an opaque model of the environment that is not transparent to the user. Thus, a plan explanation is crucial. 1.1 AI Planning Planning is the branch of AI that seeks to automate reasoning about plans, most import- antly, the reasoning that goes into formulating a plan to come up with a series of actions or procedures to accomplish a particular goal. It is crucial for AI applications because it allows machines to think ahead, adapt to changes, and act autonomously 1 . Just like humans plan their daily tasks with a goal in mind, AI systems use planning algorithms to assess the situation, identify the intended outcome, and develop a strategy that specifies 1 https://w.geeksforgeeks.org/what-is-the-role-of-planning-in-artificial-intelligence/ 1 CHAPTER 1. INTRODUCTION the steps to take to get there. It is a model-based approach: a planning system takes a model of the environment, a description of the initial situation, the actions available to change it, and the goal condition as inputs and outputs a plan composed of those actions that will accomplish the goal when executed from the initial situation. • Types of planning: There are several types of planning approaches in AI, each suited to different tasks and environments: – Classical planning (Fikes and Nilsson [1971], Ghallab et al. [2004]) is the tra- ditional form of planning approaches that restrict the view of the world over which planning is performed. It assumes that the environment is fully observ- able and static, where all actions are deterministic. The AI agent has complete knowledge of the world and operates with a fixed goal, attempting to find a sequence of actions that leads from an initial state to a goal state. – Probabilistic planning (Kushmerick et al. [1995]) extends classical planning by incorporating uncertainty and randomness into the planning process, allowing for more realistic and adaptable solutions in complex, unpredictable environ- ments. The AI system must account for the fact that actions may have different possible outcomes with associated probabilities. Probabilistic planning often uses models like Markov Decision Processes (MDPs) or Partially Observable Markov Decision Processes (POMDPs) to manage this uncertainty. – Hierarchical planning (Erol et al. [1994]) breaks down complex tasks into sim- pler, smaller sub-tasks and creates a plan for each sub-task. This hierarchical approach is especially useful for solving large-scale problems where goals can be divided into manageable steps. It often involves decomposing high-level tasks into sequences of lower-level actions. – Reactive planning (Schmidhuber [1990]) denotes a group of autonomous agents’ action selection techniques in highly dynamic and unpredictable environments. Rather than following a pre-defined plan, the AI agent continuously reacts to changes in the environment in real time. This approach doesn’t rely on creating a full plan ahead of time but focuses on immediate responses to the current situation. – Temporal planning (Ghallab and Laruelle [1994], Haslum et al. [2019]) considers time restrictions and inter-dependencies between actions for reasoning about events and their temporal relationships. It ensures that the plan is workable within a certain time limit by taking into account the duration of tasks. • Applications of planning: AI planning is now being applied in diverse fields, demonstrating its adaptability and efficiency. A few significant applications are: – Robotics: (Karpas and Magazzeni [2020]) Automated planning allows robots to navigate efficiently in environments, avoid obstacles, and perform tasks 2 1.1. AI PLANNING autonomously. For example, an Amazon warehouse robot 2 can plan its path to pick up items without collisions. – Search and rescue operations: (Jennings et al. [1997]) AI is revolutionizing search and rescue missions by enabling efficient planning, resource allocation, and real-time data analysis, ultimately improving the speed and precision of rescue operations. – Autonomous Vehicles: Self-driving cars (Garikapati and Shetiya [2024]) use planning to navigate roads, make turns, stop at traffic signals, and avoid colli- sions with pedestrians or other vehicles. – Healthcare: Planning systems are used in treatment (Schmidt et al. [2023]), where algorithms suggest optimal therapies for patients based on various factors like medical history, current health, and probability of success. – Gaming: In video games (Yannakakis [2012]), planning is used to simulate intelligent behavior in non-player characters (NPCs). NPCs can plan their strategies in real-time, providing more challenging and unpredictable gameplay. – Logistics: Planning in AI optimizes logistics, inventory, and transportation, helping businesses improve efficiency and reduce costs. It can plan the most cost-effective routes for shipping goods or the best times to restock inventory. • Challenges of planning: While AI planning has many advantages, many issues need to be resolved. Typical challenges include: – Complexity: Planning, especially in complex environments, can be computa- tionally expensive. Finding the optimal sequence of actions in large, dynamic systems can take a significant amount of processing power and time. – Uncertainty: In uncertain or unpredictable environments, creating a plan that can handle every possible outcome is challenging. Probabilistic and reactive planning methods aim to address this, but it remains a difficult problem. – Scalability: As the size of the problem or task increases, so does the difficulty of planning. Scaling up planning algorithms to handle large datasets or envir- onments with numerous variables is a technical hurdle. – Trust: Trust in AI hinges on transparency, reliability, and accountability. For AI to be trustworthy, its autonomous actions must be consistent and depend- able. Organizations should clearly show how their AI systems plan and operate to foster transparency and build explainable systems that earn user confidence. 1.1.1 Planning in Hybrid Systems Planning in hybrid systems involves creating plans for systems with both discrete and continuous variables, which are common in real-world applications such as autonomous navigation, control systems, industrial production processes, power systems, robotics, etc. 2 https://w.aboutamazon.com/news/operations/amazon-robotics-robots-fulfillment-center/ 3 CHAPTER 1. INTRODUCTION It requires specialized planning techniques to handle these complex dynamics. Therefore, planning in hybrid systems is essential for designing and controlling these systems effect- ively, ensuring they achieve desired goals while considering constraints and dynamics. We discuss the relevant background in Chapter 2. • Modeling hybrid systems: Popular techniques in literature for modeling hybrid systems are as follows: – PDDL+ (Planning Domain Definition Language +) (Fox and Long [2006]) is an extension of PDDL designed to model hybrid systems, allowing for the rep- resentation of continuous processes and events alongside discrete actions. – Hybrid Automata (HA) (Alur et al. [1993, 1995a]) are a formal framework for modeling hybrid systems, combining discrete state transitions with continuous dynamics. • Planning approaches in hybrid systems: – Satisfiability Modulo Theories (SMT) planning approach for hybrid systems encodes planning problems as first-order logic formulae in a theory, allowing for formal analysis and planning of systems with both discrete and continuous dynamics. SMT solvers (de Moura and Bjørner [2008]) provide solutions to such encoding, allowing for efficient plan generation and verification (Cashmore et al. [2020]). – Mixed Integer Linear Programming (MILP) is a mathematical optimization technique used to find the best solution to problems with both continuous and discrete variables. It can be used to formulate and solve planning problems in hybrid systems, such as path planning (Richards and How [2002]), schedul- ing, and resource allocation by modeling constraints and objectives as linear equations or inequalities. – Heuristic search has been used in hybrid systems where planning problems can be cast as sequential decision-making problems, provided some time discretiza- tions (Scala et al. [2016]). Such methods exploit the planning-via-discretizations approach where the continuous dynamics of a model is approximated with uni- form time steps and step-functions (Piotrowski [2018]). – Reinforcement learning (Busoniu et al. [2018]) can be used to learn optimal policies for controlling hybrid systems in dynamic environments. It focuses on using AI algorithms to learn optimal strategies for managing complex sys- tems with multiple components. This is particularly useful in areas like energy management for hybrid electric vehicles (HEVs) and microgrids, where RL can adapt to changing conditions and optimize resource allocation 4 1.1. AI PLANNING 1.1.2 A Case Study: Motion-planning Problem in Robotics from a Hy- brid System Perspective The motion planning problem for robots traditionally consists of finding a state traject- ory and associated inputs, connecting the initial state to a state in the goal region while satisfying the system dynamics and safety criterion, for example, avoiding obstacles. It has been widely applied to many real-world applications, such as self-driving cars (Teng et al. [2023]), unmanned aerial vehicles (UAV) (Liu et al. [2017]), free-floating space ma- nipulator (Rybus [2020]), biped robots (Huang et al. [2001]), and robotic arms (Liu and Liu [2021]), etc., to perform complex tasks. Various approaches to address the motion- planning problem exist in the literature, such as RRT (Rapidly-exploring Random Tree) ( LaValle [1998]), A ∗ (Hart et al. [1968]), SAT, and SMT-based path-planning ( Hung et al. [2014], Saha et al. [2014]). However, a class of algorithms that have been particu- larly successful in solving such problems for robot models with differential constraints are the sampling-based algorithms (Choset et al. [2005], LaValle [2006]). • Motion planning in unknown or partially known environment: The prob- lem of finding a safe (i.e., collision-free) path from an initial state to a goal state when the navigational space is a priori unknown and is incrementally observable with the robot’s movement in the environment through line-of-sight perception has ubiquitous applications. However, motion-planning problems in such a setup have received relatively little theoretical investigation as compared to the problems where the environment is known. Most of the motion-planning algorithms assume that the environment is known and thus can not be used directly (LaValle [2006]). A graph-based approach based on D ∗ algorithm is proposed in (Stentz [1993]) for path- planning in unknown, partially known, and changing environments. It models the environment as a graph where nodes represent the robot’s states and arcs repres- ent the cost of moving between two states. (Janson et al. [2018]) introduced re- planning and forward-looking biasing in a self-contained framework while avoiding inevitable collision states (ICS) (Fraichard and Asama [2004]) for motion-planning in a partially known environment. Sampling-based methods like RRT ∗ (Karaman and Frazzoli [2011]), and its many variants have had considerable success for kin- ematic motion-planning. However, the complexity of such methods heavily relies on robot dynamics, and the global computation over the entire state space for high- dimensional systems (like legged robots) in cluttered environments renders them slow. A similar approach, SweepingRRT (Albee et al. [2020]), uses a global and a local plan for motion-planning when complete environment information is not avail- able. Global plan is computed less frequently, observing the large obstacles which are a priori known, while the local plan is computed frequently as smaller obstacles are incrementally detected on the global path. • Motion planning from a hybrid systems perspective: Motion planning re- quires consideration of both continuous dynamics and discrete dynamics. The con- tinuous dynamics arise from the robot’s mechanical design, whereas the discrete dy- namics arise from its internal logic/timer to resolve tasks and interactions with the 5 CHAPTER 1. INTRODUCTION environment. Motion planning for continuous-time systems (known as kinodynamic planning aims to generate trajectories) and discrete-time systems (aims to generate discrete poses) are well-studied problems (LaValle [2006]). Most existing algorithms (LaValle [1998]) incrementally construct a search tree (rooted in the initial state) in the state space by adding random samples while trying to find a path that connects the initial and final states in the search tree. These algorithms use operations, such as concatenation, well-defined for continuous and discrete-time systems. However, the complex domain structure inherent in hybrid systems makes it exceedingly diffi- cult to establish well-defined operations on trajectories. Hybrid systems may exhibit the following behaviors: 1) evolves continuously, 2) makes discrete transitions (or jumps) all the time, 3) evolves continuously and exhibits one or multiple jumps at times, or 4) exhibits Zeno behavior (where a system undergoes an infinite number of discrete transitions within a finite amount of time. HyRRT (Wang and Sanfelice [2024]) attempts to provide mathematical definitions of operations and their analysis for motion planning algorithms for hybrid systems. (Bhatia et al. [2010]) considers the problem of motion planning for mobile robots with nonlinear hybrid dynam- ics. It uses a multi-layered framework for solving planning problems. At the higher level, it employs a discrete abstraction of the hybrid system and suggests high-level plans, and at the lower level, a sampling-based planner uses the dynamics of the hybrid system and the suggested high-level plans to explore the state-space for feas- ible solutions. (Plaku et al. [2009]) approach motion-planning from a hybrid system falsification perspective, where robot motion planning is viewed as a search problem for a witness trajectory that satisfies certain invariants, such as ensuring that the robot motion respects dynamics constraints and avoids collision with obstacles. 1.2 Explainable AI Planning (XAIP) As AI is increasingly being adopted into application-solutions, the challenge of supporting interaction with humans is becoming more apparent. Partly, this is to support integrated working styles, in which humans and intelligent systems cooperate in problem-solving. It is also a necessary step in building trust with AI-based systems. The need for explainable AI (XAI) first became prominent in machine learning, where the lack of understandable decision rationales is particularly daunting. XAI concerns the challenge of shedding light on opaque models in contexts for which transparency is important, i.e., where these mod- els could be used to solve analysis or synthesis tasks. While XAI at large is primarily concerned with learning-based approaches, model-based approaches are well-suited for ex- planation, and explainable AI planning (XAIP) (Fox et al. [2017], Kambhampati [2019]) can play an important role in addressing complex decision-making procedures. One of the recent developments towards this end is the establishment of the XAIP Workshop 3 at the International Conference on Automated Planning and Scheduling (ICAPS), the premier conference in the field. From its inception, XAIP has garnered increasing interest due to its role in designing explainable systems that bridge the gap between theoretical and 3 https://kcl-planning.github.io/XAIP-Workshops/ 6 1.2. EXPLAINABLE AI PLANNING (XAIP) algorithmic planning literature and real-world applications (Chakraborti et al. [2020]). 1.2.1 XAIP Perspectives XAIP can be discussed from different perspectives involved while considering the persona of the explainee (Chakraborti et al. [2020]) as shown in Figure 1.1. The perspective groups below are to some extent based on the work in (Gade et al. [2020], Saeed and Omlin [2023]), which are true for XAI in general but also acknowledged to be crucial to the XAIP scene as well (Langley [2019], Chakraborti et al. [2020]). Figure 1.1: XAIP perspectives • End user: These are the individuals who will use or be impacted by the implement- ation of new technology and processes. They interact with the system in the form of a user. For example, this may be a passenger on an autonomous car, or a human teammate in a human-robot team (Chakraborti et al. [2019b]) who is affected by, or is a direct stakeholder in the agent’s plans, or a user who collaborates with an automated planner in a decision support setting (Grover et al. [2020]). • Domain designer: XAIP can be important in checking that the system adheres to desired properties in the pre-deployment phase. It helps a domain expert design better models by explaining the system’s behaviour while interacting with the en- vironment during test runs. For example, a designer of goal-oriented conversation systems (Sreedharan et al. [2020]) can take help from XAIP during the model ac- quisition process. • Algorithm designer: Algorithms play a crucial role in system performance, and XAIP could be useful as a feedback system in designing better algorithms. The role of an algorithm designer is distinct and may not even have any overlap in expertise with a domain designer (Sreedharan et al. [2020]): e.g., in the context of automated planning systems, this could be someone who works on an informed search. 7 CHAPTER 1. INTRODUCTION • Regulations: As AI-based autonomous systems are being used in many areas of our daily lives, it could result in unacceptable decisions being taken by such systems in certain situations. Such decisions need an explanation to the user. Especially those that may lead to legal effects. For example, suppose that an AI system rejects one’s application for a loan. In that case, the applicant has the right to request justifications behind that decision to ensure that the system adheres to the laws and regulations (Samek and Müller [2019]). Thus, it poses a new challenge to the legislation. The General Data Protection Regulation (GDPR) 4 of the European Union establishes regulations for what is called the ’right to explanation’, by which a user is entitled to request an explanation of the decision made by the algorithm that considerably influences them (Goodman and Flaxman [2017]). • Business: Winning user trust in any AI-based system is a major challenge for the industry. XAIP plays a crucial role in building a robust, transparent, and trust- worthy system that can explain its behaviour to the end user (Chakraborti et al. [2020]). It helps in gaining the user’s trust. However, it can increase development and deployment costs. 1.2.2 Relevant XAIP concepts A planning problem Π is a sequential decision-making problem for an AI agent. We can define Π as a transition function δ Π : A×S → S× R, where A is the set of actions (or capabilities) available to the agent, and S is the set of states in which it can be. The real number R denotes the transition cost. Thus, a transition δ ∈ δ Π defines the agent’s behaviour in terms of actions, i.e., the prerequisites of actions and how it changes the state of the world. The planning algorithm A solves Π subject to a desired property τ to produce a plan φ, i.e. A : Π× τ 7→ φ. Here, τ may represent different properties such as soundness, optimality, and so on. A plan φ can be defined as a sequence of actions ⟨a 1 ,a 2 ,...,a n ⟩, a i ∈ A that transforms the current state I ∈ S of the agent to its goal G∈ S, that is, δ Π (φ, I) = ⟨G, P a i ∈φ c i ⟩, where c i is the cost of applying the action a i . • Explanation process in XAIP proceeds with a question from the explainee about a current solution or about unsolvability in the case when there is no solution for a given planning problem Π, and the explainer (the XAIP system) comes up with an explanation for it (Chakraborti et al. [2020]): – Q: "Why φ?" or "Why not φ ′ ?" - when Π is solvable. Here, φ ′ is an alternate plan or a foil (Miller [2019]) which may be explicitly, implicitly, or even partially stated in the questions. Examples of foils would be: ∗ “Why a ∈ φ?” is a partial foil where all plans with action a in them are the foils. ∗ The original question “Why φ?” where the implicit foil is “as opposed to all other plans φ ′ ”. 4 https://w.privacy-regulation.eu/en/r71.htm 8 1.3. XAIP IN HYBRID SYSTEMS – Q: "Why no plan?" - when Π is unsolvable. It is interesting when a planning problem is unsolvable for a planner. This may happen for two reasons: the planning problem is unsolvable and there is no feasible solution possible, or the planner cannot solve the problem due to its limitation and does not know whether the problem is solvable. In both cases, the XAIP system should provide reasonable explanations to the user. – A: An explanation E ensures that the explainee can compute and verify that: ∗ A : Π× τ ̸ E 7−→ φ ′ , or A : Π× τ E 7−→ φ ′ , but φ≡ φ ′ or φ > φ ′ (the comparison criterion may be cost, preferences, etc.). ∗ A : Π× τ E 7−→∅, when the planning problem is unsolvable. The Q & A continues until the explainee is satisfied, as (Smith [2012]) highlights that this approach to explanation is an iterative process. • Properties of explanation: The need for explanations arises when there is a mismatch between a proposed solution or the absence of a solution and the user’s expectation. This might be because the user may not have formed an expected plan or because a plan was successfully constructed but does not match the proposed solution. The explanations attempt to bridge the gap between these mismatched positions. Explanations can be local, regarding a specific plan and its properties (Ribeiro et al. [2016], Dey et al. [2024]), or global, focusing on the assumptions on which the plan rests, the process by which it was constructed, or how the planning system works in general (Kim et al. [2018]). (Miller [2019]) provides an insightful view on explanations from the social sciences. It outlines three key properties for consideration: social in being able to model the expectations of the explainee, se- lective in being able to select explanations among several competing hypotheses, and contrastive in being able to differentiate properties of two competing hypotheses. The contrastive property, in particular, has received much attention (Hoffmann and Magazzeni [2019], Miller [2021]) in the XAIP community. Abstraction is a useful concept in which explanations given on an abstract model of a complex decision- making system are more helpful to the explainee (Sreedharan et al. [2019], Ribeiro et al. [2016]). 1.3 XAIP in Hybrid Systems Planning for hybrid systems is an important area in AI planning, mainly motivated by the need to deal with real-world problems. Hybrid systems closely model these problems involving continuous and discrete behaviour. Such systems, also known as Cyber-Physical Systems, have hybrid dynamics, subject to (continuous) physical effects and controlled by (discrete) digital equipment. These systems are complex as they need to model complex domains involving continuous nonlinear methods, differential equations, fluid dynamics, etc. As a result, the behaviour of such systems is also complex. Therefore, any automated plan in these domains needs explanations to the end user. XAIP plays a crucial role in 9 CHAPTER 1. INTRODUCTION Research Directions in XAIP for Hybrid Systems Explaining Plans Explaining Unsolvability Figure 1.2: The proposed organization to discuss the challenges and research direc- tions in XAIP for hybrid systems. designing and planning in the pre-deployment phase for such a system and explaining its behaviour to the users afterward. The current research directions in XAIP for hybrid system planning problems can be divided into two classes, as shown in Figure 1.2. 1.3.1 Explaining Plans in Hybrid Systems Explaining plans is the oldest branch of XAIP. It aims to help humans understand the inner workings of a plan suggested by the AI system (McGuinness et al. [2007], Khan et al. [2009], Bidot et al. [2010], Sohrabi et al. [2011], Seegebarth et al. [2012], Bercher et al. [2014], Nothdurft et al. [2015]). Different XAIP perspectives (discussed in Section 1.2.1) will require different types of explanations. These explanations can be classified into two primary classes: algorithm-based explanations and model-based explanations (Chakraborti et al. [2020]). Algorithm-based explanations attempt to explain the innards of the underlying plan- ning algorithm and are generally useful for experts (i.e., algorithm designers). For example, (Magnaguagno et al. [2020]) developed a cloud-based planning tool with state-space visu- alization to illustrate the operation of the planning process and how the domain dynamics evolve during the execution of the plan. It can also visualize fail planning instances, which is useful in debugging. On the contrary, the majority of works in XAIP consider model-based explanations. This category consists of algorithm-agnostic methods for generating explanations since the properties of a solution can be evaluated independently of the method used to come up with them. Unlike debugging, requiring detailed algorithm-specific analysis, end users are primarily interested in model-based, algorithm-agnostic explanations so that services (Cashmore et al. [2019]) can be built around it. 1.3.2 Explaining Unsolvability in Hybrid Systems A special kind of “why” question is: “why didn’t you find a solution to this problem?” (Krarup et al. [2021b]). While there has been a lot of research on generating explanations for planning problems, most of the earlier works in explanation generation have focused on explaining why a given plan or action was chosen (Chakraborti et al. [2017, 2019a], Krarup et al. [2021b]). However, explaining the unsolvability of a given planning problem remains a largely open and understudied problem. The recent works that focus on ex- plaining the unsolvability of planning problems have primarily concentrated on generating 10 1.4. POPULAR APPROACHES TO EXPLANATION GENERATION IN XAIP certificates or proofs of unsolvability (Eriksson et al. [2017, 2018]), these approaches, which are more oriented towards automatic verification, may fall short in adequately explaining unsolvability in complex planning domains. (Göbelbecker et al. [2010]) argues that excuses can be made for why a plan cannot be found by identifying counterfactual alterations to the original planning task to make it solvable. (Sreedharan et al. [2019]) use hierarchical model abstractions to generate the reason for unsolvability of planning problems. These hierarchical model abstractions relax a planning problem until a solution can be found. Then, they look for landmarks of this relaxed problem that cannot be satisfied in less re- laxed versions of the problem. The unsatisfiability of these landmarks provides a succinct description of critical propositions that cannot be satisfied. Eifler et al. [2020] derives plan properties that must be exhibited by all possible plans that could serve as explanations in case of unsolvability. However, most of these works are on classical planning problems. To the best of our knowledge, not much work addresses this issue for the hybrid system planning problems. 1.4 Popular Approaches to Explanation Generation in XAIP In this section, we discuss a few important approaches to explanation generation in XAIP upon which we build our work in this thesis. However, the majority of the works in this direction are on discrete systems; we will highlight those that are relevant to hybrid systems whenever possible. 1.4.1 Contrastive Explanation Approach When there is a mismatch between a proposed plan from an automated planner and the user’s expectation, reconciliations are often required. The discrepancy could arise from either the user’s failure to develop a predictive plan or a mismatch between their pre- dicted plan and the one presented. Explanations serve to reconcile these differences. In (Fox et al. [2017]), the authors discuss how to achieve the goal of providing reasonable answers to user questions through explanations. Among the explanation properties high- lighted in Section 1.2.2, contrastive explanations received significant research interest in the literature. (Fox et al. [2017], Hoffmann and Magazzeni [2019]) highlights the role of contrastive questions in XAIP. Below, we discuss a few terminologies: • Contrastive questions: An important type of question in XAIP takes the form: – “Why action A instead of action B?” (Mueller et al. [2019]) has shown that users tend to ask "why" questions when seeking explanations about a specific part of the plan, referred to as local questions, while "how" or "what" questions are asked when seeking explanations about the plan as a whole, referred to as global questions. Insights from social sciences suggest that these "why" questions are often contrastive (Miller [2019]). • Contrastive plans: A contrastive plan or a hypothetical plan incorporates the user-suggested foil. This is done by first deriving the constraints from the contrast- 11 CHAPTER 1. INTRODUCTION ive question posed by the user. These constraints are then imposed on the planning system such that any plan generated by the planner must adhere to the user sugges- tions. • Contrastive explanations: When a contrastive question is posed about a plan, a contrastive explanation can be given, highlighting how the original plan differs from an alternative plan that incorporates the user’s suggested foil. Offering contrast- ive explanations is both an effective way to improve understanding and a simpler approach than providing a full causal analysis (Miller [2019]). Furthermore, their inherent structure facilitates comparisons between the original plan and the one incorporating the user’s suggested alternative. (Eifler et al. [2020]) provides contrastive explanations by deriving plan properties that must hold if a contrast case was in the plan. (Krarup et al. [2019]) focuses on local explanations of temporal and numeric planning problems, formally describing the compilation from user questions to constraints in a PDDL2.1 planning setting, and explaining why a planner has made a certain decision. (Kim et al. [2019]) introduce a Bayesian inference framework of linear temporal logic specifications to generate differences between plan traces for in- ferring contrastive explanations. (Bercher et al. [2014]) gives contrastive explanations to user queries to help them assemble a home theatre by providing the reasons for an ac- tion’s inclusion in the plan. (Zhao and Sukkerd [2019]) discuss how such approaches can have interesting applications in cyber-physical systems (CPSs). Contrastive explanation approach has also been applied to explain machine learning based models. Dhurandhar et al. [2018] proposes a contrastive explanations method (CEM) to generate explanations for differentiable models such as deep neural networks, where one has complete access to the model. In Dhurandhar et al. [2019], a model agnostic contrastive explanations method (MACEM) is proposed to generate contrastive explanations for any classification model where one can query only the class probabilities for a desired input. 1.4.2 Model Reconciliation Approach In most human-AI interaction scenarios, humans often have their own preconceived no- tions and expectations regarding a system (Carroll and Olson [1988]), potentially leading them to evaluate plans based on their own models, which may not align with the sys- tem’s assessment of the result or quality. In this context, a recurring theme is the model reconciliation problem (MRP) (Chakraborti et al. [2017]), a paradigm that empowers an agent (the explainer) to generate explanations by considering the “mental model" of the human user (the explainee), drawing on the theory of mind (ToM) (Premack and Woodruff [1978]) from human psychology. These model-based explanations aim to explain a plan by transferring a minimum number of necessary updates from the agent’s model to the user, effectively bringing the model of the user closer to the agent’s model (Chakraborti et al. [2017], Sreedharan et al. [2018]). The process of explanations is thus a reconciliation of the agent’s model Π A and the human mental model Π H so that both can agree on the 12 1.4. POPULAR APPROACHES TO EXPLANATION GENERATION IN XAIP property τ of the decision being made. The model reconciliation process requires that: Given : A : Π A × τ 7→ φ Π H +E 7→ ˆ Π H such that A : ˆ Π H × τ 7→ φ. Empirical evidence suggests that model reconciliation is a natural and effective approach for explaining classical planning problems to humans (Chakraborti et al. [2018], Zahedi et al. [2019]). Using map visualizations of a planning problem, these studies specifically showed that human users understood and believed model reconciliation explanations were necessary for explaining (classical planning) plans. (Chakraborti et al. [2017]) assumes that the user’s model is known and proposes a method to generate minimally complete and monotonic explanations that update the user’s model to accept a plan. Conversely, (Sreedharan et al. [2018]) produces conformant explanations applicable to multiple po- tential user models when the exact user model is unknown. Both of these approaches consider only optimal solutions in classical planning. An AI agent here creates the best possible plan based on its model Π A , and a human interprets this plan using their own understanding Π H . Explanations become necessary when the AI’s “best” plan isn’t also the best plan from the human’s perspective. However, the necessity of optimal plans for explanation is generally questionable, and optimal planning for hybrid systems is unde- cidable (Alur et al. [1995b]). (Kulkarni et al. [2019]) compute the plan distances between the agent’s plan and a user expected one. Thereafter, it uses a machine learning based re- gression model on human-annotated plans and the plan distances to compute explicability distance that is then used as the heuristic to search for explicable plans. 1.4.3 Logic-Based Approach A classical planning problem can be translated into a propositional satisfiability (SAT) problem with formulas representing the initial state, goal, and action dynamics over a maximum of n time steps, where n is usually the upper bound on the horizon of plan length (Kautz and Selman [1992]). Similarly, a hybrid system planning problem can be formalized as an SMT (Satisfiability Modulo Theories) formula in first-order logic interpreted in the theory of quantifier-free linear real arithmetic (Cashmore et al. [2016b]). These logic-based frameworks offer attractive features that are desirable in explanation generations, such as Expressivity and Traceability (Vasileiou et al. [2022]). • Expressivity refers to the expressive power of logical languages to describe various phenomena in a principled and axiomatic way and the ability to distinguish between certain structures defined in them. For instance, propositional logic uses a finite set of propositions P = p 1 ,...,p n and models M = μ 1 ,...,μ k representing truth assignments to P. In the case of a classical planning problem Π, each proposition p i encodes states, actions, and transitions up to a time horizon n. Each model μ j assigns Boolean values to these propositions, describing the truth of states and actions at a specific time step within Π’s execution. A knowledge base of these propositions can then explain events that occurred during that time. 13 CHAPTER 1. INTRODUCTION • Traceability implies that given a logical description of a problem, it is easy to trace the reasons for particular “behavior". For example, if a knowledge base KB encodes a planning problem Π, a valid plan φ is logically implied by KB. Therefore, deductive inference can trace the reasons for φ’s validity, and these reasons, expressed in the logic of KB, can explain why φ is valid. (Vasileiou et al. [2022]) presents a logic-based extension to MRP problems based on know- ledge representation and reasoning for mixed discrete-continuous domains. It provides a framework for finding a subset of the knowledge base of the agent with which to reconcile the human knowledge base for explanations. (Bercher et al. [2014]) uses a logic-based approach to answer the question “why the action a∈ φ?" by deducing a causal link chain originating at a that can be traced to the goal. There has been a long history of us- ing such information to characterize plans in the context of plan modification and reuse. (Seegebarth et al. [2012]) presents a formal approach to plan explanation. Information about plans is represented as first-order logic formulae, and explanations are constructed as proofs in the resulting axiomatic system. 1.4.4 Divide and Conquer Strategy A well-known insight into human thinking and problem solving is that humans tend to decompose a problem into sub-problems that help in progressively converging towards the goal. Many AI systems mimic this notion in the way they solve problems. For example, the main feature of the pioneering automated theorem prover logic theorist is the use of a problem-subproblem hierarchy (Newell and Simon [1956]). This has been a popular approach in many other domains, such as robotics (Krogh and Feng [1989]) and AI (Sutton et al. [1999]) apart from planning (Hoffmann et al. [2004], Lipovetzky and Geffner [2012], Richter et al. [2008]). Authors in (Hoffmann et al. [2004], Lipovetzky and Geffner [2012]) find sub-problems for a solvable planning problem of the discrete domains in terms of ordered landmarks. Landmarks are facts given as propositional formulas that must be true at some point in every valid solution plan. An innovative technique for the identification of subproblems relevant to explaining the unsolvability of a planning problem in domains with discrete dynamics has been proposed in (Sreedharan et al. [2019]). 1.5 Decidability of Planning Problems in Hybrid Systems Verifying the solvability of planning problems is undecidable for hybrid systems in general (Alur et al. [1995b]). When a planner fails to generate a valid plan for a problem, it cannot be asserted whether it is due to the underlying undecidability or that the problem is insol- uble. However, some special classes of problems within the hybrid system are known to be decidable. The state reachability problem for timed automata (TA) is decidable (Alur and Dill [1994]), which makes this an interesting sub-class of linear hybrid automata (LHA). Its complexity class is PSPACE-complete. However, some problems, like the general language inclusion problem and the determinisability problem for certain types of TA, are unde- cidable (Clemente et al. [2020]). Decidability results for TA are generalized to multirate 14 1.6. THESIS OUTLINE automata (MA), another subclass of LHA, with variables that run at any constant positive slopes (Nicollin et al. [1992], Alur et al. [1997]). Its complexity class is PSPACE-complete. The decidability problem for an initialized rectangular automata (RA) is decidable under two restrictions: 1) whenever the activity of a variable changes, the value of the variable is reinitialized; 2) the values of two variables with different activities are never compared (Henzinger et al. [1998]). Its complexity class is PSPACE. A RA is a multirate automaton (MA) if act(v) (flow) is a singleton for all vertices v of RA. An MA is a timed automaton (TA) if each variable of MA is either a clock or a memory cell. A variable is a memory cell if it has a slope of 0 at every vertex of RA. A variable is a clock if c has a slope of 1 at every vertex. A two-slope variable with slopes 0 and 1 is a stopwatch. The reachability problem is undecidable in TA for a single stopwatch (Henzinger et al. [1998]). The decision problems for classical planning, also known as PlanSAT problems, pose the question of whether there exists any plan that solves a planning problem, and are decidable (Russell and Norvig [2020]). The proof follows from the fact that the number of states is finite. However, introducing function symbols expands the state space to in- finity, rendering the problem only semi-decidable. This means we can devise an algorithm that correctly solves any solvable instance but might run indefinitely for unsolvable ones. Notably, the Bounded PlanSAT problem retains its decidability even when function sym- bols are included. For the formal proofs of these claims, refer to (Ghallab et al. [2004]). PlanSAT and its bounded variant both reside within the complexity class PSPACE, a significantly more challenging class than NP. Problems in PSPACE are solvable by a deterministic Turing machine using a polynomial amount of memory. Even under sub- stantial constraints, these problems remain hard; for instance, eliminating negative effects of actions still leaves them NP-hard. Interestingly, if we further restrict the problems by disallowing negative preconditions, PlanSAT’s complexity drops to P. Verifying the unsolvability of planning problems in the temporal-planning (TP) domain is decidable under the ANSO (action non-self-overlapping) assumption (Panjkovic et al. [2022]). Its complexity class is PSPACE-complete. In general, the complexity of TP depends on the domain of time. If time is interpreted as a discrete quantity, TP is EXPSPACE-complete (Rintanen [2007]). Instead, if time is interpreted as a dense quantity, TP is undecidable (Gigante et al. [2022]). 1.6 Thesis Outline The research directions discussed in the preceding sections motivate the research plan presented in this thesis. The rest of the chapters are organized as follows: • In Chapter 2, we first present the background concepts of hybrid systems essential to this thesis. It then presents a motion-planning problem framed from a hybrid system’s perspective. The primary goal of this exercise is to familiarize ourselves with the planning literature. We propose a motion-planning algorithm for a robot in an unknown environment based on iterative constraint-solving. An integrated software framework consisting of a simultaneous localization and mapping module, a motion 15 CHAPTER 1. INTRODUCTION planning module, and a plan execution module, specifically designed for a lizard- inspired quadruped robot, has been developed in collaboration. The techniques of motion planning and plan execution are the contributions of this thesis. We present performance of the the algorithm for planning tasks in simulation settings. The contents of this work have been published in Sarwar et al. [2021]. • In Chapter 3, we propose a contrastive plan explanation framework for hybrid system planning problems that builds a hypothetical model from the user’s questions and generates a hypothetical plan to explain by highlighting its contrast with the original plan. We discuss a few classes of contrastive questions. The proposed framework, based on the re-model and re-plan idea, advocates constructing a hypothetical model for each of these contrastive questions so that any valid plan, if generated on this model, will imitate the contrastive question. In addition, we provide a framework that verifies unsolvable planning problems (no plan instances) and proves the absence of a plan using bounded reachability analysis. The contents of this work have been published in Sarwar et al. [2023b] and Sarwar et al. [2020]. • In Chapter 4, we present a contrastive explanation tool for plans in hybrid domains. The tool consists of (1) A web-based interactive GUI for selecting questions, viewing contrastive plans and the generated explanations, and (2) A back-end implementing an iterative re-modeling and re-planning algorithm. The tool offers a collection of contrastive questions over a plan for users to select. An explanation is produced by contrasting the original plan against an alternative that meets the user’s expecta- tion implicit in the question. The tool has the provision to contrast with the best alternative that the underlying planner can generate in terms of plan metrics. The contents of this work have been published in Dey et al. [2024]. • In Chapter 5, we propose a model reconciliation framework for explaining unsolvab- ility of hybrid system planning problems. We assume that the agent has a complete model of the environment, while the human has a partial or erroneous model and expects a plan for the planning problem when there is none. The explanation prob- lem is presented as a process of continuous reconciliation between these two entities (agent and human) to make the human domain consistent with that of the agent. We use a mix of graph traversal and path analysis, along with Linear programming, to carry out the reconciliation process. In particular, we use the concept of Irreducible Infeasible Sets (IIS) to generate explanations. The contents of this work have been published in Sarwar et al. [2023a]. • In Chapter 6, we present a framework to decompose an unsolvable planning task in a hybrid system into sub-problems for analyzing and explaining unsolvability. In particular, for a given unsolvable planning problem, we propose a novel method to the waypoint identification problem by casting it to an instance of the longest common subsequence problem. As waypoints appear on every path from the source to the planning goal, this work envisions such waypoints as sub-problems of the planning problem, and the unreachability of any of these waypoints as an explanation for the 16 1.7. LIST OF CONTRIBUTIONS unsolvability of the original planning problem. The contents of this work have been published in Sarwar and Ray [2025]. • Finally, in Chapter 7, we summarize the presented methods, emphasizing their use- fulness in explanation generation and developing XAIP systems from the hybrid system perspectives, and our observations on major issues of hybrid system plan- ning problems such as scalability, complexity, and decidability. We conclude by outlining future research directions of this work. 1.7 List of Contributions The following are the primary contributions in terms of novel frameworks that have been developed as part of this thesis: • A robotic software framework that integrates SLAM, motion planning, and control for autonomous navigation for a lizard-inspired quadruped robot. We present a mo- tion planning algorithm for the robot through constraint-solving. • A Contrastive Plan Explanation Framework for explaining plans for hybrid system planning problems through the re-modeling and re-planning strength of the frame- work for iterative users’ questions. • A framework for verifying no-plan instances through bounded reachability analysis of the planning problem. • A web-based interactive Contrastive Explanation Generation Tool that facilitates experimenting with the explanation generation process for hybrid system planning problems with integrated state-of-the-art hybrid system planners. • A path-based Continuous Model Reconciliation framework for explaining unsolvab- ility of the hybrid system planning problems through a mix of graph traversal and path analysis. • A novel approach to decompose an unsolvable planning task into sub-problems through waypoint identification and explanation generation for the unsolvability via the reachability analysis of the sub-problems. 17 CHAPTER 1. INTRODUCTION 18 Chapter 2 Hybrid System Planning Chapter Abstract: This chapter first presents the background concepts of hybrid sys- tems essential to the remainder of this thesis. Building upon this background, we introduce a motion-planning problem in robotics from a hybrid system perspective. This leads to a case study on the autonomous navigation of a lizard-inspired quadruped robot in an un- known environment, where we present an integrated software framework comprising three key modules: a simultaneous localization and mapping (SLAM) module utilizing visual odo- metry, a motion-planning module based on constraint-solving, and a plan-execution module designed specifically for the robot. To demonstrate the framework’s efficacy, we present the results of several navigation tasks conducted in various indoor simulation settings, utilizing a specific model of the quadruped robot. H ybrid systems are controllable physical systems (such as robots or production plants) that combine discrete and continuous behavior. This dual nature is modeled by integrating continuous dynamics (described by time derivatives over state variables, such as ̇x = v) with discrete dynamics (instantaneous state changes). The standard modeling techniques include - PDDL+ (Planning Domain Definition Language +) (Fox and Long [2006]): An extension of PDDL used in planning to represent continuous processes/events alongside discrete actions, and Hybrid Automata (HA) (Alur et al. [1993, 1995a]): A formal framework that models hybrid systems by combining discrete state transitions with continuous dynamics, often used for verification and analysis. The system’s behavior is modeled by three main components: • Processes: Dictate the continuous dynamics, often using differential equations (ef- fects are sets of time-derivative functions). • Events: Formalize discrete changes that happen spontaneously in the environment when their conditions are met. • Actions: Formalize the agent’s decisions and what it can actively do (effects are assignments like x := ξ). 19 CHAPTER 2. HYBRID SYSTEM PLANNING Planning in hybrid systems is challenging for planners to solve due to several inter- secting factors (Piotrowski et al. [2016]): • Undecidability: Continuous variables cause the reachability problem to become un- decidable (Alur et al. [1995b]). • Search Space Explosion: The combination of discrete state variables causing state explosion and complex system dynamics (often involving non-linear behaviors) res- ults in immense search spaces. The goal of a hybrid system planning problem remains to find a timed plan of actions that successfully transitions the system from an initial state to a goal state. Section 2.1 discusses background concepts of hybrid systems and planning. Section 2.2 introduces the motion-planning problem in robotics from a hybrid system perspective. Section 2.3 presents a case study of autonomous navigation in an unknown environment. 2.1 Background This section provides an overview of the background concepts essential to this thesis. More detailed definitions will be introduced as needed in subsequent chapters. We begin with introducing the hybrid automaton, a mathematical model used to describe hybrid systems. Definition 2.1.1 A hybrid automaton (HA) is a seven tuple HA= Loc, V ar, Flow, Init, Lab, Edge, Inv where: • Loc is a finite set of vertices called locations. • V ar is a finite set of real-valued variables. A valuation v for the variables is a function that assigns a real-value v(x) ∈ R to each variable x ∈ V ar. We write V for the set of valuations. • Flow is a mapping from each location l ∈ Loc to a set of differential equations ̇x = f (x 1 ,...,x |V ar| )| x∈ V ar, where ̇x denotes the rate of change of variable x. • Init is a tuple ⟨l ini ,S⟩ such that l ini ∈ Loc and S ⊆ V . • Lab is a finite set of labels. • Edge is a finite set of transitions e = (l,a,g,r,l ′ ), each consisting of a source location l ∈ Loc, a target location l ′ ∈ Loc, label a ∈ Lab, a guard g ⊆ V and a reset map r : R |V ar| → R |V ar| . • Inv is a mapping from each location l∈ Loc to an invariant Inv(l)⊆ V . A state in an HA is a pair (l,v) consisting of a location l ∈ Loc and a valuation v ∈ V . The locations Loc and the transitions between them via the transitions in Edge model the discrete dynamics, whereas Flow(l) models the continuous change in a hybrid system. Inv(l) are constraints on the HA states requiring that v ∈ Inv(l), for every state (l,v) of the HA. A HA behaviour is defined by a run: 20 2.2. MOTION PLANNING IN ROBOTICS FROM A HYBRID SYSTEM PERSPECTIVE Definition 2.1.2 (Run) A run of a hybrid automaton is a sequence (ℓ 0 ,x 0 ) τ 0 −→ (ℓ 0 ,y 0 ) e 0 −→ (ℓ 1 ,x 1 ) τ 1 −→ (ℓ 1 ,y 1 ),..., e N−1 −→ (ℓ N ,x N ) τ N −→ (ℓ N ,y N ) such that for all i = 0,...,N − 1, (i) (ℓ 0 ,x 0 ) ∈ Init; (i) in each step i, the labeling function e i (l) maps the start location of the edge e i to ℓ i and e i (l ′ ) maps the end location of the edge e i to ℓ i+1 , where ℓ i ,ℓ i+1 ∈ Loc. (i) ∀t ∈ [0,τ i ], flow ℓ i (x i ,t) ∈ Inv(ℓ i ); (iv) flow ℓ i (x i ,τ i ) = y i ; (v) y i ∈ e i (g), where e i (g) maps to the guard g i of the edge e i ; (vi) x i+1 = e i (r)(y i ) and y N = flow ℓ N (x N ,τ N ) such that ∀t ∈ [0,τ N ], flow ℓ N (x N ,t) ∈ Inv(ℓ N ). The times τ i are called the dwell times of the system in respective locations ℓ i . 2 A planning problem in a hybrid system requires two components: a domain description and a problem description. The domain is modeled using standard techniques (such as PDDL+ or Hybrid Automata) and defines the system’s dynamics. The problem description configures the initial and goal states for the specific planning task. We formally define a hybrid system planning problem as follows: Definition 2.1.3 A planning problem Π for a hybrid system is a pair (Dom, Prob), where Dom defines a planning domain represented as hybrid automata HA/PDDL+ domain, and Prob represents a problem description defining the initial and goal configurations.2 A system’s state evolves under two dynamic modes: 1. Continuous Evolution (Time Passage): State variables evolve according to the flow defined in the current location. 2. Discrete Transition: An instantaneous change occurs when a labeled action/trans- ition is applied. This requires the state’s valuation to satisfy the transition’s guard condition. The valuation of the resulting state is then dictated by the transition’s reset map. Given these dynamics, we now define a plan for a planning problem Π as follows: Definition 2.1.4 A plan for a planning problem Π is a tuple ⟨ λ n , makespan ⟩ where λ n is a finite sequence of n pairs ⟨t i ,a i ⟩. In the pair, t i ∈ R + is the time instance of executing the action a i ∈ Lab. In the sequence, t i is non-decreasing. The makespan is the duration of the plan.2 A planning problem Π is solvable if a valid plan exists; otherwise, it is unsolvable. 2.2 Motion Planning in Robotics from a Hybrid System Perspective Motion planning 1 , also known as path planning or the navigation problem, is a computa- tional task that involves finding a valid sequence of configurations to move an object from a 1 https://en.wikipedia.org/wiki/Motion_planning 21 CHAPTER 2. HYBRID SYSTEM PLANNING starting point to a destination. This concept is used in fields like computational geometry, computer animation, robotics, and computer games. For example, navigating a mobile robot to a distant waypoint inside a building, while avoiding walls and stairs, is a task that motion planning addresses. A motion planning algorithm accepts these task descriptions as inputs and outputs the speed and turning commands for the robot’s wheels. These algorithms tackle intricate scenarios, such as multi-joint robots (like industrial arms), ob- ject manipulation tasks, diverse constraints (e.g., a car’s forward-only movement), and uncertainties in both the environment and robot models. Motion planning has several robotics applications, such as autonomy, automation, navigation, and robotic surgery etc. It requires consideration of both continuous dynamics and discrete dynamics (Wang and Sanfelice [2024]). For instance, the position and velocity of a collision-resilient multi- copter system in (Zha and Mueller [2021]) evolves continuously in open space, yet exhibits discrete state changes upon collision with a wall. In these situations, neither a purely continuous nor a purely discrete-time model is adequate for capturing the system’s beha- vior. A hybrid system model is therefore essential, capable of capturing purely continuous, purely discrete, and combined behaviors. The continuous dynamics arise from the robot’s mechanical design, whereas the discrete dynamics arise from its internal logic/timer to resolve tasks and interactions with the environment. In this chapter, we present an in- tegrated software framework for the autonomous navigation of a lizard-inspired quadruped robot (Nishad et al. [2021]) designed for a stealth surveillance operation 2 , while emphasis remains on the motion-planning problem of the robot. We design a hybrid controller that controls the navigation of the robot along a projected path. 2.3 A Case Study for Autonomous Navigation in an Un- known Environment Autonomous navigation is of central importance in robotics, with increasing use of ro- bots in various applications, such as search and rescue operations (Jennings et al. [1997]), warehouse automation (Bertazzi and Speranza [2013]), surveillance (Portugal and Rocha [2016]), etc. Many of these applications require planning and control in unknown envir- onments. Autonomously navigating a robot in an unknown scene comprises tasks such as: mapping and localization from the percepts received via sensors, safe motion-planning under partial knowledge of the scene, and finally, plan execution utilizing the available mo- tion primitives of the robot. Though many software systems support these sub-tasks, there is a lack of integrated software that supports all the sub-tasks for autonomous navigation. In this work, we present a robotic software architecture comprised of SLAM (simul- taneous localization and mapping), motion-planning, and control modules 3 . The control module is mainly designed for driving a lizard-inspired quadruped robot designed for a stealth surveillance operation. We show the utility of this robotic software in addressing various navigation tasks, emphasizing the motion-planning and control parts. To exemplify 2 This work is a part of the Science and Engineering Research Board (SERB) project with File No. IMP/2018/000523 for developing a surveillance robot for security operations. 3 This work is part of a collaboration. Motion planning and control modules are part of this thesis. 22 2.3. A CASE STUDY FOR AUTONOMOUS NAVIGATION IN AN UNKNOWN ENVIRONMENT Figure 2.1: Stealth surveillance system for monitoring hostage scenario one such application, consider a hostage scenario, depicted in Figure 2.1, where navigation and map generation in an unknown hostage environment by lizard-like robots enable the security forces to take timely measures stealthily. Though navigation in robotics is a heav- ily explored area, navigation in an unknown environment remains relatively less explored and calls for a different approach than the classical algorithms. To summarize, the main contributions of this work are: • We design an integrated robotic software framework for autonomous navigation of lizard-inspired quadruped robot in an unknown environment; • For navigation in unknown environments, the SLAM, motion-planning, and the con- trol components of the software run in synergy to generate a safe path-planning and simultaneously drive the robot to the goal. The SLAM module is not part of this thesis, which has been moved to the preliminaries of this work. • We show the reduction of the motion-planning problem as a constraint satisfaction problem and solve it by a state-of-the-art constraint solver and • Finally, we develop a working prototype of the proposed framework in Coppeli- aSim with support via ROS Interface. We present experimental results on several environments, ranging from simple to complex, to demonstrate the efficacy of the framework. The rest of the chapter is organized as follows. Section 2.3.1 gives an overview of related works on motion-planning problems. Section 2.3.2 provides preliminaries. In Section 2.3.3, we describe in detail our proposed robotic software framework. Section 2.3.4 presents experimental results in a number of simulation scenarios. Section 2.3.5 concludes the work. 23 CHAPTER 2. HYBRID SYSTEM PLANNING 2.3.1 Related Works Various approaches to address the motion-planning problem exist in the literature, such as RRT (Rapidly-exploring Random Tree) (LaValle [1998]), A ∗ (Hart et al. [1968]), SAT and SMT-based path-planning (Hung et al. [2014], Saha et al. [2014]). Most of the motion- planning algorithms assume that the environment is known and thus can not be used dir- ectly (LaValle [2006]). In (Stentz [1993]), a graph-based approach based on D ∗ algorithm is proposed for path-planning in unknown, partially known and changing environments. Although this model works well in a partially known environment, it is computationally expensive in an unknown environment. Current approaches on sampling-based planning methods like RRT ∗ (Karaman and Frazzoli [2011]), and its many variants have had consid- erable success for kinematic motion-planning. However, the complexity of such methods heavily relies on robot dynamics, and the global computation over the entire state space for high-dimensional systems (like legged robots) in cluttered environments becomes slow. A similar approach, SweepingRRT (Albee et al. [2020]) use a global and a local plan for motion-planning when complete environment information is not available. Global plan is computed less frequently, observing the large obstacles which are a priori known, while the local plan is computed frequently as smaller obstacles are incrementally detected on the global path. 2.3.2 Preliminaries 2.3.2.1 SLAM Module The SLAM module consists of two sub-modules: (1) Localization and (2) Mapping. Loc- alization is responsible for estimating the robot’s pose in a given environment, whereas mapping generates a 2D occupancy grid map of the environment. Localization Localizing a robot in an environment at a time-instance t requires the following: Measurement data z t and Control data u t . Measurement data z t provides information about the robot’s pose p t at time t. Since we are only interested in 2D pose of the robot, we define p t as follows: p t = [x(t),y(t),θ(t)], where (x(t), y(t)) and θ(t) are the estimated position and orientation of the robot at time t. We use an RGB-D camera mounted on the robot to capture RGB image and depth information of the environment. This depth information is then used to extract point cloud for each pixel in the image. Once a point cloud of an image is extracted as its 3D representation, we use Iterative Closest Point (ICP) algorithm (Segal et al. [2009]) to compute z t by transforming the current point cloud so that it will be aligned with the previous point cloud. Control data u t , on the other hand, provides information about how much the robot moves in x and y directions and the robot’s turning angle around its joint. In practice, sensors and actuators in a robot yield uncertainties due to the presence of noise in z t and u t . For example, depth information from an RGB-D camera may have noise due to improper reflection of the depth signal from objects, incorrect image taken when the camera is unstable, etc. Similarly, actions taken by the robot may have noise due to slippage, mechanical errors, etc., which yield uncertainties in control data. To overcome this, we use a sample-based probabilistic 24 2.3. A CASE STUDY FOR AUTONOMOUS NAVIGATION IN AN UNKNOWN ENVIRONMENT approach to estimate the robot’s pose and to generate the map of the environment. In particular, we use Particle Filter algorithm (Thrun [2002a]) which takes control data u t , measurement data z t , and a set S t−1 =p [1] t−1 ,p [2] t−1 ,...,p [K] t−1 of K pose samples at time t− 1 as input and returns robot’s pose p t at time t as output. Mapping Since the robot is capable of walking only on the floor, we are interested in identifying the obstacles that touch the ground and have a height greater than the ground clearance from the robot’s base link. This phase considers the point clouds that satisfy the aforesaid conditions, and the position of these point clouds in discretized 2D space is calculated using the robot’s position. The map is generated in the form of 2D occupancy grid by marking each cell with one of the values from -1, 0, 1, where -1, 0, and 1 denote the cell as unexplored, free, or occupied respectively. Since measurement data may have noise, the grid cells that should be marked with 0 can be erroneously marked as 1 or vice versa. To overcome this, we use the notion of occupancy probability of grid cells (Thrun [2002b]), which defines the probability of a cell being occupied. Initially, the occupancy probability of each cell is set to 0.5, indicating all of them as unexplored. During the robot’s navigation, the robot observes the nearby cells repeatedly over a period of time, and accordingly, the occupancy probabilities of the cells either increase or decrease. When the probability of a grid cell increases and reaches the threshold P occupied , the corresponding cell value is set to 1. Similarly, when the probability decreases and reaches the threshold P free , the corresponding cell-value is set to 0. Values for the cells having occupancy probability between P occupied and P free are set to -1. This way, eventually, an estimate of the environmental map closer to the actual map is achieved by the robot. The overall mapping algorithm is depicted in Algorithm 1. The algorithm takes robot pose p t , depth map d t at time t and the probabilistic partial map PM t−1 at time t− 1 as inputs, and it generates probabilistic partial map PM t , partial map M t at time t and snapshot map L. The algorithm begins with computing the probabilistic partial global map PM t in steps 1-18, and then it generates M t and L in steps 19-30. Steps 2-5 deal with initialization, where each cell in L and M t is set to -1 indicating unexplored, PM t is set to PM t−1 and the variables obstacle_set, obstacle_grids and free_grids for storing valid obstacles, obstacle grid locations and free grid locations respectively are set to ∅. Step 6 generates a point cloud in the robot’s frame of reference, which is then transformed to the initial frame of reference in step 7. The point clouds, which are just above the ground clearance of the robot pose, are considered as the obstacles in steps 8-12. Once obstacle grid locations and free grid locations are determined in steps 13-14, their occupancy probabilities are computed using inverse_sensor_model (Thrun [2002b]) and the log odd ratio of prior occupancy in steps 15-18. Steps 19-30 construct partial global map M t and snapshot map L depending upon whether the occupancy probabilities in the corresponding cells in PM t and L meet the threshold P occupied and P free . 2.3.3 Robotic Navigation Framework Figure 2.2 depicts the design of the Lizard-inspired quadruped robot. It has a front-link and a back-link attached to a base-link. The front-link and back-link can rotate upto 45° 25 CHAPTER 2. HYBRID SYSTEM PLANNING Algorithm 2.1: Mapping Input: Robot pose (p t ) and depth map (d t ) at time t, probabilistic partial map PM t−1 at t− 1 , camera_intrinsic_parameters. Output: Probabilistic partial map PM t , partial global map M t , snapshot map L. 1 begin 2Initialize, L i = −1, ∀L i ∈ L ; g i = −1, ∀g i ∈ M t ; PM t = PM t−1 ; 3Initialize obstacle_set = obstacle_grids = free_grids = ∅; 4PC = compute_point_cloud(d t , camera_intrinsic_parameters); 5PC transformed = transform_point_cloud(PC, p t ); 6for pc i ∈ PC transformed do 7if z-axis of pc i > ground_clearance of robot base then 8insert pc i in obstacle_set; 9end 10end 11obstacle_grids = grid_location(obstacle_set); 12free_grids = find_free_grids(obstacle_grids , p t ); 13for all cell m i ∈ obstacle_grids or free_grids do 14l 0 = log(probability(m i = 1)/probability(m i = 0)) 15PM t,i = PM t−1,i + inverse_sensor_model(m i ,p t ,d t ) - l 0 ; 16end 17for all cell pm i ∈ PM t , g i ∈ M t do 18if pm i ≥ P occupied then 19g i = 1 ;/* obstacle */ 20end 21else if pm i ≤ P free then 22g i = 0 ;/* free space */ 23end 24end 25for all cell L i ∈ L , g i ∈ M t do 26if L i ∈ free_grids or obstacle_grids then 27L i = g i ; 28end 29end 30return PM t , M t , L; 31 end 26 2.3. A CASE STUDY FOR AUTONOMOUS NAVIGATION IN AN UNKNOWN ENVIRONMENT w.r.t base-link. Two legs that can be lifted or grounded are attached to front-link and back-link each. The length, width, and height of the robot are 15cm by 10cm by 5cm, respectively. A RGB-D camera is also mounted on the robot. The overall architecture of the proposed framework is shown in Figure 2.3, which comprises two essential modules: (1) SLAM and (2) Planning and Control. The SLAM module is responsible for mapping the unknown environment and detecting the robot’s position using the odometry data and depth information from the RGB-D camera mounted on the robot. However, the SLAM module is not part of this thesis. A description of it is given in section 2.3.2. In this section, we describe the Planning and Control module in detail: 2.3.3.1 Planning and Control Module Planning and control module has two sub-modules, namely motion-planning and control. We now discuss each in detail: Motion-planning: A motion-plan for a robot is a finite sequence of way-points in the 2D environment that, when traced, leads the robot to the assigned destination. We say that a motion-plan is safe when the path that it gives (the path formed by joining the consecutive way-points through straight lines) is obstacle-free. The motion-planning problem can thus be defined as the problem of finding a finite sequence of way-points such that it is safe. In the following, we briefly illustrate its reduction to a satisfiability problem of a first-order logic formula. We reduce the motion-planning problem as a constraint-satisfaction- problem by encoding the initial, goal positions, the movement of the robot, and the obstacles in the map as first-order-logic formulae in the theory of quantifier-free nonlinear real arithmetic (Saha et al. [2014]). The t th way-point in a motion-plan is represented by a pair of real variables x t and y t . The number of way-points in a motion plan is upper bounded by a constant M, i.e., 0 ≤ t ≤ M. Our planner is restricted to generating piece-wise-linear (PWL) paths. The number of line-segments in the path and hence the precision can be tuned with the value of M. Initial and Goal State: The initial and the goal positions of the robot are tuples ⟨x init ,y init ⟩ and ⟨x g ,y g ⟩ respectively which are represented by the following constraints: Figure 2.2: CoppeliaSim model of the quadruped robot 27 CHAPTER 2. HYBRID SYSTEM PLANNING Figure 2.3: The robotic software architecture for SLAM, path-planning, and control. Init : (x 0 = x init )∧ (y 0 = y init ) Goal : _ 1≤t≤M (x t = x g )∧ (y t = y g ) 1≤t<M (x t = x g )∧ (y t = y g ) =⇒ (x t+1 = x g )∧ (y t+1 = y g ) . It encodes that the first way-point must be the initial position, and at least one of the way-points is the goal position. The second clause encodes that the robot remains in the goal position after reaching there. Obstacles: Each obstacle Obs represents a rectangular region in the map bounded by the four corner points, where (x tl ,y tl ), (x tr ,y tr ), (x bl ,y bl ) and (x br ,y br ) denote the top-left, top-right, bottom-left and bottom-right corner points, respectively. We further inflated each obstacle region by r grid units on each side, where r > radius of the circumscribed circle for our robot. The i th obstacle obs i is defined as follows: obs i x tl = obs i x tl − r, ∧ obs i y tl = obs i y tl + r ∧ obs i x tr = obs i x tr + r ∧ obs i y tr = obs i y tr + r ∧ obs i x bl = obs i x bl − r ∧ obs i y bl = obs i y bl − r ∧ obs i x br = obs i x br + r ∧ obs i y br = obs i y br − r. Obstacle Free Path: The constraints given by Obs_freepath ensures an obstacle free path given by the way-points. If (x t ,y t ) and (x t+1 ,y t+1 ) are any two consecutive way- points in the projected path, it must be ensured that the line joining them does not pass through any obstacle region. Let (obs j x tl ,obs j y tl ), (obs j x tr ,obs j y tr ), (obs j x bl ,obs j y bl ) and (obs j x br ,obs j y br ) are the corner points of the j th obstacle respectively. For every rectangular obstacle, say obs j , the idea is to search for a separating line a j .x +b j .y +c j = 0 such that any pair of way-points (x t ,y t ) and (x t+1 ,y t+1 ) lie on one side of the line whereas the four corner points of obs j are on the other side of the line. This ensures that the line 28 2.3. A CASE STUDY FOR AUTONOMOUS NAVIGATION IN AN UNKNOWN ENVIRONMENT joining the way-points does not pass through the obstacle. Obs_freepath : 1≤t<M 0≤j<N (a tj x t−1 + b tj y t−1 + c tj < 0)∧ (a tj x t + b tj y t + c tj < 0)∧ (a tj obs j x bl + b tj obs j y bl + c tj > 0)∧ (a tj obs j x br + b tj obs j y br + c tj > 0)∧ (a tj obs j x tl + b tj obs j y tl + c tj > 0)∧ (a tj obs j x tr + b tj obs j y tr + c tj > 0) _ (a tj x t−1 + b tj y t−1 + c tj > 0)∧ (a tj x t + b tj y t + c tj > 0)∧ (a tj obs j x bl + b tj obs j y bl + c tj < 0)∧ (a tj obs j x br + b tj obs j y br + c tj < 0)∧ (a tj obs j x tl + b tj obs j y tl + c tj < 0)∧ (a tj obs j x tr + b tj obs j y tr + c tj < 0) where t∈1,...,M− 1,j ∈0,...,N− 1, a tj ,b tj ,c tj ∈ R and N denotes the number of obstacle regions. Additionally, the robot’s movement between two consecutive way-points is bounded by v units in both X and Y axes, which is encoded as Mov : 0≤t<M [abs(x t+1 − x t ) < v]∧ [abs(y t+1 − y t ) < v] where v ∈ Z + is less than the minimum of height and width of the environment. This constraint ensures that consecutive way-points are within a reasonable distance. The Map-Analyzer generates a bounding-box approximation of the obstacle regions and returns a list of obstacles from the partial map, assuming the unknown regions as free spaces. This allows for an easy representation of obstacles as constraints, in the SMT- Constraint-Generator sub-module. An SMT-LIB file is generated from SMT-Constraint- Generator sub-module representing the free-spaces, obstacle regions in the partial map, the robot’s initial position, goal position, and safe-movement as constraints. Then, the constraints in the SMT-LIB file are solved for satisfiability by the Z3 (de Moura and Bjørner [2008]) SMT-Solver. When satisfiable, a motion plan is extracted from the satis- fying assignments as a sequence of way-points. Control: This sub-module is responsible for moving the lizard-inspired quadruped robot along the computed way-points using a LoS algorithm (see Algo. 2.2) and a hybrid con- troller module. However, before discussing the algorithm and the controller module, we first give a brief description of the motion-primitives of the robot. Motion-primitives: Motion primitives are a set of control actions that can be provided as commands to our robot for execution. Our robot has four basic primitives: Move- forward and Move-backward for linear movement and Rotate-clockwise (RotClk), Rotate- anti-clockwise (RotAclk) for rotational movements. Rotational movement has four dis- cretization angles- 6°, 4.5°, 3°, and 1.5°. Each motion-primitive has two phases, and when applied on the lizard-inspired quadruped robot will perform one linear or rotational movement. The details of the phases of motion-primitives Move-forward and for one discretization angle of RotClk (see in Figure 2.4) is given below: 29 CHAPTER 2. HYBRID SYSTEM PLANNING (a) Move-forward(b) RotClk Figure 2.4: Phases of Move-forward and Rotate-Clockwise maneuvering of the robot 1) Move-forward: The motion primitive Move-forward is used to perform a forward direc- tion movement for the robot. In the first phase, the robot lifts its front-right and back-left legs together while front-joint and back-joint are rotated by 30° and −30° angles, re- spectively. As front-left and back-right legs are grounded at that time, the rotations of front-joint and back-joint make the front-right and back-left legs move forward (see Figure 2.4a first phase, where θ f =−θ b = 30°). In the second phase, the robot lifts its front-left and back-right legs together while front-joint and back-joint are rotated with −30° and 30° angles, respectively (see Figure 2.4a second phase, where −θ f = θ b = 30°). Now, these two legs move forward and thus complete one forward movement. As a result of one such forward movement, the effective displacement of the robot is 6.46 cm (calculated via simulation). 2) Rotate-clockwise (RotClk): The motion primitive RotClk will cause the robot to make a clockwise angular rotation. We now present the mechanism for 6° clockwise rotation (RotClk-6): In the first phase, the robot lifts its front-left and back-right legs together while front-joint and back-joint are rotated with θ f and θ b , respectively (see Figure 2.4b first phase, where−θ f = θ b = 45°). In the second phase, the robot lifts its front-right and back- left legs together while front-joint and back-joint are rotated with 20° and −20° angles, respectively (see Figure 2.4b second phase, where θ f = −θ b = 20°). RotClk movement will rotate the robot clockwise α degrees which is the effective angular rotation. This α depends on γ, given as: γ = K ∗ α, where γ = ∥θ 1 ∥−∥θ 2 ∥, θ 1 and θ 2 are angle of rotation of front-joint and back-joint in the first and second phase respectively while K is the proportional gain and obtained by simulations. In this case, ∥α∥ = 6° as γ = 25°. Rot (∥θ 1 ∥, ∥θ 2 ∥)Rot-diff (γ)Eff-rot (α)Ratio (K) 45° , 20°25°5.947°≈ 6°4.204 40° , 20°20°4.535°≈ 4.5°4.414 35° , 20°15°3.072°≈ 3°4.883 30° , 20°10°1.485°≈ 1.5°6.734 30° , 30°0°0°− Table 2.1: Clockwise and anti-clockwise rotations. θ 1 and θ 2 are angles of rotation of the joints in the first and second phases, respectively. 30 2.3. A CASE STUDY FOR AUTONOMOUS NAVIGATION IN AN UNKNOWN ENVIRONMENT In Table 2.1, we have shown different discretization of clockwise and anti-clockwise rotational movements of our robot, which are achieved by applying different θ 1 and θ 2 rotations to the joints in both phases of the movements. Rot-diff denotes rotational difference in both phases, Eff-rot denotes effective rotation of the robot, and K is the ratio of γ/α. LoS algorithm: The LoS algorithm is responsible for the line-of-sight navigation of the robot between the consecutive way-points on a projected path. The algorithm receives the robot’s positional information (x t , y t ) from the current pose and fetches the next way- point w p (w x , w y ) from the sequence of way-points wps produced by the motion-planning sub-module. The algorithm begins by calculating the angle φ between the robot’s current position and the waypoint with respect to the positive X-axis. It then calculates the effect- ive angle θ, which is the angle the robot needs to rotate considering its current orientation θ t . Then, repeatedly apply rotational motion-primitives to rotate the robot towards the waypoint accordingly. Each rotational movement is followed by a Move-backward move- ment to minimize positional changes during rotation due to the unavailability of in-place rotation for our robot. The algorithm then calculates the Euclidean distance dist between (x t , y t ) and (w x , w y ) and the number of Move-forward motion primitives it needs to apply to reach the waypoint w p by dividing dist with a constant value of 6.46cm which is the robot’s effective displacement for one application of Move-forward movement. It calls the hybrid controller, which we discuss in the next section, in synergy with the movement to ensure the robot does not deviate from its path. Algorithm 2.2: LoS algorithm Input: Robot’s pose p t and way-point w p . 1 begin 2 Calculate the angle φ; 3 Calculate the effective angle, θ ← φ− θ t ; 4 if θ > 180 then 5θ ← θ− 360; 6 end 7 Apply rotational motion-primitives to cover θ ; /* Each rotational movement is followed by a move backward movement */ 8 Calculate the distance dist← q (w x − x t ) 2 + (w y − y t ) 2 ; 9 Calculate the n_steps← dist/6.46 ; /* The number of linear movements needed to cover the distance */ 10 for i = 0; i < n_steps; i++ do 11Apply the move-forward motion-primitive; 12Calls hybrid controller ; /* Calculates the deviation from the path, and moves accordingly. */ 13 end 14 end 31 CHAPTER 2. HYBRID SYSTEM PLANNING Designing a hybrid controller for the Lizard-inspired quadruped robot: We propose designing a hybrid controller that controls the navigation of the robot along a projected path. Let (x,y) and θ define the robot’s current position and orientation in the environment, respectively, and the robot has a fixed speed v as shown in Figure 2.5. Thus, the velocity of the robot in the X and Y axes direction can be derived as v∗ cosθ and v∗ sinθ, respectively. The robot can rotate with an angular speed w, for simplicity, which can be either of three discrete values −6, 0, and 6 degrees/second. When w = 0, the direction θ remains unchanged, and the robot is going straight. When w = −6, the direction θ is decreasing, and thus the robot is attempting to turn right. Conversely, when w = 6, the direction θ is increasing, and thus the robot is attempting to turn left. Figure 2.5: Robot movement The movement of the robot following a projected path is shown in Figure 2.6, which is made for illustration and may not be an actual path in the environment. The green trace represents the robot’s trajectory. The red points are the waypoints, and the lines joining them represent the projected path from start to goal location. Figure 2.6: Robot’s trajectory We assume a buffer zone of distance ±E around the projected path as shown in Figure 2.6. We continuously monitor the robot’s deviation d from the projected path as shown 32 2.3. A CASE STUDY FOR AUTONOMOUS NAVIGATION IN AN UNKNOWN ENVIRONMENT in Figure 2.7, where the line AB joins the current waypoint and the next waypoint on the projected path and the line CB joins the robot’s current position and the next waypoint. We calculate the slopes m 1 and m 2 of the lines AB and CB with X-axis as below: m 1 = y 2 − y 1 x 2 − x 1 m 2 = y 2 − y x 2 − x Figure 2.7: Robot’s deviation from the path The angle φ between the line AB and CB can be derived as: φ = arctan∥ m 2 − m 1 1 + m 2 ∗ m 1 ∥ The length a of CB is calculated as: a = q (x 2 − x) 2 + (y 2 − y) 2 The robot’s deviation from the projected path is calculated as: d = a∗ sinφ Hybrid controller: The hybrid controller for our robot is shown in Figure 2.8. It has four modes- Move Forward/Backward, Rotate-Clockwise, Rotate-Anticlockwise, and Stop. The LoS algorithm calls the hybrid controller to control the robot’s motion dynamics to minimize deviation from the projected path. 2.3.3.2 Integrating SLAM, Planning and Control The Navigation algorithm controls the navigation of our lizard-inspired quadruped ro- bot in an unknown environment by integrating the SLAM, Motion-planning, and Control modules. Through SLAM module, it incrementally builds a partial global map of the observable terrain and localizes the robot in that space. It then uses the motion-planning module to find a safe path-plan to a given goal position through iterative re-planning applied on the progressively gathered knowledge about the environment. Finally, the con- trol module drives the robot along the path to reach the goal. These modules work in synergy to generate a safe path-plan and simultaneously drive the robot to the goal. The 33 CHAPTER 2. HYBRID SYSTEM PLANNING Figure 2.8: Hybrid controller details of this integration are shown in Algorithm 2.3. The algorithm takes the goal po- sition (x g , y g ), a set of pose samples S t−1 , measurement data z t , control data u t , depth map d t and probabilistic partial map PM t−1 as inputs. The algorithm begins with ini- tializing the time t, S t−1 , PM t−1 , and the problem instance prob in line 2. In line 3, it receives the robot’s current pose p t , pose samples set S t , partial global map M t , snapshot map L, and probabilistic partial map PM t from the SLAM module. Lines 4-15 perform motion-planning task which starts with Map-Analyzer to build a list of obstacle regions obs list using the information in M t in line 4. Line 5 initializes two variables max_wps and max_dist that correspondingly set the limits for the maximum number of way-points in a plan and the maximum distance between the two consecutive way-points. In lines 6-10, the algorithm checks iteratively whether the problem instance is satisfiable with a maximum of 10 way-points in a plan starting with an initial value of 2. For each iteration in line 7, SMT-Constraint-Generator formulates the constraint-satisfaction problem by encoding the constraints obs list , robot’s current pose p t , goal position (x g , y g ), max_wps and max_dist as a first-order-logic formulae φ such that the satisfiability of φ implies the existence of a safe motion plan. The formula φ is then solved by the SMT-Solver Z3 in line 8, which returns either of the two values ’SAT’ or ’UNSAT’ to indicate whether the problem instance is satisfiable or not in the current iteration. When satisfiable, the motion plan is constructed from the satisfying assignment as a sequence of way-points wps in line 12. Lines 16-24 are responsible for driving the robot along the path to reach the goal. In line 17, every way-point w p in wps is processed sequentially and checked in line 19 against the snapshot map L, to verify whether w p ∈ free_spaces (free_spaces is the set of free spaces in L). If the w p is found to be in free space, the LoS algorithm is called in line 20 to generate commands in the form of motion-primitives to move the robot towards the way-point and control its movement along a path by using the hybrid controller. Otherwise, the algorithm replans with the updated M t , which is continuously being updated with the robot’s movement in the environment. 34 2.3. A CASE STUDY FOR AUTONOMOUS NAVIGATION IN AN UNKNOWN ENVIRONMENT Algorithm 2.3: Navigation Input: Goal position (x g ,y g ), a set of pose samples S t−1 , control data u t , measurement data z t , depth map d t and probabilistic partial map PM t−1 . 1 begin 2 t = 1; S 0 = ∅; PM 0 = ∅; prob = UNSAT; 3 p t , S t = Localization(S t−1 , u t , z t ); M t , L, PM t = Mapping(p t , d t , PM t−1 ); 4 obs list = Map-Analyzer(M t ); 5 max_wps = 2; max_dist = 100; /* Max way-points in a plan and max distance between two consecutive way-points. */ 6 while (max_wps≤ 10) ∧ (prob == UNSAT) do 7φ = SMT-Constraint-Generator(obs list , p t , (x g , y g ), max_wps, max_dist); 8prob = SMT-Solver(φ); 9max_wps = max_wps + 1; 10 end 11 if prob==SAT then 12Construct wps from SAT assignment of φ; 13 else 14return no-plan; 15 end 16 while wps̸=∅ do 17w p = wps.delete; 18t = t + 1; 19if w p ∈ free_spaces then 20Call LoS(p t , w p ); 21else 22Go to step 2; 23end 24 end 25 end 2.3.4 Simulation and Experimental Evaluation We now present our simulation results 4 on various indoor environments. The simulations are performed on CoppeliaSim 5 and ROS 6 integrated platform, on a system with Intel® Core™ i5-8250U CPU @ 1.60GHz, 8 cores, 8GB RAM, Ubuntu 18.04. We use the standard SMT-LIB format to represent our constraints for motion-planning. These constraints are solved by using Z3 solver 7 . Figure 2.9 shows various stages of our navigation software at work in an indoor simulation environment, where plan generation and navigation of 4 The source codes of our implementation are available at: w.iitp.ac.in/~halder/IRIA2021/Codes. zip 5 https://w.coppeliarobotics.com/ 6 https://w.ros.org/ 7 https://github.com/Z3Prover/z3 35 CHAPTER 2. HYBRID SYSTEM PLANNING Table 2.2: Simulation Results Environment obsvarcnstrturnsreplans T plan distT traverse cov(%)Remarks SceneDimInsComplexity(in Sec)(in Meter)(in Sec) Scene-110× 10m 2 ins1Complex2252779180773123.8113.2991936.59Success ins2Complex1173056166176Time Out2.6152621.16Failure ins3Simple6048110.125.3221120.21Success ins4Complex136240515655261.987.3653937.63Success Scene-25× 5m 2 ins1Complex116224616115317.16.9251061Success ins2Simple55454331211.274.1229026.12Success ins3Moderate435944314310.13.76.133.1Success Scene-310× 10m 2 ins1Complex1922439158742410.243226Success ins2Complex2011073076719511.129.4120039.11Success ins3Simple10110389111.22.613020.63Success ins4Moderate12515181091348.44.2736723.2Success our robot are shown at different time points. The robot’s starting position and goal position are marked by the green and red circles, respectively, in the environment (in Figure 2.9a). Figures 2.9b-2.9f show the robot’s perception of the environment at different time-instances during planning and navigation steps. The orange dot indicates the robot’s current position in the perceived environment. The purple and cyan areas are, respectively, the unknown and free spaces, whereas the yellow regions represent obstacle regions. The white trajectories are the projected path from the robot’s current position to the goal after plan-generation, considering the partial information gathered till that point, and the red trajectories represent the traversed path by the robot along the projected path. In Table 2.2, we present the results of our simulation experiments on different scenarios of varying complexity. Dim and Ins represent the dimension and problem instances in the scene. obs denotes the number of obstacle regions detected in the environment, whereas var and cnstr denote the numbers of SMT variables and constraints used, respectively, to solve the problem instances. turns reports the number of turns taken by the robot during its traversal to the goal, re-plans records the number of re-plans during this traversal, and T plan is the accumulated duration of time for the generation of these plans. dist and T traverse are the distance and time to traverse by the robot from the initial to the goal position. The percentage of the total area covered by the robot during this navigation is presented as cov. We consider the ‘Complexity’ of an environment based on the dimensions of the environment, the number of obstacles detected, and the number of turns needed to reach the goal position. We classify the problem instances as ‘Simple’, ‘Moderate’ and ‘Complex’ by observing the no. of constraints and variables used in their first-order- logic formula encoding (2.3.3(B)), given as: Simple: (var < 1000) ∧ (cnstr < 1000), Moderate: (1000 < var < 2000)∧ (1000 < cnstr < 1500), Complex: (var > 2000) ∧ (cnstr > 1500). We report whether the robot successfully reaches the goal position within the plan-generation timeout of 60 seconds under ‘Remarks’. Our navigation software successfully solves 10 out of the 11 navigation tasks. 2.3.5 Conclusion The work presents an integrated software framework for autonomous navigation of a lizard- inspired quadruped robot in an unknown environment. The SLAM, motion-planning, and control components of the framework run in synergy to generate a safe path plan and 36 2.3. A CASE STUDY FOR AUTONOMOUS NAVIGATION IN AN UNKNOWN ENVIRONMENT (a)(b)(c) (d)(e)(f) Figure 2.9: Planning and Navigation: simulation results of plan generation and nav- igation in an office-like environment using CoppeliaSim and ROS: (a) is shown here. The initial plan is given in (b) when most of the environment was unknown, assuming the unknown as free spaces. (c) shows detection of obstacles on the path, (d) and (e) present two re-planning steps at different time points during navigation, and (f) depicts the robot’s actual traversed path while following the generated plans. simultaneously drive the robot to the goal. The motion-planning problem is solved by reducing the constraint-satisfaction problem, which in turn is solved with a state-of-the- art constraint solver, Z3. The robotic software architecture has been tested with several planning problem instances in various indoor simulation scenarios, and the results met our objectives. 37 CHAPTER 2. HYBRID SYSTEM PLANNING 38 Chapter 3 A Contrastive Plan Explanation Framework for Hybrid System Models Chapter Abstract: In AI planning, having an explanation of a plan given by an AI planner is often desirable. The ability to explain various aspects of a synthesized plan to an end-user not only brings trust in the AI-based system but also reveals insights into the planning domain and the planning process. Contrastive explanation has been a popular approach in the literature (Fox et al. [2017], Hoffmann and Magazzeni [2019]). It is not only an effective way for enhancing understanding but also simpler to generate than full causal analyses (Miller [2019]). Additionally, their inherent design makes it easy to compare the original plan with the user’s alternative. Contrastive questions such as “Why action A instead of action B?” can be answered with a contrastive explanation that compares the properties of the original plan containing A against the contrastive plan containing B. In this chapter, we explore a set of contrastive questions that a user of a planning tool may raise and propose a re-model and re-plan framework to provide explanations to such questions. Earlier work has reported this framework on planning instances for discrete problem domains described in the Planning Domain Definition Language (PDDL) and its variants. This work proposes an extension for planning instances described by PDDL+ for hybrid systems that portray a mix of discrete-continuous dynamics. Specifically, given a mixed discrete-continuous system model in PDDL+ and a plan describing the set of desirable actions on the same to achieve a destined goal, we present a framework that can integrate contrastive questions in PDDL+ and synthesize alternate plans. We present a detailed case study on our approach and propose a comparison metric to compare the original plan with the alternate ones. A rtificial Intelligence (AI) planning has been an active area of research for several dec- ades, and is a core component of all autonomous systems today. Classical AI planning 39 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS concerns with finding a sequence of feasible actions from an initial configuration of a sys- tem to a desired goal. Planning has been an active area of research for AI practitioners, leading to the development of planners for diverse planning domains, goal descriptions, and varied optimization objectives. Techniques ranging from graph traversals to recent de- velopments around constraint solvers for efficient and scalable planning have been explored widely. Planning with hybrid domains modelled in PDDL+ (Fox and Long [2006]) has been gaining research interest in the automated planning community in recent years. Hy- brid domains capture a more accurate representation of real-world problems that involve an interplay of continuous and discrete processes. However, solving problems represented as PDDL+ domains is very challenging due to the complex system dynamics, including non-linear processes and events. In planning literature over the past years, a number of languages for expressing planning problem specifications have evolved (e.g. STRIPS Fikes and Nilsson [1971], ADL Pednault [1989], PDDL Howe et al. [1998] and PDDL+) and sophisticated planners (e.g. GraphPlan Smith and Weld [1998], SatPlan Kautz et al. [2006], F Hoffmann [2001], Hoffmann and Nebel [2001], FastDownward Helmert [2006], HSP Bonet and Geffner [1999], ENHSP Scala et al. [2016], UPMurphi Penna et al. [2012], DiNo Piotrowski et al. [2016], SMTPlan+ Cashmore et al. [2020] and LPG Gerevini and Serina [2002]) have advanced the field of planning. Explainable Planning (XP), also termed as Explainable AI Planning (XAIP) (Hoff- mann and Magazzeni [2019]) has picked up as a problem of immense recent importance, given the multitude of application domains where autonomous plans are being envisioned to automate and replace manually crafted ones. Indeed, questions like safety, robustness, and trustworthiness of automatically learned and synthesized plans are being analyzed be- fore deployment. More importantly, explainable planning has been an interesting area of research in recent times, given the increasing number of application areas in which humans and autonomous agents collaborate with mutual trust and where mutual understanding and cooperative plans are important for achieving a desired objective. With the advent of automated planning being applied in safety-critical systems, the need for plan explanation to a human expert responsible for implementing the plan has become ever more important. Before accepting and executing the plan, the human expert ought to be convinced about the safety and rationality of the plan. In the first place, there may be a mismatch between the knowledge of the human agent about the planning domain and the domain knowledge of the autonomous planning agent. Therefore, many obvious choices of actions anticip- ated by the human agent might not be apparent to the autonomous agent. Further, the autonomous agent often generates a plan based on an abstract model of the domain. Due to such an abstraction, the plan may not be executable in the physical world. Explanation of plans is thus crucial. To address the explanation concern, a popular approach in recent literature is that of contrastive explanations. In particular, contrasting a given plan with alternate ones, in order to produce an argument about why the generated plan is to be chosen for execution over the possible alternatives, has been proposed in (Krarup et al. [2019]), in the context of discrete systems and problem domains. This forms a foundation for the motivation of our work. In recent times, with the advent of Cyber-Physical Systems (CPS) and Internet-of- 40 Things, there has been a renewed research interest on planning for hybrid systems that exhibit an interplay of discrete and continuous dynamics. Planning in hybrid systems poses particular challenges to classical AI planners, due to the interplay of continuous and discrete dynamics. This has inspired recent research on planning for hybrid systems, and hybrid system planners (e.g. SMTPlan+, UPMurphi and ENHSP) have been at the forefront of planning research in recent literature. These tools accept the planning problem description in the PDDL+ (Fox and Long [2006]) modelling language, which allows to describe processes with mixed (continuous and discrete) dynamics. Contributions: In this chapter, we propose a contrastive explanation paradigm for plans in hybrid systems, as done for their discrete counterparts, by highlighting the con- trasts of the original plan with an alternate one. We aim to explain a plan by a con- trastive explanation framework that provides answers to contrastive questions posed by a plan user. Through such answers, a user gains insights as to why a particular plan can be trusted and deployed, or on the other-hand, why there may be a need for re-planning. Our framework works by generating alternate plans to the planning problem by constructing a hypothetical planning problem, and this construction is such that any valid plan for the hypothetical problem will meet the user’s expectation phrased in the contrastive question. We present the construction of such hypothetical planning problems for 8 classes of con- trastive questions and present proofs of their soundness. In summary, the contributions of this work are as follows: 1. We propose a set of eight contrastive questions on plans for hybrid systems. This set of questions is certainly not exhaustive and only portrays some of the critical questions that we believe a user of a planning tool may be interested in posing on plans for hybrid systems. 2. An explanation framework based on the re-model and re-plan idea is proposed that advocates the construction of a hypothetical model for each of these contrastive questions in such a way that any valid plan, if generated on this model, will imitate the contrastive question. 3. We introduce a set of metrics for the comparison of a contrastive plan with that of the original plan, thus generating contrastive explanations for the user’s questions that may help in understanding the planning domain and the behaviour of the planner in a better way. 4. In addition, we propose to prove the absence of a plan in a hybrid domain using verification tools such as dReach (Gao et al. [2014]). Since planning problems are undecidable for hybrid systems in general (Alur et al. [1995a]), when a planner fails to generate a valid plan for a given problem instance, it is not clear whether this is due to the underlying undecidability or because the problem instance does not admit a valid plan. To explain this to a user, we leverage bounded reachability analysis of the domain. 5. To illustrate our explanation framework, we consider a car-domain which is modeled in PDDL+ and we synthesize plans using the SMTPlan+ planner. The compilation 41 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS of the hypothetical models are shown for each of the questions, and the resulting con- trasting plans are compared with the original plan following a set of metrics similar to the ones introduced in (Cashmore et al. [2019]). Experimental evaluation of the framework has been presented for two other domains, namely generator-events and planetary-lander, along with the car domain using the three planners SMTPlan+, UPMurphi, and ENHSP. The rest of this chapter is organized as follows. Section 3.1 presents an overview of relevant preliminaries and an example problem context. Section 3.2 illustrates our framework of contrastive explanations, and Section 3.3 elucidates a case study on an example domain. Section 3.4 elaborates on the explanation of no-plans, while Section 3.5 shows implement- ation and results. Section 3.6 presents related literature and Section 3.7 discusses the limitations of our approach. Section 3.8 summarizes the contributions and our findings from this work. 3.1 Preliminaries In this section, we present an overview of some background concepts that are needed for this work. We begin with a brief description of the Planning Domain Description Language PDDL+. PDDL+ extends PDDL2.1 for representing mixed discrete-continuous domains and planning problems on them. The key features additionally supported in PDDL+ are the ability to model exogenous events and continuous changes as processes. We begin with the definition of a planning instance in PDDL+ and then touch upon each of the above-mentioned features that it brings forth using an example of a planning problem. Definition 3.1.1 A planning instance Π in PDDL+ (Fox and Long [2006]) is a pair (Dom, Prob), where Dom is a 6-tuple ⟨Fs, Rs, As, Es, Ps, arity⟩ called the domain consisting of a finite set of function symbols Fs, predicate symbols Rs, action symbols As, event symbols Es, process symbols Ps and an arity function that maps each of these symbols to their respective arities. The arity of a symbol specifies the number of arguments it takes. Prob is a triplet ⟨Os,I,G⟩ called the planning problem consisting of the set of objects Os in the planning instance, the initial state I and the goal condition G.2 We consider a domain consisting of a car that has to travel a specified distance, respecting certain constraints. The planning domain represented in PDDL+ is shown in Listing 1. It has six functions symbols: d, v, a, upLimit, downLimit and runningTime, three predic- ate symbols: running, engineBlown and goalReached, three action symbols accelerate, decelerate and stop, one event symbol engineExplode and one process symbol moving. The arity maps each of these symbols to 0. The problem instance that we have used in this domain is shown in Listing 2 where I is the initial state and G is the goal condition as discussed later. Listing 3.1: The car domain in PDDL+. ( define ( domain car ) ( : predicates ( running ) ( engineBlown ) ( goalReached )) 42 3.1. PRELIMINARIES ( : functions (d) (v) ( a ) ( upLimit ) ( downLimit ) ( runningTime )) ( : process moving : parameters () : precondition ( and ( running )) : e f f e c t ( and ( increase (v) (∗ #t ( a ) ) ) ( increase (d) (∗ #t (v ) ) ) ( increase ( runningTime ) (∗ #t 1 ) ) ) ) ( : action a c c e l e r a t e : parameters () : precondition ( and ( running ) (< ( a ) ( upLimit ) ) ) : e f f e c t ( and ( increase ( a ) 1))) ( : action d e c e l e r a t e : parameters () : precondition ( and ( running ) (> ( a ) ( downLimit ) ) ) : e f f e c t ( and ( decrease ( a ) 1))) ( : event engineExplode : parameters () : precondition ( and ( running ) (>= ( a ) 1) (>= (v) 100)) : e f f e c t ( and ( not ( running )) ( engineBlown ) ( assign ( a ) 0))) ( : action stop : parameters () : precondition ( and (= (v) 0) (>= (d) 30) ( not ( engineBlown ) ) ) : e f f e c t ( goalReached ) ) ) Grounding of any symbol in PDDL+ is defined as follows: Definition 3.1.2 Given a planning instance Π, grounding of a symbol χ ∈ Fs ∪ Rs ∪ As ∪ Es ∪ Ps is formed by instantiating the arguments of the symbol with its actual parameters while respecting the arity of the symbol.2 A set of atomic propositions P is obtained from grounding the predicate symbols in Rs. For example, the atomic proposition (running) can be directly obtained from the predicate symbol running since its arity is 0. An atomic proposition (available unit1) is an example of a grounded predicate of arity 1 in the Planetary Lander domain described in Appendix 11.2. Here, available is a predicate and unit1 is an object that is an argument to the predicate available. The set of Primitive Numeric Expressions (PNEs) (Fox and Long [2006]) is defined in the following. Definition 3.1.3 For a given planning instance Π, the Primitive Numeric Expres- sions (PNEs) are the terms constructed from the grounded function symbols of the domain with the number of parameters given by their arities.2 Like grounded predicate symbols result in atomic propositions, grounded function symbols result in numeric values. An example of a PNE in our domain is (a), which denotes a numeric value modeling the acceleration of the car at an instance of time. A domain may contain numeric expressions formed with arithmetic operations on the PNEs, such as a = 0 or a = a + 1, which denote initialization of acceleration or increase in acceleration of the car or formation of a constraint in the domain, etc. We denote the set of grounded actions by A where a ground action is defined in the following: Definition 3.1.4 A ground action (Fox and Long [2006]) act is obtained from an action symbol in As by substituting objects for each of its parameters. The components of act 43 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS consist of a precondition Pre(act) and an effect Eff(act). Pre(act) is a formula that needs to be satisfied to make the action applicable. An Eff(act) is the action’s postcondition consisting of assignments to atomic propositions and PNEs. The effect Eff can be Eff + or Eff − . Eff + signifies addition of an effect with the existing ones. Similarly, Eff − signifies deletion of an effect from the existing ones.2 An example of a ground action in our domain is accelerate, which has the precondition running ∧ (a < upLimit) as its precondition where ∧ denotes Boolean AND. The postcondition is an assignment of a + 1 to the PNE a, shown with the increase construct in Listing 1. PDDL+ supports discrete instantaneous actions as well as durative actions, i.e., actions which occur for a finite duration. Durative actions can also be mapped into an equivalent start-process-stop representation. The reason for performing the mapping is in order to give durative actions a semantics in terms of the underlying constructs of PDDL+, which are themselves given an interpretation in terms of hybrid automata as discussed in (Fox and Long [2006]). However, the semantics of PDDL+ can also be defined without going through hybrid automata, as explained by (Shin and Davis [2005]) and (Percassi et al. [2021]). The definition of a ground event is the same as Definition 3.1.4, with the restriction that events are required to have at least one numeric precondition, which makes them a special case of actions (Fox and Long [2006]). An example of an event in our car domain is engineExplode. Actions in PDDL+ are executable by the planner, whereas the events are not, since they are triggered by the environment. In our car domain, the execution of actions accelerate and decelerate respectively increases and decreases the acceleration of the car by one unit. The event engineExplode is triggered when the velocity and acceleration of the car reach a certain threshold. The set of ground processes pr is obtained from the process symbols in Ps. A ground process is defined as follows: Definition 3.1.5 A ground process (Fox and Long [2006]) proc is an instantiation of a process symbol in Ps having a name together with its actual parameters. It consists of a precondition Pre(proc) and an effect Eff(proc). Pre(proc) is a propositional precondition for the process’s activation and consists of atomic propositions formed over either the ground atoms in the planning domain or else relational terms constructed from arithmetic operations applied to PNEs or real values. An Eff(proc) is a numeric postcondition that is a conjunction of additive assignment propositions, the values of which are expressions that are of the form (∗ #t exp) where exp is #t free (exp does not contain any sub-expression that uses #t). Continuous effects are represented by update expressions that refer to the special variable #t.2 The variable #t in the postcondition is syntactic and signifies that the effect of a process is time-dependent, which represents the continuous system dynamics. An example of a ground process in our domain is moving with the atomic proposition running as its activation precondition. The process has a postcondition increase (v) (∗ #t (a)), that is semantically equivalent to dv dt = a. Similarly, the continuous change in distance (d) covered by the car and the time elapsed while moving (runningTime) are modeled with differential equations. 44 3.1. PRELIMINARIES A state s of a PDDL+ domain consists of a time t ∈ R, a logical constituent s l ⊆ P, and a numeric constituent s v that describes the values for the PNEs at that state. The goal condition G is a proposition that can include both atoms formed from the relation symbols and objects of the planning instance and numeric propositions between primitive numeric expressions and numbers (Fox and Long [2006]). The initial state I is the state of the model at time t = 0. The set of objects Os is the entity of interest in the domain. An example of a planning problem in PDDL+ is shown in Listing 2. The problem has an empty set of objects Os, the initial condition I specifies that the predicate running is initially true, the initial value of the functions a, v, d and runningTime are assigned 0 and that of upLimit, downLimit are assigned 1 and -1 respectively. The goal for the car is to travel a minimum distance of 30 units in less than 50 units of time while avoiding an engine explosion, which is caused when the velocity of the car is greater than or equal to 100 units and the acceleration is greater than or equal to 1 unit. This is specified in G with the predicates goalReached and the negation of engineBlown, together with the numeric condition runningTime≤ 50. Listing 3.2: The planning problem in the car domain in PDDL+. ( define ( problem car_prob ) ( : domain car ) ( : i n i t ( running ) (= ( runningTime ) 0) (= ( upLimit ) 1) (= ( downLimit ) −1) (= d 0) (= a 0) (= v 0)) ( : goal ( and ( goalReached ) ( not ( engineBlown ) ) (≤ ( runningTime ) 50 ) ) ) Hybrid automata will be presented pictorially alongside the PDDL+ representation of planning domains in the work, for ease of illustration. We now briefly describe a mapping between the PDDL+ constructs and the components of HA. The variables x∈ V ar in HA map to the PNEs of a PDDL+ model. Recall that a PNE is a grounded function and assumes numeric values. Each location l ∈ Loc maps to a logical state of the PDDL+ model, i.e., a subset of predicates P which are true. The tuple Init =⟨l ini ,S⟩ maps to the initial condition I of the PDDL+ model. l ini represents the predicates in P which are true initially and S represents initial assignment of values to the PNEs. The flow in a location Flow(l) maps to the continuous change due to a process in PDDL+. The invariant in a location Inv(l) maps to a subset of PNEs in PDDL+ that is used to specify the condition on the numeric state of the PDDL+ model that should hold throughout the execution of a process. The set of labels Lab maps to the set of ground actions and events in PDDL+. Note that there is no distinction between an action and an event in an HA. A transition e = (l,a,g,r,l ′ ) maps to a discrete change of logical state from l to l ′ in PDDL+ due to any action or event a. The guard g maps to the precondition, and the reset r maps to the postcondition of the action/event on the PNEs respectively. We present a representation of the car domain as a hybrid automaton in Figure 3.1 below: Definition 3.1.6 A plan φ is a tuple ⟨λ,makespan⟩. For a planning instance with a set of ground actions A, λ is a finite set of triplets ⟨t,act,dur⟩ together with the plan makespan ∈ R. In the triplet λ, t∈ R + is the time instant of executing the action act∈ A and dur ∈ R + is the duration for which the action act remains active in the plan. The makespan is the overall duration of the plan.2 45 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS running goalReached engineBlown accelerate a<upLimit a=a+1 initial a=0 v=0 d=0 t=0 stop (v==0) & (d>=30) engineExplode (a>=1) & (v>=100) decelerate a>downLimit a=a-1 Inv: t<=50; Figure 3.1: The hybrid automaton model of the Car domain Note that a plan does not report events and processes. This is because neither the occur- rence of an event nor its duration can be controlled by a planner. For the same reason, since processes model continuous changes in the system, they are not under the control of the planner. The duration of instantaneous actions in a plan is always zero. makespan captures the total time spent by a system during which a plan is active. A system may spend time dwelling due to the application of processes. As processes are not visible in the plan, such dwelling times are captured in the passage of time between consecutive applications of actions. The dwell time before any action appears in the plan is captured by the time point at which the first action appears. However, the dwell time that may appear after the application of the last action in the plan before reaching the goal state needs to be derived from the makespan. Consider Case 1 in Figure 3.2 where for a given goal state such as (B)∧ (b == 5), a plan may consist only of an application of the action act. However, the system needs to dwell 5 time units even after the application of the action act to reach the goal state, which is not visible in the plan but captured in the makespan. Another typical situation may arise when a goal state could be reached only by dwelling for a duration. In such a case, the plan may not consist of any action. For example, consider Case 2 in Figure 3.2 where a goal state (A)∧ (a == 5) can be reached only by dwelling 5 time units in the location A. The plan in this case consists of an empty list of actions. Here, the dwelling time of the system is captured by the makespan for such an empty plan. (a) Case1(b) Case2 Figure 3.2: Example scenarios where makespan is useful. In the figure, t encodes time, and a and b are variables. In our contrastive plan explanation framework, which is discussed in Section 3.2, some of the contrastive questions are concerning the sequence of actions appearing in a plan. We 46 3.2. EXPLANATION FRAMEWORK now therefore define an action sequence in a plan φ. Definition 3.1.7 Given a planning instance Π and a plan φ, an action sequence is an ordered set of ground actions in φ ordered by their time of appearance in φ.2 Listing 3 shows the generated plan on the planning problem shown in Listing 2 on the car domain of Listing 1, generated by SMTPlan+, a planning tool for hybrid systems. Listing 3.3: A plan generated by SMTPlan+ on the car domain on a planning problem in Listing 2. The plan duration is 32.0 units. Time ActionDuration 0 . 0 :( a c c e l e r a t e )[ 0 . 0 ] 1 . 0 :( d e c e l e r a t e )[ 0 . 0 ] 3 1 . 0 :( d e c e l e r a t e )[ 0 . 0 ] 3 2 . 0 :( stop )[ 0 . 0 ] makespan: = 32.0 units. In the following, we describe the intuition behind contrastive questions for hybrid system plans, and discuss how the questions can be translated to PDDL+ for use in SMTPlan+ for alternative plan generation. The hybrid automata description of the corresponding PDDL+ models are also shown for clarity on model semantics and as a visual aid for the readers. 3.2 Explanation Framework In this section, we address contrastive questions on a plan in a hybrid system, such as (a) why execute action A and not action B at a point in the plan? (b) why apply an action for τ duration at a point in the plan and not more or less? (c) why does an action sequence a i ,a i+1 ,...,a i+n appear in the plan and not any other sequence? (d) why does a plan have a given duration/length and not less? To address the class (a) questions, there are finitely many contrastive plan alternatives that are to be considered, given that the number of discrete actions is finitely many. We propose to address explanations of contrastive questions over the actions by a contrastive explanation framework, where the human agent may ask contrastive questions over the action space, and our framework answers with alternative plans and their costs until the human agent builds trust over the generated plan. Addressing contrastive questions of class (b) requires other novel techniques since we cannot explicitly consider all possible alternate dwell times and re- plan options as there are infinitely many. Here, we intend to explore solutions by building hypothetical models, with added time variables and constraints on them, and then, re-plan over these hypothetical ones to obtain an explanation. For example, we may introduce a new time variable π and add a constraint such as (π < τ ) in the original model to obtain a hypothetical model H. Similarly, we can also obtain another hypothetical model H’ by adding the constraint (π > τ ) and executing a re-plan step. Re-planning on H and H’ and comparing the results with the original plan may provide an explanation of (b). The class (c) questions require preserving the action sequence that appears before and after the action sequence a i ,a i+1 ,...,a i+n while allowing any sequence of actions other than the 47 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS one specified to happen in between. The contrastive questions of class (d) relate to queries to investigate if there are plans of shorter duration or shorter length. Such questions can use the iterative strength of our explanation framework to find an optimal plan in terms of duration/length. In this work, we propose the following collection of contrastive questions and techniques to provide their explanation: 1. Why did the planner choose to do action A and not B instead? 2. Why did the planner not choose to do an action later in the plan? 3. Why did the planner not choose to do an action earlier in the plan? 4. Why did the planner choose to do an action in the plan, instead of not doing it? 5. Why not have fewer occurrences of an action in the plan? 6. Why is the accumulative duration of the plan not less? 7. Why did an action sequence appear in the plan? 8. Why is the length of the plan not less? 3.2.1 Contrastive Explanation Framework In this section, we present an explanation framework for contrastive questions about a planning instance for a hybrid system. The framework is an iterative and collaborative model based on (Krarup et al. [2021a]), where the collaboration comes as the four-stage mixed-initiative process as: (i) a user asks a contrastive question by observing a plan; (i) a formal question is formed by deriving the constraints from the user question; (i) these constraints are compiled into a hypothetical planning model; (iv) a solution for the hypothetical model is formed. The alternative solution thus derived is called a hypothetical plan. It contains the contrast cases expected by the user, and hence can be compared with the original plan. A comparison between the plans serves as an explanation for the user’s query. However, before initiating the discussion on our contrastive plan explanation frame- work for a hybrid system, we formally define an explanation problem as below: Definition 3.2.1 An explanation problem is a tuple E = ⟨Π,φ,Q⟩ (Cashmore et al. [2019]), where Π represents the planning instance (see Def. 3.1.1), φ is the plan generated by a planner (see Def. 3.1.6), and Q represents the contrastive question posed by the user. 2 To bring forth explanations to the above-mentioned set of contrastive questions, we pro- pose a Contrastive Plan Explanation Framework based on (Krarup et al. [2019]) which is given in the context of discrete systems and problem domains. We extend their framework to hybrid system models. Additionally, we propose a set of algorithms for each contrastive question that constructs the hypothetical models from the corresponding user questions. 48 3.2. EXPLANATION FRAMEWORK The framework takes an explanation problem E and produces a contrastive explanation CE (see Def. 3.2.6), by contrasting a given plan with alternate ones produced with hypo- thetical models, as an argument about why the generated plan is to be chosen for execution over the possible alternatives. A schematic of this framework is shown in Figure 3.3. HModel Generation HModel HPlan Synthesis HPlan Model Planner Plan Contrastive Explanation User Question ? Figure 3.3: The Contrastive Plan Explanation Framework. Based on the contrastive question, the original model is modified to a hypothetical model that enforces the planner to generate a contrastive plan as expected by the user. We denote the original planning instance as Model Π and the hypothetical model formed after incorporating the necessary changes in the original model as HModel Π ′ . A plan obtained for an HModel is a hypothetical plan denoted as HPlan φ ′ . To formalize the discussion of contrastive plan explanation, we now introduce the relevant definitions. A constraint property is formed from the question Q, which represents some user-imposed constraints over a plan φ and can be defined as follows. Definition 3.2.2 For a planning instance Π and a plan φ, a constraint property (Krarup et al. [2021a]) is a quantifier-free first-order logic predicate ψ over φ representing constraints from a user question Q.2 The constraint operator, which encapsulates a constraint property within a planning model, is given as follows. Definition 3.2.3 (Krarup et al. [2021a]) A constraint operator × is defined so that, for a planning instance Π and any constraint property ψ, Π× ψ constructs a HModel Π ′ , satisfying the condition that any plan for Π ′ is a plan for Π that also satisfies ψ.2 Definition 3.2.4 HModel is a new planning instance Π ′ constructed from the planning instance Π which encapsulates a constraint property ψ of a user question Q. Π ′ can be defined as: Π ′ = (⟨Fs ′ ,Rs ′ ,As ′ ,Es ′ ,Ps ′ ,arity ′ ⟩,⟨Os ′ ,I ′ ,G ′ ⟩) where Fs’, Rs’, As’, Es’, Ps’, arity’, Os’, I’ and G’ indicate modifications on the corres- ponding components.2 49 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS Definition 3.2.5 HPlan is a hypothetical plan φ ′ produced by the planner over an HModel Π ′ which satisfies the constraint(s) posed in question Q by a user.2 Contrastive Explanation Metrics: we compare the HPlan φ ′ with the original plan φ. The comparison is done by adopting a contrastive explanation metric defined as follows. Definition 3.2.6 To show the differences between φ and an HPlan φ ′ , a contrastive ex- planation CE (Cashmore et al. [2019]) of a plan φ on a hybrid-system domain D for a given question Q is a tuple ⟨E,Q⟩, where E consists of the following six components: • Remove (φ,φ ′ ): The set of actions removed from the original plan. • Add (φ,φ ′ ): The set of actions added to the alternate plan. • Common (φ,φ ′ ): The set of actions present in both the original and the alternate plan. • dwell-diff: is the set (p,t) | p ∈ Ps and t = t orig − t alt . The set contains a pair (p,t) for every p in the set of process symbols Ps where the corresponding t shows the difference of the dwell time in process p due to this alternate plan. Here, t orig and t alt denote the dwell time in p in the original plan φ and the alternate plan φ ′ , respectively. • diff-cost τ : The makespan of the alternate plan minus the makespan of the original plan. • diff-cost len : The number of occurrences of actions in the alternate plan minus the number of occurrences of actions in the original plan. Now in the light of PDDL+, from a given planning instance Π, and a contrastive question Q, we denote a compilation of a hypothetical planning instance by Compilation(Π,Q). The result of this compilation is a modified planning instance Π ′ = (⟨Fs ′ , Rs ′ , As ′ , Es ′ , Ps ′ , arity ′ ⟩,⟨O ′ s, I ′ , G ′ ⟩). Here, the HModel Π ′ is derived as Π × ψ, where ψ is the constraint property derived from Q. Fs ′ , Rs ′ , As ′ , Es ′ , Ps ′ , arity ′ , O ′ s, I ′ , and G ′ represent modifications on the corresponding elements. However, if the user wants to ask questions iteratively about the model, then Π ′ can also be used as an input iteratively. This allows the user to stack questions, further increasing the understanding of the plan through iterative questioning. 3.3 Contrastive Questions In this section, we consider the case study of the car domain (Listing 1) and an instance of a planning problem (Listing 2) to illustrate our explanation framework for each of the earlier discussed contrastive questions. We use SMTPlan+ as the planner for our experiments. We start by taking a user question, a formal question is formed by deriving the constraints, and these constraints are compiled into a hypothetical planning model HModel. Then, a solution for the HModel is formed. The alternative solution derived is 50 3.3. CONTRASTIVE QUESTIONS called a hypothetical plan HPlan. The compilation of the hypothetical model (HModel) is shown for the corresponding question, and the resulting HPlan generated by SMTPlan+ is compared with the original plan. A contrastive explanation, as per our comparison metric, is reported for each case to show the differences between the original plan and the alternate one. These comparisons between the plans serve as an explanation for the user’s query. 3.3.1 Replacing an Action by Another in the Plan A user might question the appearance of an action at a certain stage of the plan, and may look for an alternate option. For example, for a given plan φ, a contrastive question Q can be asked of the form: Why did the planner choose to perform action A in state S rather than action B? We are given a plan φ consisting of an action sequence ⟨a 1 , a 2 , . . . , a i , . . . , a n ⟩ that leads the system from an initial state I to the goal G. Let the system be in state S after the application of the grounded action a i−1 . Then, the question can be more precisely asked as ‘why was the action ground(a i ) at state S chosen to be applied, rather than the action ground(b)?’. For example, by observing the example plan in Listing 3, the user might ask the question: ‘Is it possible to replace the first instance of decelerate with accelerate’? An intuitive argument supporting the preference to replace the decelerate action with the accelerate action early in the plan might be due to the fact that it can enable the car to travel faster and the appearance of decelerate action earlier in the plan can slow down the car. To generate the HPlan φ ′ , a compilation is formed such that the action ground(b) appears in the plan in place of the action ground(a i ) resulting in a new state S ′ , followed by a new sub-plan φ s containing an arbitrary sequence of grounded actions ⟨c 1 , c 2 , . . . , c m ⟩ leading to the goal G. The HPlan is then the initial set of actions of the original plan φ concatenated with b and the new sub-plan φ s as below: φ ′ :⟨a 1 ,a 2 ,...,a i−1 ,b,c 1 ,c 2 ,...,c m ⟩ To obtain the state S ′ , we need to preserve the sequence of grounded actions ⟨a 1 , a 2 , . . . , a i−1 ⟩, which is followed by ground(b) in a valid plan. For that, the corresponding HModel is constructed as follows: For each action a j in ⟨a 1 , a 2 , . . . , a i−1 ⟩, we introduce a new action act j which can only appear in the preserved sequence and a new predicate has_done_act j which represents that the action act j has been applied. For the action b, we introduce a new action b1 and a new predicate has_done_b. b1 will only appear to replace a i in a plan, whereas b may appear in any other place in a plan. The goal is extended to include the grounded predicate has_done_b to enforce that the user suggested action ground(b) replaces the action ground(a i ) in the original plan φ. This is achieved by constructing HModel Π ′ as follows. 51 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS Π ′ = (⟨Fs,Rs ′ ,As ′ ,Es,Ps,arity ′ ⟩,⟨Os,I,G ′ ⟩) where • Rs ′ : Rs∪(has_done_act j , ∀j ∈ [1,i− 1]),has_done_b; • As ′ : As∪(act j , ∀j ∈ [1,i− 1]),b1; • arity ′ (act j ) = arity(a j ), ∀j ∈ [1,i− 1]; • arity ′ (b1) = arity(b); • arity ′ (has_done_act j ) = 0, ∀j ∈ [1,i− 1]; • arity ′ (has_done_b) = 0; • G ′ : G∧ ground(has_done_b); where the first action in the sequence act 1 has been extended with an added precondition ¬(has_done_act 1 ) Pre(act 1 ) = Pre(a 1 )∧¬(has_done_act 1 ) and an added effect (has_done_act 1 ), which indicates that the action has been applied in the plan, Pre(act 1 ) = Pre(a 1 )∧ (has_done_act 1 ) for all j ∈ [2,i−1], each act j has been extended with added precondition has_done_act j−1 and ¬(has_done_act j ) which ensures that the previous action in the sequence has been applied already, Pre(act j ) = Pre(a j )∧ (has_done_act j−1 )∧¬(has_done_act j ) and added effect has_done_act j , which indicates that the action has been applied in the plan, Eff + (act j ) = Eff(a j )∪has_done_act j Action b1 extends the action b with an added precondition has_done_a i−1 and an added effect has_done_b. The precondition has_done_a i−1 ensures that b1 will appear only after the sequence ⟨act 1 , act 2 , . . . , act i−1 ⟩ and the effect has_done_b would enforce its application in a valid plan. Pre(b1) = Pre(b)∧ (has_done_act i−1 ) Eff + (b1) = Eff(b)∪has_done_b All the existing actions of the original model Π are extended with the added precondition has_done_b, which indicates that they may appear in the rest of the plan. Pre(a) = Pre(a)∧ (has_done_b),∀a∈ As ′ All the added actions have semantics similar to their original counterparts in the domain; we only restrict where they can appear in a valid plan. Therefore, act j in the HPlan φ ′ 52 3.3. CONTRASTIVE QUESTIONS represents a j in the original plan φ and should not be treated as a different action. Simil- arly, b1 and b represent the same action in the HPlan except for a different precondition. Now, we show that the above construction of the HModel is sound, in other words, it produces an HPlan which is expected by the user. Lemma 3.3.1.1 The construction of the HModel is sound. Proof: A valid HPlan φ ′ for the HModel Π ′ consists of an action sequence ⟨act 1 , act 2 , . . . , act i−1 b1⟩ which is followed by some sequence of actions in As ′ . As (has_done_b) is included as a goal constraint to ensure that b1 appears in a plan whereas it’s precondition has_done_act i−1 ensures that it can appear only after the action sequence ⟨act 1 , act 2 , . . . , act i−1 ⟩ has appeared in the plan. Moreover, the precondition Pre(act j ) = Pre(a j )∧ (has_done_act j−1 )∧¬(has_done_act j ) ensures that actions appearing in the preserved sequence do not repeat. Other domain actions have the precondition has_done_b, which means that they can appear only after b1 has been applied in a plan. Hence, any valid plan must start with an action sequence ⟨act 1 , act 2 , . . . , act i−1 ⟩ that is followed by b1 which replaces the action a i in the original plan φ and finally, succeeded by some action sequence ⟨c 1 , c 2 , . . . , c m ⟩ leading to the goal. This shows that our construction is sound. 2 Example 3.3.1.1 Consider the user question above where b is accelerate and a i is the grounded decelerate action (the second ground action) in the original plan φ. The plan φ consists of a sequence of grounded actions: φ :⟨accelerate,decelerate,decelerate,stop⟩ To reflect the user suggestions in the HPlan φ ′ , a valid plan must preserve the first groun- ded action accelerate in the plan whereas the second grounded action decelerate must be replaced with another grounded accelerate action. To generate the HPlan, the compilation is formed as discussed above. We introduce a new action accelerate1 which appears only at the start of a valid plan preserving the place of the first ground action accelerate in the original plan φ and a new predicate has_done_acc1 which represents the fact that action accelerate1 has been applied. Sim- ilarly, we introduce another new action accelerate2 to replace the second grounded action decelerate in the original plan φ to form a valid HPlan φ ′ and a predicate has_done_acc2. The latter indicates that the grounded action accelerate2 has been applied to replace the grounded decelerate action from a valid HPlan. All existing actions of the domain have been extended with the precondition has_done_acc2. The modifications are shown below: • Pre(accelerate1) = Pre(accelerate)∧ ¬(has_done_acc1) • Eff + (accelerate1) = Eff(accelerate)∪ (has_done_acc1) • Pre(accelerate2) = Pre(accelerate)∧ (has_done_acc1)∧ ¬(has_done_acc2) • Eff + (accelerate2) = Eff(accelerate)∪(has_done_acc2) • Pre(accelerate) = Pre(accelerate)∧ (has_done_acc2) 53 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS • Pre(decelerate) = Pre(decelerate)∧ (has_done_acc2) • Pre(stop) = Pre(stop)∧ (has_done_acc2) The modified domain is shown in Listing 4: Listing 3.4: Modified domain with accelerate, accelerate1 and accelerate2 action. ( : p r e d i c a t e s ( running ) ( engineBlown ) ( goalReached ) ( has_done_acc1 ) ( has_done_acc2 )) ( : f unc ti ons ( d) ( v ) ( a) ( upLimit ) ( downLimit ) ( runningTime ) ) . . . ( : action a c c e l e r a t e 1 : parameters () : precondition ( and ( running ) (< ( a) ( upLimit )) ( not ( has_done_acc1 )) ) : e f f e c t ( and ( increase (a ) 1) ( has_done_acc1 ) )) ( : action a c c e l e r a t e 2 : parameters () : precondition ( and ( running ) ( has_done_acc1 ) ( not ( has_done_acc2 )) ) : e f f e c t ( and ( increase (a ) 1) ( has_done_acc2 ) )) ( : action a c c e l e r a t e : parameters () : precondition ( and ( running ) (< ( a) ( upLimit )) ( has_done_acc2 )) : e f f e c t ( and ( increase (a ) 1))) ( : action d e c e l e r a t e : parameters () : precondition ( and ( running ) (> ( a) ( downLimit )) ( has_done_acc2 )) : e f f e c t ( and ( decrease (a ) 1))) ( : action stop : parameters () : precondition ( and (= ( v ) 0) (>= (d ) 30) ( not ( engineBlown )) ( has_done_acc2 )) : e f f e c t ( goalReached )) ) The initial and goal states are modified as shown below: ( : i n i t ( running ) (= ( runningTime ) 0) (= ( upLimit ) 1) (= ( downLimit )−1) (= d 0) (= a 0) (= v 0) (= c 0)) ( : goal ( and ( goalReached ) ( not ( engineBlown )) (<= ( runningTime ) 50) ( has_done_acc2 )) ) The hybrid automaton model of the modified domain is shown in Figure 3.4, where two new actions accelerate1 and accelerate2 are added to the model. It may be noted that the precondition (a < upLimit) of the accelerate action in the original model is omitted for the accelerate2 action in the HModel as it restricts two consecutive occurrences of accelerate actions in the plan. The generated HPlan by SMTPlan+ is: Time ActionDuration 0 . 0 :( a c c e l e r a t e 1 )[ 0 . 0 ] 1 . 0 :( a c c e l e r a t e 2 )[ 0 . 0 ] 2 . 0 :( d e c e l e r a t e )[ 0 . 0 ] 3 . 0 :( d e c e l e r a t e )[ 0 . 0 ] 8 . 0 :( d e c e l e r a t e )[ 0 . 0 ] 12 .0 :( stop )[ 0 . 0 ] 54 3.3. CONTRASTIVE QUESTIONS running goalReached engineBlown accelerate1 has_done_acc1==false a<upLimit a=a+1 has_done_acc1=true accelerate2 has_done_acc1==ture has_done_acc==false a=a+1 has_done_acc2=true accelerate has_done_acc2==true a<upLimit a=a+1 initial a=0 v=0 d=0 t=0 stop (v==0) & (d>=30) has_done_acc2==true engineExplode (a>=1) & (v>=100) decelerate has_done_acc2==true a>downLimit a=a-1 Inv: t<=50, has_done_acc2; Figure 3.4: The hybrid automaton for the corresponding HModel of the question is addressed in sect 4.1. makespan: = 12.0 units. We observe that the synthesized plan is shorter in makespan but longer in length, with a greater number of actions. Contrastive Explanation: The contrastive explanation for the above user question is as follows: • Remove (φ,φ ′ ): accelerate action is removed from the original plan; • Add (φ,φ ′ ): accelerate1 and accelerate2 are added to the alternate plan; • Common (φ,φ ′ ): decelerate and stop; • dwell-diff : moving, 20; • diff-cost τ : −20; • diff-cost len : 2. A user of the framework can draw the following conclusion from the contrastive explana- tion: Conclusion: Replacing decelerate with an accelerate action at time instant 1 provides a shorter plan in terms of the makespan but a longer plan in terms of the number of applied actions. 3.3.2 Restricting an Action to Appear After a Certain Time A user might be interested in seeing the consequences of delaying the appearance of an action in a plan. In such a case, a formal question Q can be: Why did the planner choose to do an action at a particular time instant, why not later? For example, given a plan φ which consists of an ordered sequence of ground actions ⟨a 1 , a 2 , . . . , a i , . . . , a n ⟩, the user might ask ‘why the ground(a i ) appeared in the plan at time t, why not later?’. For the example plan in Listing 3, a question might be: 55 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS ‘Why is the ground action (decelerate) used at time 1.0, why not later?’ Since the goal is to travel a distance of at least 30 units within 50 units of time, this question attempts to understand the rationale of a decelerate action that will slow down the car, which may result in either the car taking more than 50-time units to cover the distance or the car coming to a halt before traveling a distance of 30 units. To generate the HPlan φ ′ to answer such a question, a compilation is formed such that the action ground(a i ) is restricted to appear after the time t in a plan to be valid. Let T v be the time variable that captures the time elapsed in the system during which a plan is active. For that, we create a separate process that remains active throughout the execution of the plan and updates the value of T v using the #t literal. A new constraint (T v > t) is included, which is used to extend the precondition of the action a i , forcing it to appear only after the time t. The compiled HModel Π ′ is: Π ′ = (⟨Fs ′ ,Rs,As ′ ,Es,Ps,arity ′ ⟩,⟨Os,I,G⟩) where • Fs ′ = Fs∪T v • As ′ = a ′ i ∪ As\a i , (here, \ represents the set difference operation). • arity ′ (T v ) = arity ′ (a ′ i ) = arity(a i ) • arity ′ (a ′ i ) = arity(a i ) where the new action a ′ i extends a i with the added precondition (T v > t), i.e. Pre(a ′ i ) = Pre(a i )∧ (T v > t) The action a ′ i represents the original action a i in the domain and should be considered as the same action. It is only restricted to appear in a plan after a certain time. The con- struction proposed for the HModel is sound, and it produces an HPlan which is expected by the user. Lemma 3.3.2.1 The construction of the HModel is sound. Proof: A valid HPlan must apply a ground (a ′ i ) action after a time t. As the action a ′ i has an added precondition (T v > t), it can appear only after time t in any plan.2 Example 3.3.2.1 Consider the user question above. Let a i be the grounded decelerate action that appeared at time 1.0 in the plan φ. To answer this question, a compilation to form the HModel follows the above discussion, where decelerate1 replaces the decelerate action in the domain. The grounded decelerate1 action is constrained to appear later than the time 1.0 in the HPlan. The function symbol runningTime, which is updated by the process moving, encodes the elapsed time in the domain. The precondition of the decelerate1 action is extended with an additional constraint (runningTime > 1). Pre (decelerate1) = Pre (decelerate)∧ (runningTime > 1) where decelerate ∈ As. The changes encoded in PDDL+ are shown below: 56 3.3. CONTRASTIVE QUESTIONS Listing 3.5: The modified decelerate action with the added constraint. ( : action d e c e l e r a t e 1 : parameters () : precondition ( and ( running ) (> (a) ( downLimit )) (> ( runningTime ) 1)) : e f f e c t ( and ( decrease ( a) 1))) The modified hybrid automaton model is shown in Figure 3.5 where runningTime is mapped to the variable t. The constraint (t >1) is added in the self-transition with the label decelerate1. running goalReached engineBlown accelerate a<upLimit a=a+1 initial a=0 v=0 d=0 t=0 stop (v==0) & (d>=30) engineExplode (a>=1) & (v>=100) decelerate1 a>downLimit t > 1 a=a-1 Inv: t<=50; Figure 3.5: The hybrid automaton for the corresponding HModel of the question is addressed in sect. 4.2. The resulting HPlan given by SMTPlan+ is: Time ActionDuration 0 . 0 :( a c c e l e r a t e )[ 0 . 0 ] 2 . 0 :( d e c e l e r a t e 1 )[ 0 . 0 ] 16 .0 :( d e c e l e r a t e 1 )[ 0 . 0 ] 18 .0 :( stop )[ 0 . 0 ] makespan: = 18.0 units. We observe that the plan originally synthesized by the planner is not optimal on the plan duration as the contrastive plan is of shorter makespan. Contrastive Explanation: The contrastive explanation for the above user question is as follows: • Remove (φ,φ ′ ): The decelerate action is removed from the original plan; • Add (φ,φ ′ ): The decelerate1 action is added to the alternate plan; • Common (φ,φ ′ ): accelerate, and stop; • dwell-diff : moving, 14; • diff-cost τ : −14; • diff-cost len : 0, as the original plan has same length. A user of the framework can draw the following conclusion from the contrastive explana- tion: 57 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS Conclusion: The original plan generated by the planner is not optimal with respect to the duration of the plan for the planning problem instance. The alternate plan is better than the original, having a lesser makespan. 3.3.3 Restricting an Action to Appear Before a Certain Time In contrast to the question in Section 3.3.2, a user might be interested in hastening the appearance of an action in a plan. In such a case, for a given plan φ, a formal question Q can be asked of the form: Why did the planner choose to do an action at a particular time instant, why not earlier? The user may be interested to know whether the goal could be achieved by choosing to do an action earlier in the plan. For example, given a plan φ consisting of ground actions ⟨a 1 , a 2 , . . . , a i , . . . , a n ⟩, the user might ask ‘why the ground(a i ) appears in the plan at time t, why not earlier?’. For the plan in Listing 3, a pertinent question can be: ‘Why is the action (decelerate) used at time instant 1.0, why not earlier?’ For a contrastive explanation of a question of this type, a compilation is formed such that an action ground(a i ) strictly does appear before time t in HPlan. To construct the HModel, we extend the original action a i to a i1 while also retaining a i in the domain. The action a i1 is constrained to appear in an HPlan strictly before time t. We introduce a new function symbol T v to encode the system time, which is then used to form a constraint (T v < t). Similar to the previous discussion, a process that remains active throughout the plan duration is created to update the value of T v . The precondition of the action a i1 is extended with the constraint (T v < t), and a new predicate symbol do_before_t is introduced, which is used to extend the effect of the action a i1 . The predicate is also used as a goal constraint to enforce that the action a i1 always appears before time t in a valid plan. The action a i acts similarly as in the original model, such that it may also be available to use later in the plan. The compiled HModel Π ′ is: Π ′ = (⟨Fs ′ ,Rs ′ ,As ′ ,Es,Ps,arity ′ ⟩,⟨Os,I,G ′ ⟩) where • Fs ′ = Fs∪T v • Rs ′ = Rs∪do_before_t • As ′ = As∪a i1 • arity ′ (T v ) = 0 • arity ′ (do_before_t) = 0 • arity ′ (a i1 ) = arity(a i ) • G ′ : G∧ ground(do_before_t) 58 3.3. CONTRASTIVE QUESTIONS where the new action a i1 extends a i with the added precondition and effect, i.e. Pre(a i1 ) = Pre(a i )∧ (T v < t) Eff + (a i1 ) = Eff(a i )∪(do_before_t) The added action a i1 has similar semantics as the original action a i in the domain; it is only restricted to appear in a plan before a certain time. The actions a i1 and a i should be viewed as actions with the same interpretation except that their preconditions and effects are different. The constructed HModel is sound and it produces an HPlan which is expected by the user. Lemma 3.3.3.1 The construction of HModel is sound. Proof: A valid HPlan must apply a ground (a i1 ) action before a time t. As the action a i1 has an added precondition (T v < t), it can be applied only before the time instance t in a plan.2 Example 3.3.3.1 Consider the user question above, let a i be the grounded decelerate action that appears at time 1.0 in the plan φ in Listing 3. To answer this question, the compilation to form the HModel follows the above discussion. A new action decelerate1 is introduced. The grounded decelerate1 action is constrained to appear earlier than the time 1.0 in an HPlan. The function symbol runningTime, which is updated by the process moving, encodes the elapsed time in the domain. The precondition of the decelerate1 action is extended with an additional constraint (runningTime < 1). Pre(decelerate1) = Pre(decelerate)∧ (runningTime < 1) and the effect of decelerate1 is extended with the predicate (do_before_1). Eff + (decelerate1) = Eff + (decelerate)∪(do_before_1) where decelerate ∈ As and do_before_1 is a new predicate symbol introduced. The goal is also extended with an additional constraint (do_before_1). The modifications in the domain are shown in Listing 6: Listing 3.6: The modified domain with the decelerate1 action. ( : action d e c e l e r a t e 1 : parameters () : precondition ( and ( running ) (> ( a) ( downLimit )) (< ( runningTime ) 1)) : e f f e c t ( and ( decrease (a ) 1) ( do_before_1 )) ) The goal state is modified as shown below: ( : goal ( and ( goalReached ) ( not ( engineBlown )) ( do_before_1 ) (<= ( runningTime ) 50))) The model as a modified hybrid automaton is shown in Figure 3.6 where the action decelerate1 is introduced. We map runningTime to the variable t. The proposition (do_before_1) is used in the goal state to force the ground action decelerate1 to appear in the plan before time t. The resulting HPlan given by SMTPlan+ is shown below: 59 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS running goalReached engineBlown accelerate a<upLimit a=a+1 initial a=0 v=0 d=0 t=0 stop (v==0) & (d>=30) engineExplode (a>=1) & (v>=100) decelerate1 a>downLimit t < 1 a=a-1 decelerate_before=ture Inv: t<=50, (decelerate_before) decelerate1 a>downLimit a=a-1 Figure 3.6: The hybrid automaton for the corresponding HModel of the question is addressed in sect. 4.3. TimeActionDuration 0 . 0 :( a c c e l e r a t e )[ 0 . 0 ] 0. 75 :( d e c e l e r a t e 1 )[ 0 . 0 ] 40.75:( d e c e l e r a t e )[ 0 . 0 ] 41 .5 :( stop )[ 0 . 0 ] makespan: = 41.5 units. We see that the decelerate action now appears earlier. However, the plan is of a longer makespan. This observation stands as an explanation of why the decelerate action was taken at time instance 1 in the original plan. Contrastive Explanation: • Remove (φ,φ ′ ): No action is removed; • Add (φ,φ ′ ): decelerate1 is added; • Common (φ,φ ′ ): accelerate, decelerate and stop; • dwell-diff : moving, −9.5; • diff-cost τ : 9.5; • diff-cost len : 0, as the original plan has same length. A user of the framework can draw the following conclusion from the contrastive explana- tion: Conclusion: The application of decelerate action later than 1-time unit results in a longer plan in terms of the makespan. 3.3.4 Barring an Action from Appearing in the Plan The appearance of a certain action may seem confusing to a user at times, and barring the action from appearing in a plan may seem reasonable. To simulate such a scenario, the following contrastive question Q can be asked: 60 3.3. CONTRASTIVE QUESTIONS Why is an action used in the plan, rather than not being used? For example, given a plan φ of ground actions ⟨a 1 , a 2 , . . . , a i , . . . , a n ⟩, the user might ask ‘why is the action a i used in the plan, rather than not being used?’. To construct the HModel to answer this question, a compilation is formed such that the action ground(a i ) is barred from appearing in a generated HPlan. To do this, dropping the action a i from the original model will suffice to serve the purpose. The compiled HModel Π ′ is: Π ′ = (⟨Fs,Rs,As ′ ,Es,Ps,arity⟩,⟨Os,I,G⟩) where As ′ = As \ a i , (\ represents the set difference operation). The constructed HModel is sound and will produce an HPlan which is expected by the user, if there exists any. Lemma 3.3.4.1 The construction of the HModel is sound. Proof: A valid HPlan must not contain the specified action a i . As the action a i is deleted from the set of available actions in the domain, the planner is bound to choose actions only from the available set of actions in a valid plan.2 Example 3.3.4.1 In the plan of Listing 3, the action a i can be the decelerate action in the domain. A user might think the action decelerate is slowing down the car and may be detrimental to an efficient plan towards achieving the goal of traveling 30 units in a short time. The contrastive question that could be asked: ‘Why is the decelerate action used in the plan, rather than not being used?’ To answer this contrastive question, the HModel is formulated following the above discus- sion. We drop the action decelerate from our domain shown in Listing 1 so that it cannot be used in the HPlan. The modified hybrid automaton model of the domain is shown in Figure 3.7, where the transition with the label decelerate is now removed. running goalReached engineBlown accelerate a<upLimit a=a+1 initial a=0 v=0 d=0 t=0 stop (v==0) & (d>=30) engineExplode (a>=1) & (v>=100) Inv: t<=50; Figure 3.7: The hybrid automaton for the corresponding HModel of the question addressed in sect 4.4 As a result of this change in the model, the planner is unable to generate a valid plan. The contrastive explanation is shown below: 61 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS • Remove (φ,φ ′ ): Not available, as there is no alternate plan; • Add (φ,φ ′ ): Not available, as there is no alternate plan; • Common (φ,φ ′ ): Not available, as there is no alternate plan; • dwell-diff : Not available; • diff-cost τ : ∞; • diff-cost len : ∞. We observe that not having the decelerate action in the model results in no valid plan being synthesized by the planner. Hence, in the car domain, a user of the framework can draw the following conclusion from the contrastive explanation: Conclusion: Barring the decelerate action from appearing in the plan leads to no valid plan in this domain. Remark: The planning problem is undecidable for hybrid systems in general (Alur et al. [1995a]). When the planner cannot generate a valid plan for a planning problem, it cannot be asserted whether it is due to the underlying undecidability or that the planning problem admits no valid plan. This leads to a limitation in our explanation framework since incorrect explanations may result in such cases. To address this limitation to a certain extent, we propose a technique in Section 3.4. 3.3.5 Restricting an Action to Occur Less Than a Certain Number of Times At times, from a human perspective, it may not be trivial to figure out why an action is performed a certain number of times. Thus, it is legitimate to ask the following contrastive question: Why did the planner choose to do an action n number of times, why not fewer? For example, given a plan φ of ground actions, a grounded action b may appear multiple times in the plan. Let the action b appear n times in φ. Then, a user might ask ‘why did the planner choose to do the action ground(b) n number of times, why not fewer?’. From our plan in Listing 3, the question can be: ‘Why is the action (decelerate) taken twice in the plan, why not once?’ To generate the HPlan, a compilation is formed such that the number of occurrences of the grounded action b is less than n in a valid plan. A new function symbol s is introduced, which keeps track of the number of grounded actions b in a plan, and is initially set to 0. The action b ′ extends b with an additional add effect (s = s + 1). The goal condition is also extended with an additional constraint (s < n) such that a valid plan will always contain fewer than n grounded b actions. The compiled HModel Π ′ is: Π ′ = (⟨Fs ′ ,Rs,As ′ ,Es,Ps,arity ′ ⟩,⟨Os,I ′ ,G ′ ⟩) where 62 3.3. CONTRASTIVE QUESTIONS • Fs ′ = Fs ∪ s • As ′ = b ′ ∪ As\b, (here, \ represents the set difference operation) • arity ′ (s) = 0 • arity ′ (b) = arity(b) • I ′ = I ∪PNE(s = 0), s∈ Fs ′ • G ′ = G∧ (s < n) where b ′ extends b with an additional add effect, i.e. Eff + (b ′ ) = Eff(b)∪(s = s + 1) The action b ′ represents the original action b in the domain; it is only restricted to appear less than a specified number of times in a plan. The constructed HModel is sound and will always produce an HPlan where b appears less than n times, if any such plan exists. Lemma 3.3.5.1 The construction of the HModel is sound. Proof: In a valid HPlan the ground(b ′ ) action must appear less than n times. On each occurrence of the action b ′ in a plan, the added effect of b ′ increases the value of the function symbol s by 1, which is initially set to 0. Hence, the added goal constraint (s < n) ensures that only those plans are generated where b ′ has appeared less than n times.2 Example 3.3.5.1 Consider the user question above, where b is the decelerate action. The compilation to form the HModel follows the above discussion. A new function symbol s is introduced, whose initial value is set to 0 in the initial state. Let n be two here, which implies that the appearance of decelerate action should be less than twice in a plan. The action decelerate is modified to decelerate1, which is allowed to occur only once in the plan. The action decelerate1 now has an added effect to increase the value of s, together with the other effects of the decelerate action in the original domain: • Pre (decelerate1) = Pre (decelerate) • Eff + (decelerate1) = Eff(decelerate) ∪ (s = s + 1), s∈ Fs. Additionally, a constraint (s < 2) is added in the goal state such that the action decelerate1 appears only once in a plan. The changes made in the domain are shown below: ( : f unc ti ons ( d) ( v ) ( a) ( upLimit ) ( downLimit ) ( runningTime ) ( s )) . . . ( : action d e c e l e r a t e 1 : parameters () : precondition ( and ( running ) (> ( a) ( downLimit )) ) : e f f e c t ( and ( decrease (a ) 1) ( increase ( s ) 1))) The constraint (s < 2) is added in the goal state as shown below: 63 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS running goalReached engineBlown accelerate a<upLimit a=a+1 initial a=0 v=0 d=0 t=0 s=0 stop (v==0) & (d>=30) engineExplode (a>=1) & (v>=100) decelerate1 a>downLimit a=a-1 s=s+1 Inv: t<=50, s<2; Figure 3.8: The hybrid automaton for the corresponding HModel of the question addressed in sect. 4.5. ( : goal ( and ( goalReached ) ( not ( engineBlown )) (<= ( runningTime ) (< ( s ) 2))) Figure 3.8 represents the relevant modifications in the hybrid automaton representation of the domain. As a result of these modifications, no valid plan can be generated by the planner. The contrastive explanation is: • Remove (φ,φ ′ ): Not available, as there is no alternate plan; • Add (φ,φ ′ ): Not available, as there is no alternate plan; • Common (φ,φ ′ ): Not available, as there is no alternate plan; • dwell-diff : Not available; • diff-cost τ : ∞; • diff-cost len : ∞. A user of the framework can draw the following conclusion from the contrastive explana- tion: Conclusion: In this domain, no valid plan can be generated that has fewer than two executions of the action decelerate. 3.3.6 Questioning the Optimality of Plan Duration A user might want to know whether the observed plan is optimal in terms of duration or whether there exists any other plan with a shorter plan-duration. The following contrastive question can be considered: Why is the makespan of the plan not less? 64 3.3. CONTRASTIVE QUESTIONS Recall that makespan is the duration of a plan. As we have seen earlier, the planner may not provide an optimal plan. Therefore, an iterative question ‘Why the makespan is t to complete a task, and not less?’ where t is the makespan of the generated plan in the last successful iteration, can help in questioning the optimality of the plan. In our car domain, a user might ask: ‘Why is the makespan of the plan not less than 32?’ For such a question, we propose an iterative HModel formation that addresses a sequence of user questions. A user may interact iteratively in this framework and can successively view generated HPlans and seek counter explanations by imposing additional constraints on the planning problem. The collection of HModels that can be successively built forms a hierarchical structure rooted at the original model and extended by the incremental addition of new constraint properties (Definition 3.2.2). An iterative Hmodel formation is defined as follows. Definition 3.3.1 Iterative HModel Formation: For a planning instance Π and a plan φ, let ψ i be the set of user imposed constraint properties derived from φ i−1 which is initially empty, i.e. ψ 0 = ∅. Each stage i (initially 0) of this process starts with the planner producing a HPlan φ ′ i for the HModel Π ′ i = Π× ψ i , where × is the constraint operator (Definition 3.2.3) which encapsulates ψ i in Π ′ i .2 Algorithm 3.1 constructs the iterative HModel for the iterative user queries. Algorithm 3.1: Iterative HModel Construction Input: The HModel Π ′ i−1 , and the HPlan φ ′ i−1 where Π ′ 0 and φ ′ 0 are the original model Π and the original plan φ correspondingly; Output: The HModel Π ′ i 1 begin 2 ×: The constraint operator; 3 i = 1; 4while true do 5if (φ ′ i−1 ̸= no-plan) then 6Π ′ i = Π ′ i−1 × ψ i (φ ′ i−1 ) ;/* ψ i (φ ′ i−1 ) is the user imposed constraint from φ ′ i−1 */ 7i = i + 1; 8else 9return HModel cannot be constructed. 10end 11end 12 end Example 3.3.6.1 Consider the user’s question on the optimality of the makespan of a plan. The constraint ψ i can be updated from a valid plan HPlan φ ′ i−1 generated from the previous iteration, as shown below. if (φ ′ i−1 ̸= no-plan) ψ i = (runningTime < makespan(φ ′ i−1 )) where makespan(φ ′ i−1 ) denotes the duration of the plan φ ′ i−1 . For example, in the original problem (see Listing 2), the constraint ψ = (runningTime ≤ 50) leads to a plan of a duration of 32.0 (see Listing 3). So in the next iteration, the constraint ψ 1 in Π ′ 1 is 65 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS modified to runningTime < 32 to find a plan with a duration of less than 32.0-time units to check if any such plan exists. This iterative process is continued till we can conclude that no plan exists for the given runningTime constraint. The iterative HModel Π ′ i is given below: Π ′ i = Π ′ i−1 ×ψ i (φ ′ i−1 )(3.1) where Π ′ i is the HModel formed in the i-th iteration, φ ′ i−1 is the HPlan generated on Π ′ i−1 and ψ i (φ ′ i−1 ) defines a constraint generated by observing φ ′ i−1 . The new Π ′ i is formed from Π ′ i−1 by imposing the constraint ψ i . The constraint-operator × is defined in Definition 3.2.3. The constructed iterative HModel is sound and will produce in each iteration a valid HPlan with shorter makespan than the HPlan of the previous round. Lemma 3.3.6.1 The construction of the iterative HModel is sound. Proof: In each iteration a valid HPlan must have a shorter makespan than the HPlan of the previous round. The goal constraint in the i-th iteration is ψ i which imposes the constraint that the runningTime of φ ′ i should be less than the makespan of φ ′ i−1 (runningTime < makespan(φ ′ i−1 )). Thus, it ensures that φ ′ i has a shorter makespan than φ ′ i−1 .2 In Table 3.1, we present the results in which makespan is the duration of the plan, constr is the upper-bound on the constraint on runningTime for each iteration, whereas dwell-diff and diff-cost τ are the components of contrastive explanation. Roundconstrmakespandwell-diffdiff-cost τ 1≤ 5032− 2< 3231.5moving, 0.5−0.5 3< 3118moving, 13.5−13.5 4< 1817.5moving, 0.5−0.5 5< 1714moving, 3.5−3.5 6< 1413.5moving, 0.5−0.5 7< 1312moving, 1.5−1.5 8< 1211.75moving, 0.25−0.25 9< 1110.96moving, 0.79−0.79 10< 10.9610.95moving, 0.01−0.01 11< 10.95∞NA∞ Table 3.1: The makespan of the plan for each re-plan iteration We can see in the table that for runningTime ≤ 10.95, the planner is unable to find a valid plan for the planning instance. Therefore, via the contrastive questions, a user may improve the quality of the plan in terms of the makespan metric. The contrastive explanation is: • Remove (φ,φ ′ ): No action is removed from the original plan; 66 3.3. CONTRASTIVE QUESTIONS • Add (φ,φ ′ ): No new action is added in each iteration; • Common (φ,φ ′ ): accelerate, decelerate and stop; • The dwell-diff and diff-cost τ of HModel Π ′ for each iteration are given in Table 3.1 • The diff-cost len for each iteration is 0 as all alternate plans are of the same length except for the no-plan where the length is denoted by ∞. A user of the framework can draw the following conclusion from the contrastive explana- tion: Conclusion: The makespan of the original plan is not optimal, and there can be alternate plans with lesser makespan. 3.3.7 Question on the Sequence of Actions in the Plan A certain sequence of actions in a plan may seem costly to a user, or the user might have a preference in mind for a sequence of actions, and the absence of that sequence in a plan may give rise to the question of: Why this sequence of actions, why not any other instead? An action sequence is defined in Definition 3.1.7. To address such user questions on a sequence of actions appearing in a plan, the following contrastive question can be asked: Why does the action sequence a i − a i+1 − ...− a i+k appear in the plan ⟨a 1 − a 2 −...−a i−1 −a i −a i+1 −...−a i+k −a i+k+1 −...−a m ⟩, why not some other sequence? In the plan in Listing 3, a question can be asked: ‘Why does the action sequence decelerate-decelerate appear in the plan⟨accelerate- decelerate-decelerate-stop⟩, why not some other sequence?’ To explain this question, we intend to design a hypothetical model such that any hypo- thetical plan in the model preserves the action sequence that appears before and after the sequence in the question in the original plan. We refer to these as pre-sequence and post-sequence respectively. For example, in the question above, the pre-sequence a 1 − a 2 − ...− a i−1 and the post-sequence a i+k+1 − a i+k+2 − ...− a m are needed to be preserved in the hypothetical plan. In addition, we need to ensure that the action sequence in the question a i − a i+1 − ...− a i+k does not appear in the hypothetical plan. In the following section, we first discuss a general approach to sequence modification and then come back to our original question in the car domain. Note that our proposed solution introduces primed actions where necessary. For example, a is an action in the domain, a ′ is a primed action newly introduced into the HModel. These primed actions have the same interpretation as their unprimed counterparts, except that they may have different preconditions and effects. 67 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS Preserving pre-sequence: To preserve the pre-sequence in the plan, we treat each action that appears in the pre-sequence as an independent action that can only appear in the pre- sequence of a plan. We modify each action that appears in the sequence a 1 −a 2 −...−a i−1 in the following way: For each action a j ,∀j ∈1, 2,...,i−1, a new action a ′ j is constructed while also retaining the a j in the HModel. The action a ′ j will appear only in the pre-sequence while the action a j may appear later in a plan. For each action a ′ j (j ′ ∈ 1, 2,...,i− 1), new predicate symbols has_done_a ′ j are used to preserve their order of appearance in the plan. A predicate symbol has_done_preseq is introduced, which represents that the pre-sequence has been applied to the plan. The action a 1 is extended to a ′ 1 with an added precondition ¬(has_done_a ′ 1 ) and an added effect has_done_a ′ 1 , i.e. Pre(a ′ 1 ) = Pre(a 1 )∧¬(has_done_a ′ 1 ) Eff + (a ′ 1 ) = Eff(a 1 )∪(has_done_a ′ 1 ) For each action a j (j ∈ 2, 3,...,i− 1), a j is extended to a ′ j with added precondition has_done_a ′ j−1 and ¬(has_done_a ′ j ) and an added effect has_done_a ′ j . This ensures that the actions of the pre-sequence appear in order. Pre(a ′ j ) = Pre(a j )∧ (has_done_a ′ j−1 )∧¬(has_done_a ′ j ) Eff + (a ′ j ) = Eff(a j )∪(has_done_a ′ j ) Additionally, the action a ′ i−1 has an added effect has_done_preseq. Eff + (a ′ i−1 ) = Eff(a i−1 )∪(has_done_a ′ i−1 ), (has_done_preseq) All other domain actions are modified with an added precondition has_done_preseq such that they may appear in a plan only after the pre-sequence. has_done_preseq is also used as a goal constraint to ensure the pre-sequence is preserved in a plan. Restricting the sequence in the question: To restrict the action sequence a i −a i+1 − ...− a i+k in a plan, we modify each action that appears in the sequence in the following way: We introduce a new function symbol c that represents a counter indicating how much of the forbidden sequence has been seen so far. It is set to -1 initially. For the first action a i in the sequence, we create two actions a ′ i and a ′ i . a ′ i is extended with added preconditions has_done_preseq and c = −1 indicating that it can appear only at the beginning of a sequence after the pre-sequence has been applied in the plan and an added effect to assign c to the value 1 to indicate the first action of the forbidden sequence has appeared in the plan. Pre(a ′ i ) = Pre(a i )∧ (has_done_preseq)∧ (c ==−1) Eff + (a ′ i ) = Eff(a i )∪(c = 1) a ′ i can appear at any other place in the sequence when appearing out of order. a ′ i is extended with added preconditions has_done_preseq and c̸=−1 to indicate that it can 68 3.3. CONTRASTIVE QUESTIONS appear after the pre-sequence has been applied in the plan and an added effect to assign c to the value 0 to indicate that it is appearing out of order and hence the forbidden sequence is broken. Pre(a ′ i ) = Pre(a i )∧ (has_done_preseq)∧ (c̸=−1) Eff + (a ′ i ) = Eff(a i )∪(c = 0) For ∀j in 1, 2,...,k− 1, each a i+j is flattened to a ′ i+j and a ′ i+j . a ′ i+j has been extended with added preconditions has_done_preseq and (c = j) and an added effect (c = j + 1). This is how the consecutive appearance of the actions in the forbidden sequence are remembered. a ′ i+j has been extended with added preconditions has_done_preseq and (c̸= j), and an added effect (c = 0) which resets the counter. The last action in the sequence a i+k is barred from appearing in a plan when all previous actions of the forbidden sequence have appeared in order. a i+k is modified to a ′ i+k with added preconditions has_done_preseq and (c ̸= k) to ensure that it can appear only out of order, and an added effect (c = 0) to indicate that the forbidden sequence has not appeared in the plan. The modifications are shown below: • Pre(a ′ i+j ) = Pre(a i+j )∧has_done_preseq∧(c == j), • Eff + (a ′ i+j ) = Eff (a i+j )∪(c = j + 1), • Pre(a ′ i+j ) = Pre(a i+j )∧ (c̸= j)∧ (has_done_preseq), • Eff + (a ′ i+j ) = Eff (a i+j )∪(c = 0), • Pre(a ′ i+k ) = Pre(a i+k )∧ (c̸= k)∧ (has_done_preseq), • Eff + (a ′ i+k ) = Eff (a i+j )∪(c = 0). The rest of the actions in the domain are modified to include an added effect (c = 0) to indicate that the forbidden sequence has not appeared in the plan. Preserving post-sequence: Similar to the pre-sequence, the post-sequence also needs to be preserved in a plan, and we treat each action that appears in the post-sequence as an independent action that can appear only at the post-sequence in a plan. We modify each action that appears in the sequence a i+k+1 − a i+k+2 − ...− a m as given below: For each action a j , ∀j ∈ i + k + 1,i + k + 2,...,m, a new action a ′ j is constructed while also retaining the a j in the HModel. The action a ′ j will appear only in the post- sequence while the action a j may appear elsewhere in a plan. A new predicate symbol start_postsequence is introduced, which is used to mark the beginning of the post-sequence in a plan. For each action a ′ j (j ′ ∈i + k + 1,i + k + 2,...,m), a new predicate symbol has_done_a ′ j is introduced to ensure their order of appearance in the plan. Finally, an- other predicate symbol has_done_postsequence is introduced, which represents that the post-sequence has been applied to the plan. The first action a i+k+1 in the post-sequence is extended to a ′ i+k+1 with added preconditions has_done_preseq, ¬(has_done_a ′ i+k+1 ) 69 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS and¬(start_postsequence), and with added effects has_done_a ′ i+k+1 and start_postsequence. has_done_preseq ensures that it can appear only after the pre-sequence, while start_postsequence marks the beginning of the post-sequence. The duality of the precondition¬(has_done_a ′ i+k+1 ) and the effect has_done_a ′ i+k+1 prevents it from further repetitions in a plan. Pre (a ′ i+k+1 ) =Pre (a i+k+1 )∧ (has_done_preseq)∧¬(start_postsequence)∧ ¬(has_done_i ′ i+k+1 ) Eff + (a ′ i+k+1 ) = Eff (a i+k+1 )∪(has_done_a ′ i+k+1 ), (start_postsequence) For all j in i + k + 2,i + k + 3,...,m, a j is extended to a ′ j with added preconditions (has_done_a ′ j−1 ) and ¬(has_done_a ′ j ) and an added effect (has_done_a ′ j ), i.e. Pre (a ′ j ) = Pre (a j )∧ (has_done_a ′ j−1 )∧¬(has_done_a ′ j ) Eff + (a ′ j ) = Eff (a j )∪(has_done_a ′ j ) The action a m is extended to a ′ m with an added effect has_done_postsequence, i.e. Eff + (a ′ m ) = Eff (a m )∪(has_done_a ′ m ), (has_done_postsequence) To enforce that the post-sequence is preserved in a plan, the constraint (has_done_postsequence) is also used as a goal condition. All other actions in the domain are extended with added precondition¬(start_postsequence) so that they cannot be applied after the post-sequence begins in a plan. The HModel Π ′ is: Π ′ = (⟨Fs ′ ,Rs ′ ,As ′ ,Es,Ps,arity ′ ⟩,⟨Os,I ′ ,G ′ ⟩) where • Fs ′ = Fs∪c • Rs ′ = Rs ∪ has_done_preseq, has_done_postsequence, start_postsequence, has_done_a ′ j , ∀j ′ ∈ 1, 2,...,i ∪ i + k + 1,i + k + 2,...,m • As ′ = a ′ 1 ,a ′ 2 ,...,a ′ i ,a ′ i+1 ,a ′ i+1 ,...,a ′ i+k−1 ,a ′ i+k−1 ,a ′ i+k ,a ′ i+k+1 ,...,a ′ m ∪ As \ a i ,a i+1 ,...,a i+k , (here, \ represents the set difference operation) • arity ′ (As ′ ) = arity(As) • I ′ = I ∪ (c =−1) • G ′ : G ∧ ground(has_done_preseq) ∧ ground(has_done_postsequence) The constructed HModel of restricting an action sequence in a plan is sound and will produce an HPlan as expected by the user. Lemma 3.3.7.1 The construction of the HModel is sound. 70 3.3. CONTRASTIVE QUESTIONS Proof: A valid HPlan must preserve the pre-sequence and the post-sequence while there can be an arbitrary action sequence in between them in a plan. The pre-sequence and the post-sequence are preserved in a valid HPlan through the use of the goal constraints has_done_preseq and has_done_postsequence. A sequence of actions other than the forbidden sequence may appear in between the pre-sequence and the post-sequence. In the pre-sequence, each action is treated uniquely, and necessary preconditions and effects are added to bar their repeated occurrences in the pre- and post-sequences. While the primed actions will appear only in the pre-sequence, their unprimed counterparts may appear later in the plan. The forbidden sequence without a prefix or suffix cannot appear in a valid plan, as we remember the actions in their order of occurrence. If a sequence matches all the actions of the forbidden sequence except the last one, we forbid the last action of the forbidden sequence to appear in the plan. The primed actions of that section appear in the plan when they appear in the same order as the forbidden sequence and hence increase the counter, whereas the double-primed counterparts appear when they break the sequence. Whenever an action appears out of order, it erases the counter to indicate that the forbidden sequence is broken. Similar to the pre-sequence, each action in post- sequence is treated uniquely. The primed actions will appear only in the post-sequence, whereas their unprimed counterparts may appear elsewhere in the plan.2 Example 3.3.7.1 Considering the user question on action sequence in our car domain shown above, we have the following constructs: • pre-sequence: accelerate • forbidden-sequence: decelerate-decelerate • post-sequence: stop To preserve the pre-sequence in a plan following the above discussion, we introduce a new action accelerate’ which extends the accelerate action while also retaining it in the HModel. A new predicate symbol has_done_preseq, which represents the pre-sequence, is added in the planning instance. As our pre-sequence consists of only one grounded accelerate action, we do not need any additional predicate symbol to maintain the order. The accelerate’ action is extended with an added precondition ¬(has_done_preseq) and an added effect (has_done_preseq). Pre (accelerate ′ ) = Pre (accelerate)∧¬(has_done_preseq) Eff + (accelerate ′ ) = Eff (accelerate)∪(has_done_preseq) The constraint (has_done_preseq) is also added as a goal condition which enforces the pre-sequence into a plan. All other domain actions include a precondition (has_done_preseq) such that they can appear in a plan only after the pre-sequence. To restrict the appearance of the forbidden-sequence decelerate-decelerate in a plan, we modify the actions of the do- main. The first action in the sequence decelerate is modified to decelerate’, which can only appear at the start of a sequence that succeeds the pre-sequence. We introduce a new func- tion symbol c used as a counter variable, which is initially set to -1. The decelerate’ action 71 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS has added preconditions (has_done_preseq) and (c =−1) where (has_done_preseq) rep- resents it can appear in a plan only after the pre-sequence and a negative value of c suggests the forbidden sequence has not started yet, respectively, and an added effect which assign c to the value 1. We further modify the decelerate action to decelerate” which includes pre- conditions (has_done_preseq), (c̸= 1), and (c̸=−1), to indicate that it appears after the pre-sequence and not preceding a decelerate’ action respectively, and an added effect c = 0 indicates that the forbidden sequence is broken. The rest of the actions in the domain, ac- celerate and stop, are modified to include an added precondition (has_done_preseq) such that they may appear only after the pre-sequence and an added effect (c = 0) to indicate the forbidden sequence has not appeared in the plan. The modifications are shown below: • Pre (decelerate ′ ) = Pre (decelerate)∧ (has_done_preseq)∧ (c ==−1), • Eff + (decelerate ′ ) = Eff (decelerate)∪(c = 1), • Pre (decelerate ′ ) = Pre (decelerate)∧ (has_done_preseq)∧ (c̸= 1)∧ (c̸=−1), • Eff + (decelerate ′ ) = Eff (decelerate)∪(c = 0), • Pre (accelerate) = Pre (accelerate)∧ (has_done_preseq), • Pre (stop) = Pre (stop)∧ (has_done_preseq), • Eff + (accelerate) = Eff (accelerate)∪(c = 0), • Eff + (stop) = Eff (stop)∪(c = 0). To preserve the post-sequence in a plan, a new predicate symbol has_done_postsequence is introduced, which represents the post-sequence that has been applied to the plan. As the post-sequence consists of only one grounded stop action, we do not need any additional predicate symbol to maintain the order. The original stop action in the domain is extended to stop’. The added preconditions (has_done_preseq) and ¬(has_done_postsequence) ensure that the action stop’ can be applied in a plan only after the pre-sequence and it does not repeat itself, respectively. The added effect (has_done_postsequence) ensures that the post-sequence always appears in a plan, i.e. Pre (stop ′ ) = Pre (stop)∧ (has_done_preseq)∧¬(has_done_postsequence) Eff + (stop ′ ) = Eff (stop)∪(has_done_postsequence) The constraint (has_done_postsequence) is used as a goal condition to enforce the ap- pearance of the post-sequence in a plan. All other actions in the domain are extended with an additional precondition ¬(has_done_postsequence) to prevent them from appearing after the postsequence in a plan. The modifications in the domain are shown below: • Fs’ = Fs ∪ c, • Rs’ = Rs ∪has_done_preseq,has_done_postsequence • As’ = accelerate ′ ,decelerate ′ ,decelerate ′ ,stop ′ ∪ As \decelerate, 72 3.3. CONTRASTIVE QUESTIONS • arity’(As’) = arity(As), • I’ = I ∪ PNE (c =−1), • G’ = G ∧(has_done_preseq)∧ (has_done_postsequence). The modifications in the domain are shown below: ( : p r e d i c a t e s ( running ) ( engineBlown ) ( goalReached ) ( has_done_presequence ) ( has_done_postsequence ) ( : f unc ti ons ( d) ( v ) ( a) ( upLimit ) ( downLimit ) ( runningTime ) ( c )) . . . ( : action a c c e l e r a t e : parameters () : precondition ( and ( running ) (< ( a) ( upLimit )) ( has_done_presequence ) ( not ( has_done_postsequence )) ) : e f f e c t ( and ( increase (a ) 1) ( assign ( c ) 0))) ( : action accelerate ’ : parameters () : precondition ( and ( running ) (< ( a) ( upLimit )) ( not ( has_done_presequence )) ) : e f f e c t ( and ( increase (a ) 1) ( has_done_presequence )) ) ( : action decelerate ’ : parameters () : precondition ( and ( running ) (> ( a) ( downLimit )) ( has_done_presequence ) (= ( c )−1) ( not ( has_done_postsequence ) )) : e f f e c t ( and ( decrease (a ) 1) ( assign ( c ) 1) )) ( : action decelerate ’ ’ : parameters () : precondition ( and ( running ) (> ( a) ( downLimit )) ( not (= ( c ) 1)) ( not (= ( c )−1)) ( has_done_presequence ) ( not ( has_done_postsequence ))) : e f f e c t ( and ( decrease (a ) 1) ( assign ( c ) 0))) ( : action stop : parameters () : precondition ( and (= ( v ) 0) (>= (d ) 30) ( not ( engineBlown )) ( has_done_presequence ) ( not ( has_done_postsequence ))) : e f f e c t ( and ( goalReached ) ( assign ( c ) 0) )) ( : action stop ’ : parameters () : precondition ( and (= ( v ) 0) (>= (d ) 30) ( not ( engineBlown ) ( has_done_presequence ) ( not ( has_done_postsequence ))) : e f f e c t ( and ( goalReached ) ( has_done_postsequence )) ) The initial and goal states are modified as shown below: ( : i n i t ( running ) (= ( runningTime ) 0) (= ( upLimit ) 1) (= ( downLimit )−1) (= d 0) (= a 0) (= v 0) (= c−1)) ( : goal ( and ( goalReached ) ( not ( engineBlown )) (<= ( runningTime ) 50) ( has_done_presequence ) ( has_done_postsequence )) ) The modifications made are shown in the corresponding hybrid automata representation of the HModel in Figure 3.9: The resulting HPlan given by SMTPlan+ is: Time ActionDuration 73 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS running goalReached engineBlown accelerate a<upLimit has_done_preseq==true has_done_postseq==false a=a+1 c=0 initial a=0 v=0 d=0 t=0 c=-1 stop (v==0) & (d>=30) has_done_preseq==true has_done_postseq==false c=0 engineExplode (a>=1) & (v>=100) decelerate' a>downLimit has_done_preseq==true has_done_postseq==false c==-1 a=a-1 c=1 Inv: t<=50, (has_done_preseq) (has_done_postseq) accelerate' has_done_preseq==false a<upLimit a=a+1 has_done_preseq=true decelerate′ a>downLimit has_done_preseq==true has_done_postseq==false a=a-1 c=0 stop' (v==0) & (d>=30) has_done_preseq==true has_done_postseq==false has_done_postseq=true Figure 3.9: The hybrid automaton for the corresponding HModel of the question addressed in sect. 4.7. 0 . 0 :( accelerate ’ )[ 0 . 0 ] 1 . 0 :( decelerate ’ )[ 0 . 0 ] 3.0( a c c e l e r a t e )[ 0 . 0 ] 5.0( decelerate ’ ’ )[ 0 . 0 ] 13 .0 :( decelerate ’ ’ )[ 0 . 0 ] 16 .0 :( stop ’ )[ 0 . 0 ] makespan: = 16.0 units. Recall that primed actions have the same interpretation as their unprimed counterparts, except that they may have different preconditions and effects. The contrastive explanation is: • Remove (φ,φ ′ ): stop and decelerate are removed in the alternate plan; • Add (φ,φ ′ ): accelerate’, decelerate’, decelerate” and stop’ are added in the alternate plan; • Common (φ,φ ′ ): accelerate; • dwell-diff : moving, 16; • diff-cost τ : −16; • diff-cost len : 2. A user of the framework can draw the following conclusion from the contrastive explana- tion: Conclusion: For the given problem instance, there exist plans that consist of action sequences other than the sequence decelerate-decelerate, however, the plan length is greater than that of the original one. 74 3.3. CONTRASTIVE QUESTIONS 3.3.8 Questioning the Optimality of Plan Length Similar to the question in Section 3.3.6, a user might be interested in a plan with a shorter plan length. For an explanation of the plan length, the following contrastive question can be asked, Why is the length of the plan not less? Similar to addressing the question in Section 3.3.6, an iterative question ‘Why is the length of the plan k and not less?’ where k is the length of the valid plan successfully generated by the planner in the last iteration, can help to address the optimality of the planner in terms of the plan length. In our car domain, a user might ask: ‘Why is the length of the plan not less than 4?’ To answer such a contrastive question, the planning instance is compiled to introduce a new function symbol s set to 0 initially. Each action n ∈ As is modified to have a new added effect that increases the value of s by 1 on each occurrence of the action in a plan. Eff + (n) ′ = Eff(n)∪PNE(s = s + 1),s∈ Fs, n∈ As where Eff + (n) ′ indicates the modified post-condition of the action n. A constraint PNE(s < k) is added as a goal condition to check whether a plan of length less than k exists. It- eratively lowering the value of k using a model similar to Equation (3.1) can provide an optimal plan in terms of plan length. The new planning instance is: Π ′ = (⟨Fs ′ ,Rs,As,Es,Ps,arity ′ ⟩,⟨Os,I ′ ,G ′ ⟩) where • Fs ′ = Fs ∪s • arity ′ (s) = 0 • I ′ = I ∪ PNE (s = 0) • G ′ = G ∪ constraint(PNE (s < k)) The constructed iterative HModel is sound and will produce an HPlan having a plan-length shorter than the HPlan of the previous round, if there exists any. Lemma 3.3.8.1 The construction of the iterative HModel is sound. Proof: In each iteration a valid HPlan must be shorter in plan-length than the HPlan of the previous round. The function symbol s keeps track of the applied action in a plan. The goal constraint (s < k) imposes that the plan-length of HPlan φ ′ i in the i-th iteration should be less than k, where k is the plan-length of the HPlan of the previous round. Thus, it ensures that φ ′ i has a shorter plan-length than φ ′ i−1 .2 75 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS Example 3.3.8.1 Consider the user question above, where we look for a plan with a plan- length less than 4. For that, a new function symbol s is introduced and initialized to 0. Each action in the domain is extended with an added effect to increase the value of s by one on its appearance in a plan. The goal condition is additionally extended with the constraint (s < k), where k is the plan-length of the HPlan obtained in the previous iteration. For example, in the first iteration, the HModel Π ′ 1 contains the goal constraint (s < 4). The corresponding modification in the hybrid automaton representation of the domain is shown in Figure 3.10. running goalReached engineBlown accelerate a<upLimit a=a+1 s=s+1 initial a=0 v=0 d=0 t=0 s=0 stop (v==0) & (d>=30) s=s+1 engineExplode (a>=1) & (v>=100) decelerate a>downLimit a=a-1 s=s+1 Inv: t<=50, s<k; Figure 3.10: The hybrid automaton for the corresponding HModel for question 8. Restricting the plan length to be less than 4 results in no plan being generated by the planner. The contrastive explanation is: • Remove (φ,φ ′ ): Not available, as there is no alternate plan; • Add (φ,φ ′ ): Not available, as there is no alternate plan; • Common (φ,φ ′ ): Not available, as there is no alternate plan; • dwell-diff : Not available; • diff-cost τ : ∞; • diff-cost len : ∞. A user of the framework can draw the following conclusion from the contrastive explana- tion: Conclusion: For the problem instance, no valid plan can be generated that has a plan length less than 4. Summary Table: A summary of the contrastive explanations of the above-discussed user questions is provided in Table 3.2. In the table, Q. No. marks the section number and represents the particular contrastive question that we discussed in that section. Rem, Add, and Com are the number of actions that are removed from the original plan, the number of new actions that appear in the alternate plan HPlan, and the number of actions that 76 3.4. PROVING THE ABSENCE OF A PLAN appear in both the original and the alternate plans, respectively. Dwell-diff exhibits the dwell time differences of the processes in the original and the HPlan, where a positive value indicates that the original plan has a longer dwell time in the respective location. DC τ and DC l show the differences of makespan and plan-length of the original and the HPlan. A positive value of DC τ and DC l indicates the HPlan has a longer makespan and a greater plan-length than the original. Remark draws the conclusion on the quality of the generated HPlan and the experimental observations. Q. No.RemAddComDwell-diffDC τ DC l Remark 3.3.1023moving, 20-202 HPlan has a shorter makespan but a longer plan in terms of applied actions. The original plan is better. 3.3.2112moving, 14-140 The original plan is not optimal with respect to the duration of the plan. The HPlan is better. 3.3.3012moving, -9.59.50 The decelerate action being applied later than 1-time unit, resulted in a longer plan. The original plan is better. 3.3.4NANANANA∞ Barring the decelerate action from appearing in the plan leads to no valid plan in this domain. 3.3.5NANANANA∞ No valid plan can be generated that has less than two executions of the action decelerate. 3.3.6003see Table 3.1see Table 3.10 The makespan of the original plan is not optimal and there can be alternate plans with shorter makespan. 3.3.7241moving, 16-162 The HPlan consisting of action sequence other than decelerate-decelerate has greater plan-length than the original one. 3.3.8NANANANA∞ No valid plan can be generated that has a plan length less than 4. Table 3.2: A brief summary of the contrastive explanations generated for the user questions discussed above. 3.4 Proving the Absence of a Plan The planning problem is undecidable for hybrid systems in general (Alur et al. [1995a]). When a planner fails to generate a valid plan for a planning problem, it cannot be asserted whether it is due to the underlying undecidability or that the planning problem admits no valid plan. This leads to a limitation in our explanation framework since incorrect explanations may result in such cases. To address this issue, we attempt to prove the non- existence of a plan by reachability analysis in the hybrid domain of the planning problem for which we get no-plan (we refer to them as no-plan-models in the text that follows). Reachability analysis checks whether the system can reach a target state from an initial state under the dynamics of the system. Observe that reachability analysis can be useful in proving the non-existence of a plan because unreachability of the goal-state from the initial state under the dynamics of the planning domain proves and also explains, in some sense, why a sound planner does not return a valid plan for a given planning problem instance. Note that reachability analysis is undecidable for hybrid systems in general. However, allowing δ approximation in the analysis may often lend to decidability as shown in (Gao et al. [2012b]). We leverage this observation in the explanation of no-plan being returned by a planner by proving the absence of a valid plan. A brief discussion on the notion of δ 77 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS approximation in reachability analysis follows in the text. The key idea of δ-approximate reachability analysis is to create a bounded abstraction of the system and reduce it to the satisfiability problem of a first-order logic formula over reals with Type-2 computable functions (Gao et al. [2012a,b]). The presence of bounded quantifiers and the infusion of a δ-relaxed notion of satisfiability in the resulting formula ensures decidability for a positive rational number δ. We leverage this δ-approximate bounded reachability analysis feature of dReach (Gao et al. [2014]) on our no-plan-models to explain the non-existence of plans in the no-plan instances that we encounter with SMTPlan+. In our reachability analysis of the no-plan-models, we have set a bound of k-depth (we refer to as plan-depth) as it corresponds to checking the existence of any plan upto k-length. In the following sections, we first describe how the δ-approximation is applied and its significance in reachability analysis. We then discuss the explanations for the no-plan-models and also provide an algorithm for the same. Remark: Note that with δ-approximate bounded reachability analysis, we can only provide a proof that the problem instance does not admit a valid plan. Further insights on why there is no valid plan is not addressed by this method. There are notable works which provide reasons for unsolvability of planning problem instances for discrete domains such as (Eriksson and Helmert [2020], Eriksson et al. [2017]). For hybrid domains, we are not aware of any such work, and we plan to address this as future work. For certain PDDL+ problem instances, planners like ENHSP (Scala et al. [2016]) can deduce unsolvability and report it. In problem instances beyond the scope of ENHSP, the proposed reachability analysis based method can be used with dReach, which can analyze a large class of non-linear systems. 3.4.1δ-Approximation The δ-approximation of a system can be best explained in terms of showing the δ- weakening (Gao et al. [2014]) of the corresponding hybrid automata. Let δ ∈ Q + ∪0 be an arbitrary rational number and H= Loc, V ar, Flow, Init, Lab, Edge, Inv represent a hybrid automaton. The δ-weakening of H can be obtained by weakening the flow and invariants in each location, initial valuation of variables, guards, and assignments of each transition of the automaton. We represent a δ-weakened hybrid system as H δ : H δ = Loc,V ar,Flow δ ,Init δ ,Lab,Edge δ ,Inv δ It may be noted from the syntactic construction described above that H δ represents an over-approximation of H. Thus, the set of runs in H δ subsumes the set of runs in H. Let H(runs) denote the set of all runs in H. H(runs) ⊆ H δ (runs). Consider the no-plan- model of Section 3.3.4, for which the corresponding hybrid automaton is shown in Figure 3.7. δ-weakening of the model in Section 3.3.4 corresponds to the weakening of flows and invariants in each location, guards and assignments of each transition by the value δ. In bounded reachability analysis, for a given δ ∈ Q + and a bounded depth k, if dReach concludes that a given problem instance is unsatisfiable, then it implies the following: 78 3.5. RESULTS 1. The problem instance is also unsatisfiable for all values ≤ δ for a given bound k. 2. The problem instance is also unsatisfiable for all the depths < k for a given δ. Next, it can be shown that if there is no plan in H δ for any given problem instance, there is no plan in H as well. Lemma 3.4.1.1 For a given problem instance Π, the absence of any plan in H δ implies the absence of any plan in H.2 Proof: Since H(runs)⊆ H δ (runs) and a plan φ is an instantiation of a run r ∈ H(runs), it is self-evident that for the given problem instance Π, a precision δ, and plan-depth k if there exists no plan in H δ there exists no plan in H.2 3.5 Results In this section, we present the performance statistics of our contrastive explanation frame- work. We have chosen three benchmark PDDL+ domains: the Car domain (KCL-Planning [Last accessed on Nov 2025]), the Generator-events domain (KCL-Planning [Last accessed on Nov 2025]) and the Planetary-lander domain (Fox and Long [2006]) to present a com- parative look on the scalability of the contrastive explanation framework on these planning problems over these domains. Additionally, we evaluate the planning problem instances over some of the state-of-the-art planners, which further shows the applicability of our framework across planners. For a problem instance, we use different planners to generate plans. Contrastive questions are formed based on those plans, and corresponding HModels are constructed for each contrastive question. These HModels are used with the planners to generate the HPlans. Below, we provide a brief description of the planning domains. 3.5.1 Benchmarks Car Domain (CD): The planning problem consists of a car (initially at rest) that has to cover a given distance within a specified time bound. Further, it is needed to ensure that the car has a zero velocity at the end as well. The domain comprises of six function symbols: d, v, a, upLimit, downLimit and runningTime, three predicate symbols: run- ning, engineBlown and goalReached, three instantaneous actions accelerate, decelerate and stop, one event engineExplode and one process moving. The function symbols d, v, and a respectively represent the distance, velocity, and acceleration of the car, whereas upLimit and downLimit specify the upper and lower bounds on the acceleration. runningTime en- codes the elapsed time of the car. The predicates running, engineBlown, and goalReached define the states of the car. The actions accelerate and decelerate respectively increase and decrease the acceleration of the car by one unit. The stop action is applicable when the car covers the specified distance and has a velocity of zero. The event engineExplode is triggered when the velocity and acceleration of the car reach a certain threshold and the car is in the engineBlown state. The process moving is activated when the car is in the running state specified by a predicate running. This process updates the distance and the velocity of the car as a function of time. Initially, the car is in the running state with a zero 79 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS velocity. The goal is to traverse a distance of 30 units within 50 units of time. The planning domain represented in PDDL+ is shown in Listing 1. The initial condition init specifies that the predicate running is initially true, the initial values of the functions a, v, d, and runningTime are assigned 0. Across the problem instances, we vary the value of upLimit and downLimit, which specify the bounds on the acceleration of the car. For problem instance 1 (see Listing 2), upLimit and downLimit are assigned 1 and -1, respectively, whereas for problem instance 2 (see Appendix 8.1) these values are set to 2 and -2, respectively. The goal for the car is to travel a minimum distance of 30 units in less than or equal to 50 units of time while avoiding an engine explosion, which is caused when the velocity is greater than or equal to 100 units and the acceleration is greater than or equal to upLimit. This is specified as the goal condition goal with the predicates goalReached and the negation of engineBlown, together with the numeric condition runningTime≤ 50. Generator-Events Domain (GD): The generator-events domain consists of a generator that has to run for a specified amount of time. The domain comprises of four function symbols: fuelLevel, capacity, fuelInTank and ptime, four predicate symbols: generator-ran, available, using and safe, one durative-action generate which operates for a fixed duration of 1000 time units, one instantaneous action refuel, one process refuelling and two events: tankEmpty and generatorOverflow. We need to instantiate objects of types generator and tank, which will be set as parameters to different domain symbols while respecting their arities. The fuelLevel and the capacity are associated with the generator and indicate the current fuel level and fuel storage capacity of the generator, whereas the fuelInTank and the ptime are associated with the tanks, indicating fuel reserve in a tank and fuel pouring time from a tank while in use, respectively. The generator-ran indicates whether the generator ran successfully for a specified amount of time (1000 time units), available indicates tanks that are in store for use, using indicates tanks that are being used by the generator, and safe indicates if the generator is operating safely. The durative-action generate has precondition fuelLevel ≥ 0 and safe to indicate that the current fuel level in the generator needs to be greater than 0 and the generator needs to be in the safe mode. It runs for a fixed duration of time (1000 units) and has an effect to decrease the fuelLevel at a constant rate. At the end, this makes generator-ran to be true to reflect that the gen- erator ran successfully for the specified duration. The instantaneous-action refuel enables the generator to use an available tank and activates the process refuelling, which in turn increases the fuelLevel at a rate proportional to the pouring time ptime while decreasing the fuelInTank at the same rate. The event emptyTank triggers to indicate that a tank is unavailable for use when fuelInTank becomes 0 for it, whereas the event generatorOverflow drives the generator to an unsafe-mode when its fuelLevel exceeds the capacity threshold. For creating different problem instances, we vary the number of fuel reserve tanks and the initial fuelLevel of the generator. For problem instance 1 (see Appendix 8.2), the initial state specifies that the current fuelLevel of the generator gen is 940 units, whereas the capacity to hold fuel in the generator is 1600 units. The tanks available to use are t1 and t2, and the fuel reserve in each tank is 40 units. In problem instance 2 (see Appendix 8.2), the initial fuelLevel of the generator gen is 900 units, and the tanks available to use 80 3.5. RESULTS are t1, t2, and t3, and the fuel reserve in each tank is 40 units. Initially, the generator is in the safe mode. The predicate and function symbols that are not initialized explicitly in the initial state are initialized with a default value of false and 0, respectively. In both cases, the goal condition comprises of a single constraint (generator-ran) which needs to be true in the goal state, indicating that the generator ran successfully for a specified amount of time (1000 time units). The PDDL+ representation of the domain is shown in Appendix 8.2. Additionally, a hybrid automaton model for the generator-events domain for problem instance 1 is shown for visual representation in Appendix 8.2. Planetary-lander domain (PD): The planetary-lander domain is based on a simplified model of a solar-powered lander. The domain has 21 function symbols: demand, supply, soc, charge_rate, daytime, heater_rate, dusk, dawn, fullprepare_durtime, prepareobs1_durtime, prepareobs2_durtime, observe1_durtime, observe2_durtime, obs1_rate, obs2_rate, A_rate, B_rate, C_rate, D_rate, safeLevel and solar_const, which control different parameters of the domain. It has 7 predicate symbols: day, commsOpen, readyForObs1, readyForObs2, gotObs1, gotObs2 and available. The domain has five durative-actions: fullPrepare, pre- pareObs1, prepareObs2, observe1 and observe2. Each action performs a specific task and draws a fixed power throughout the operation. observe1 and observe2 are two observe actions, which observe two different phenomena. However, prior to performing any obser- vation task, the system must prepare the observers either by using a single long action, called fullPrepare, or by using two shorter actions, called prepareObs1 and prepareObs2, which are specific to one of the observation actions. The shorter actions cumulatively have higher power requirements for their execution than the single preparation action. The lander is required to execute both observation actions before a communication link is established, which sets a deadline on the activities. These activities are all carried out against a backdrop of fluctuating power supply. The lander is equipped with a regen- erative solar power resource. The domain contains four processes: generating, charging, discharging and night_operations, and two events: nightfall and daybreak. The generating process generates solar power for the lander, which is governed by the position of the sun. At night, there is no power generated, which rises smoothly to a peak at midday, falling back to zero at dusk. The nightfall event stops the generating process, and the lander enters the night_operations mode. This process draws a constant power require- ment for a heater used to protect its instruments, in addition to any requirements for instruments. The daybreak event stops the night_operations process, and the generating process restarts. Both these events are triggered by a simple clock that is driven by the twin processes of generating and night_operations and reset by the events. The lander is equipped with a battery, allowing it to store excess energy as charge through the process charging while excess demand must be supplied from the battery through the process discharging. The planning problem comprises an initial state, goal conditions, and object instantiations. An object of type equipment is instantiated. The initial state specifies the initial values of the different parameters of the domain. The goal of the planning problem is to observe the two phenomena of the environment, which are specified by two predicate symbols gotObs1 and gotObs2. These predicates are initially set to false, which indicates 81 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS that those phenomena have not been observed yet. The actions observe1 and observe2 set them to true after observing those phenomena. In this case, to create the different instances, we vary the initial state of charge soc and the safe operating charge level safe- Level for the lander. For problem instance 1, soc is set to 93 units and safeLevel is set to 15 units, whereas in problem instance 2, soc is set to 50 units and safeLevel is set to 25 units. The PDDL+ representation of the domain and the planning problem instances 1 and 2 are shown in Appendix 8.3. 3.5.2 Evaluation We present here the performance of our framework with respect to the contrastive ques- tions presented in Section 3.2, working with three benchmark PDDL+ planning domains and three state-of-the-art planners SMTPlan+ (SM), ENHSP (EN), and UPMurphi (UP). The results are presented in Table 3.3. In the table, BM represents the benchmark problem domains in PDDL+. Ins represents the problem instance of a domain. For each domain, we present two problem instances in the table. PL reports the hybrid system planner chosen for solving the planning problem instance. For each problem instance in each domain, we execute all the three planners for each of the eight questions, with a time bound of one hour and peak memory usage bound of 4 GB. InDur and Len show the original plan’s makespan and plan-length for the problem instance. QN represents the contrastive question addressed in Section 3.3 (3.3.1-3.3.8). For each of these con- trastive questions, an HPlan is generated. H-dur and H-len show the makespan and the plan-length of the corresponding HPlan. Diff-cost τ and Diff-cost len are the contrastive explanation (CE) metrics discussed in Definition 3.2.6. For abbreviation, we use DC τ and DC l to denote these metrics in the table. DC τ presents the plan-duration difference and DC l presents the plan-length difference of the original plan and the HPlan. A positive DC τ indicates a longer makespan, whereas a positive DC l indicates a longer plan-length of the HPlan in comparison to the original one. CM reports the number of constraint modifications that are made in the domain (inclusive of constraint additions and dele- tions). Time and H-time represent the plan generation time whereas Mem and H-mem report the memory usages for the original plan and the HPlan, respectively. Finally, HQ represents an evaluation of the quality of the HPlan against the original plan while con- sidering the contrastive explanation metrics DC τ and DC l , where a ✓ indicates that the original plan is better than the HPlan, a × indicates that the HPlan is better than the original plan, and a = indicates that both the plans are of equal quality. A ✓ in the HQ field for a contrastive question provides an explanation on why the planner has provided such a plan over the hypothetical one. However, × in the HQ field exhibits that there is a better plan possible for the problem instance than the original plan, considering plan makespan and length as quality evaluation metrics. A = in the HQ field indicates that the produced HPlan is an alternate plan of the same quality with respect to the original plan. In some of the problem instances, the HModels for the corresponding contrastive questions produce no plans which is either reported by the planner (indicated as NP in the table) or the planner exceeds the resource bounds set by us (indicated as RO in the table). Q6 questions the optimality of plan-length of a plan (see Section 3.3.6), and is not 82 3.5. RESULTS included in the table since it represents an iterative questioning and compilation (details can be seen in Table 3.1). Q5 is also not included in the table for the planetary-lander domain since this question is not applicable to the considered planning problem instances in which the original plan is such that the actions that appear in the plan do not repeat. Analysis of Results: In the car domain, for problem instance 1, SMTPlan+ produces HPlan for the contrastive questions QN1, QN2, QN3, QN6, and QN7, but is not able to produce HPlans for QN4, QN5, and QN8 within the time bound. We understand that QN4, QN5, and QN8 are actually unsolvable as ENHSP, which is a sound and complete planner, reports the non-existence of a plan for these instances. Additionally, it may be noted that UPMurphi reports no plan for QN4, however, it reports a memory overflow for QN5 and QN8. Similarly, for problem instance 2 of the car domain, SMTPlan+ reports plans for the contrastive questions QN1, QN2, QN3, QN6, and QN7, but does not produce any plan for QN4, QN5, and QN8 due to timeout. In this case, both ENHSP and UPMurphi produce plans for all questions except QN4, for which both of them report a no plan. This leads us to conclude that the three planners agree on the positive questions. Further, we note the fact that QN4 is actually unsolvable for this instance. Table 3.3 shows that for the generator-events domain, for problem instance 1, SMTPlan+ produces HPlans for the contrastive questions QN1, QN2, QN3, QN6, and QN7, but does not produce any plan for QN4, QN5, and QN8 due to timeout. For problem instance 2, SMTPlan+ produces HPlans for the questions QN2, QN3, and QN6, and does not produce any plan for QN1, QN4, QN5, QN7 and QN8 due to timeout. In both the problem instances, both ENHSP and UPMurphi fail to produce any plan on the original domain and for all the questions as well, due to resource overflow. A similar interpretation follows for the entries on the Planetary-lander domain. Contrastive Explanation: We observe that for most of the problem instances, the original plan is better than the contrasting alternate one (HPlan) marked by ✓ in Table 3.3 and thus justifies the original plan to the user. In instances marked with a ×, a user may consider to re-plan given that an alternate hypothetical plan is better in comparison to the original one. Performance Evaluation: We compare the time and memory usage to generate the HPlan with the time and memory usage to generate the original plan by the planners SMTPlan+, ENHSP, and UPMurphi, presented in Figure 3.11 and Figure 3.12, re- spectively. The y-axis represents time (in seconds) in Figure 3.11 and memory usage (in MB) in Figure 3.12, while the x-axis represents the alternate plans generated for the con- trastive questions. OP represents the original plan and QN1 to QN8 denote the questions addressed in Section 3.3.1 through 3.3.8 for which the alternate plan is generated. We have included only those planning instances where a valid plan is generated or the in- stance is reported as unsolvable by the planner. The original plan and the HPlans for problem instance 1 are marked in Red, whereas the instances of problem instance 2 are marked in Green. We can conclude that the time and memory usage to generate the 83 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS BMInsPLDurLenQNH-durH-len CE metrics CM TimeH-timeMemH-mem HQ DC τ DC l (in sec)(in sec)(in MB)(in MB) Q1126-202120.046<1✓ Q2184-14010.024<1× Q341.549.5040.036<1✓ SM324Q4RORORORO10.027RO<1RO✓ Q5RORORORO3RORO✓ Q7186-142210.066<1✓ Q8RORORORO5RORO✓ Q15028111270.40470.7✓ Q240141-210.33766.1✓ Q339220640.37178.6✓ p1EN3916Q4NPNPNPNP10.4250.28582.366.3✓ Q5NPNPNPNP32.351481.3✓ Q749291013850.537114.1✓ Q8NPNPNPNP54.1351198.9✓ Q19.5056-1.4982104.672166.4✓ Q211.50340.5014.972161.3✓ Q311.50340.5046.472161.5✓ UP11.0034Q4NPNPNPNP14.773.62159.81968.8✓ Q5RORORORO4RORO✓ Q711.00560.0022215.822168.6✓ CDQ8RORORORO6RORO✓ Q1126-202120.048<1✓ Q2184-14010.025<1× Q341.549.5040.031<1✓ SM324Q4RORORORO40.028RO<1RO✓ Q5RORORORO3RORO✓ Q7186-142210.075<1✓ Q8RORORORO5RORO✓ Q145611946150.714155.6✓ Q25037242213.242872.5✓ Q31519-9440.46179.8✓ p2EN2615Q4NPNPNPNP10.5350.475133.5110.4✓ Q5501424-131.023259.1✓ Q747642149792.378350.9✓ Q8244-2-1151.001230.5✓ Q17.5089-0.4982156.62169.6✓ Q28.50670.5015.562162.5✓ Q38.006700410.462162.5= UP8.0067Q4NPNPNPNP18.113.82160.92161.7✓ Q59.50451.498-2415.32159.6✓ Q78.00890.0022419.212175.3✓ Q811.00342.997-367.082161.5✓ Q110003-64040.19232.1× Q2107238040.22332.2✓ Q310013-63040.071<1× p1SM10643Q4RORORORO10.068RO<1RO✓ Q5RORORORO3RORO✓ Q71064300175.26333.2= Q8RORORORO4RORO✓ GDQ1RORORORO3RORO✓ Q21072480414.50837.3✓ Q310014-63040.23732.0× p2SM10644Q4RORORORO11.089RO32.7RO✓ Q5RORORORO3RORO✓ Q7RORORORO24RORO✓ Q8RORORORO4RORO✓ Q118.00340.0011864.792222.1✓ Q218.002300542.062224.2= Q318.00340.0011536.682224.5✓ p1UP18.0023Q418.00340.0011142.30428.382225.52213.1✓ Q5NPNPNPNP454.942221.9✓ Q718.0023002137.672241.3= Q8NPNPNPNP7245.462224.8✓ PDQ118.00340.00118114.902225.5✓ Q218.002300586.402221.7= Q318.00340.0011565.832224.6✓ p2UP18.0023Q418.00340.0011176.4247.062223.32213.1✓ Q5NPNPNPNP482.802224.7✓ Q718.0023002160.902241.0= Q8NPNPNPNP7237.112224.6✓ Table 3.3: BM represents the problem domains in PDDL+,and Ins represents the problem instances of a domain. PL reports the hybrid system planner chosen for solving the instances, where SM, EN, and UP represent SMTPlan+, ENHSP, and UPMurphi, respectively. Dur and Len represent the original plan-duration and plan length for a problem instance, QN represents the set of contrastive questions presen- ted in Section 3.3, H-dur and H-len represent the plan-duration and the plan-length of the hypothetical plan HPlan. DC τ and DC l represent the plan-duration difference and the plan-length difference of the HPlan with the original plan, respectively; these are the contrastive explanation (CE) metrics illustrated in Def 3.2.6. CM reports the number of constraint modifications that are made in the domain (inclusive of con- straint addition and deletion). Time and H-time represent the plan generation time whereas Mem and H-mem report the memory usages for the original plan and the HPlan, respectively. HQ represents an evaluation on the quality of the HPlan against the original plan. 84 3.5. RESULTS contrastive alternate plans (HPlans) by the planners are similar to the time and memory usage to generate the original plan, except in some of the cases. The contrastive explan- ation framework does not, therefore, incur much performance overhead. The scalability of the framework is dependent on the scalability of the planner used for plan generation. Note that the HPlan generation for QN8 for problem instance 1 in Figure 3.11b and QN8 for problem instance 1 and 2 in Figure 3.11g take considerable time than the genera- tion time of the original plan. The HModels corresponding to these problem instances generate no valid plans. Deducing whether there exist no plans is more performance- intensive than generating a plan for a problem instance, which justifies the added time taken to solve these instances. QN7 for the problem instance 1 in Figure 3.11e repres- ents the compilation that questions an action sequence in a plan, which imposes many constraints and additional actions in the domain, which may be the cause for taking the additional time to find a solution in this instance. For similar reasons, QN8 for prob- lem instance 1 in Figure 3.12b, and Q7 for problem instances 1 and 2 in Figure 3.12g require additional memory in solving the planning instances. The experiments were per- formed on a machine with 8 GB RAM, Intel Core i5-8250U@1.60GHz, 8 8-core processor with Ubuntu 18.04 64-bit OS. All the example domains, the problem files, and the con- structed hypothetical models (HModels) for the contrastive questions can be found at: https://gitlab.com/Sazwar/contrastive-explanations. 3.5.3 Performance Evaluation of Proving Absence of Plan To prove the absence of a plan with bounded reachability analysis in our running example, we present the experimental observations with dReach. We set a δ to a reasonably small value of 0.01, and k is assigned a value of 10 and 20 in 2 separate experiments. For all the experiments, dReach reports the no-plan instances as unsatisfiable, which implies that there is no plan for these problem instances within the plan-depth k. The reachability analysis results of the no-plan-models are shown in Table 3.4, where prob_ins represents the dReach encoding of the corresponding problem instances presented in Section 3.3.4, 3.3.5 and 3.3.8, plan-depth is the bound on the plan length k, time represents the time taken by dReach to complete the reachability analysis and satisfiability reports whether the problem instance is reported as satisfiable by dReach (where SAT and UNSAT indicates satisfiable and unsatisfiable respectively). The planning problem instance of Section 3.3.5 did not terminate within 48 + hrs for a plan-depth of more than 13, and thus we could not draw any conclusion from this instance beyond a depth of 13. All our no-plan-models were unsatisfiable (UNSAT) in H δ which proved that there are no plans within the plan-depth k in the corresponding H system. However, the same can not be asserted if a problem instance is observed to be satisfiable (SAT) in the corres- ponding H δ model for a given δ and k. To cover such issues, we propose an algorithm that iteratively lowers the value of δ and checks the reachability of the problem upto a certain precision (Algorithm 3.2). The algorithm takes an HModel and plan-depth k as inputs. Initially, we set the precision δ to be 0.01. It then checks the satisfiability of HModel in the corresponding δ-perturbed hybrid system and iteratively lowers the δ precision (upto 10 −6 ) when the problem is SAT. If the problem instance becomes UNSAT, it concludes 85 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS (a) The HPlan generation times using SMTPlan+ for two problem instances in Car domain. (b) The HPlan generation times using ENHSP for two problem instances in Car domain. (c) The HPlan generation times using UPMurphi for problem instance 1 in Car domain. (d) The HPlan generation times using UPMurphi for problem instance 2 in Car domain. (e) The HPlan genera- tion times using SMT- Plan+ for problem instance 1 in Generator-events do- main. (f) The HPlan genera- tion times using SMT- Plan+ for problem instance 2 in Generator-events do- main. (g) The HPlan genera- tion times using UPMurphi for instances in Planetary- lander domain. Figure 3.11: Scatter graph comparing the HPlan generation time over the original plan. OP represents the original plan whereas QN1, QN2, QN3, QN4, QN5, QN7 and QN8 represent the HPlans generated against the compilations for the contrastive questions discussed in Sec. 3.3.1, 3.3.2, 3.3.3, 3.3.4, 3.3.5, 3.3.7 and 3.3.8, respectively. 86 3.5. RESULTS (a) The memory usage in HPlan generation us- ing SMTPlan+ for prob- instances in Car domain. (b) The memory usage in HPlan generation using ENHSP for prob-instances in Car domain. (c) The memory usage in HPlan generation using UPMurphi for prob-ins 1 in Car domain. (d) The memory usage in HPlan generation using UPMurphi for prob-ins 2 in Car domain. (e) The memory usage in HPlan generation using SMTPlan+for prob-ins 1 in Generator-events. (f) The memory usage in HPlan generation using SMTPlan+ for prob-ins 2 in Generator-events. (g) The memory usage of the HPlan generation by UPMurphi for instances in Planetary-lander. Figure 3.12: Scatter graph comparing the memory usages by the planner for the generation of HPlans and the original plan. OP represents the original plan whereas QN1, QN2, QN3, QN4, QN5, QN7 and QN8 represent the HPlan corresponding to the contrastive questions addressed in Section 3.3.1, 3.3.2, 3.3.3, 3.3.4, 3.3.5, 3.3.7 and 3.3.8, respectively. 87 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS prob_insbound on plan-depth (k)time (sec)satisfiability Sec. 3.3.4 100.288UNSAT 201.277UNSAT Sec. 3.3.5 10541.423UNSAT 134010.524UNSAT Sec. 3.3.8 101.931UNSAT 208.152UNSAT Table 3.4: Bounded reachability analysis of the no-plan-models for a given δ perturb- ation of 0.01. that there is no plan. However, the explanation of the non-existence of a plan remains inconclusive when the algorithm returns SAT for the δ perturbation less than 10 −6 . Algorithm 3.2: algorithm to prove absence of plan by bounded reachability analysis Input: HModel, plan-depth k. 1 begin 2 δ = 0.01; /* Initialize the value of δ */ 3 while (δ ̸= 10 −6 ) do /* Checking the value of δ */ 4res = dReach(HModel, δ, k); /* DReach call */ 5if res is SAT then /* Satisfiability check */ 6δ = δ∗ 0.1; /* Lowering the value of δ by 0.1 */ 7else 8Print "Goal state unreachable from the initial state in the domain"; 9return; 10end 11 end 12 Print "Cannot explain the non-existence of a plan"; 13 return; 14 end 3.6 Related Works A roadmap for explainable artificial intelligence planning with contrastive questions such as “Why did the planner do A rather than B?” or simply “why” and “why not” questions have been addressed in (Fox et al. [2017]) where the authors discuss things that need to be explained, features of planning that facilitate explanations and how to achieve the goal of providing reasonable answers to user questions through the explanations. The work in (Miller [2018]) extended the structural causal model approach to the case of contrastive explanations and defined contrastive explanations for two types of questions: alternative questions and congruent questions, where the author argues that an alternative question of the form “Why A rather than B?” is a contrastive question. (Lim et al. [2009]) shows 88 3.6. RELATED WORKS that these why and why not questions benefit in terms of objective understanding and feelings of trust. A detailed overview of the explainable artificial intelligence planning landscape and different terms used in this domain are introduced in (Chakraborti et al. [2018]), while (Chakraborti et al. [2017]) considered explanation as a model reconciliation problem assuming that the agent and the human may have possibly different models of the environment. In such a scenario, the agent explains those actions to the human that are not expected to be executed, taking the human model as a reference. Explanation here can be seen as a reconciliation between the agent and the human. In (Dhurandhar et al. [2018]), a method, called Contrastive Explanations Method (CEM) was proposed to generate contrastive explanations for differentiable models such as deep neural networks, where one has complete access to the model. In (Dhurandhar et al. [2019]), Model Ag- nostic Contrastive Explanations Method (MACEM) was proposed to generate contrastive explanations for any classification model where one is able to only query the class probab- ilities for a desired input which allows them to generate contrastive explanations for not only neural networks but models such as random forests, boosted trees and even arbitrary ensembles that are still among the state-of-the-art when learning on structured data. In (Smith [2012]), Smith put forward the challenge of planning as an iterative process for better modeling preferences and providing explanations. In our paper, we have also taken this concept of iterative explanations for our model. While improving the user’s level of understanding and building trust in the system are the main purposes of these explana- tions, they can be local (regarding a specific plan) or global (concerning how the planning system works in general). (Krarup et al. [2019]) focus on local explanations of temporal and numeric planning problems, introducing a formal description of the compilation from user questions to constraints in a PDDL2.1 planning setting, and explaining why a planner has made a certain decision. In contrast, our paper applies a similar approach to both local and global planning scenarios for modeling mixed discrete-continuous planning prob- lems using the PDDL+ planning setting. (Cashmore et al. [2019]) presents a prototype framework to facilitate Explainable Planning as a service, limited to discrete systems. Our framework is built as a plug-in on top of SMTPlan+, the state-of-the-art planner for hybrid system domains. A different direction to explainable AI planning is explaining the unsolvability of a planning problem when there is no-plan. There are notable works in this direction trying to explain the unsolvability of a planning problem. (Göbelbecker et al. [2010]) argues that excuses can be produced for why a plan cannot be found, and the work proposes a formalization of counterfactual alterations to the original planning task such that the new planning task will be solvable. Based on that, they provide an algorithm to find these ex- cuses in a reasonable time. (Sreedharan et al. [2019]) uses hierarchical model abstractions to generate the reason for the unsolvability of planning problems. This hierarchical model abstraction relaxes a planning problem until a solution can be found. Then, they look for landmarks of this relaxed problem that cannot be satisfied in less relaxed versions of the problem. The unsatisfiability of these landmarks provides a succinct description of critical propositions that cannot be satisfied. (Eifler et al. [2020]) has taken a somewhat different approach by deriving properties that must be exhibited by all possible plans that could 89 CHAPTER 3. A CONTRASTIVE PLAN EXPLANATION FRAMEWORK FOR HYBRID SYSTEM MODELS serve as explanations in case of unsolvability. We have observed that some of the contrastive questions result in no-plans in our ex- ample domain, which serves as the motivation to verify the unsolvability of those no-plan problem instances. Though similar work on verifying the unsolvability of the planning problem has been done previously (Eriksson and Helmert [2020], Eriksson et al. [2017]), these are on discrete domains. Verifying the unsolvability of the planning problems in hybrid systems comes with an additional challenge since the planning problem is undecid- able in general hybrid domains. In this work, we attempt to prove the non-existence of a plan in the hybrid domain by reachability analysis. We do not provide explanations for the non-existence of a plan in this paper. Addressing this issue is left for future work. 3.7 Limitation In the car domain, we observe that SMTPlan+ does not guarantee an optimal plan generation, and the plan produced is rather just one among the many possible plans for a given problem instance in the domain. Due to this, some of the conclusions may be weak, which are based on comparing the hypothetical plan with the original one. In the no-plan explanation framework, we check the existence of a plan using δ-reachability analysis of DReach of the no-plan model upto a given plan-depth k. The non-existence of a plan for a problem instance upto k depth does not imply that there does not exist a plan with length > k. However, the choice of k has significance on the usability of a plan in real- world applications, and concluding the non-existence of plans of bounded length can be insightful to a user. 3.8 Conclusion In this chapter, we explore a set of contrastive questions that a user of a planning tool may raise, and we propose a re-model and re-plan framework to provide explanations to such questions. Specifically, given a hybrid system model in PDDL+ and a plan describing the set of desirable actions on the same to achieve a destined goal, our framework can integrate contrastive questions in PDDL+ and synthesize alternate plans using a hypothetical model (HModel) constructed by imposing constraints drawn from the questions. We present a detailed case study on our approach, and with a comparison metric, we compare the original plan with the alternate ones. We show that our contrastive explanations can draw conclusions about the planning domain and the planning tool as well, such as identifying that the plan is not always necessarily cost-optimal, that one or some actions need to appear later in the plan, etc. Even the no-plans are helpful to figure out the critical actions for a certain problem. Further, we provide a no-plan explanation algorithm for our no-plan models through bounded reachability analysis to verify the reachability of the problem instances. We demonstrate experimental results on three planning domains using three state-of- the-art planners. We believe our frameworks can be of immense importance to the hybrid systems planning community for synthesizing better, explainable plans. 90 Chapter 4 A Contrastive Explanation Tool for Plans in Hybrid Domains Chapter Abstract: This work presents a web-based tool for generating contrastive ex- planations of plans in hybrid domains. The tool offers a collection of contrastive questions over a plan, such as "Why use action A?", for users to select. An explanation of the user question is produced by contrasting the original plan against an alternative that meets the user’s expectation implicit from the question. The tool has the provision to contrast with the best alternative that the underlying planner can generate in terms of plan makespan (duration of a plan) and length. The current version supports two planners in the hybrid domain, namely SMTPlan+, and ENHSP, which a tool user can select. The tool consists of (1) A web-based interactive GUI for selecting questions, viewing contrastive plans and the generated explanations, and (2) A back-end implementing an iterative re-modeling and re-planning algorithm. We demonstrate the working of the tool over two case studies. T here has been a surge of applications where automated planners are deployed and where humans and autonomous agents collaborate to achieve a desired objective. En- suring the safety, robustness, and trustworthiness of such systems is of utmost importance. Explainable Artificial Intelligence Planning (XAIP) (Chakraborti et al. [2020], Fox et al. [2017], Hoffmann and Magazzeni [2019], Zhang et al. [2017]) is a promising field of research that aims to provide AI planning systems the ability to explain the rationale behind a de- cision of an autonomous agent to engage in trustworthy collaborations with humans. A roadmap for XAIP is proposed in (Fox et al. [2017]) where the authors discuss a taxonomy of user questions that should be addressed, things that need to be explained, features of planning that facilitate explanations and how to achieve the goal of providing reasonable answers to user questions through the explanations. When there is a mismatch between a plan obtained from an automated planner and the user’s expectations, reconciliations are often required. (Mueller et al. [2019]) has shown that users tend to ask "why" questions when seeking explanations about a specific part of the plan, referred to as a local ques- tion, while "how" or "what" questions are asked when seeking explanations about the plan 91 CHAPTER 4. A CONTRASTIVE EXPLANATION TOOL FOR PLANS IN HYBRID DOMAINS as a whole, referred to as global questions. Insights from social sciences (Miller [2019]) suggest that "why" questions are often contrastive, taking the form of "Why A rather than some B?" Based on these observations, (Fox et al. [2017]) demonstrates that when the planning domain is well-known to the user, it is more common to ask more local, contrastive "why" questions than global "how" or "what" questions. A contrastive ques- tion can be addressed with a contrastive explanation by virtue of its ability to highlight the differences between the original plan and an alternative plan that accommodates the user’s suggested alterations. This mode of explanation has proven to be highly effective in enhancing comprehension and is also simpler to execute than a full-fledged causal analysis (Krarup et al. [2021b]). Furthermore, contrastive explanations have several comparative advantages, such as enabling a clear and direct comparison between the planner’s given plan and the user’s conceived plan. The paradigm of contrastive explanations has been ex- plored in the context of planning domains having discrete state-transition representations (PDDL2.1 Fox and Long [2003]) (Cashmore et al. [2019]). In our earlier work (Sarwar et al. [2022]), we presented a general framework for contrastive explanation of plans in do- mains represented as discrete state-transition systems, together with continuous variables with their evolution given as differential equations. Such domains are known as hybrid domains in literature and modeled with PDDL+ (Cashmore et al. [2016a]). In this work, we present an interactive web interface for a contrastive explanation of plans in hybrid domains. The explanation algorithms implemented in the tool are based on the work in (Sarwar et al. [2022]). It offers a user-friendly interface that can be used by AI practitioners for explainable planning. The key contributions of this work are: • A web-based interactive tool interface supporting XAIP in hybrid domains. • An iterative re-modeling and re-planning algorithm to find a competitive contrastive plan with respect to the comparison metrics (e.g., plan makespan, plan length) that the underlying planner can provide. • A provision to experiment with two planners for hybrid domains and compare and contrast the explanations produced thereof. Note on Collaboration and Authorship: This work was developed in collaboration and has been previously published in Dey et al. [2024]. For this thesis, the framework design and the algorithmic backbone of the tool are the primary contributions. The first author primarily implemented and carried out experimental validation. The rest of this chapter is organized as follows. In Section 4.1, we provide an overview of our framework. Section 4.2 presents implementation and results. Section 4.3 discusses related works, and Section 4.4 concludes this work. 4.1 Tool Overview We first present the workflow of the tool as shown in Fig 4.1. The tool takes as input a planning problem Π (Sarwar et al. [2022]) which comprises a pair (Dom,Prob), where Dom is a representation of the planning domain in PDDL+, and 92 4.1. TOOL OVERVIEW HModel Generator Planner Selector Planner HModel Explanator Plan Contrastive Explanation ? HPlan Plan Planning Problem Interpreter Question Template Selection Question Binder Q Human-XAIP Interface Iterative Re-planning Process Step1 Step3 Step2 Step4 Step5 Step6 Step7 Step8 Figure 4.1: Tool Workflow Prob is a planning problem instance that constitutes an initial state and a goal condition. The planning problem Π is fed to a hybrid system planner SMTPlan+ (Cashmore et al. [2020]) or ENHSP (Scala et al. [2016]) based on the user’s choice. The planner produces a plan φ which is presented to the user. This plan is referred to as the original plan. A plan is a time-annotated sequence of actions, where the annotation denotes the time at which the action is to be applied (Sarwar et al. [2022]). The user chooses a template question Q on the original plan φ. The tool provides a drop-down menu for choosing the question. Our tool offers the following set of seven templatized contrastive questions to the users: (a) Why did the planner choose to do action A and not B instead? (b) Why did the planner not choose to do action A later in the plan? (c) Why did the planner not choose to do action A earlier in the plan? (d) Why did the planner take action A in the plan, instead of not taking it? (e) Why not have fewer occurrences of action A in the plan? (f) Why is the accumulative duration of the plan not less? (g) Why is the length of the plan not less? This set of questions is not exhaustive, but we believe these questions are important in these domains. Once the question is chosen, the user is prompted to bind the free variables in the templatized question to the corresponding values in the plan context. The original planning problem Π is then altered to a hypothetical planning problem (HModel) Π ′ based on the constructions proposed in (Sarwar et al. [2022]) implemented in the HModel Generator module. The hypothetical planning problem Π ′ is fed to the same planner chosen in Step-2, and a hypothetical plan (HPlan) φ ′ as expected by the user is generated. The tool interface displays the HPlan φ ′ along with a contrastive explanation by comparing it with the original plan φ. The tool supports the following comparison metrics: (a) Plan makespan and (b) Plan length. A shorter makespan and length of a plan signify that the goal can be achieved relatively quickly with less number of applied actions, which may incur costs. We now discuss the major building blocks of the tool in the following subsection. 93 CHAPTER 4. A CONTRASTIVE EXPLANATION TOOL FOR PLANS IN HYBRID DOMAINS 4.1.1 Module Description Planner Selector: A user can choose one among these 2 state-of-the-art hybrid system planners SMTPlan+ and ENHSP via the tool interface as shown in Figure 4.2a. We try to make this interface generic for any planner that is capable of dealing with PDDL+ syntax and can be plugged in as a plan generation engine. Consider a planning domain that models the motion of a car. The function symbols represent the distance, velocity, and acceleration of the car, while the predicates define the states of the car. The goal is to safely traverse a distance of 30 units within 50 units of time. The planning domain represented in PDDL+ is shown in Listing 4.1 and a problem instance that we have used in this domain is shown in Listing 4.2. A detailed description of the domain can be found at https://github.com/manabjamin2nadved1947/XAIP.git. Listing 4.1: The car domain in PDDL+. ( define ( domain car ) ( : predicates ( running ) ( engineBlown ) ( goalReached )) ( : functions (d) (v) ( a ) ( upLimit ) ( downLimit ) ( runningTime )) ( : process moving : parameters () : precondition ( and ( running )) : e f f e c t ( and ( inc reas e (v) (∗ #t ( a ) ) ) ( in crea se (d) (∗ #t (v ) ) ) ( in cre ase ( runningTime ) (∗ #t 1 ) ) ) ) ( : action a c c e l e r a t e : parameters () : precondition ( and ( running ) (< ( a ) ( upLimit ) ) ) : e f f e c t ( and ( incr eas e ( a ) 1))) ( : action d e c e l e r a t e : parameters () : precondition ( and ( running ) (> ( a ) ( downLimit ) ) ) : e f f e c t ( and ( decrease ( a ) 1))) ( : event engineExplode : parameters () : precondition ( and ( running ) (>= ( a ) 1) (>= (v) 100)) : e f f e c t ( and ( not ( running ) ) ( engineBlown )( assign ( a ) 0))) ( : action stop : parameters () : precondition ( and(=(v)0)(>=(d) 30)( not ( engineBlown ) ) ) : e f f e c t ( goalReached ) ) ) Listing 4.2: The planning problem in the car domain in PDDL+. ( define ( problem car_prob ) ( : domain car ) ( : i n i t ( running ) (= ( runningTime ) 0) (= ( upLimit ) 1) (= ( downLimit ) −1) (= d 0) (= a 0) (= v 0)) ( : goal ( and ( goalReached ) ( not ( engineBlown )) (<= ( runningTime ) 50)))) A domain and problem file is entered (Fig. 4.2a) alongside a selected planner (SMTPlan+), which returns a plan as follows: Listing 4.3: A plan generated by SMTPlan+ on car domain on the problem in Listing 4.2. Time ActionDuration 0 . 0 :( a c c e l e r a t e )[ 0 . 0 ] 94 4.1. TOOL OVERVIEW 1 . 0 :( d e c e l e r a t e )[ 0 . 0 ] 3 1 . 0 :( d e c e l e r a t e )[ 0 . 0 ] 3 2 . 0 :( stop )[ 0 . 0 ] makespan: = 32.0 units. (a) Loading planning problem and choosing a planner. (b) Observing original plan and selecting a templatized question. Figure 4.2: A snapshot of the tool’s GUI Interpreter: This module involves transforming the user’s contrastive inquiry into a formalized question. This is accomplished by feeding the user-chosen templatized con- trastive question together with template bindings of actions, time-instances, and frequency of actions of the domain, leading to the generation of a concrete question. In the plan presented in Listing 4.3, a user may wish to ask why the planner chose to decelerate at the 1 st time unit rather than not earlier, anticipating that it would have yielded a better plan. We take this question as the running example to illustrate the rest of the blocks. By selecting the third templatized question from the menu as shown in Figure 4.2b, that is "Restricting an Action to Appear Before a Certain Time", the user is redirected to an input form to specify the relevant action and the time to bind with the placeholders in the question as shown in Figure 4.3. After gathering the user input, the question binder generates a concrete question. For example, when the user questions the use of the de- celerate action at time instant 1.0 rather than earlier, the question binder concretizes the template question with the action instance decelerate and a time instance t which is less than 1. 95 CHAPTER 4. A CONTRASTIVE EXPLANATION TOOL FOR PLANS IN HYBRID DOMAINS Figure 4.3: Binding templatized question with instances. HModel generator: This module automates the generation of HModel by receiving the concrete question generated by the Interpreter and the original planning problem as inputs. It applies pattern-matching techniques with arguments specified as regular expressions (regex) and utilizes string manipulation to compile a set of constraints in PDDL+ to generate the HModel. In our example, the domain is updated by introducing a new action called decelerate_new, such that any valid plan must contain this action, which always appears before the time instance 1.0 unit (See Listing 4). Listing 4.4: The update in the domain with the decelerate_new action. . . . ( : action decelerate_new : parameters () : precondition ( and (< ( running_time ) 1) ( running ) (> ( a ) ( down_limit ) ) ) : e f f e c t ( and ( do_before_new ) ( decrease ( a ) 1))) . . . The goal state in the problem file is updated with additional requirements (Listing 5): Listing 4.5: The modified goal state with the do_before_new predicate. ( : goal ( and(do_before_new) ( goal_reached ) ( not ( engineBlown )) (<= ( running_time ) 50) ( transmission_fine ) ) ) Explanator: After an HPlan is generated by the planner from the HModel, it is passed to this module along with the original plan for generating a contrastive explanation. The quality of the hypothetical plan is compared with the original plan with respect to the metrics of makespan and plan length (the number of actions appearing in the plan). When the makespan as well as the length of HPlan is longer than that of the original plan, the tool reports this to the user as a justification for using the original plan instead of the alternate user-expected hypothetical plan. On the other hand, if either the makespan or the plan length of the hypothetical plan is better than that of the original plan, the tools report this to the user, in which case a replan may be considered. The following HPlan (see Figure 4.4) is received for the contrastive question of our running example. Upon comparison with the original plan as shown in Listing 4.2, it is evident that the original plan has the same length as the contrastive plan, but has a better makespan. 96 4.1. TOOL OVERVIEW This is highlighted to the user as an explanation of the original plan. The provision of the show optimal and show optimal length buttons shown in Figure 4.4 entails the generation of an optimal plan in terms of makespan and length, respectively, based on the user’s specification for each question. The approach employed to achieve this is described below. Figure 4.4: An HPlan generated by SMTPlan+ according to the contrastive question. Iterative Re-planning: This module harnesses the power of iterative modeling to fa- cilitate repeated user questioning. Sometimes users may need to ask repeated questions to refine their queries and obtain better answers, particularly when it comes to optimizing plan length or duration (makespan). For instance, a user can repeatedly ask questions (f) and (g) (Section 2, point 4) to converge towards a makespan and length optimal plan. Our tool allows for iterative execution of steps 5-8 (as outlined in Section 4.1) to support these repeated queries. This process can be repeated until no further hypothetical plans can be generated for the given query. Iterative Re-planning with Bisection: A limitation of the iterative re-planner mod- ule is that of repeated manual tool invocation to progressively converge towards an optimal hypothetical plan with respect to either makespan or plan length. Additionally, a tool user may wish to contrast the original plan against that hypothetical plan, which is makespan or length optimal, instead of contrasting with just any. This tool has a provision that progressively searches for better alternate plans and terminates with the best-found hy- pothetical plan to be contrasted with. This is achieved using a bisection algorithm (see Algorithm 4.1). The algorithm begins by setting an upper and lower bound of the op- timization objective, which is either makespan or plan length. The lower bound is set to 0, and the upper bound is set to the makespan (length) returned for a hypothetical plan by the planner. We use variables high and low respectively to store these values. In each iteration, the algorithm probes whether a plan exists with a makespan (length) ≤ mid = low+high 2 . If such a plan is found, we improve the upper bound high to the makespan (length) of that plan. Conversely, in the absence of a feasible plan, we improve the lower bound (low) to mid. The algorithm terminates when the difference between the upper and lower bound is less than a given tolerance tol, and the plan having a makespan (length) equal to high is returned as the best found contrastive plan. 97 CHAPTER 4. A CONTRASTIVE EXPLANATION TOOL FOR PLANS IN HYBRID DOMAINS Algorithm 4.1: Bisection (Π.dom, Π.prob, low, high, tol) Input: low, high, tol Output: makespan of the contrastive plan (high), the contrastive plan φ 1 begin 2if abs(high - low) < tol then 3modifyProblemFile(Π.prob, high) ;/* modify makespan bound */ 4φ = runPlanner(Π.dom, Π.prob) ;/* get HPlan */ 5return (high, φ); 6end 7mid = (low+high)/2; 8modifyProblemFile(Π.prob, mid); 9 φ = runPlanner(Π.dom, Π.prob); 10if φ == null then 11return Bisection(Π.dom, Π.prob, mid, high, tol) ;/* No plan found */ 12end 13if φ != null then/* A plan exists */ 14high = getMakespan(φ) ;/* high = getLength(φ) for length objective */ 15return Bisection(Π.dom, Π.prob, low, high, tol) 16end 17 end 4.2 Implementation and Results Implementation: We use Python and its regular expression library package re, Javas- cript for designing the web interface, Nodejs for the server environment, Express framework for web application features, Bootstrap 5.2.3 and EJS for designing front-end GUI. The ex- periments are performed in a machine with 8 GB RAM, an Intel Core i5-8250U@1.60GHz 8-core processor, with Ubuntu 20.04 64-bit OS. Case Study: We demonstrate the tool’s performance on two benchmark PDDL+ plan- ning domains, the car domain and the generator-events domain. A brief description of the car domain is discussed in Section 4.1.1. The Generator-events Domain consists of a generator and two fuel tanks. The generator consumes fuel while running and has a fuel capacity. The fuel in the tanks can be poured into the generator. The dynamics of fuel consumption and fuel pouring from tanks into the generator are given. The planning task is to run the generator for 1000 time units safely, that is, without an overflow or underflow of the fuel. The source code of the tool and a detailed description of the domains can be found at https://github.com/manabjamin2nadved1947/XAIP.git. Evaluation: A performance summary of generating explanations is provided in Table 4.1. For each question, we construct an HModel, and the HModel Compilation Time shows the time needed to generate these HModels. The mean HModel compilation time in our framework is approximately 0.055 seconds, which is significantly small. In the SMTPlan+ and ENHSP columns, we report the corresponding HPlan generation time. The mean HPlan generation time for SMTPlan+ in the car domain and the generator- events domain is, respectively, 0.05 seconds and 0.079 seconds, whereas for ENHSP the mean time is 0.53 seconds in the car domain. The generator-events domain is evaluated using SMTPlan+ only, as the domain is not supported by ENHSP. However, for question 98 4.3. RELATED WORKS 4, which specifies to exclude one action from the original plan, neither SMTPlan+ nor ENHSP is able to generate a solution within 10 seconds in both domains. We also note the time required to generate a contrastive plan by iterative re-planning with Bisection, with minimum makespan objective, shown under the Best Contrastive Plan Time in the table. The average time taken to generate the best contrastive plan is 59.26 seconds and 98.09 seconds for the car and generator-events domains. The results indicate that the best contrastive plan generation process is expensive in comparison to HModel generation and plan execution. This can be attributed to the iterations in the Bisection algorithm. HModelBest Contrastive BenchmarkQ No.SMTPlan+ENHSPCompilation TimePlan Time q10.090s0.842s0.038s1m2.396s q20.059s0.520s0.026s1m28.344s Car q30.041s0.384s0.029s0m51.703s domain q4NPNP0.041sNP q50.036s0.492s0.092s0m52.123s q60.038s0.436s0.025s1m30.856s q70.068s0.511s0.244s10.106s q10.034s0.036s2m32.106s q20.034s0.027s1m31.017s Generatorq30.049s NA 0.037s2m23.407s -eventsq4NP0.067sNP domainq50.158s0.034s1m30.850s q60.039s0.036s1m30.924s q70.164s0.038s10.235s Table 4.1: Table depicting the mean response time of our tool to each of the seven contrastive questions in the car and the generator-events domain. Q. No. is the contrastive question from the set of questions in Section 4.1. The columns SMTPlan+ and ENHSP show the time taken by these planners to generate a plan. The column HModel Compilation Time shows the time taken to construct the Hmodel, and Best Contrastive Plan Time reports the time to find the best contrastive plan by iterative re-modeling with Bisection. NP denotes no plan produced within a given time-bound. NA denotes not applicable. 4.3 Related Works Contrastive explanation: (Miller [2021]) extends the definition of explanation using structural causal models to contrastive explanation. They distinguish contrastive explan- ations for two categories of questions: counterfactual questions like "why P rather than Q?" and bi-factual questions, which are of the form "why P but Q?" The work asserts that contrastive explanations are integral to understanding how people seek explanations, as evidenced by research in philosophy and social science. Specifically, the preference for contrastive questions like "Why P rather than Q?" over simple questions like "Why 99 CHAPTER 4. A CONTRASTIVE EXPLANATION TOOL FOR PLANS IN HYBRID DOMAINS P?" highlights the importance of considering alternatives when seeking explanations. To produce contrastive explanations for differentiable models like deep neural networks, a method called the contrastive explanations method (CEM) is suggested by (Dhurandhar et al. [2018]). Iterative modeling: In (Smith [2021]), the authors propose explainable planning as an iterative process allowing a user to refine the hypothetical model and iterate the process by posing new questions. This permits the user to set extra constraints that can be incorporated into the hypothetical model if the explanation remains unsatisfactory. XAIP as a service: In (Cashmore et al. [2019]), the authors propose a prototype framework for explainable planning as a service which can be used with any planner capable of reasoning with PDDL2.1. They present a console-based and a GUI-based interface, allowing users to view the plan and select a formal query from a list of questions. The system produces a visual representation of the original plan and a hypothetical Plan. This work extends such a tool interface to the hybrid planning domain. It implements the algorithms presented in (Sarwar et al. [2022]). 4.4 Conclusion In this work, we present an interactive web interface for contrastive explanation of plans that implements the presented algorithms. This tool provides a way to experiment with different planning domains in hybrid systems and to plug in different hybrid system plan- ners as plan generation engines. It explores a set of contrastive questions that a user of a planning tool may raise, and provides an interface for contrastive explanations of such questions. It also provides provisions for finding a competitive contrastive plan based on the comparison metrics (e.g., plan makespan, plan length) that the underlying planner can provide. We demonstrate experimental results on two planning domains using two state-of-the-art planners. 100 Chapter 5 Explaining Unsolvability of Planning Problems in Hybrid Systems with Model Reconciliation Chapter Abstract: A recent problem of interest in Explainable AI Planning is that of ex- plaining the unsolvability of planning problems. Though there has been a lot of research on generating explanations of solutions to planning problems, explaining the absence of solu- tions remains a largely open and understudied problem. Model reconciliation has been a popular approach for generating explanations for such problems in recent literature, which involves an AI agent and a human planner, who have different models of the planning domain, and each explains to the other the differences they have in their domain repres- entations and attempts to arrive at a consensus. More often than not, it is assumed that the AI agent has a correct and complete view of the domain, of which the human only has a partial view. Through reconciliation, the human domain is updated to be consistent with what the AI agent has. Most of the works in this direction are targeted toward classical planning problems on domains represented typically as discrete state transition systems or variants. In this chapter, we provide an approach towards model reconciliation for plan- ning problems in hybrid systems represented as a mix of discrete and continuous domains. We assume that the agent has a complete model of the environment, while the human has a partial or erroneous model and expects a plan for the planning problem when there is none. The explanation problem is presented as a process of continuous reconciliation between these two entities (agent and human) to make the human domain consistent with that of the agent. To this effect, we use a mix of graph traversal and path analysis, along with Linear programming to carry out the reconciliation process. In particular, we use the concept of Irreducible Infeasible Sets (IIS) to generate explanations. Experimental results on 2 representative hybrid domains show the efficacy of our approach. 101 CHAPTER 5. EXPLAINING UNSOLVABILITY OF PLANNING PROBLEMS IN HYBRID SYSTEMS WITH MODEL RECONCILIATION E xplainable planning is an active area of research in recent times, given the increasing number of application areas in which humans and autonomous agents collaborate. In such application areas, cooperative plans derived with mutual trust and understanding are important for achieving a desired objective. With automated planning being applied in safety-critical systems, the need for explanation and trust in the agent’s behaviour has become ever more important to a human user. The ability to explain the rationale behind a decision of an autonomous agent is widely regarded as one of the precursors needed for humans to engage in trustworthy collaborations with autonomous agents. While there has been a lot of research on generating explanations to planning problems, most of the earlier works in explanation generation have focused on explaining why a given plan or action was chosen. However, explaining the unsolvability of a planning problem remains a largely open and understudied problem. In the context of unsolvability explanation, recent works have looked at generating certificates or proofs of unsolvability (Eriksson et al. [2017, 2018]). Finding counterfactual alterations to the original planning task such that the new planning task can be made solvable, i.e, an excuse for the unsolvability of the planning problem has been proposed in (Göbelbecker et al. [2010]). Such certificates or proofs of unsolvability gear towards automatic verification rather than explaining unsolvability of planning problems. However, the excuses generated by altering the planning task may not always be adequate to explain unsolvability in complex planning domains. In most human-AI interaction scenarios for a given planning task on which human-agent collaboration is sought, the human may have a preconceived domain that may differ from the actual domain known to the agent. In such scenarios, the Model Reconciliation Problem (MRP) (Chakraborti et al. [2017]) has been a popular approach to explain the agent’s domain knowledge to the human where reconciliation between these two models is done by making the models equivalent such that a plan in one entails a plan in the other and vice-versa. To do so, MRP attempts to provide an explanation that can be used to update the human model such that the agent’s behaviour is also amenable to the human user. However, to the best of our knowledge, most of the works in MRP literature have been applied to planning problems in discrete domains. In this chapter, we present a model reconciliation framework for explaining unsolv- ability of planning problems in hybrid domains that exhibit an interplay of discrete and continuous dynamics. In such a domain, a feasible plan has to satisfy not only the discrete dynamics but also the constraints imposed by the presence of the continuous dynamics of the domain. This makes the MRP problem more involved as well, since the reconciliation involves concurrence on both the discrete and continuous behaviors concerning the given planning task. In our setting, we assume that the planning problem is solvable in the human model but unsolvable in the agent model. In other words, a feasible plan exists in the human model while no such plan is entailed by the agent’s model of the domain. We explain the unsolvability of the planning problem through a reconciliation process between the human and agent models. In other words, we examine the concurrence of each plan 102 6 123 45 78 9 10 1112 13 14 15 1617 18 19 2021 22 23 24 8 9 1011 12 13 14 1516 17 18 24 232221 20 19 12 34 5 6 7 Robot's starting location Charging Station Blocked cell Goal location Oil spilled area with higher charge depletion Plan Human modelAgent model Figure 5.1: The human and the agent view of the warehouse environment is depicted. Each cell is associated with an identifier shown in the top-left corner of the cell. The Blue dotted line represents an expected plan by a human under its knowledge of the warehouse. Our explanation algorithm examines this plan in the agents’ view of the world and detects the presence of an obstacle in cell number 20. This knowledge is reported to the user as an explanation of the infeasibility of the plan. The Green dashed line represents a candidate plan in the human model when abstracting its continuous dynamics. This plan is refuted by our algorithm when the feasibility of the path is checked against the continuous dynamics of the human model. that the human model entails with its counterpart in the agent domain. To do so, we have a three-step procedure. In the first step, we examine the feasibility of a plan obtained by restricting only to the discrete dynamics of the human model while abstracting away the continuous one. If this fails, we provide an explanation of the plan’s infeasibility by high- lighting the difference in the discrete dynamics between the human and the agent model. We then continue examining the next plan for further reconciliations. On the other hand, where the plan is feasible in the agent model, we proceed to the next step, where we rein- troduce the continuous dynamics together with the discrete one in the human model and re-examine the feasibility of the plan in the human model. When the plan is infeasible in the human model itself, it is discarded as far as reconciliation is concerned, being infeasible in both models and thus causing no conflict. On the other hand, if the plan is feasible, we proceed to the last step, where we check the feasibility of the same plan in the agent model. Note that the motivation behind the stratification of the human model based on its discrete and discrete-continuous dynamics is to efficiently reconcile the discrete dynamics with the agent model without needlessly spending planning effort in the entire complex model. As an outcome of the last step, the plan turns out to be infeasible, though feasible in the human model, since there are no feasible solutions to the planning problem at hand. In this case, the explanation is produced using the concept of irreducible infeasible sets (IIS) (Chinneck and Dravnieks [1991]), and its application to bounded model checking of linear hybrid automata (Fränzle and Herde [2004], Bu et al. [2008]). For this, we construct a linear constraint set from the plan using linear programming (LP) such that the satis- fiability of this constraint set implies the existence of a plan in the agent model. We then extract the IIS from the encoding using an underlying LP solver. This continues until 103 CHAPTER 5. EXPLAINING UNSOLVABILITY OF PLANNING PROBLEMS IN HYBRID SYSTEMS WITH MODEL RECONCILIATION we exhaustively enumerate all plans that the human can find in its model and the agent rules out each, either in the discrete setup or considering the continuous dynamics. The final explanation is a summary obtained in the three steps above. We demonstrate our approach on a warehouse automation and a water-level monitoring system model. The average time for model reconciliation is 1387.31 seconds, where we consider all the possible plans for a planning problem. The average time to generate explanations of unsolvability of a plan is 0.2 seconds. This shows that our algorithm can quickly explain the causes of unsolvability. The stratification of the model dynamics and techniques to focus only on relevant dynamics of the model while searching for explanations expedite the explan- ation generation process. For instance, in the warehouse automation domain, 43.95% of the plans are discarded for searching explanations. In the water-level monitoring domain, 72.22% of plans are similarly pruned from generating any explanation. In summary, the contributions of this work are as follows. • We propose a path-based continuous model reconciliation framework for explaining unsolvability of planning problems for hybrid systems. The explanation problem is formulated as a continuous reconciliation process between the agent and the human models to make the human knowledge base consistent with the agent. • We show that a discrete and continuous path analysis approach can be leveraged to derive explanations of the unsolvability of a planning problem in hybrid domains. To this effect, we use a mix of graph traversal and path analysis, along with Linear programming to carry out the reconciliation process. • The discrete path analysis falsifies a path by mapping each location and transition of the path in the human model to the agent model by a simple graph traversal. The continuous path analysis leverages reachability analysis, combining with minimal inconsistent constraint sets to find the infeasible path-segments for which the paths become unsatisfiable. • To illustrate our explanation framework, we demonstrate a warehouse automation system as an example planning domain, along with a water-level monitoring system, and present results for problem instances on each domain with varying plan depth. The rest of this chapter is organized as follows. In Section 5.1, we sketch a motivating example on which we demonstrate this work. Section 5.2 provides an overview of the prob- lem statement. Section 5.3 illustrates our methodology and the framework of explanation. Section 5.4 discusses implementation and results. Section 5.5 presents related literature. Finally, Section 5.6 summarizes the contributions and findings of this work and discusses possible future directions. 5.1 Motivating Example In this section, we present an MRP scenario in the context of warehouse automation where a robot operates to manage the inventories of the warehouse. The warehouse is divided 104 5.1. MOTIVATING EXAMPLE into cells. The discrete dynamics here capture the connectivity of the cells along with the presence of objects in certain cells, which are interpreted as obstacles through which the robot cannot move. The movement of the robot is restricted to one of its adjacent cells, and movement to diagonal cells is prohibited. The continuous dynamics capture the battery charge depletion rate of the robot within a cell. Within each cell, the robot follows the dynamics particular to that cell. When the robot makes a transition from one cell to another, it starts to follow the dynamics of the new cell instantaneously. The robot is assigned the task of carrying a consignment to a designated cell while the number of cell visits is restricted to ≤ D cells. The robot starts from the yellow-colored cell where the planning problem requires it to transport the black box to the goal cell (red-colored cell). The robot depletes its charge according to the cell dynamics while on the move. There is a charging station shown as a green-colored cell. The robot may visit this cell to recharge its battery. The orange-colored cells represent the oil-spilled areas where the robot depletes battery charge at a higher rate due to the slippery floor condition. The grey-colored cells are blocked with obstacles. The robot is equipped with a rechargeable battery. The initial battery charge is 10 units. Each cell has a charge depletion rate of 2 units (modelling the continuous dynamics) except the orange cells, for which the corresponding value is 4. The robot has complete knowledge of the warehouse, whereas the human has only been exposed to a partial view. In Figure 5.1, we have shown a representation of the human and the agent’s view of the warehouse. Note that some of the obstacles in the warehouse are unknown to the human. Additionally, the human is unaware of the oil-spilled areas of the warehouse as well and is thereby not aware of the higher charge depletion rates in those cells. Interestingly, there are a few candidate plans that are obtained when abstracting out the continuous dynamics of the human model and are detected as infeasible in the human model itself during the feasibility analysis with its continuous dynamics. In particular, the plans that avoid a visit to the recharge station are infeasible due to the battery drainage before reaching the goal, given that the charge depletion rate is at-least 2 units in each cell and the Manhattan distance (the distance between two points on a grid is the sum of the vertical and horizontal distances between them) to the destination for all such plans is ≥ 6. One such infeasible plan is shown as a Green dashed line in the figure, whose Manhattan distance to the destination is 6. Henceforth, the human would expect any plan via the recharge station as a solution to the planning task. The Blue dotted line in the figure represents such a plan. However, this plan goes through cell number 20, having an obstacle in the warehouse that the human is unaware of. When examining this plan in the agent’s model of the warehouse, our explanation algorithm detects the presence of an obstacle in this cell and reports this to the human as an explanation of the plan’s infeasibility. This knowledge transfer via an explanation results in a modified human view of the warehouse. As a result of this knowledge, the human would not enumerate any plan via this obstacle. Now, an alternate plan that the human may expect is shown as the Blue dotted line in Figure 5.2. When this plan is examined by our algorithm, it is found to be feasible in the agent’s world when the continuous dynamics are abstracted in the first step. This is going to prompt our algorithm to now analyze the feasibility 105 CHAPTER 5. EXPLAINING UNSOLVABILITY OF PLANNING PROBLEMS IN HYBRID SYSTEMS WITH MODEL RECONCILIATION 6 123 45 7 89101112 13 1415 16 17 18 192021 22 2324 8 9 1011 12 13 14 1516 17 18 24 232221 20 19 1 23 45 6 7 Human modelAgent model Figure 5.2: Infeasibility of the plan explained by continuous dynamics, particularly the higher charge depletion rate at orange cells of this plan by considering the continuous dynamics as well. This is when our algorithm is going to detect that the plan goes via the cells with an oil-spill, and therefore, due to the higher battery depletion rate here (4 units instead of 2), the robot’s battery drains completely before reaching the charging station. Consequently, our algorithm concludes the infeasibility of this plan. Similarly, the remaining plans in the human’s world would also be detected as infeasible due to the battery drainage before reaching the charging station. The explanation will include the orange cell with oil-spill in the IIS to indicate it as a potential cause of the unsolvability in the agent’s model. In the following section, we formally describe the MRP problem and the domain rep- resentation. We represent the agent model and the human model as hybrid automata HA A and HA H , respectively (Alur et al. [1995a]). Each cell of the domain is represented as a location in both the automata models HA A and HA H , preserving the dynamics of that cell. 5.2 Problem Overview In the planning context, a transition e ∈ Edge in HA (see Defination 2.1.1) depicts an action act of the domain. The name of the action is the transition’s label e.a, the pre- condition is the transition’s guard e.g, and the post-condition is the transition’s reset e.r. The Init of the automaton defines the initial condition of the planning problem. The change in the discrete state of the domain due to the action is captured through the transition’s source and target location e.l and e.l ′ , respectively. A state of a HA is a pair (l,v) consisting of a location l ∈ Loc and a valuation v ∈ V . l denotes the discrete state, whereas v denotes the continuous state of the system. A state can change either due to an application of an action or due to the passage of time following the continuous flow dynamics. The continuous state change is given by flow l (v,t), which is the solution of the differential equation Flow(l). We refer to the former as action-transition and the later as timed-transition. A planning problem for a hybrid system consists of a hybrid automaton representation of the planning domain and a problem description, which is defined as: Definition 5.2.1 A planning problem Π for a hybrid system HA is a tuple (Dom, Prob, 106 5.2. PROBLEM OVERVIEW Depth), where • Dom is represented as a hybrid automation HA defining the planning domain (Sar- war et al. [2023b]). • Prob is a tuple⟨Init,Goal⟩ representing a problem description, where Init and Goal define the initial and the goal states. Init is a tuple ⟨l init ,S init ⟩ such that l init ∈ Loc and S init ⊆ V and Goal is a tuple ⟨l goal ,S goal ⟩ such that l goal ∈ Loc and S goal ⊆ V . • Depth D defines the bound on the length of a plan.2 Definition 5.2.2 A plan φ is a tuple ⟨λ,makespan⟩. For a planning problem with a set of actions A, λ is a finite set of pairs ⟨t,act⟩ together with the plan makespan ∈ R. In the pair, t ∈ R + is the time instance of executing the action act ∈ A. The makespan is the overall duration of the plan.2 We refer to the length of the plan to be the number of actions appearing in λ. The model reconciliation problem (MRP) is defined below. Definition 5.2.3 Given a human model HA H , an agent model HA A , a Prob for which a plan φ exists for (HA H ,Prob) but no plan exists for (HA A ,Prob) within a plan of length D, the model reconciliation problem is to provide an updated model HA ′ H to the human such that φ does not exist for (HA ′ H ,Prob).2 Inv: Flow: Flow: Inv: Inv: Flow:Flow:Flow: Flow: Flow: Flow: Flow:Flow:Flow: Flow:Flow: Inv: Inv:Inv:Inv: Inv: Inv: Inv:Inv: Inv:Inv: Figure 5.3: The HA model for the warehouse automation problem, where locations represent the corresponding cells shown in Figure 5.1. The automaton provided here is a part of the warehouse automation domain. In our setup, as discussed earlier, we assume both HA H and HA A are represented as hybrid automata. Figure 5.3 shows the hybrid automata for the agent and human models for the problem in Figure 5.1. The differences between the human and the agent models are depicted in red. Location 20 (l20) represents the blocked cell 20 in the agent model, which the human assumes is free. The transitions, which are shown in red, are present in the human model but not in the agent model. The dynamics that are given in red represent the human’s erroneous assumptions. The complete automaton could not be included as part of Figure 5.3 due to space constraints. 107 CHAPTER 5. EXPLAINING UNSOLVABILITY OF PLANNING PROBLEMS IN HYBRID SYSTEMS WITH MODEL RECONCILIATION 5.3 Our Reconciliation Methodology In this section, we discuss the details of the reconciliation process. We assume that the human model and the agent model may differ on two points (a) the human may not have information about the inaccessibility of some location, and (b) the human may not have information about the correct continuous dynamics. In our setup, we consider an iterative MRP process. In each iteration, the human produces a plan in HA H , the agent provides an explanation to refute the existence of that plan in HA A , following which the human updates its HA H and searches for another plan. This continues till no plan exists in the human model. To expedite this plan search-validate-reconcile process, the human first creates an abstraction of the hybrid domain HA H by abstracting out the continuous dynamics and keeping only the abstract graph structure G H consisting of the automaton locations and transitions underlyingHA H . The initial location l init and goal location l goal of the planning problem Prob are respectively marked as the initial and goal locations in G H , and a path p H reaching the goal from the initial location within the specified length is extracted from G H . The path p H serves as a candidate plan when only the discrete dynamics are considered; however, it may or may not be feasible when the continuous dynamics are brought in. For the path p H , the agent model is consulted to check if the same path can be reproduced from the corresponding initial to the goal location in the graph structure G A underlying the agent domain automaton HA A . If not, the first edge that appears in the human plan but not in the agent is marked as an explanation, and also deleted from G H such that no further plan involving this edge is generated from G H . Reconciliation then proceeds with the next initial-to-goal path in G H . However, if p H can be reproduced in the agent graph G A (considering only the locations and transitions), the human checks to see if the continuous dynamics involved on p H (reconstructed by bringing in the location dynamics, invariants, flow, edge guards) entails a feasible solution in HA H . If not, reconciliation again proceeds with the next path in G H . However, if p H is feasible inHA H , it is passed on for validation in the agent modelHA A , considering the full discrete + continuous setting. Evidently, since no plan for Prob exists in HA A , the feasibility check for p H turns out to be negative, and the first location that is not able to meet the constraints is produced as an explanation. This continues till no further initial to goal paths can be generated from G H . Our method has 3 key steps as outlined below. • Discrete path generation and reconciliation • Path feasibility considering continuous dynamics • Explanation and human model update We discuss each in detail below. The overall architecture of our explanation framework is shown in Figure 5.4. 5.3.1 Discrete Path Generation and Reconciliation The objective of this step is to examine whether a path in the human model is also valid in the agent model, considering only the discrete dynamics. The main motivation for this 108 5.3. OUR RECONCILIATION METHODOLOGY INPUT: MRP Problem (HA A , HA H , Prob, D) Start G A = Compute_Graph (HA A ), G H = Compute_Graph (HA H ), E = null, S = null, IP = null; p H = Compute_Path (G H , D) Discrete_Feasibility (G A , p H ) Path replayed ? Compute graph structure of the human and agent models Compute a path in the human graph structure G H Replay p H in the agent graph structure G A C H _Feasibility (HA H , p H ) C A _Infeasibility (HA A , p A ), Find IIS; Yes Yes No No OUTPUT: Return E, S; End Return invalid-edge e Path feasibility analysis with continuous dynamics in the human model Path feasibility analysis with continuous dynamics in the agent model to find IIS Return infeasible path Return IIS path-segments Path feasible ? Figure 5.4: The flow diagram of our model reconciliation framework step is to quickly rule out invalid plans in the human model that are due to an incorrect understanding of the locations and transitions of the planning domain, without getting into the complex continuous dynamics. To this effect, we first define the concept of an abstract graph corresponding to a hybrid automaton obtained by abstracting out the continuous dynamics and considering only the location-location edge relationship. Definition 5.3.1 For a hybrid automaton HA = Loc, Var, Flow, Init, Lab, Edge, Inv , the graph of the automaton is defined as G HA = (Loc, E) where E ⊆ Loc× Loc such that for every (l, a, g, r, l ′ ) ∈ Edge, there is an edge (l, l ′ ) ∈ E.2 As the first step, the abstract graphs G A and G H corresponding to HA A and HA H re- spectively are obtained. Our approach then proceeds to extract a path p H from G H . In this graph, a path is a plan from the initial location to the goal location in the domain Dom specified in the planning problem Π. We now define the structure of a path. Definition 5.3.2 A path p in a graph structureG of a hybrid automaton HA is a sequence of vertices and edges l 0 e 0 −→ l 1 e 1 −→ l 2 e 2 −→ ... e n−1 −→ l N where l i ∈ Loc, e i ∈ Edge, l i = e i .l, l i+1 = e i .l ′ .2 For reconciliation, we proceed to check if p H is a valid or spurious path inG A . An important assumption we make here is that the location namespace of G A and G H match, in other words, for each location appearing on p H , we can uniquely identify a corresponding location in G A . Algorithm 5.1 presents the pseudo-code for discrete feasibility analysis of a path, which takes p H and G A as inputs. It checks for the existence of a path in G A , which is a counterpart of p H . If such a path exists, we say p H is feasible in G A . If p H is not feasible, the algorithm returns the first invalid-edge detected in p H not present in G A . We add the invalid-edge to the explanation-edge set E, which we define as follows: Definition 5.3.3 An invalid-edge invalid e is the first edge along a path p H which is present in G H but not in G A . invalid e corresponds to a transition e ∈ Edge in HA H and not present in HA A . 109 CHAPTER 5. EXPLAINING UNSOLVABILITY OF PLANNING PROBLEMS IN HYBRID SYSTEMS WITH MODEL RECONCILIATION Definition 5.3.4 The explanation-edge set E is the collection of invalid-edges along dif- ferent paths of the human graph structure G H which are not feasible in the agent graph structure G A . Algorithm 5.1: discrete_feasibility(p H , G A ) Input: A path p H from G H , and agent’s graph G A . Output: The first invalid-edge of p H if spurious, True otherwise 1 begin 2for i = 0 to n− 1 do 3if (e i .l == l i ) & (e i .l ′ == l i+1 ), where e i is the i-th edge on p H , and l i , l i+1 ∈ Loc of G A . then 4continue; 5else 6Add e i to E; 7return e i ; 8end 9end 10return True 11 end Intuitively, we attempt to reconstruct the entire location-edge-location sequence in p H starting from the initial to the goal in the agent graph G A . If this succeeds, we conclude p H is valid, spurious otherwise. In the latter case, we identify the first edge e on p H that is invalid, i.e., does not have an existence in G A . We mark e as invalid and delete it from G H so that further paths generated from G H cannot include the spurious edge e. Example 5.3.1.1 Consider Figure 5.1 in which the blue dotted line represents a plan with the following sequence of locations and edges l 7 -e 7 13 -l 13 -e 13 19 -l 19 -e 19 20 -l 20 -e 20 21 -l 21 -e 21 22 -l 22 - e 22 23 -l 23 -e 23 17 -l 17 -e 17 18 -l 18 . In the human model, the path goes through locations 19 and 20 as the human erroneously assumes location 20 is a free cell. Therefore, the transition from location 19 to location 20 is valid in the human model. However, the agent knows that location 20 is blocked. Thus, there is no such transition in the agent model, which is depicted by the cross-mark on the path there in the figure. Hence, when we consider discrete feasibility analysis of the path from the human abstract graph in the agent abstract graph, this transition e 19 20 from location 19 to location 20 is identified as an invalid-edge. 2 The path generation and validation step is carried out first on the abstract graphs to expedite the reconciliation process. Paths that appear as candidate plans in the human model due to his lack of awareness of the complete location structure can be simply ruled out without considering the complex continuous dynamics, when they are attempted for reconstruction on G A . Also, every path generated from G H does not correspond to a valid path in HA H . We now proceed with the next validation step with the continuous dynamics. 5.3.2 Path Feasibility in the Continuous Dynamics Once a path p H obtained from G H is identified as feasible in the abstract graph G A , we proceed to check if the same is feasible inHA H in the presence of the continuous dynamics. 110 5.3. OUR RECONCILIATION METHODOLOGY Algorithm 5.2 presents the pseudo-code for feasibility checking, which takes the path p H and the model HA H as inputs. If p H is infeasible in HA H itself, we mark it as infeasible and add it to the infeasible-path set IP. Any infeasible path and all its extensions are discarded to ensure that further paths are not generated involving such infeasible paths as sub-paths. This reduces the number of paths generated in our approach. However, if p H is feasible in HA H , it corresponds to a plan φ H in the human model. The first step in our algorithm encodes the constraints associated with a path arising due to the continuous dynamics, while the second Solve step invokes a solver to check if the constraints encoding the path entail a feasible solution. The encoding is such that a solution to the constraints implies the existence of a run of the system that leads to the goal state from an initial state. The existence of a run in turn implies the existence of a plan to solve the planning problem. A path p H abstracts zero, one, or many runs in HA H , mainly due to the non- determinism that it embodies with respect to the dwell time at a location. Corresponding to a path p H in G H , we first define the notion of a corresponding run in HA H . Algorithm 5.2: c H _feasibility(p H , HA H ). Input: A path p H , human model HA H . Output: Return true or false. 1 begin 2 c H = Get constraints of p H in HA H ; 3 result = Solve(c H ) ;/* Use BACH */ 4if result == SAT then 5return true ;/* When p H is feasible */ 6else 7add p H to IP; 8return false ;/* When p H is infeasible */ 9end 10 end Definition 5.3.5 A run corresponding to a path l 0 e 0 −→ l 1 e 1 −→ l 2 e 2 −→ ... e n−1 −→ l N is an alternating sequence of timed and action transitions of the hybrid automaton depicted as: (l 0 ,v 0 ) τ 0 −→ (l 0 ,v ′ 0 ) e 0 −→ (l 1 ,v 1 ) τ 1 −→ (l 1 ,v ′ 1 ) e 1 −→ ... e n−1 −→ (l n ,v n ) τ n −→ (l n ,v ′ n ) where (i) v 0 ∈ S, (i) e i ∈ Edge such that l i = e i .l, l i+1 = e i .l ′ , v ′ i ∈ e i .g, v i+1 = e i .r(v i ), ∀i ∈ 0,...,n− 1. (i) ∀t ∈ [0,τ i ], flow l i (v i ,t) ∈ Inv(l i ), ∀i ∈ 0,...,n. (i) flow l i (v i ,τ i ) = v ′ i ; (iv) v i+1 = e i .r(v i ). (vi) (l n ,v ′ n ) satisfies goal condition G.2 It can be seen that the existence of a run corresponding to a path also implies the existence of a feasible plan for the planning problem which can be obtained by extracting the actions associated with the transitions e i together with the time of taking the transitions τ i as the action, time pairs of a valid plan. We now elaborate on the encoding of a path as constraints in detail below. Constraint Encoding of a Path We encode a path as a set of linear constraints. Thus, the problem of checking the reachability of the location l n (representing the terminal location on a path) and thus 111 CHAPTER 5. EXPLAINING UNSOLVABILITY OF PLANNING PROBLEMS IN HYBRID SYSTEMS WITH MODEL RECONCILIATION validating the existence of a plan is reduced to a linear program encoded as a set of path constraints as discussed in (Bu et al. [2008]). This linear encoding is illustrated with a simple example. Example 5.3.2.1 Consider Figure 5.3 which shows the hybrid automaton modeling of a part of the warehouse. Consider the path p in the agent model ⟨ init −→ l 7 e 7 8 −→ l 8 e 8 9 −→ l 9 e 9 10 −→ l 10 e 10 16 −→ l 16 e 16 22 −→ l 22 e 22 23 −→ l 23 e 23 17 −→ l 17 e 17 18 −→ l 18 ⟩. The linear encoding of the path as constraints is defined below: 1. A variable t i is associated with each location of the path, representing the timed transition in the respective location. For each t i (0≤ i≤ 8), we add non-negativity constraints t i ≥ 0 in the linear program. 2. For flow conditions in every location along the path, we generate the corresponding flow constraints. For instance, given the flow condition −1 ≤ ̇x ≤ 1 in location l 7 , we generate a constraint x out l 7 = x in l 7 +t 0 ∗ ̇x, where x in l 7 represents the value of x when a run enters location l 7 and x out l 7 represents the value of x when a run is to transition to l 8 after having stayed at location l 7 for t 0 units of time. 3. For location invariants in every location along the path, we generate the invariant constraints. For instance, the invariant 0 ≤ x ≤ 1 in location l 7 is encoded as two constraints: 0≤ x in l 7 ≤ 1, and 0≤ x out l 7 ≤ 1. 4. For reset on an edge along the path, we generate corresponding reset constraints. For instance, for x = 0.5 on edge init, we get x in l 7 = 0.5. 5. For guard conditions along each edge in the path, we generate the corresponding con- straints. For instance, the guard x == 1 in edge e 7 8 is represented as the constraint x out l 7 = 1. Conjunction of all the constraints constitutes the path formula, which we check for satis- fiability. Below are the location-wise constraints for the path in the example. Location l 7 : x in l 7 = 0.5, y in l 7 = 1.5, c in l 7 = 10, t 0 ≥ 0, x out l 7 = x in l 7 + t 0 ∗ ̇x, y out l 7 = y in l 7 + t 0 ∗ ̇y, x out l 7 = 1, c out l 7 = c in l 7 + (−2)∗ t 0 , 0≤ x in l 7 ≤ 1, 0≤ x out l 7 ≤ 1, 1≤ y in l 7 ≤ 2, 1≤ y out l 7 ≤ 2, c in l 7 ≥ 0.1, c out l 7 ≥ 0.1; Location l 8 : x in l 8 = x out l 7 , y in l 8 = y out l 7 , c in l 8 = c out l 7 , t 1 ≥ 0, x out l 8 = x in l 8 + t 1 ∗ ̇x, y out l 8 = y in l 8 + t 1 ∗ ̇y, x out l 8 = 2, c out l 8 = c in l 8 + (−2)∗ t 1 , 1≤ x in l 8 ≤ 2, 1≤ x out l 8 ≤ 2, 1≤ y in l 8 ≤ 2, 1≤ y out l 8 ≤ 2, c in l 8 ≥ 0.1, c out l 8 ≥ 0.1; Location l 9 : x in l 9 = x out l 8 , y in l 9 = y out l 8 , c in l 9 = c out l 8 , t 2 ≥ 0, x out l 9 = x in l 9 + t 2 ∗ ̇x, y out l 9 = y in l 9 + t 2 ∗ ̇y, x out l 9 = 3, c out l 9 = c in l 9 + (−2)∗ t 2 , 2≤ x in l 9 ≤ 3, 2≤ x out l 9 ≤ 3, 1≤ y in l 9 ≤ 2, 1≤ y out l 9 ≤ 2, c in l 9 ≥ 0.1, c out l 9 ≥ 0.1; 112 5.3. OUR RECONCILIATION METHODOLOGY Location l 10 : x in l 10 = x out l 9 , y in l 10 = y out l 9 , c in l 10 = c out l 9 , t 3 ≥ 0, x out l 10 = x in l 10 + t 3 ∗ ̇x, y out l 10 = y in l 10 + t 3 ∗ ̇y, y out l 10 = 2, c out l 10 = c in l 10 + (−4)∗ t 3 , 3≤ x in l 10 ≤ 4, 3≤ x out l 10 ≤ 4, 1≤ y in l 10 ≤ 2, 1≤ y out l 10 ≤ 2, c in l 10 ≥ 0.1, c out l 10 ≥ 0.1; Location l 16 : x in l 16 = x out l 10 , y in l 16 = y out l 10 , c in l 16 = c out l 10 , t 4 ≥ 0, x out l 16 = x in l 16 + t 4 ∗ ̇x, y out l 16 = y in l 16 + t 4 ∗ ̇y, y out l 16 = 3, c out l 16 = c in l 16 + (−4)∗ t 4 , 3≤ x in l 16 ≤ 4, 3≤ x out l 16 ≤ 4, 2≤ y in l 16 ≤ 3, 2≤ y out l 16 ≤ 3, c in l 16 ≥ 0.1, c out l 16 ≥ 0.1; Location l 22 : x in l 22 = x out l 16 , y in l 22 = y out l 16 , c in l 22 = c out l 16 , t 5 ≥ 0, x out l 22 = x in l 22 + t 5 ∗ ̇x, y out l 22 = y in l 22 + t 5 ∗ ̇y, x out l 22 = 4, c out l 22 = c in l 22 + t 5 ∗ 0, 3≤ x in l 22 ≤ 4, 3≤ x out l 22 ≤ 4, 3≤ y in l 22 ≤ 4, 3≤ y out l 22 ≤ 4, c in l 22 ≥ 0.1, c out l 22 ≥ 0.1; Location l 23 : x in l 23 = x out l 22 , y in l 23 = y out l 22 , c in l 23 = 10, t 6 ≥ 0, x out l 23 = x in l 23 + t 6 ∗ ̇x, y out l 23 = y in l 23 + t 6 ∗ ̇y, c out l 23 = c in l 23 + (−2)∗ t 6 , 4≤ x in l 23 ≤ 5, 4≤ x out l 23 ≤ 5, 3≤ y in l 23 ≤ 4, 3≤ y out l 23 ≤ 4, c in l 23 ≥ 0.1, c out l 23 ≥ 0.1, y out l 23 = 3; Location l 17 : x in l 17 = x out l 23 , y in l 17 = y out l 23 , c in l 17 = c out l 23 , t 7 ≥ 0, x out l 17 = x in l 17 + t 7 ∗ ̇x, y out l 17 = y in l 17 + t 7 ∗ ̇y, c out l 17 = c in l 17 + (−2)∗ t 7 , 4≤ x in l 17 ≤ 5, 4≤ x out l 17 ≤ 5, 2≤ y in l 17 ≤ 3, 2≤ y out l 17 ≤ 3, c in l 17 ≥ 0.1, c out l 17 ≥ 0.1, x out l 17 = 5; Location l 18 : x in l 18 = x out l 17 , y in l 18 = y out l 17 , c in l 18 = c out l 17 , t 8 ≥ 0, 5≤ x in l 18 ≤ 6, 2≤ y in l 18 ≤ 3, c in l 18 ≥ 0.1, Path Feasibility Analysis in the Human model For a feasible path p H obtained from G H , we check if a feasible run can be obtained by solving the path constraints for satisfiability. This is then solved by a Linear Programming (LP) tool (Li et al. [2006]). If the set of constraints is satisfiable (SAT), a run exists which implies that the path p H has a plan. The result of executing this plan from the initial state Init leads to the goal state Goal, and satisfies the goal condition, along with each location dynamics, beginning from the initial location, as specified in the problem Prob. Plan Infeasibility Analysis in the Agent model Once a plan in the human model is obtained corresponding to p H , we proceed to analyze it in the agent model HA A using a similar constraint encoding strategy as shown above using the corresponding path p A obtained from G A . Evidently, since Prob is unsolvable in the agent model, no valid run corresponding to p A can be constructed in HA A . Thus, the Solve step as done in Step 3 of Algorithm 5.3 returns unsatisfiable and we proceed to extract the IIS (Chinneck and Dravnieks [1991]) to find the set of infeasible constraints that are collectively unsatisfiable for the given linear program encoding the run in the agent model. IIS can be defined as below: Definition 5.3.6 IIS (Chinneck and Dravnieks [1991]): An irreducible infeasible set (IIS) for a set of path constraints is a minimal set of inconsistent constraints.2 113 CHAPTER 5. EXPLAINING UNSOLVABILITY OF PLANNING PROBLEMS IN HYBRID SYSTEMS WITH MODEL RECONCILIATION Many software packages are available that support the efficient analysis of a linear con- straint set and locating of the IIS, such as MINOS (Chinneck [1994]), IBM CPLEX (Con- tributors [Last accessed on Nov 2025]), and LINDO (INC. [Last accessed on Nov 2025]). Now coming back to our discussion on the path constraints, the IIS of Example 5.3.2.1 is of the path c in l 10 = c out l 9 , c out l 10 = c in l 10 + (−4) ∗ t 3 , c in l 10 ≥ 0.1, c out l 10 ≥ 0.1, c in l 16 = c out l 10 , c out l 16 = c in l 16 + (−4)∗t 4 , c in l 16 ≥ 0.1, c out l 16 ≥ 0.1, c in l 22 = c out l 16 , c in l 22 ≥ 0.1, c out l 22 ≥ 0.1 This IIS can be mapped back to the original elements in the path as illustrated in (Bu et al. [2011]) to find the path-segment for which the whole path becomes infeasible. For example, the IIS constraints set of the above can be mapped to the path-segment l 10 e 10 16 −→ l 16 e 16 22 −→ l 22 , which is actually infusing inconsistency for the whole path p to become infeasible. We denote these path-segments as IIS path-segments. Definition 5.3.7 IIS path-segment: An IIS path-segment ips is a segment of a path p for which p becomes infeasible as the underlying set of constraints of ips is inconsistent.2 The IIS path-segment p ′ for our example path p is l 10 e 10 16 −→ l 16 e 16 22 −→ l 22 . Further, this path segment can be used for pruning paths in the human model in our algorithm. This is because any path p ′ that contains the path segment p ′ will certainly be infeasible (Bu et al. [2011]) in the agent model, and the explanation of the infeasibility is derived in the IIS. We can therefore discard p ′ for explanation generation in our algorithm. Now coming back to our Algorithm 5.3, since p A is infeasible in HA A , we add p A to the set IP. All extensions of p A are also discarded to reduce the number of paths being checked. We store the IIS path-segment in the set S. Algorithm 5.3: c A _infeasibility(p A , HA A ) Input: A path p A , and the agent’s model HA A . Output: Return IIS path-segments. 1 begin 2 c A = Get constraints of p A in HA A ; 3Solve(c H ); 4IIS-Path_Segments = Find_IIS(c A ) ;/* Use BACH */ 5Add p A to IP; 6return IIS-Path_Segments; 7 end We give an explanatory example of continuous path analysis through which IIS path- segments along a path can be identified. These IIS path-segments are conveyed as explan- ations for the infeasibility of the path. Example 5.3.2.2 Consider the path in the human model of Example 5.3.2.1 shown as a Blue dotted line in Figure 5.2, which is feasible in the abstract graph structure G A of the agent model. Continuous feasibility analysis in the human model reveals that the path is feasible, i.e., there is a corresponding plan in the human model. We take the corresponding path of the agent model and perform infeasibility analysis of the path with the continuous dynamics of the agent model which returns the IIS path-segment l 10 -e 10 16 -l 16 -e 16 22 -l 22 for which the path becomes infeasible. We add the IIS path-segment to the bad-segments set S. In following iterations, any path that consists of this IIS path-segment is considered an 114 5.4. EVALUATION infeasible path and not further assessed. For example, the path l 7 -e 7 1 -l 1 -e 1 2 -l 2 -e 2 3 -l 3 -e 3 9 -l 9 - e 9 10 -l 10 -e 10 16 -l 16 -e 16 22 -l 22 -e 22 23 -l 23 -e 23 17 -l 17 -e 17 18 -l 18 also consists of the IIS path-segment l 10 -e 10 16 - l 16 -e 16 22 -l 22 as a sub-path, and thus is known to be infeasible with IIS already extracted. Thus, this path will not be considered for explanation generation. 5.3.3 Explanation and Human Model Update Our method provides explanations for the unsolvable planning problems in hybrid systems through a path-oriented continuous reconciliation process between the human model and the agent model. The MRP involves the AI agent providing an explanation or model update to the human so that in the new updated human model, the unsolvability of the planning problem can be understood. As plans are projected to paths, the explanations provided by our method are of the following types: Path-wise explanation and model update: A path-wise update of model differences, either in terms of transitions (through discrete feasibility analysis) or in terms of IIS path- segments (through continuous feasibility analysis) provides a short and precise explanation with respect to a single path. Unsolvability explanation by model reconciliation: Providing a consolidated col- lection of model differences in terms of transitions and IIS path-segments along all paths. Optimization To optimize our technique, we prune paths from the human model while searching for a candidate path to reduce the number of path analysis steps, which is quite resource- consuming. We prune paths as follows: • We prune an invalid-edge from the human graph structure to reduce the number of path computations. • If a path is an extension of any infeasile path from IP, we discard the path. • If a path consists of any of the IIS path-segments as a sub-path from the set S, we discard the path. 5.4 Evaluation In this section, we discuss the application of our framework on 2 case studies of planning problem domains of hybrid systems, (a) a warehouse automation system and (b) a water- level monitoring system. A brief description of the warehouse automation domain is presented in Section 5.1, and a pictorial overview is shown in Figure 5.1. It has 24 locations in both models (the human model and the agent model). We have used three planning problem instances on this domain, where they differ in the initial, goal, and blocked locations, as well as in the number of transitions in the domain. The water-level monitoring domain is presented with one planning problem instance. This domain has 6 115 CHAPTER 5. EXPLAINING UNSOLVABILITY OF PLANNING PROBLEMS IN HYBRID SYSTEMS WITH MODEL RECONCILIATION TransNo. of (in HM)PathsInvalidIISTimeMem BenchmarkInsDepLocHMAMpathsinf. pathsreplayededgessegs(in sec)(in MB) 7303030.9664.9 Prob0110245650646581552.9077.6 1598862213527751034151313119.161993.7 Warehouse 7220101.2071.1 automation Prob021024565038326331.4870.3 15576944178215912334762837.011211.3 7220100.7275.7 Prob0310245448262198643563.3675.5 152183615196664033226677.72544.8 Water-level 6101010.5361 monitoring Prob0120666502020.9660.2 501202021.1761.4 Table 5.1: Results on the warehouse automation and water-level monitoring systems. HM and AM represent Human Model and Agent Model, respectively. locations and 6 transitions in both models. All experiments are performed on a machine with 8 GB RAM, an Intel Core i5-8250U@1.60GHz, 8 8-core processors with Ubuntu 18.04 64-bit OS. The example warehouse automation domain, the problem files, and the code base can be found at: https://gitlab.com/Sazwar/XSpeed-plan. Table 5.1 shows results for three planning problem instances of the warehouse auto- mation system and one planning problem instance of the water level monitoring system. Each planning problem instance is presented with varying plan depth to show the utility of this framework on larger plans, as well as on the plan space exploration. Benchmark represents the planning domain, warehouse automation, and the water level monitoring system. Ins represents the planning problem instances of the domains, while Dep presents the bound on the plan length. Loc shows the number of locations in the hybrid automata of the human and the agent models. Trans denotes the number of edges in the hybrid automata of the human model (HM) and the agent model (AM), respectively. No. of paths specifies the number of abstract paths obtained from the initial to the goal location for the planning problem instance in the human model obtained from the abstract graph structure. Now, recall that first, a path is to be checked in the agent’s discrete graph. The path may or may not be feasible with the continuous dynamics of the human model. Once the path is found in the agent’s discrete graph, we proceed to check for its feasibility in the continuous dynamics of the human model. No. of inf. paths specifies the number of paths that are infeasible in the human model itself when the continuous dynamics are brought in. Paths replayed presents the number of paths of the human model that are successfully replayed in the abstract graph structure of the agent model. Invalid edges shows the number of edges found that are present in the human model but not in the agent model. IIS segs shows the number of IIS path-segments found. Time and Mem report the corresponding execution time and memory usage incurred by our framework for reconciliation. Table 5.2 reports a path-wise average explanation generation time incurred by us. Since our reconciliation is path-based, our framework can also be used to analyze a single path to quickly pinpoint the infeasibility of a given plan. As earlier, Benchmark, Ins, and Depth represent the domains, the planning problem instances, and the plan depth, respectively. The column Avg time presents the average explanation generation time of a single path on those instances. 116 5.5. RELATED WORKS BenchmarkInsDepth Avg time (in sec) Prob01 70.32 100.045 150.132 Warehouse Prob02 70.6 automation 100.038 150.049 Prob03 70.36 100.013 150.031 Water-level Prob01 60.53 monitoring 200.192 500.097 Table 5.2: Average explanation generation time. Analysis of results: For all problem instances of the warehouse automation and the problem instance of the water-level monitoring system, the human model and the agent model have the same number of locations. However, the number of transitions in these two models varies in the warehouse automation domain, whereas they have the same number of transitions in the water-level monitoring domain. For each of the problem instances, we present results for varying plan depth. However, the number of paths explored increases exponentially with an increase in plan depth in the warehouse automation domain. The difference between the No. of paths in the human model (HM) and the Paths replayed in the agent model shows the number of paths pruned for optimization to speed up the process. For the three planning problem instances in the warehouse automation domain, 21.58%, 72.42%, and 69.66% of the paths are discarded, respectively, while searching for explanations. In the water-level monitoring domain, 72.22% of paths are similarly pruned from generating any explanation. The Invalid edges presents only the explanation edge set that is not present in the agent model along the checked paths. Once an invalid- edge is found along a path, that path is not assessed further to speed up the execution. Hence, an invalid-edge that might appear later in the path is not checked. The IIS segs presents the number of distinct path-segments along which the planning problem is unsolvable. The Time and the Mem usages indicate that our framework works reasonably well with time consumption and memory utilization. However, with the increase of plan depth, the time and memory usage increase as the number of paths enumerated increases exponentially. The average time for model reconciliation to the MRP problem is 1387.31 seconds, where we consider all the possible plans for a planning problem. The average explanation generation time of a path is 0.2 second, which is quite small and signifies that we can quickly reconcile. 5.5 Related Works Our work leverages the intersection of reachability analysis and planning − our tech- niques are inspired by path-oriented bounded model checking (BMC) (Biere et al. [2003]) 117 CHAPTER 5. EXPLAINING UNSOLVABILITY OF PLANNING PROBLEMS IN HYBRID SYSTEMS WITH MODEL RECONCILIATION approach, and the concept of locating minimal infeasible constraint sets in linear programs (Chinneck and Dravnieks [1991]) and its application to BMC of linear hybrid automata (Fränzle and Herde [2004], Bu et al. [2008]). We apply these techniques to explain unsolv- ability of the planning problem through model reconciliation (Chakraborti et al. [2017]), which was introduced by the planning community in the context of explainable AI planning (XAIP) (Hoffmann and Magazzeni [2019]). A significant amount of research in XAIP focuses on generating explanations of solu- tions to planning problems, i.e., the problem of explaining why a given plan or action was chosen. However, explaining unsolvability of a planning problem remains a largely open and understudied problem in this area of research. Some notable works that have tried to address unsolvability of planning problems mostly looked at verifying the unsolvability by generating certificates (Eriksson and Helmert [2020]), (Eriksson et al. [2017]) or proofs (Eriksson et al. [2018]) rather than explaining the causalities of unsolvability of the plan- ning problem. Such certificates or proofs of unsolvability are not enough to increase the human understandability of why the problem was unsolvable. Additionally, most of these works are on classical planning problems on discrete domains. Verifying unsolvability of planning problems in hybrid systems comes with an additional challenge since these plan- ning problems are undecidable in general (Alur et al. [1995a]). (Sarwar et al. [2023b]) provides a way for addressing unsolvability of a planning problem in hybrid domains by δ-approximate bounded reachability analysis (Gao et al. [2014]). Few notable works that are directed towards explaining the unsolvability of a plan- ning problem are, similarly, limited to classical planning problems. (Göbelbecker et al. [2010]) argues that excuses can be produced for why a plan cannot be found. This work proposes a formalization of counterfactual alterations to the original planning task, such that the new planning task turns out to be solvable, and provides an algorithm to find these excuses. (Sreedharan et al. [2019]) uses hierarchical model abstractions to generate the reason for unsolvability of planning problems. These hierarchical model abstractions relax a planning problem until a solution can be found. Then, they look for landmarks of this relaxed problem that cannot be satisfied in less relaxed versions of the problem. The unsatisfiability of these landmarks provides a succinct description of critical propositions that cannot be satisfied. (Eifler et al. [2020]) has taken a somewhat different approach by deriving properties that must be exhibited by all possible plans that could serve as ex- planations in case of unsolvability. However, generating excuses, hierarchically abstracting models, or deriving plan properties in terms of propositional formulas may not be enough to understand why a problem was unsolvable for complex domains like planning problems of hybrid systems, which encode mixed discrete and continuous dynamics. This motivates a reconciliation step. MRP (Chakraborti et al. [2017]) has been a popular theme in this direction, where explanations are provided as the reconciliation between two models (the human model and the AI model) by bringing them closer. However, most of the works in the MRP literature have been applied to classical planning problem domains. A logic-based extension to the MRP has been applied to mixed discrete-continuous domains (Vasileiou et al. [2022]). In this work, the authors approach the MRP based on knowledge representation and reason- 118 5.6. CONCLUSION ing. It provides a framework that finds a subset of the knowledge base of the agent with which to reconcile the human knowledge base for explanations. However, the explanation is provided for why a plan is feasible in a model rather than addressing the unsolvability of a planning problem head-on, as is done by us. In contrast, our approach to MRP is based on a path-based continuous model reconciliation process through communications between the agent and the human models that are presented as hybrid automata. Hy- brid automata present a natural description formalism for hybrid domains, in addition to specialized language extensions (e.g., PDDL+). We believe that the hybrid automaton structure, being state-based, provides a useful design methodology for such otherwise com- plex domains. The adoption of hybrid automata and explanations produced thereof makes our work quite novel. To the best of our knowledge, explanations and reconciliations on hybrid automatons for hybrid domain models have not been addressed in the literature. Added to that is our approach of dealing with the reconciliation task in two different steps, first in the abstract graph, and then, in the continuous dynamics is new as well. This distinguishes our work from other approaches available in MRP literature. 5.6 Conclusion In this work, we explore the area of explaining the unsolvability of planning problems for hybrid systems. We propose a path-based continuous model reconciliation framework for explaining unsolvability. We show that a discrete and continuous path analysis approach can be leveraged to derive explanations of the unsolvability of a planning problem in hybrid domains. While discrete path analysis falsifies a path by mapping each location and transition of the path in the human model to the agent model, continuous path analysis leverages reachability analysis, combining with minimal inconsistent constraint sets to find the infeasible path-segments for which the paths become unsatisfiable. We demonstrate a warehouse automation system as an example planning domain, along with a water-level monitoring system, and present results for problem instances on each domain with varying plan depth. We show that the average explanation generation time of this framework is significantly small, which can be utilized to quickly pinpoint any infeasibilities of a given plan. While the path-oriented analysis explores the plan space to find all the possible paths, it can track most of the causes that contribute to the unsolvability of the planning problem. As future work, we plan to include provisions to update and reconcile the continuous dynamics which is not considered in this version. Additionally, we intend to explore more domain models and instances going forward. 119 CHAPTER 5. EXPLAINING UNSOLVABILITY OF PLANNING PROBLEMS IN HYBRID SYSTEMS WITH MODEL RECONCILIATION 120 Chapter 6 Exploring Inevitable Waypoints for Unsolvability Explanation in Hybrid Planning Problems Chapter Abstract: Explaining unsolvability of planning problems is of significant re- search interest in Explainable AI Planning. A number of research efforts on generating explanations of solutions to planning problems have been reported in AI planning literat- ure. However, explaining the unsolvability of planning problems remains a largely open and understudied problem. A widely practiced approach to plan generation and automated problem solving, in general, is to decompose tasks into sub-problems that help progress- ively converge towards the goal. In this work, we propose to adopt the same philosophy of sub-problem identification as a mechanism for analyzing and explaining unsolvability of planning problems in hybrid systems. In particular, for a given unsolvable planning problem, we propose to identify common waypoints, which are universal obstacles to plan existence; in other words, they appear on every plan from the source to the planning goal. This work envisions such waypoints as sub-problems of the planning problem and the unreachability of any of these waypoints as an explanation for the unsolvability of the ori- ginal planning problem. We propose a novel method of waypoint identification by casting the problem as an instance of the longest common subsequence problem, a widely popu- lar problem in computer science, typically considered as an illustrative example for the dynamic programming paradigm. Once the waypoints are identified, we perform symbolic reachability analysis on them to identify the earliest unreachable waypoint and report it as the explanation of unsolvability. We present experimental results on unsolvable planning problems in hybrid domains. I n human-computer interaction (HCI), humans engage in trustworthy collaborations with autonomous agents, and one of the precursors of such collaborations is that an autonom- ous agent must explain the rationale behind its decision to the human. With the emergence of artificial intelligence (AI) and the multitude of application domains where AI plan- 121 CHAPTER 6. EXPLORING INEVITABLE WAYPOINTS FOR UNSOLVABILITY EXPLANATION IN HYBRID PLANNING PROBLEMS ning is being envisioned to replace plans generated by humans, Explainable AI Planning (XAIP) (Fox et al. [2017], Hoffmann and Magazzeni [2019]) has emerged as an important connection between HCI and AI for designing explainable systems that bridges the gap between theoretical and algorithmic planning and real-world applications (Chakraborti et al. [2020]). While there has been a lot of research on generating explanations to planning problems, most of the earlier works in explanation generation have focused on explaining why a given plan or action was chosen (Chakraborti et al. [2017, 2019a], Krarup et al. [2021b], Sarwar et al. [2023b]). However, explaining the unsolvability of a given planning problem remains a largely open and understudied problem. The recent works that focus on explaining the unsolvability of planning problems have primarily concentrated on gen- erating certificates or proofs of unsolvability (Eriksson et al. [2017, 2018]), or on identifying counterfactual alterations to the original planning task to make it solvable, often referred to as "excuses" (Göbelbecker et al. [2010]). These approaches, which are more oriented towards automatic verification, may fall short in adequately explaining unsolvability in complex planning domains. A well-known insight into human thinking and problem-solving is that humans tend to decompose a problem into sub-problems that help in progressively converging towards the goal. Many AI systems mimic this notion in the way they solve problems. For instance, the main feature of the pioneering automated theorem prover, logic theorist, is the use of problem-subproblem hierarchy (Newell and Simon [1956]). An innovative technique for identification of sub-problems relevant for explaining unsolvability of a planning problem in domains with discrete dynamics has been proposed in (Sreedharan et al. [2019]). In this work, we propose adopting the same philosophy of sub-problem identification as an efficient mechanism for analyzing and explaining the unsolvability of planning problems in hybrid domains, which are domains with a combination of discrete and continuous dynamics. In particular, for a given unsolvable planning problem, we propose to identify a sequence of waypoints that are universal obstacles to plan existence; in other words, they appear on every path on every plan from the source to the planning goal. This work envisions such waypoints as sub-problems of the planning problem and the unreachability of any of these waypoints as an explanation for the unsolvability of the original problem at hand. We propose a novel method of waypoint identification by casting the problem as an instance of the longest common subsequence problem, a widely popular problem in computer science, typically considered as an illustrative example for the dynamic programming paradigm. Once the waypoints are identified, we perform symbolic reachability analysis on them to identify the earliest unreachable waypoint and report it as the explanation of unsolvability. We present experimental results on unsolvable planning problems in hybrid domains. In summary, the key contributions of this work are: (a) A proposal for an artifact for explaining unsolvability of hybrid planning problems based on the identification of inevitable waypoints. (b) A method to generate the explanation artifact by casting it as an instance of the longest common subsequence problem, and subsequently using symbolic reachability analysis on the hybrid automaton. 122 6.1. MOTIVATING EXAMPLE The rest of this chapter is organized as follows. In Section 6.1, we present a motivat- ing example to demonstrate this work. Section 6.2 provides a background and problem overview. Section 6.3 illustrates our methodology and the framework of explanation. Sec- tion 6.4 discusses implementation and results. Section 6.5 presents related literature. Fi- nally, Section 6.6 summarizes the contributions and the findings of this work and discusses possible future directions. 6.1 Motivating Example In this section, we present a motivating example in the context of a planning problem for a planetary rover that explores a planetary site and collects samples for experiments. The agent (an autonomous battery-powered rover) possesses knowledge about the topography of the exploration site as a planar grid. Figure 6.1 shows the topography as a 5×5 grid of cells. The rover is initially positioned at cell 11, and there is a base-station at cell 25. The task of the rover is to collect soil and rock samples from designated sites in cells 1 and 24, respectively, and then reach the base-station. Mountainous regions and craters in the terrain are marked as impassable (cells 7, 12, etc). There are inclined areas in the terrain shown in orange cells. The motion dynamics of the rover is interpreted as a hybrid system. The rover’s continuous motion and battery discharge dynamics can be different in each cell. For instance, the motion and battery depletion dynamics in an inclined region is different from the dynamics in flat regions and regions where soil and rock samples are collected. When a rover makes a transition from one cell to another, it starts to follow the dynamics of the new cell instantaneously. The discrete dynamics here capture the connectivity of the cells in the presence of mountains and craters where the rover cannot move. The rover’s movement is restricted to one of its adjacent cells, and movement to diagonal cells is prohibited. When a planner reports the planning task as unsolvable, our algorithm identifies ordered waypoints that ought to be reached in any plan to achieve the task. For in- stance, in the discussed domain, our algorithm detects that the cells 6-1-2-3-8-13-14-24 must be visited by any valid plan to solve the planning task. These are marked as ordered waypoints w1− w8 in the figure. Our proposed algorithm envisions such waypoints as sub-goals of the planning problem. The unreachability of any of these waypoints under the domain dynamics is reported as an explanation for the unsolvability of the original planning problem. The computational challenge lies in finding the waypoints, finding an order between them, and lastly, finding the earliest waypoint that is unreachable under the dynamics. We propose a method of waypoint identification by casting the problem as an instance of the longest common subsequence problem, a widely popular problem in computer science, typically considered as an illustrative example for the dynamic pro- gramming paradigm. Unreachability of a waypoint is determined using a bounded model checker. For instance, given the initial rover battery charge of 10 units and the battery depletion rates of the cells (depletion rate of 1 unit in cells except in cell 1 and cell 24 where soil and rock sampling depletes battery at a higher rate of 2 units, and in the cells with inclination having a depletion rate of 3 units), reachability analysis determines that 123 CHAPTER 6. EXPLORING INEVITABLE WAYPOINTS FOR UNSOLVABILITY EXPLANATION IN HYBRID PLANNING PROBLEMS Figure 6.1: The rover domain is depicted. Initially, the rover is at cell 11. Mountains and craters are impassable regions of the terrain, such as cells 7, 12, etc. The rover needs to collect soil samples from cell 1 and rock samples from cell 24. The base station is at cell 25. The up-slope areas in the terrain are shown in orange. The inevitable waypoints in the domain are cells w1 to w8 marked in blue. cell 13 is the first unreachable waypoint and reports this as an explanation of unsolvability. Some of the waypoints as sub-goals may be explicitly known from the planning prob- lem description itself. For example, the planetary rover domain has two sub-goals explicitly mentioned, namely the collection of soil and rock samples from cell 1 and cell 24, respect- ively, marked as waypoints w2 and w8. There may be sub-goals that are not apparent from the problem description explicitly, but they are implicitly mandatory to complete the bigger planning task. For example, the implicit waypoints in our planetary rover do- main are marked w1 and w3−w7 in the figure. Our waypoint detection algorithm detects both the explicit as well as implicit waypoints. We term these as inevitable waypoints. In the following section, we formally describe the domain representation and the explanation problem we intend to solve. 6.2 Problem Overview In this section, we provide a formal definition of a planning problem in a hybrid system. A hybrid system exhibits an interplay of discrete and continuous dynamics. Hybrid automata H (see Definition 2.1.1) are a well-known mathematical model for such systems (Alur et al. [1995a, 1993]). A state of H is a pair (l,v) consisting of a location l∈ Loc and a valuation v ∈ Inv(l). The component Init defines the set of initial states of the automaton. Figure 6.2 shows the hybrid automaton for the planetary rover domain, as an example. Each cell in Figure 6.1 is represented as a location of the automaton with an invariant that is the region enclosed by the cell. The battery charge depletion rate c and motion dynamics (described over the 124 6.2. PROBLEM OVERVIEW position variables x and y) of the rover within the cell are modeled as flow equations of the location. The cell-to-cell movement of the rover is given as transitions between locations. The initial location is shown in green. The yellow locations represent the regions where the rover collects soil and rock samples. The orange locations represent the inclined regions. The base station, the rover’s destination, is shown in red. An impassable location (loc7) is shown in grey with no incoming or outgoing edges. Loc1 Flow: Loc6 Loc11 Loc2Loc3 Loc8 Loc13 Loc7 Loc14 Loc19 Loc24 Loc25 Flow: Inv: Flow:Flow:Flow: Flow:Flow:Flow:Flow: Flow:Flow:Flow: Inv: Inv: Inv: Inv: Inv: Inv: Inv:Inv: Inv:Inv: Inv: init e1 6 e11 6 e6 11 e6 1 e1 2 e2 1 e2 3 e3 2 e8 3 e3 8 e13 8 e8 13 e13 14 e14 13 e19 14 e14 19 e19 24 e24 19 e24 25 e25 24 Figure 6.2: A hybrid automaton model of the planetary rover domain (partially shown). Definition 6.2.1 The graph of a hybrid automaton H is defined as G H = (V , E), where V = Loc and E ⊆ Loc× Loc such that for every (l,a,g,r,l ′ ) ∈ Edge, there is an edge (l,l ′ )∈ E.2 A discrete evolution of the system is then represented as a path on the automaton graph. Definition 6.2.2 A path p between locations l 0 and l n in G H is a sequence of locations and edges given as: l 0 e 0 −→ l 1 e 1 −→ l 2 e 2 −→ ... e n−1 −→ l n where l i ∈ Loc, e i ∈ Edge, and l i , l i+1 are the source and destination of e i respectively. The length of a path is the number of edges it contains.2 A planning problem typically consists of a domain description and the initial and goal states of the planning task. In this work, as part of the problem description, we also consider a bound on how many times the actions of the domain can be applied and a set of constraints that every valid plan must satisfy. As an underlying assumption, we consider the planning problem to have one initial and one goal location. This is not a significant limitation, since we can explain the unsolvability of a planning problem with multiple initial and goal locations by solving many explanation problems, each for a planning problem with an initial and goal location pair. We define a planning problem for a hybrid system as follows: 125 CHAPTER 6. EXPLORING INEVITABLE WAYPOINTS FOR UNSOLVABILITY EXPLANATION IN HYBRID PLANNING PROBLEMS Definition 6.2.3 A planning problem Π for a hybrid system is a three-tuple (Dom, Prob, Depth), where • Dom (Sarwar et al. [2023b]) is represented as a hybrid automaton H. • Prob is a tuple⟨Init,Goal⟩ representing a problem description, where Init and Goal define the initial and the goal states. • Depth defines the bound on how many actions can be applied in a plan.2 The set of initial states of the planning problem is given by Init, that is a tuple ⟨l 0 ,S 0 ⟩ such that l 0 ∈ Loc and S 0 ⊆ Inv(l 0 ). The tuple represents initial states (l 0 ,v)| v ∈ S 0 . Moreover, Init must be a subset of the set of initial states of the automaton H. The Goal states of the planning problem are given as a tuple⟨l goal ,S goal ⟩ such that l goal ∈ Loc and S goal ⊆ Inv(l goal ). The tuple represents goal states (l goal ,v)| v ∈ S goal . The set of labels of the hybrid automaton corresponds to the available actions for a plan. A state can change either due to a transition in the automaton or due to the passage of time, where the variables evolve according to the flow in a location. We refer to the former as discrete transition and the latter as timed transition. A discrete transition happens due to the application of an action given by the label of the transition, provided the valuation of the state satisfies the guard of the transition. On taking a discrete transition, the valuation of the new state must follow the reset map of the transition. We now define a plan for a planning problem Π: Definition 6.2.4 A plan for a planning problem Π is a tuple ⟨ λ n , makespan ⟩ where λ n is a finite sequence of n pairs ⟨t i ,a i ⟩. In the pair, t i ∈ R ≥0 is the time instance of executing the action a i ∈ Lab. In the sequence, t i is non-decreasing. The makespan is the duration of the plan.2 We refer to the length of a plan to be the number of pairs ⟨t i ,a i ⟩ in λ n . An executable plan on a H is defined as follows: Definition 6.2.5 A plan⟨λ n ,makespan⟩ for a planning problem Π is called executable on the domain H of Π if and only if the application of the plan on H results in an alternating sequence of timed and discrete transitions, called a run of the H depicted as: (l 0 ,v 0 ) τ 0 ⇝ (l 0 ,v ′ 0 ) a 1 −→ (l 1 ,v 1 ) τ 1 ⇝ (l 1 ,v ′ 1 ) a 2 −→ ... a n −→ (l n ,v n ) τ n ⇝ (l n ,v ′ n ) where (i) (l 0 ,v 0 ) ∈ Init (i) a i is a label of some edge e i ∈ Edge such that l i−1 is the source and l i is the destination location of e i , v ′ i−1 ∈ g and v i ∈ r where g is the guard and r is the reset map of e i , ∀i ∈J1..nK (i) The transitions labeled τ i ∈ R ≥0 represent timed transitions with τ i being the time of dwelling in the location, with the constraint that ∀t ∈ [0,τ i ], the timed transition (l i ,v i ) t ⇝ (l i ,v + i ) has v + i ∈ Inv(l i ), ∀i ∈J0..nK . (iv) ⟨ P i−1 j=0 τ j ,a i ⟩ is a pair in λ n , ∀i∈J1..nK (v) P n i=0 τ i = makespan.2 The length of a run is the number of discrete transitions it contains. In control-theoretic terms, a plan is a control strategy that acts on a plant, a hybrid automaton in our context. 126 6.3. METHODOLOGY The application of a control strategy on a plant results in a controlled execution of the plant, which we call a run in our context. Due to uncertainties modeled in a H, an application of a plan may result in more than one run. A plan is called valid if it is executable, i.e., its applications on H results in a run from a state in Init to a state in Goal, and the length of the plan is less than or equal to Depth. In the following text, we write "a run of a valid plan" as a short-form of saying "a resulting run of the domain H of Π due to the application of a valid plan". A planning problem Π is called solvable if it admits a valid plan. If no such plan exists, then the planning problem is said to be unsolvable. Definition 6.2.6 A planning problem Π is unsolvable if it admits no valid plan.2 The problem addressed in this work is as follows: Problem Statement 1 Given an unsolvable planning problem Π, generate an artifact automatically that explains why Π is unsolvable. In the subsequent sections, we describe the details of the explanation artifact and the algorithm to generate the same. 6.3 Methodology Start INPUT: Planning Problem π(H, Prob, D) OUTPUT: Explanation (π) G H = ComputeGraph (H) Is length(LCS) ≤ 2 ? Y < = ConstructChain (LCS) k = |Y < | LCS = FindLCS (PS) End Compute the graph structure of the model Compute all paths in graph structure G H Yes No Yes No Yes No Find the longest common location sequence of all path strings Construct the chain of sub- problems i = 0 Perform reachability analysis of a sub-problem π i Fetch the next sub-problem The planning problem is discretely infeasbile Returns the first unachievable sub- problem π i Unable to find sub-problem PS = ComputePaths (G H ) Is PS = ∅ ? Is result = SAT ? i ++ Yes result = Reachability (π i ) Is i < k ? The last sub- problem π k is unachievable No π i ∈Y < Figure 6.3: Proposed Explanation Framework. In this section, we describe our explanation algorithm, which takes an unsolvable planning problem Π as input and computes an artifact Explanation(Π), which we define later in the text (Def. 6.3.4). Our explanation algorithm attempts to divide an unsolvable planning problem Π into several sub-problems, following the common divide-and-conquer paradigm of problem-solving. These sub-problems have the property that each must be solvable for Π to be solvable. Identifying these sub-problems is computationally challenging and is the key to generating the explanation artifact. The proposed algorithm takes a layered approach. The sub-problems are determined by taking into consideration only the discrete dynamics of the hybrid automaton. Once the sub-problems are identified, the 127 CHAPTER 6. EXPLORING INEVITABLE WAYPOINTS FOR UNSOLVABILITY EXPLANATION IN HYBRID PLANNING PROBLEMS explanation artifact is generated by considering the hybrid dynamics in its entirety. The abstraction of the continuous dynamics in the first phase allows us to work in the domain of graphs, and consequently, we show a reduction from finding sub-problems to finding the longest common sub-sequence of a finite set of strings, a well-known problem in algorithms. Finally, the feasibility of the identified sub-problems is verified using symbolic reachability analysis, and the explanation is generated, which highlights which of the sub-problems is responsible for the unsolvability of Π. A schematic diagram of our explanation framework is shown in Figure 6.3. We now present the details. 6.3.1 Decomposition into Sub-Problems We present here the central idea of this work, the binary relation among subproblems. Before presenting the idea of subproblems, we define below when a run of a valid plan of Π is said to intersect with a set of states of the domain of the planning problem. Definition 6.3.1 Given a run r of a valid plan of Π and a set of states ⟨l,S⟩ of the domain H of Π, the run r is said to intersect with ⟨l,S⟩ when the following conditions hold: (i) there exists a timed transition (l,v) τ ⇝ (l,v ′ ) in r, (i) there exists a dwelling time t∈ [0,τ ] such that the timed transition (l,v) t ⇝ (l,x) has x∈ S. We now define the notion of a sub-problem as a relation between hybrid planning problems. Definition 6.3.2 Given planning problems Π i = (Dom i , Prob i , Depth i ) and Π j = (Dom j , Prob j , Depth j ), we say Π i is a sub-problem of Π j when the following conditions hold: (a) Dom i = Dom j , (b) Depth i = Depth j , and (c) Prob i , Prob j are tuples ⟨ Init i , Goal i ⟩ and ⟨ Init j , Goal j ⟩ resp. where • Init i = Init j • For any run of a valid plan of Π j to exist, the run must intersect with Goal i . Note that if Π i is a subproblem of Π j , then Π i must be solvable for Π j to be solvable. In other words, it is impossible to have a run of a valid plan of Π j that does not intersect the goal states of Π i . In the definition, there is no assumption of solvability or unsolvability of Π j . For an unsolvable Π j where there exists no real run of a valid plan, the definition aims to convey that any "hypothetical" run of a valid plan of Π j must visit Goal i . Example 6.3.1.1 Consider a planning problem in a rover-like domain explained above, the state-space shown in Figure 6.4. The states within each cell belong to the invariant of a distinct location of a H model with nine locations. The initial states are in the green region, and the goal states are in the red region. The shaded regions are impassable. Some of the runs of valid plans are shown as red trajectories from an initial state to a goal state. Observe that for any run of a valid plan to exist, it must pass through the region enclosed by the blue cell. This means that the region is an inevitable waypoint. In the following text, we interchangeably refer to a sub-problem as a waypoint. We write Π i ≤ Π j to say that Π i is a sub-problem of Π j . A planning problem Π can have multiple sub-problems. We denote the set of all sub-problems of Π by Π ⊔ . Π ⊔ =Π ′ | Π ′ ≤ Π(6.1) 128 6.3. METHODOLOGY 1 2 3 4 5 6 789 Figure 6.4: All runs of valid plans from the initial set to the goal set must pass through the blue doorway. We envision the blue region as an inevitable waypoint to the planning problem. 6.3.1.1 Abstraction over Planning Problems The cardinality of Π ⊔ can be potentially infinite. We therefore present the following construction of a finite set of planning problems Π ∗ for a given Π induced by itsH domain. We also define an abstraction function α that maps each Π ′ ∈ Π ⊔ to an element of Π ∗ such that if Π ′ ≤ Π then α(Π ′ ) ≤ Π. We then proceed with our analysis on this finite abstraction Π ∗ . Definition 6.3.3 Let Π = (H, ⟨Init, Goal⟩, Depth) be a planning problem. We define a finite set of planning problems Π ∗ as follows: Π ∗ = [ ℓ∈Loc Π ℓ (6.2) where Loc is the finite set of locations of H, Π ℓ = ⟨ H, ⟨Init, Goal ℓ ⟩, Depth ⟩ and Goal ℓ =⟨ℓ,Inv(ℓ)⟩. The cardinality of Π ∗ will be the cardinality of Loc ofH. For example, the cardinality of Π ∗ for the problem Π of Figure 6.4 is nine, where each problem will have the tuple consisting of a location of the automaton and the corresponding invariant as its Goal. The definition of α : Π ⊔ → Π ∗ for a given Π ′ = (H, ⟨Init, Goal⟩, Depth), where Goal = ⟨ℓ goal ,S goal ⟩ is given as: α(Π ′ ) = Π ℓ goal (6.3) PROPOSITION 6.3.1 If Π ′ ∈ Π ⊔ then α(Π ′ )∈ Π ⊔ . Proof 6.3.1 The goal states of Π ′ constitute a subset of the goal states of α(Π ′ ) and therefore, if all valid runs of Π intersect the goal states of Π ′ , then they also intersect the goal states of α(Π ′ ). Hence, α(Π ′ )≤ Π and thus α(Π ′ )∈ Π ⊔ . PROPOSITION 6.3.2 The ordered pair ⟨Π ∗ ,≤⟩ is a partially ordered set (poset). Proof 6.3.2 For ⟨Π ∗ ,≤⟩ to be a poset, the binary relation ≤ on Π ∗ should be reflexive, anti-symmetric, and transitive, that is, ≤ must be a partial order relation. From definition 6.3.2, it is easy to see that every planning problem in Π ∗ is a sub-problem to itself and hence reflexive. Transitivity and anti-symmetry also follow from the definition of sub-problem. 129 CHAPTER 6. EXPLORING INEVITABLE WAYPOINTS FOR UNSOLVABILITY EXPLANATION IN HYBRID PLANNING PROBLEMS Chains Recall that a subset Y < ⊆ Π ∗ of a partially ordered set ⟨Π ∗ ,≤⟩ is a chain if ∀Π i , Π j ∈ Y < : (Π i ≤ Π j )∨ (Π j ≤ Π i ). The length of a chain is the number of elements in Y < . A poset can have more than one chain of longest length. We now describe the explanation artifact that we intend to generate with our explanation algorithm: Definition 6.3.4 Given an unsolvable planning problem Π, an explanation artifact Explanation(Π) is a planning problem Π i ∈ Π ∗ such that: (i) Π i ∈ Y < for some Y < such that Π i is unsolvable. (i) ∀ Π j ∈ Y < such that Π j ≤ Π i and Π j ̸= Π i , Π j is solvable. The explanation is thus the first unsolvable sub-problem of Π in a chain of sub-problems in the poset ⟨Π ∗ ,≤⟩. As an illustration, assume that Y < =Π 1 , Π 2 , ..., Π n is a chain of sub-problems of Π in⟨Π ∗ ,≤⟩ having a total order as Π 1 ≤ Π 2 ≤ ...≤ Π n , Explanation(Π) is the Π i ∈ Y < that is unsolvable where ∀Π j ∈ Y < such that Π j ≤ Π i , Π j is solvable. For instance, consider the planning problem of the motivating example. Our abstraction will render 25 planning problems in Π ∗ , each having one of the cells as the goal. An example of a chain of sub-problems is the chain consisting of 8 sub-problems shown as waypoints w1−w8 with a total order w1≤ w2≤ ...≤ w8 amongst them. Observe that this chain is also the longest possible chain of sub-problems in the poset. In this chain, the explanation generated will be the earliest w i that is unsolvable. The goal of generating the proposed explanation artifact is to assist a human expert/con- trol engineer in diagnosing the causes of unsolvability by localizing the earliest cause of unsolvability. The detection of the earliest waypoint, which is infeasible, localizes the primitive cause of unsolvability in that sense. The intuition behind finding a chain of sub-problems is to have a causal analysis of the unsolvability of the planning problem. We now present the algorithm to find Explanation(Π) in the following section. Our initial step for an unsolvable planning problem involves examining all graphically con- nected paths of a bounded depth, extending from the initial to the goal location within the domain’s graph structure. Subsequently, we identify a common sequence of locations present on each of these paths. This location sequence is pivotal to constructing a chain of sub-problems for the unsolvable planning problem. We first show a reduction from finding a chain of sub-problems to finding a longest-common-subsequence of finitely many strings. 6.3.2 Reduction to Longest Common Subsequence (LCS) Problem The computation of Explanation(Π) first requires finding a chain of sub-problems of Π in ⟨Π ∗ ,≤⟩. We now show a reduction of this problem to the problem of finding an LCS of finitely many strings. Recall that reduction is a way of converting one problem into another problem such that the solution of the second problem can be used to solve the first problem. To present the reduction to LCS, we use the graph of a hybrid automaton. We represent a path in a graph as a string ps of the location sequence while eliminating the edges. For example, a path l 0 e 0 −→ l 1 e 1 −→ l 2 e 2 −→ ... e n−1 −→ l n is represented as a string 130 6.3. METHODOLOGY "l 0 l 1 l 2 ...l n ". Now, for the given unsolvable problem Π = (H,⟨Init,Goal⟩,Depth), we can compute all paths of length less than or equal to Depth between l 0 and l goal , the initial and the goal location in Init and Goal respectively. When Π includes explicit sub-tasks of visiting a certain set of locations, we compute only those paths between l 0 to l goal that visit the given set of locations. Since we are interested in paths of bounded length, there will be finitely many such paths. The string representations of all such paths are denoted by the set PS(Π). The graph of the hybrid automaton provides a higher abstraction of the domain in the sense that if there is no path from l 0 to l goal in G H , then there cannot be any valid run of a plan from Init to Goal and hence the planning problem is unsolvable. We may then identify the cause of unsolvability to be in the discrete dynamics, oblivious to the continuous dynamics of the domain. More importantly, as we shall see now, a chain of sub-problems can be identified from the longest common subsequence of the strings in PS(Π). Finding an LCS between strings is a classic computer science problem. An LCS measures the closeness of two or more strings by finding the maximum number of identical symbols in them in the same order (Bergroth et al. [2000], Maier [1978], Hirschberg [1974]). Recall that a subsequence is different from a substring, which additionally requires that the common symbols present in the strings are without gaps. We now present the main result of the work. PROPOSITION 6.3.3 Given a planning problem Π, computing a chain of sub-problems Y < in the poset ⟨Π ∗ ,≤⟩ can be reduced to computing a longest common subsequence of PS(Π). S X A Y B Z D Figure 6.5: A depiction of a graph of a H. Let S and D be the source and the goal locations in a planning problem Π. Every path from S to D visits locations A and B in sequence. Thus, S-A-B-D is the LCS of paths in PS(Π). Clearly, if a valid run of a plan exists in the H, the run must visit the invariants of S, A, B, and D sequentially. Proof 6.3.3 G H = (V , E) is implicitly present in H of Π. Any graph search algorithm, such as breadth-first search, can compute paths in PS(Π). Let "l 0 l i l j ...l n " be an LCS of path strings in PS(Π). Being a common subsequence, every path from l 0 to l n visits these nodes in sequence. This implies that every valid run of Π must intersect the in- variant of these locations in sequence, that is: Π l 0 ≤ Π l i ≤ Π l j ≤ ... ≤ Π l n . ∴ Y < = Π l 0 , Π l i , Π l j ,..., Π n is a chain in ⟨Π ∗ ,≤⟩. Figure 6.5 shows a sketch of the proof idea. PROPOSITION 6.3.4 Given a planning problem Π, the length of LCS of PS(Π) is bounded by the length of the shortest path in PS(Π). 131 CHAPTER 6. EXPLORING INEVITABLE WAYPOINTS FOR UNSOLVABILITY EXPLANATION IN HYBRID PLANNING PROBLEMS Proof 6.3.4 Let the length of the LCS be l, and the length of the shortest path p in PS(Π) be |p|, the number of locations in p. As the locations in LCS are common to all paths in PS(Π). Therefore, the length of LCS cannot be greater than p, i.e., l ̸> |p|. Discussion: If PS(Π) is empty, our explanation algorithm terminates and reports that the planning problem is unsolvable due to the discrete dynamics, since there is no path between the initial and the goal locations. Observe that there is a connection between articulation points or cut-vertices of G H and the locations in an LCS of PS(Π). An articulation point is a vertex of a graph removal of which, along with its incident edges, results in an increase in the connected components in the graph. One can argue that every articulation point ofG H whose removal results in distinct components such that one contains l 0 and the other contains l goal , will be a member of the LCS. This is because every path from l 0 to l goal must contain such articulation points and therefore will be captured in the LCS. Let us call such articulation points as disconnecting articulation points in the sense that their removal disconnects l 0 and l goal . Note that every vertex in an LCS need not be such an articulation point ofG H . This is because an LCS is computed over paths in PS(Π), which contains only paths of length bounded by a depth specified in the problem instance. There may be paths in G H of longer length which does not pass through one or more vertices in the LCS. Such vertices in the LCS cannot be disconnecting articulation points whose removal disconnects l 0 and l goal . Consequently, if our algorithm finds a trivial LCS string of length two, which is l 0 − l goal , then G H has no disconnecting articulation point. Such an LCS is trivial because any valid plan of course must meet the invariant of l 0 followed by the invariant of l goal . Therefore, Y < = Π l 0 , Π l goal is always a chain of inevitable sub-problems for any planning problem Π. Although finding a trivial LCS does not mean there are no articulation points inG H . All articulation points in G H can be computed in polynomial time, but computing Explanation(Π) additionally requires finding disconnecting articulation points and an ordering on them based on the sub-problem relation. Thus, the polynomial-time algorithm does not suffice. LCS of strings can be computed using the standard dynamic programming paradigm (Bergroth et al. [2000]). For instance, in the motivating example of planetary rover domain, the longest common sequence for the strings of paths of length bounded by 15 from l 11 to l 25 is l 11 -l 6 -l 1 -l 2 -l 3 -l 8 -l 13 -l 14 -l 24 -l 25 . In the next section, we show the computation of Explanation(Π). 6.3.3 Explanation Generation by Reachability Analysis The computed LCS is converted to sub-problems from the locations in the LCS. For each location l in the LCS, we construct a sub-problem Π l (recall definition 6.3.3). Thus, we have the chain Y < = Π l 0 , Π l i , Π l j ,..., Π l n for an LCS say "l 0 l i l j ...l n ". We verify the solvability of the sub-problems in Y < (inevitable waypoints) using bounded reachability analysis on the hybrid automaton domain by reintroducing the continuous dynamics (by reintroducing the location invariants, flow, transition guards, and resets). Given a hybrid automaton, a set of initial and goal states, and a bound of analysis, say d, bounded reachability analysis is the method of computationally deciding whether any goal state 132 6.3. METHODOLOGY is reachable from any initial state by a run of the automaton of length bounded by d. Therefore, bounded reachability of the goal states from initial states of a planning problem bounded by Depth implies the existence of a run of a valid plan, which in turn implies the solvability of the given planning problem. In contrast, unreachability of the goal states implies the non-existence of any valid run and hence absence of a plan. As our planning problem under analysis is unsolvable, one or more of the waypoints in Y < must be unreachable. The reachability analysis of the waypoints is performed in the order in which they appear in the chain Y < . If the sub-problem Π i is found to be reachable, we proceed to check the reachability of the next sub-problem Π i+1 in the chain. The first Π i which is unreachable is returned as Explanation(Π), implying that it is the first unreachable sub-problem/waypoint in a chain of sub-problems/waypoints. We use a bounded reachability analysis tool Bach (Bu et al. [2008]) for reachability analysis of the waypoints. Bach can analyse linear hybrid automata (LHA) (Henzinger [1996], Li et al. [2006]) and reports a reachability problem instance as satisfiable when a run exists from an initial state to a goal state in the corresponding hybrid automaton for the planning problem of length bounded by a given depth, deciding the solvability of the corresponding planning problem. Otherwise, Bach reports the instance as unsatisfiable, when no such run exists. Algorithm 6.1 takes a chain of sub-problems Y < as input. It returns the first unreachable sub-problem in the chain as an explanation of unsolvability. Algorithm 6.1: Generating Explanation(Π) using Reachability Analysis input : A chain of sub-problems Y < = ⟨Π l 0 , Π l i , Π l j ,..., Π l n ⟩ output: First unreachable sub-problem Π l k in the chain. 1 n← length(Y < ) /* No. of sub-problems in Y < */ 2 for k ← 0 to n do 3 R← reachabilityProblem(H, Init, Goal l k , Depth), where Goal l k is the goal of Π l k /* Reduced to reachability problem */ 4 result← BACH (R) /* Use BACH */ 5 if result == SAT then 6continue /* When sub-problem Π l k is reachable. */ 7 else 8return Π l k /* Returns first unreachable sub-problem as the explanation. */ 9 end 10 end 6.3.4 Complexity Analysis In a H with n locations and given a source and destination location, the worst-case com- plexity of computing all paths in G H of length at most d from the given source to the destination is O n d . The complexity of computing LCS of strings corresponding to the paths (say m many) of maximum length d using standard dynamic programming is O d m . It is known to be an NP-hard problem (Djukanovic et al. [2020], Maier [1978]). Although 133 CHAPTER 6. EXPLORING INEVITABLE WAYPOINTS FOR UNSOLVABILITY EXPLANATION IN HYBRID PLANNING PROBLEMS finding the inevitable waypoints with the proposed algorithm turns out to be inefficient asymptotically, for problem instances of small size (H with a few locations and for a small depth d), we show empirically that our algorithm can generate inevitable waypoints and the explanation artifact efficiently. Complexity of Reachability Analysis : The model checker Bach performs a path- oriented reachability analysis of a linear hybrid automaton. A path from the initial to the goal location is encoded into a set of linear constraints and consequently solved using a linear programming (LP) problem solver. Details of the path encoding can be found in (Bu et al. [2008], Sarwar et al. [2023a]). Though practical LP solvers use variants of the Simplex algorithm, which runs efficiently on most practical problems, it is not a polynomial-time algorithm in general. In theory, linear programming has been shown to be in class P (Khachiyan [1979, 1980]). Our algorithm calls Bach for each sub-problem in the computed chain. As the length of the computed chain, the LCS, is bounded by the length of the shortest path in PS(Π) (proposition 6.3.4), which is again bounded by d, the complexity is upper bounded by d times the complexity of LP solving. 6.4 Results and Implementation In this section, we present the performance of our framework on several unsolvable hybrid planning problem instances. 6.4.1 Experimental Setup Benchmarks : A brief description of the planning domains is given as follows: Plan- etary rover domain is presented in Section 6.1 and a pictorial overview is shown in Figure 6.1. The planning task for the rover is to reach the base station from its initial location after collecting soil and rock samples from the designated sites. City route- network domain presents a route-network of a city where important places are given as junctions in the network. The planning problem for a battery-powered car is to navig- ate through the city’s route network to reach its destination. Warehouse automation domain (Sarwar et al. [2023a]) represents a scenario where a robot operates to manage the inventories of a warehouse. The floor map of the warehouse is given as grid cells. Few cells in the warehouse are blocked, whereas on a few cells, the robot depletes more energy due to the condition of the surface, such as an oil spillage or being bumpy. The planning problem is for the robot to carry a consignment from its initial location to a goal location. We have crafted warehouse scenarios of varying grid dimensions for evaluating our algorithm. Water-level monitor (Bu et al. [2008]) represents a system that controls the water level in a reservoir. The system goes into an unsafe state if the water level in the reservoir meets underflow or overflow conditions. The planning task is to drive the system to an unsafe state from a given initial state. NAV (Bu et al. [2024]) models the motion of a point robot in a 2-dimensional plane, partitioned into 3 2 rectangular regions, and each such region is associated with a vector field described by the flow equations. The planning problem is to find a trajectory from an initial state to a goal state. NRS (Wang [2005], Bu 134 6.4. RESULTS AND IMPLEMENTATION Benchmarks#Locs#TransDepth|PS(Π)||Y < | #Feas. Exp(Π) TimeMemory wps(in sec)(in MB) Planetary 2540 15244 106Loc13 0.4711.5 rover (PR)20204773.60447.3 City 1025 10468 42Loc7 0.7310.1 route (CR)15921725.931075.8 6x42450 1036 64Loc17 1.578.8 15409984.17507.5 Warehouse 6x63678 12128 4Loc28 0.549.2 17581664.47884.9 automation (WA)8x864100 121697 Loc41 1.3717.5 1710214318.621454.2 10x10100178 122117 Loc57 4.13133.9 15788425.141445.3 Water-level 66 205 32Loc6 0.055.7 monitor (WLM)50120.055.7 NAV924 102325 21Loc6 0.399.5 151497333.41773.2 NRS2730 15312 21Loc25 0.037.6 2078120.1422.1 Table 6.1: Explanation generation on unsolvable planning problem instances. et al. [2024]) represents a nuclear reactor system consisting of 2 rods that absorb neutrons from heavy water when inserted, and a controller that schedules the insertion of the rods into the heavy water. The system is considered safe if there is exactly one rod absorbing neutrons in the heavy water at any instant of time. The planning problem is to find an unsafe execution of the system from a given initial state. A detailed description of the city route and warehouse automation domains is given in the Appendix. We constructed the Planetary rover and the City route-network domains for evaluating our algorithm. The Warehouse automation domain is taken from (Sarwar et al. [2023a]). The rest of the domains are from verification problem instances in linear hybrid systems. For instance, the Water-level monitor domain is a benchmark taken from (Bu et al. [2008]), whereas NAV and NRS are benchmarks taken from Arch-comp 24 pcdb category (Bu et al. [2024]). In these domains, we pose the safety property verification problem as planning problem instances. The planning problems taken for evaluation are all known to be unsolvable. Implementation : All experiments are performed on a machine with 8 GB RAM, Intel Core i5-8250U@1.60GHz, and 8-core processor with Ubuntu 18.04 64-bit OS. All benchmark domains, the problem files, and the code base can be found at: https:// gitlab.com/Sazwar/Sub-goal-Construction. 6.4.2 Evaluation Table 6.1 shows results of our framework on several hybrid systems benchmark domains. Each of the domains is presented with an unsolvable planning problem instance with varying bounds on the plan depth to test the scalability of the framework. Benchmark represents the planning domains together with the planning problem, while #Loc and #Trans report the number of locations and edges in the hybrid automaton of the domain, respectively, showing the size of the domain. Depth presents the bound on the plan length. We have presented results for two plan depths on each domain. |PS(Π)| specifies the number of path strings corresponding to the paths from the initial to the goal location 135 CHAPTER 6. EXPLORING INEVITABLE WAYPOINTS FOR UNSOLVABILITY EXPLANATION IN HYBRID PLANNING PROBLEMS BenchmarksDepth Time (in secs) AT (a) PE(b) Finding Y < (c) RA PR 150.030.010.430.47 201.991.140.473.60 CR 100.030.010.690.73 154.720.230.985.93 WA 6x4 100.010.011.541.57 152.290.161.724.17 6x6 120.010.010.510.54 173.670.040.764.47 8x8 120.050.011.311.37 177.530.031.068.62 10x10 120.520.023.594.13 1521.680.023.4425.14 WLM 200.010.010.030.05 500.010.010.030.05 NAV 100.010.010.370.39 152.780.210.423.41 NRS 150.010.010.010.03 200.120.010.010.14 Table 6.2: Performance analysis of our framework. PE indicates Path-exploration time, RA indicates Reachability analysis time, and AT indicates Accumulative Time = (a) + (b) + (c). of the planning problem instance in the graph structure of the H domain. Recall that we look at all these paths while computing the longest common location sub-sequence, which gives us the inevitable waypoints in the chain of sub-problems in Y < . |Y < | gives us the chain length, which emphasizes the number of inevitable sub-problems detected by our framework, and #Feas. wps denotes the number of solvable sub-problems/waypoints in the chain. Exp(Π) presents the first unsolvable sub-problem in Y < , and thereby, the first infeasible waypoint for the planning problem. A location loc in the Exp(Π) column represents the first sub-problem Π loc in the chain Y < that is unsolvable. For example, Loc13 corresponding to the entry of Planetary rover domain reports the sub-problem Π loc13 as the explanation of unsolvability of the problem instance. Time and Memory report the corresponding execution time and memory usage incurred by our framework. Table 6.2 presents a detailed diagnosis of the execution time taken for explanation generation, showing the time taken for computing all initial to goal paths (PS), Comput- ing a chain of inevitable sub-problems (Y < ), and reachability analysis to find the first unsolvable planning problem in the chain (Explanation(Π)). 136 6.4. RESULTS AND IMPLEMENTATION 6.4.3 Analysis of results Table 6.1 shows that our framework identifies a chain of inevitable waypoints and an explanation of unsolvability efficiently. Performance degrades with an increase in the depth bound of the planning problem instance. This is clearly because increasing depth results in an exponential increase in the number of paths from the initial to the goal location, which also increases the time to compute LCS of path strings. In NAV and NRS, our algorithm reports the trivial chain of waypoints which is visiting the initial location followed by visiting the goal location as inevitable. Note that this is because the graph of these domains do not have any disconnecting articulation point (refer to the discussion section). Memory usage exceeds 500 MB in a few instances. This is because of a bfs (breadth-first search) based path exploration where the size of the bfs queue increases exponentially at each level due to branching factor. Table 6.2 shows the performance of the three major components of the algorithm. The path-exploration time and reachability analysis by the bounded model checker dominates the overall time taken by the algorithm. The results emphasize that it can quickly identify the sub-problems for a planning problem. (a) W 6 is the first unreachable waypoint and serves as an explanation of the plan- ning problem being unsolvable. Warehouse automation domain W1 W2 W3 W4W5W6 123456 7 89101112 1314 15 161718 192021222324 28272625 2930 313233343536 (b) W 4 is the first unreachable waypoint for the planning problem. Figure 6.6: Illustration of results in the example scenarios. In Figure 6.6, we illustrate our method in the example scenarios of Planetary rover and Warehouse automation domains. In the motivating example problem instance in Planetary rover domain, the algorithm identified 8 sub-problems W 1,W 2,...,W 8 in Y < with the total order W 1≤ W 2≤ ...≤ W 8, each representing an inevitable waypoint. Our explanation algorithm detects five waypoints (W 1-W 5) as reachable, depicted by green ticks, and reports W 6 as the first unreachable waypoint shown by a red cross in Figure 6.6a. The unreachability of W 6 can lead the human expert to deduce that the rover’s initial battery charge is insufficient to drive it past the ascending regions in W 4 and W 5. Figure 6.6b shows the identified waypoints and an explanation on a 6×6 warehouse domain for a planning problem where a robot needs to carry a consignment from its initial location to the goal location. In the domain, blue and red cells are the initial and goal 137 CHAPTER 6. EXPLORING INEVITABLE WAYPOINTS FOR UNSOLVABILITY EXPLANATION IN HYBRID PLANNING PROBLEMS locations, respectively. Yellow cells have surfaces with oil-spillage and therefore, the robot has a greater rate of battery depletion in these cells. Grey cells are blocked. The green cell is the only charging station. The explanation algorithm identified 6 sub-problems W 1,W 2,...,W 6 for a planning problem depth bound of 12, and 4 sub-problems W 1 to W 4 for a depth bound of 17, respectively. The waypoints W 5 and W 6 are inevitable only when the depth bound is 12, since a path longer than 12 in length may not mandatorily visit these waypoints. Note that these vertices (W 5 and W 6) are not articulation points, whereas the other waypoints (W 1− W 4) are articulation points of the warehouse grid graph. In both problem instances, our algorithm reports that the robot cannot reach the waypoint W 4 under the dynamics, which is an explanation of unsolvability. A control engineer can deduce that the initial battery charge and the charge capacity of the robot are not sufficient to reach the waypoint directly or via the recharging station. Therefore, a higher charge capacity or a better placement of the charging station close to the waypoint W 4 may be a workaround to make the task solvable. Discussion: Depth has a significant impact on the efficiency of our approach by restricting the number of paths explored in a planning domain. It also has an effect on identifying the waypoints for a problem. For example, the waypoints w5 and w6 are not the articulation points of the 6×6 warehouse domain shown in Figure 6.6b. They appear solely because of the given depth bound on the planning problem. The results presented in Table 6.1 and Table 6.2 highlight the effect of depth. 6.4.4 Domain Descriptions City-network domain: The context for this domain is a car that wants to reach a destination through a route network of a city. Figure 6.7 shows the route network of the city. It has 10 important junctures. Blue-colored and green-colored routes connect these junctures. They respectively represent both-way and one-way traffic in the city. The direction of the traffic in green routes is shown with a directed arrow. The car is initially at juncture A. The juncture A is shown as the green-colored node in the figure. The car has a battery that depletes energy at a constant rate represented by a variable b. Initially, it has 20 units of battery charge available. Similarly, the juncture-to-juncture movement delay for the car is represented by a variable d where each route has a different delay. From a juncture, the car can only move to the adjacent junctures following the route between them. The junctures H (red-colored), I (yellow-colored), and J (blue-colored) are the destinations of three different planning problems of the domain. The orange-colored nodes in the figure are the waypoints that appear in every source-to-destination path for a planning problem of A to H. The routes represent the discrete dynamics of the domain that captures the connectivity of the junctures of the city. The continuous dynamics of the domain involve energy depletion and the juncture-to-juncture movement delay of the car due to different traffic patterns. Warehouse automation domain: We present the context of warehouse automation (Sarwar et al. [2023a]) where a robot operates to manage the inventories of the warehouse. The warehouse is divided into cells. The discrete dynamics here capture the connectivity of the cells along with the presence of objects in certain cells, which are interpreted as 138 6.4. RESULTS AND IMPLEMENTATION Figure 6.7: The city route network is depicted. Each node represents the important junctures of the city. The blue-colored routes represent both-way transportation between junctures. The green-colored routes represent one-way traffic. Initially, the car is at juncture A (shown as the green-colored node). The junctures I (yellow- colored), H (red-colored), and J (blue-colored) are the destinations of three different planning problems of the domain. The orange-colored nodes in the figure are the waypoints that appear in every source-to-destination path for a planning problem of A to H. 19 20 21 22 23 24 13 14 15 16 17 18 7 8 9 10 11 12 1 2 3 4 5 6 Figure 6.8: Warehouse automation domain. obstacles through which the robot cannot move. The movement of the robot is restricted to one of its adjacent cells, and movement to diagonal cells is prohibited. The continuous dynamics capture the battery charge depletion rate of the robot within a cell. Within each cell, the robot follows the dynamics particular to that cell. When the robot makes a transition from one cell to another, it starts to follow the dynamics of the new cell instantaneously. The robot is assigned the task of carrying a consignment to a designated cell while the number of cell visits is restricted to ≤ D cells. The robot starts from the yellow-colored cell where the planning problem requires it to transport the black box to the goal cell (red-colored cell). The robot depletes its charge according to the cell dynamics while on the move. There is a charging station shown as a green-colored cell. The robot may visit this cell to recharge its battery. The grey-colored cells are blocked with obstacles. The robot is equipped with a rechargeable battery. The initial battery charge is 10 units. Each cell has a charge depletion rate of 2 units (modeling the continuous dynamics). In Figure 6.8, we have shown a representation of the warehouse automation domain. Now, consider the planning problem where the robot needs to carry the consignment to the goal from its initial location. Every feasible path for the robot must go through the cells marked with hatched lines (orange-colored) as shown in the figure. We consider these cells as landmarks for the planning problem. A landmark here means a cell that a robot must visit on its way to the goal. 139 CHAPTER 6. EXPLORING INEVITABLE WAYPOINTS FOR UNSOLVABILITY EXPLANATION IN HYBRID PLANNING PROBLEMS 6.5 Related Works Some notable works addressing the unsolvability of planning problems, mostly looked at verifying the unsolvability by generating certificates (Eriksson and Helmert [2020]), (Eriks- son et al. [2017]) or proofs (Eriksson et al. [2018]) rather than explaining the causalities of unsolvability of the planning problem. Such certificates or proofs of unsolvability are not enough to increase the human understandability of why the problem was unsolvable. Most of these works focus on planning problems in discrete domains. Verifying the un- solvability of planning problems in hybrid systems comes with an additional challenge since these planning problems are undecidable in general (Alur et al. [1995a]). In (Sarwar et al. [2023b]), authors provide an approach to addressing the unsolvability of a planning problem in hybrid domains by δ-approximate bounded reachability analysis (Gao et al. [2014]). However, this work also verifies unsolvability rather than explaining it. Few not- able works that are directed towards explaining the unsolvability of a planning problem are, similarly, limited to classical planning problems. Authors in (Göbelbecker et al. [2010]) argue that excuses can be produced by counterfactual alterations to the original planning task such that the new planning task turns out to be solvable, and provides excuses for why a plan cannot be found. In (Eifler et al. [2020]), authors derive properties of a plan which could serve as explanations in case of unsolvability. However, generating excuses, or deriving plan properties in terms of propositional formulas may not be enough to un- derstand why a problem was unsolvable for complex domains like planning problems of hybrid systems which encode mixed discrete and continuous dynamics. In (Vasileiou et al. [2022]), an approach based on knowledge representation and reasoning has been applied to these domains. It provides explanations by finding a subset of the agent’s knowledge base with which to reconcile the human knowledge base for explanations. However, it does not address unsolvability problems, rather, explains why a plan is feasible in a model. A path-oriented reconciliation process between the agent and human models of hybrid sys- tems is provided in (Sarwar et al. [2023a]). It performs the reachability analysis along a path and uses the concept of irreducible infeasible sets (IIS) to generate explanations for unsolvability. In this work, we propose to decompose an unsolvable planning problem into sub- problems motivated by the well-known insight that humans tend to break down sequential planning problems in terms of the sub-problems they need to achieve (Newell and Si- mon [1956], VanLehn [1986]). This has been a popular approach in many domains such as robotics (Krogh and Feng [1989]) and AI (Sutton et al. [1999]) apart from planning (Hoffmann et al. [2004], Lipovetzky and Geffner [2012], Richter et al. [2008]). (Hoffmann et al. [2004], Lipovetzky and Geffner [2012]) find sub-problems for a solvable planning problem of the discrete domains in terms of ordered landmarks. Landmarks are facts given as propositional formulas that must be true at some point in every valid solution plan. In (Sreedharan et al. [2019]), authors use hierarchical model abstractions to re- lax a planning problem until a solution can be found and looks for landmarks of this relaxed problem. They use these landmarks to identify the unachievable sub-problem for the planning problem. These works are in discrete domains. In contrast, our frame- 140 6.6. CONCLUSION work decomposes an unsolvable planning problem of hybrid domains into several smaller sub-problems by reducing it to an instance of longest common subsequence problem and consequently generating explanations using reachability analysis. 6.6 Conclusion In this work, we explore the area of explaining the unsolvability of planning problems for hybrid systems by means of detecting the inevitable sub-problems that must be solvable in order for the bigger problem to be solvable. We show a reduction from the problem of finding sub-problems and an ordering between them to finding the LCS of a finite set of path strings. We present an explanation artifact through these sub-problems and by conducting reachability analysis. Results emphasize that our framework can efficiently identify inevitable sub-problems and the first infeasible one among them as an explanation for unsolvability of a planning problem. We believe that explanations reported by our algorithm can help a control engineer, an AI planner, or a human supervisor to comprehend the cause of unsolvability of the planning problem at hand. 141 CHAPTER 6. EXPLORING INEVITABLE WAYPOINTS FOR UNSOLVABILITY EXPLANATION IN HYBRID PLANNING PROBLEMS 142 Chapter 7 Conclusion and Future Work X AIP has garnered significant research interest due to its role in designing explainable systems. In this dissertation, we look into XAIP from a hybrid system perspective as these systems closely model real-world scenarios. We explain the behavior of such systems. Problems are discussed from two different directions: one that aims to explain when the planning problem is solvable, and there exists an automatically generated plan; the other case tries to explain when the planning problem is unsolvable, and there exists no such plan. We start our investigation with the motion planning problem for an autonomous robot. In chapter 2, we demonstrate a case study on the motion planning problem. We present an integrated software framework for autonomous navigation of an lizard-inspired quadruped robot in an unknown environment. While the framework has three components: SLAM, motion-planning, and control, emphasis remains on motion planning and control. The motion-planning problem is solved by reducing it to the constraint-satisfaction prob- lem, which is solved with a state-of-the-art constraint solver, Z3. A hybrid controller then executes the solution to move the robot efficiently in the environment. We have exper- imented with several planning problem instances in various indoor simulation scenarios, and the results met our objectives. The proposed solution has many interesting applica- tions, including surveillance in an unknown environment, information gathering about a hostage scenario, etc. In chapter 3, we provide a contrastive explanation framework that aims to generate explanations of a plan for a hybrid system planning problem. Given a hybrid system model in PDDL+ and a plan describing the set of desirable actions on the same to achieve a desired goal: 1. Our framework can integrate users’ questions in PDDL+ and synthesize alternate plans using a hypothetical model (HModel) constructed by imposing constraints drawn from the questions. 2. The framework incorporates a re-model and re-plan approach to facilitate explana- tion to the iterative user’s question. While the primary aim of these explanations is to build trust in AI-based systems, they also help to understand the inner dynamics of the planning domain and the planner. 143 CHAPTER 7. CONCLUSION AND FUTURE WORK Additionally, it can identify the modeling flaws to design a better planning model. We present a detailed case study on our approach, with comparison metrics to compare the original plan with the alternate ones. Furthermore, we provide a no-plan explanation algorithm for our unsolvable planning problem instances through bounded reachability analysis. We believe our framework can be of immense importance to the hybrid systems planning community for synthesizing better, explainable plans. At the end of this chapter, we present a web-based contrastive explanation tool that implements the iterative re- modeling and re-planning algorithm, and provides a provision to experiment with different planning domains in hybrid systems and plug in different hybrid system planners as a plan generation engine. Explaining unsolvability has been an interesting direction. In chapter 5, we explore this direction for hybrid system planning problems. We assume that the AI agent and the human have different knowledge bases. While the agent has a complete model of the environment, the human has a partial or erroneous model and expects a plan for the planning problem when there is none. We present a path-based continuous model reconciliation framework that updates the human with the causes of unsolvability to make the human domain consistent with that of the agent. 1. Our approach performs a discrete path analysis to quickly falsify a path by mapping each location and transition of the path in the human model to the agent model. 2. In continuous path analysis, we leverage reachability analysis by combining it with minimal inconsistent constraint sets to find the infeasible path segments for which the paths become unsatisfiable. We demonstrate our work on two hybrid system planning domains (i.e., warehouse auto- mation system and water-level monitoring system). Our framework generates explanations quickly, allowing for swift identification of the reasons behind a plan’s failure. Further- more, its path-oriented analysis thoroughly explores the planning space, enabling it to uncover the majority of factors contributing to the problem’s unsolvability. In chapter 6, we aim to explain unsolvability by identifying the inevitable sub-problems for hybrid system planning problems. The proposed method performs the following: 1. A graph traversal on the abstract graph of the domain to enumerate all feasible paths from source to goal. 2. Identifies inevitable waypoints by casting it to a LCS problem. 3. Decomposes the planning problem into sub-problems based on waypoints. 4. Finally, generate an explanation artifact through these sub-problems by conducting reachability analysis. We present our work on several hybrid system benchmarks, and results emphasize that it can efficiently identify inevitable sub-problems and generate explanation artifacts. We believe that explanations reported by our algorithm can help a control engineer, an AI planner, or a human supervisor to comprehend the cause of unsolvability of the planning problem at hand. 144 Limitation and Future Work. Planning for hybrid systems still holds many open challenges. However, the key issues that are inherently present for such problems are com- plexity and scalability. Our knowledge of the computational complexity of many practical planning fragments is still limited, even in the discrete-time setting. A significant contrib- utor to the complexity of planning for hybrid systems is the use of unbounded numeric and continuous variables, making it challenging to develop procedures with guaranteed termination over potentially infinite state spaces. Despite the inherent complexity of un- bounded numerics, many real-world scenarios can be effectively modeled using numeric variables with predefined bounds. Although initial studies exploring these bounded cases have begun to appear (Gnad et al. [2023], Gigante and Scala [2023]), significant research is still required to understand how these theoretical results can be practically applied through new algorithms and heuristics that exploit these assumptions for more efficient reasoning, and expand to encompass temporal planning. The works discussed so far primarily focus on generating explanations retrospectively, after a plan has been generated (or the search for one has failed). However, explanation can be integrated directly into an agent’s decision-making process. Just as humans tend to make better choices when required to justify them, incorporating this philosophy into XAIP could lead to enhanced, more human-aware systems. This could be one interesting direction to look into in the future. For work presented in chapter 5 and 6, we plan to include provisions for a more fine-grained analysis of the unsolvability and derive causes from the continuous dynamics that are not considered in the current versions. Further- more, we aim to explore a more personalized way of generating explanations. In human interactions, explainers naturally adjust the level of detail and choose a conceptual model they believe will align with the listener’s understanding. Consequently, explanations can be provided at varying levels of abstraction, relying on different conceptual models. For the work presented in chapter 2, we plan to extend the software framework to address dynamic environments and motion planning of a swarm of robots. 145 CHAPTER 7. CONCLUSION AND FUTURE WORK 146 Bibliography Keenan Albee et al. Real-time motion planning in unknown environments for legged robotic planetary exploration. In 2020 IEEE Aerospace Conference, pages 1–9, 2020. doi: 10.1109/AERO47225.2020.9172596. 5, 24 R. Alur, C. Courcoubetis, N. Halbwachs, T.A. Henzinger, P.-H. Ho, X. Nicollin, A. Olivero, J. Sifakis, and S. Yovine. The algorithmic analysis of hybrid systems. Theoret- ical Computer Science, 138(1):3 – 34, 1995a. ISSN 0304-3975. doi: https://doi. org/10.1016/0304-3975(94)00202-T. URL http://w.sciencedirect.com/science/ article/pii/030439759400202T. Hybrid Systems. 4, 19, 41, 62, 77, 106, 118, 124, 140 Rajeev Alur and David L. Dill. A theory of timed automata. Theor. Comput. Sci., 126(2): 183–235, 1994. doi: 10.1016/0304-3975(94)90010-8. URL https://doi.org/10.1016/ 0304-3975(94)90010-8. 14 Rajeev Alur, Costas Courcoubetis, Thomas A. Henzinger, and Pei Hsin Ho. Hybrid auto- mata: An algorithmic approach to the specification and verification of hybrid systems. In Robert L. Grossman, Anil Nerode, Anders P. Ravn, and Hans Rischel, editors, Hy- brid Systems, pages 209–229, Berlin, Heidelberg, 1993. Springer Berlin Heidelberg. ISBN 978-3-540-48060-0. 4, 19, 124 Rajeev Alur, Costas Courcoubetis, Nicolas Halbwachs, Thomas A. Henzinger, Pei-Hsin Ho, Xavier Nicollin, Alfredo Olivero, Joseph Sifakis, and Sergio Yovine. The algorithmic analysis of hybrid systems. Theor. Comput. Sci., 138(1):3–34, 1995b. doi: 10.1016/ 0304-3975(94)00202-T. URL https://doi.org/10.1016/0304-3975(94)00202-T. 13, 14, 20 Rajeev Alur, Costas Courcoubetis, and Thomas A. Henzinger. Computing accumulated delays in real-time systems. Formal Methods Syst. Des., 11(2):137–155, 1997. doi: 10.1023/A:1008626013578. URL https://doi.org/10.1023/A:1008626013578. 15 Pascal Bercher, Susanne Biundo, Thomas Geier, Thilo Hoernle, Florian Nothdurft, Felix Richter, and Bernd Schattenberg. Plan, repair, execute, explain - how planning helps to assemble your home theater. In Steve A. Chien, Minh Binh Do, Alan Fern, and Wheeler Ruml, editors, Proceedings of the Twenty-Fourth International Conference on Automated Planning and Scheduling, ICAPS 2014, Portsmouth, New Hampshire, USA, June 21-26, 2014. AAAI, 2014. URL http://w.aaai.org/ocs/index.php/ICAPS/ ICAPS14/paper/view/7901. 10, 12, 14 147 BIBLIOGRAPHY Lasse Bergroth, Harri Hakonen, and Timo Raita. A survey of longest common subsequence algorithms. In Pablo de la Fuente, editor, Seventh International Symposium on String Processing and Information Retrieval, SPIRE 2000, A Coruña, Spain, September 27-29, 2000, pages 39–48. IEEE Computer Society, 2000. doi: 10.1109/SPIRE.2000.878178. URL https://doi.org/10.1109/SPIRE.2000.878178. 131, 132 Luca Bertazzi and M. Grazia Speranza. Inventory routing problems with multiple cus- tomers. EURO Journal on Transportation and Logistics, 2(3):255–275, 2013. ISSN 2192-4376. doi: https://doi.org/10.1007/s13676-013-0027-z. URL https://w. sciencedirect.com/science/article/pii/S2192437620301199. 22 Amit Bhatia, Lydia E. Kavraki, and Moshe Y. Vardi. Motion planning with hybrid dynamics and temporal goals. In Proceedings of the 49th IEEE Conference on Decision and Control, CDC 2010, December 15-17, 2010, Atlanta, Georgia, USA, pages 1108– 1115. IEEE, 2010. doi: 10.1109/CDC.2010.5717440. URL https://doi.org/10.1109/ CDC.2010.5717440. 6 Julien Bidot, Susanne Biundo, Tobias Heinroth, Wolfgang Minker, Florian Nothdurft, and Bernd Schattenberg. Verbal plan explanations for hybrid planning. In Mat- thias Schumann, Lutz M. Kolbe, Michael H. Breitner, and Arne Frerichs, editors, Multikonferenz Wirtschaftsinformatik, MKWI 2010, Göttingen, Deutschland, 23.- 25.2.2010, Proceedings, pages 2309–2320. Universitätsverlag Göttingen, 2010. URL http://webdoc.sub.gwdg.de/univerlag/2010/mkwi/03_anwendungen/planen_ scheduling/06_verbal_plan_explanations_for_hybrid_plannings.pdf. 10 Armin Biere, Alessandro Cimatti, Edmund M. Clarke, Ofer Strichman, and Yunshan Zhu. Bounded model checking. Adv. Comput., 58:117–148, 2003. doi: 10.1016/ S0065-2458(03)58003-2. URL https://doi.org/10.1016/S0065-2458(03)58003-2. 117 Blai Bonet and Hector Geffner. Planning as heuristic search: New results. In Susanne Biundo and Maria Fox, editors, Recent Advances in AI Planning, 5th European Confer- ence on Planning, ECP’99, Durham, UK, September 8-10, 1999, Proceedings, volume 1809 of Lecture Notes in Computer Science, pages 360–372. Springer, 1999. doi: 10.1007/10720246\_28. URL https://doi.org/10.1007/10720246_28. 40 Lei Bu, You Li, Linzhang Wang, and Xuandong Li. BACH : Bounded reachability checker for linear hybrid automata. In Alessandro Cimatti and Robert B. Jones, editors, Formal Methods in Computer-Aided Design, FMCAD 2008, Portland, Oregon, USA, 17-20 November 2008, pages 1–4. IEEE, 2008. doi: 10.1109/FMCAD.2008.ECP.13. URL https://doi.org/10.1109/FMCAD.2008.ECP.13. 103, 112, 118, 133, 134, 135 Lei Bu, Yang Yang, and Xuandong Li. Iis-guided DFS for efficient bounded reachab- ility analysis of linear hybrid automata. In Kerstin Eder, João Lourenço, and Onn Shehory, editors, Hardware and Software: Verification and Testing - 7th International Haifa Verification Conference, HVC 2011, Haifa, Israel, December 6-8, 2011, Revised 148 BIBLIOGRAPHY Selected Papers, volume 7261 of Lecture Notes in Computer Science, pages 35–49. Springer, 2011. doi: 10.1007/978-3-642-34188-5\_7. URL https://doi.org/10.1007/ 978-3-642-34188-5_7. 114 Lei Bu, Atanu Kundu, Rajarshi Ray, and Yuhui Shi. Arch-comp24 category report: Hybrid systems with piecewise constant dynamics and bounded model checking. In Goran Frehse and Matthias Althoff, editors, Proceedings of the 11th Int. Workshop on Applied Verification for Continuous and Hybrid Systems, volume 103 of EPiC Series in Computing, pages 1–14. EasyChair, 2024. doi: 10.29007/nv67. URL /publications/paper/ZSb4. 134, 135 Lucian Busoniu, Tim de Bruin, Domagoj Tolic, Jens Kober, and Ivana Palunko. Re- inforcement learning for control: Performance, stability, and deep approximators. Annu. Rev. Control., 46:8–28, 2018. doi: 10.1016/J.ARCONTROL.2018.09.005. URL https://doi.org/10.1016/j.arcontrol.2018.09.005. 4 John M. Carroll and Judith Reitman Olson. Chapter 2 - mental models in human- computer interaction. In MARTIN HELANDER, editor, Handbook of Human-Computer Interaction, pages 45–65. North-Holland, Amsterdam, 1988. ISBN 978-0-444-70536- 5. doi: https://doi.org/10.1016/B978-0-444-70536-5.50007-5. URL https://w. sciencedirect.com/science/article/pii/B9780444705365500075. 12 M. Cashmore, M. Fox, D. Long, and D. Magazzeni. A Compilation of the Full PDDL+ Language into SMT. In International Conference on Automated Planning and Schedul- ing, 26:79–87, 3 2016a. 92 Michael Cashmore, Maria Fox, Derek Long, and Daniele Magazzeni. A compilation of the full PDDL+ language into SMT. In Amanda Jane Coles, Andrew Coles, Stefan Edelkamp, Daniele Magazzeni, and Scott Sanner, editors, Proceedings of the Twenty- Sixth International Conference on Automated Planning and Scheduling, ICAPS 2016, London, UK, June 12-17, 2016, pages 79–87. AAAI Press, 2016b. URL http://w. aaai.org/ocs/index.php/ICAPS/ICAPS16/paper/view/13101. 13 Michael Cashmore, Anna Collins, Benjamin Krarup, Senka Krivic, Daniele Magazzeni, and David Smith. Towards explainable ai planning as a service. In 2nd ICAPS Workshop on Explainable Planning, USA, 7 2019. URL https://strathprints.strath.ac.uk/ 69987/. 10, 42, 48, 50, 89, 92, 100 Michael Cashmore, Daniele Magazzeni, and Parisa Zehtabi. Planning for hybrid systems via satisfiability modulo theories. J. Artif. Intell. Res., 67:235–283, 2020. doi: 10.1613/ jair.1.11751. URL https://doi.org/10.1613/jair.1.11751. 4, 40, 93 Tathagata Chakraborti, Sarath Sreedharan, Yu Zhang, and Subbarao Kambhampati. Plan explanations as model reconciliation: Moving beyond explanation as soliloquy. In Carles Sierra, editor, Proceedings of the Twenty-Sixth International Joint Conference on Artifi- cial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 156–163. 149 BIBLIOGRAPHY ijcai.org, 2017. doi: 10.24963/ijcai.2017/23. URL https://doi.org/10.24963/ijcai. 2017/23. 10, 12, 13, 89, 102, 118, 122 Tathagata Chakraborti, Anagha Kulkarni, Sarath Sreedharan, David E. Smith, and Sub- barao Kambhampati. Explicability? Legibility? Predictability? Transparency? Pri- vacy? Security? The Emerging Landscape of Interpretable Agent Behavior. arXiv e-prints, art. arXiv:1811.09722, November 2018. 89 Tathagata Chakraborti, Sarath Sreedharan, Sachin Grover, and Subbarao Kambhampati. Plan explanations as model reconciliation - an empirical study. CoRR, abs/1802.01013, 2018. URL http://arxiv.org/abs/1802.01013. 13 Tathagata Chakraborti, Anagha Kulkarni, Sarath Sreedharan, David E. Smith, and Sub- barao Kambhampati. Explicability? legibility? predictability? transparency? privacy? security? the emerging landscape of interpretable agent behavior. In J. Benton, Nir Lipovetzky, Eva Onaindia, David E. Smith, and Siddharth Srivastava, editors, Pro- ceedings of the Twenty-Ninth International Conference on Automated Planning and Scheduling, ICAPS 2019, Berkeley, CA, USA, July 11-15, 2019, pages 86–96. AAAI Press, 2019a. URL https://ojs.aaai.org/index.php/ICAPS/article/view/3463. 10, 122 Tathagata Chakraborti, Sarath Sreedharan, and Subbarao Kambhampati. Balancing ex- plicability and explanations in human-aware planning. In Sarit Kraus, editor, Pro- ceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 1335–1343. ijcai.org, 2019b. doi: 10.24963/IJCAI.2019/185. URL https://doi.org/10.24963/ijcai.2019/185. 7 Tathagata Chakraborti, Sarath Sreedharan, and Subbarao Kambhampati. The emerging landscape of explainable automated planning & decision making. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 4803–4811. ijcai.org, 2020. doi: 10.24963/IJCAI.2020/ 669. URL https://doi.org/10.24963/ijcai.2020/669. 7, 8, 10, 91, 122 John W. Chinneck. Minos(iis): Infeasibility analysis using minos. Computers & Op- erations Research, 21(1):1–9, 1994. ISSN 0305-0548. doi: https://doi.org/10.1016/ 0305-0548(94)90057-4. URL https://w.sciencedirect.com/science/article/ pii/0305054894900574. 114 John W. Chinneck and Erik W. Dravnieks. Locating minimal infeasible constraint sets in linear programs. INFORMS J. Comput., 3(2):157–168, 1991. doi: 10.1287/ijoc.3.2.157. URL https://doi.org/10.1287/ijoc.3.2.157. 103, 113, 118 Howie Choset, Kevin M. Lynch, Seth Hutchinson, George Kantor, Wolfram Burgard, Lydia E. Kavraki, and Sebastian Thrun. Principles of robot motion: Theory, algorithms, and implementations [book review]. IEEE Robotics & Automation Magazine, 12:110– 110, 2005. URL https://api.semanticscholar.org/CorpusID:10730563. 5 150 BIBLIOGRAPHY Lorenzo Clemente, Slawomir Lasota, and Radoslaw Piórkowski. Determinisability of one- clock timed automata. In Igor Konnov and Laura Kovács, editors, 31st International Conference on Concurrency Theory, CONCUR 2020, September 1-4, 2020, Vienna, Austria (Virtual Conference), volume 171 of LIPIcs, pages 42:1–42:17. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2020. doi: 10.4230/LIPICS.CONCUR.2020.42. URL https://doi.org/10.4230/LIPIcs.CONCUR.2020.42. 14 CPLEX Optimizer Contributors.Cplex. http://w-01.ibm.com/software/ integration/optimization/cplex-optimizer/, Last accessed on Nov 2025. 114 Leonardo de Moura and Nikolaj Bjørner. Z3: An efficient smt solver. In C. R. Ramakrish- nan and Jakob Rehof, editors, Tools and Algorithms for the Construction and Analysis of Systems, pages 337–340, Berlin, Heidelberg, 2008. Springer Berlin Heidelberg. 4, 29 Devdan Dey, Mir Md Sajid Sarwar, Rajarshi Ray, and Ansuman Banerjee. A contrastive explanation tool for plans in hybrid domains. In Sujit Kumar Charkrabarti, Raghavan Komondoor, Raveendra Kumar Medicherla, Aseem Rastogi, and Sudipto Ghosh, editors, Proceedings of the 17th Innovations in Software Engineering Conference, ISEC 2024, Bangalore, India, February 22-24, 2024, pages 15:1–15:5. ACM, 2024. doi: 10.1145/ 3641399.3641424. URL https://doi.org/10.1145/3641399.3641424. 9, 16, 92 Amit Dhurandhar, Pin-Yu Chen, Ronny Luss, Chun-Chen Tu, Pai-Shun Ting, Karthikeyan Shanmugam, and Payel Das. Explanations based on the missing: To- wards contrastive explanations with pertinent negatives. In Advances in Neural In- formation Processing Systems 31: Annual Conference on NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 590–601, 2018. URL https://proceedings.neurips. c/paper/2018/hash/c5f2543b53f4c0ad3819a36752467b-Abstract.html. 12, 89, 100 Amit Dhurandhar, Tejaswini Pedapati, Avinash Balakrishnan, Pin-Yu Chen, Karthikeyan Shanmugam, and Ruchir Puri. Model Agnostic Contrastive Explanations for Structured Data. arXiv e-prints, art. arXiv:1906.00117, May 2019. 12, 89 Marko Djukanovic, Günther R. Raidl, and Christian Blum. Finding longest common subsequences: New anytime a ∗ search results. Appl. Soft Comput., 95:106499, 2020. doi: 10.1016/J.ASOC.2020.106499. URL https://doi.org/10.1016/j.asoc.2020. 106499. 133 Rebecca Eifler, Michael Cashmore, Jörg Hoffmann, Daniele Magazzeni, and Marcel Stein- metz. A new approach to plan-space explanation: Analyzing plan-property dependen- cies in oversubscription planning. In The Thirty-Fourth AAAI Conference on Artifi- cial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Ad- vances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 9818–9826. AAAI Press, 2020. URL https://ojs.aaai.org/index.php/AAAI/ article/view/6534. 11, 12, 89, 118, 140 151 BIBLIOGRAPHY Salomé Eriksson and Malte Helmert. Certified unsolvability for SAT planning with prop- erty directed reachability. In J. Christopher Beck, Olivier Buffet, Jörg Hoffmann, Erez Karpas, and Shirin Sohrabi, editors, Proceedings of the Thirtieth International Con- ference on Automated Planning and Scheduling, Nancy, France, October 26-30, 2020, pages 90–100. AAAI Press, 2020. URL https://ojs.aaai.org/index.php/ICAPS/ article/view/6649. 78, 90, 118, 140 Salomé Eriksson, Gabriele Röger, and Malte Helmert. Unsolvability certificates for clas- sical planning. In Laura Barbulescu, Jeremy Frank, Mausam, and Stephen F. Smith, ed- itors, Proceedings of the Twenty-Seventh International Conference on Automated Plan- ning and Scheduling, ICAPS 2017, Pittsburgh, Pennsylvania, USA, June 18-23, 2017, pages 88–97. AAAI Press, 2017. URL https://aaai.org/ocs/index.php/ICAPS/ ICAPS17/paper/view/15734. 11, 78, 90, 102, 118, 122, 140 Salomé Eriksson, Gabriele Röger, and Malte Helmert. A proof system for unsolvable planning tasks. In Mathijs de Weerdt, Sven Koenig, Gabriele Röger, and Matthijs T. J. Spaan, editors, Proceedings of the Twenty-Eighth International Conference on Automated Planning and Scheduling, ICAPS 2018, Delft, The Netherlands, June 24- 29, 2018, pages 65–73. AAAI Press, 2018. URL https://aaai.org/ocs/index.php/ ICAPS/ICAPS18/paper/view/17760. 11, 102, 118, 122, 140 Kutluhan Erol, James A. Hendler, and Dana S. Nau. UMCP: A sound and complete procedure for hierarchical task-network planning. In Kristian J. Hammond, editor, Proceedings of the Second International Conference on Artificial Intelligence Planning Systems, University of Chicago, Chicago, Illinois, USA, June 13-15, 1994, pages 249– 254. AAAI, 1994. URL http://w.aaai.org/Library/AIPS/1994/aips94-042.php. 2 Richard E. Fikes and Nils J. Nilsson. Strips: A new approach to the application of theorem proving to problem solving. Artificial Intelligence, 2(3):189–208, 1971. ISSN 0004-3702. doi: https://doi.org/10.1016/0004-3702(71)90010-5. URL https://w. sciencedirect.com/science/article/pii/0004370271900105. 2, 40 M. Fox and D. Long. Modelling mixed discrete-continuous domains for planning. Journal of Artificial Intelligence Research, 27:235–297, 10 2006. ISSN 1076-9757. 4, 19, 40, 41, 42, 43, 44, 45, 79, 167 Maria Fox and Derek Long. PDDL2.1: an extension to PDDL for expressing temporal planning domains. J. Artif. Intell. Res., 20:61–124, 2003. doi: 10.1613/jair.1129. URL https://doi.org/10.1613/jair.1129. 92 Maria Fox, Derek Long, and Daniele Magazzeni. Explainable Planning. arXiv e-prints, art. arXiv:1709.10256, September 2017. 88 Maria Fox, Derek Long, and Daniele Magazzeni. Explainable planning. CoRR, abs/1709.10256, 2017. URL http://arxiv.org/abs/1709.10256. 6, 11, 39, 91, 92, 122 152 BIBLIOGRAPHY Thierry Fraichard and Hajime Asama. Inevitable collision states — a step towards safer robots? Advanced Robotics, 18(10):1001–1024, 2004. doi: 10.1163/1568553042674662. URL https://doi.org/10.1163/1568553042674662. 5 Martin Fränzle and Christian Herde. Efficient proof engines for bounded model checking of hybrid systems. In Juan Bicarregui, Andrew Butterfield, and Alvaro Arenas, edit- ors, Proceedings of the Ninth International Workshop on Formal Methods for Industrial Critical Systems, FMICS 2004, Linz, Austria, September 20-21, 2004, volume 133 of Electronic Notes in Theoretical Computer Science, pages 119–137. Elsevier, 2004. doi: 10.1016/j.entcs.2004.08.061. URL https://doi.org/10.1016/j.entcs.2004.08.061. 103, 118 Krishna Gade, Sahin Cem Geyik, Krishnaram Kenthapadi, Varun Mithal, and Ankur Taly. Explainable AI in industry: practical challenges and lessons learned: implications tutorial. In Mireille Hildebrandt, Carlos Castillo, L. Elisa Celis, Salvatore Ruggieri, Linnet Taylor, and Gabriela Zanfir-Fortuna, editors, FAT* ’20: Conference on Fair- ness, Accountability, and Transparency, Barcelona, Spain, January 27-30, 2020, page 699. ACM, 2020. doi: 10.1145/3351095.3375664. URL https://doi.org/10.1145/ 3351095.3375664. 7 Sicun Gao, Jeremy Avigad, and Edmund M. Clarke. δ-complete decision procedures for satisfiability over the reals. In Bernhard Gramlich, Dale Miller, and Uli Sattler, edit- ors, Automated Reasoning, pages 286–300, Berlin, Heidelberg, 2012a. Springer Berlin Heidelberg. ISBN 978-3-642-31365-3. 78 Sicun Gao, Jeremy Avigad, and Edmund M. Clarke. Delta-decidability over the reals. In Proceedings of the 27th Annual IEEE Symposium on Logic in Computer Science, LICS 2012, Dubrovnik, Croatia, June 25-28, 2012, pages 305–314. IEEE Computer Society, 2012b. doi: 10.1109/LICS.2012.41. URL https://doi.org/10.1109/LICS.2012.41. 77, 78 Sicun Gao, Soonho Kong, Wei Chen, and Edmund M. Clarke. Delta-complete analysis for bounded reachability of hybrid systems. CoRR, abs/1404.7171, 2014. URL http: //arxiv.org/abs/1404.7171. 41, 78, 118, 140 Divya Garikapati and Sneha Sudhir Shetiya. Autonomous vehicles: Evolution of artificial intelligence and the current industry landscape. Big Data Cogn. Comput., 8(4):42, 2024. doi: 10.3390/BDCC8040042. URL https://doi.org/10.3390/bdcc8040042. 3 Alfonso Gerevini and Ivan Serina. LPG: A planner based on local search for planning graphs with action costs. In Malik Ghallab, Joachim Hertzberg, and Paolo Traverso, editors, Proceedings of the Sixth International Conference on Artificial Intelligence Plan- ning Systems, April 23-27, 2002, Toulouse, France, pages 13–22. AAAI, 2002. URL http://w.aaai.org/Library/AIPS/2002/aips02-002.php. 40 Malik Ghallab and Hervé Laruelle. Representation and control in ixtet, a temporal planner. In Kristian J. Hammond, editor, Proceedings of the Second International Conference on 153 BIBLIOGRAPHY Artificial Intelligence Planning Systems, University of Chicago, Chicago, Illinois, USA, June 13-15, 1994, pages 61–67. AAAI, 1994. URL http://w.aaai.org/Library/ AIPS/1994/aips94-011.php. 2 Malik Ghallab, Dana S. Nau, and Paolo Traverso. Automated planning - theory and practice. Elsevier, 2004. ISBN 978-1-55860-856-6. 2, 15 Nicola Gigante and Enrico Scala. On the compilability of bounded numeric planning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intel- ligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, China, pages 5341–5349. ijcai.org, 2023. doi: 10.24963/IJCAI.2023/593. URL https://doi.org/10.24963/ ijcai.2023/593. 145 Nicola Gigante, Andrea Micheli, Angelo Montanari, and Enrico Scala. Decidability and complexity of action-based temporal planning over dense time. Artif. Intell., 307: 103686, 2022. doi: 10.1016/J.ARTINT.2022.103686. URL https://doi.org/10.1016/ j.artint.2022.103686. 15 Daniel Gnad, Malte Helmert, Peter Jonsson, and Alexander Shleyfman. Planning over integers: Compilations and undecidability. In Sven Koenig, Roni Stern, and Mauro Val- lati, editors, Proceedings of the Thirty-Third International Conference on Automated Planning and Scheduling, Prague, Czech Republic, July 8-13, 2023, pages 148–152. AAAI Press, 2023. doi: 10.1609/ICAPS.V33I1.27189. URL https://doi.org/10. 1609/icaps.v33i1.27189. 145 Moritz Göbelbecker, Thomas Keller, Patrick Eyerich, Michael Brenner, and Bernhard Nebel. Coming up with good excuses: What to do when no plan can be found. In Ronen I. Brafman, Hector Geffner, Jörg Hoffmann, and Henry A. Kautz, editors, Pro- ceedings of the 20th International Conference on Automated Planning and Scheduling, ICAPS 2010, Toronto, Ontario, Canada, May 12-16, 2010, pages 81–88. AAAI, 2010. URL http://w.aaai.org/ocs/index.php/ICAPS/ICAPS10/paper/view/1453. 11, 89, 102, 118, 122, 140 Bryce Goodman and Seth R. Flaxman. European union regulations on algorithmic decision-making and a "right to explanation". AI Mag., 38(3):50–57, 2017. doi: 10.1609/AIMAG.V38I3.2741. URL https://doi.org/10.1609/aimag.v38i3.2741. 8 Sachin Grover, Sailik Sengupta, Tathagata Chakraborti, Aditya Prasad Mishra, and Sub- barao Kambhampati. RADAR: automated task planning for proactive decision support. Hum. Comput. Interact., 35(5-6):387–412, 2020. doi: 10.1080/07370024.2020.1726751. URL https://doi.org/10.1080/07370024.2020.1726751. 7 David Gunning and David W. Aha. Darpa’s explainable artificial intelligence (XAI) program. AI Mag., 40(2):44–58, 2019. doi: 10.1609/AIMAG.V40I2.2850. URL https://doi.org/10.1609/aimag.v40i2.2850. 1 154 BIBLIOGRAPHY Peter E. Hart, Nils J. Nilsson, and Bertram Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics, 4(2):100–107, 1968. doi: 10.1109/TSSC.1968.300136. 5, 24 Patrik Haslum, Nir Lipovetzky, Daniele Magazzeni, and Christian Muise. Temporal Plan- ning, pages 103–122. Springer International Publishing, Cham, 2019. ISBN 978-3- 031-01584-7. doi: 10.1007/978-3-031-01584-7_5. URL https://doi.org/10.1007/ 978-3-031-01584-7_5. 2 Malte Helmert. The fast downward planning system. J. Artif. Intell. Res., 26:191–246, 2006. doi: 10.1613/jair.1705. URL https://doi.org/10.1613/jair.1705. 40 Thomas A. Henzinger. The theory of hybrid automata. In Proceedings, 11th Annual IEEE Symposium on Logic in Computer Science, New Brunswick, New Jersey, USA, July 27-30, 1996, pages 278–292. IEEE Computer Society, 1996. doi: 10.1109/LICS. 1996.561342. URL https://doi.org/10.1109/LICS.1996.561342. 133 Thomas A. Henzinger, Peter W. Kopke, Anuj Puri, and Pravin Varaiya. What’s decidable about hybrid automata? J. Comput. Syst. Sci., 57(1):94–124, 1998. doi: 10.1006/JCSS. 1998.1581. URL https://doi.org/10.1006/jcss.1998.1581. 15 DS Hirschberg. On Finding Maximal Common Subsequences. Computer Science Labor- atory, Princeton University, 1974. 131 Jörg Hoffmann. F: the fast-forward planning system. AI Mag., 22(3):57–62, 2001. doi: 10.1609/aimag.v22i3.1572. URL https://doi.org/10.1609/aimag.v22i3.1572. 40 Jörg Hoffmann and Daniele Magazzeni. Explainable AI planning (XAIP): overview and the case of contrastive explanation (extended abstract). In Markus Krötzsch and Daria Stepanova, editors, Reasoning Web. Explainable Artificial Intelligence - 15th International Summer School 2019, Bolzano, Italy, September 20-24, 2019, Tu- torial Lectures, volume 11810 of Lecture Notes in Computer Science, pages 277–282. Springer, 2019. doi: 10.1007/978-3-030-31423-1\_9. URL https://doi.org/10.1007/ 978-3-030-31423-1_9. 9, 11, 39, 40, 91, 118, 122 Jörg Hoffmann and Bernhard Nebel. The F planning system: Fast plan generation through heuristic search. J. Artif. Intell. Res., 14:253–302, 2001. doi: 10.1613/jair.855. URL https://doi.org/10.1613/jair.855. 40 Jörg Hoffmann, Julie Porteous, and Laura Sebastia. Ordered landmarks in planning. J. Artif. Intell. Res., 22:215–278, 2004. doi: 10.1613/JAIR.1492. URL https://doi.org/ 10.1613/jair.1492. 14, 140 Adele Howe, Craig Knoblock, Drew McDermott, Ashwin Ram, Manuela Veloso, Daniel Weld, David Wilkins, Anthony Barrett, Dave Christianson, et al. Pddl: The planning domain definition language. Technical report, Technical Report, 1998. 40 155 BIBLIOGRAPHY Qiang Huang, Kazuhito Yokoi, Shuuji Kajita, Kenji Kaneko, Hirohiko Arai, Noriho Koyachi, and Kazuo Tanie. Planning walking patterns for a biped robot. IEEE Trans. Robotics Autom., 17(3):280–289, 2001. doi: 10.1109/70.938385. URL https: //doi.org/10.1109/70.938385. 5 W. N. N. Hung, X. Song, J. Tan, X. Li, J. Zhang, R. Wang, and P. Gao. Motion planning with satisfiability modulo theories. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 113–118, 2014. doi: 10.1109/ICRA.2014.6906597. 5, 24 LINDO SYSTEMS INC. Lindo. https://w.lindo.com/index.php/products/ lindo-api-for-custom-optimization-application, Last accessed on Nov 2025. 114 Lucas Janson, Tommy Hu, and Marco Pavone. Safe motion planning in unknown envir- onments: Optimality benchmarks and tractable policies. CoRR, abs/1804.05804, 2018. URL http://arxiv.org/abs/1804.05804. 5 J. S. Jennings, G. Whelan, and W. F. Evans. Cooperative search and rescue with a team of mobile robots. In 1997 8th International Conference on Advanced Robotics. Proceedings. ICAR’97, pages 193–200, 1997. doi: 10.1109/ICAR.1997.620182. 3, 22 Subbarao Kambhampati. Synthesizing explainable behavior for human-ai collaboration. In Edith Elkind, Manuela Veloso, Noa Agmon, and Matthew E. Taylor, editors, Pro- ceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’19, Montreal, QC, Canada, May 13-17, 2019, pages 1–2. Inter- national Foundation for Autonomous Agents and Multiagent Systems, 2019. URL http://dl.acm.org/citation.cfm?id=3331663. 6 Sertac Karaman and Emilio Frazzoli. Sampling-based algorithms for optimal motion planning. The International Journal of Robotics Research, 30(7):846–894, 2011. doi: 10.1177/0278364911406761. URL https://doi.org/10.1177/0278364911406761. 5, 24 Erez Karpas and Daniele Magazzeni.Automated planning for robot- ics.Annu. Rev. Control. Robotics Auton. Syst., 3:417–439, 2020.doi: 10.1146/ANNUREV-CONTROL-082619-100135. URL https://doi.org/10.1146/ annurev-control-082619-100135. 2 Henry Kautz, Bart Selman, and Joerg Hoffmann. Satplan: Planning as satisfiability. In 5th international planning competition, volume 20, page 156, 2006. 40 Henry A. Kautz and Bart Selman. Planning as satisfiability. In Bernd Neumann, editor, 10th European Conference on Artificial Intelligence, ECAI 92, Vienna, Austria, August 3-7, 1992. Proceedings, pages 359–363. John Wiley and Sons, 1992. 13 KCL-Planning. Pddl+benchmarks. https://github.com/KCL-Planning/SMTPlan/ tree/master/benchmarks, Last accessed on Nov 2025. 79 156 BIBLIOGRAPHY Leonid Genrikhovich Khachiyan. A polynomial algorithm in linear programming. In Doklady Akademii Nauk, volume 244, pages 1093–1096. Russian Academy of Sciences, 1979. 134 L.G. Khachiyan. Polynomial algorithms in linear programming. USSR Computational Mathematics and Mathematical Physics, 20(1):53–72, 1980. ISSN 0041-5553. doi: https://doi.org/10.1016/0041-5553(80)90061-0. URL https://w.sciencedirect. com/science/article/pii/0041555380900610. 134 Omar Zia Khan, Pascal Poupart, and James P. Black. Minimal sufficient explanations for factored markov decision processes. In Alfonso Gerevini, Adele E. Howe, Amedeo Cesta, and Ioannis Refanidis, editors, Proceedings of the 19th International Conference on Automated Planning and Scheduling, ICAPS 2009, Thessaloniki, Greece, September 19-23, 2009. AAAI, 2009. URL http://aaai.org/ocs/index.php/ICAPS/ICAPS09/ paper/view/726. 10 Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Viégas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learn- ing, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 2673–2682. PMLR, 2018. URL http://proceedings.mlr.press/v80/kim18d.html. 9 Joseph Kim, Christian Muise, Ankit Shah, Shubham Agarwal, and Julie Shah. Bayesian inference of linear temporal logic specifications for contrastive explanations. In Sarit Kraus, editor, Proceedings of the Twenty-Eighth International Joint Conference on Ar- tificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 5591–5598. ijcai.org, 2019. doi: 10.24963/IJCAI.2019/776. URL https://doi.org/10.24963/ ijcai.2019/776. 12 Benjamin Krarup, Michael Cashmore, Daniele Magazzeni, and Tim Miller. Model-based contrastive explanations for explainable planning. In ICAPS 2019 Workshop on Explain- able AI Planning (XAIP). AAAI Press, USA, 7 2019. URL https://strathprints. strath.ac.uk/69957/. 12, 40, 48, 89 Benjamin Krarup, Senka Krivic, Daniele Magazzeni, Derek Long, Michael Cashmore, and David E. Smith. Contrastive explanations of plans through model restrictions. CoRR, abs/2103.15575, 2021a. URL https://arxiv.org/abs/2103.15575. 48, 49 Benjamin Krarup, Senka Krivic, Daniele Magazzeni, Derek Long, Michael Cashmore, and David E. Smith. Contrastive explanations of plans through model restrictions. J. Artif. Intell. Res., 72:533–612, 2021b. doi: 10.1613/jair.1.12813. URL https://doi.org/10. 1613/jair.1.12813. 10, 92, 122 B.H. Krogh and D. Feng. Dynamic generation of subgoals for autonomous mobile robots 157 BIBLIOGRAPHY using local feedback information. IEEE Transactions on Automatic Control, 34(5):483– 493, 1989. doi: 10.1109/9.24200. 14, 140 Anagha Kulkarni, Yantian Zha, Tathagata Chakraborti, Satya Gautam Vadlamudi, Yu Zhang, and Subbarao Kambhampati. Explicable planning as minimizing distance from expected behavior. In Edith Elkind, Manuela Veloso, Noa Agmon, and Matthew E. Taylor, editors, Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’19, Montreal, QC, Canada, May 13-17, 2019, pages 2075–2077. International Foundation for Autonomous Agents and Multiagent Systems, 2019. URL http://dl.acm.org/citation.cfm?id=3332015. 13 Nicholas Kushmerick, Steve Hanks, and Daniel S. Weld. An algorithm for probabilistic planning. Artif. Intell., 76(1-2):239–286, 1995. doi: 10.1016/0004-3702(94)00087-H. URL https://doi.org/10.1016/0004-3702(94)00087-H. 2 Pat Langley. Varieties of explainable agency. In In XAIP Workshop, ICAPS 2019, 2019. 7 Steven M. LaValle. Rapidly-exploring random trees: A new tool for path planning. Tech- nical report, Department of Computer Science, Iowa State University, 1998. 5, 6, 24 Steven M. LaValle. Planning algorithms. Cambridge University Press, 2006. ISBN 978-0- 521-86205-9. 5, 6, 24 Xuandong Li, Sumit Jha Aanand, and Lei Bu. Towards an efficient path-oriented tool for bounded reachability analysis of linear hybrid systems using linear programming. In Ofer Strichman and Armin Biere, editors, Proceedings of the Fourth International Workshop on Bounded Model Checking, BMC@FLoC 2006, Seattle, WA, USA, August 15, 2006, volume 174 of Electronic Notes in Theoretical Computer Science, pages 57–70. Elsevier, 2006. doi: 10.1016/j.entcs.2006.12.023. URL https://doi.org/10.1016/j. entcs.2006.12.023. 113, 133 Brian Y. Lim, Anind K. Dey, and Daniel Avrahami. Why and why not explana- tions improve the intelligibility of context-aware intelligent systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’09, page 2119–2128, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605582467. doi: 10.1145/1518701.1519023. URL https://doi.org/10.1145/ 1518701.1519023. 88 Nir Lipovetzky and Hector Geffner. Width and serialization of classical planning problems. In Luc De Raedt, Christian Bessiere, Didier Dubois, Patrick Doherty, Paolo Frasconi, Fredrik Heintz, and Peter J. F. Lucas, editors, ECAI 2012, volume 242 of Frontiers in Artificial Intelligence and Applications, pages 540–545. IOS Press, 2012. doi: 10.3233/ 978-1-61499-098-7-540. URL https://doi.org/10.3233/978-1-61499-098-7-540. 14, 140 158 BIBLIOGRAPHY Shuai Liu and Pengcheng Liu. A review of motion planning algorithms for robotic arm systems. Lecture Notes in Mechanical Engineering, 2021. URL https://api. semanticscholar.org/CorpusID:242941370. 5 Sikang Liu, Nikolay Atanasov, Kartik Mohta, and Vijay Kumar. Search-based mo- tion planning for quadrotors using linear quadratic minimum time control. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2017, Vancouver, BC, Canada, September 24-28, 2017, pages 2872–2879. IEEE, 2017. doi: 10.1109/IROS.2017.8206119. URL https://doi.org/10.1109/IROS.2017.8206119. 5 Mauricio Cecilio Magnaguagno, Ramon Fraga Pereira, Martin D. Móre, and Felipe Me- neguzzi. Web planner: A tool to develop, visualize, and test classical planning domains. In Mauro Vallati and Diane E. Kitchin, editors, Knowledge Engineering Tools and Tech- niques for AI Planning, pages 209–227. Springer, 2020. doi: 10.1007/978-3-030-38561-3\ _11. URL https://doi.org/10.1007/978-3-030-38561-3_11. 10 David Maier. The complexity of some problems on subsequences and supersequences. J. ACM, 25(2):322–336, 1978. doi: 10.1145/322063.322075. URL https://doi.org/10. 1145/322063.322075. 131, 133 Deborah L. McGuinness, Alyssa Glass, Michael Wolverton, and Paulo Pinheiro da Silva. Explaining task processing in cognitive assistants that learn. In David Wilson and Geoff Sutcliffe, editors, Proceedings of the Twentieth International Florida Artificial Intelligence Research Society Conference, May 7-9, 2007, Key West, Florida, USA, pages 284–289. AAAI Press, 2007. URL http://w.aaai.org/Library/FLAIRS/ 2007/flairs07-059.php. 10 Tim Miller. Contrastive Explanation: A Structural-Model Approach. arXiv e-prints, art. arXiv:1811.03163, November 2018. 88 Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artif. Intell., 267:1–38, 2019. doi: 10.1016/J.ARTINT.2018.07.007. URL https://doi.org/ 10.1016/j.artint.2018.07.007. 8, 9, 11, 12, 39, 92 Tim Miller. Contrastive explanation: a structural-model approach. Knowl. Eng. Rev., 36:e14, 2021. doi: 10.1017/S0269888921000102. URL https://doi.org/10.1017/ S0269888921000102. 9, 99 Shane T. Mueller, Robert R. Hoffman, William J. Clancey, Abigail Emrey, and Gary Klein. Explanation in human-ai systems: A literature meta-review, synopsis of key ideas and publications, and bibliography for explainable AI. CoRR, abs/1902.01876, 2019. URL http://arxiv.org/abs/1902.01876. 11, 91 A. Newell and H. Simon. The logic theory machine–a complex information processing system. IRE Transactions on Information Theory, 2(3):61–79, 1956. doi: 10.1109/TIT. 1956.1056797. 14, 122, 140 159 BIBLIOGRAPHY Xavier Nicollin, Alfredo Olivero, Joseph Sifakis, and Sergio Yovine. An approach to the description and analysis of hybrid systems. In Robert L. Grossman, Anil Nerode, Anders P. Ravn, and Hans Rischel, editors, Hybrid Systems, volume 736 of Lecture Notes in Computer Science, pages 149–178. Springer, 1992. doi: 10.1007/3-540-57318-6\_28. URL https://doi.org/10.1007/3-540-57318-6_28. 15 S. Nishad, R. Halder, G. Banda, R. Ray, A. Bhattacharya, and A. Thakur. A lizard- inspired quadruped robot based on pressure sensitive adhesion mechanism for wall climbing. In Proceedings of the 5th International Conference on Advances in Robot- ics (AIR 2021), IIT Kanpur, India, June 30–July 4 2021. (w.iitp.ac.in/~halder/ IRIA2021/AIR2021.pdf). 22 Florian Nothdurft, Gregor Behnke, Pascal Bercher, Susanne Biundo, and Wolfgang Minker. The interplay of user-centered dialog systems and AI planning. In Proceed- ings of the SIGDIAL 2015 Conference, The 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 2-4 September 2015, Prague, Czech Republic, pages 344–353. The Association for Computer Linguistics, 2015. doi: 10.18653/V1/W15-4646. URL https://doi.org/10.18653/v1/w15-4646. 10 Stefan Panjkovic, Andrea Micheli, and Alessandro Cimatti. Deciding unsolvability in tem- poral planning under action non-self-overlapping. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applica- tions of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022, pages 9886–9893. AAAI Press, 2022. doi: 10.1609/AAAI.V36I9.21225. URL https://doi.org/10.1609/aaai.v36i9.21225. 15 Edwin P. D. Pednault. Adl: Exploring the middle ground between strips and the situation calculus. In Proceedings of the First International Conference on Principles of Know- ledge Representation and Reasoning, page 324–332, San Francisco, CA, USA, 1989. Morgan Kaufmann Publishers Inc. ISBN 1558600329. 40 Giuseppe Della Penna, Daniele Magazzeni, and Fabio Mercorio. A universal plan- ning system for hybrid domains. Appl. Intell., 36(4):932–959, 2012. doi: 10.1007/ s10489-011-0306-z. URL https://doi.org/10.1007/s10489-011-0306-z. 40 Francesco Percassi, Enrico Scala, and Mauro Vallati. Translations from discretised PDDL+ to numeric planning. In Susanne Biundo, Minh Do, Robert Goldman, Michael Katz, Qiang Yang, and Hankz Hankui Zhuo, editors, Proceedings of the Thirty-First Inter- national Conference on Automated Planning and Scheduling, ICAPS 2021, Guang- zhou, China (virtual), August 2-13, 2021, pages 252–261. AAAI Press, 2021. URL https://ojs.aaai.org/index.php/ICAPS/article/view/15969. 44 Wiktor Mateusz Piotrowski. Heuristics for AI planning in hybrid systems. PhD thesis, King’s College London, UK, 2018. URL https://ethos.bl.uk/OrderDetails.do? uin=uk.bl.ethos.754884. 4 160 BIBLIOGRAPHY Wiktor Mateusz Piotrowski, Maria Fox, Derek Long, Daniele Magazzeni, and Fabio Mer- corio. Heuristic planning for hybrid systems. In Dale Schuurmans and Michael P. Wellman, editors, Proceedings of the Thirtieth AAAI Conference on Artificial Intelli- gence, February 12-17, 2016, Phoenix, Arizona, USA, pages 4254–4255. AAAI Press, 2016. URL http://w.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12394. 20, 40 Erion Plaku, Lydia E. Kavraki, and Moshe Y. Vardi. Hybrid systems: from verification to falsification by combining motion planning and discrete search. Formal Methods Syst. Des., 34(2):157–182, 2009. doi: 10.1007/S10703-008-0058-5. URL https://doi.org/ 10.1007/s10703-008-0058-5. 6 David Portugal and Rui P. Rocha. Cooperative multi-robot patrol with bayesian learn- ing. Auton. Robots, 40(5):929–953, June 2016. ISSN 0929-5593. doi: 10.1007/ s10514-015-9503-7. URL https://doi.org/10.1007/s10514-015-9503-7. 22 David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind? Be- havioral and Brain Sciences, 1(4):515–526, 1978. doi: 10.1017/S0140525X00076512. 12 Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should I trust you?": Ex- plaining the predictions of any classifier. In Balaji Krishnapuram, Mohak Shah, Alexan- der J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi, editors, Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 1135–1144. ACM, 2016. doi: 10.1145/2939672.2939778. URL https://doi.org/10.1145/2939672.2939778. 9 Arthur Richards and Jonathan P. How. Aircraft trajectory planning with collision avoid- ance using mixed integer linear programming. In American Control Conference, ACC 2002, Anchorage, Alaska, USA, May 8-10 2002, pages 1936–1941. IEEE, 2002. doi: 10.1109/ACC.2002.1023918. URL https://doi.org/10.1109/ACC.2002.1023918. 4 Silvia Richter, Malte Helmert, and Matthias Westphal. Landmarks revisited. In Di- eter Fox and Carla P. Gomes, editors, Proceedings of the Twenty-Third AAAI Confer- ence on Artificial Intelligence, AAAI 2008, Chicago, Illinois, USA, July 13-17, 2008, pages 975–982. AAAI Press, 2008. URL http://w.aaai.org/Library/AAAI/2008/ aaai08-155.php. 14, 140 Jussi Rintanen. Complexity of concurrent temporal planning. In Mark S. Boddy, Maria Fox, and Sylvie Thiébaux, editors, Proceedings of the Seventeenth Interna- tional Conference on Automated Planning and Scheduling, ICAPS 2007, Providence, Rhode Island, USA, September 22-26, 2007, pages 280–287. AAAI, 2007. URL http: //w.aaai.org/Library/ICAPS/2007/icaps07-036.php. 15 Stuart Russell and Peter Norvig. Artificial Intelligence: A Modern Approach (4th Edition). Pearson, 2020. ISBN 9780134610993. URL http://aima.cs.berkeley.edu/. 15 161 BIBLIOGRAPHY Tomasz Rybus. Point-to-point motion planning of a free-floating space manipulator using the rapidly-exploring random trees (RRT) method. Robotica, 38(6):957–982, 2020. doi: 10.1017/S0263574719001176. URL https://doi.org/10.1017/S0263574719001176. 5 Waddah Saeed and Christian W. Omlin. Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities. Knowl. Based Syst., 263:110273, 2023. doi: 10.1016/J.KNOSYS.2023.110273. URL https://doi.org/10.1016/j.knosys. 2023.110273. 7 I. Saha, R. Ramaithitima, V. Kumar, G. J. Pappas, and S. A. Seshia. Automated compos- ition of motion primitives for multi-robot systems from safe ltl specifications. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1525– 1532, 2014. doi: 10.1109/IROS.2014.6942758. 5, 24, 27 Wojciech Samek and Klaus-Robert Müller. Towards explainable artificial intelligence. In Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller, editors, Explainable AI: Interpreting, Explaining and Visualiz- ing Deep Learning, volume 11700 of Lecture Notes in Computer Science, pages 5–22. Springer, 2019. doi: 10.1007/978-3-030-28954-6\_1. URL https://doi.org/10.1007/ 978-3-030-28954-6_1. 8 Mir Md Sajid Sarwar and Rajarshi Ray. Exploring inevitable waypoints for unsolvability explanation in hybrid planning problems. ACM Trans. Embed. Comput. Syst., 24(6), October 2025. ISSN 1539-9087. doi: 10.1145/3767745. URL https://doi.org/10. 1145/3767745. 17 Mir Md Sajid Sarwar, Rajarshi Ray, and Ansuman Banerjee. A contrastive plan explana- tion framework for hybrid system models. In 18th ACM/IEEE International Conference on Formal Methods and Models for System Design, MEMOCODE 2020, Jaipur, India, December 2-4, 2020, pages 1–11. IEEE, 2020. doi: 10.1109/MEMOCODE51338.2020. 9315040. URL https://doi.org/10.1109/MEMOCODE51338.2020.9315040. 16 Mir Md Sajid Sarwar, Rajeshwar Yadav, Sudip Samanta, Rajarshi Ray, Raju Halder, Gourinath Banda, Ansuman Bhattacharya, and Atul Thakur. A robotic software frame- work for autonomous navigation in unknown environment. In 2021 International Sym- posium of Asian Control Association on Intelligent Robotics and Industrial Automation (IRIA), pages 345–350, 2021. doi: 10.1109/IRIA53009.2021.9588693. 16 Mir Md Sajid Sarwar, Rajarshi Ray, and Ansuman Banerjee. Explaining unsolvability of planning problems in hybrid systems with model reconciliation. In Reinhard von Hanxleden, Stephen A. Edwards, Jens Brandt, and Qi Zhu, editors, 21st ACM-IEEE International Symposium on Formal Methods and Models for System Design, MEMO- CODE 2023, Hamburg, Germany, September 21-22, 2023, pages 47–58. ACM / IEEE, 2023a. URL https://ieeexplore.ieee.org/document/10316224. 16, 134, 135, 138, 140 162 BIBLIOGRAPHY Mir Md Sajid Sarwar, Rajarshi Ray, and Ansuman Banerjee. A contrastive plan explan- ation framework for hybrid system models. ACM Trans. Embed. Comput. Syst., 22(2): 22:1–22:51, 2023b. doi: 10.1145/3561532. URL https://doi.org/10.1145/3561532. 16, 107, 118, 122, 126, 140 MMS Sarwar, R. Ray, and A. Banerjee. A Contrastive Plan Explanation Framework for Hybrid System Models. ACM TECS, 9 2022. 92, 93, 100 Enrico Scala, Patrik Haslum, Sylvie Thiébaux, and Miquel Ramírez. Interval-based re- laxation for general numeric planning. In Gal A. Kaminka, Maria Fox, Paolo Bouquet, Eyke Hüllermeier, Virginia Dignum, Frank Dignum, and Frank van Harmelen, editors, ECAI 2016 - 22nd European Conference on Artificial Intelligence, 29 August-2 Septem- ber 2016, The Hague, The Netherlands - Including Prestigious Applications of Artificial Intelligence (PAIS 2016), volume 285 of Frontiers in Artificial Intelligence and Applic- ations, pages 655–663. IOS Press, 2016. doi: 10.3233/978-1-61499-672-9-655. URL https://doi.org/10.3233/978-1-61499-672-9-655. 4, 40, 78, 93 Jürgen Schmidhuber. An on-line algorithm for dynamic reinforcement learning and plan- ning in reactive environments. In IJCNN 1990, International Joint Conference on Neural Networks, San Diego, CA, USA, June 17-21, 1990, pages 253–258. IEEE, 1990. doi: 10.1109/IJCNN.1990.137723. URL https://doi.org/10.1109/IJCNN. 1990.137723. 2 Matthew C. Schmidt, Christopher D. Abraham, Jiayi Huang, Clifford G. Robinson, Geof- frey Hugo, Nels C. Knutson, Baozhou Sun, Chipo Raranje, Erno Sajo, Piotr Zyg- manski, Marian Jandel, Peter Szentivanyi, Jessica Hilliard, Jessica Hamilton, and Francisco J. Reynoso. Clinical application of a template-guided automated planning routine. Journal of Applied Clinical Medical Physics, 24(3):e13837, 2023. doi: https: //doi.org/10.1002/acm2.13837. URL https://aapm.onlinelibrary.wiley.com/doi/ abs/10.1002/acm2.13837. 3 Bastian Seegebarth, Felix Müller, Bernd Schattenberg, and Susanne Biundo. Making hybrid plans more clear to human users - A formal approach for generating sound ex- planations. In Lee McCluskey, Brian Charles Williams, José Reinaldo Silva, and Blai Bonet, editors, Proceedings of the Twenty-Second International Conference on Auto- mated Planning and Scheduling, ICAPS 2012, Atibaia, São Paulo, Brazil, June 25- 19, 2012. AAAI, 2012. URL http://w.aaai.org/ocs/index.php/ICAPS/ICAPS12/ paper/view/4691. 10, 14 Aleksandr Segal, Dirk Haehnel, and Sebastian Thrun. Generalized-icp. In Robotics: sci- ence and systems, volume 2, page 435. Seattle, WA, 2009. 24 Ji-Ae Shin and Ernest Davis. Processes and continuous change in a sat-based planner. Artif. Intell., 166(1-2):194–253, 2005. doi: 10.1016/j.artint.2005.04.001. URL https: //doi.org/10.1016/j.artint.2005.04.001. 44 163 BIBLIOGRAPHY David Smith. Planning as an iterative process. Proceedings of the AAAI Conference on Artificial Intelligence, 26(1):2180–2185, 9 2021. doi: 10.1609/aaai.v26i1.8449. URL https://ojs.aaai.org/index.php/AAAI/article/view/8449. 100 David E. Smith. Planning as an iterative process. In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, AAAI’12, page 2180–2185. AAAI Press, 2012. 9, 89 David E. Smith and Daniel S. Weld. Conformant graphplan. In Proceedings of the Fifteenth National/Tenth Conference on Artificial Intelligence/Innovative Applications of Artifi- cial Intelligence, AAAI ’98/IAAI ’98, page 889–896, USA, 1998. American Association for Artificial Intelligence. ISBN 0262510987. 40 Shirin Sohrabi, Jorge A. Baier, and Sheila A. McIlraith. Preferred explanations: Theory and generation via planning. In Wolfram Burgard and Dan Roth, editors, Proceed- ings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2011, San Francisco, California, USA, August 7-11, 2011, pages 261–267. AAAI Press, 2011. doi: 10.1609/AAAI.V25I1.7845. URL https://doi.org/10.1609/aaai.v25i1.7845. 10 Sarath Sreedharan, Tathagata Chakraborti, and Subbarao Kambhampati. Handling model uncertainty and multiplicity in explanations via model reconciliation. In Mathijs de Weerdt, Sven Koenig, Gabriele Röger, and Matthijs T. J. Spaan, editors, Proceedings of the Twenty-Eighth International Conference on Automated Planning and Scheduling, ICAPS 2018, Delft, The Netherlands, June 24-29, 2018, pages 518–526. AAAI Press, 2018. URL https://aaai.org/ocs/index.php/ICAPS/ICAPS18/paper/view/17783. 12, 13 Sarath Sreedharan, Siddharth Srivastava, David E. Smith, and Subbarao Kambhampati. Why can’t you do that hal? explaining unsolvability of planning tasks. In Sarit Kraus, editor, Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 1422–1430. ijcai.org, 2019. doi: 10.24963/ijcai.2019/197. URL https://doi.org/10.24963/ijcai.2019/ 197. 9, 11, 14, 89, 118, 122, 140 Sarath Sreedharan, Tathagata Chakraborti, Christian Muise, Yasaman Khazaeni, and Subbarao Kambhampati. - D3WA+ - A case study of XAIP in a model acquisition task for dialogue planning. In J. Christopher Beck, Olivier Buffet, Jörg Hoffmann, Erez Karpas, and Shirin Sohrabi, editors, Proceedings of the Thirtieth International Con- ference on Automated Planning and Scheduling, Nancy, France, October 26-30, 2020, pages 488–498. AAAI Press, 2020. URL https://ojs.aaai.org/index.php/ICAPS/ article/view/6744. 7 Anthony Stentz. Optimal and efficient path planning for unknown and dynamic envir- onments. INTERNATIONAL JOURNAL OF ROBOTICS AND AUTOMATION, 10: 89–100, 1993. 5, 24 164 BIBLIOGRAPHY Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artif. Intell., 112(1- 2):181–211, 1999. doi: 10.1016/S0004-3702(99)00052-1. URL https://doi.org/10. 1016/S0004-3702(99)00052-1. 14, 140 Siyu Teng, Xuemin Hu, Peng Deng, Bai Li, Yuchen Li, Yunfeng Ai, Dongsheng Yang, Lingxi Li, Zhe Xuanyuan, Fenghua Zhu, and Long Chen. Motion planning for autonom- ous driving: The state of the art and future perspectives. IEEE Trans. Intell. Veh., 8(6): 3692–3711, 2023. doi: 10.1109/TIV.2023.3274536. URL https://doi.org/10.1109/ TIV.2023.3274536. 5 S. Thrun. Particle filters in robotics. In Proceedings of the 17th Annual Conference on Uncertainty in AI (UAI), 2002a. 25 Sebastian Thrun. Probabilistic robotics. Communications of the ACM, 45(3):52–57, 2002b. 25 Kurt VanLehn. John r. anderson, the architecture of cognition. Artif. Intell., 28(2): 235–240, 1986. doi: 10.1016/0004-3702(86)90084-6. URL https://doi.org/10.1016/ 0004-3702(86)90084-6. 140 Stylianos Loukas Vasileiou, William Yeoh, Tran Cao Son, Ashwin Kumar, Michael Cash- more, and Daniele Magazzeni. A logic-based explanation generation framework for classical and hybrid planning problems. J. Artif. Intell. Res., 73:1473–1534, 2022. doi: 10.1613/jair.1.13431. URL https://doi.org/10.1613/jair.1.13431. 13, 14, 118, 140 Farn Wang. Symbolic parametric safety analysis of linear hybrid systems with bdd-like data-structures. IEEE Trans. Softw. Eng., 31(1):38–51, January 2005. ISSN 0098-5589. doi: 10.1109/TSE.2005.13. URL https://doi.org/10.1109/TSE.2005.13. 134 Nan Wang and Ricardo G. Sanfelice. Motion planning for hybrid dynamical sys- tems: Framework, algorithm template, and a sampling-based approach. CoRR, abs/2406.01802, 2024. doi: 10.48550/ARXIV.2406.01802. URL https://doi.org/ 10.48550/arXiv.2406.01802. 6, 22 Georgios N. Yannakakis. Game AI revisited. In John Feo, Paolo Faraboschi, and Oreste Villa, editors, Proceedings of the Computing Frontiers Conference, CF’12, Caligari, Italy - May 15 - 17, 2012, pages 285–292. ACM, 2012. doi: 10.1145/2212908.2212954. URL https://doi.org/10.1145/2212908.2212954. 3 Zahra Zahedi, Alberto Olmo Hernandez, Tathagata Chakraborti, Sarath Sreedharan, and Subbarao Kambhampati. Towards understanding user preferences for explanation types in model reconciliation. In 14th ACM/IEEE International Conference on Human-Robot Interaction, HRI 2019, Daegu, South Korea, March 11-14, 2019, pages 648–649. IEEE, 2019. doi: 10.1109/HRI.2019.8673097. URL https://doi.org/10.1109/HRI.2019. 8673097. 13 165 BIBLIOGRAPHY Jiaming Zha and Mark W. Mueller. Exploiting collisions for sampling-based multicopter motion planning. In IEEE International Conference on Robotics and Automation, ICRA 2021, Xi’an, China, May 30 - June 5, 2021, pages 7943–7949. IEEE, 2021. doi: 10.1109/ICRA48506.2021.9561166. URL https://doi.org/10.1109/ICRA48506. 2021.9561166. 22 Yu Zhang, Sarath Sreedharan, Anagha Kulkarni, Tathagata Chakraborti, Hankz Hankui Zhuo, and Subbarao Kambhampati. Plan explicability and predictability for robot task planning. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 1313–1320, 2017. doi: 10.1109/ICRA.2017.7989155. 91 Ellin Zhao and Roykrong Sukkerd. Interactive explanation for planning-based systems: WIP abstract. In Xue Liu, Paulo Tabuada, Miroslav Pajic, and Linda Bushnell, ed- itors, Proceedings of the 10th ACM/IEEE International Conference on Cyber-Physical Systems, ICCPS 2019, Montreal, QC, Canada, April 16-18, 2019, pages 322–323. ACM, 2019. doi: 10.1145/3302509.3313322. URL https://doi.org/10.1145/3302509. 3313322. 12 166 Chapter 8 Appendix 8.1 Car Domain ( define ( problem car_prob ) ( : domain car ) ( : i n i t ( running ) (= ( runningTime ) 0) (= ( upLimit ) 2) (= ( downLimit ) −2) (= d 0) (= a 0) (= v 0)) ( : goal ( and ( goalReached ) ( not ( engineBlown )) (<= ( runningTime ) 50))) Listing 8.1: The planning problem instance 2 in the car domain. 8.2 Generator Events Domain Hybrid Automatom Model of Generator-events Domain: To represent the generator-events domain (see 3.5.1) for the planning problem instance 1 into a hybrid automaton model, the behaviour of the durative-action generate is captured with the start-process-stop paradigm in PDDL+ Fox and Long [2006]. The durative-action generate can be viewed as a sequence of four distinct but causally related parts: an instantaneous-action generateStart, a pro- cess generateProcess, another instantaneous-action generateEnd and an event generateFail. The generateStart marks the start of the generate action (shown by the name extension St in Figure 8.1), it then activates the generateProcess which in turn starts to decrease the fuelLevel of the generator at a constant rate. The time elapsed is captured by the variable d. The generateEnd updates (generatorRan) to true when d becomes 1000. The event that is a part of the start-process-stop model monitors any violation of invariant of the durative-action. In our model, the event generateFail is monitoring only the generator underflow condition (where fuelLevel becomes zero), since the invariant (safe gen) of the generate action is already being monitored by the event generatorOverflow that observes the generator overflow condition. The corresponding hybrid automaton model for the planning problem instance is depicted in Figure 8.1. The locations in the automata are formed with subsets of predicate symbols that hold on that location along with the name extension St (indicating the generate action has started) whenever applicable. 167 CHAPTER 8. APPENDIX ( define ( domain generatorplus ) ( : requirements : f l u e n t s : durative−actions : duration−i n e q u a l i t i e s : adl : typing : time ) ( : types generator tank ) ( : predicates ( generator−ran ) ( a v a i l a b l e ? t − tank ) ( using ? t − tank ?g − generator ) ( s a f e ?g − generator )) ( : functions ( fuelLevel ?g − generator ) ( capacity ?g − generator ) ( fuelInTank ? t − tank ) ( ptime ? t − tank )) ( : durative−action generate : parameters (? g − generator ) : duration (= ? duration 1000) : condition ( and ( over a l l (>= ( fuelLevel ?g ) 0)) ( over a l l ( s a f e ?g ) ) ) : e f f e c t ( and ( decrease ( fuelLevel ?g ) (∗ #t 1)) ( at end ( generator−ran ) ) ) ) ( : action r e f u e l : parameters (? g − generator ? t − tank ) : precondition ( and ( not ( using ? t ?g )) ( a v a i l a b l e ? t )) : e f f e c t ( and ( using ? t ?g ) ( not ( a v a i l a b l e ? t ) ) ) ) ( : process r e f u e l l i n g : parameters (? g − generator ? t −tank ) : precondition ( and ( using ? t ?g )) : e f f e c t ( and ( increase ( ptime ? t ) (∗ #t 1)) ( decrease ( fuelInTank ? t ) (∗ #t (∗ 0.001 (∗ ( ptime ? t ) ( ptime ? t ) ) ) ) ) ( increase ( fuelLevel ?g ) (∗ #t (∗ 0.001 (∗ ( ptime ? t ) ( ptime ? t ) ) ) ) ) ) ) ( : event tankEmpty : parameters (? g − generator ? t − tank ) : precondition ( and ( using ? t ?g ) (<= ( fuelInTank ? t ) 0)) : e f f e c t ( and ( not ( using ? t ?g ) ) ) ) ( : event generatorOverflow : parameters (? g − generator ) : precondition ( and (> ( fuelLevel ?g ) ( capacity ?g )) ( s a f e ?g )) : e f f e c t ( and ( not ( s a f e ?g ) ) ) ) ) Listing 8.2: The generator-events domain in PDDL+. ( define ( problem run−generatorplus ) ( : domain generatorplus ) ( : objects gen − generator t1 t2 − tank ) ( : i n i t (= ( fuelLevel gen ) 940) (= ( capacity gen ) 1600) (= ( fuelInTank t1 ) 40) (= ( fuelInTank t2 ) 40) ( a v a i l a b l e t1 ) ( a v a i l a b l e t2 ) ( s a f e gen )) ( : goal ( generator−ran ) ) ) Listing 8.3: The planning problem instance 1 in the generator-events domain in PDDL+. 168 8.2. GENERATOR EVENTS DOMAIN ( define ( problem run−generatorplus ) ( : domain generatorplus ) ( : objects gen − generator t1 t2 t3 − tank ) ( : i n i t (= ( fuelLevel gen ) 900) (= ( capacity gen ) 1600) (= ( fuelInTank t1 ) 40) (= ( fuelInTank t2 ) 40) (= ( fuelInTank t1 ) 40) ( a v a i l a b l e t1 ) ( a v a i l a b l e t2 ) ( a v a i l a b l e t3 ) ( s a f e gen )) ( : goal ( generator−ran ) ) ) Listing 8.4: The planning problem instance 2 in the generator-events domain in PDDL+. Sf-Nut1-Nut2 Sf-Nut1-Nut2-St Sf-Ut1-Nut2 Sf-Ut1-Ut2 Sf-Nut1-Ut2 Sf-Ut1-Nut2-St Sf-Nut1-Ut2-St notSafe-Fail generatorRan Sf-Ut1-Ut2-St (refuel t1) (available t1) not(available t1) (refuel t1, t2) (available t1) & (available t2) not(available t1)not(available t2) (refuel t2) (available t2) not(available t2) (tankEmpty t1) (ft1<=0) (tankEmpty t2) (ft2<=0) (refuel t2) (available t2) not(available t2) (tankEmpty t2) (ft2<=0) (tankEmpty t1) (ft1<=0) (refuel t1) (available t1) not(available t1) generateStart (fl>=0) (d=0) generateEnd (fl>=0) & (d==1000) (refuel t1) (available t1) not(available t1) (tankEmpty t1) (ft1<=0) generateStart (fl>=0) (d=0) generateEnd (fl>=0) & (d==1000) generateEnd (fl>=0) & (d==1000) generateEnd (fl>=0) & (d==1000) generatorOverflow (fl>=c) generatorOverflow (fl>=c) generatorOverflow (fl>=c) generateStart (fl>=0) (d=0) (refuel t1) (available t1) not(available t1) (tankEmpty t1) (ft1<=0) (refuel t2) (available t2) not(available t2) (tankEmpty t2) (ft2<=0) generateStart (fl>=0) (d=0) (refuel t2) (available t2) not(available t2) generatorOverflow (fl>=c) generatorOverflow (fl>=c) generatorOverflow (fl>=c) (tankEmpty t2) (ft2<=0) Abbreviations: fl = fuelLevel,ft1 = (fuelInT ank t1), ft2 = (fuelInT ank t2), pt1 = (ptime t1),pt2 = (ptime t2),x = 0.001*(ptime t1)*(ptime t2),y = 0.001*(ptime t2)*(ptime t2),d = duration,c = capacity , Sf = (safe gen),Ut1 = (using t1 gen),Ut2 = (using t2 gen), Nut1 = not(using t1 gen),Nut2 = not(using t2 gen),St = generator started. initial fl = 940, ft1 = 40,ft2 = 40, c = 1600, (available t1),(available t2); Inv: d<=1000, ft2>=0, fl<=c, fl>=0; Inv: ft1>=0, fl<=c; Inv: ft1>=0, ft2>=0 fl<=c; Inv: d<=1000, fl>=0; Inv: ft2>=0, fl<=c; Inv: d<=1000, ft1>=0, fl<=c, fl>=0; generateFail (fl<0) generateFail (fl<0) generateFail (fl<0) generateFail (fl<0) Inv: d<=1000, ft1>=0, ft2>=0, fl<=c, fl>=0; Figure 8.1: A hybrid automaton model of the generator-events domain for the planning problem instance 1. 169 CHAPTER 8. APPENDIX Listing 11 and Listing 12 show the generated plan for the planning problem instances of Listing 9 and Listing 10 by SMTPlan+, respectively. TimeActionDuration 0 . 0 :( r e f u e l gen t1 )[ 0 . 0 ] 0 . 0 :( r e f u e l gen t2 )[ 0 . 0 ] 6 4 . 0 :( generate gen )[ 1 0 0 0 . 0 ] Listing 8.5: A plan generated by SMTPlan+ on the generator-events domain for the planning problem instance 1 in Listing 9. The plan duration is 1064.0 units. TimeActionDuration 0 . 0 :( r e f u e l gen t1 )[ 0 . 0 ] 0 . 0 :( r e f u e l gen t2 )[ 0 . 0 ] 0 . 0 :( r e f u e l gen t3 )[ 0 . 0 ] 6 4 . 0 :( generate gen )[ 1 0 0 0 . 0 ] Listing 8.6: A plan generated by SMTPlan+ on the generator-events domain for the planning problem instance 2 in Listing 10. The plan duration is 1064.0 units. 8.3 Planetary Lander Domain ( define ( domain power ) ( : requirements : typing : durative−actions : f l u e n t s : time : negative−preconditions : timed−i n i t i a l −l i t e r a l s ) ( : types equipment ) ( : predicates ( day ) (commsOpen) ( readyForObs1 ) ( readyForObs2 ) ( gotObs1 ) ( gotObs2 ) ( a v a i l a b l e ?e − equipment )) ( : functions (demand) ( supply ) ( soc ) ( charge_rate ) ( daytime ) ( heater_rate ) ( dusk ) (dawn) ( fullprepare_durtime ) ( prepareobs1_durtime ) ( prepareobs2_durtime ) ( observe1_durtime ) ( observe2_durtime ) ( obs1_rate ) ( obs2_rate ) ( A_rate ) ( B_rate ) ( C_rate ) ( D_rate ) ( safeLevel ) ( solar_const )) ( : process charging : parameters () : precondition ( and (< (demand) ( supply )) ( day )) : e f f e c t ( and ( increase ( soc ) (∗ #t (∗ (∗ (− ( supply ) (demand ) ) ( charge_rate )) (− 100 ( soc ) ) ) ) ) ) ) ( : process discharging : parameters () : precondition (> (demand) ( supply )) : e f f e c t ( decrease soc (∗ #t (− (demand) ( supply ) ) ) ) ) ( : process generating : parameters () : precondition ( day ) : e f f e c t ( and ( increase ( supply ) (∗ #t (∗ (∗ ( solar_const ) ( daytime )) (+(∗( daytime ) (−(∗ 4 ( daytime )) 90)) 450)))) ( increase ( daytime ) (∗ #t 1 ) ) ) ) ( : process night_operations : parameters () : precondition ( not ( day )) 170 8.3. PLANETARY LANDER DOMAIN : e f f e c t ( and ( increase ( daytime ) (∗ #t 1)) ( decrease ( soc ) (∗ #t ( heater_rate ) ) ) ) ) ( : event n i g h t f a l l : parameters () : precondition ( and ( day ) (>= ( daytime ) ( dusk ) ) ) : e f f e c t ( and ( assign ( daytime ) (− (dawn ) ) ) ( not ( day ) ) ) ) ( : event daybreak : parameters () : precondition ( and ( not ( day )) (>= ( daytime ) 0)) : e f f e c t ( day )) ( : durative−action fullPrepare : parameters (? e − equipment ) : duration (= ? duration ( fullprepare_durtime )) : condition ( and ( at s t a r t ( a v a i l a b l e ?e )) ( over a l l (> ( soc ) ( s a f e l e v e l ) ) ) ) : e f f e c t ( and ( at s t a r t ( not ( a v a i l a b l e ?e ) ) ) ( at s t a r t ( increase (demand) ( A_rate ) ) ) ( at end ( a v a i l a b l e ?e )) ( at end ( decrease (demand) ( A_rate ) ) ) ( at end ( readyForObs1 )) ( at end ( readyForObs2 ) ) ) ) ( : durative−action prepareObs1 : parameters (? e − equipment ) : duration (= ? duration ( prepareobs1_durtime )) : condition ( and ( at s t a r t ( a v a i l a b l e ?e )) ( over a l l (> ( soc ) ( s a f e l e v e l ) ) ) ) : e f f e c t ( and ( at s t a r t ( not ( a v a i l a b l e ?e ) ) ) ( at s t a r t ( increase (demand) ( B_rate ) ) ) ( at end ( a v a i l a b l e ?e )) ( at end ( decrease (demand) ( B_rate ) ) ) ( at end ( readyForObs1 ) ) ) ) ( : durative−action prepareObs2 : parameters (? e − equipment ) : duration (= ? duration ( prepareobs2_durtime )) : condition ( and ( at s t a r t ( a v a i l a b l e ?e )) ( over a l l (> ( soc ) ( s a f e l e v e l ) ) ) ) : e f f e c t ( and ( at s t a r t ( not ( a v a i l a b l e ?e ) ) ) ( at s t a r t ( increase (demand) ( C_rate ) ) ) ( at end ( a v a i l a b l e ?e )) ( at end ( decrease (demand) ( C_rate ) ) ) ( at end ( readyForObs2 ) ) ) ) ( : durative−action observe1 : parameters (? e − equipment ) : duration (= ? duration ( observe1_durtime )) : condition ( and ( at s t a r t ( a v a i l a b l e ?e )) ( at s t a r t ( readyForObs1 )) ( over a l l (> ( soc ) ( s a f e l e v e l ) ) ) ( over a l l ( not (commsOpen ) ) ) ) : e f f e c t ( and ( at s t a r t ( not ( a v a i l a b l e ?e ) ) ) ( at s t a r t ( increase (demand) ( obs1_rate ) ) ) ( at end ( a v a i l a b l e ?e )) ( at end ( decrease (demand) ( obs1_rate ) ) ) ( at end ( not ( readyForObs1 ) ) ) ( at end ( gotObs1 ) ) ) ) ( : durative−action observe2 : parameters (? e − equipment ) : duration (= ? duration ( observe2_durtime )) : condition ( and ( at s t a r t ( a v a i l a b l e ?e )) ( at s t a r t ( readyForObs2 )) ( over a l l (> ( soc ) ( s a f e l e v e l ) ) ) ( over a l l ( not (commsOpen ) ) ) ) : e f f e c t ( and ( at s t a r t ( not ( a v a i l a b l e ?e ) ) ) ( at s t a r t ( increase (demand) ( obs2_rate ) ) ) ( at end ( a v a i l a b l e ?e )) ( at end ( decrease (demand) ( obs2_rate ) ) ) ( at end ( not ( readyForObs2 ) ) ) ( at end ( gotObs2 ) ) ) ) ) Listing 8.7: The planetary-lander domain in PDDL+. Listing 16 shows the generated plan on the planning problem instance 1 in Listing 14 by UPMurphi. 171 CHAPTER 8. APPENDIX ( define ( problem planety_prob ) ( : domain planety ) ( : objects unit1 − equipment ) ( : i n i t ( at 0 ( not (commsOpen ) ) ) ( a v a i l a b l e unit1 ) (= A_rate 4.0) (= B_rate 2.0) (= C_rate 2.5) (= D_rate 1.5) (= obs1_rate 4) (= obs2_rate 5) (= fullprepare_durtime 3) (= observe1_durtime 7) (= observe2_durtime 8) (= prepareobs1_durtime 1) (= prepareobs2_durtime 2) (= demand 0) (= supply 0) (= soc 93) (= charge_rate 0.0095) (= daytime 4) (= heater_rate 15.1) (= dusk 9.0) (= dawn 3.0) (= safeLevel 15.0) (= solar_const 0.03) ( not ( gotObs1 )) ( not ( gotObs2 )) ( not ( readyforobs1 )) ( not ( readyforobs2 )) ( day )) ( : goal ( and ( gotObs1 ) ( gotObs2 ) ) ) ) Listing 8.8: The planning problem instance 1 in the planetary-lander domain in PDDL+. ( define ( problem planety_prob ) ( : domain planety ) ( : objects unit1 − equipment ) ( : i n i t ( at 0 ( not (commsOpen ) ) ) ( a v a i l a b l e unit1 ) (= A_rate 4.0) (= B_rate 2.0) (= C_rate 2.5) (= D_rate 1.5) (= obs1_rate 4) (= obs2_rate 5) (= fullprepare_durtime 3) (= observe1_durtime 7) (= observe2_durtime 8) (= prepareobs1_durtime 1) (= prepareobs2_durtime 2) (= demand 0) (= supply 0) (= soc 50) (= charge_rate 0.0095) (= daytime 4) (= heater_rate 15.1) (= dusk 9.0) (= dawn 3.0) (= safeLevel 25.0) (= solar_const 0.03) ( not ( gotObs1 )) ( not ( gotObs2 )) ( not ( readyforobs1 )) ( not ( readyforobs2 )) ( day )) ( : goal ( and ( gotObs1 ) ( gotObs2 ) ) ) ) Listing 8.9: The planning problem instance 2 in the planetary-lander domain in PDDL+. TimeActionDuration 0.000:( f u l l p r e p a r e 1 unit1 ) [ 3 . 0 0 0 ] 3.001:( observe1 unit1 )[ 7 . 0 0 0 ] 10.002: ( observe22 unit1 )[ 8 . 0 0 0 ] Listing 8.10: A plan generated by UPMurphi on the planetary-lander domain on planning problem in Listing 14. The plan duration is 18.002 units. 172