Paper deep dive

How Reliable are AMR Parser Outputs? Uncertainty Estimation for AMR Parsing

Anuj Kumar, Shintaro Ozaki, Yamuna Prasad, Katsuhiko Hayashi, Hidetaka Kamigaito, Taro Watanabe

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 59

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%

Last extracted: 10/12/2026, 7:04:50 AM

Summary

This paper presents the first systematic study of uncertainty estimation (UE) for Abstract Meaning Representation (AMR) parsers in a black-box setting. Using top-k beam-search candidates, the authors estimate uncertainty via pairwise candidate similarities (lexical and graph-based methods) and evaluate reliability using the Prediction Rejection Ratio (PRR). The study analyzes the correlation between parsing performance and uncertainty reliability, the impact of beam size, and the influence of evaluation metrics. Results indicate that higher-performing parsers generally yield more reliable uncertainty estimates, moderate beam sizes are optimal, and SPRING parser offers the best balance of performance and reliability.

Entities (15)

AMR · concept · 99%Prediction Rejection Ratio · metric · 98%SPRING · model · 98%AMR 3.0 · dataset · 95%DegMat · method · 95%ECC · method · 95%EigV · method · 95%HMG-AMR-BART · model · 95%HMG-AMR-T5 · model · 95%IBM-Neural · model · 95%IBM-Parser · model · 95%LexSim · method · 95%MBSE · model · 95%SemBLEU · metric · 95%Smatch · metric · 95%

Relation Signals (8)

Prediction Rejection Ratio → evaluates → Uncertainty Estimation Reliability

confidence 98% · The reliability of these uncertainty estimates is evaluated using the Prediction Rejection Ratio (PRR)

SPRING → hasbestbalance → Performance and Reliability

confidence 95% · SPRING (Bevilacqua et al., 2021) achieves the strongest overall balance, obtaining the highest mean PRR (0.265)

EigV → isbasedon → Laplacian Eigenvalues

confidence 95% · EigV estimates uncertainty from eigenvalue deviations

LexSim → isbasedon → ROUGE-L Similarity

confidence 95% · lexical uncertainty is estimated from pairwise ROUGE-L (Lin, 2004) similarity

Evaluation Metric → influences → Uncertainty Assessment

confidence 92% · Different evaluation metrics can lead to different conclusions about their reliability

Beam Size → affects → Uncertainty Reliability

confidence 90% · Moderate beam sizes yield the most reliable uncertainty estimates, while larger beams introduce diversity that weakens the relationship

HMG-AMR-T5 → exhibits → Steep Degradation

confidence 90% · HMG-AMR-T5 exhibits the steepest degradation, approaching zero or negative PRR at k = 8 and k = 10

SPRING → uses →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Recent advances have significantly improved the performance of Abstract Meaning Representation (AMR) parsers, yet determining when their outputs can be trusted remains largely unexplored. This work presents the first study of uncertainty estimation for AMR parsers in a black-box setting using top-k beam-search candidates. Uncertainty is estimated from pairwise candidate similarities and evaluated using the Prediction Rejection Ratio. This work characterises the relationship between parsing performance and uncertainty, evaluates the reliability of beam candidate distributions, analyses the effect of evaluation metrics on uncertainty assessment, and identifies parsers that provide the strongest balance between performance and reliability. Experimental results show the value of uncertainty estimation as a complementary dimension for evaluating AMR parsers.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

57,198 characters extracted from source content.

Expand or collapse full text

How Reliable are AMR Parser Outputs? Uncertainty Estimation for AMR Parsing Anuj Kumar 1,2 , Shintaro Ozaki 2 , Yamuna Prasad 1 , Katsuhiko Hayashi 2,3 , Hidetaka Kamigaito 2 , Taro Watanabe 2 1 Indian Institute of Technology Jammu, India 2 Nara Institute of Science and Technology (NAIST), Japan 3 The University of Tokyo, Japan anuj,yamuna.prasad@iitjammu.ac.in, katsuhiko-hayashi@g.ecc.u-tokyo.ac.jp ozaki.shintaro.ou6,kamigaito.h,taro.watanabe@naist.ac.jp Abstract Recent advances have significantly improved the performance of Abstract Meaning Repre- sentation (AMR) parsers, yet determining when their outputs can be trusted remains largely un- explored. This work presents the first study of uncertainty estimation for AMR parsers in a black-box setting using top-kbeam-search can- didates. Uncertainty is estimated from pairwise candidate similarities and evaluated using the Prediction Rejection Ratio. This work charac- terises the relationship between parsing perfor- mance and uncertainty, evaluates the reliability of beam candidate distributions, analyses the effect of evaluation metrics on uncertainty as- sessment, and identifies parsers that provide the strongest balance between performance and reliability. Experimental results show the value of uncertainty estimation as a complementary dimension for evaluating AMR parsers. 1 Introduction Abstract Meaning Representation (AMR) repre- sents sentence meaning as a rooted semantic graph that captures concepts and their relations beyond surface text (Banarescu et al., 2013). By mod- elling predicate–argument structure, AMR supports meaning-centric NLP applications including infor- mation extraction (Han et al., 2025), question an- swering (Rakshit and Flanigan, 2025), summarisa- tion (Qiu et al., 2024), multilingual semantic pars- ing (Vanroy and Van de Cruys, 2024), and trustwor- thy AI (Regan et al., 2024; Pust et al., 2015). However, manually constructing AMR graphs is expensive and difficult to scale, motivating the de- velopment of automatic AMR parsers that convert text into structured semantic representations. Ad- vances in pretrained language models have substan- tially improved parsers such as SPRING (Bevilac- qua et al., 2021), IBM-Parser (Fernandez As- tudillo et al., 2020), HMG-AMR (Sataer et al., 2024), MBSE (Lee et al., 2022), and IBM- Neural (Drozdov et al., 2022), making AMR pars- ing increasingly important for downstream applica- tions where graph quality directly affects reasoning and decision-making. Despite recent advances in AMR parsing, an important question remains unresolved:how confident should a parser be in its prediction? Recent studies have explored uncertainty esti- mation for large language models using prob- abilistic, diversity-based, and graph-based ap- proaches (Kuhn et al., 2023; Lin et al., 2024; Hashimoto et al., 2025; Vashurin et al., 2025), highlighting the importance of estimating predic- tion reliability beyond output generation. How- ever, Uncertainty Estimation (UE) for AMR pars- ing remains largely unexplored. Existing evalua- tion protocols rely on graph similarity metrics (Cai and Knight, 2013; Opitz et al., 2020; Anchiêta et al., 2019; Song and Gildea, 2019; Opitz, 2023; Opitz et al., 2021), which compare predicted graphs against gold annotations and measure aggregate parser performance. Nevertheless, these metrics provide limited insight into the reliability of in- dividual predictions and into whether a parser’s confidence aligns with its actual performance. To the best of our knowledge, this work presents the first systematic study of uncertainty estimation for AMR parsing in a black-box setting and addresses the following research questions: RQ1:Do performance and uncertainty correlate in AMR parsing? RQ2: How reliable are beam candidates? RQ3:Is relying on a single metric enough to judge the uncertainty of AMR parsers? RQ4:Which parser is best in terms of both perfor- mance and reliability? 1 Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing Oct 24–29, 2026 ©2026 Association for Computational Linguistics arXiv:2610.13236v1 [cs.CL] 9 Oct 2026 Our experiments 1 evaluate uncertainty estima- tion across multiple AMR parsers in a black-box setting. For each input, the parser generates a set of beam candidates, from which an uncertainty es- timator assigns a scalar uncertainty score; higher scores indicate greater uncertainty. In this work, Uncertainty Estimation (UE) refers to the process of assigning these scores, while Reliability refers to how well the scores reflect the actual quality of predictions. The reliability of these uncertainty es- timates is evaluated using the Prediction Rejection Ratio (PRR) (Malinin et al., 2017), which measures how effectively the uncertainty scores identify low- quality predictions: a reliable estimator should as- sign higher uncertainty to predictions with lower parsing quality, such that rejecting the most un- certain predictions improves the remaining aver- age quality. Results show a general tendency for higher-performing parsers to produce more reliable uncertainty estimates, although parser architecture and candidate-generation behaviour also influence reliability. Moderate beam sizes yield the most re- liable uncertainty estimates, while different evalua- tion metrics can lead to different conclusions about their reliability. These findings demonstrate that uncertainty estimation provides a complementary perspective beyond parsing accuracy for evaluating AMR parsers. 2 Problem Definition LetD = (s i ,g ∗ i ) n i=1 denote a dataset of size n, wheres i denotes an input sentence andg ∗ i its corresponding gold AMR graph. Given sentence s i , an AMR parser generates k beam candidates C i =g (1) i ,...,g (k) i , g (1) i = ˆg i ,(1) whereC i denotes the set ofkcandidate AMR graphs,g (j) i thej-th beam prediction, andˆg i the top-1 prediction. In the black-box setting, parsing is defined asP : s i 7→C i without access to model probabilities or internal states. The objective is to learn an uncertainty estimatorUthat maps beam candidates to a scalar uncertainty score, U :C i 7→ u i , u i ∈R(2) yielding one uncertainty score per input sentence and uncertainty vectoru = (u 1 ,...,u n ) ∈R n . Prediction quality is computed between the top-1 1 https://github.com/naist-nlp/UE_Method_AMR_ Parser prediction and gold graph producing quality vector q = (q 1 ,...,q n ) ∈R n . A desirable estimator satisfies u i ∝ (1− q i ) and maximises PRR. 3 Methodology This work estimates uncertainty for AMR parsing in a black-box setting, where model probabilities and internal states are unavailable. The key intu- ition is that a set of beam candidates with high semantic agreement indicates lower uncertainty, whereas greater disagreement among candidates indicates higher uncertainty. For each input sen- tence, pairwise similarities among the candidate AMR graphs are computed to construct a similar- ity matrix (§3.1), which is then converted into a scalar uncertainty score (§3.2). The quality of these uncertainty estimates is evaluated using the Predic- tion Rejection Ratio (PRR) (§3.3), which measures how effectively the uncertainty scores identify low- quality predictions and whether rejecting the most uncertain predictions improves the quality of the remaining predictions. 3.1 Pairwise Similarity Matrix For each candidate setC i , pairwise candidate agree- ment is represented by [S i ] jl = F sim (g (j) i ,g (l) i ),S i ∈R k×k ,(3) whereF sim denotes an AMR similarity function, [S i ] jl ∈ [0, 1]measures similarity between beam candidatesg (j) i andg (l) i , and[S i ] j = 1. This pro- cedure produces one similarity matrixS i per sen- tence. 3.2 Uncertainty Estimation Each uncertainty estimator maps beam candidates to an uncertainty score asU : C i 7→ S i 7→ u i , whereS i ∈R k×k is the pairwise similarity matrix, u i ∈Ris the scalar uncertainty score for sentence s i , andu = (u 1 ,...,u n ) ∈R n is the uncertainty vector. We use the following estimators: LexSim Following Kuhn et al. (2023), lexical uncertainty is estimated from pairwise ROUGE- L (Lin, 2004) similarity: u LexSim i =− 1 k(k− 1) X j̸=l [R i ] jl (4) where[R i ] jl ∈ [0, 1]denotes the ROUGE-L simi- larity between beam candidates g (j) i and g (l) i . 2 Graph-Based Methods Following Lin et al. (2024),S i is interpreted as a weighted graph. The symmetrized adjacency matrixA∈R k×k and de- gree matrix D∈R k×k are defined as [A] jl = [S i ] jl + [S i ] lj 2 ,[D] j = X l [A] jl (5) and transformed into the normalised Laplacian L = I− D −1/2 AD −1/2 ,(6) whereL∈R k×k andλ 1 ,...,λ k denote its eigen- values. The normalised Laplacian provides a spec- tral representation of candidate connectivity: the multiplicity of its zero eigenvalue equals the num- ber of connected components in the graph (von Luxburg, 2007, Proposition 4). This property mo- tivates the use of the Laplacian eigenvalues and eigenvectors in the following EigV of Equation 7 and Ecc of Equation 8. EigV estimates uncertainty from eigenvalue de- viations, where larger deviations(1− λ j )corre- spond to greater candidate disagreement: u EigV i = k X j=1 max(0, 1− λ j ).(7) Ecc estimates uncertainty from candidate spread in the Laplacian eigenspace. Letv 1 ,...,v k ∈R k denote the eigenvectors ofL. Each candidateg (j) i is embedded ase j = (v 1,j ,...,v k,j ) ⊤ ∈R k and mean-centred as ̃ e j = e j − 1 k P k l=1 e l ; uncertainty is the Frobenius norm of the centred embeddings: u Ecc i = v u u t k X j=1 ∥ ̃ e j ∥ 2 .(8) DegMat does not rely on the normalized Lapla- cian and estimates uncertainty from average candi- date agreement, where smaller P j,l [A] jl indicates higher uncertainty: u DegMat i = 1− tr(D) k 2 = 1− 1 k 2 X j,l [A] jl . (9) 3.3 Prediction Rejection Evaluation An uncertainty estimator should assign a low uncer- tainty score to reliable predictions and a high un- certainty score to unreliable predictions. PRR (Ma- linin et al., 2017) evaluates this behaviour by mea- suring quality improvement after rejecting uncer- tain outputs. Prediction quality is computed between the top-1 prediction ˆg i and the gold graph g ∗ i : q i =F qual (ˆg i ,g ∗ i )∈R(10) yielding quality vectorq = (q 1 ,...,q n ) ∈ R n . Given uncertainty vectoru ∈R n , predic- tions are sorted by increasing uncertaintyπ = argsort(u) ∈Z n . At rejection rateα ∈ [0, 1], the⌊αn⌋most uncertain predictions are removed, and the remaining mean quality is Q(α) = 1 n−⌊αn⌋ n−⌊αn⌋ X i=1 q π i ∈R(11) whereπ i is thei-th element ofπ. The area under the resulting prediction–rejection curve (AUCPR) is normalised against oracle and random rejection: PRR = AUCPR unc − AUCPR random AUCPR oracle − AUCPR random ∈R (12) wherePRR = 1denotes oracle performance, PRR = 0random rejection, andPRR < 0harm- ful uncertainty estimates. Further details on the algorithmic framework and AUCPR derivation are provided in Appendix A. 4 Experiments and Results 4.1 Experimental Setup Experiments are conducted on the AMR 3.0 test set (Knight et al., 2020), comprisingn = 1, 898 sentence–graph pairs, using six AMR parsers 2 : SPRING (Bevilacqua et al., 2021), IBM Transition Parser (Fernandez Astudillo et al., 2020), HMG- AMR-BART (Sataer et al., 2024), HMG-AMR- T5 (Sataer et al., 2024), MBSE (Lee et al., 2022), and IBM-Neural (Drozdov et al., 2022). For each parser, beam search generates candidate sets with k ∈ 3, 5, 8, 10. Fewer than 1% of generated beam candidates are invalid across parsers and beam sizes and are assigned zero similarity when computing graph-based similarity measures. AMR similarity function defined in Equation(3) is measured using Smatch (Cai and Knight, 2013), Smatch++ (Opitz, 2023), S 2 Match (Opitz et al., 2020), SEMA (Anchiêta et al., 2019), and WLK (Opitz et al., 2021), while prediction quality defined in Equation(10)is evaluated against gold 2 Parsers were selected based on reported performance and code availability. LLM baselines were excluded due to sub- stantially lower parsing performance. Additional details are provided in Appendix C.1 3 ParserSmatchSmatch++ SPRING (Bevilacqua et al., 2021)0.84490.8438 MBSE (Lee et al., 2022)0.83740.8378 HMG-AMR-BART (Sataer et al., 2024)0.83580.8380 IBM-Parser (Fernandez Astudillo et al., 2020)0.81280.8181 IBM-Neural (Drozdov et al., 2022)0.81150.8174 HMG-AMR-T5 (Sataer et al., 2024)0.76930.7797 Table 1: Top-1 AMR parsing performance measured using Smatch and Smatch++. SPRING IBM-Parser HMG-AMR-T5 HMG-AMR-BART MBSE IBM-Neural 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Mean PRR (all UE methods, all beams) 0.269 0.211 0.100 0.154 0.204 0.191 0.291 0.264 0.136 0.191 0.225 0.239 0.236 0.129 0.073 0.128 0.120 0.127 Smatch++ S²match SemBLEU Figure 1: Mean PRR for each parser and prediction quality metric, averaged across all beam sizes and un- certainty estimators. graphs using Smatch++ (Opitz, 2023), S 2 Match, and SemBLEU (Song and Gildea, 2019). Addi- tional implementation details, parser checkpoints, similarity metrics, and hardware configuration are provided in Appendix B. 4.2 Results and Analysis RQ1: Do performance and uncertainty corre- late in AMR parsing?Table 1 and Figure 1 com- pare parsing performance and PRR score across parsers. Overall, parsers with higher top-1 Smatch scores tend to achieve higher PRR, indicating a pos- itive relationship between parsing accuracy and reli- ability. However, parsers with comparable Smatch scores exhibit substantially different PRR values; for instance, MBSE and HMG-AMR-BART have mean PRRs of 0.183 and 0.154, respectively, under Smatch++, despite near-identical parsing accuracy, suggesting that candidate-generation behaviour and architectural design also independently influence reliability. Additional results are provided in Ap- pendix C.1 and Appendix C.2, where positive cor- relations are observed consistently across evalu- ation metrics. Quantitatively, the Spearman cor- relation between parsing performance and mean PRR isρ = 0.388for Smatch++,ρ = 0.317for S 2 Match, andρ = 0.360for SemBLEU, with the strongest correlation observed under Smatch++. These results further support the positive relation- 35810 Beam size k 0.05 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Mean PRR (UE methods × quality metrics) SPRING IBM-Parser HMG-AMR-T5 HMG-AMR-BART MBSE IBM-Neural Figure 2: Mean PRR averaged across uncertainty esti- mation methods and prediction quality metrics for beam sizes k ∈3, 5, 8, 10. ship between parsing performance and uncertainty- estimation reliability. RQ2:How reliable are beam candidates? Beam candidates provide meaningful uncertainty estimates, as shown in Figure 2. However, PRR declines across most parsers as beam size increases, suggesting that larger beams introduce a more di- verse set of candidates, weakening the relation- ship between uncertainty and prediction quality. The rate of degradation varies substantially: HMG- AMR-T5 falls below zero atk = 10, indicating that the uncertainty estimates are actively harmful, whereas transition-based parsers remain compara- tively stable, potentially because their architecture explores structured action sequences rather than relying solely on graph linearisation. Overall, mod- erate beam sizes provide the strongest trade-off between candidate diversity and uncertainty qual- ity. Additional analysis across beam sizes and un- certainty estimators is provided in Appendix C.3, where LEXSIM is the only estimator whose PRR consistently increases with beam size, whereas ECC exhibits the steepest degradation. These re- sults show that increasing the beam size does not affect all uncertainty estimators equally, as their ability to capture candidate agreement varies. RQ3: Is relying on a single metric enough to judge the uncertainty of AMR parsers? No, the uncertainty of AMR parsers cannot be judged by using a single evaluation metric. As illus- trated in Figure 3, PRR varies across metrics, with transition-based parsers achieving stronger PRR un- der S 2 Match than Smatch++, indicating that uncer- tainty assessment depends on the notion of graph similarity being measured. Additional analysis is provided in Appendix C.4, where graph-based 4 LexSim (Smatch++) EigVal (Smatch++) DegMat (Smatch++) Ecc (Smatch++) LexSim (S²match) EigVal (S²match) DegMat (S²match) Ecc (S²match) LexSim (SemBLEU) EigVal (SemBLEU) DegMat (SemBLEU) Ecc (SemBLEU) -0.10 0.00 0.10 0.20 0.30 Parser Reliability Profile | Beam 10 (mean PRR: LexSim, EigVal, DegMat, Ecc × Smatch++, S²match, SemBLEU) SPRING IBM-Parser HMG-AMR-T5 HMG-AMR-BART MBSE IBM-Neural Figure 3: Each value shows the mean PRR averaged across uncertainty estimation methods and prediction quality metrics. estimators show strong mutual agreement under Smatch++, while LEXSIM shows near-zero or neg- ative correlation with most graph-based methods. Under SemBLEU, estimator agreement becomes more uniform, indicating that the relationship be- tween uncertainty estimators also changes with the evaluation metric. These observations reinforce that reliability is jointly determined by the uncer- tainty estimator and the metric used to assess pre- diction quality, making multi-metric evaluation im- portant for a robust assessment. RQ4: Which parser is best in terms of both per- formance and reliability?SPRING (Bevilacqua et al., 2021) achieves the strongest overall balance, obtaining the highest mean PRR (0.265) while maintaining positive uncertainty estimates across all beam sizes and evaluation metrics as shown in Figure 1 and 2. Its seq2seq formulation with graph- isomorphic linearisation reduces reliance on heuris- tic processing, preserving semantically coherent candidate sets as beam diversity increases. Addi- tional parser reliability results across beam sizes are provided in Appendix C.5, where IBM-Parser remains the most stable across all metrics. At the same time, HMG-AMR-T5 exhibits the steepest degradation, approaching zero or negative PRR at k = 8andk = 10. Detailed estimation results atk = 10are provided in Appendix C.6, where DEGMAT-SEMA achieves the strongest PRR for SPRING and HMG variants, while LEXSIM out- performs graph-based methods for IBM-Parser and IBM-Neural. These results highlight that parser selection involves a trade-off between overall pars- ing performance and the stability of uncertainty estimation across beam sizes. While SPRING pro- vides the strongest overall balance, IBM-Parser may be preferable when stable uncertainty estima- tion across different beam sizes is prioritised. 4.3 Additional Analysis To further investigate factors affecting uncertainty estimation and reliability, the analysis examines sentence length and agreement between uncertainty estimators. Short sentences (≤ 9tokens) receive substantially higher mean uncertainty scores than medium- and long-length sentences; however, PRR indicates that uncertainty estimates are most re- liable for medium-length sentences, followed by long and short sentences. The analysis further shows strong agreement among graph-based es- timators. Additional analysis is provided in Ap- pendix C.3 and C.6 showing that parser reliability varies substantially with beam size: some parsers remain stable as the beam expands, whereas oth- ers exhibit pronounced degradation. At larger beam sizes, the most reliable UE also varies across parsers, indicating that no single estimator consis- tently provides the strongest reliability. 5 Conclusion This work presents UE for AMR parsers in a black- box setting. Across multiple AMR parsers, beam sizes, and evaluation metrics, it is observed that more accurate parsers generally yield more reliable uncertainty estimates. At the same time, parser reliability also varies with beam size and parser architecture. The results further show that the re- liability of UE depends on the evaluation metric and that moderate beam sizes tend to yield more reliable UE than larger beams. Importantly, the experiments demonstrate that parsing performance and uncertainty estimation with reliability provide complementary information for comparing AMR parsers. Parsing accuracy measures the quality of the predicted AMRs, whereas PRR evaluates how effectively the estimated uncertainty identifies potentially low-quality predictions. Since parsers with similar parsing accuracy can exhibit substan- tially different PRR, considering both performance and reliability provides a more comprehensive as- sessment of parser behaviour. Overall, the find- ings support the use of both conventional accuracy- based evaluation and uncertainty-based reliability evaluation when comparing AMR parsers. 5 Limitations This work studies uncertainty estimation for AMR parsing under a black-box setting using beam- search candidates as the sole source of uncertainty. Although beam decoding is adopted by most pub- licly available AMR parsers and therefore provides a consistent evaluation setting across models, the behaviour of uncertainty under alternative decoding strategies (e.g., sampling-based or diverse decod- ing methods) remains unexplored and may yield different candidate distributions and reliability char- acteristics. A second limitation concerns evaluation. Un- certainty quality in this study is measured using the PRR alongside existing AMR quality metrics. However, current AMR evaluation metrics capture distinct aspects of graph quality and can yield dif- ferent conclusions about uncertainty, as observed in our experiments. This suggests that uncertainty- aware evaluation for AMR remains an open prob- lem and that future work may benefit from devel- oping reliability-oriented evaluation protocols that complement conventional parsing metrics. Finally, this study focuses on post-hoc uncer- tainty estimation without modifying parser architec- tures or training objectives. Integrating uncertainty directly into AMR parser training and decoding may further improve reliability. Acknowledgment The authors would like to thank the Japan Science and Technology Agency (JST) for supporting the research activity for this project at the Nara Institute of Science and Technology (NAIST) through the LOTUS Programme (India-Japan Circulation of Talented Youths in Science Programme). Ethical Considerations AI-assisted tools were used to support parts of the research workflow. ChatGPT 3 was used for writing support and language refinement, Claude Code 4 was used to accelerate research workflows, and Grammarly 5 was used for grammar correction and proofreading. All experiments, analyses, research decisions, and final manuscript content were con- ducted and verified by the authors. 3 https://chat.openai.com/ 4 https://w.anthropic.com/claude-code 5 https://w.grammarly.com/ References Rafael Torres Anchiêta, Marco Antonio Sobrevilla Cabezudo, and Thiago Alexandre Salgueiro Pardo. 2019. Sema: an extended semantic evaluation metric for amr. ArXiv, abs/1905.12069. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguis- tic Annotation Workshop and Interoperability with Discourse, pages 178–186. Michele Bevilacqua, Rexhina Blloshmi, and Roberto Navigli. 2021. One spring to rule them both: Sym- metric amr semantic parsing and generation without a complex pipeline. Proceedings of the AAAI Confer- ence on Artificial Intelligence, 35(14):12564–12573. Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceed- ings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748–752. Andrew Drozdov, Jiawei Zhou, Radu Florian, Andrew McCallum, Tahira Naseem, Yoon Kim, and Ramón Astudillo. 2022. Inducing and using alignments for transition-based AMR parsing. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1086–1098. Ramón Fernandez Astudillo, Miguel Ballesteros, Tahira Naseem, Austin Blodgett, and Radu Florian. 2020. Transition-based parsing with stack-transformers. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 1001–1007. Peitao Han, Lis Pereira, Fei Cheng, Wan Jou She, and Eiji Aramaki. 2025. AMR-RE: Abstract Meaning Representations for retrieval-based in-context learn- ing in relation extraction. In Proceedings of the 2025 Conference of the Nations of the Americas Chap- ter of the Association for Computational Linguistics: Human Language Technologies (Volume 4: Student Research Workshop). Wataru Hashimoto, Hidetaka Kamigaito, and Taro Watanabe. 2025. Decoding uncertainty: The impact of decoding strategies for uncertainty estimation in large language models. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2025, pages 14601–14613. Kevin Knight, Bianca Badarau, Laura Baranescu, Claire Bonial, Madalina Bardocz, Kira Griffitt, Ulf Herm- jakob, Daniel Marcu, Martha Palmer, Tim O’Gorman, and Nathan Schneider. 2020.Abstract Mean- ing Representation (AMR) Annotation Release 3.0. LDC2020T02. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for un- certainty estimation in natural language generation. 6 In The Eleventh International Conference on Learn- ing Representations. Young-Suk Lee, Ramón Astudillo, Hoang Thanh Lam, Tahira Naseem, Radu Florian, and Salim Roukos. 2022. Maximum Bayes Smatch ensemble distilla- tion for AMR parsing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5379–5392. Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024. Generating with confidence: Uncertainty quantifica- tion for black-box large language models. Transac- tions on Machine Learning Research. Alexander H. Liu, Kartik Khandelwal, Sandeep Sub- ramanian, Victor Jouault, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, Alexandre Sablayrolles, Amélie Héliou, Amos You, Andy Ehrenberg, Andy Lo, Anton Eliseev, Antonia Calvi, Avinash Soori- yarachchi, Baptiste Bout, and 101 others. 2026. Ministral 3. Preprint, arXiv:2601.08584. Andrey Malinin, Anton Ragni, Kate Knill, and Mark Gales. 2017. Incorporating uncertainty into deep learning for spoken language assessment. In Pro- ceedings of the 55th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 2: Short Papers), pages 45–50. Juri Opitz. 2023. SMATCH++: Standardized and ex- tended evaluation of semantic graphs. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1595–1607. Juri Opitz, Angel Daza, and Anette Frank. 2021. Weisfeiler-leman in the bamboo: Novel AMR graph metrics and a benchmark for AMR graph similarity. Transactions of the Association for Computational Linguistics, 9:1425–1441. Juri Opitz, Letitia Parcalabescu, and Anette Frank. 2020. AMR similarity metrics from principles. Transac- tions of the Association for Computational Linguis- tics, 8:522–538. Michael Pust, Ulf Hermjakob, Kevin Knight, Daniel Marcu, and Jonathan May. 2015. Parsing English into Abstract Meaning Representation using syntax- based machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1143–1154. Haoyi Qiu, Kung-Hsiang Huang, Jingnong Qu, and Nanyun Peng. 2024. AMRFact: Enhancing summa- rization factuality evaluation with AMR-driven nega- tive samples generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Geetanjali Rakshit and Jeffrey Flanigan. 2025. ASQ: Automatically generating question-answer pairs us- ing AMRs. In Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing - Natural Language Processing in the Generative AI Era, pages 1003–1011. Michael Regan, Shira Wein, George Baker, and Emilio Monti. 2024. MASSIVE multilingual Abstract Mean- ing Representation: A dataset and baselines for hallu- cination detection. In Proceedings of the 13th Joint Conference on Lexical and Computational Semantics (*SEM 2024). Yikemaiti Sataer, Yunlong Fan, Bin Li, Miao Gao, Chuanqi Shi, and Zhiqiang Gao. 2024. Hierarchi- cal information matters! improving amr parsing with multi-granularity representation interactions. Infor- mation Processing & Management, 61(3):103698. Linfeng Song and Daniel Gildea. 2019. SemBleu: A ro- bust metric for AMR parsing evaluation. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4547–4552. Bram Vanroy and Tim Van de Cruys. 2024. Less is enough: Less-resourced multilingual AMR parsing. In Proceedings of the 20th Joint ACL - ISO Work- shop on Interoperable Semantic Annotation @ LREC- COLING 2024, pages 82–92. Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. 2025. Benchmarking uncertainty quan- tification methods for large language models with LM-polygraph. Transactions of the Association for Computational Linguistics, 13. Ulrike von Luxburg. 2007. A tutorial on spectral clus- tering. Preprint, arXiv:0711.0189. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Trans- formers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Day- iheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. 7 Algorithm 1: Black-box Uncertainty Esti- mation for AMR Parsing Input: DatasetD =(s i ,g ∗ i ) n i=1 , parser P , beam size k Output: PRR PRR∈R foreach (s i ,g ∗ i )∈D do Generate k beam candidates via beam search; C i = P(s i ) =g (1) i ,...,g (k) i Compute pairwise candidate similarity matrix; S i = F sim (C i )∈R k×k ,[S i ] j = 1, [S i ] jl ∈ [0, 1] UE methods; u LexSim i = 1− 1 k(k− 1) X j̸=l ROUGE(g (j) i ,g (l) i ), u DegMat i = 1− 2 k(k− 1) X j<l [S i ] jl , u EigV i = 1 k k X r=1 (λ r − 1) 2 , u Ecc i =   k X j=1 e j − 1 k k X l=1 e l 2 2   1/2 . Higher u i indicates greater uncertainty; Compute prediction quality against the gold graph; q i =F qual (g (1) i ,g ∗ i )∈R Construct uncertainty and quality vectors; u = (u 1 ,...,u n )∈R n , q = (q 1 ,...,q n )∈R n Sort by increasing uncertainty (π 1 = least uncertain); π = argsort(u)∈Z n At rejection rate α, remove⌊αn⌋ most uncertain predictions and compute remaining mean quality; Q(α) = 1 n−⌊αn⌋ n−⌊αn⌋ X r=1 q π r Compute area under the prediction–rejection curve; AUCPR = Z 1 0 Q(α) dα Normalise against oracle and random baselines; PRR = AUCPR unc − AUCPR random AUCPR oracle − AUCPR random return PRR A Algorithm A.1 Algorithm Algorithm 1 summarises the complete procedure. A.2 Computational Complexity The dominant cost of the proposed framework is pairwise similarity computation, which re- quiresO(k 2 )graph comparisons per sentence. Alignment-based metrics (Smatch, Smatch++, S 2 Match) scale asO(|V| 2 r)per comparison, where|V|is the number of graph nodes andr the number of alignment restarts; SEMA scales asO(|V| +|E|); and WLK scales asO(|V||E|h), where|E|is the number of edges andhthe number of Weisfeiler–Lehman iterations. The overall run- time acrossPparsers,Bbeam configurations, and Nsentences is thereforeO(P · B· N · k 2 · C sim ), whereC sim denotes the per-comparison cost of the chosen metric. Uncertainty estimation adds neg- ligible overhead: DEGMAT and ECC operate in O(Nk 2 ), EIGV inO(Nk 3 ), and PRR evaluation inO(N logN)per estimator–metric pair. To avoid redundant computation, all pairwise similarity ma- trices are cached as compressed.npzarrays after the first computation. B Experiment Setup This Appendix provides implementation details re- quired to reproduce all experiments. B.1 Dataset Experiments are conducted on the AMR 3.0 (https://catalog.ldc.upenn.edu/ LDC2020T02 ) test set containing 1,898 sentence– graph pairs. Each example consists of an input sentence and its corresponding gold AMR graph represented in PENMAN notation. All uncertainty estimation experiments are performed on parser outputs generated from this test split. B.2 AMR Parsers Six publicly available AMR parsers are evaluated in this study. All experiments use official imple- mentations and publicly released pretrained check- points without modifying model architectures or retraining. •SPRING (Bevilacqua et al., 2021) (https:// github.com/SapienzaNLP/spring) 8 •IBM-Parser (Fernandez Astudillo et al., 2020)(https://github.com/IBM/ transition-amr-parser) •MBSE (Lee et al., 2022) using checkpoint: amr3joint_ontowiki2_g2g-structured-bart- large-seed44, loaded through the IBM implementation(https://github.com/ IBM/transition-amr-parser). •IBM-Neural (Drozdov et al., 2022) using checkpoint:amr3.0-structured-bart-large- neur-al-seed42, loaded through the IBM implementation(https://github.com/ IBM/transition-amr-parser). •HMG-AMR-T5(Sataeretal.,2024) (https://github.com/YIKMAT/HMG-AMR/ tree/main/HMG-AMR(T5)) • HMG-AMR-BART (Sataer et al., 2024) (https://github.com/YIKMAT/HMG-AMR/ tree/main/HMG-AMR(Bart)) For reference, two instruction-tuned LLM base- lines (Qwen3-32B and Ministral-14B) were addi- tionally evaluated using direct AMR generation without task-specific training using the transformer library (Wolf et al., 2020). Due to substantially lower performance on AMR 3.0 as shown in Ta- ble 3 and the absence of dedicated AMR parsing capability, these models were excluded from uncer- tainty estimation experiments. B.3 Similarity Construction and Evaluation Pairwise candidate similarity matricesS i ∈R k×k are constructed using four AMR-specific similarity metrics: •Smatch(CaiandKnight,2013): https://github.com/flipz357/ amr-metric-suite • S 2 Match(Opitzetal.,2020): https://github.com/flipz357/ amr-metric-suite • SEMA(Anchiêtaetal.,2019): https://github.com/flipz357/ amr-metric-suite • WLK(Opitzetal.,2021): https://github.com/flipz357/ weisfeiler-leman-amr-metrics Prediction quality for PRR evaluation is mea- sured using three metrics: •Smatch++ (Opitz, 2023):https://github. com/flipz357/smatchpp.git •S 2 Match(Opitzetal.,2020): https://github.com/flipz357/ amr-metric-suite •SemBLEU(SongandGildea,2019): https://github.com/freesunshine0316/ sembleu B.4 Uncertainty Estimation and Evaluation Uncertainty is estimated using one lexical and three graph-based approaches: •LEXSIM (Kuhn et al., 2023): estimates uncer- tainty from mean pairwise ROUGE-L simi- larity across beam candidates; lower lexical overlap indicates greater candidate diversity and higher uncertainty. •EIGV (Lin et al., 2024): constructs a similar- ity graph over candidates and measures uncer- tainty from the eigenvalues of the normalised Laplacian; larger eigenvalue deviations from one indicate weaker graph connectivity and greater candidate disagreement. •DEGMAT (Lin et al., 2024): measures uncer- tainty as the complement of the normalised total pairwise similarity; a lower sum of edge weights in the similarity graph indicates higher uncertainty. •ECC (Lin et al., 2024): embeds each candidate in the Laplacian eigenspace and measures un- certainty as the spread of embeddings around their centroid; a larger Frobenius norm indi- cates greater candidate dispersion and higher uncertainty. Prediction quality for PRR evaluation is measuredusingSmatch++(Opitz,2023), S 2 Match (Opitz et al., 2020), and SemBLEU (Song and Gildea, 2019). Uncertainty quality is evaluated using the PRR (Malinin et al., 2017). B.5 Hardware and Runtime All uncertainty estimation and PRR calculations were performed in a CPU-only environment with- out GPU acceleration. Each job was allocated 4 CPU cores and 64 GB of RAM. 9 ConfigurationRuntime Per parser (all beams) ∼89,000–90,000 s (≈25 h) Per parser per beam∼22,000 s (≈6 h) Full experiment (6 parsers) >150 h wall-clock Table 2: Hardware allocation and observed execution time were measured from job logs. ParserSmatch SPRING (Bevilacqua et al., 2021)0.8449 MBSE (Lee et al., 2022)0.8374 HMG-AMR-BART (Sataer et al., 2024)0.8358 IBM-Parser (Fernandez Astudillo et al., 2020)0.8128 IBM-Neural (Drozdov et al., 2022)0.8115 HMG-AMR-T5 (Sataer et al., 2024)0.7693 Qwen3-32B (Yang et al., 2025)0.1214 Ministral-14B (Liu et al., 2026)0.0897 Table 3: Top-1 AMR parsing performance measured using Smatch. Table 2 reports the observed runtime measured from execution logs. Generating beam candidates and computing uncertainty metrics required approx- imately 25 hours per parser across all beam sizes (k ∈3, 5, 8, 10). Processing a single beam con- figuration required approximately 6 hours. Run- ning the complete evaluation pipeline across all six parsers took approximately 150 hours of wall-clock time. C Additional Results This section presents additional analyses support- ing the claims in the main paper, including parser- level performance comparisons, the correlation be- tween AMR parsing quality and uncertainty esti- mation, and the stability of uncertainty estimators under varying beam sizes. C.1 AMR Parsing Performance Table 3 reports parser Smatch score performance on the gold dataset reproduced using the publicly re- leased implementations and default hyperparameter settings provided in the original papers. To ensure consistent comparison across parsers, all results are evaluated before BLINKification. Under this setting, SPRING achieves the highest parsing ac- curacy (0.8449), followed by MBSE (0.8374) and HMG-AMR-BART (0.8358), indicating that recent encoder–decoder approaches remain highly com- petitive for AMR generation. IBM-Parser and IBM- Neural obtain comparable performance around 0.81 Smatch, whereas HMG-AMR-T5 shows lower ac- curacy relative to other specialised AMR parsers. For reference, two instruction-tuned LLM base- lines (Qwen3-32B and Ministral-14B) are addition- ally evaluated using direct AMR generation with- out task-specific training and exhibit substantially lower performance, highlighting the difficulty of generating valid AMR structures through general- purpose language modelling alone. Due to their poor performance on the AMR 3.0 test set, these models are excluded from subsequent analysis. C.2 Relationship Between Parsing Performance and Reliability Figure 4 examines whether AMR parsing accuracy translates into more reliable uncertainty estimates. Each point represents a parser–beam combination (6 parsers×4 beam sizes = 24 points per panel), where the x-axis shows the parser’s fixed top-1 Smatch accuracy and the y-axis shows mean PRR averaged across all 13 uncertainty estimators for that parser and beam size. Because Smatch accu- racy is fixed per parser, the four points per parser are stacked vertically, reflecting how PRR varies across beam sizes at the same accuracy level. Across all evaluation metrics, a moderate pos- itive relationship is observed (ρ = 0.388for Smatch++,ρ = 0.317for S 2 Match, andρ = 0.360for SemBLEU), indicating that parsers with stronger top-1 accuracy tend to achieve higher mean PRR across beam configurations. How- ever, the correlations confirm that accuracy alone does not fully determine the quality of uncertainty. Parsers with comparable Smatch scores often oc- cupy different vertical regions of the PRR space, suggesting that beam decoding behaviour and ar- chitectural design also independently influence re- liability, beyond accuracy. The vertical spread of points within each parser further illustrates the effect of beam size on un- certainty quality: some parsers maintain stable PRR across beams, while others experience sub- stantial degradation as beam size increases, rein- forcing the findings from RQ2. Finally, the varia- tion inρacross evaluation metrics indicates that the strength of the performance–uncertainty relation- ship depends on how prediction quality is defined, consistent with the findings reported in RQ3. Overall, parser accuracy provides only a partial explanation of uncertainty quality, and reliability emerges as a complementary evaluation dimension that is not captured by conventional AMR parsing metrics alone. 10 0.780.800.820.84 Top-1 Smatch (parser accuracy) 0.05 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Mean PRR (across UE methods) = 0.388 Smatch++ 0.780.800.820.84 Top-1 Smatch (parser accuracy) 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Mean PRR (across UE methods) = 0.317 S²match 0.780.800.820.84 Top-1 Smatch (parser accuracy) 0.0 0.1 0.2 0.3 Mean PRR (across UE methods) = 0.360 SemBLEU PRR vs Parser Accuracy (Smatch) Does better accuracy mean better UE? SPRING IBM-Parser HMG-AMR-T5 HMG-AMR-BART MBSE IBM-Neural k=3 k=5 k=8 k=10 Figure 4: Relationship between parser accuracy (Smatch) and uncertainty quality (PRR) across beam sizes and evaluation metrics. k=3k=5k=8k=10 LexSim EigV-Smatch EigV-S²match EigV-SEMA EigV-WLK DegMat-Smatch DegMat-S²match DegMat-SEMA DegMat-WLK Ecc-Smatch Ecc-S²match Ecc-SEMA Ecc-WLK 0.1490.1570.1670.171 0.2880.2360.1530.138 0.3010.2420.1570.143 0.2520.2340.1680.157 0.2750.1950.1280.113 0.2880.2360.1550.139 0.3000.2420.1590.144 0.2560.2340.1890.182 0.2770.1980.1310.116 0.3080.2080.0980.098 0.3190.2050.0970.098 0.2890.2070.1060.099 0.2740.1650.0710.072 Smatch++ k=3k=5k=8k=10 LexSim EigV-Smatch EigV-S²match EigV-SEMA EigV-WLK DegMat-Smatch DegMat-S²match DegMat-SEMA DegMat-WLK Ecc-Smatch Ecc-S²match Ecc-SEMA Ecc-WLK 0.1870.2010.2090.215 0.3180.2750.1940.178 0.3350.2850.2020.186 0.2820.2680.2060.196 0.3040.2220.1580.144 0.3190.2750.1970.179 0.3330.2850.2040.188 0.2840.2660.2260.220 0.3060.2240.1610.147 0.3340.2470.1410.140 0.3450.2460.1420.141 0.3130.2440.1510.143 0.2990.1920.1050.104 S²match k=3k=5k=8k=10 LexSim EigV-Smatch EigV-S²match EigV-SEMA EigV-WLK DegMat-Smatch DegMat-S²match DegMat-SEMA DegMat-WLK Ecc-Smatch Ecc-S²match Ecc-SEMA Ecc-WLK 0.0840.0900.0960.097 0.2310.1750.0850.068 0.2400.1750.0830.066 0.2040.1920.1230.108 0.2410.1700.1000.080 0.2320.1740.0880.069 0.2390.1750.0850.067 0.2030.1840.1420.132 0.2430.1740.1030.083 0.2580.1580.0330.031 0.2670.1540.0310.027 0.2470.1730.0640.056 0.2330.1410.0360.034 SemBLEU 0.3 0.2 0.1 0.0 0.1 0.2 0.3 Mean PRR 0.3 0.2 0.1 0.0 0.1 0.2 0.3 Mean PRR 0.2 0.1 0.0 0.1 0.2 Mean PRR UE Method Stability Across Beam Sizes (mean PRR over 6 parsers) Figure 5: Mean PRR of uncertainty estimators across beam sizes aggregated over all parsers. Stable estimators exhibit less variation in performance across beam sizes. C.3 Uncertainty Estimator Stability Across Beam Sizes Figure 5 reports the mean PRR aggregated across all six parsers for each uncertainty estimator and beam size, directly supporting RQ2. The over- all pattern is consistent with the main results: uncertainty-estimation reliability is highest atk = 3and generally declines as the beam size increases, with the sharpest drops occurring betweenk = 5 and k = 8. Degradation rates differ markedly across estima- tors. ECC-based methods start among the strongest atk = 3, with ECC-S 2 MATCH reaching 0.345 un- der S 2 Match, yet show the steepest decline, falling to 0.141 atk = 10. EIGV variants follow a similar trajectory, dropping from approximately0.30–0.34 atk = 3to0.14-0.20atk = 10, depending on the metric. Thus, both estimator families show sub- stantial degradation in reliability as the beam size increases. In contrast, LEXSIM is the only estimator whose mean PRR increases with beam size, rising from 0.149 to 0.171 under Smatch++ and from 0.187 to 0.215 under S 2 Match, suggesting that its un- certainty estimates are less sensitive to increasing candidate diversity. Among graph-based meth- ods, SEMA-based variants show greater robustness: DEGMAT-SEMA retains PRR values of 0.182 and 0.220 atk = 10under Smatch++ and S 2 Match, re- spectively, substantially outperforming other graph- 11 LexSim EigV-Smatch EigV-S²match EigV-SEMA EigV-WLK DegMat-Smatch DegMat-S²match DegMat-SEMA DegMat-WLK Ecc-Smatch Ecc-S²match Ecc-SEMA Ecc-WLK LexSim EigV-Smatch EigV-S²match EigV-SEMA EigV-WLK DegMat-Smatch DegMat-S²match DegMat-SEMA DegMat-WLK Ecc-Smatch Ecc-S²match Ecc-SEMA Ecc-WLK 1.000.260.25-0.01-0.160.250.280.31-0.140.260.260.120.01 0.261.000.980.770.601.000.980.680.600.870.890.850.71 0.250.981.000.730.530.991.000.610.520.890.910.840.65 -0.010.770.731.000.780.780.730.870.790.750.740.860.78 -0.160.600.530.781.000.580.530.661.000.650.610.750.92 0.251.000.990.780.581.000.990.670.590.870.890.840.69 0.280.981.000.730.530.991.000.630.530.890.910.840.65 0.310.680.610.870.660.670.631.000.680.710.690.770.72 -0.140.600.520.791.000.590.530.681.000.640.600.750.92 0.260.870.890.750.650.870.890.710.641.000.990.950.81 0.260.890.910.740.610.890.910.690.600.991.000.930.78 0.120.850.840.860.750.840.840.770.750.950.931.000.88 0.010.710.650.780.920.690.650.720.920.810.780.881.00 UE Method Pairwise Correlation (Spearman ) Smatch++ across 6 parsers × 4 beams 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 Spearman Figure 6: Pairwise Spearman correlation between uncertainty estimators evaluated using Smatch++. Graph-based estimators exhibit strong agreement, while LEXSIM captures a distinct uncertainty signal. based methods at larger beam sizes. Notably, S 2 Match consistently yields higher mean PRR than Smatch++ across all estimators and beam sizes. For example, EIGV-S 2 MATCH obtains 0.335 compared with 0.301 atk = 3. This indi- cates that the observed effect of increasing beam size also depends on how prediction quality is mea- sured, with S 2 Match generally producing stronger uncertainty-reliability scores in this analysis. Over- all, moderate beam sizes (k = 3tok = 5) provide the strongest trade-off between candidate diversity and uncertainty-estimation reliability. C.4 Agreement Between UE Figures 6, 8, and 11 report pairwise Spearman cor- relations between uncertainty estimators across all parser and beam configurations. These results pro- vide additional evidence for RQ3 by examining whether uncertainty estimators produce consistent rankings under different evaluation metrics. Across all metrics, EIGV, DEGMAT, and ECC exhibit consistently strong mutual agreement. Un- der Smatch++, most pairwise correlations exceed ρ = 0.87, with estimators using the same sim- ilarity function approaching near-perfect agree- ment (e.g. EIGV-SMATCH and DEGMAT-SMATCH: ρ = 1.00). Agreement is similarly high under S 2 Match, where several pairs reachρ = 1.00, and remains strong under SemBLEU (ρ ≥ 0.85for most graph-based pairs). This indicates that graph- based estimators generally preserve similar uncer- tainty rankings, although their agreement depends on the similarity function used. In contrast, LEXSIM shows substantially weaker correlation with graph-based estimators across met- rics. Under Smatch++, correlations range from 12 SPRINGIBM -ParserHMG-T5HMG-BARTMBSEIBM-Neural UE MethodS++SBS++SBS++SBS++SBS++SBS++SB LexSim0.22090.16950.26600.1615-0.0395-0.09040.08580.03950.22940.13320.26590.1685 EigV-Smatch0.19560.14210.22180.1251-0.0838-0.11990.07560.03590.18710.08020.23250.1431 EigV-S2match0.19120.13220.23190.1232-0.0794-0.11760.07510.03300.19310.07520.24480.1473 EigV-SEMA0.25040.22540.19190.13020.0334-0.00600.10800.08110.16660.08590.19450.1328 EigV-WLK0.24690.22310.18070.12430.0348-0.00020.06050.05120.15550.08110.00000.0000 DegMat-Smatch0.19710.14380.22200.1250-0.0826-0.11830.07760.03800.18780.08070.23280.1434 DegMat-S2match0.19250.13360.23210.1234-0.0755-0.11410.07720.03510.19420.07630.24520.1477 DegMat-SEMA0.27090.24650.19840.13530.06850.02320.13690.10910.20610.13340.21040.1472 DegMat-WLK0.25100.22750.18140.12480.04190.00740.06430.05500.15800.08320.00000.0000 Ecc-Smatch0.18530.13110.20510.1023-0.1136-0.14520.0268-0.01760.11010.00730.17680.1075 Ecc-S2match0.18170.12220.20340.1011-0.1117-0.14770.0231-0.02510.10850.00350.18500.1102 Ecc-SEMA0.18000.15060.14710.0911-0.0207-0.05470.0271-0.00150.10410.03850.15420.1117 Ecc-WLK0.18210.15130.16010.0981-0.0089-0.04490.0168-0.01040.08240.01040.00000.0000 Table 4: PRR at beam sizek = 10for all uncertainty estimation methods across parsers and quality metrics (Smatch++ (S++) and SemBLEU (SB)). Higher PRR indicates better alignment between uncertainty estimates and prediction quality. Green cells denote the best-performing uncertainty estimator within each parser–metric pair. Consistent with the main findings, SPRING achieves the strongest overall reliability, while DEGMAT-SEMA frequently emerges as the most effective graph-based uncertainty estimator at larger beam sizes. 0.1 0.0 0.1 0.2 0.3 PRR (SemBLEU) SPRINGHMG-AMR-T5HMG-AMR-BART 35810 Beam size k 0.1 0.0 0.1 0.2 0.3 PRR (SemBLEU) IBM-Parser 35810 Beam size k MBSE 35810 Beam size k IBM-Neural PRR vs Beam Size SemBLEU LexSimEigValDegMatEccRandom Figure 7: Parser-level PRR as a function of beam size evaluated using SemBLEU. Each subplot corresponds to one parser and compares uncertainty estimators across beam sizes. ρ =−0.16(EIGV-WLK) toρ = 0.31(DEGMAT- SEMA), while under SemBLEU they decrease to as low asρ = −0.14(DEGMAT-WLK). This indicates that lexical overlap captures an uncer- tainty signal that differs substantially from struc- tural graph agreement, and that the degree of this difference varies with the evaluation metric. Metric-dependent differences are also observed in the PRR results. Figure 5 shows consistently higher mean PRR under S 2 Match than Smatch++ across estimators and beam sizes; for example, EIGV-S 2 MATCH obtains 0.335 compared with 0.301 atk = 3. Under SemBLEU, WLK-based estimators show stronger agreement with other graph-based methods (ρ ≥ 0.85), reducing the separation between estimator families observed un- der Smatch++ and S 2 Match. Overall, these results show that changing the evaluation metric can alter both uncertainty reliability and agreement between estimators, indicating that no single metric provides a complete view of uncertainty reliability. 13 LexSim EigV-Smatch EigV-S²match EigV-SEMA EigV-WLK DegMat-Smatch DegMat-S²match DegMat-SEMA DegMat-WLK Ecc-Smatch Ecc-S²match Ecc-SEMA Ecc-WLK LexSim EigV-Smatch EigV-S²match EigV-SEMA EigV-WLK DegMat-Smatch DegMat-S²match DegMat-SEMA DegMat-WLK Ecc-Smatch Ecc-S²match Ecc-SEMA Ecc-WLK 1.000.310.300.14-0.110.310.340.37-0.130.310.310.23-0.01 0.311.001.000.750.571.001.000.690.560.920.920.930.62 0.301.001.000.740.561.000.990.670.550.920.930.930.61 0.140.750.741.000.680.750.750.860.690.750.750.790.59 -0.110.570.560.681.000.570.560.651.000.660.640.670.94 0.311.001.000.750.571.001.000.690.560.920.920.930.62 0.341.000.990.750.561.001.000.700.550.930.930.930.62 0.370.690.670.860.650.690.701.000.640.760.770.760.61 -0.130.560.550.691.000.560.550.641.000.650.630.660.93 0.310.920.920.750.660.920.930.760.651.000.990.960.75 0.310.920.930.750.640.920.930.770.630.991.000.960.73 0.230.930.930.790.670.930.930.760.660.960.961.000.75 -0.010.620.610.590.940.620.620.610.930.750.730.751.00 UE Method Pairwise Correlation (Spearman ) S²match across 6 parsers × 4 beams 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 Spearman Figure 8: Pairwise Spearman correlation between uncertainty estimators evaluated using S 2 Match. C.5 Parser Reliability Across Beam Sizes Figures 7, 9, and 10 visualise parser-level PRR across beam sizes, with each curve reporting the mean PRR for a given uncertainty estimator. SPRING maintains the strongest and most con- sistent reliability profile across all metrics. Under Smatch++, PRR remains above0.18atk = 10for all estimators except ECC, which converges with others at larger beams. Under S 2 Match and Sem- BLEU, SPRING similarly retains positive PRR across all estimators and beam sizes, confirming its robustness to candidate expansion. IBM-Parser is the most stable parser across beam sizes. Under all three metrics, PRR remains nearly flat fromk = 3tok = 10, with LEXSIM consistently outperforming graph-based methods and remaining the strongest estimator throughout. IBM-Neural shows a similar stability pattern, with LEXSIM again achieving the highest PRR at larger beams. HMG-AMR-T5 exhibits the steepest degrada- tion across all metrics. Under Smatch++, graph- based estimators drop below zero byk = 8, and LEXSIM also approaches zero byk = 10. Under SemBLEU, all estimators reach negative PRR at k = 8andk = 10, indicating that beam candidates become actively misleading. HMG-AMR-BART shows similar sensitivity, particularly under Sem- BLEU, where ECC drops below zero at k = 10. MBSE achieves a competitive PRR atk = 3 under all metrics. However, it degrades noticeably at larger beams, particularly under S 2 Match where ECC falls to≈ 0.12atk = 10, substantially below LEXSIM and DEGMAT. Overall, these results rein- force RQ4: the most reliable parser is the one that maintains reliable uncertainty estimates as candi- date diversity increases not the one with the highest parsing accuracy. 14 0.0 0.1 0.2 0.3 0.4 PRR (Smatch++) SPRINGHMG-AMR-T5HMG-AMR-BART 35810 Beam size k 0.0 0.1 0.2 0.3 0.4 PRR (Smatch++) IBM-Parser 35810 Beam size k MBSE 35810 Beam size k IBM-Neural PRR vs Beam Size Quality: Smatch++ LexSimEigVDegMatEccRandom Figure 9: Parser-level PRR as a function of beam size evaluated using Smatch++. Each subplot corresponds to one parser and compares uncertainty estimators across beam sizes. 0.0 0.1 0.2 0.3 0.4 PRR (S²match) SPRINGHMG-AMR-T5HMG-AMR-BART 35810 Beam size k 0.0 0.1 0.2 0.3 0.4 PRR (S²match) IBM-Parser 35810 Beam size k MBSE 35810 Beam size k IBM-Neural PRR vs Beam Size S²match LexSimEigValDegMatEccRandom Figure 10: Parser-level PRR as a function of beam size evaluated using S 2 Match. Each subplot corresponds to one parser and compares uncertainty estimators across beam sizes. C.6 Detailed Uncertainty Estimation Results Table 4 reports complete PRR results atk = 10 across all parsers, uncertainty estimators, and evalu- ation metrics. These results provide a more detailed view of RQ4 by examining whether the reliability of a parser is consistent across different uncertainty estimators at a large beam size. DEGMAT-SEMA achieves the strongest PRR for SPRING under both Smatch++ (0.271) and SemBLEU (0.247), and 15 LexSim EigV-Smatch EigV-S²match EigV-SEMA EigV-WLK DegMat-Smatch DegMat-S²match DegMat-SEMA DegMat-WLK Ecc-Smatch Ecc-S²match Ecc-SEMA Ecc-WLK LexSim EigV-Smatch EigV-S²match EigV-SEMA EigV-WLK DegMat-Smatch DegMat-S²match DegMat-SEMA DegMat-WLK Ecc-Smatch Ecc-S²match Ecc-SEMA Ecc-WLK 1.000.050.060.01-0.110.040.070.31-0.140.090.100.05-0.03 0.051.000.990.920.761.000.990.770.730.890.880.910.85 0.060.991.000.900.710.991.000.750.680.900.900.910.83 0.010.920.901.000.800.930.910.900.770.860.860.900.85 -0.110.760.710.801.000.780.710.710.990.740.710.810.93 0.041.000.990.930.781.000.990.790.750.880.880.910.86 0.070.991.000.910.710.991.000.760.680.900.890.910.83 0.310.770.750.900.710.790.761.000.670.780.780.800.75 -0.140.730.680.770.990.750.680.671.000.710.690.790.91 0.090.890.900.860.740.880.900.780.711.001.000.980.89 0.100.880.900.860.710.880.890.780.691.001.000.980.88 0.050.910.910.900.810.910.910.800.790.980.981.000.94 -0.030.850.830.850.930.860.830.750.910.890.880.941.00 UE Method Pairwise Correlation (Spearman ) SemBLEU across 6 parsers × 4 beams 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 Spearman Figure 11: Pairwise Spearman correlation between uncertainty estimators evaluated using SemBLEU. also leads for HMG-AMR-BART (0.137, 0.109) and HMG-AMR-T5 (0.069, 0.023). In contrast, LEXSIM achieves the strongest PRR for IBM- Parser (0.266, 0.162), IBM-Neural (0.266, 0.169), and MBSE (0.229, 0.133). Thus, the estimator providing the most reliable uncertainty signal at k = 10varies across parsers, indicating that parser comparison based on reliability is also influenced by the interaction between the parser and the un- certainty estimator. ECC-based methods perform weakest overall at k = 10, with several parsers recording negative PRR under SemBLEU, such as HMG-AMR-BART (−0.025) and HMG-AMR-T5 (−0.148). This sug- gests that eigenspace-based candidate spread be- comes less informative as beam size increases for these parsers. Overall, the detailed results show that although SPRING provides the strongest over- all reliability, its advantage does not imply that a single uncertainty estimator is optimal across all parsers. 16

Instant research discovery