Paper deep dive
Impact of enriched meaning representations for language generation in dialogue tasks: A comprehensive exploration of the relevance of tasks, corpora and metrics
Alain Våzquez, Maria Inés Torres
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 4/1/2026, 1:35:34 AM
Summary
This paper presents a comprehensive analysis of using enriched Meaning Representations (MRs) with task demonstrators to enhance Natural Language Generation (NLG) in dialogue systems. By fine-tuning GPT-2 models on four diverse datasets (E2E, ViGGO, MultiWOZ, and EMPATHIC), the authors demonstrate that enriched inputs improve generation quality, particularly in complex tasks and small datasets. The study also evaluates the effectiveness of five metrics (BLEU, BLEURT, LaBSE, Slot Accuracy, and Dialogue Act Accuracy), finding that semantic metrics and those trained on human ratings are superior to lexical metrics for assessing generation quality.
Entities (10)
Relation Signals (3)
Enriched Meaning Representations â enhances â NLG
confidence 95% · Our key insight is that the proposed enriched inputs are effective for complex tasks
GPT-2 â evaluatedon â E2E
confidence 90% · We compare datasets spanning single-domain (E2E)... for this comparative study, we selected GPT-2
BLEURT â evaluates â NLG
confidence 90% · The analysis involves five metrics... that focus on different linguistic aspects
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Conversational systems should generate diverse language forms to interact fluently and accurately with users. In this context, Natural Language Generation (NLG) engines convert Meaning Representations (MRs) into sentences, directly influencing user perception. These MRs usually encode the communicative function (e.g., inform, request, confirm) via DAs and enumerate the semantic content with slot-value pairs. In this work, our objective is to analyse whether providing a task demonstrator to the generator enhances the generations of a fine-tuned model. This demonstrator is an MR-sentence pair extracted from the original dataset that enriches the input at training and inference time. The analysis involves five metrics that focus on different linguistic aspects, and four datasets that differ in multiple features, such as domain, size, lexicon, MR variability, and acquisition process. To the best of our knowledge, this is the first study on dialogue NLG implementing a comparative analysis of the impact of MRs on generation quality across domains, corpus characteristics, and the metrics used to evaluate these generations. Our key insight is that the proposed enriched inputs are effective for complex tasks and small datasets with high variability in MRs and sentences. They are also beneficial in zero-shot settings for any domain. Moreover, the analysis of the metrics shows that semantic metrics capture generation quality more accurately than lexical metrics. In addition, among these semantic metrics, those trained with human ratings can detect omissions and other subtle semantic issues that embedding-based metrics often miss. Finally, the evolution of the metric scores and the excellent results for Slot Accuracy and Dialogue Act Accuracy demonstrate that the generative models present fast adaptability to different tasks and robustness at semantic and communicative intention levels.
Tags
Links
- Source: https://arxiv.org/abs/2603.29518v1
- Canonical: https://arxiv.org/abs/2603.29518v1
Trouble viewing inline? Open PDF directly â
Full Text
113,318 characters extracted from source content.
Expand or collapse full text
IMPACT OF ENRICHED MEANING REPRESENTATIONS FOR LANGUAGE GENERATION IN DIALOGUE TASKS: A COMPREHENSIVE EXPLORATION OF THE RELEVANCE OF TASKS, CORPORA AND METRICS. A PREPRINT Alain VĂĄzquez Speech Interactive Group University of the Basque Country (UPV/EHU) Sarriena s/n, 48940 Leioa, Spain alain.vazquez@ehu.eus Maria InĂ©s Torres Speech Interactive Group University of the Basque Country (UPV/EHU) Sarriena s/n, 48940 Leioa, Spain manes.torres@ehu.eus April 1, 2026 ABSTRACT Conversational systems should generate diverse language forms to interact fluently and accurately with a variety of users. In this context, Natural Language Generation (NLG) engines convert Meaning Representations (MRs) into sentences, directly influencing user perception. These MRs usually encode the communicative function (e.g., inform, request, confirm) via DAs and enumerate the semantic content with slotâvalue pairs. In this work, our objective is to analyse whether providing a task demonstrator to the generator enhances the generations of a fine-tuned model. This demonstrator is an MRâsentence pair extracted from the original dataset that enriches the input at training and inference time. The analysis involves five metrics (BLEU, BLEURT, LaBSE, Slot Accuracy and Dialogue Act Accuracy) that focus on different linguistic aspects, and four datasets (E2E, ViGGO, MultiWOZ, and EMPATHIC) that differ in multiple features, such as domain, size, lexicon, MR variability, and acquisition process. To the best of our knowledge, this is the first study on dialogue NLG, implementing a comparative analysis of the impact of MRs on the generation quality across domains, corpora characteristics and the metrics used to evaluate these generations. Our key insight is that the proposed enriched inputs are effective for complex tasks and small datasets with high variability in MRs and sentences. They are also beneficial in zero-shot settings for any domain. Moreover, the analysis of the metrics shows that the semantic metrics capture the generation quality more accurately than lexical metrics. In addition, among these semantic metrics, those trained with human ratings can detect omissions and other subtle semantic issues that embedding-based metrics often miss. Finally, the evolution of the metric scores and the excellent results for Slot Accuracy and Dialogue Act Accuracy demonstrate that the generative models present fast adaptability to different tasks and robustness at semantic and communicative intention levels. Keywords Natural Language Generation·Dialogue Systems·Dialogue Acts·Prompt-based Learning·Evaluation of NLG 1 Introduction The impressive development of Transformers, along with the rapid increase in computing capabilities and the accessi- bility of vast amounts of data, has revolutionised natural language processing (NLP) in recent years. Large language models (LLMs) have excelled in tasks such as transcription, translation or summarisation, among others. However, LLMs present well-known challenges, including the opacity of training data and the lack of clarity in their behaviour. In conversational systems, a major concern is their tendency to hallucinate, generate misinformation and lack reasoning. arXiv:2603.29518v1 [cs.CL] 31 Mar 2026 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT Furthermore, while LLMs could recognise lexical and structural patterns, they struggle with semantic inferences and extralinguistic reasoning essential for real-world knowledge tasks [Lappin, 2024]. Conversational systems should be able to generate diverse language forms to interact fluently and accurately with a variety of users. In this context, natural language generation (NLG) plays a crucial role in dialogue, directly influencing user perception. In open-domain dialogues, the NLG must be able to express opinions and emotions, convey facts, provide information, ask contextual questions (who, what, where, when, why), confirm user inputs or give instructions. Conversely, closed-domain dialogues must align with the specific task, objectives, and conversational context. In both cases, generated sentences should be semantically controlled, accurate and free of hallucinations. Although end-to-end architectures have advanced in the design of dialogue systems [Ji et al., 2024, LĂłpez Zorrilla and Torres, 2022], they still have difficulty achieving dialogue coherence and interpretability. In this context, the concepts of dialogue acts (DA), which represent a sentenceâs intentionality or purpose, and meaning representation (MR), which adds specific attributes and entities in the shape of slot-value pairs, remain essential for managing conversational flow. Specifically, NLG generates MR-based sentences, i.e., it takes the meaning representation as input and produces sentences as output. When combined with LLMs, both closed- and open-domain dialogue systems [Wang et al., 2024] benefit from improved adaptability to complex tasks [Galland et al., 2024], as well as enhanced coherence and response accuracy [Griol and Callejas, 2024, Vazquez et al., 2024, Hu et al., 2022]. In this work, we aim to explore different ways of representing meaning for sentence generation. Our objective is to analyse whether providing the generator model with additional information, in the form of a task demonstrator, enhances generation quality. This task demonstrator is a sample from the original dataset consisting of an MRâsentence pair and enriches the input both at training and inference time. Specifically, we investigate whether these one-shot representations improve generation compared to inputs containing only the MR. The comparison requires evaluation metrics. In NLG, classic metrics, some originally from translation or language model evaluation, do not always capture relevant differences in generated sentences. Nevertheless, each metric assesses a different generation aspect, making their comparison essential for an in-depth study like ours. In this work, we use reference-based metrics such as BLEU [Papineni et al., 2002], BLEURT [Sellam et al., 2020], and LABSE [Feng et al., 2022], as well as reference-free metrics like Slot Accuracy [Li et al., 2020], complemented by the more recent Dialogue Act Accuracy (DAC) [Ramirez et al., 2023]. Our goal is to analyse each metricâs ability to identify differences in sentences generated from the proposed MR formats. Four corpora have been selected for this study: E2E [Novikova et al., 2017], VIGGO [Juraska et al., 2019], MULTIWOZ [Budzianowski et al., 2018], and EMPATHIC [VĂĄzquez et al., 2023]. All four are dialogue-oriented NLG corpora in which each sample consists of a DA-based MR and its corresponding example sentence. These datasets differ significantly in domain, size, lexicon, input variability, and acquisition process, among other features. We will analyse the suitability of each MR and assess the discriminative power of each metric based on the characteristics of each corpus. In summary, this work aims to study the impact of meaning representations on language generation in dialogue tasks. To this end, we thoroughly analyse the relevance of the tasks, the characteristics of the corpora, and the aspects each metric evaluates in the experimental results. For this comparative study, we selected GPT-2 [Radford et al., 2019], which is computationally more accessible and manageable than recent state-of-the-art LLMs such as GPT-4 [Achiam et al., 2023], Gemini [Team et al., 2023], and LLaMA 3 [Dubey et al., 2024]. In addition, recent work has shown that enriched inputs with task demonstrators can substantially enhance the generation capabilities of medium-sized models [Liu et al., 2023, VĂĄzquez et al., 2023]. The main contributions can be outlined as follows: âąAnalysis of the impact of prompt forms on the quality of generated sentences, as well as their zero-shot capabilities. âą Comparative evaluation of each metricâs ability to assess different quality aspects of generated sentences. âą Assessment of the suitability of classifying generated sentences into dialogue acts as an evaluation metric for generation quality. âą Analysis of the influence of corpus design and characteristics, such as the number of references and attributes, on the quality of sentence generation. âąCross-analysis of the distribution of the resulting quality scores in terms of representations, corpora and metrics. âą Human qualitative analysis of the generated sentences. 2 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT To the best of our knowledge, this is the first in-depth study on NLG in the context of dialogue, implementing a comparative analysis of the impact of meaning representations on the generation quality. This thorough analysis involves multiple domains, tasks, and corpora characteristics, as well as the metrics used to evaluate these generations. Overall, our experiments yield several practical insights. At the representation level, our proposed inputs are particularly helpful in complex domains and small datasets with high variability in terms of MRs, as well as the lexicon and semantics of their sentences. They also show zero-shot capabilities across domains. In terms of metrics, we observe that semantic metrics capture the quality of the generations more accurately than lexical ones based on n-gram overlap. In addition, when comparing semantic metrics, those trained with human ratings can detect omissions and other subtle semantic issues that embedding-based metrics often miss. Finally, the evolution of metric scores during fine-tuning, together with strong results on Slot Accuracy and DAC, indicates that the models adapt quickly to the task and are robust at the semantic and communicative-intent levels. The remainder of the paper is structured as follows. Section 2 explores the related work. Section 3 introduces the corpora used in our experiments and highlights their key differences. Section 4 explains the explored meaning representations, including the novel enriched representations and the methodology for selecting task demonstrators. Section 5 defines the evaluation metrics. Section 6 details the experimental results and discusses their key findings. Section 7 summarises the work with the main conclusions. 2 Related work 2.1 Natural Language Generation for dialogue tasks Task-oriented dialogue systems commonly adopt modular pipeline architectures due to the higher understandability and controllability of the process than the end-to-end systems [Algherairy and Ahmed, 2024]. Among these modules, NLG is crucial, transforming structured inputsâMRsâinto coherent and fluent natural language outputs. These MRs typically consist of a DA indicating the communicative intent accompanied by slot-value pairs that encode the semantic content to be expressed. Advanced NLG engines that leverage diverse DA-based MRs contribute to generating more dynamic and engaging dialogues, making NLG an interesting area of research [VĂĄzquez and Torres, 2025, Vazquez et al., 2024, Ramirez et al., 2023, Reed et al., 2022]. This field experienced a revolution driven by Transformer-based architectures and LLMs. The Transformers [Vaswani, 2017] significantly improved the handling of linguistic long-range dependencies, i.e., relationships between tokens or discourse elements that are far apart in the input sequence, and allowed for efficient parallelisation. These architectures laid the foundation for the first LLMs, such as GPT-2 [Radford et al., 2019], T5 [Raffel et al., 2020], and BERT [Devlin et al., 2019], which have proven highly effective for NLG tasks. Models like DialoGPT [Zhang et al., 2020] and SC-GPT [Peng et al., 2020] demonstrate the adaptability of GPT-2 for dialogue tasks. On the one hand, DialoGPT, a tunable response generation model trained with conversations, can be integrated into open-domain or domain-specific systems with small-scale fine-tuning. On the other hand, SC-GPT is pre-trained on a large annotated NLG corpus, acquiring the capability to controllably transform MRs into natural language outputs. Finally, this model with a small fine-tuning has also shown capabilities to generate adequate responses for spoken dialogue systems in noisy environments [Mousavi et al., 2024]. The new era of LLMsâGPT-4 [Achiam et al., 2023], Gemini [Team et al., 2023], and LLaMA 3 [Dubey et al., 2024]âmuch larger than their precursors, have further advanced the state of the art by showcasing capabilities to perform various NLG tasks with minimal or no fine-tuning. ChatGPT 1 represents this evolution, demonstrating the ability to handle multi-turn dialogues with coherent, fluent and logical responses. Additionally, Ramirez et al. [2023] have shown that these models can produce semantically almost perfect generations without fine-tuning through prompt learning techniques that enrich the input with demonstrators of the task to perform. Similarly, Vazquez et al. [2024] have built a high-quality synthetic dialogue dataset across multiple domains that presents fewer hallucinations and omissions than corpora written by humans. However, the computational requirements of these new LLMs pose significant challenges, such as hardware requirements, memory constraints or real-time performance. Consequently, some authors still prefer using smaller models, such as GPT-2 [LĂłpez Zorrilla et al., 2023]. Liu et al. [2023], VĂĄzquez et al. [2023] have demonstrated that enriched inputs with task demonstrators enhance the generation capabilities of these models. Inspired by them, we perform a detailed study of the impact of enriched MRs in the performance of GPT-2 models, with the innovation of fine-tuning with these representationsâa relatively underexplored approach. 1 https://chatgpt.com/ 3 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT 2.2 Meaning representations for NLG NLG inputs are wide and varied, as are their tasks. In the bibliography, we find works about description generation of the information given in a table, summarisation of an article from the news or sentence generation of the meaning encapsulated in a graph [Gehrmann et al., 2021, Parikh et al., 2020, Moryossef et al., 2019], among others. However, DA-based MRs remain the predominant input type used in most dialogue tasks [Wu et al., 2023, Wang et al., 2022, Du et al., 2020]. These representations became more important with the rise of neural models and Transformer architectures, which resulted in an increment of the number of annotated datasets [Rastogi et al., 2020, Budzianowski et al., 2018, Novikova et al., 2017]. For instance, the well-known E2E dataset, annotated in terms of DA, was created as part of the E2E NLG challenge [Novikova et al., 2017]. This challenge demonstrated the capability of diverse models and architectures to generate fluent and varied outputs using DA-based MRs. Peng et al. [2020] also showed that models capable of transforming these inputs to sentences in one domain can adapt to new domains with a small fine-tuning. Furthermore, these MRs have been employed with relevant contributions in works that employed methodologies like continual learning [Mi et al., 2020] or reinforcement learning [Wang et al., 2021]. Finally, MRs are a practical way to encapsulate information for the NLG module in dialogue system architectures [VĂĄzquez, 2026, VĂĄzquez et al., 2023, Wang et al., 2022]. Efforts to improve traditional DA-based MRs have led to significant advancements. For instance, Du et al. [2020] enriched inputs with descriptions about the nature of DAs and slots and VĂĄzquez et al. [2023] transformed DAs into their descriptive forms. These two strategies enhanced semantic accuracy, robustness, and diversity. In a different approach, Abstract Meaning Representations (AMRs) emerged as a novel way to capture the meaning of a sentence by abstracting its syntax, focusing on relationships between concepts and their roles [Hryhoryeva, 2025, Yang et al., 2024, Banarescu et al., 2013]. They have also produced inspiring advances in tasks like machine comprehension, machine translation, and summarisation [Jin et al., 2024, Baptista et al., 2024, Tohidi and Dadkhah, 2022]. In dialogue, although AMRs are predominantly used in natural language understanding modules [Bonial et al., 2023, Brutti et al., 2022], some studies highlight their potential in improving NLG as well [Hryhoryeva, 2025, Yang et al., 2024]. LLMs have revolutionised the field by enabling most NLP tasks to be tackled effectively through prompt-based learning techniques [Gu et al., 2022, Gao et al., 2021, Brown et al., 2020]. These methodologies involve augmenting inputs with task-specific instructions, explanations, or task demonstrators to leverage LLMsâ zero-shot and few-shot learning capabilities. Specifically, in this study, we explore new enriched representations that include a task demonstrator in the input. Building on prior work that demonstrated their benefits for a specific domain [VĂĄzquez et al., 2023], we present a novel comparison that explores their impact on multiple domains and datasets. 2.3 NLG datasets for dialogue High-quality datasets have facilitated progress in the research on NLG for dialogue systems. They provide a basic tool to train and evaluate models. Therefore, over the years, researchers have created a variety of datasets tailored to different dialogue tasks, ranging from single-domain scenarios to multi-domain and knowledge-grounded interactions. Wen et al. [2015] and Novikova et al. [2017] were pioneers in creating annotated datasets designed to generate task-specific sentences. Wen et al. [2015] define a series of DAs and slots to develop dialogue systems that provide information about restaurants and hotels. Similarly, the E2E dataset [Novikova et al., 2017], focused on restaurant recommendations, was created to address the problems of classic NLG approaches. Following these early works, multi-domain datasets were developed to address the growing need for systems capable of handling more diverse and complex dialogues. The Schema-Guided Dialogue dataset [Rastogi et al., 2020] introduced a unified schema for intents and slots that enhances scalability and generalisation across domains, even unseen ones. Similarly, MultiWOZ [Budzianowski et al., 2018] is a large-scale dataset that offers richly annotated dialogues in domains like restaurants, hotels, and transportation. This dataset is an evaluation benchmark for tasks such as MR-to-text generation or dialogue state tracking. Recently, the need to create systems that generate knowledge-grounded outputs has resulted in a new trend of datasets. These datasets incorporate information from external sources in their MRs. For instance, ViGGO [Juraska et al., 2019], a dataset in the domain of video games, includes the information extracted from WikiData in its MRs as the content of their attribute-value pairs. In an extension of this work, Vazquez et al. [2024] develop a systematic methodology that combines the ViGGO DAs with WikiData knowledge graphs to create a similar dataset, WikiDialog, with seven new domains. Similarly, OpenDialKG, which obtains external information from the database Freebase, generates knowledge-rich dialogues across domains like music, movies, and sports [Moon et al., 2019]. 4 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT In this study, we compare datasets spanning single-domain (E2E), multi-domain (MultiWOZ), and knowledge-grounded dialogues (ViGGO). Additionally, we have incorporated the EMPATHIC corpus [Torres et al., 2019, Montenegro et al., 2019, Olaso et al., 2021, VĂĄzquez et al., 2023, VĂĄzquez, 2026]. This corpus is extracted from health coaching sessions where a virtual coach helps users change their unhealthy habits in the domain of nutrition. Therefore, we add this scarcely explored and particularly challenging task, coaching, to our already novel and thorough comparison of the impact of an alternative enriched representation across a diverse range of datasets. 2.4 NLG metrics The evaluation metrics for NLG have advanced significantly in recent years [Sai et al., 2022]. However, metrics originally developed for machine translation or summarisation tasks keep being the most widely used for NLG [SchmidtovĂĄ et al., 2024, Sai et al., 2022, Celikyilmaz et al., 2020]. Metrics such as ROUGE [Lin, 2004], METEOR [Banerjee and Lavie, 2005], NIST [Doddington, 2002], and mainly BLEU [Papineni et al., 2002]âwhich rely on n-gram overlapâare almost universally present in evaluations of NLG systems [Vazquez et al., 2024, Ramirez et al., 2023, Wu et al., 2023] and included in the most prominent NLG benchmarks and challenges [Gehrmann et al., 2021, Novikova et al., 2017, Colin et al., 2016]. Consequently, we selected BLEU for our study as a standard metric in NLG and to include a widely used lexical perspective in our evaluation, which complements the other metrics and evaluation approaches considered in this work. Metrics that evaluate semantic accuracy are also widely used in the NLG community [SchmidtovĂĄ et al., 2024, Ramirez et al., 2023, Ruiz et al., 2023, Reed et al., 2022], as they tend to align more closely with human ratings. Among these, BLEURT [Sellam et al., 2020] is particularly relevant due to its strong correlation with human evaluations, as it has been explicitly trained with human ratings. In this work, along with BLEURT, we introduce a new dimension to the analysis using LaBSE [Feng et al., 2022]. LaBSE, a language-agnostic embedding model, computes semantic similarity as the cosine similarity of sentence embeddings. While conceptually distinct, BLEURT and LaBSE have not yet been compared in detail regarding their performance in prior research. NLG metrics typically evaluate outputs based on ground-truth references, but metrics that compare the outputs with their corresponding input have gained importance in recent years [SchmidtovĂĄ et al., 2024]. In dialogue systems, where MRs containing slots are prevalent, slot error rate or Slot Accuracy is often used to evaluate a generatorâs robustness [Wang et al., 2023, Du et al., 2020, Li et al., 2020]. This metric determines how many values, usually entities, in the input are present in the output. Recently, some studies have adopted an alternative approach to analyse DA-based input-output alignment: the Dialogue Act Accuracy (DAC) [VĂĄzquez and Torres, 2025, Vazquez et al., 2024, Ramirez et al., 2023, VĂĄzquez et al., 2023, Reed et al., 2022]. In these works, DA classifiers are proposed to assess output quality, where a generated sentence is considered acceptable if the classifier can predict its source DA. Based on the findings of these works, we incorporated the underexplored DAC along with Slot Accuracy to broaden the scope of our analysis. In summary, our study evaluates NLG engines and representation approaches using a diverse range of automatic metrics, including underexplored dimensions. Additionally, unlike previous works that primarily validate models and methodologies through metrics, we explore metricsâ focus and provide a detailed analysis of how these metrics interact with features of representations, datasets and tasks to shape outcomes. 3 Corpora The datasets define the domains/tasks in which we validate our proposed representations. As we aim to analyse the suitability of different representations across multiple domains, languages, and corpus characteristics, we selected four datasets that differ in domain, size, lexicon, input variability, and acquisition process, among other features. In particular, we have included two single-domain datasets obtained through crowdsourcing, the E2E dataset with only informative samples [Novikova et al., 2017] and the ViGGO corpus with more DA variability [Juraska et al., 2019], one multi-domain dataset that contains complete simulated and annotated dialogues, the MultiWOZ corpus [Budzianowski et al., 2018], and a corpus extracted from simulated human-machine health coaching sessions labelled with ad-hoc MRs, the EMPATHIC corpus [Torres et al., 2019, Montenegro et al., 2019, Olaso et al., 2021, VĂĄzquez et al., 2023, VĂĄzquez, 2026]. These datasets are dialogue-related corpora whose samples consist of MRs and their corresponding example sentences. These MRs, despite differences in corpus characteristics, are all DA-based MRs, where the DA primarily represents the communicative intent of the sentence, while the slot 2 âvalue pairs encapsulate the semantic content. This representation serves as the baseline of our experiments, as we explain in Section 4. Tables 1â4 show that this representation structure is shared across all datasets. 2 The slots can also be referred to as attributes. 5 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT E2E dataset 3 E2E dataset [Novikova et al., 2017] is a well-known corpus in the domain of restaurants obtained through crowdsourcing. Although it was created for dialogue tasks, it was collected through crowdsourcing by writing sentences for predefined MRs without providing any contextual dialogue. Table 1 shows that all the MRs of this dataset use the same DA, inform, with multiple combinations of attributes and values. The two samples in the table inform about two different restaurants, sharing one slot-value pair (eatT ype= pub) and multiple attributes such aspriceRangeornear, but also presenting attributes (customerRatingandfood) and values ("The Vault" vs "The Cambridge Bleu" or "more than ÂŁ30" vs "cheap") that are different. Table 1: Samples of the E2E dataset. MRSentence inform (name = The Vaults ; eatType = pub ; priceRange = more than ÂŁ30 ; customerRating = 5 out of 5 ; near = CafĂ© Adriatic) The Vaults pub near CafĂ© Adriatic has a 5 star rating. Prices start at ÂŁ30. inform ( name = The Cambridge Blue ; eatType = pub ; food = English ; priceRange = cheap ; near = CafĂ© Brazil ) Close to CafĂ© Brazil, The Cambridge Blue pub serves delicious Tuscan Beef for the cheap price of ÂŁ10.50. Delicious Pub food. ViGGO corpus 4 ViGGO corpus [Juraska et al., 2019] is a dataset in the domain of video games collected with the same protocol as E2E. However, ViGGO contains more variability in terms of DA. Table 2 presents samples of three of the nine different communicative intentions: inform, give an opinion or ask about a preference for a specific video game characteristic. The attributes and values appear in multiple slot-value pair combinations in the MRs. The attributes are domain-specific, such as release_year or esrb in the samples of Table 2. Table 2: Samples of the ViGGO corpus MRSentence inform ( name = The Forest of Doom ; release_year = 2014 ; genres = role-playing, text adventure ; has_multiplayer = no ) The Forest of Doom is a role-playing text adventure game released in 2014. It is a single-player only game. give_opinion ( name = Undertale ; esrb = E (for Everyone) ; rating = excellent ) Undertale is a downright excellent game, and itâs rated E so everyone can enjoy it. request_attribute ( has_multiplayer = ? )Do you prefer to game with others, I mean, multiplayer? MultiWOZ corpus 5 MultiWOZ corpus [Budzianowski et al., 2018] is a large-scale dataset that contains dialogues covering seven domains: restaurants, hotels, attractions, taxis, trains, hospitals, and police. It includes annotated complete dialogues between a user and an assistant. In these dialogues, the user asks for information in one of these domains, and the assistant, thought to be replaced by an automatic system, provides this information. They are simulated conversations since they were obtained by employing crowdsourcing in a Wizard-of-Oz (WOZ) framework, where the crowd workers wrote the next turn for a given dialogue context previously created by other workers. As the complete dialogues were annotated, Table 3 presents turns of users (USERin DAs) and assistants (SYSTEM). Unlike the other datasets, MultiWOZ contains MRs that combine simple DAs, like the first example of Table 3. In these cases, the DA is the union of simple DAs, so in this example of the table, the DA is the union ofSYSTEM_Booking_BookandSYSTEM_Restaurant_Inform. This DA is domain-specific, but MultiWOZ also presents open-domain DAs, such as the DA of the second example of the table:USER_General_Thank. Finally, the attributes are domain-specific, likebookdayoraddress, and are the entities that appear in the dialogues. 3 Available free on https://github.com/tuetschek/e2e-dataset 4 Available free on https://nlds.soe.ucsc.edu/viggo 5 Available free on https://github.com/budzianowski/multiwoz 6 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT Table 3: Samples of the MultiWOZ corpus MRSentence SYSTEM_Booking_Book ( bookday = Wednesday ; bookpeople = five ; booktime = 11 am ; name = this restaurant ; ref = DU8IWQZ2 ) & SYSTEM_Restaurant_Inform ( address = 2 Rose Crescent City Centre ; name = The Gardenia ) The Gardenia is located at 2 Rose Crescent City Centre. I have booked your table at this restaurant for five people, Wednesday at 11 am. Your reference number is DU8IWQZ2. USER_general_thank ( )No that will be all, thank you. EMPATHIC corpus 6 The EMPATHIC corpus was gathered during the EMPATHIC project [Torres et al., 2019, Montenegro et al., 2019, Olaso et al., 2021, VĂĄzquez et al., 2023, VĂĄzquez, 2026]. This project aims to change usersâ unhealthy habits through questions and consequent reflection. For this end, a dialogue system was built to carry out health coaching sessions following a behaviour model, the GROW model [Leach, 2020]. Table 4: Samples of the EMPATHIC corpus MRSentence Gen_Hello ( user_name = Elisabeth )Hello Elisabeth! Int_know_coaching ( ) Do you know what coaching is? GSQ_what_obj ( action = change ) Would you like to change anything? RQ_curr_sit ( food = water ) Do you think youâre drinking enough water? During this project, the EMPATHIC corpus was collected through WoZ experiments. These experiments were simulated coaching sessions where a real user spoke with a wizard who played the role of a virtual coach. From these spoken and complete interactions, only the coach turns were annotated in terms of DAs. Table 4 shows four samples of the EMPATHIC corpus, where the ad-hoc MRs designed for this corpus can be observed. This corpus presents two similarities between this dataset and MultiWOZ. First, it contains open-domain DAs such asGen_Helloand close-domain DAs such asGSQorRQ. This second group is highly bound to the GROW behavioural model based on questions, soGSQandRQstand for Goal Set Question and Reality Question, respectively. Second, the attributes, which are close-domain, were the entities found in the previously acquired conversations. Corpora features Table 5 summarises the characteristics of the four datasets and shows how different they are. The first two columns have already been mentioned in the definition of the datasets. So, in terms of domains, only E2E and MultiWOZ share a domain, restaurants. Meanwhile, only E2E and ViGGO employed the same protocol for the acquisition processes. Table 5: Characteristics of the four selected corpora. Domains stands for the topics the datasets cover. Acquisition process means the methodology to obtain the datasets. Numbers of DAs, attributes, and MRs quantify the diversity of inputs. Corpus size stands for the number of MR-sentence pairs. Running words include all the words in the corpus, whereas Vocabulary is the number of different words. In the Number of DAs for MultiWOZ, 798 represents the total number of DAs that appear as a combination of the 52 simple DAs defined for the corpus. DatasetDomains Acquisition process Number of DAs Number of attributes Number of MRs Corpus size Running words Vocabulary E2ERestaurantsCrowd186.0k51k1.0M5.1k ViGGOVideo gamesCrowd9142.3k6.9k150k4.4k MultiWOZ7 domainsWOZ + Crowd798 (52)2554k140k1.9M30k EMPATHIC CoachingWOZ78211.2k2.3k13k1.7k 6 Available at low cost on https://catalog.elra.info/en-us/repository/browse/ELRA-S0414/ 7 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT The number of DAs and attributes in relation to the number of MRs also makes a difference between the corpora. While E2E only employ one DA and eight attributes for 6k MRs, EMPATHIC employ many more DAs (78) and attributes (21) for a reduced quantity of MRs (1.2k). These numbers represent a big difference in the variability of the MRs. Only the large-scale MultiWoZ contains higher variability than EMPATHIC in the representations, with 52 simple DAs combined for a total of 798 DAs, due to its special MRs. Another important difference among the corpora is their size, i.e, the number of samples. In this regard, MultiWOZ and E2E are much bigger than the other two datasets. The datasets also differ in the number of references. The references are the sentences associated with the same MR in the corpus. This feature does not appear in the table but is highly bound with the Corpus size and the Number of MRs. The first feature is higher than the second for all the datasets, i.e., all the datasets present multiple sentences for the same MR. In Section 6.4, we analyse the importance of this dataset characteristic. Finally, while Running words are the total words in the corpus, Vocabulary stands for the number of different words. In this regard, E2E adds a low variability of the lexicon og their sentences to its low MR variability with 1M of running words but only 5.1k different words. By contrast, the rest of the datasets show a notable lexicon variability, where, for instance, ViGGOâs vocabulary is very close to E2Eâs with one order less in Corpus size and Running words. 4 Meaning representations In this section, we propose different ways to represent the input to feed NLG systems. First, we chose the original MRs of the corpora (Table 1â4) as Baseline representations. As we described in Section 3, these MRs consist of a DA and a set of slot-value pairs. Next, building on the promising research in VĂĄzquez et al. [2023], we propose including a task demonstrator in the input connected to the Baseline MR. This Baseline representation is also referred to as input MR because it is the MR for the generation in all these representations. Meanwhile, the task demonstrator is a sample extracted from the corpora, consisting of an MR (demonstrator MR) and its corresponding sentence (demonstrator sentence). These enriched representations change both the input at inference time and the training data. Figure 1: Meaning representations overview. One Baseline representation and three Prompt representations. The Baseline representation, which is the MR found in the original datasets, is the input MR (orange boxes) for the generation in all the representations. Prompt representations are created by connecting a task demonstrator with the input MR. This task demonstrator includes an MR, the demonstator MR (green), and its corresponding example sentence, the demonstrator sentence (grey). The colours of the boxes, the letters n and m for the number of the attributes and the use (or not) of apostrophes inside Input and Demonstrator MR boxes are used to indicate which elements are necessarily identical (or not) among the representations. Figure 1 shows a schema of the four representations of the meaning studied in this work. The figure shows that Prompt 1, Prompt 2, and Prompt 3 follow the same structure, but they differ in how similar demonstrator MR is to the input MR. In this regard, the conditions to select the demonstrator for each Prompt representation are as follows: âąPrompt 1: The DA of the demonstrator MR must be the same as the input MR, as it is indicated with the light blue boxes in Figure 1. âąPrompt 2: Demonstrator MR must include the same DA (light blue boxes) and number of attributes (n) as the input MR 8 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT âąPrompt 3: The Demonstrator MR must contain the same DA (light blue boxes) and attributes (light yellow boxes) as the input MR. Table 6: Examples of task demonstrators linked to an input MR for each dataset. DAs and attributes are in bold and coloured to highlight identical elements between the demonstrator MRs and the input MR. Note that brown colour is employed for those attributes that do not appear in the input MR, but they do in any demonstrator MR. Prompt 1 was discarded for E2E because the selection methodology would assign the same task demonstrator for all the E2E MRs. Dataset Example type Examples E2E Input MR inform ( name = Zizzi ; eatType = coffee shop ; customer rating = high ; near = Burger King ) Prompt 2 demonstrator inform ( name = Loch Fyne ; eatType = restaurant ; familyFriendly = yes ; food = English ) + A child friendly restaurant that has English food is Loch Fyne. Prompt 3 demonstrator inform ( name = Wildwood ; near = CafĂ© Rouge ; customer rating = low ; eatType = restaurant ) + Wildwood, a restaurant with a low customer rating, is near CafĂ© Rouge. ViGOO Input MR give_opinion ( name = Super Mario World ; player_perspective = side view ; rating = excellent ; has_multiplayer = yes ) Prompt 1 demonstrator give_opinion ( name = Guitar Hero: Smash Hits ; rating = poor ; developer = Beenox ) + Guitar Hero: Smash Hits is just not my kind of game. I find Beenox games are often not very fun. Prompt 2 demonstrator give_opinion ( name = Worms: Reloaded ; player_perspective = side view ; rating = average ; available_on_Steam = yes ) + Worms: Reloaded is an average quality side view game you can at least find on Steam. Prompt 3 demonstrator give_opinion ( name = Portal 2 ; player_perspective = first person; has_multiplayer = yes ; rating = excellent ) + Iâve been having an absolute blast playing Portal 2. First person games are always my favorite, but the multiplayer makes it even better. MultiWOZ Input MR SYSTEM_Booking_Book ( bookstay = 2 ; ref = VH33JKKF ) SYSTEM_general_reqmore ( ) Prompt 1 prompt SYSTEM_Booking_Book ( ref = GS2CO4IV ) SYSTEM_general_reqmore ( ) + Your table is booked and will be reserved for 15 minutes. Your reference number is GS2CO4IV. May I help you with anything else today? Prompt 2 demonstrator SYSTEM_Booking_Book ( name = Cambridge Belfry ; ref = LJOEY6H2 ) SYSTEM_general_reqmore ( ) + Okay. Your booking at the Cambridge Belfry was successful. The reference number is LJOEY6H2 . Can I help you with anything else today? Prompt 3 demonstrator SYSTEM_Booking_Book ( bookstay = 4 ; ref = 3VUMBFZ0 ) SYSTEM_general_reqmore ( ) + Great, youâre booked for 4 nights with reference number 3VUMBFZ0 . May I help you with anything else? EMPATHIC Input MRRQ_curr_sit ( action = cook ) Prompt 1 demonstrator RQ_curr_sit ( action = tell ; freq = daily ) + Can you tell me about your daily eating habits? Prompt 2 demonstrator RQ_curr_sit ( food = water ) + And do you think you drink enough water? Prompt 3 demonstrator RQ_curr_sit ( action = change ) + Would you like to change your eating habits? These conditions for selecting a demonstrator make task demonstrators from Prompt 3 more specific to the Baseline than those from Prompt 2, and in turn, those from Prompt 2 more specific than those from Prompt 1. Table 6 presents real examples, one for each dataset, of input MRs (first row of each dataset) with their corresponding demonstrator for the three Prompt representations (next rows). They illustrate that the conditions stated above are satisfied and show the specificity of the demonstrators for each proposed representation. In MultiWOZ, for instance, the DA for the Baseline representation and for the Demonstrator MR of the three enriched representations isSYSTEM_Booking_Book & SYSTEM_general_reqmore. Next, the number of attributes of the Input MR in this dataset is two, which matches the number of them in the corresponding Demonstrator MR of Prompt 2 and Prompt 3. Finally,bookstayandref (highlighted in red and orange in all the rows of this dataset) are the attributes for the Baseline representation and the Demonstrator MR of Prompt 3. By contrast, Prompt 1 and Prompt 2 present attributes that do not appear in the original representation (highlighted in brown). The same restrictions are followed in the other datasets. Prompt 1 was discarded for E2E because the selection methodology, which we explain next, would assign the same task demonstrator for all the E2E MRs. 9 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT Table 7: Number of different demonstrators in each dataset for the three Prompt representations. Dataset Prompt 1Prompt 2Prompt 3 E2E- 7 6120 ViGOO921473 MultiWOZ798228111644 EMPATHIC 78197537 In this proposal, we also aim to reduce the number of different demonstrators in training, making our approach extensible even to datasets with reduced size. Thus, our selection methodology of demonstrators assigns the same demonstrators to all MRs that satisfy identical conditions. In other words, all the MRs from the original dataset with the same DA are connected to the same demonstrator for Prompt 1, similarly, those with the same DA and number of attributes for Prompt 2, and the same DA and exact attributes for Prompt 3. Therefore, the number of different demonstrators in the training set matches the number of DAs for Prompt 1, the combination of DAs and the number of attributes for Prompt 2, and the combinations of DAs and attributes for Prompt 3. Table 7 shows the number of demonstrators for each dataset in each Prompt representation. This table confirms that the number of demonstrators for Prompt 1 matches the number of DAs (Table 5). Consequently, the number of different demonstrators for Prompt 1 in E2E would be one. So, we discarded this representation for this dataset since all input MRs would have been linked to the same demonstrator. In addition, as expected, the more specific the demonstrators, the higher the number of them, i.e., Prompt 3 always contains more than Prompt 2, which, in turn, contains more than Prompt 1. Lastly, the corpora with more MR variability, such as MultiWOZ and EMPATHIC, employ more demonstrators than, for instance, E2E, with a very low variability in this regard. 5 Metrics The metrics play an important role in this work. We aim to explore what each metric can detect and understand how the features of tasks, corpora and input representations affect the model evaluation with each metric. To this end, we have selected a range of metrics that allow us to evaluate different aspects of the outputs: lexical, semantics or coherence with the input, among others. They can be split into reference-driven and reference-less metrics and are as follows: âąReference-driven metrics: Metrics whose scores are obtained by comparing each generated sentence with the multiple references found in the corpus. We have selected three reference-driven metrics that cover lexical and semantic aspects as well as, to some extent, human criteria. âBLEU [Papineni et al., 2002]: Metric universally employed in the NLG-related works, although it was originally created for the translation task. It is a good indicator of lexical accuracy as its evaluation is based on n-gram overlapping. In this work, we employ BLEU-4 (with n-grams from uni-grams to 4-grams) with a smoothing function [Chen and Cherry, 2014]. âBLEURT [Sellam et al., 2020]: Machine-learned metric that aims to replicate human evaluation to some extent, thanks to its training process. This process employed a BERT model trained with a public collection of 430k sentence pairs and ratings based on human perception for each pair. As a result, the model returns a similarity value for each pair of sentences. âLaBSE [Feng et al., 2022]: Metric based on sentence embeddings. They are obtained with a BERT dual encoder characterized by being a language-agnostic model that supports more than 100 languages. The metric scores are the cosine similarity between the sentence embeddings of generation and references. âą Referenceless metrics: They can also be named source-based metrics because they compare the generated sentence (the output) with the MR (the input). In this case, we have selected two metrics that evaluate the coherence of the generation in terms of semantic content and communicative intention. â Slot Accuracy [Li et al., 2020]: Metric that focuses on the values associated with the attributes in the MRs. It evaluates whether the model includes them in the generated sentences. So, the score is the percentage of values in the input that appear in the output, giving an indicator of the robustness of the 7 There is no Prompt 1 representation for E2E, because E2E only has one DA, so all the MRs from the original corpus would be connected to the same demonstrator for this representation. 10 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT NLG in terms of how much input information is communicated in the output. Note that this metric does not include generated sentences obtained from MR without attributes. Therefore, the evaluation set could be smaller than the one used in other metrics. â Dialogue Act Accuracy [Ramirez et al., 2023]: The DAC is an underexplored metric that quantifies the accuracy of classifiers to predict the source DA of a generated sentence. The scores are the results of classification problems where the inputs are the generated sentences, and the outputs are the DAs. In this work, four corpora were selected that differ in the number and typology of DAs. So, different types of DA classifiers were trained. There was no classifier for E2E because it only has one DA. Then, ViGGO and EMPATHIC classifiers are multiclass, as only one DA can be assigned to each generated sentence. By contrast, the MultiWOZ classifier is multilabel because its MR accept DAs that are a combination of simple DAs. For the classifiers, we employed DistilBERT models [Sanh, 2019] because they have demonstrated great training capabilities despite their reduced size. In addition, we split the original datasets in a proportion of 95/5 for training and validation, where the validation set gave us reference values to determine when a generation is reasonably good in this regard. 6 Results & discussion This section presents a thorough comparative study that defines the objective of our work. This comparison allows us to analyse three different aspects: a) which representation of the meaning best leverages the capabilities of the generative models, b) the impact of these meaning representations on the generation quality across datasets, and c) the influence of dataset and task features in the model evaluation with each metric. Therefore, we will have a comprehensive view of how these three elements â representations, datasets, and metrics â interact and affect the final results. In general terms, the study has shown that enriched representations perform particularly well on small datasets involving complex tasks and high variability in MRs, as well as the lexicon and semantics of the sentences. However, they can act as distractors if the baseline MR is already long and contains a lot of semantic information. These enriched representations are also effective in zero-shot 8 settings, obtaining better results than the Baseline before training. The analysis of the metrics reveals that those that evaluate semantic aspects capture output quality more accurately than those that focus on the lexicon. Finally, the generative models stand out for their capacity to adapt to different tasks and for their robustness in handling the input at both semantic and communicative intention levels. To explain this detailed study with the highest clarity, we structure the section as follows. First, we present the experimental framework for training the models and generating the sentences in Section 6.1. Next, we show and discuss the results obtained in the evaluation of the generated sentences. The analysis of the results is split into results with reference-driven (Section 6.2) and referenceless metrics (Section 6.3). Next, Section 6.4 goes deeper into evaluating the importance of dataset features in the results, exploring specific corpus features such as the number of references and attributes per MR. Finally, Section 6.5 describes a human study of the generations that allows us to determine the role of metrics and representations in the results. 6.1 Experimental framework We selected one of the pre-trained GPT-2 models [Radford et al., 2019] due to their acceptable computational require- ments and their demonstrated suitability for fine-tuning and prompt-based learning approaches [Peng et al., 2020, Liu et al., 2023, VĂĄzquez et al., 2023]. In particular, we perform experiments with the GPT-2 Medium model to validate the proposed MRs and compare their impact on the generations across domains. These experiments involve fine-tuning procedures with 5-fold cross-validation, ensuring that the modelsâ performances are rigorously evaluated across different subsets of the data. For consistency with the pre-training setup, we utilise the same byte-pair tokeniser [Sennrich et al., 2016] during the fine-tuning process. In training, we perform over five epochs using a learning rate scheduler with a linear warm-up that starts 5e-5, a batch size of 8 and an Adam optimiser with weight decay [Kingma and Ba, 2014]. For the generation phase, we generate five distinct outputs per MR. Each output is constrained to a maximum length of 80 tokens, whereas we set the temperature to 1.0, stimulating output variability. 8 Strictly speaking, when using representations that include one demonstrator, these experiments correspond to a one-shot setting rather than a pure zero-shot one. Here, we use the term âzero-shotâ to emphasise that no additional task-specific training or fine-tuning is performed. 11 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT 6.2 Analysis of the results with reference-driven metrics To assess how well the different representations support generation quality across domains, we begin by analysing the scores obtained for the reference-driven metrics, namely BLEU, BLEURT, and LaBSE. These metrics calculate their scores by comparing the generated outputs, five per MR, with the references related to such MRs in the original corpora. This analysis offers insight into the behaviour of enriched and baseline representations across different datasets, using as evaluation criteria metrics that rely on such predefined outputs. To obtain the results for each metric, first, a score (generation score) is computed for each one of the five generations per input by comparing them separately with all the references. The scores for each MR (MR scores) are the average of the five generation scores. Finally, the average scores for each metric is the average over all the MR scores. These scores are calculated for each combination of metrics, representations, and datasets. Note that in this analysis, we focus on the average scores after fine-tuning (Table 8) and the evolution of these scores before training, epoch 0, and after each epoch (Figure 3), as well as the distribution of the MR scores for the outputs generated with the fine-tuned models (Figure 2). Average scores after fine-tuning Tables 8a, 8b, and 8c show the scores obtained after fine-tuning the models with each representation-dataset pair for BLEU, BLEURT, and LaBSE, respectively. These tables provide an initial overview of the modelsâ performance differences across metrics and allow us to identify which representation obtains the best results (denoted in bold) for each dataset. Table 8: Average scores after fine-tuning for each reference-driven metric, representation, and dataset. Base corresponds to the baseline representation, whereas Pi is the Prompt i representation with i=1,2,3. The datasets (Data) are E2E, ViGGO (ViG), MultiWOZ (MWOZ), and EMPATHIC (EMP). No Prompt 1 for E2E. The best score for each dataset is denoted in bold. (a) BLEU DataBaseP1P2P3 E2E0.15-0.120.13 ViG0.210.150.170.19 MWOZ0.160.140.140.17 EMP 0.170.190.200.26 (b) BLEURT DataBaseP1P2P3 E2E0.60-0.540.54 ViG0.650.580.610.63 MWOZ0.600.570.580.59 EMP 0.550.560.570.60 (c) LaBSE DataBaseP1P2P3 E2E0.76-0.690.70 ViG0.820.730.770.79 MWOZ0.680.660.670.67 EMP 0.660.680.680.70 Low BLEU resultsTable 8 shows very low results for BLEU. Its values (no higher than 0.3) are much lower than the results obtained with the other two metrics (no lower than 0.5). This discrepancy with the other two metrics suggests that lexical similarity based on n-gram overlap may not adequately quantify the output quality. Demonstrators effective in small datasets with complex tasks Table 8 shows that Prompt representations are effective in datasets with the special characteristics of EMPATHIC. It seems that the demonstrators take more relevance when the dataset is small, with a high variety in the input and the output, and a complex task. For E2E, ViGGO and MultiWOZ, the results with the enriched representations are satisfactory, but the models perform better with the Baseline MR. Prompt 3: the best prompt Prompt representations obtain better results when they include more specific task demonstrators in their input, across all datasets. In other words, Prompt 3 consistently outperforms Prompt 2, which in turn is better than Prompt 1. These more specific representations require a higher number of demonstrators during training, as shown in Table 7 of Section 4. This may suggest that good performance depends on having a sufficient number of demonstrators. However, the results obtained with Prompt 1 for EMPATHIC show that enriched representations can still be effective even when the number of training demonstrators is low. Distribution of the scores We introduce in this section a new type of analysis: the distribution of the scores assigned to each MR, i.e., the distribution of the MR scores. This analysis enables us to determine whether the average values reported in the previous 12 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT (a) E2E (BLEU)(b) E2E (BLEURT)(c) E2E (LaBSE) (d) ViGGO (BLEU)(e) ViGGO (BLEURT)(f) ViGGO (LaBSE) (g) MultiWOZ (BLEU)(h) MultiWOZ (BLEURT)(i) MultiWOZ (LaBSE) (j) EMPATHIC (BLEU)(k) EMPATHIC (BLEURT)(l) EMPATHIC (LaBSE) Figure 2: Distribution of the MR scores of each reference-driven metric after the fine-tuning for each representation in each dataset. subsection stem from consistent behaviour across all examples or from a combination of highly variable outputs. In other words, it helps determine whether the observed averages reflect normally distributed scores around a central value or instead emerge from large disparities between high- and low-quality generations. Figure 2 presents these score distributions for each combination of metric, dataset, and representation. Each row in the figure corresponds to a specific dataset (E2E, ViGGO, MultiWOZ, and EMPATHIC, in that order), and each column corresponds to a specific metric (BLEU, BLEURT, and LaBSE, respectively). Within each subplot, the distribution of the MR scores is plotted separately for each representation. Metrics, the main difference The distributionsâ shapes are mainly associated with the metric. While BLEU is characterised by having most of its values close to 0.0, BLEURT and LaBSE generally follow Gaussian distributions with a higher average for LaBSE. Singularities of EMPATHIC and MultiWOZ The distributions highlight the different behaviour of the model on EMPATHIC. Unlike the Gaussian-shaped curves observed for BLEURT and LaBSE in the other datasets, EMPATHIC displays bimodal-like distributions, particularly pronounced in LaBSE scores. Additionally, it is the only dataset with a 13 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT (a) E2E (BLEU)(b) E2E (BLEURT)(c) E2E (LaBSE) (d) ViGGO (BLEU)(e) ViGGO (BLEURT)(f) ViGGO (LaBSE) (g) MultiWOZ (BLEU)(h) MultiWOZ (BLEURT)(i) MultiWOZ (LaBSE) (j) EMPATHIC (BLEU)(k) EMPATHIC (BLEURT)(l) EMPATHIC (LaBSE) Figure 3: Evolution of the average score of each reference-driven metric during the fine-tuning for each representation in each dataset. considerable portion of BLEU scores spread across the full range, including high-end values. Meanwhile, MultiWOZ exhibits very similar distributions across representations for each metric. We hypothesise that representation influence diminishes when fine-tuning is performed on large-scale datasets, such as MultiWOZ. 14 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT Evolution of the scores To complement the analysis of average scores, we examine how model performance evolves throughout the fine-tuning process. This dynamic perspective helps us understand not only the final outcomes but also how different representations support the modelâs progressive adaptation to the task. In particular, tracking the evolution of scores across epochs allows us to assess the zero-shot effectiveness of the representations and their impact on the modelâs learning. Figure 3 presents learning curves for each combination of metric, dataset, and representation, showing the average score before training (zero-shot) and after each of the five epochs. The figure follows the same format as Figure 2, with datasets arranged by row and metrics by column. Within each plot, one learning curve is shown for each representation. Prompt-representations trigger model zero-shot capabilitiesAll the learning curves present higher values for the Prompt representations than for the Baseline before training (epoch 0), except for BLEU (the first column of plots), whose curves start very close to 0.0 across all datasets and representations. In other words, the zero-shot capabilities of the generative models are activated by the enriched representations. Even in datasets that do not benefit from these representations after fine-tuning, this improvement is observed in the zero-shot setting. Therefore, we show that basic enriched inputs with a single task demonstrator can improve generation quality before training, even in moderately sized models like GPT-2. Good task adaptability The curves generally show most of their improvement during the first epoch, reaching reasonably high scoresâexcept for BLEU. Thus, we confirm that a capable generative model, once fine-tuned, can be easily adapted to multiple tasks and domains. MultiWOZ: slow adaptability with overly general demonstrators The curves for Prompt 1 and Prompt 2 in MultiWOZ exhibit a more gradual learning process compared to the general behaviour. Although the final scores after five epochs are comparable, the curves do not show most of the improvement in the first epoch. This behaviour may be explained by two factors: a) the limited number of demonstrators for Prompt 1 and Prompt 2 (Table 7) compared to the size of MultiWOZ (Table 5), and b) the high output variability of MultiWOZ. These two factors might result in generic task demonstrators that fail to cover the output variability in MultiWOZ. As a result, the demonstrators seem not to be informative enough for all MR, making the adaptation process more challenging for the generative model. 6.3 Analysis of the results with referenceless metrics Reference-driven metrics need scenarios with predefined references to compare, and they work better as long as these references are representative enough for the given inputs. In addition, their outcomes can be affected by the number of references associated with the MRs, as we will present in Section 6.4. In this work, we decided to employ referenceless metrics to overcome this reference necessity and widen the perspective with two completely different metrics: Slot Accuracy and Dialogue Act Accuracy. We discuss the results of these two metrics separately because Slot Accuracy can be evaluated as a generation problem, while DAC is a classification problem. Slot Accuracy Table 9: Slot accuracy average scores after fine-tuning for each representation and dataset. Base corresponds to the Baseline representation, whereas Pi is the Prompt i representation with i=1,2,3. The datasets (Data) are E2E, ViGGO (ViG), MultiWOZ (MWOZ) and EMPATHIC (EMP). No Prompt 1 for E2E. The best score for each dataset is denoted in bold. DataBaseP1P2P3 E2E0.79-0.650.67 ViG0.800.690.720.72 MWOZ0.940.910.910.91 EMP 0.880.860.870.90 The Slot Accuracy measures how accurately the generated sentence expresses the slot values specified in the input. To analyse the scores for this metric, first Table 9 presents the average scores after fine-tuning, then Figure 4 shows the distribution of the MR scores, and finally Figure 5 depicts the evolution of the average scores over training. Note that, 15 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT unlike reference-driven metrics, a generation score is computed by comparing such generation directly with the source MRs, rather than with multiple reference sentences. After that, the process to calculate the MR scores and the average scores is the same as for the reference-driven metrics. High semantic coherence Table 9 presents excellent results mainly for EMPATHIC and MultiWOZ. These two datasets obtain around 90% of accuracy for all the MRs. E2E and ViGGO also obtain good results, mainly with the Baseline representation. These results mean that the fine-tuned models are semantically coherent with the input. (a) E2E (Slot Accuracy)(b) ViGGO (Slot Accuracy) (c) MultiWOZ (Slot Accuracy)(d) EMPATHIC (Slot Accuracy) Figure 4: Distribution of the Slot Accuracy MR scores after fine-tuning for each representation in each dataset. (a) E2E (Slot Accuracy)(b) ViGGO (Slot Accuracy) (c) MultiWOZ (Slot Accuracy)(d) EMPATHIC (Slot Accuracy) Figure 5: Evolution of the average scores of Slot Accuracy during the fine-tuning for each representation in each dataset. 16 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT Two distribution behaviours The distributions (Figure 4) depict two kinds of behaviours, which are a direct consequence of the distribution of the number of attributes in each dataset, as we explain in Section 6.4. MultiWOZ and EMPATHICâs distributions are concentrated around the top values due to the low number of attributes in their inputs. Conversely, the higher variance in the number of attributes for E2E and ViGGO results in multiple peaks in their distribution of scores. Slot Accuracy confirms the model behaviour Slot Accuracy confirms the model behaviour with the different representations and datasets. First, we observe the benefit of the Prompt representations for datasets like EMPATHIC (Table 9). Next, the performance improvement in zero-shot settings across all the datasets with the enriched inputs is even more evident for this metric (Figure 5). Finally, this figure also depicts the fast adaptability of all the models to the new tasks, except for the representations with very generic demonstrators for MultiWOZ. Dialogue Act accuracy The DAC scores are the results of the classification of the generated sentences in terms of DAs. To discuss them, Table 10 presents the average scores after fine-tuning, while Figure 6 depicts the evolution of those scores during the fine-tuning process. Note that E2E is excluded from this analysis, as it contains only one DA, i.e., only one possible classification output. The analysis also does not present the distribution of MR scores because 0 (incorrect prediction DA) and 1 (correct) are the only possible outcomes of the classifiers. In addition, the table and figure show the results obtained using the validation partition as a reference for a good accuracy value in each dataset. The sentences evaluated in this validation set are from the corpus, whereas the rest of the evaluation is over sentences generated by the model. Table 10: Dialogue Act Accuracy average scores after fine-tuning for each representation in each dataset. Val and Base correspond to the Validation dataset and Baseline representation, respectively, whereas Pi are the Prompt i representations with i=1,2,3. The datasets (Data) are ViGGO (ViG), MultiWOZ (MWOZ) and EMPATHIC (EMP). The best score for each dataset is denoted in bold, the Validation result is excluded from this comparison. DataValBaseP1P2P3 ViG0.980.990.650.890.94 MWOZ0.850.860.520.680.83 EMP 0.680.650.720.700.70 (a) ViGGO (DAC)(b) MultiWOZ (DAC)(c) EMPATHIC (DAC) Figure 6: Evolution of the Dialogue Act Accuracy average scores during the fine-tuning for each representation in each dataset. No E2E as it only contains the inform DA. High coherence with communicative intentions The results in Table 10 illustrate the excellent classification capabilities of the selected models and the coherence of the generated sentences with the communicative intentions given by the DAs. First, the Validation column proves the exceptional performance of the DistilBERT models as classifiers. 98% with 9 DAs for ViGGO, 85% with 52 DAs in a multilabel classification problem for MultiWOZ, and 68% with 78 DAs for EMPATHIC are excellent results, considering the number of DAs for each dataset. Secondly, the results obtained with the generations for the different representations (last four columns) prove that these models can generate coherent sentences regarding communicative intention. Some results have even exceeded those obtained with the validation set. 17 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT Same model behaviour across all the metricsFigure 6 confirms some features observed for the model behaviour in the interaction with the datasets and representations, such as the zero-shot capabilities with the Prompt representations, the fast (or slow in some specific cases) model adaption to the task or the prompt influence in the generation for each dataset. This is very relevant because the metrics evaluate different aspects of the generated sentences. 6.4 Impact of the corpus characteristic on the results The datasets used in this work are highly diverse, as described in Section 3. In this section, we analyse whether this diversity in corpus characteristics significantly affects the results. Specifically, we focus on two features shaped by the design and acquisition protocols of the datasets: the number of references and the number of attributes per MR. The number of references determines how many reference sentences are available for comparison in reference-driven metrics, potentially influencing these scores. In turn, the number of attributes is considered a measure of the complexity of the input. We examine the impact of these two corpus features on both lexical (BLEU) and semantic (LaBSE) metrics computed over the fine-tuned models. The section first describes the distribution of these features across datasets, and then analyses their impact on the results separately. Relevant corpus characteristics for generation For the study of the influence of these two features, we first examine the distribution of MRs in each dataset by number of attributes and references. (Figure 7). The plots of this figure show the percentage of MRs that contain the number of references or attributes indicated on the x-axis of each plot. In the case of the number of attributes (Figure 7b), the numbers go from 0 to 8, whereas the x-axis of Figure 7a presents both specific numbers of references and intervals, such as from 6 to 10 (6-10) or more than 1000 (>1000). (a) Number of references(b) Number of attributes Figure 7: MR distributions in terms of the number of references/attributes. The number of references/attributes is given on the x-axis with specific numbers of references/attributes or intervals. The distributions are in percentages. Figure 7a shows that all the MRs of ViGGO have 3 references 9 , while E2E, acquired with the same protocol, has been designed to have MRs that principally contain between 6 and 20 references. By contrast, half of the MultiWOZ and EMPATHIC MRs have only one reference, but these datasets also have a significant number of MRs with more than 100 references or even more than 1000 in the case of MultiWOZ. These distributions are influenced by the acquisition process and the design of the datasets. Datasets like E2E and ViGGO, which write and select sentences for predefined MR, have more control over this feature than those that need to annotate previously created conversations, such as MultiWOZ and EMPATHIC. The design of the datasets also influences the distribution of the number of attributes per MR (Figure 7b), with similarities between the same datasets. E2E and ViGGO were created to train NLG engines that can manage inputs with multiple attributes. Consequently, these two datasets always present at least one attribute and usually more than three, with a predominance of MRs between 3 and 4 attributes in ViGGO and between 5 and 6 in E2E. By contrast, the attributes for MultiWOZ and EMPATHIC are the entities found in the conversations during the annotation process. As a result, more than half of the MRs of these two datasets present no attributes or only one, whereas the proportion for MRs with more than four attributes is almost irrelevant. 9 There are some exceptions because some MRs were included several times to increase their training weight. But in practice, all the MRs are linked to 3 references. 18 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT Influence of the number of references Figure 8 shows the distribution of scores in terms of the number of references across datasets. This figure includes three subplotsâone per datasetâeach showing the BLEU (top) and LaBSE (bottom) scores obtained with the fine-tuned models. Note that the ViGGO dataset is not included in this analysis because all its MRs have three references. Each point in the plots represents the average score over all MRs that share the same number (or interval) of references as indicated on the x-axis. For example, a point over the label 6â10 corresponds to the average score computed across all MRs that have between 6 and 10 references. (a) E2E(b) MultiWOZ (c) EMPATHIC Figure 8: Results for BLEU (top) and LaBSE (bottom) scores in terms of the number of references per MR with the fine-tuned models. The x-axis establishes specific numbers of references or intervals of numbers of references. No meaningful differences across metrics and representations The results for BLEU and LaBSE in each dataset only differ in the absolute values; the overall trends are the same. In the comparison between representations, the differences observed in the results mainly appear in the MRs with one reference. So, we can assume that the impact of the representations on the model is not affected by the number of references. Higher impact in datasets with lexically and semantically different references The comparison across datasets reveals a distinct effect of the number of references on the scores. The scores for E2E remain very similar regardless of the number of references. However, MultiWOZ and EMPATHIC exhibit a clear decrease in their scores as the number of references increases. We hypothesise that this is related to the output variability of the datasets: high for MultiWOZ and EMPATHIC, and low for E2E, as shown in Section 3. The high variability in MultiWOZ and EMPATHIC implies that the references often differ in both lexicon and semantics. For example, the MRSYSTEM_general_bye ( )from MultiWOZ presents 5394 references that contain very lexically and semantically different sentences, such as "I hope 19 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT you enjoy your stay. Contact us anytime. Goodbye.", "Thanks for using our system today!" and "You have a good day too.". This reduces the likelihood that a generated sentence will obtain a high generation score against all the references, producing a lower average. In contrast, in E2E, even the MRs with the highest number of references contain very similar sentences. In fact, the MR with the highest number of references, 46, for this dataset isinform ( name = Loch Fyne ; eatType = restaurant ; food = English ; familyFriendly = yes )with references like "A child friendly restaurant that has English food is Loch Fyne.", "A good kid-friendly, English restaurant is Loch Fyne." and "Loch Fyne serves English food. Itâs a family friendly restaurant.". Consequently, the number of reference have little impact on this dataset. Influence of the number of attributes (a) E2E(b) ViGGO (c) MultiWOZ(d) EMPATHIC Figure 9: Results for BLEU (top) and LaBSE (bottom) scores in terms of the number of attributes per MR with the fine-tuned models. Figure 9 shows the effect of the distribution of the number of attributes over the four datasets. The figure is the equivalent of Figure 8 but for the number of attributes. In this case, the ViGGO dataset is included in the study as its MRs do present variability in the number of attributes (Figure 7b). Increasing impact of the meaning representation with the number of attributesFigure 9 shows consistent trends for BLEU and LaBSE: the higher the number of attributes, the bigger the differences among the representations. Therefore, the structure of the MRs becomes more relevant when the input includes a high number of attributes. Issue with artificial perfect scores Figure 9d shows scores of 1.0 for EMPATHIC MRs with 6 attributes in both metrics. Although these values may look highly positive, they actually reflect an undesired behaviour. Due to the small 20 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT size of EMPATHIC and the very limited number of MRs with 6 attributes (Figure 7b), we found that these perfect scores come from MRs with a single reference that also appears in the task demonstrator of the Prompt 3 representation. In these cases, the model simply copies the demonstrator sentence, which matches the only reference used for evaluation, leading to artificially perfect scores. For example, the MRRQ_curr_sit_freq (action = think ; action = eat ; number = five ; food = fruits ; food = vegetables ; freq = per day )is the only MR with this DA and these attributes. Therefore, its reference and the Prompt 3 demonstrator are identical: "And do you think you eat five fruits and vegetables per day?". Moreover, the 5 generated samples are exactly the same sentence. Note that the effect on the average scores is negligible, since this happens in less than 1% of the samples. Still, these observations suggest that datasets that include complex MRs with few references may benefit from preprocessing steps such as paraphrasing or other data augmentation techniques to reduce this behaviour. Direct relation with Slot Accuracy resultsThe number of attributes has a direct impact on the Slot Accuracy results. If we go back to the distributions of results for the Slot Accuracy (Figure 4) and the distributions of the number of attributes per MR Figure 7b, we can observe one behaviour for MultiWOZ and EMPATHIC and another for E2E and ViGGO. The first two datasets tend to have fewer than three attributes in their MRs, so the percentages of Slot Accuracy are usually 0, 50, or 100%. Consequently, the distributions of results for these two datasets are mostly observed around these values. Conversely, E2E and ViGGO usually present inputs with between three and eight attributes. So, the possible values for the Slot Accuracy percentage are varied, and the distribution of Slot Accuracy scores presents a significant proportion of MRs across the entire range. In addition, the size of the evaluation set for Slot Accuracy is drastically reduced for MultiWOZ and EMPATHIC, since inputs with no attributes are not included in their evaluation, which accounts for a substantial portion of these two datasets (around 40%). 6.5 Qualitative analysis of the generations To complement the quantitative evaluation with a more in-depth understanding of the model outputs, we provide a qualitative analysis of the generated sentences. For this human evaluation, a manual selection of examples from ViGGO and EMPATHIC was carried out. We have selected these two datasets because they present notable differences in both design and performance. In this assessment, we first analyse why BLEU scores are much lower than those obtained with the other two reference-driven metrics: BLEURT and LaBSE. Then, we carry out a similar and novel comparison between these two semantic reference-based metrics. Finally, we study the opposing effect of the Prompt 3 representation in ViGGO and EMPATHIC, which was selected as the best among the enriched representations. For all the comparisons, a set of samples was selected from the outputs of the fine-tuned models. The metric comparisons include generated sentences that show the largest discrepancies between the metrics analysed. Meanwhile, the comparison between the Baseline and Prompt 3 focuses on those MRs that, on average, show the largest differences in reference-based metric scores between the two representations, i.e., the MRs with the highest average absolute difference in MR scores across the three reference-based metrics. From this qualitative inspection, we confirm that metrics focusing on lexical aspects do not reliably reflect generation quality. In contrast, the human-trained BLEURT metric appears to capture omissions and semantic nuances that go undetected by LaBSE. Finally, the comparison of representations sheds light on the different impact of the enriched representations on ViGGO and EMPATHIC generations. Specifically, adding a demonstrator to some long ViGGO inputs can induce a forgetting effect in the models, while in EMPATHIC, the demonstrator help fine-tuned models to learn sentence structures principally. Analysis of metric scores Tables 11 and 12 present representative generation examples that show the different behaviour of the metrics. Each table contains ViGGO and EMPATHIC examples. These examples consist of the MR, the references (with the number of them) linked to this MR, the generated sentence, the dataset and the generation scores for all the metrics, except DAC, whose individual scores are irrelevant. In the tables, the headers of the input elements are orange, while those of the output are purple. For the compared metrics, we highlighted the lowest values in red and the highest ones in green. BLEU is not adequate for evaluation of output qualityTable 11 shows three examples (one for ViGGO and two for EMPATHIC) of good generations with very low BLEU (in red in the table) and reasonably high BLEURT and LaBSE scores (in green). All three generations accurately reflect their corresponding MR. The ViGGO exampleâs generation, "Crysis is an action-adventure shooter released in 2007. It has received good reviews", contains two omissions (esrb andhas_multiplayer), but it is a goodinformsentence that conveys nearly all the intended semantic content. Regarding the EMPATHIC generations, "Do you like to cook fish?" and "Hello!" are perfectly acceptable outputs for an open question about cooking fish and a greeting, respectively. Thus, the poor BLEU scores in these examples confirm 21 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT Table 11: Examples to compare BLEU with BLEURT and LaBSE. Slot Accuracy (Slot acc) is also included. Each example contains, from top to bottom: MR, references, a generation, as well as the dataset and scores for all the metrics in the last row. Relevant elements are highlighted in bold and different colours as follows: orange for the headers related to the inputs, purple for those related to the outputs, blue for the datasets, red for BLEU scores, and green for BLEURT and LaBSE scores. MRinform ( name = Crysis ; release_year = 2007 ; esrb = M (for Mature) ; rating = good ; genres = action-adventure, shooter ; has_multiplayer = yes ) References3 references: - A solid M-rated action-adventure shooter, Crysis, came out in 2007. The game offers multiplayer. - An action-adventure shooter game for mature audiences is the 2007 game, Crysis. It was a fun game offering also a multiplayer mode. - Crysis is an action-adventure shooter that was released in 2007, rated M (for Mature). It is considered good quality, and has multiplayer mode. GenerationCrysis is an action-adventure shooter released in 2007. It has received good reviews. DatasetViGOOBLEU0.06BLEURT0.66LaBSE0.81Slot acc0.80 MRGen_Open_question ( action = cook ; food = fish ) References1 reference: Whatâs your favourite fish to cook? GenerationDo you like to cook fish? DatasetEMPATHICBLEU0.07BLEURT0.71LaBSE0.83Slot acc1.00 MRGen_Hello ( ) References25 references:- Good morning- Hello, how are you?- Hey, whatâs up? good morning.- Hello.- (21 references more) GenerationHello! DatasetEMPATHICBLEU0.05BLEURT0.64LaBSE0.59Slot acc- Table 12: Examples to compare BLEURT with LaBSE. BLEU and Slot Accuracy (Slot acc) are also included. Each example contains, from top to bottom: MR, references, a generation, as well as the dataset and scores for all the metrics in the last row. Relevant elements are highlighted in bold and different colours as follows: orange for the headers related to the inputs, purple for those related to the outputs, blue for the datasets, red for BLEURT scores, and green for LaBSE scores. MRinform ( name = Tony Hawkâs Pro Skater 3 ; release_year = 2001 ; esrb = T (for Teen) ; genres = sport ; player_perspective = third person ) References3 references: - Tony Hawkâs Pro Skater 3 was released in 2001. It is a third person sports game that is rated T (for Teen). - Tony Hawkâs Pro Skater 3 was a third person sport game released in 2001. It was rated T (for Teen). - Another third person perspective sport game is Tony Hawkâs Pro Skater 3. It came out in 2001 and is rated T (for Teen). GenerationThe 2001 sport game, Tony Hawkâs Pro Skater 3, was a third person game. It has a t (for teen) rating and is rated t (for physical). DatasetViGGOBLEU0.34BLEURT0.61LaBSE0.92Slot acc1.00 MRGen_Agreement ( ) References91 references:- Absolutely- Of course.- Yeah- Right- I totally understand.- (86 references more) GenerationSure. DatasetEMPATHICBLEU0.02BLEURT0.09LaBSE0.56Slot acc- that it may not be suitable for assessing output quality. In addition, in the first EMPATHIC example, the generated sentence contains the n-gram "to cook fish?", which uses the same words as the reference n-gram "fish to cook?". However, this does not lead to a high score, presumably due to the different word order. Therefore, BLEU appears to be highly sensitive to lexical order. BLEURT detects omissions and semantic nuances Table 12 shows representative examples of the comparison performed between BLEURT and LaBSE. The two examples (one for each dataset) present higher scores for LaBSE. However, the human-trained BLEURT seems to capture more subtle aspects than LaBSE. For instance, the ViGGO example presents a hallucination in its generation, where the slotesrb = T (for Teen)is generated twice, and one of them contains incorrect information: "It has a t (for teen) rating and is rated t (for physical)". This error goes unnoticed by LaBSE, which gives an almost perfect score of 0.92, whereas BLEURT produces a more adequate score of 0.61. In the EMPATHIC example, the generation "Sure." can be considered good for expressing agreement and obtains 0.09 from BLEURT and 0.56 from LaBSE. We attribute this difference to the fact that while all references are appropriate to convey agreement, not all express it with the same degree of emphasis. These semantic nuances in the meanings of "Yeah", "I totally understand", or the generated "Sure" appear to influence BLEURT scores, which may provide evidence of BLEURTâs sensitivity to such distinctions. 22 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT Table 13: Examples of the generations obtained with the Baseline and Prompt 3 representations. Each example contains from top to bottom: MR, references, the task demonstrator linked to the MR in the Prompt 3 representation, generations obtained with the Baseline representation, generations obtained with the Prompt 3 representation, and dataset and scores for all the metrics for both representations (Baseline/Prompt 3) in the last row. The highest score for each metric is highlighted in bold. MRinform ( name = HSS ; genres = adventure, hack-and-slash, indie ; platforms = PlayStation, Xbox, PC ; available_on_steam = yes ; has_linux_release = no ; has_mac_release = no ) References3 references: - HSS is an indie adventure, hack-and-slash game for PlayStation, Xbox, and PC. It is available for Steam, however, it is not out for Linux or even Mac. - HSS is an indie hack-and-slash adventure game available on Steam. You can play it on PlayStation, Xbox, and PC, but not on Linux or Mac. - HSS is an indie, adventure, hack-and-slash game thatâs available on PlayStation, Xbox and PC. You can also get it from Steam, but it does not have Linux or Mac releases. Prompt 3 task demonstrator inform ( name = The Sims ; genres = simulation, strategy ; platforms = PlayStation, Xbox, Nintendo, PC ; available_on_steam = no ; has_linux_release = no ; has_mac_release = yes ) + The Sims is a simulation-strategy game that is available for PlayStation, Xbox, Nintendo, and PC. It can operate on Mac, but it is not released for Linux. It is not available on Steam. Baseline Generations 15 generations: - HSS is an indie adventure hack-and-slash game. Itâs available on the following platforms: PlayStation, Xbox, and PC. Steam is also the home of the game, though it isnât available for Linux or Mac. - If youâre into indie adventure and hack-and-slash games, try HSS. Itâs an indie game released for PlayStation, Xbox, PC, and Mac. You can get it on Steam now. - If you like indie hack-and-slash adventures on the PC, try HSS. Itâs available on Steam, and you can play it on all platforms, such as the PlayStation, Xbox, and PC. There was no Linux or Mac release.- (12 generations more) Prompt 3 Generations 15 generations: - HSS is an indie adventure-shooter with puzzle solving elements. - HSS is an indie adventure-hack and slash with a Mac twist. - HSS is an indie adventure-hack-and-slash on the PC that has been released on Steam.- (12 generations more) DatasetViGGOBLEU0.31 / 0.15BLEURT0.70 / 0.56LaBSE0.89 / 0.68Slot acc0.82 / 0.33 MRGen_Open_topic ( hobbies = travel ) References1 reference: Letâs talk about travel. Prompt 3 task demonstrator Gen_Open_topic ( hobbies = cinema ) + Letâs talk about cinema. Baseline Generations 5 generations: - We are going to travel, arenât we?- We will travel together.- Weâre going to travel together! - Which of these trips did you like the most?- Then weâl travel together. Prompt 3 Generations 3 generations: - Letâs talk about travel.- We can have a trip to the islands.- Letâs talk about travelling DatasetEMPATHICBLEU0.03 / 0.45BLEURT0.36 / 0.69LaBSE0.54 / 0.78Slot acc0.80 / 0.67 Analysis of the influence of Prompt representation for different datasets Table 13 shows two representative examples of generations for Baseline and Prompt 3 inputs (one for ViGGO and one for EMPATHIC). In these examples, the input elements (with headers in orange in the table) are the selected Input MR (labelled MR), the references (including their number), and the task demonstrator used in the Prompt 3 representation. For the output (headers in purple), the table shows both the Baseline and Prompt 3 generations (including their number) and the MR scores separately (Baseline/Prompt 3). The highest MR scores for each metric are highlighted in bold. Finally, the dataset is specified (in blue). In this comparison, although metric scores were used to select the examples, we focus on comparing the generated sentences to understand why enriched representations have an opposite effect across the two datasets. Prompt representations produce omissions in ViGGO ViGGO generations with Prompt 3 tend to be shorter and contain omissions when the inputs contain high semantic context, i.e, a high number of attributes. The ViGGO example of Table 13 illustrates it perfectly: all the Prompt 3 generations included in the table contain multiple omissions and are much shorter than the references, whereas only the second Baseline generation omitshas_linux_releaseand hallucinates (incorrect generation ofhas_mac_release). As a consequence, we may assume that concatenating the MR and the task demonstrator in the Prompt 3 representation results in an excessively long input, which makes the model omit some information. Model learns from the demonstrator in EMPATHIC The inspection of the EMPATHIC MRs shows Prompt 3 generations where the model seems to learn from the task demonstrator, especially when it fails to correctly generate the MR with the Baseline representation. In the EMPATHIC example of Table 13, the generations "Letâs talk about travel." and "Letâs talk about travelling" clearly adopt the structure of Letâs talk about cinema from the task demonstrator. Although the other Prompt 3 generation "We can have a trip to the islands." is incorrect forGen_Open_topic ( 23 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT hobbies = travel ), the Prompt 3 generations are more accurate overall than the Baseline generations, all of which are incorrect. Therefore, we can assume that the proposed representations provide useful guidance in cases where fine-tuning alone is insufficient with the Baseline MRs. In conclusion, the enriched representation proves effective in smaller datasets where fine-tuning on the original MRs is insufficient to capture the generation patterns. However, in datasets with inputs containing a high density of semantic content, such as ViGGO, the resulting enriched MRs tend to become excessively long, leading the model to omit relevant information. 7 Conclusions In this work, we have carried out a comprehensive analysis of the impact of different meaning representations on the performance of fine-tuned NLG models across multiple dialogue datasets. In particular, we propose prompt representations that integrate a task demonstrator in the meaning representation structure. To validate our approach, we assess them across four different datasets using five distinct metrics, offering a detailed analysis of the interactions between representation, dataset and metric features. Our study shows that the proposed representations of meaning are especially effective for small datasets for complex tasks and high-output variability. In this context, the demonstrator enriches the input for MRs where the reduced fine-tuning is insufficient for an accurate generation. Conversely, in other datasets, the models perform better when they are fine-tuned with the original MRs. These datasets usually present long inputs with high semantic content, and the extra information of the task demonstrator might produce a forgetting effect in the models that leads to short generations with omission. Another positive effect of the enriched representations appears in zero-shot settings, i.e., before training the models. We have observed that even in those datasets where these enriched inputs have not been effective after fine-tuning, the models without training perform better with prompt representations than the baseline one. Finally, the comparison between prompt representations proves that more specific demonstrators in the input are more beneficial for the models, while the more generic ones may act as distractors, making the learning process for the models more difficult. The thorough analysis of the metrics has also displayed relevant findings. First, metrics that evaluate lexical aspects based on n-gram overlapping do not seem adequate to assess output quality. In this regard, semantic metrics are more accurate. In a novel comparison of semantic metrics, we have observed that those metrics trained with human ratings detect omissions and semantic nuances not appreciable by others based on sentence embeddings. Finally, the reference-less metrics have confirmed the effect of the representations and datasets on the modelâs behaviour and revealed the model coherence with the input in terms of communicative intention and semantics. This study also highlights the suitability of GPT-2 models for generating language in the context of dialogue. Particularly, they present good adaptation capabilities. The excellent performance observed after just one training epoch, regardless of the representation or dataset, confirms this feature. Future work should extend this study to the new era of LLMs to explore how prompt-enhanced MRs interact with the few-shot and zero-shot learning capabilities of these new models. Another promising direction is to explore whether prompt-based MRs enhance robustness and adaptability across different languages, contributing to the development of more inclusive and generalized dialogue systems. Acknowledgments This work has been partially supported by the CRYSTAL HORIZON-MSCA-SE grant 101182965, Spanish MCIU by the BRAINS project grant PID2024-155948OB-C51 and by the Basque Government under grant PRE 2020 1 0274. References Shalom Lappin. Assessing the strengths and weaknesses of large language models. Journal of Logic, Language and Information, 33(1):9â20, 2024. 24 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT Shengpeng Ji, Yifu Chen, Minghui Fang, Jialong Zuo, Jingyu Lu, Hanting Wang, Ziyue Jiang, Long Zhou, Shujie Liu, Xize Cheng, et al. WavChat: A survey of spoken dialogue models. arXiv preprint arXiv:2411.13577, 2024. Asier LĂłpez Zorrilla and M. InĂ©s Torres. A multilingual neural coaching model with enhanced long-term dialogue structure. ACM Trans. Interact. Intell. Syst., 12(2), July 2022. ISSN 2160-6455. doi:10.1145/3487066. URL https://doi.org/10.1145/3487066. Zihan Wang, Jiayu Xiao, Mengxiang Li, Zhongjiang He, Yongxiang Li, Chao Wang, and Shuangyong Song. To- wards robustness and diversity: Continual learning in dialog generation with text-mixup and batch nuclear-norm maximization. In 2024 International Joint Conference on Neural Networks (IJCNN), pages 1â8. IEEE, 2024. Lucie Galland, Catherine Pelachaud, and Florian Pecune. Generating unexpected yet relevant user dialog acts. In Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 192â203, 2024. David Griol and Zoraida Callejas. Combining statistical dialog management and intent recognition for enhanced response selection. Logic Journal of the IGPL, page jzae045, 2024. Alain Vazquez, Angela Maria Ramirez, Neha Pullabhotla, Nan Qiang, Ranran Haoran Zhang, Marilyn Walker, and Maria InĂ©s Torres. Knowledge-grounded dialogue act transfer using prompt-based learning for controllable open- domain NLG. In Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 78â91, 2024. Xiangkun Hu, Junqi Dai, Hang Yan, Yi Zhang, Qipeng Guo, Xipeng Qiu, and Zheng Zhang. Dialogue meaning representation for task-oriented dialogue systems. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 223â237, 2022. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311â318, 2002. Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881â7892, 2020. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language-agnostic BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878â891, 2022. Yangming Li, Kaisheng Yao, Libo Qin, Wanxiang Che, Xiaolong Li, and Ting Liu. Slot-consistent NLG for task- oriented dialogue systems with iterative rectification network. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 97â106, Online, July 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.acl-main.10. URL https://aclanthology.org/2020.acl-main.10. Angela Ramirez, Kartik Agarwal, Juraj Juraska, Utkarsh Garg, and Marilyn Walker. Controllable generation of dialogue acts for dialogue systems via few-shot response generation and ranking. In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 355â369, 2023. Jekaterina Novikova, Ond Ë rej DuĆĄek, and Verena Rieser. The E2E dataset: New challenges for end-to-end generation. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 201â206, 2017. Juraj Juraska, Kevin Bowden, and Marilyn Walker. ViGGO: A video game corpus for data-to-text generation in open-domain conversation. In Proceedings of the 12th International Conference on Natural Language Generation, pages 164â172, 2019. PaweĆ Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. MultiWOZ-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016â5026, 2018. Alain VĂĄzquez, Asier LĂłpez Zorrilla, Javier Mikel Olaso, and MarĂa InĂ©s Torres. Dialogue management and language generation for a robust conversational virtual coach: Validation and user study. Sensors, 23(3):1423, 2023. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 25 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv e-prints, pages arXivâ2407, 2024. Sijie Liu, Yiquan Fang, Hua Cheng, Yiming Pan, Yufei Liu, and Caiting Gao. Large language models guided generative prompt for dialogue generation. In 2023 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC), pages 10â17. IEEE, 2023. Atheer Algherairy and Moataz Ahmed. A review of dialogue systems: current trends and future directions. Neural Computing and Applications, 36(12):6325â6351, 2024. Alain VĂĄzquez and Maria Ines Torres. Prompt-based language generation for complex conversational coaching tasks across languages. In Proceedings of the 26th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 601â608, Avignon, France, August 2025. Association for Computational Linguistics. URL https://aclanthology.org/2025.sigdial-1.48/. Lena Reed, Cecilia Li, Angela Ramirez, Liren Wu, and Marilyn Walker. Jurassic is (almost) all you need: Few-shot meaning-to-text generation for open-domain dialogue. In Conversational AI for Natural Human-Centric Interaction: 12th International Workshop on Spoken Dialogue System Technology, IWSDS 2021, Singapore, pages 99â119. Springer, 2022. A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1â67, 2020. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171â4186, 2019. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and William B Dolan. DialoGPT: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270â278, 2020. Baolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li, Jinchao Li, Michael Zeng, and Jianfeng Gao. Few-shot natural language generation for task-oriented dialog. In Trevor Cohn, Yulan He, and Yang Liu, editors, Findings of the Association for Computational Linguistics: EMNLP 2020, pages 172â182, Online, November 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.findings-emnlp.17. URLhttps://aclanthology.org/2020. findings-emnlp.17. Seyed Mahed Mousavi, Gabriel Roccabruna, Simone Alghisi, Massimo Rizzoli, Mirco Ravanelli, and Giuseppe Riccardi. Are LLMs robust for spoken dialogues? arXiv preprint arXiv:2401.02297, 2024. Asier LĂłpez Zorrilla, MarĂa InĂ©s Torres, and Heriberto CuayĂĄhuitl.Audio embedding-aware dialogue pol- icy learning.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:525â538, 2023. doi:10.1109/TASLP.2022.3225658. Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D Dhole, et al. The GEM benchmark: Natural language generation, its evaluation and metrics. In 1st Workshop on Natural Language Generation, Evaluation, and Metrics 2021, pages 96â120. Association for Computational Linguistics, 2021. Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. ToTTo: A controlled table-to-text generation dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1173â1186, 2020. Amit Moryossef, Yoav Goldberg, and Ido Dagan. Step-by-step: Separating planning from realization in neural data- to-text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2267â2277, 2019. Qingyang Wu, James Gung, Raphael Shu, and Yi Zhang. DiactTOD: Learning generalizable latent dialogue acts for controllable task-oriented dialogue systems. In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 255â267, 2023. 26 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT Weizhi Wang, Zhirui Zhang, Junliang Guo, Yinpei Dai, Boxing Chen, and Weihua Luo. Task-oriented dialogue system as natural language generation. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, pages 2698â2703, 2022. Yuheng Du, Shereen Oraby, Vittorio Perera, Minmin Shen, Anjali Narayan-Chen, Tagyoung Chung, Anu Venkatesh, and Dilek Hakkani-Tur. Schema-guided natural language generation. In 13th International Conference on Natural Language Generation, INLG 2020, pages 283â295. Association for Computational Linguistics (ACL), 2020. Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8689â8696, 2020. Fei Mi, Liangwei Chen, Mengjie Zhao, Minlie Huang, and Boi Faltings. Continual learning for natural language generation in task-oriented dialog systems. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3461â3474, 2020. Jianhong Wang, Y Zhang, Tae-Kyun Kim, and Yunjie Gu. Modelling hierarchical structure between dialogue policy and natural language generator with option framework for task-oriented dialogue system. In Ninth International Conference on Learning Representation (ICLR). The International Conference on Learning Representations (ICLR), 2021. Alain VĂĄzquez. The Role of Meaning Representations in Natural Language Generation for Dialogue. PhD thesis, University of the Basque Country, 2026. Alain VĂĄzquez, Asier LĂłpez Zorrilla, and M. InĂ©s Torres. How should we represent dialog acts to leverage pretrained natural language generators? In Presented at 13th International Workshop on Spoken Dialogue Systems Technology, 2023. Darya Hryhoryeva. Data-to-text Generation with Detailed Inputs. PhD thesis, Univerzita Karlova, Matematicko- fyzikĂĄlnĂ fakulta, 2025. Bohao Yang, Chen Tang, and Chenghua Lin. Improving medical dialogue generation with abstract meaning representa- tions. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11826â11830. IEEE, 2024. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. Abstract meaning representation for sembanking. In Proceedings of the 7th linguistic annotation workshop and interoperability with discourse, pages 178â186, 2013. Zhijing Jin, Yuen Chen, Fernando Gonzalez Adauto, Jiarui Liu, Jiayi Zhang, Julian Michael, Bernhard Schölkopf, and Mona Diab. Analyzing the role of semantic representations in the era of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3781â3798, 2024. Jorge Baptista, SĂłnia Reis, JoĂŁo Dias, and Pedro A Santos. Lexicalized meaning representation (LMR). In Workshop on Designing Meaning Representation, pages 101â111. ELRA; ICCL, 2024. Nasim Tohidi and Chitra Dadkhah. A short review of abstract meaning representation applications. Modeling and Simulation in Electrical and Electronics Engineering, 2(3):1â9, 2022. Claire Bonial, Julie Foresta, Nicholas C Fung, Cory Hayes, Philip Osteen, Jacob Arkin, Benned Hedegaard, and Thomas Howard. Abstract meaning representation for grounded human-robot communication. In Proceedings of the Fourth International Workshop on Designing Meaning Representations, pages 34â44, 2023. Richard Brutti, Lucia Donatelli, Kenneth Lai, and James Pustejovsky. Abstract meaning representation for gesture. In Proceedings of the thirteenth language resources and evaluation conference, 2022. Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. PPT: Pre-trained prompt tuning for few-shot learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8410â8423, 2022. Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 2021. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877â1901, 2020. 27 Impact of enriched meaning representations for language generation in dialogue tasks.A PREPRINT Tsung-Hsien Wen, Milica GaĆĄi Ì c, Nikola MrkĆĄi Ì c, Pei-Hao Su, David Vandyke, and Steve Young. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In LluĂs MĂ rquez, Chris Callison-Burch, and Jian Su, editors, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711â1721, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi:10.18653/v1/D15-1199. URL https://aclanthology.org/D15-1199. Seungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. OpenDialKG: Explainable conversational reasoning with attention-based walks over knowledge graphs. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 845â854, 2019. MarĂa InĂ©s Torres, Javier Mikel Olaso, CĂ©sar Montenegro, Roberto Santana, Alain VĂĄzquez, Raquel Justo, JosĂ© Antonio Lozano, Stephan Schlögl, GĂ©rard Chollet, Nazim Dugan, et al. The EMPATHIC project: mid-term achievements. In Proceedings of the 12th ACM International Conference on Pervasive Technologies Related to Assistive Environments, pages 629â638, 2019. CĂ©sar Montenegro, Asier LĂłpez Zorrilla, Javier Mikel Olaso, Roberto Santana, Raquel Justo, Jose A Lozano, and MarĂa InĂ©s Torres. A dialogue-act taxonomy for a virtual coach designed to improve the life of elderly. Multimodal Technologies and Interaction, 3(3):52, 2019. Javier M Olaso, Alain VĂĄzquez, Leila Ben Letaifa, Mikel De Velasco, Aymen Mtibaa, Mohamed Amine Hmani, Dijana Petrovska-DelacrĂ©taz, GĂ©rard Chollet, CĂ©sar Montenegro, Asier LĂłpez-Zorrilla, et al. The EMPATHIC virtual coach: A demo. In Proceedings of the 2021 International Conference on Multimodal Interaction, pages 848â851, 2021. Ananya B Sai, Akash Kumar Mohankumar, and Mitesh M Khapra. A survey of evaluation metrics used for NLG systems. ACM Computing Surveys (CSUR), 55(2):1â39, 2022. PatrĂcia SchmidtovĂĄ, Saad Mahamood, Simone Balloccu, Ond Ë rej DuĆĄek, Albert Gatt, Dimitra Gkatzia, David M Howcroft, Ond Ë rej PlĂĄtek, and Adarsa Sivaprasad. Automatic metrics in natural language generation: A survey of current evaluation practices. In Proceedings of the 17th International Natural Language Generation Conference, pages 557â583, 2024. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. Evaluation of text generation: A survey. arXiv preprint arXiv:2006.14799, 2020. Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74â81, 2004. Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65â72, 2005. George Doddington. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the second international conference on Human Language Technology Research, pages 138â145, 2002. Emilie Colin, Claire Gardent, Yassine Mârabet, Shashi Narayan, and Laura Perez. The WebNLG challenge: Generating text from DBPedia data. In 9th International Natural Language Generation conference, pages 163â167. Association for Computational Linguistics, 2016. Eneko Ruiz, MarĂa InĂ©s Torres, and Arantza del Pozo.Question answering models for humanâmachine interaction in the manufacturing industry.Computers in Industry, 151:103988, 2023.ISSN 0166- 3615. doi:https://doi.org/10.1016/j.compind.2023.103988. URLhttps://w.sciencedirect.com/science/ article/pii/S0166361523001380. Yufan Wang, Bowei Zou, Rui Fan, Aiti Aw, and Tingting He. DSPM-NLG: A dual supervised pre-trained model for few-shot natural language generation in task-oriented dialogue system. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12389â12402, 2023. Sarah Leach. Behavioural coaching: The GROW model. In The Coachesâ Handbook, pages 176â186. Routledge, 2020. Boxing Chen and Colin Cherry. A systematic comparison of smoothing techniques for sentence-level BLEU. In Proceedings of the ninth workshop on statistical machine translation, pages 362â367, 2014. V Sanh. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715â1725, 2016. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 28