Paper deep dive
ESAinsTOD: A Unified End-to-End Schema-Aware Instruction-Tuning Framework for Task-Oriented Dialog Modeling
Dechuan Teng, Chunlin Lu, Libo Qin, Wanxiang Che
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/13/2026, 1:04:31 AM
Summary
ESAinsTOD is a unified end-to-end schema-aware instruction-tuning framework designed to improve task-oriented dialog (TOD) modeling. By leveraging full-parameter fine-tuning of LLMs and introducing instruction and schema alignment mechanisms, the framework enables models to generalize across heterogeneous TOD datasets, improve data efficiency in low-resource settings, and mitigate cascading errors common in modular pipeline architectures.
Entities (6)
Relation Signals (3)
ESAinsTOD → evaluatedon → MultiWOZ
confidence 99% · ESAinsTOD outperforms state-of-the-art models by a significant margin on end-to-end task-oriented dialog modeling benchmarks: CamRest676, In-Car and MultiWOZ
ESAinsTOD → improves → Task-Oriented Dialog Modeling
confidence 98% · ESAinsTOD, a unified End-to-end Schema-Aware Instruction-tuning framework for general Task-Oriented Dialog modeling.
ESAinsTOD → utilizes → LLaMA-2
confidence 95% · In this work, we use Llama 2 7B as the backbone model
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Existing end-to-end modeling methods for modular task-oriented dialog systems are typically tailored to specific datasets, making it challenging to adapt to new dialog scenarios. In this work, we propose ESAinsTOD, a unified End-to-end Schema-Aware Instruction-tuning framework for general Task-Oriented Dialog modeling. This framework introduces a structured methodology to go beyond simply fine-tuning Large Language Models (LLMs), enabling flexible adaptation to various dialogue task flows and schemas. Specifically, we leverage full-parameter fine-tuning of LLMs and introduce two alignment mechanisms to make the resulting system both instruction-aware and schema-aware: (i) instruction alignment, which ensures that the system faithfully follows task instructions to complete various task flows from heterogeneous TOD datasets; and (ii) schema alignment, which encourages the system to make predictions adhering to the specified schema. In addition, we employ session-level end-to-end modeling, which allows the system to access the results of previously executed task flows within the dialogue history, to bridge the gap between the instruction-tuning paradigm and the real-world application of TOD systems. Empirical results show that while a fine-tuned LLM serves as a strong baseline, our structured approach provides significant additional benefits. In particular, our findings indicate that: (i) ESAinsTOD outperforms state-of-the-art models by a significant margin on end-to-end task-oriented dialog modeling benchmarks: CamRest676, In-Car and MultiWOZ; (ii) more importantly, it exhibits superior generalization capabilities across various low-resource settings, with the proposed alignment mechanisms significantly enhancing zero-shot performance; and (iii) our instruction-tuning paradigm substantially improves the model's robustness against data noise and cascading errors.
Tags
Links
- Source: https://arxiv.org/abs/2603.09691v1
- Canonical: https://arxiv.org/abs/2603.09691v1
Trouble viewing inline? Open PDF directly →
Full Text
154,677 characters extracted from source content.
Expand or collapse full text
Springer Nature 2021 L A T E X template ESAinsTOD: A Unified End-to-End Schema-Aware Instruction-Tuning Framework for Task-Oriented Dialog Modeling Dechuan Teng 1 , Chunlin Lu 2 , Libo Qin 2* and Wanxiang Che 1* 1 Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology, Harbin, China. 2 School of Computer Science and Engineering, Central South University, Changsha, China. *Corresponding author(s). E-mail(s): lbqin@csu.edu.cn; car@ir.hit.edu.cn; Contributing authors: dcteng@ir.hit.edu.cn; 234711290@csu.edu.cn; Abstract Existing end-to-end modeling methods for modular task-oriented dialog systems are typically tailored to specific datasets, making it challenging to adapt to new dialog scenarios. In this work, we propose ESAinsTOD, a unified E nd-to-end S chema-Aware Instruction-tuning framework for general T ask-O riented D ialog modeling. This framework introduces a structured methodology to go beyond simply fine-tuning Large Language Models (LLMs), enabling flexible adaptation to various dialogue task flows and schemas. Specifically, we leverage full-parameter fine-tuning of LLMs and introduce two alignment mechanisms to make the resulting system both instruction-aware and schema-aware: (i ) instruction align- ment, which ensures that the system faithfully follows task instructions to complete various task flows from heterogeneous TOD datasets; and (i ) schema alignment, which encourages the system to make predictions adhering to the specified schema. In addition, we employ session-level end-to-end modeling, which allows the system to access the results of previously executed task flows within the dialogue history, to bridge the 1 arXiv:2603.09691v1 [cs.CL] 10 Mar 2026 Springer Nature 2021 L A T E X template 2A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling gap between the instruction-tuning paradigm and the real-world appli- cation of TOD systems. Empirical results show that while a fine-tuned LLM serves as a strong baseline, our structured approach provides sig- nificant additional benefits. In particular, our findings indicate that: (i ) ESAinsTOD outperforms state-of-the-art models by a significant margin on end-to-end task-oriented dialog modeling benchmarks: CamRest676, In-Car and MultiWOZ; (i ) more importantly, it exhibits superior generalization capabilities across various low-resource settings, with the proposed alignment mechanisms significantly enhancing zero-shot performance; and (i ) our instruction-tuning paradigm substantially improves the model’s robustness against data noise and cascading errors. Keywords: Task-oriented dialog system, Instruction tuning, Multi-task learning, Dialogue understanding, Dialogue generation 1 Introduction Task-oriented dialogue (TOD) systems are designed to assist users in accom- plishing a wide range of tasks, such as travel planning and restaurant reservation, through natural language communications. These systems are fun- damental in developing various virtual assistants and have recently garnered extensive attention (Wen et al, 2017; Ham et al, 2020; Yang et al, 2021; Su et al, 2022; He et al, 2022b). A typical TOD system adopts a modularized pipeline architecture (Young et al, 2013; Gao et al, 2019), where the natural language understanding (NLU) module is responsible for parsing user utterances into semantic frames consisting of intents and slot-value pairs, the dialog manage- ment (DM) module for maintaining the dialog state (i.e., user goal), managing database queries, and then deciding next system actions, and the natural lan- guage generation (NLG) module for converting the system actions into natural language responses. With the rapid development of pre-trained language mod- els (PLMs), many methods have been proposed to improve the performance of each module (Qin et al, 2019; Kim et al, 2020; Heck et al, 2020; Chen et al, 2022). However, the pipeline architecture often suffers from error propagation due to the independent training of each module, where the accumulated errors between modules will lead to the overall performance degradation of the dia- log system. Therefore, much work explores modeling the modularized TOD system in an end-to-end (E2E) trainable manner to alleviate error propaga- tion (Hosseini-Asl et al, 2020; Yang et al, 2021; Peng et al, 2021). For example, UBAR (Yang et al, 2021) incorporates dialogue state tracking (DST), policy planning (POL), and natural language generation in a single auto-regressive language model. Subsequently, dialog pre-training, where vanilla PLMs are pre-trained on large-scale dialog corpora with well-designed dialog objectives, is also proposed to build pre-trained conversation models (PCMs) that enrich Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling3 conversational knowledge and can be further fine-tuned to improve down- stream dialog tasks (Mehri et al, 2020; Wu et al, 2020; Zhang et al, 2020d; Bao et al, 2020; Su et al, 2022; He et al, 2022b). While these methods have advanced the field, the advent of Large Language Models (LLMs) presents new opportunities and challenges. A straightforward approach is to simply fine-tune a powerful LLM on existing TOD datasets. However, this often fails to address several persistent drawbacks: • Weak adaptability. Existing models are tailored to specific datasets covering certain domains and dialog task flows. Moreover, the core idea of existing PCMs is to extract and summarize general dialog knowledge from various TOD datasets during the dialog pre-training stage. This paradigm does not explicitly account for the tight coupling between data annotations and their corresponding schema (e.g., database structure and available APIs). Con- sequently, adapting these models to new dialogue scenarios with different underlying schemas remains a significant challenge. Fig. 1a illustrates the general process of applying PCMs to the TOD field. • Insufficient annotation exploitation. To exploit various public heterogeneous TOD datasets that have incomplete annotations (e.g., data only annotated with DST or POL), existing methods decouple them into different sub-task datasets on which PLMs are pre-trained with multi-task learning objec- tives (Su et al, 2022; Yang et al, 2023), without considering the correlation between tasks. Therefore, these resulting PCMs exhibit limited performance on end-to-end TOD modeling. In this work, to address the above two issues, we propose a simple yet effective instruction-tuning framework, ESAinsTOD (E nd-to-end S chema- Aware Instruction-tuning for general T ask-O riented D ialog modeling). Rather than relying solely on the inherent capabilities of an LLM backbone, our framework introduces a structured approach to unify heterogeneous data and tasks, enabling the model to generalize across diverse dialogue scenarios, as shown in Fig. 1b. Specifically, we introduce instruction alignment to link different TOD tasks with corresponding annotations, allowing the model to faithfully follow task instructions to complete given dialog tasks, thus unify- ing various TOD datasets with heterogeneous annotations. Since task outputs are related to the corresponding schema, we argue that it is non-trivial to make the model aware of the schema information. Therefore, we introduce schema alignment to force model predictions to be consistent with the specified schema, providing an opportunity to generalize to any dialog sce- nario with different schemas. Based on the proposed framework, we construct a multi-turn end-to-end instruction-tuning TOD corpus from 11 public TOD datasets, which cover a number of domains and scenarios and are partially or fully annotated with intents, dialogue states, database results, and dialogue acts. The detailed statistics of the corpus are shown in Table 1. In total, our corpus encompasses over 471k dialogue turns across 336 domain schemas. Springer Nature 2021 L A T E X template 4A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling Our experiments are designed to carefully distinguish the performance gains attributable to the powerful LLM backbone from the benefits conferred by our framework itself. The results demonstrate that while a fine-tuned LLM establishes a strong baseline, our proposed framework (1) significantly enhances generalization capabilities across diverse dialogue scenarios, (2) improves data efficiency in low-resource settings, and (3) effectively mitigates error propagation that is a persistent challenge in both pipeline and end- to-end systems. This simple framework can fully consolidate the annotations from various TOD datasets into a single model, efficiently transferring to new dialogue scenarios through instruction and schema alignment. Furthermore, schema-aware instruction-tuning establishes clearer boundaries between end- to-end TOD modeling in various scenarios, thereby reducing the frequency of cascading errors by mitigating the interference among training data. In summary, our contributions are three-fold: • To the best of our knowledge, we are the first to explore a unified instruction-tuning framework for end-to-end task-oriented dialog modeling. The framework systematically structures heterogeneous dialog data to effec- tively unlock the generalization capabilities of large language models for complex, multi-domain conversations. • We construct a multi-turn end-to-end instruction-tuning corpus and obtain an efficient task-oriented dialogue model by fine-tuning an open-source LLM on this corpus. The open corpus covers various dialog scenarios, with each dialog session annotated with specific schema definitions, task instructions, and their corresponding task outputs. This construction paradigm facilitates the establishment of a benchmark for developing and evaluating general- izable TOD systems. The trained model, datasets, and codes are publicly available at https://github.com/AaronTengDeChuan/ESAinsTOD. • Results on multiple datasets demonstrate the effectiveness of our proposed task-oriented instruction-tuning framework. We also conduct extensive experiments to verify and analyze the strong generalization, data efficiency, and error mitigation benefits of ESAinsTOD. We believe that our work will provide a new perspective for building robust and adaptable TOD systems in the era of LLMs. 2 Preliminaries In this section, we first formally define the workflow of the traditional modularized TOD system and then present how to model it in an end-to- end manner. Finally, we introduce the backbone models for our framework, including Llama (Touvron et al, 2023) and Qwen2.5 (Yang et al, 2024), open- source decoder-only large language models, which are naturally suitable for end-to-end TOD modeling. Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling5 Pre-trained Language Model Dialog Corpora Pre-trained on Pre-trained Conversation Model Corpus A — Annotations DST DB POL NLG — — Annotations DST DB POL NLG — Corpus A — Annotations DST DB POL NLG — Corpus B — Annotations NLU DST DB POL NLG — — Annotations NLU DST DB POL NLG — Corpus B — Annotations NLU DST DB POL NLG — Fine-tuned on PCM APCM B ...... — — — — — Dialog Actions Dialog State Dialog Response Intent Dialog State Dialog Actions Dialog Response — — — — — (a) Conventional PCMs Large Language Model Corpus A — — Task Instructions Annotations DST DB POL NLG — — — — Task Instructions Annotations DST DB POL NLG — — Corpus B — — Task Instructions Annotations NLU DST DB POL NLG — — — — Task Instructions Annotations NLU DST DB POL NLG — — ...... Fine-tuned on TOD Instruction-tuned LLM Task Flow Task Flow Dialog State Intent Dialog Response Dialog State Intent Dialog Actions Dialog Response — Task Instructions NLU DST DB NLG — Schema CSchema C — Task Instructions NLU DST DB NLG — Schema C Schema ASchema ASchema BSchema B — Task Instructions NLU DST DB POL NLG — Schema BSchema B — Task Instructions NLU DST DB POL NLG — Schema B (b) Our Instruction-tuned LLM for TOD Fig. 1: Comparison between conventional PCMs and our instruction-tuned LLM for task-oriented dialog systems. In Figure (a), PCM A and PCM B are derived by fine-tuning pre-trained models on specific corpora A and B, respectively. Each PCM is tailored for a particular workflow and lacks the flexibility to apply to other dialogue scenarios. In contrast, as depicted in Figure (b), the TOD Instruction-tuned LLM is developed by fine-tuning a large language model on multiple corpora, employing both instruction-aware and schema-aware mechanisms. This design enables it to generalize effectively to rare or unseen dialogue scenarios. 2.1 Problem definition 2.1.1 Notations Let C = (U 1 ,R 1 ), (U 2 ,R 2 ),..., (U T ,R T ) denote a complete dialogue with T turns, where U and R are the messages from the user and the system. d = d 1 ,...,d M represents the M domains supported by the dialog system, and each domain d m corresponds to a databaseDB d m containing many entries, each of which consists of multiple attributes. Moreover, databases in many TOD systems predefine a set of dedicated API interfaces (i.e., intents) for querying or updating, whose calling parameters consist of specific attribute- value pairs. In this work, we refer to domain-attribute tuples as slots, typically categorized into informable slots that users need to specify and the system needs to track, and requestable slots about which users can make inquiries. Given a set of all informable slots S = S 1 ,S 2 ,...,S J , the dialog state at turn t is defined as B t = n S j : v (t) j | 1≤ j ≤ J,v (t) j ∈V j o , where V j is the value space of slot S j . By putting the value spaces of all slots together, we construct the global state space O =(S 1 ,V 1 ), (S 2 ,V 2 ),..., (S J ,V J ), also known as the ontology. Springer Nature 2021 L A T E X template 6A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling Table 1: Detailed statistics of our end-to-end instruction-tuning TOD corpus Dataset Annotations # Sess. # Turn # Dom. # Sch. NLU DST DB POL NLG CamRest676 ×✓ ×✓6762,74411 (0) In-Car×✓ ×✓3,0287,87633 (0) MultiWOZ 2.1 ×✓10,43371,48477 (0) SGD✓22,593229,15620270 (528) Frames×✓ × ×✓1,3679,57911 (0) BiToD✓3,68936,23155 (7) STAR✓ ×✓5,80353,5581313 (24) BANKING77✓ × × × ×13,24213,24211 (0) CLINC150✓ × × × ×22,50022,5001010 (0) HWU64✓ × × × ×11,03611,0361818 (0) SNIPS✓ × × × ×14,48414,48477 (0) TOTAL108,851471,89086336 (559) These datasets include either subsets or the complete set of annotations pertinent to task-oriented dialogue. In this context, “DB” denotes database query results, “Sess.” refers to the number of dialogue sessions, and “Dom.” indicates the number of domains. In the “Sch.” column, the figures outside the parentheses reflect the number of domain schemas, while those inside the parentheses represent the count of intent schemas. The schema of a TOD system, which structurally describes the applicable scenarios and services that the system can provide, encompasses the subordi- nate slots and intents of each domain, the slots associated with the intent, and the ontology. 2.1.2 Modularized task-oriented dialog systems In a modularized TOD system, each turn of interaction with the user requires understanding user utterances, tracking dialog states, querying the databases when necessary, making decisions, and replying to the user in natural language, where the execution of each task depends on the output of the previous task, known as the task flow. At each turn t, the system first applies a natural language understanding (NLU) module to identify the domains, intents, and slot-value pairs involved in U t : n (d (t) k ,i (t) k ,s (t) k ) o K k=1 = NLU (U t ), (1) where K is the number of identified intents, i (t) k is the k-th intent, d (t) k and s (t) k are the domain and slot-value pairs corresponding to i (t) k , respectively. Then, conditioned on the current dialog context C t = (U 1 ,R 1 ),..., (U t−1 ,R t−1 ),U t and the previous dialog state B t−1 , a dia- log state tracking (DST) module maintains the dialog state B t by modifying Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling7 existing slots, adding new slot-value pairs, or deleting invalid slots in B t−1 : B t = DST (C t ,B t−1 ). (2) The predicted B t is then used to interact with the database when necessary. Concretely, for each intent i (t) k , the dialog system extracts the slot-value pairs required for the API call from B t and then queries the corresponding database DB d (t) k to obtain the results E (t) d (t) k . Subsequently, a dialog policy (POL) module plans the next action based on the DB results and dialog state, such as recom- mending entities that meet user needs, requesting more information from the user when conditions are insufficient, and informing the user of the relevant information requested: A t = POL C t ,B t ,E (t) ,(3) where A t is a set of (domain,act,slot) triplets, and act indicates the opera- tion applied to the slot, such as informing the slot value. Finally, a natural language generation (NLG) module transforms the action A t into a fluent and informative response R t : R t = NLG (C t ,A t ). (4) For the training of a modularized TOD system, each module is individually designed and optimized on its respective task data. 2.1.3 End-to-end task-oriented dialog modeling Separately modeling each module of a TOD system can lead to error propaga- tion, where errors made by previous modules are continuously accumulated and amplified in subsequent modules, causing the system’s behavior to deviate sig- nificantly from the expected. To alleviate the error propagation problem, much work proposes various joint training mechanisms on multiple modules (Chen et al, 2019; Wang et al, 2020; Zhang et al, 2020c), enabling a single model or multiple models to learn to solve multiple tasks together. For instance, Liang et al (2020) jointly train four decoders to sequentially perform NLU, DST, POL, and NLG tasks, while Lee (2021) optimizes an auto-regressive decoder to generate system action and response. Recently, PLMs such as GPT-2 (Radford et al, 2019), T5 (Raffel et al, 2020), and UniLM (Dong et al, 2019) have been introduced into TOD systems to simplify system design and improve performance. PPTOD (Su et al, 2022) formalizes all TOD subtasks into the text-to-text format and trains a unified T5 model to generate the corresponding task outputs based on different task prefixes. On the other hand, PLMs also have pushed joint modeling towards a more end-to-end manner, allowing a single model to efficiently execute the entire task flow of a TOD system for each dialog turn. Hosseini-Asl et al Springer Nature 2021 L A T E X template 8A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling (2020) and Peng et al (2021) sequentially concatenate various TOD subtasks and database results in the causal decoding mode of auto-regressive models, allowing a single model to complete multiple tasks and more effectively extract the relationships between these tasks. Formally, the learning objective of end- to-end TOD joint modeling for each turn t is to minimize the negative log- likelihood of task outputs given the dialog context C t : L turn =− logp θ R t ,A t ,E (t) ,B t |C t =− logp θ (B t |C t )· p θ A t |C t ,B t ,E (t) · p θ R t |C t ,B t ,E (t) ,A t , (5) where θ is the model parameters to be optimized and E (t) is derived from the database API calls as described in Section 2.1.2. UBAR (Yang et al, 2021) further extends the end-to-end modeling to a more practical TOD setting by modeling on the dialog session level, where the intermediate task annotations of previous turns are also included in the dialog history during training. Thus, the loss function of session-level end-to-end TOD modeling can be defined as: L session =− T X t=1 logp θ R t ,A t ,E (t) ,B t |H t , H t = ( H t−1 ⊕ B t−1 ,E (t−1) ,A t−1 ,R t−1 ,U t , if t > 1; U 1 ,if t = 1, (6) where the operator ⊕ represents the string concatenation operation. This allows the model to mine more characteristics for each task from the coher- ent session-level workflow, such as incremental updates of dialog states, the coherence of action decisions, and the consistency of response generation. 2.2 Task-oriented dialog datasets Benefiting from the broad prospects of intelligent assistants, task-oriented dialog systems have received widespread attention in recent years, and many task-oriented dialog datasets have been proposed to verify the sys- tem designs (Mrksic et al, 2017; Wen et al, 2017; Budzianowski et al, 2018; Mehri et al, 2020; Byrne et al, 2019; Rastogi et al, 2020; Lin et al, 2021). These datasets can be roughly divided into three categories: (1) language understanding datasets, (2) dialogues with modular annotations, and (3) dia- logues with end-to-end annotations. Language understanding datasets label the intents or slots in user utterances, aiding in the development of the natu- ral language understanding module. A complete task-oriented dialogue system typically needs to sequentially accomplish user utterance understanding, dia- logue state tracking, database querying, system action decision-making, and response generation. Therefore, multi-turn dialog datasets with intermediate task annotations have been introduced, where each turn of the dialog contains Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling9 not only user utterances and system responses but also dialog states, database query results, etc. End-to-end annotated dialog datasets, on the other hand, only provide the underlying databases for each dialog, requiring the dialog system to directly retrieve the databases to reply to the user. In this work, we focus on modeling modularized task-oriented dialogue systems in a task-flow manner. We carefully select 11 task-oriented dialog datasets to construct the instruction-tuning corpus, including four language understanding datasets and seven multi-turn dialog datasets with modular annotations. Table 1 shows the statistics of each dataset and the involved dialog tasks. 2.3 Backbone models Backbone models. Both Llama series models open-sourced by Meta AI 1 and Qwen2.5 series developed by Alibaba Cloud 2 are built on the decoder- only transformer architecture. These models are pre-trained on a large-scale text corpus sourced from publicly available data using a straightforward auto- regressive language modeling objective. The Llama and Qwen2.5 models have demonstrated a powerful ability to comprehend context and a great potential in faithfully following TOD-related task instructions. In this work, we use Llama 2 7B as the backbone model, which comprises 32 transformer blocks, each with 32 attention heads and a hidden dimension of 4096. Furthermore, to validate the generalizability of our proposed framework, we introduce three additional models from the Qwen2.5-Instruct series (Yang et al, 2024) with varying parameter counts: 0.5B, 1.5B, and 3B. This allows us to assess the effectiveness and scalability of the framework across different model sizes and architectures, thereby providing a more comprehensive eval- uation of its performance. The pre-training configurations of Llama 2 and Qwen2.5 enable the models to handle input sequences of up to 4096 and 32K tokens, respectively. Instruction fine-tuning. Instruction fine-tuning (IFT) (Wei et al, 2022) is a supervised learning paradigm employed to enhance the capabilities of pre-trained language models (PLMs). Following the initial large-scale, self- supervised pre-training phase, instruction fine-tuning further trains the model on a curated dataset composed of (instruction, input, output) triples. These instructions are typically formulated in natural language and describe a wide variety of tasks (e.g., summarization, question answering, translation, reasoning). The primary objective of this process is to teach the model to generalize across different tasks by understanding and following human-provided com- mands. By being exposed to a diverse set of explicit instructions and their corresponding high-quality responses, the methodology has been shown to be highly effective in aligning models with human preferences (Ouyang et al, 1 https://ai.meta.com/meta-ai/ 2 https://qwen.ai/ Springer Nature 2021 L A T E X template 10A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling Domain Schema Intent Schema Database Results MusicBusesHotels ...... FindBusBuyBusTicket What is a good hotel in New York City? Of the 10 I see, I recommend 1 Hotel Brooklyn Bridge, a 4 star place. ...... What else you got? How about a 3 star? Of the 8 3 stars I have, I'd say go with 11 Howard. What about a bus for 2 people leaving from Philly to go there on the 5th? ["Buses", "Hotels"] Domain Identification "Buses": ["FindBus"] Intent Detection "Hotels": "destination": "New York City", "has_wifi": "True", "hotel_name": "11 Howard", "star_rating": "3", "Buses": "from_location": "Philly", "leaving_date": "the 5th", "to_location": "New York City", "travelers": "2" Dialog State Tracking [Buses] [inform_count] [recommend] leaving_time fare transfers Got it. I have [inform_count] buses. There's an [leaving_time] with [transfers] transfers for [fare]. Got it. I have 8 buses. There's an 8:30 am with 0 transfers for $24. ...... Ok. Now how about that 1 hotel room for 2 days starting next Wednesday? 1 room at 11 Howard in New York on March 6th for 2 days? ...... Buses: 8 matched entities, [1] "fare": "24", "from_location": "Philadelphia", "from_station": "Intercity Terminal", "leaving_date": "2019-03-05", "leaving_time": "08:30", "to_location": "New York", "to_station": "Port Authority Station", "transfers": "0", "travelers": "2" [2] ...... ...... Domain Schema Informable slots from_loc; leaving_date; leaving_time; to_loc; travelers: "possible_values": [1, 2, 3, 4, 5] Requestable slots from_station; transfers; to_station; fare Intents FindBus; BuyBusTicket Intent Schema Required slots ["from_loc", "to_loc", "leaving_date"] Optional slots "travelers": 1 Result slots ["from_loc", "to_loc", "from_station", "to_station", "leaving_date", "leaving_time", "fare", "travelers", "transfers"] UserSystem SchemaDatabase Intermediate Inference Fig. 2: Illustration of the instruction-tuning paradigm for task-oriented dialog systems 2022). This significantly improves its zero-shot and few-shot performance on unseen tasks (Wei et al, 2022), making it a more versatile and practical tool that can be prompted to perform new functions without requiring task-specific training. 3 Methodology Overall, we propose an instruction-tuning paradigm for task-oriented dialogs, upon which a generalizable TOD system ESAinsTOD is built via session- level end-to-end modeling on dialogues with modular annotations. To enhance the applicability of ESAinsTOD to a diverse set of TOD tasks and a wide range of domains, we employ three key mechanisms to fully leverage multiple TOD datasets with partially overlapping tasks, including an instruction-aware mechanism, a schema-aware mechanism, and a session-level end-to-end model- ing mechanism. In the subsequent sections, we will detail (1) how to construct an instruction-tuning corpus from various TOD datasets in a unified manner, and (2) how to leverage this corpus to accomplish end-to-end TOD modeling. Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling11 Table 2: Instruction template variations across datasets. The Full Task Instructions part presents example output for each task instruction. Note that the example output is not included in the instruction template. System Instruction PleaseactasanAIassistanttointeractwiththeuserinatask-orienteddialoguescenarioto meethis/herneeds. Foreachmessagefromtheuser,followtheinstructionsbelowtogenerateintermediateresults untiltheassistantreplies: Full Task Instructions DIPleaseidentifythedomainsinvolvedintheusermessagefrom:domainlist. Example : ["banking"] ; ["hotel", "train"] IDSelectthecorrectintent(s)expressedintheusertextamongtheprovidedintents. Example : "RentalCars": ["ReserveARentalCar"], "Media": ["MovieSearchGenre"] DSTPleasemaintaintheuser’sneedsfromthebeginningofthedialoguetothepresent inthefollowingformatofslot-valuepairs:dstformat. Example : "hotel": "pricerange": "cheap", "area": "west", "train": "dest": "norwich" SADBeforegeneratingtheassistant’sreply,summarizethesystemactiondecisions. Example : [hotel] [recommend] name [inform] type stars parking internet Delex RGGeneratedelexicalizedassistantreply. Example : How about [valuename]? It has free parking, and it’s a [valuestars]-star hotel. Conc RGGenerateconcreteassistantreply. Example : How about University Arms Hotel? It has free parking, and it’s a 4-star hotel. Dataset System Prompt CamRest System Instruction +① DI② DST③ Delex RG In-Car System Instruction +① DI② DST③ Delex RG④ Conc RG MultiWOZ System Instruction +① DI② DST③ SAD④ Delex RG⑤ Conc RG SGD System Instruction +① DI② ID③ DST④ SAD⑤ Delex RG⑥ Conc RG Frames System Instruction +① DI② DST③ Conc RG BiToD System Instruction +① DI② DST③ SAD④ Delex RG⑤ Conc RG dst format : "relations": ["equal to", "atleast", "not", "oneof"], "examples": "slot1": "oneof(val1, val2)", "slot2": "equalto(val3)" STAR System Instruction +① DI② ID③ SAD④ Conc RG BANKING System Instruction +① ID CLINC System Instruction +① DI② ID HWU System Instruction +① DI② ID SNIPS System Instruction +① DI② DST The numbers (①-⑥) denote the sequential order of the instructions and the corresponding tasks to be executed. Except for BiToD, the dstformat of the datasets containing DST annotations is"format": "slotname": "slotvalue", "examples": "slot1": "val1", "slot2": "val2". 3.1 Alignment between dialog scenarios and annotations Fig. 2 provides an illustrative example of transforming a dialogue session from the SGD dataset into instruction-tuning data, showcasing the comprehensive process of session-level end-to-end modeling and inference. Instruction-aware mechanism. PPTOD (Su et al, 2022) jointly learns all TOD tasks by recasting them as text generation problems with task- specific prompts, without explicitly modeling the relationships between tasks. Springer Nature 2021 L A T E X template 12A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling Although existing methods (Hosseini-Asl et al, 2020; Peng et al, 2021; Yang et al, 2021) successfully achieve end-to-end modeling of a fixed task flow in TOD systems, they overlook the diversity in task sets and annotation for- mats that different TOD datasets may present. Motivated by previous work, we establish a series of task instructions prior to initiating a conversation. These instructions delineate detailed guidelines for a TOD system, for example, generating dialogue states according to a specified annotation format, and the order in which the TOD system should complete each task. It is worth noting that there are differences in instruction templates across datasets, which are summarized in Table 2. This variability allows the instruction-tuned system to adapt flexibly to diverse task flows and develop instruction awareness. Schema-aware mechanism. The schemas, defined in Section 2.1.1, are typically incompatible across different datasets, posing a challenge for exist- ing methods to make consistent predictions with the corresponding schema. Different from specialized TOD systems tailored for specific datasets, recent research has shifted towards building generalizable subtask modules by explic- itly providing the corresponding schema in the task input. Zhang et al (2023) constrain the LLM to generate dialog states according to the given slot names and possible values, and Lee et al (2021) supervise T5 to understand slot descriptions and generate corresponding slot values. Recognizing the critical role of schema information in general-purpose TOD systems, we integrate it into our dialogue context modeling to advance LLMs to generate outputs that are aware of the specific schema. As illustrated in Fig. 2, once the domains associated with each user utterance are identified, the schema of each acti- vated domain — encompassing its slots, possible values, and list of intents — is directly appended to the domain recognition results. In this way, subsequent tasks, such as intent detection or dialog state tracking, can utilize the dialog history to identify intents and slots mentioned by users within the predefined schema, thereby preventing the system from yielding unexpected outcomes. Similarly, if the dataset predefines the schema for intents, the schemas cor- responding to the recognized intents will also be incorporated into the intent recognition results. 3.2 Session-level end-to-end modeling Turn-level end-to-end TOD modeling is centered on jointly learning the task flow for each user message, taking both the current user message U t and the dialogue history C 1:t = (U 1 ,R 1 ),..., (U t−1 ,R t−1 ) as input. However, this turn-level modeling restricts the system from accessing historical execu- tion records for each task, which could compromise the behavioral consistency vital to the functionality of a practical TOD system. Following UBAR (Yang et al, 2021), we preserve the task flow execution results of each dialogue turn within the dialogue history, enabling session-level end-to-end TOD modeling. Specifically, for the t-th turn of the dialogue in Fig. 2, our instruction-tuning Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling13 framework defines the dialogue context as: H t = ( H t−1 ⊕ d (t) ,i (t) ,B t−1 ,E (t−1) ,A t−1 ,R t−1 ,U t , if t > 1; inst,U 1 ,if t = 1. (7) In Equation 7, inst denotes the task instructions, while d (t) and i (t) respec- tively represent the domain and intent recognition results of the user message U t , with the corresponding schemas included in both. Moving forward, we will describe the implementation details of each TOD-related task and formalize the process of session-level end-to-end TOD modeling process. Task: Domain identification. Given the impracticality of incorporating the schemas for all domains supported by a TOD system within the context, it is essential to first identify the domains relevant to the user utterance at each turn. We approach the domain identification task — a multi-label classi- fication challenge — as a generative problem, where domain labels are directly produced as a serialized Python list to facilitate easy parsing. The learning objective of generating the domain label sequence d (t) is defined using the standard negative log-likelihood loss: L domain =− logp θ d (t) |H t =− L d X l=1 logp θ d (t) l |H t ,d (t) <l , (8) where L d is the number of tokens derived from tokenizing the sequence d (t) , and d (t) <l denotes all tokens that precede the l-th token in d (t) . Subsequently, the active domains can be easily parsed from the generated d (t) and the corresponding schemas are appended to d (t) : ̃ d (t) = parse d (t) , d (t) = d (t) ⊕schema dom dom∈ ̃ d (t) . (9) Here, ̃ d (t) represents the set of identified domains obtained through the parsing function parse(·), such as deserialization. Task:Intentdetection. Toaccommodatemulti-intentscenar- ios, wherein users express several intents within a single domain or across multiple domains in a single turn, we utilize a Python dic- tionary structure to represent intents. This structure is formatted as domain 1 : intentlist 1 , domain 2 : intentlist 2 ,.... The training objective of intent detection is computed as: L intent =− logp θ i (t) |H t ,d (t) =− L i X l=1 logp θ i (t) l |H t ,d (t) ,i (t) <l , (10) Springer Nature 2021 L A T E X template 14A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling Algorithm 1 Schema management Require: Dialog session C with T turns Require: Schema type type∈domain, intent and schema set S type Require: Maximum number of history turns W 1: Initialize an empty mapping M type ⇐ ∅ 2: for each turn (U t ,R t ) in C do 3:Obtain parsed labels y (t) of U t , such as domains ̃ d (t) and intents ̃ i (t) 4:for each label y (t) k in y (t) do 5:if y (t) k /∈M type or M type h y (t) k i > W then 6:Append schema S type h y (t) k i to corresponding recognition results 7:M type h y (t) k i ⇐ 0 8:end if 9:end for 10:for e in M type do 11: M type [e]⇐M type [e] + 1 12:end for 13: end for where i (t) is the serialized intent label sequence of length L i . Similar to domain identification, the schemas of the parsed intents ̃ i (t) are appended to i (t) . Schema management. In practice, when multiple dialogue turns discuss the same topic, the same schema may be repeated numerous times within the dialogue context. Therefore, we employ a schema management algorithm to control the incorporation of schemas during data construction and end-to-end inference. This technique allows schemas to be shared across multiple dialogue turns, as detailed in Algorithm 1. Task: Dialog state tracking. As illustrated in Fig. 2, the dialog state is represented as a serialized nested dictionary, with the state of each domain comprising slot-value pairs. The generation loss of the dialog state B t : L dst =− L B X l=1 logp θ B t,l |H t ,d (t) ,i (t) ,B t,<l . (11) The parsed dialog state is then used to interact with the underlying databases. Unlike prior work, our database query results E (t) not only include the number of matched entities or the success of the API call, but also contain detailed API outcomes. Task: System action decision and response generation. Lastly, the system action A t and response R t are sequentially produced, grounded in the dialogue contextH t and the outputs of previous tasks. The corresponding loss Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling15 functions are: L pol =− L A X l=1 logp θ A t,l |H t ,d (t) ,i (t) ,B t ,E (t) ,A t,<l ,(12) L nlg =− L R X l=1 logp θ R t,l |H t ,d (t) ,i (t) ,B t ,E (t) ,A t ,R t,<l .(13) 3.3 Instruction-tuning and inference In summary, the core of our proposed instruction-tuning architecture lies in three key mechanisms: (1) the instruction-aware mechanism, which unifies TOD datasets with varying task sets, ensuring the fine-tuned model can fol- low instructions to accomplish the corresponding tasks, (2) the schema-aware mechanism, which aligns the dialog-specific schema (e.g., intent names, slot names) with the task annotations of each dialog, enabling the model to make predictions consistent with the given schema, and (3) the session-level end-to- end modeling mechanism, which captures the information flow between tasks and the behavioral changes of each task across dialog turns. Full learning objective. Due to the considerable overhead of anno- tating task-oriented dialog data, not all datasets are as comprehensively annotated as the SGD dataset, which includes all five TOD tasks T = domain, intent, dst, pol, nlg. This results in heterogeneous dialog corpora. In this scenario, we aim to employ instruction-tuning across various heteroge- neous datasets to develop a generalizable TOD system that adheres closely to task instructions and schemas. Let D =∪ N i=1 D i denote the instruction-tuning corpus, which comprises N TOD datasets. Each dataset D i includes a sub- set T i of total tasks T and is constructed according to our instruction-tuning paradigm. The total lossL θ (D) over our instruction-tuning corpusD combines multiple training objectives from (8) to (13): L θ (D) = N X i=1 |D i | X n=1 X task∈T i L task (x n ), (14) where x n is the n-th dialog in the i-th dataset. In the instruction-tuning stage, the model parameters θ are learned via mini-batch optimization, as detailed in Section 4.2. Inference. For a new dialogue from dataset D i , the system is first fed with the task instructions inst defined by the task subsetT i and the user utterance U 1 to initiate the inference process. Then the generated content of each task is retained within the task flow of the current turn, and the task flow execution results of each turn are preserved in the dialogue context to facilitate session- level end-to-end TOD inference, as illustrated in Fig. 2. Springer Nature 2021 L A T E X template 16A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling Error Handling. To ensure system robustness against malformed or unparsable outputs from the language understanding module, particularly in complex multi-intent scenarios, we employ a strict error-handling protocol during the parsing stage. Our framework utilizes a standard JSON parser to convert the model’s textual output into structured data formats, such as nested dictionaries and lists, which are essential for schema retrieval and database interaction. In instances where the output is syntactically incorrect and fails to parse, the system is designed to return a default empty structure (e.g., an empty dic- tionary or list) instead of raising an exception. This prevents system crashes and allows the dialogue to proceed. Crucially, the original, unparsed string generated by the model is always retained in the dialogue context. This design choice ensures no information is lost and provides an opportunity for down- stream error analysis or recovery mechanisms. Empirically, we have found that such parsing failures are infrequent, even with smaller-scale models. Context Management. As stated in Section 3.2, our model ESAinsTOD is fine-tuned and evaluated in a session-level end-to-end manner. In this paradigm, the context ideally includes all previous user messages, system out- puts for various dialogue tasks, and potentially extensive schema definitions and database results. For multi-turn dialogues, this concatenated input can certainly exceed the model’s token limit. To manage inputs that exceed the model’s maximum token limit (e.g., 4096 tokens for LLaMA 2 7B), we employ a straightforward yet effective sliding-window strategy for truncation as follows: • During Fine-tuning: For long dialogue sessions, we segment them into multiple training samples using a sliding window. The window is carefully selected to ensure that any turn that would otherwise be truncated still retains sufficient historical context to be a meaningful training example. In our implementation, we allocate up to 60% of the maximum token length for the preceding dialogue history. • During Inference: At each turn of the conversation, we preserve the most recent historical turns that fit within approximately three-quarters (75%) of the maximum token limit. The objective of retaining this sufficient context is to ensure that the model can fully model the relationship between the dialogue history and the current dialogue turn, which is crucial for coherent and context-aware responses. Beyond the primary sliding-window approach, we have implemented two other mechanisms to further mitigate token consumption: • Schema Management: As presented in Algorithm 1, we employ a schema management technique to reduce the token usage associated with schema definitions. • Output Deduplication: During inference, we also reduce the token count of task results by identifying and removing repetitive sequences from overly long generated text. Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling17 In Section 5.2.4, we will analyze the impact of sliding window size on end-to- end dialogue modeling performance of ESAinsTOD by limiting the number of history turns. 4 Experiment setup In this section, we will first introduce the benchmarks used in our experiments, along with the evaluation metrics and baselines for each task. Then, we will briefly describe the implementation details of our instruction-tuning framework and our experimental settings. 4.1 Benchmarks We test ESAinsTOD on three types of task-oriented benchmarks, includ- ing language understanding, dialogue state tracking, and end-to-end dialog modeling. 4.1.1 Language understanding Natural language understanding (NLU) is a fundamental task in task-oriented dialog systems, which assists in subsequent modules (i.e., dialogue management and response generation) by classifying user intents and recognizing the related slots in user utterances (Tur and De Mori, 2011; Young et al, 2013; Qin et al, 2021b; Teng et al, 2024). Evaluated datasets. Intent detection (ID) aims to recognize the intentions expressed in the current user utterance. We choose three popular single-turn benchmarks: BANKING77 (Casanueva et al, 2020) with 77 intents in the bank- ing domain, HWU64 (Liu et al, 2019) with 64 intents spanning 21 domains, and CLINC150 (Larson et al, 2019) with 150 intents spanning 10 domains. Slot filling (SF) is to recognize the slot values mentioned in the user utter- ances based on a predefined set of slots. The SF datasets usually contain intent annotations as well, and the two tasks are closely coupled, e.g., recog- nizing user intents may help determine which slots should be filled. Therefore, considering the interaction between the two tasks can achieve a more accu- rate understanding of user utterances. SNIPS (Coucke et al, 2018) is a widely used benchmark, which contains 7 intent types and 72 slot labels in the home automation domain. Baselines. Wecompare ESAinsTODwithseveralclassification- basedapproaches,includingConvBERT(Mehrietal,2020), USE+ConveRT (Casanueva et al, 2020), Example+Observer (Mehri and Eric, 2021), and ConvFit (Vulic et al, 2021). Among them, Exam- ple+Observer and ConvFit are similarity-based classification methods, which treat all sentences belonging to an intent as diverse surface instances, and predict the intent by finding the most similar instance to the test sentence from the training set. On SNIPS, three joint models are compared, including Springer Nature 2021 L A T E X template 18A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling BERT-Joint (Castellucci et al, 2019), Stack-Propgation+BERT (Qin et al, 2019), and Co-Interactive transformer+BERT (Qin et al, 2021a). It is worth noting that most methods model intent recognition as a sentence classification task and slot filling as a sequence labeling task. In contrast, our method directly generates the intent text and textual slot-value pairs based on user utterances and schema information, making it more flexible to adapt to new domains and settings, such as multi-intent recognition. Evaluation metrics. We adopted turn accuracy as the evaluation met- ric for intent recognition, which is the percentage of user turns where the model correctly predicts the active intent. Following Goo et al (2018) and Qin et al (2019), SNIPS is evaluated by slot F1 score, intent accuracy, and overall accuracy. 4.1.2 Dialogue State Tracking (DST) As a core component of task-oriented dialog systems, DST aims to extract and maintain user constraints consisting of slot-value pairs. DST extends the single-turn slot filling task into the interactive process between users and dia- logue agents, necessitating the model to accurately track user needs throughout the dialogue history, which encompasses various linguistic phenomena such as reference and ellipsis. Evaluated datasets. MultiWOZ (Budzianowski et al, 2018) is a challenging dataset for multi-domain DST, which contains many multi-domain dialogues, i.e., a dialogue session may involve domain or topic transitions multiple times. Baselines. All baselines for DST can be divided into three categories: 1) classification-based methods which select the value for each slot from a pre- defined candidate set, such as TripPy (Heck et al, 2020) and FPDSC (Zhou et al, 2021); 2) generation-based methods which directly generate slot values without being restricted to a fixed ontology, such as SOM-DST (Kim et al, 2020), SimpleTOD (Hosseini-Asl et al, 2020), Seq2seq-DU (Feng et al, 2021), UBAR (Yang et al, 2021), and SOLOIST (Peng et al, 2021); 3) hybrid methods which extract slot values from the dialogue history for non-categorical slots and select correct options for categorical slots, including DSTQA (Zhou and Small, 2019), DS-DST (Zhang et al, 2020a), and DSS-DST (Guo et al, 2021). Evaluation metrics. Joint Goal Accuracy (JGA) is used by much previous work (Wu et al, 2019; Heck et al, 2020; Zhang et al, 2020a) to measure the performance of dialogue state tracking, which is the ratio of user turns that all the slot-value pairs predicted by the model exactly match the ground truth. Different from previous methods that train and evaluate DST separately, we report the performance of DST in end-to-end modeling to verify the advantages of our instruction-tuning framework in multi-task joint modeling. Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling19 4.1.3 End-to-end (E2E) Dialog Modeling E2E dialog modeling is a challenging task that can measure the overall per- formance of a dialog system, where an agent needs to first perform language understanding, dialogue state tracking and database (DB) query, and then decide the next system actions based on the dialogue history and DB query results, and finally generate natural language responses. This fully end-to- end evaluation setting is closer to reality, where the system’s actions and responses are not only related to the dialogue context but also conditioned on the generated dialogue state and DB query results. Evaluated datasets. Following He et al (2022b), our model is evaluated on four widely used task-oriented dialog datasets: CamRest676 (Wen et al, 2017) with 676 dialogues on assisting users to find restaurants, In-Car (Eric et al, 2017) where the car assistant responds to user requests about three distinct domains (i.e., calendar scheduling, weather information retrieval, and point-of-interest navigation), MultiWOZ2.0 (Budzianowski et al, 2018) and MultiWOZ2.1 (Eric et al, 2020). MultiWOZ2.1 is a newer version of MultiWOZ2.0 with more accurate state annotations. Baselines. To demonstrate the effectiveness of our proposed ESAinsTOD in end-to-end TOD modeling, we compare it with the following state-of-the-art baselines: 1) SimpleTOD, UBAR, SOLOIST, MinTL (Lin et al, 2020), TOP+NOD (Liu et al, 2021), and MTTOD (Lee, 2021) for MultiWOZ; 2) SEDST (Jin et al, 2018), TSCP (Lei et al, 2018), FSDM (Shu et al, 2019), and LABES (Zhang et al, 2020b) for CamRest676 and In-Car. Evaluation metrics. The Combined Score (Mehri et al, 2019) is a com- mon overall metric for E2E dialog modeling, which is the weighted sum of three sub-metrics: Comb = BLEU + 0.5 × (Inform Rate + Success Rate). BLEU (Papineni et al, 2002) measures the similarity between the gener- ated response and the ground truth. Inform Rate and Success Rate reflect whether the agent offers a suitable entity and answers all the requested infor- mation in the generated response, respectively. For CamRest676 and In-Car, we report the Match and SuccF1 from Lei et al (2018), which correspond to Inform Rate and Success Rate, respectively. When calculating evaluation metrics on MultiWOZ datasets, the generated dialogue states are used to query the databases for DB results, which are then compared with the ground-truth DB results to obtain Inform Rate. 4.1.4 Comparison with Pre-trained conversation models Except for the above task-specific models, we also compare with three strong PCMs, including TOD-BERT (Wu et al, 2020), PPTOD (Su et al, 2022), and SPACE (He et al, 2022b). These PCMs are further pre-trained on large- scale TOD datasets with various well-designed pre-training objectives, and then fine-tuned on the above three tasks. Specifically, TOD-BERT is pre- trained with masked language modeling (MLM) and response selection (RS), Springer Nature 2021 L A T E X template 20A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling and PPTOD recasts all TOD-related tasks into input-output pairs on which a single multi-task model is pre-trained. SPACE is a unified semi-supervised PCM learned from both labeled and unlabeled dialog corpora. It comprises four successive components and is modeled within a TOD task flow with span-based MLM for the dialog encoder, semi-supervised contrastive learning objectives for dialog understanding and dialog policy decoders, and language modeling for the response decoder. 4.2 Implementation details We utilize Llama 2 7B (Touvron et al, 2023) as our backbone model and perform supervised fine-turning using PyTorch (Paszke et al, 2019), Trans- formers library (Wolf et al, 2020), FlashAttention (Dao et al, 2022), and DeepSpeed (Rasley et al, 2020). Instruction-tuning. Given that our preliminary experiments showed the superior performance of full-parameter fine-tuning over parameter-efficient fine-tuning (specifically, LoRA (Hu et al, 2022)), we employ fully fine-tuned LLMs to build our task-oriented dialogue system. We set the maximum sequence length to 4096 tokens and the batch size to 64. For multi-turn dia- logues, each sample consists of multiple consecutive dialogue turns in a dialogue session, where each dialogue turn contains the user message, intermediate task results, and system response. Long dialogues are split into multiple samples, while sufficient context is retained for each dialogue segment to ensure that the model can fully model the relationship between the dialogue history and the current dialogue turn. For single-turn language understanding datasets, we concatenate multiple examples to fully utilize the sequence length and use a special token for separation. Optimization. During the fine-tuning phase, we use the Fused Adam opti- mizer with L2 regularization (commonly referred to as AdamW (Loshchilov and Hutter, 2019)), with a weight decay set at 0.1 and gradient clipping capped at 1.0. This optimization is applied to an autoregressive language modeling objective. A cosine learning rate scheduler is adopted, initiating with a warmup ratio of 0.1 and reaching a peak learning rate of 5e-5. Notably, loss calcula- tion is restricted solely to the answer tokens for each task, ignoring all other tokens. The fine-tuning process for Llama 2 7B on our instruction-tuning cor- pus is completed in approximately 6 hours over 2 epochs, using 8 NVIDIA Tesla A100 80GB GPUs. Inference details. A high-throughput Large Language Model (LLM) serv- ing engine, named vLLM (Kwon et al, 2023), is employed to accelerate the inference processes in dialog systems. 3 A greedy decoding strategy is used for generating task results across various benchmarks. For each task, generation terminates either upon reaching the maximum token length or upon encounter- ing the designated stop tokens. To ensure the stability of the inference process, 3 https://github.com/vllm-project/vllm Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling21 multiple inferences are performed on each sample using different random seeds, and the average metrics are reported. 4.3 Experimental settings To investigate the effectiveness and generalization ability of our instruction- tuning framework, we conduct full training, low-resource, and zero-shot evaluations. Full training evaluation. Complete training sets from all datasets are mixed for fine-tuning, and the trained model is evaluated on the test sets of multiple benchmarks. The purpose of this setting is to explore the upper performance of our framework on various task-oriented dialog benchmarks. Low-resource and zero-shot evaluations. Recent studies have paid increasing attention to the transferability of dialog models to rare or unseen dialog scenarios. We combine the training sets from five task-oriented dialog datasets — namely, In-Car, SGD, Frames, BiTOD, and STAR — with differ- ent percentages of MultiWOZ data (i.e., 0%, 5%, 10%, 20%) for fine-tuning the backbone model, where 0% means that the model has never seen any Multi- WOZ data during training (i.e., zero-shot setting). Subsequently, we evaluate the model on the MultiWOZ test set. The zero-shot evaluation serves to mea- sure our framework’s ability to support extending to new domains or intents without retraining from scratch. Furthermore, we also simulate another low-resource scenario by varying the percentage of randomly sampled data from the instruction-tuning corpus to study the effectiveness of our method in data-scarce situations. To ensure the stability of the results in the low-resource setting, we report the average scores by sampling and training multiple times with three different random seeds. 5 Experiment results In this section, we present the experimental results and analysis of our proposed method for the task-oriented dialog system. 5.1 Main results The evaluation results of individual tasks under the full-data setting are shown in Tables 3, 4, 5, 6 and 7. End-to-end dialog modeling. Tables 3 and 4 present the end-to-end dia- log modeling results on CamRest676, In-Car, and MultiWOZ datasets. Our instruction-tuned model, ESAinsTOD, outperforms previous state-of-the-art methods across all four datasets. From Table 3, we can see that our model reaches the highest scores on all metrics for the CamRest676 and In-Car datasets. For both MultiWOZ 2.0 and 2.1 datasets, ESAinsTOD achieves absolute improvements of 4.78% and 3.92% in combined scores over the pre- vious strongest baseline, SPACE, as shown in Table 4. ESAinsTOD also Springer Nature 2021 L A T E X template 22A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling Table 3: End-to-end dialog modeling results on CamRest676 and In-Car Model CamRest676In-Car MatchSuccF1BLEUCombMatchSuccF1BLEUComb SEDST92.7075.4023.60107.6584.5082.9019.30103.00 TSCP92.7085.4025.30114.3584.5081.1021.90104.70 LABES96.4083.0025.50115.2085.8077.0022.80104.20 FSDM93.5086.2025.80115.6584.8082.1021.50104.95 SPACE ⋆ 97.74 88.2423.68116.6785.2683.1622.92107.13 ESAinsTOD (Llama 2 7B) 98.50 88.45 26.92 120.39 90.58 88.09 27.87 117.21 Best results are in bold, and the second best results are underlined. ⋆ Corresponding pre-trained conversational model (PCM) has been further fine-tuned on each downstream dataset individually. Table 4: End-to-end dialog modeling results on MultiWOZ Model MultiWOZ 2.0MultiWOZ 2.1 Inform Success BLEUCombInform Success BLEUComb SimpleTOD84.4070.1015.0192.2685.0070.5015.2392.98 UBAR † 85.1071.0216.2194.2786.2070.3216.4894.74 SOLOIST85.5072.9016.5495.74---- MinTL † 84.8874.9117.8997.78---- TOP+NOD86.9076.2020.58102.13---- PPTOD ⋆ 89.2079.4018.62102.9287.0979.0819.17102.26 MTTOD90.9982.5820.25107.0490.9982.0819.68106.22 SPACE ⋆ 91.5084.7019.30107.4093.0084.1019.91108.46 ESAinsTOD (Llama 2 7B) 94.30 87.10 21.48 112.18 94.40 87.50 21.41 112.38 Llama 2 7B (FT)92.8084.9020.95109.8092.3084.6021.33109.78 ESAinsTOD (GT) (Llama 2 7B) 88.2081.6021.87106.7787.6081.1021.92106.27 † During inference at current turn, corresponding methods require the orcale dialogue states of previous turns. ⋆ Corresponding PCMs have been further fine-tuned on each dataset individually. Llama 2 7B (FT) signifies that Llama 2 7B model was fine-tuned exclusively on the training set of each dataset, without cross-dataset training. ESAinsTOD (GT) indicates that results are based on ground-truth system actions and responses from previous turns. attains the highest SuccF1 or Success scores on In-Car and two MultiWOZ benchmarks, outperforming SPACE by 4.93, 2.40, and 3.40 points. In partic- ular, the BLEU score of ESAinsTOD surpasses those of all baselines across all four benchmarks. These advantages indicate that our dialog system can more accurately fulfill user requests and deliver smoother responses. Notably, all compared baselines have been specifically fine-tuned on their respective Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling23 Table 5: DST evaluation: Joint Goal Accuracy on Multi- WOZ ModelMultiWOZ 2.0MultiWOZ 2.1 Classification-based Approaches TOD-BERT-48.00 DST-Picklist † 54.3953.30 SST † 51.1755.23 TripPy-55.29 CHAN † 52.6858.55 FPDSC-turn † 55.0357.88 FPDSC-dual † 53.1759.07 Hybrid Approaches DSTQA † 51.3649.67 DS-DST52.2451.21 DSS-DST † 56.9360.73 Generation-based Approaches SOM-DST51.3852.57 MinTL52.1053.62 SimpleTOD-55.76 Seq2seq-DU-56.10 UBAR (DST)52.5956.20 SOLOIST (DST)53.2056.85 PPTOD ⋆ (DST)53.8957.45 ESAinsTOD (Llama 2 7B) 55.9058.68 Llama 2 7B (FT)54.9256.74 ESAinsTOD (GT) (Llama 2 7B) 57.2360.76 The end-to-end methods tagged with (DST) include versions specifi- cally trained for dialogue state tracking (DST) task. † Corresponding methods necessitate a full ontology in which all pos- sible values are predefined for each slot. ⋆ Corresponding PCM has been further fine-tuned on each down- stream dataset individually. datasets, with PPTOD and SPACE undergoing dialogue pre-training before fine-tuning. In contrast, ESAinsTOD is able to follow task instructions and schema information to adapt to various datasets that involve different domains and dialog tasks, making it more flexible and generalizable. Dialog state tracking. The accuracy of tracking dialog states is a key deter- minant of end-to-end dialog modeling performance. Therefore, we evaluate various methods for DST task, including both task-specific and end-to- end approaches, as detailed in Table 5. SOM-DST, Seq2seq-DU, and all classification-based and hybrid approaches are specifically designed for DST, whereas the remaining methods are end-to-end. Without relying on a complete ontology or any specialized post-processing, ESAinsTOD obtains the highest Springer Nature 2021 L A T E X template 24A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling joint goal accuracy (JGA) among all generation-based baselines. Unlike other end-to-end methods that require specialized training to improve dialog state generation, our model consistently achieves superior results in both DST and response generation tasks within end-to-end dialog modeling. In the MultiWOZ benchmarks, we adopt a more realistic setting where a TOD system can only access the dialog context containing its own generated content from previous turns, rather than relying on the ground-truth dialog states, system actions, and responses. For a fair comparison, we also report DST results based on both generated dialogue states and ground-truth sys- tem actions and responses from previous turns, referred to as “ESAinsTOD (GT)” in Table 5. When given actual dialog history, our model performs com- parably to, or even outperforms, state-of-the-art specialized methods such as FPDSC and DSS-DST in the DST task, highlighting its substantial poten- tial for dialog understanding. Similarly to the findings reported in UBAR, we observe that incorporating ground-truth responses into the dialog history leads to a significant decline in both the inform rate and success rate, as detailed in Table 4. This decline occurs because ESAinsTOD relies on its historical actions to make context-aware decisions for the current turn, and introducing ground-truth actions may disrupt the coherence of its dialog policy. Language understanding. We also report results on the test sets of the language understanding tasks included in our instruction-tuning cor- pus. According to Tables 6 and 7, ESAinsTOD demonstrates competitive performance compared to existing methods in intent detection and slot fill- ing benchmarks, with the exception of SPACE. It is worth mentioning that these intent detection datasets do not provide specific descriptions or expla- nations for abbreviated intent labels, which may make classification-based discriminative methods more appropriate. Consequently, we suggest that such datasets may not be conducive to end-to-end modeling tasks and could neg- atively affect performance on other intent detection datasets. Furthermore, we individually fine-tune Llama 2 7B on each benchmark and present the evaluation results in the “Llama 2 7B (FT)” row of Table 6. Compared to another generation-based method, PPTOD, Llama 2 7B (FT) achieves only a marginal improvement, indicating that merely increasing the model size does not yield significant benefits for intent detection tasks. We attribute this phenomenon to the absence of natural language descriptions for intent labels, making it difficult for LLMs to align these labels with the user’s natural language inputs. This issue will be further explored in future work. 5.2 Further analysis To further explore the advantages of our instruction-tuning framework, we conduct a comprehensive study focusing on the following key questions: • Question 1 : Are large language models (LLMs) suitable for direct appli- cation in task-oriented dialog modeling? (Refer to Section 5.2.1) Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling25 Table 6: Comparison results on intent detection tasks ModelBANKING77 CLINIC150 HWU64 Classification-based Approaches ConvBERT92.9597.0790.43 USE+ConvRT93.3697.1692.62 Example+Observer93.8397.3193.03 ConvFiT94.1697.3492.42 SPACE ⋆ 94.9497.8994.14 Generation-based Approaches PPTOD ⋆ 93.8697.1392.84 ESAinsTOD (Llama 2 7B) 92.8997.3192.75 Llama 2 7B (FT)94.3597.2993.01 Best results are in bold, and the second best results are underlined. ⋆ Corresponding PCMs have been further fine-tuned on each downstream dataset individually. Table 7: Comparison results on SNIPS dataset ModelIntent AccSlot F1Overall Acc BERT-Joint99.0096.2091.60 Stack-Propgation+BERT99.0097.0092.90 Co-Interactive transformer+BERT98.8097.1093.10 ESAinsTOD (Llama 2 7B) 99.4396.7692.14 Best results are in bold, and the second best results are underlined. • Question 2 : What is the generalization capability of ESAinsTOD in end- to-end dialog modeling? (Refer to Section 5.2.2) • Question 3 : Can schema information improve the data efficiency of the model? (Refer to Section 5.2.3) • Question 4 : What impact does dialog context have on session-level dialog modeling? (Refer to Section 5.2.4) • Question 5 : Can schema information reduce cascading errors in end-to-end dialog modeling? (Refer to Section 5.2.4) 5.2.1 Explore the strengths of large language models To explore “Question 1: Are large language models (LLMs) suitable for direct application in task-oriented dialog modeling?”, we first assess the base appli- cability of LLMs. ESAinsTOD leverages LLMs that have demonstrated exceptional instruction adherence and conversational proficiency. To determine their suitability for TOD modeling, we directly fine-tune Llama 2 7B on the MultiWOZ dataset under both full-resource and low-resource settings. Springer Nature 2021 L A T E X template 26A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling Table 8 : Low-resource end-to-end evaluation on MultiWOZ 2.0 Model 5% of training data 10% of training data 20% of training data Inform Succ. BLEU Comb Inform Succ. BLEU Comb Inform Succ. BLEU Comb MD-Sequicity 49.40 19.70 10.30 44.85 58.10 34.70 11.40 57.80 64.40 42.10 13.00 66.25 DAMD 52.50 31.80 11.60 53.75 55.30 30.30 13.00 55.80 62.60 44.10 14.90 68.25 SOLOIST 69.30 52.30 11.80 72.60 69.90 51.90 14.60 75.50 74.00 60.10 15.25 82.29 MinTL 75.48 60.96 13.98 82.20 78.08 66.87 15.46 87.94 82.48 68.57 13.00 88.53 PPTOD 79.86 63.48 14.89 86.55 84.42 68.36 15.57 91.96 84.94 71.70 17.01 95.32 ESAinsTOD ( Llama 2 7B ) 88.70 79.10 16.14 100.04 90.30 82.20 16.37 102.62 91.80 84.20 20.69 108.69 Llama 2 7B (FT) 86.00 76.20 14.28 95.38 89.90 80.70 15.15 100.45 89.20 82.40 18.21 104.01 Best results are in bold, and the second best results are underlined.Instruction-tuning data comprises the complete training sets of five end-to-end (E2E) dialog datasets, excluding CamRest676,along with corresponding proportions of the MultiWOZ 2.0 dataset.Results of all baselines are taken from Su et al (2022). “Succ.” denotes the success rate. Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling27 Table 9: Low-resource DST evaluation on Multi- WOZ 2.0 Model Training Percentage (%) 51020 SimpleTOD16.1422.3731.22 MinTL21.2830.3235.96 SOLOIST26.5332.4238.68 PPTOD base 40.2043.4546.96 PPTOD large 43.6145.9648.95 ESAinsTOD (Llama 2 7B) 50.41 51.48 52.89 Llama 2 7B (FT)47.6948.9451.26 ESAinsTOD (GT) (Llama 2 7B) 50.16 52.60 54.76 Llama 2 7B (FT, GT)48.7750.8152.14 Instruction-tuning data comprises the complete training sets of five end-to-end (E2E) dialog datasets, excluding CamRest676, along with corresponding proportions of the MultiWOZ 2.0 dataset. The resulting model, referred to as “Llama 2 7B (FT)”, consistently outperforms all baselines, including the pre-trained conversational models PPTOD and SPACE, as detailed in Tables 4 and 5. In low-resource setting, we used only 5%, 10%, and 20% of the training data for fine-tuning. The results, as listed in the “Llama 2 7B (FT)” row of Tables 8 and 9, show that Llama 2 7B achieves substantial advantages in terms of generalization over PPTOD across different data proportions. To be specific, Llama 2 7B, utilizing just 5% of the training data, attains a combined score of 95.38, which is equiv- alent to PPTOD’s 95.32 achieved with 20% of the training data. Moreover, Llama 2 7B with 20% of the training data achieves a score of 104.01, surpass- ing PPTOD trained on the entire dataset, which scored 102.92. These results reveal the potential of LLMs in transfer learning for task-oriented dialog mod- eling and imply that continuous dialogue pre-training may not be necessary for such models. 5.2.2 Analysis of generalization capability in end-to-end (E2E) dialog modeling This subsection addresses “Question 2: What is the generalization capability of ESAinsTOD in end-to-end dialog modeling?”. As discussed in Section 3.1, our instruction-tuning method is designed to adapt flexibly to various dialog tasks and domains by leveraging task instructions and schema information. To verify the effectiveness and generalization power of our approach, we conduct both low-resource and zero-shot end-to-end evaluations on the MultiWOZ datasets. Springer Nature 2021 L A T E X template 28A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling Table 10: Zero-shot evaluation results under different ablation settings ModelJGAInformSuccessBLEUCombined Turn-level evaluation on MultiWOZ 2.1 test set ESAinsTOD (Llama 2 7B) 63.0091.6073.506.6089.15 w/o ia56.1787.8067.006.6184.01 w/o sa48.6769.2039.306.8861.13 w/o ia, sa43.3867.3031.706.6556.15 Session-level evaluation on MultiWOZ 2.1 test set ESAinsTOD (Llama 2 7B) 26.9086.0063.306.6781.32 w/o ia14.8462.3040.206.4957.74 w/o sa7.1428.7013.206.6327.58 w/o ia, sa3.4523.309.505.9422.34 For turn-level evaluation, where oracle historical dialog states are included in the dialog context, the Joint Goal Accuracy (JGA) primarily measures the ability of the dialogue system to recognize the user’s needs in the current turn. Low-resource evaluation. The model is fine-tuned on the training sets of five other task-oriented dialog datasets, supplemented with varying propor- tions (i.e., 5%, 10%, 20%) of the MultiWOZ 2.0 dataset, and then evaluated on the MultiWOZ 2.0 test set. We compare ESAinsTOD with several state-of- the-art methods, including MD-Sequicity (Zhang et al, 2020c), DAMD (Zhang et al, 2020c), SOLOIST, MinTL, and PPTOD. The experimental results, pre- sented in Tables 8 and 9, indicate that ESAinsTOD consistently outperforms all baselines across different amounts of MultiWOZ 2.0 training data. Notably, ESAinsTOD surpasses PPTOD by significant margins in terms of Inform and Success rates, particularly when only 5% of the training data is avail- able (i.e., 88.70% vs. 79.86% in Inform rate, 79.10% vs. 63.48% in Success rate). This performance underscores ESAinsTOD’s capability to maintain robust dialogue modeling and understanding even in low-resource settings. Additionally, ESAinsTOD slightly outperforms Llama 2 7B (FT), meaning that our instruction-tuning framework can efficiently integrate diverse dialog datasets with varied annotation schemas, domains, and tasks. This integra- tion facilitates the transfer of dialog-related knowledge, such as general dialog policies, resulting in further performance enhancements in both full-resource and low-resource settings. When trained on 20% of the MultiWOZ 2.0 data, our ESAinsTOD is even comparable to the previous state-of-the-art baseline SPACE with full data, as detailed in Table 4. Zero-shot evaluation. To assess the adaptability of our proposed instruction-tuning framework in unseen dialog scenarios, we fine-tune Llama 2 7B on five TOD datasets before applying it directly to the MultiWOZ test set. In particular, in addition to the conventional session-level evaluation, we adopt a turn-level evaluation setting in which intermediate task results from historical turns in the dialog context are considered as ground truth. This Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling29 5102050100 Percentage of instruction-tuning data (%) 84 86 88 90 92 94 96 Inform Rate (%) with schema without schema (a) Inform Rate on MultiWOZ 2.1 5102050100 Percentage of instruction-tuning data (%) 76 78 80 82 84 86 88 Success Rate (%) with schema without schema (b) Success Rate on MultiWOZ 2.1 Fig. 3: Evaluation results of ESAinsTOD trained on varying percentages of our instruction-tuning corpus with or without schema information evaluation mode concentrates on assessing the system’s comprehension of the dialogue context as well as its ability to complete tasks at each turn. Table 10 presents the zero-shot evaluation results under different fine-tuning settings. When the schema-aware mechanism is removed from the instruction-tuning process, the new model (denoted as w/o sa) experiences a substantial reduc- tion in performance, with an average decrease of 39.85% in the Inform rate and 42.15% in the Success rate across both evaluation settings. This emphasizes the crucial role of aligning dialogue schemas with task outputs in enhancing the model’s generalization capabilities. Furthermore, removing task instruc- tions from the fine-tuning data (denoted as w/o ia) leads to significant declines in four out of five metrics, implying the importance of task instructions in distinguishing diverse task flows within different dialog scenarios. Compared to models fine-tuned with only one alignment mechanism (either w/o ia or w/o sa), the model without any alignment mechanism (denoted as w/o ia, sa) suffers more severe performance degradation. From the opposite perspective, employing both schema-aware and instruction-aware mechanisms simultane- ously enhances the generalization capabilities of the dialog system, ensuring optimal zero-shot performance on new dialogue schemas and task flows. Both low-resource and zero-shot evaluation results affirm the effectiveness of our proposed framework in low-resource task-oriented dialog modeling and emphasize the critical role of these two alignment mechanisms in improving the performance of dialog systems in unseen dialog scenarios. Springer Nature 2021 L A T E X template 30A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling 5.2.3 Data-efficiency analysis To answer “Question 3: Can schema information improve the data efficiency of the model in end-to-end dialog modeling?”, we conduct a data-scaling exper- iment. Concretely, we first randomly sample certain percentages of training data (i.e., 5%, 10%, 20%, and 50%) from seven E2E dialog datasets and then mix them together to construct the instruction-tuning data for fine-tuning Llama 2 7B. The ablation study in Fig. 3 illustrates how dataset-specific schema information influences task completion of our task-oriented dialog system. With limited training data, particularly at 5% and 10%, ESAinsTOD demonstrates improved performance by delivering more relevant entities and responding more accurately to user requests, compared to our implemented baseline that lacks schema information. This enhancement is quantitatively reflected in higher Inform and Success rates, underscoring the critical role of schema information in boosting the model’s generalization capabilities in extremely low-resource scenarios. By incorporating schema information, the model can effectively overlook the discrepancies between different datasets and learn more generalized dialogue policies. As the scale of training data increases (i.e., 20% and 50%), the performance gains from schema information gradually diminish. We speculate that with more data, the model effectively discerns the unique characteristics of each dataset, including the language styles of user and system utterances (e.g., length, case), as well as dataset-specific features (e.g., domain names, task names). Consequently, this comprehensive under- standing reduces the relative importance of schema information. Nonetheless, our ESAinsTOD still achieves a higher completion rate even with 100% of the training data, indicating that schema alignment can further elevate the performance ceiling of end-to-end task-oriented dialog modeling. To conclude, the integration of schemas enables the model to achieve comparable performance with less training data, thereby enhancing the data efficiency of the model. 5.2.4 Influence of dialog context Instead of the traditional turn-level approach, ESAinsTOD is fine-tuned and evaluated in a session-level end-to-end manner. Specifically, the outputs of vari- ous tasks in previous turns are retained in the dialog context and may influence task predictions in subsequent turns. Therefore, we conduct several experi- ments to investigate two related questions: “Question 4: What impact does dialog context have on session-level dialog modeling?” and “Question 5: Can schema information reduce cascading errors in end-to-end dialog modeling?”. Context window size. We first restrict the number of dialog history turns that ESAinsTOD can access when inferring each dialog turn to investigate the relationship between the context window size and inference results, as shown in Fig. 4. As the context window increases, the results on MultiWOZ 2.1 show a trend of rapid growth followed by a slow decline, with the combined Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling31 0123456 # window size 20 40 60 80 100 Metric Value 56.81 93.10 84.10 21.10 109.70 58.32 93.30 86.10 21.36 111.06 58.71 94.00 86.80 21.47 111.87 58.68 94.40 87.50 21.41 112.36 58.55 94.20 87.30 21.51 112.26 58.53 94.20 87.30 21.51 112.26 Combined Inform Success JGA BLEU Fig. 4: Results of ESAinsTOD on MultiWOZ 2.1 with varying context win- dow sizes Table 11: Session-level cascading error analysis of ESAinsTOD on Multi- WOZ 2.1 Schema Context Current InformSuccessBLEUCombined BeliefBelief ×GTGen94.5085.8019.61109.76 ×GenGen93.40 (−1.1) 84.70 (−1.1) 19.54 (−0.07)108.59 (−1.17) ✓GTGen94.5087.8021.33112.48 ✓GenGen94.40 (−0.1) 87.50 (−0.3) 21.41 (+0.08) 112.36 (−0.12) We report the inference results conditioned on dialogue history turns that contain either ground-truth or generated dialogue states. The numbers in parentheses denote the cascading impact of the model’s predictions from previous turns on the overall performance of dialogue modeling. score peaking when the context window is 4. It is evident that sufficient dia- log history is necessary for the model to plan reasonable and coherent dialog behaviors. However, the redundancy of excessive dialog context may slightly interfere with state tracking and decision-making, signifying opportunities for future research. Cascade error analysis. Cascading error propagation is prevalent among various components of TOD systems and serves as a major obstacle to the robustness of dialog systems. Next, by controlling the source of dialog states, we analyze how our proposed instruction-tuning framework alleviates session- level and task-level cascading errors in end-to-end dialog modeling. Table 11 shows that the inference results of ESAinsTOD on MultiWOZ 2.1 would Springer Nature 2021 L A T E X template 32A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling Table 12: Task-level cascading error analysis of ESAinsTOD on MultiWOZ 2.1 Schema Context Current InformSuccessBLEUCombined BeliefBelief ×GenGT94.8086.1019.70110.15 ×GenGen93.40 (−1.4) 84.70 (−1.4) 19.54 (−0.16)108.59 (−1.56) ✓GenGT94.1086.9021.52112.02 ✓GenGen94.40 (+0.3) 87.50 (+0.6) 21.41 (−0.11) 112.36 (+0.34) ×GTGT94.7086.4019.76110.31 ×GTGen94.50 (−0.2) 85.80 (−0.6) 19.61 (−0.15)109.76 (−0.55) ✓GTGT93.9086.7021.57111.87 ✓GTGen94.50 (+0.6) 87.80 (+1.1) 21.33 (−0.24) 112.48 (+0.61) decrease when the dialog context includes generated dialogue states from previous turns, compared to using ground-truth dialog states. However, the introduction of schema information significantly mitigates this performance degradation, with a notably smaller reduction in the combined score (−1.17 vs. −0.12), which underscores enhanced system stability. We attribute this to the fact that aligning the dialog-specific schema with the task outputs helps avoid interference from other dialogues in similar domains but with different schemas during fine-tuning, thereby facilitating more fine-grained dialog mod- eling. Additionally, a more interesting phenomenon emerges in the task-level cascading error analysis, where the database query, dialog policy planning, and response generation for each dialog turn are based on ground-truth dialog states of the current turn. As illustrated in Table 12, under both settings — whether or not ground-truth previous dialog states are included in the dialog context — we observe that the use of schema information allows the model to achieve higher Inform and Success rates with the generated dialog states of the current turn than using the ground-truth dialog states. In contrast, models without schema information exhibits significant cascading error propagation. This finding suggests that our instruction-tuning framework offers consider- able robustness against annotation errors found in the MultiWOZ 2.1 dataset, highlighting a notable level of resistance to data noise. 5.3 Ablation study In this section, to rigorously evaluate the contributions of our proposed com- ponents and dissect their individual impacts, we conduct a series of ablation studies. These experiments are designed to investigate: (1) the effectiveness of our schema and instruction alignment mechanisms in a fair and controlled setting; (2) the influence of backbone model scale on overall performance; and (3) the efficiency gains afforded by our schema management algorithm. Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling33 5.3.1 Effectiveness of alignment mechanisms A primary concern is to isolate the benefits of our methodology from the influ- ence of the backbone model. To ensure a fair comparison, we conduct a direct head-to-head evaluation where two representative baselines are fine-tuned on the same Qwen2.5 backbone models and the same unified corpus as our pro- posed ESAinsTOD. Furthermore, to precisely quantify the importance of each alignment mechanism, we compare our full ESAinsTOD model against two ablated variants. All models were trained on the unified corpus and evaluated on the MultiWOZ 2.1 test set. The selected baselines are as follows: • ESAinsTOD w/o ia is trained without instruction alignment. • ESAinsTOD w/o sa is trained without schema alignment. • UBAR (Yang et al, 2021), which involves session-level dialogue modeling and is functionally equivalent to our framework but without the schema and instruction alignment mechanisms. • PPTOD (Su et al, 2022), which adopts a multi-task learning paradigm by decomposing dialogue data into (task prompt, dialog input, task output) triplets. This method does not explicitly model the relationships between different dialogue tasks within a turn. The results, presented in Table 13, lead to several key insights. First, ESAinsTOD consistently outperforms all baselines and ablated variants across all model sizes, demonstrating the synergistic benefits of our com- plete framework. The significant performance delta between ESAinsTOD and UBAR directly validates the efficacy of our proposed alignment mechanisms, as this is the primary architectural difference between them. Second, the results allow us to dissect the distinct contributions of each component. Schema alignment generally has a more substantial impact on performance, particularly for dialogue state tracking. For instance, its removal (w/o sa ) results in the poorest state tracking performance among the ablated variants. Across most model sizes, schema alignment contributes more to the Inform and Success rates than instruction alignment. These findings suggest that schema alignment provides the model with a generalizable ability to follow structured schema definitions, while instruction alignment offers the flexibility to adapt to diverse dialogue task flows. Finally, ESAinsTOD’s superiority over PPTOD highlights the advantage of our session-level modeling approach, which better captures the complex inter-task dependencies inherent in realistic, multi-domain conversations. 5.3.2 Effect of backbone model scale An important question is how model scale influences the capacity of different TOD methodologies to perform complex dialogue tasks. To investigate this, we evaluate UBAR, PPTOD, and ESAinsTOD using various sizes of the Qwen2.5 series as the backbone. Springer Nature 2021 L A T E X template 34A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling Table 13: Ablation study on MultiWOZ 2.1 using Qwen2.5 model family MethodSession-LevelJGAInformSuccessBLEUComb Backbone model: Qwen2.5-0.5B-Instruct UBAR✓55.1786.8078.4018.37100.97 PPTOD×51.2289.9081.3019.78105.38 ESAinsTOD✓56.43 93.4085.3019.56108.91 w/o ia✓53.6888.2077.5020.07102.92 w/o sa✓52.8288.7080.9019.41104.21 Backbone model: Qwen2.5-1.5B-Instruct UBAR✓55.4085.9077.4019.44101.09 PPTOD×53.7391.1082.3019.88106.58 ESAinsTOD✓57.30 93.4085.1020.06 109.31 w/o ia✓56.5892.3083.0019.79107.44 w/o sa✓56.1986.9079.3019.85102.95 Backbone model: Qwen2.5-3B-Instruct UBAR✓56.0886.2078.4018.95101.25 PPTOD×54.1992.0083.5019.80107.55 ESAinsTOD✓57.58 94.7086.7019.90 110.60 w/o ia✓56.8493.3081.9019.39106.99 w/o sa✓53.5387.2080.0019.45103.05 Session-Level stands for session-level end-to-end task-oriented dialog modeling. The results are also summarized in Table 13. The performance of the baseline UBAR shows almost no improvement as the model size increases, suggesting that simply scaling up the model without a robust methodologi- cal framework yields diminishing returns. In contrast, both PPTOD and our ESAinsTOD exhibit moderate and consistent performance gains with larger backbone models. This finding leads to a crucial insight: merely increasing the size and power of the backbone LLM is insufficient to guarantee improved task-oriented dialogue capabilities. Instead, it is the methodological design—such as the alignment mechanisms in ESAinsTOD and the multi-task formulation in PPTOD—that is key to effectively unlocking and leveraging the inherent potential of the underlying language models. Our approach demon- strates superior scaling properties, indicating its effectiveness in harnessing the power of larger models. 5.3.3 Impact of schema management Next, we present further empirical analyses on the schema management strat- egy introduced in Algorithm 1. Algorithm 1 aims to enhance both training and inference efficiency by managing the repetition of schemas. We quantify its impact from two perspectives. First, on training data volume, Algorithm 1 avoids redundant schema def- initions within training samples, significantly reducing the total number of tokens required for training. As detailed in Table 14, applying this algorithm reduces the total token count of our unified corpus by approximately 44%, from Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling35 Table 14: Impact of schema management on the statistics of our unified instruction-tuning dataset Schema Management Samples ↓Turns per Sample ↑Tokens per Turn ↓ Total Tokens (Billion) ↓ ×63,0087.53401.70.184 ✓47,4898.63252.80.103 Tokens are counted by the Qwen2.5 tokenizer. ↑: higher is better and ↓: lower is better. The maximum token length per sample is 4096; dialogues exceeding this limit are split into multiple samples (see Section 3.3 for details). Table 15: Runtime efficiency comparison on MultiWOZ 2.1 using Qwen2.5- 0.5B-Instruct as the backbone MethodSM End-to-End Dialogue Modeling Inference Measurement JGA Inform Success BLEUCombLatency (ms) ↓ Speedup ↑ PPTOD-51.2289.9081.30 19.78 105.3820.941.00× UBAR-55.1786.8078.4018.37100.9725.960.81× ESAinsTOD × 56.52 93.5085.0019.54108.7940.590.52× ✓56.4393.40 85.3019.56 108.9130.650.68× SM stands for Schema Management. ↑: higher is better and ↓: lower is better. The latency of each method is measured on a single Nvidia A100 80GB GPU using offline inference with vLLM and is calculated by dividing the total inference time by the total number of dialogue turns. 0.184 billion to 0.103 billion, thereby lowering computational costs. This man- agement strategy also enhances the scope and training efficiency of session-level dialogue modeling by increasing the average number of turns per sample from 7.53 to 8.63. By reducing the average tokens per turn, more conversational context can be accommodated within a single training instance, broadening the model’s effective receptive field. Second, on inference performance, we report the end-to-end dialogue mod- eling results and average inference latency on the MultiWOZ 2.1 test set. As shown in Table 15, applying schema management provides ESAinsTOD with a notable reduction in inference latency (from 40.59 ms to 30.65 ms), corre- sponding to a 1.32x speedup. This is primarily because eliminating repeated schema definitions reduces the prefilling latency during inference. Crucially, this efficiency gain is achieved with no degradation in task performance metrics, as evidenced by a marginal difference of 0.12 in combined scores between configurations with and without schema management. This result further underscores the robustness of ESAinsTOD in handling contextual redundancy. Moreover, we compare our ESAinsTOD with two strong baselines, PPTOD and UBAR. Although our method incurs a slightly higher latency Springer Nature 2021 L A T E X template 36A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling Table 16: Failure cases from ESAinsTOD on BANKING77 test set InputGolden Prediction Similar Intent Group: carddeliveryestimate (cde), cardarrival (ca) Error Rate: 14/80=17.50% User: I am waiting for my card to arrive. cde ca User: How long will it take to arrive? cde ca User: How long does a card delivery take? ca cde Similar Intent Group: compromisedcard (c), cardpaymentnotrecognised (cpnr), directdebitpaymentnotrecognised (ddpnr) Error Rate: 10/120=8.33% User: I see random purchases to my account, was it hacked? c cpnr User: Somebody used my card to make a purchase cpnr c User: I have an unauthorized transaction on my statement cpnr ddpnr User: I was charged on my account that shouldn’t be there. ddpnr cpnr User: I am concerned about the security in my account and would like to make a dispute. ddpnr c Similar Intent Group: declinedcardpayment (dcp), declinedtransfer (dt), failedtransfer (ft) Error Rate: 9/118=7.63% User: How can I fix my card, it got declined twice. dt dcp User: I can’t transfer money from my account. dt ft User: My transfer did not go through. ft dt Similar Intent Group: toppingupbycard (tubc), transferintoaccount (tic) Error Rate: 4/79=5.06% User: How can someone add money to my account? tubc tic User: Can I add funds to the card directly from my bank account? tic tubc User: How do I top up my card? tic tubc The reported error rates exclusively account for classification errors within intent groups. (30.65 ms) than PPTOD (20.94 ms), it delivers substantial improvements in both dialogue state tracking and task-oriented dialogue modeling. Specifically, our ESAinsTOD yields absolute gains of 5.21% in Joint Goal Accuracy (JGA) and 3.53 points in the combined score. This result highlights that our pro- posed framework strikes an effective balance between achieving state-of-the-art performance and maintaining practical runtime efficiency. 5.4 Case study To provide deeper qualitative insights into the characteristics, strengths, and weaknesses of our proposed framework, we present a detailed case study across two primary tasks: language understanding (specifically intent classification) and end-to-end (E2E) dialogue modeling. 5.4.1 Analysis of language understanding failures This subsection investigates the classification errors generated by ESAin- sTOD, with its backbone being the Llama 2 7B model, in the BANKING77 Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling37 test set. As illustrated by the failure cases in Table 16, these errors mainly occur between semantically similar intents, such as card deliveryestimate and cardarrival. The primary challenge stems from the inherent design of the original intent schema: the reliance on abbreviated intent labels that lack explicit natural language descriptions. This deficiency introduces ambiguity that makes it dif- ficult for the model to differentiate between similar intents, essentially forcing the model to rely excessively on its internal memorization capabilities rather than generalizable understanding. This characteristic of these datasets favors discriminative and classification-based methods but hinders effective general- ization when training on mixed datasets. Ultimately, it restricts the flexible and effective scaling of our framework to unseen intent labels. These specific failure cases explain the results reported in Table 6, where the Llama 2 7B (FT) model fine-tuned solely on a single dataset achieves scores equal to or higher than the multi-dataset learning paradigm employed by ESAinsTOD. In other words, ESAinsTOD struggles to summarize common semantic features among diverse datasets and learn generalizable understand- ing. We posit that integrating explicit natural language descriptions and rich illustrative examples directly into the schema definition can effectively resolve the ambiguity between similar intents, thereby significantly enhancing the generalization of our framework in language understanding. 5.4.2 Analysis of end-to-end dialogue modeling superiority This analysis focuses on the E2E dialogue modeling capabilities of ESAin- sTOD in comparison to the PPTOD baseline, both of which are built on the Qwen2.5-3B-Instruct model. Table 17 presents comparative cases on the MultiWOZ 2.1 test set Error Type: Incorrect Dialogue State. In this category, the baseline PPTOD model exhibits significant errors in dialogue understanding, resulting in no matching items in subsequent database queries. • Dialogue MUL2060 (Turn 4): The user explicitly requests “a place that doesn’t need to have free parking ”, implying parking=dontcare as it’s not a hard constraint. However, PPTOD incorrectly updates the dialogue state to parking=no. • Dialogue PMUL1920 (Turn 5): PPTOD fails to accurately track the user’s preference shift, retaining the area=centre constraint despite the user indi- cating an intention to accept “any area is fine”, which should result in area=dontcare. In contrast, our ESAinsTOD model correctly predicts the desired dia- logue states throughout both dialogues, demonstrating superior state tracking competence which is essential for successful task completion. Error Type: Inappropriate Policy Planning. The failure of PPTOD in ‘MUL1376’ underscores a clear advantage for ESAinsTOD in strategic Springer Nature 2021 L A T E X template 38A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling Table 17: Case study on E2E dialog modeling from MultiWOZ 2.1 test set Error Type: Incorrect Dialogue State Dialogue: MUL2060 Turn 4 User: Great I also need to find a place to stay that doesn’t need to have free parking and is in the cheap price range. Oracle [State] hotel: parking=dontcare, pricerange=cheap [DB] hotel: 10 Ours [State] hotel: parking=dontcare, pricerange=cheap [DB] hotel: 10 pptod [State] hotel: parking=no, pricerange=cheap [DB] hotel: 0 Dialogue: PMUL1920 Turn 5 User: it should be in the town center [State] ... [DB] hotel: 0 [Delex] ... Turn 6 User: ok then i will go ahead and take any of the moderately priced ones with free wifi. please recommend one. any area is fine. Oracle [State] hotel: area=dontcare, internet=yes, pricerange=moderate [DB] hotel: 12 Ours [State] hotel: area=dontcare, internet=yes, pricerange=moderate [DB] hotel: 12 pptod [State] hotel: area=centre, internet=yes, pricerange=moderate [DB] hotel: 0 Error Type: Inappropriate Policy Planning Dialogue: MUL0947 Turn 2 User: Perhaps the center of town. Can you provide me with information regard- ing the entrance fee and a telephone number too? [State] attraction: area=centre, type=museum [DB] attraction: 11 Oracle [Delex] The [value name] is located in the [valuearea] is [valueprice] to enter and the phone number is [value phone]. Ours [Delex] [valuename] is [valueprice] and their phone number is [valuephone]. pptod [Delex] There are [valuechoice] [valuetype] in the [valuearea]. Dialogue: MUL1376 Turn 4 User: I would be interested in a cheap guesthouse if that is available. Will you check for me? [State] hotel:pricerange=cheap, stars=4, type=guest house [DB] hotel: 7 [Delex] There are [value choice] available that match what you’re looking for, do you have any preference for what area you want to stay in? Turn 5 User: I need free internet and I need their postcode. [State] hotel: internet=yes, pricerange=cheap, stars=4, type=guest house [DB] hotel: 7 Oracle [Delex] [valuename] has internet. It is located in [valuepostcode] postal code. Ours [Delex] [valuename] is available. The postcode is [valuepostcode]. pptod [Delex] There are [valuechoice] [valuetype] that meet your needs. Is there a particular area of town you would like to stay in? Error Type: Missing Requested Information Dialogue: MUL2405 Turn 1 User: Hi. I am having trouble finding a place to eat called prezzo. Can you locate it for me and tell me more? [State] restaurant: name=prezzo [DB] restaurant: 1 Oracle [Delex] [valuename] is located at [valueaddress], in the [valuearea]. It’s an [valuefood] restaurant. [valueprice] price range. Ours [Delex] [valuename] is an [valueprice] [valuefood] restaurant in the [valuearea] part of town. The address is [valueaddress] and the phone number is [valuephone]. pptod [Delex] [valuename] is an [valueprice] [valuefood] restaurant in the [valuearea]. Detailed database results and system actions are omitted for brevity. decision-making. Both methods initially choose to inquire about the user’s area preference in Turn 4. However, in Turn 5, PPTOD executes a redundant dialogue act, leading to an inappropriate response and a failed interaction. Conversely, ESAinsTOD promptly pivots its policy toward recommending an available hotel in the subsequent turn, enabling a smooth task completion. This validates the superior ability to plan policies embedded within our framework. Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling39 Error Type: Missing Requested Information. A third common fail- ure mode for PPTOD involves omission of critical information. In response to the user’s request in Turn 1 of ‘MUL2405’, PPTOD neglects to provide the requested address information concerning the restaurant. ESAinsTOD, however, retrieves and presents all necessary entity attributes, ensuring task success and complete user satisfaction. Collectively, these cases validate the superior dialogue understanding, state tracking, and policy planning abilities of ESAinsTOD compared to the baseline method. 6 Limitations and future directions In this section, we discuss the limitations of our proposed framework and outline promising directions for future research. Design trade-offs in modeling language understanding. Natural Lan- guage Understanding (NLU), encompassing tasks like intent detection and dialogue state tracking, is a cornerstone of task-oriented dialogue (TOD) sys- tems. Our framework models these tasks within a schema-aware generative paradigm. Compared to non-scalable, classification-based approaches that are tailored for specific tasks, this generative formulation may not achieve the same peak performance on certain NLU benchmarks. However, the primary advantage of our approach lies in its flexibility and scalability; the schema- aware generative model can be extended to known or unseen scenarios with remarkable adaptability, often without the need for complete retraining. Framework complexity vs. Simple LLM fine-tuning. A pertinent ques- tion is whether simply fine-tuning a large language model (LLM) for each specific scenario offers a better trade-off between simplicity and performance. Indeed, fine-tuning an LLM on a single data-rich domain is a straightforward method to achieve considerable results. In contrast, our unified framework, ESAinsTOD, requires more upfront efforts in curating schema definitions and organizing data from multiple scenarios. While this may yield only modest performance gains in single-domain settings, its strength becomes evident in practical applications characterized by diverse dialogue scenarios. The alignment mechanisms in ESAinsTOD are designed not only to mitigate interference between different scenarios but also to facilitate knowledge transfer and utilization among similar ones. In other words, our framework demonstrates superior generalization in low- resource settings. In summary, while direct fine-tuning is suitable for isolated, data-abundant tasks, ESAinsTOD presents a more balanced and robust solu- tion for the majority of real-world applications involving multiple, varied, or low-resource domains. Scalability constraints. As discussed in Section 5.2.2, ESAinsTOD sig- nificantly outperforms baseline methods in low-resource and unseen scenarios, Springer Nature 2021 L A T E X template 40A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling demonstrating strong generalization capabilities in end-to-end dialogue model- ing. The proposed alignment mechanisms contribute to this enhanced extensi- bility to unseen domains without retraining. Nevertheless, there remains room for improvement. For instance, as shown in Table 10, the performance on cer- tain tasks in unseen domains, such as the 26.9% joint goal accuracy for dialogue state tracking, has not yet reached the desired level. To address the perfor- mance gaps in unseen domains, a primary focus of our future work will be on data-centric approaches. We believe that incorporating more diverse datasets and developing sophisticated data augmentation techniques can substantially improve the model’s robustness and generalization capabilities. Additionally, our framework’s ability to generalize is constrained when faced with unseen intent labels that are highly abbreviated and lack clear semantic definitions. The absence of natural language explanations for these labels forces the model to rely heavily on memorization, hindering its flex- ibility and effectiveness. To address this, future improvements could involve exploring the integration of rich, explicit natural language descriptions into the schema. Such an enhancement could resolve ambiguity among similar intents and reduce the model’s reliance on memorization. Ultimately, this may enable our framework to effectively unify a wide range of intent classification datasets, paving the way for a more universal dialogue model. Real-world applicability assessment. The high cost of annotation means that most real-world TOD datasets are limited in scale and scenario diversity, which in turn constrains the performance evaluation of methods in practical applications. Although some recent work has used LLMs to synthesize training data in various domains (Li et al, 2022; Niu et al, 2024), a significant quality gap persists between the synthetic and real data. Consequently, synthetic data is typically used only for the initial pre-training phase, followed by fine-tuning on real data. Therefore, synthesizing high-quality, diverse data to improve or dynamically evaluate TOD models remains a promising avenue for future work. Equally important is the development of more flexible and fine-grained evaluation methodologies (Acikgoz et al, 2025; Chalamalasetti et al, 2025). Current automatic metrics often fail to capture critical intermediate errors that can occur during multi-turn user-agent interactions. Future work should aim to design novel evaluation techniques that complement traditional metrics, providing a more holistic and realistic assessment of the applicability of a dialogue system in the real world. 7 Related work 7.1 Pre-trained conversation models (PCMs) Transformer-based (Vaswani et al, 2017) pre-trained language models (PLMs) (Devlin et al, 2019; Yang et al, 2019; Raffel et al, 2020; Dong et al, 2019), which master the deep contextualized understanding of natural language texts, have garnered increasing attention and achieved remarkable success in Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling41 a variety of natural language processing (NLP) tasks, such as text classifi- cation, sequence labeling, and text generation. However, many prior studies have highlighted the intrinsic differences in linguistic patterns between human conversations and the general texts used for pre-training PLMs (Zhang et al, 2020d). PLMs directly fine-tuned on downstream dialog tasks often struggle to capture conversational linguistic features, failing to achieve the significant improvements observed in other NLP tasks. To address this issue, further pre-training PLMs on large-scale dialog corpora with well-designed dialog objectives has been proposed to build pre-trained conversation models (PCMs) that enrich conversational knowledge (Mehri et al, 2020; Zhang et al, 2020d; He et al, 2022b). These PCMs offer superior initialization parameters for fine- tuning on downstream dialog tasks and can be broadly classified into two categories: those focusing on open-domain conversations and those tailored for task-oriented dialogs. DialoGPT (Zhang et al, 2020d) is a representative PCM based on the GPT- 2 (Radford et al, 2019) architecture, pre-trained on a large-scale dialog corpus collected from Reddit comment chains to enhance the relevance, informative- ness, and fluency of open-domain dialog responses. Blender (Roller et al, 2021) extended the scale of open-domain dialog data for pre-training, demonstrating powerful response generation capabilities. Furthermore, additional pre-training tasks have been employed to improve performance (Henderson et al, 2020), such as latent act recognition in PLATO (Bao et al, 2020) and response selection. Another majority of PCMs are dedicated to tackling various challenges in task-oriented dialog scenarios, such as multi-turn dialog understanding, dialog state tracking, and dialog policy planning. These PCMs can be fur- ther categorized into three types. The first type of PCMs aims to learn semantic representations that improve dialogue understanding performance by pre-training on conversational data without relying on any additional human annotations (Zeng et al, 2023b). For instance, ConvBERT (Mehri et al, 2020) further trained BERT using a masked language modeling objective, while TOD-BERT (Wu et al, 2020) additionally incorporated a response selection objective. DialogueBERT (Zhang et al, 2021) replaced the conventional masked language modeling in TOD-BERT with a masked utterance modeling objec- tive. The second type defines pre-training objectives based on the annotations of TOD subtasks, empowering PLMs with the necessary skills required to complete specific dialogue tasks. GALAXY (He et al, 2022c) explicitly cap- tured dialog policy in labeled and unlabeled dialogs through semi-supervised learning of dialog act prediction task. SPACE-2 (He et al, 2022a) transformed the semantic frame of each dialog turn into a semantic tree structure (STS) and established a contrastive learning objective based on semantic tree simi- larity. The pre-trained SPACE-2 achieved state-of-the-art performance across multiple dialog understanding tasks, including intent prediction, slot filling, semantic parsing, and dialog state tracking. PPTOD (Su et al, 2022) conducted multi-task dialog pre-training using labeled data from various TOD tasks. Springer Nature 2021 L A T E X template 42A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling The third type involves constructing PCMs that explicitly or implicitly model the task flows inherent in task-oriented dialog data in an end-to-end manner. SOLOIST (Peng et al, 2021) fine-tuned the GPT-2 model on two TOD datasets to sequentially generate the results of multiple TOD subtasks. SPACE-3 (Dong et al, 2019) maintained a task flow consisting of four modules within a single transformer and completed semi-supervised end-to-end pre-training by lever- aging only the semantic frame as an intermediate task annotation. The latter two types of methods bridge the gap between pre-training and downstream fine-tuning by directly modeling the relationship between dialog sessions and various task annotations. 7.2 Large Language Models (LLMs) The transition from PLMs to LLMs was characterized by a substantial increase in both model size and the scale of training data. Pre-training billions of parameters on trillions of tokens spanning diverse text genres (e.g., news, books, Wikipedia articles, programming codes) allows LLMs to better com- prehend the context and produce more coherent content through a simple next-token prediction objective, such as Llama series (Touvron et al, 2023), GPT-3 series (Brown et al, 2020; Ouyang et al, 2022), GLM series (Du et al, 2022; Zeng et al, 2023a), Qwen series (Yang et al, 2024), and so on. LLMs have revolutionized the implementation of various NLP applications, achieving remarkable success across a range of NLP tasks while demonstrating impressive few-shot and even zero-shot generalization capabilities. Instruction fine-tuning (IFT) (Wei et al, 2022), which involves fine-tuning a pre-trained LLM on a set of (instruction, input, output) triples, further enhances LLMs’ fine-grained understanding and execution capabilities for complex instructions, thus broad- ening their applicability to new tasks and scenarios. IFT has motivated researchers to reconsider TOD architectures. For instance, the AutoTOD (Xu et al, 2024) approach proposes a shift from complex modularity to a zero-shot autonomous agent. This model requires only a general-purpose instruction- following LLM (e.g., GPT-4) and simple task descriptions to autonomously decide all dialogue actions. Spec-TOD (Nguyen et al, 2025) incorporates explicit task instructions into the training framework to obtain a specialized instruction-tuned LLM. Spec-TOD also focuses on reducing the reliance on massive amounts of labeled data, particularly in low-resource scenarios. Our work is inspired by the success of IFT on LLMs and aims to explore the potential of empowering LLMs with superior performance in handling task-oriented dialog (TOD) workflows under regular and low-resource settings. Benefiting from the extraordinary text modeling ability of LLMs, compared with PCMs, LLMs have advantages in reducing the degree of catastrophic forgetting and generating structured contents for building TOD systems. Specifically, continually pre-training and then fine-tuning PLMs on dialog data leads to increased catastrophic forgetting, which limits their generalization performance on downstream dialog tasks. In contrast, directly performing IFT on LLMs can leverage annotated information from various TOD datasets to Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling43 build an efficient and generalizable task-oriented dialog system while preserv- ing much of the general knowledge acquired during pre-training. Additionally, LLMs excel at converting natural texts into structured forms (e.g., dictio- nary and list objects in Python 4 ), which aids in generating and parsing the results for multi-intent recognition, dialog state tracking, and other TOD tasks. This capability simplifies the design and implementation of TOD systems by facilitating the structured output crucial for these tasks. As LLMs increase the sophistication of TOD systems, traditional evalua- tion metrics prove insufficient. The TD-EVAL framework (Acikgoz et al, 2025) unifies evaluation by combining fine-grained turn-level analysis with holistic dialogue-level comparisons, demonstrating excellent alignment with human judgment. Furthermore, Baidya et al (2025) proposes a framework to quantify and analyze the discrepancy between AI agent and human expert behavior in complex TOD systems. This work identifies that the performance gap in zero- shot LLM agents is significantly correlated with the widening of the behavior gap, including discrepancies in dialog acts, tool usage, and knowledge utiliza- tion. Increasing task complexity highlights the need for improved behavioral alignment strategies. 8 Conclusion In this work, we introduce ESAinsTOD, an innovative instruction-tuning framework for end-to-end task-oriented dialog (TOD) modeling. Our approach moves beyond simply fine-tuning a powerful LLM backbone by introducing a structured methodology that enables a single model to seamlessly adapt to a variety of annotation schemas and execute diverse TOD workflows within a unified framework. To facilitate effective alignments of task instructions with corresponding workflows as well as dialogue schemas with task outputs, we integrate two novel alignment mechanisms during the construction and mod- eling of the TOD instruction fine-tuning data. Meanwhile, our framework manages dialogue interactions in a session-level end-to-end manner, where tasks such as language understanding, policy planning, and response genera- tion are sequentially executed for each turn, conditioned on both the dialog history and the outcomes of previous task flows. Our experimental results demonstrate that this approach is highly effective. While achieving competi- tive performance against specialized models, our primary contribution is not focused on state-of-the-art metrics alone. Instead, we have shown that the true strength of ESAinsTOD lies in its ability to systematically leverage the power of LLMs for complex real-world TOD challenges. Specifically, our in- depth analysis confirms that the proposed framework significantly enhances generalization capabilities, improves data utilization efficiency in low-resource settings, and strengthens robustness against error propagation in multi-turn dialog interactions. By structuring the learning process around schemas and 4 https://w.python.org/ Springer Nature 2021 L A T E X template 44A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling instructions, our work provides a new perspective for building robust and adaptable TOD systems in the era of LLMs. Acknowledgments We thank all anonymous reviewers for their constructive comments. This work was supported by the National Natural Science Foundation of China (NSFC) via grant 62306342, 62236004, 62206078 and 62441603, and Du Xiaoman (Beijing) Science Technology Co., Ltd. Data availability All datasets listed in Table 1 are publicly available. SGD: https://github.com/ google-research-datasets/dstc8-schema-guided-dialogue. Frames dataset is available at https://github.com/awslabs/pptod. BiToD: https://github.com/ HLTCHKUST/BiToD. STAR: https://github.com/RasaHQ/STAR. SNIPS: https://github.com/LeePleased/StackPropagation-SLU. Other datasets can be downloaded from https://github.com/AlibabaResearch/DAMO-ConvAI/ tree/main/space-3. Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling45 References Acikgoz EC, Guo C, Dey S, et al (2025) TD-EVAL: revisiting task-oriented dia- logue evaluation by combining turn-level precision with dialogue-level com- parisons. CoRR abs/2504.19982. https://doi.org/10.48550/ARXIV.2504. 19982, URL https://doi.org/10.48550/arXiv.2504.19982, https://arxiv.org/ abs/2504.19982 Baidya A, Das K, Gao X (2025) The behavior gap: Evaluating zero-shot LLM agents in complex task-oriented dialogs. In: Che W, Nabende J, Shutova E, et al (eds) Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025. Association for Compu- tational Linguistics, p 23455–23472, URL https://aclanthology.org/2025. findings-acl.1205/ Bao S, He H, Wang F, et al (2020) PLATO: pre-trained dialogue genera- tion model with discrete latent variable. In: Jurafsky D, Chai J, Schluter N, et al (eds) Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020. Asso- ciation for Computational Linguistics, p 85–96, https://doi.org/10.18653/ V1/2020.ACL-MAIN.9, URL https://doi.org/10.18653/v1/2020.acl-main.9 Brown TB, Mann B, Ryder N, et al (2020) Language models are few-shot learners. In: Larochelle H, Ranzato M, Hadsell R, et al (eds) Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, URL https://proceedings.neurips.c/paper/2020/hash/ 1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html Budzianowski P, Wen T, Tseng B, et al (2018) Multiwoz - A large-scale multi- domain wizard-of-oz dataset for task-oriented dialogue modelling. In: Riloff E, Chiang D, Hockenmaier J, et al (eds) Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018. Association for Computational Linguistics, p 5016–5026, URL https://aclanthology.org/D18-1547/ Byrne B, Krishnamoorthi K, Sankar C, et al (2019) Taskmaster-1: Toward a realistic and diverse dialog dataset. In: Inui K, Jiang J, Ng V, et al (eds) Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019. Association for Computational Linguistics, p 4515–4524, https://doi.org/ 10.18653/V1/D19-1459, URL https://doi.org/10.18653/v1/D19-1459 Casanueva I, Temˇcinas T, Gerz D, et al (2020) Efficient intent detection with dual sentence encoders. In: Wen TH, Celikyilmaz A, Yu Z, et al (eds) Proceedings of the 2nd Workshop on Natural Language Processing Springer Nature 2021 L A T E X template 46A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling for Conversational AI. Association for Computational Linguistics, Online, p 38–45, https://doi.org/10.18653/v1/2020.nlp4convai-1.5, URL https:// aclanthology.org/2020.nlp4convai-1.5 Castellucci G, Bellomaria V, Favalli A, et al (2019) Multi-lingual intent detec- tion and slot filling in a joint bert-based model. CoRR abs/1907.02884. URL http://arxiv.org/abs/1907.02884, https://arxiv.org/abs/1907.02884 Chalamalasetti K, Hakimov S, Schlangen D (2025) clem:todd: A framework for the systematic benchmarking of llm-based task-oriented dialogue system realisations. CoRR abs/2505.05445. https://doi.org/10.48550/ARXIV.2505. 05445, URL https://doi.org/10.48550/arXiv.2505.05445, https://arxiv.org/ abs/2505.05445 Chen W, Chen J, Qin P, et al (2019) Semantically conditioned dialog response generation via hierarchical disentangled self-attention. In: Korhonen A, Traum DR, M`arquez L (eds) Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers. Association for Computa- tional Linguistics, p 3696–3709, https://doi.org/10.18653/V1/P19-1360, URL https://doi.org/10.18653/v1/p19-1360 Chen W, Gong Y, Wang S, et al (2022) Dialogved: A pre-trained latent variable encoder-decoder model for dialog response generation. In: Muresan S, Nakov P, Villavicencio A (eds) Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022. Association for Computational Lin- guistics, p 4852–4864, https://doi.org/10.18653/V1/2022.ACL-LONG.333, URL https://doi.org/10.18653/v1/2022.acl-long.333 Coucke A, Saade A, Ball A, et al (2018) Snips voice platform: an embedded spoken language understanding system for private-by-design voice inter- faces. CoRR abs/1805.10190. URL http://arxiv.org/abs/1805.10190, https: //arxiv.org/abs/1805.10190 Dao T, Fu DY, Ermon S, et al (2022) Flashattention: Fast and memory- efficient exact attention with io-awareness. In: Koyejo S, Mohamed S, Agarwal A, et al (eds) Advances in Neural Information Process- ing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, URL http://papers.nips.c/paper files/paper/2022/ hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html Devlin J, Chang M, Lee K, et al (2019) BERT: pre-training of deep bidirec- tional transformers for language understanding. In: Burstein J, Doran C, Solorio T (eds) Proceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling47 Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers). Association for Computa- tional Linguistics, p 4171–4186, https://doi.org/10.18653/V1/N19-1423, URL https://doi.org/10.18653/v1/n19-1423 Dong L, Yang N, Wang W, et al (2019) Unified language model pre- training for natural language understanding and generation. In: Wallach HM, Larochelle H, Beygelzimer A, et al (eds) Advances in Neural Infor- mation Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, p 13042–13054, URL https://proceedings.neurips.c/paper/ 2019/hash/c20b2d9a50d5ac1f713f8b34d9aac5a-Abstract.html Du Z, Qian Y, Liu X, et al (2022) GLM: general language model pretraining with autoregressive blank infilling. In: Muresan S, Nakov P, Villavicencio A (eds) Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022. Association for Computational Linguistics, p 320–335, https://doi.org/10.18653/V1/2022.ACL-LONG.26, URL https:// doi.org/10.18653/v1/2022.acl-long.26 Eric M, Krishnan L, Charette F, et al (2017) Key-value retrieval networks for task-oriented dialogue. In: Jokinen K, Stede M, DeVault D, et al (eds) Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, Saarbr ̈ucken, Germany, August 15-17, 2017. Association for Computational Linguistics, p 37–49, https://doi.org/10.18653/V1/W17-5506, URL https: //doi.org/10.18653/v1/w17-5506 Eric M, Goel R, Paul S, et al (2020) Multiwoz 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines. In: Cal- zolari N, B ́echet F, Blache P, et al (eds) Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020. European Language Resources Association, p 422–428, URL https://aclanthology.org/2020.lrec-1.53/ Feng Y, Wang Y, Li H (2021) A sequence-to-sequence approach to dia- logue state tracking. In: Zong C, Xia F, Li W, et al (eds) Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021. Association for Computational Linguistics, p 1714–1725, https://doi.org/10.18653/V1/2021.ACL-LONG.135, URL https://doi.org/ 10.18653/v1/2021.acl-long.135 Gao J, Galley M, Li L (2019) Neural approaches to conversational AI. Found Trends Inf Retr 13(2-3):127–298. https://doi.org/10.1561/1500000074, URL Springer Nature 2021 L A T E X template 48A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling https://doi.org/10.1561/1500000074 Goo C, Gao G, Hsu Y, et al (2018) Slot-gated modeling for joint slot filling and intent prediction. In: Walker MA, Ji H, Stent A (eds) Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers). Association for Computational Linguistics, p 753–757, https://doi.org/10. 18653/V1/N18-2118, URL https://doi.org/10.18653/v1/n18-2118 Guo J, Shuang K, Li J, et al (2021) Dual slot selector via local reliability ver- ification for dialogue state tracking. In: Zong C, Xia F, Li W, et al (eds) Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Vir- tual Event, August 1-6, 2021. Association for Computational Linguistics, p 139–151, https://doi.org/10.18653/V1/2021.ACL-LONG.12, URL https: //doi.org/10.18653/v1/2021.acl-long.12 Ham D, Lee J, Jang Y, et al (2020) End-to-end neural pipeline for goal-oriented dialogue systems using GPT-2. In: Jurafsky D, Chai J, Schluter N, et al (eds) Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, ACL 2020, Online, July 5-10, 2020. Association for Computational Linguistics, p 583–592, https://doi.org/10.18653/V1/2020. ACL-MAIN.54, URL https://doi.org/10.18653/v1/2020.acl-main.54 He W, Dai Y, Hui B, et al (2022a) SPACE-2: tree-structured semi-supervised contrastive pre-training for task-oriented dialog understanding. In: Calzolari N, Huang C, Kim H, et al (eds) Proceedings of the 29th International Con- ference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022. International Committee on Computational Linguistics, p 553–569, URL https://aclanthology.org/2022.coling-1.46 He W, Dai Y, Yang M, et al (2022b) Unified dialog model pre-training for task-oriented dialog understanding and generation. In: Amig ́o E, Castells P, Gonzalo J, et al (eds) SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022. ACM, p 187–200, https://doi.org/10.1145/ 3477495.3532069, URL https://doi.org/10.1145/3477495.3532069 He W, Dai Y, Zheng Y, et al (2022c) GALAXY: A generative pre-trained model for task-oriented dialog with semi-supervised learning and explicit policy injection. In: Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Arti- ficial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling49 - March 1, 2022. AAAI Press, p 10749–10757, https://doi.org/10.1609/ AAAI.V36I10.21320, URL https://doi.org/10.1609/aaai.v36i10.21320 Heck M, van Niekerk C, Lubis N, et al (2020) Trippy: A triple copy strategy for value independent neural dialog state tracking. In: Pietquin O, Muresan S, Chen V, et al (eds) Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGdial 2020, 1st virtual meeting, July 1-3, 2020. Association for Computational Linguistics, p 35–44, URL https://aclanthology.org/2020.sigdial-1.4/ Henderson M, Casanueva I, Mrksic N, et al (2020) Convert: Efficient and accurate conversational representations from transformers. In: Cohn T, He Y, Liu Y (eds) Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, Findings of ACL, vol EMNLP 2020. Association for Computational Linguistics, p 2161–2174, https://doi.org/10.18653/V1/2020.FINDINGS-EMNLP.196, URL https:// doi.org/10.18653/v1/2020.findings-emnlp.196 Hosseini-Asl E, McCann B, Wu C, et al (2020) A simple language model for task-oriented dialogue. In: Larochelle H, Ranzato M, Hadsell R, et al (eds) Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, URL https://proceedings.neurips.c/paper/2020/hash/ e946209592563be0f01c844ab2170f0c-Abstract.html Hu EJ, Shen Y, Wallis P, et al (2022) Lora: Low-rank adaptation of large language models. In: The Tenth International Conference on Learning Rep- resentations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, URL https://openreview.net/forum?id=nZeVKeeFYf9 Jin X, Lei W, Ren Z, et al (2018) Explicit state tracking with semi- supervisionfor neural dialogue generation. In: Cuzzocrea A, Allan J, Paton NW, et al (eds) Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM 2018, Torino, Italy, Octo- ber 22-26, 2018. ACM, p 1403–1412, https://doi.org/10.1145/3269206. 3271683, URL https://doi.org/10.1145/3269206.3271683 Kim S, Yang S, Kim G, et al (2020) Efficient dialogue state tracking by selectively overwriting memory. In: Jurafsky D, Chai J, Schluter N, et al (eds) Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, ACL 2020, Online, July 5-10, 2020. Association for Computational Linguistics, p 567–582, https://doi.org/10.18653/V1/2020. ACL-MAIN.53, URL https://doi.org/10.18653/v1/2020.acl-main.53 Kwon W, Li Z, Zhuang S, et al (2023) Efficient memory management for large language model serving with pagedattention. In: Flinn J, Seltzer MI, Druschel P, et al (eds) Proceedings of the 29th Symposium on Operating Springer Nature 2021 L A T E X template 50A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023. ACM, p 611–626, https://doi.org/10.1145/3600006.3613165, URL https: //doi.org/10.1145/3600006.3613165 Larson S, Mahendran A, Peper J, et al (2019) An evaluation dataset for intent classification and out-of-scope prediction. In: Inui K, Jiang J, Ng V, et al (eds) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019. Association for Computational Linguistics, p 1311–1316, https://doi.org/10.18653/V1/D19-1131, URL https://doi.org/ 10.18653/v1/D19-1131 Lee C, Cheng H, Ostendorf M (2021) Dialogue state tracking with a lan- guage model using schema-driven prompting. In: Moens M, Huang X, Specia L, et al (eds) Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021. Association for Com- putational Linguistics, p 4937–4949, https://doi.org/10.18653/V1/2021. EMNLP-MAIN.404, URL https://doi.org/10.18653/v1/2021.emnlp-main. 404 Lee Y (2021) Improving end-to-end task-oriented dialog system with A sim- ple auxiliary task. In: Moens M, Huang X, Specia L, et al (eds) Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021. Association for Computational Linguistics, p 1296–1303, https://doi.org/10.18653/ V1/2021.FINDINGS-EMNLP.112, URL https://doi.org/10.18653/v1/2021. findings-emnlp.112 Lei W, Jin X, Kan M, et al (2018) Sequicity: Simplifying task-oriented dia- logue systems with single sequence-to-sequence architectures. In: Gurevych I, Miyao Y (eds) Proceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers. Association for Computational Lin- guistics, p 1437–1447, https://doi.org/10.18653/V1/P18-1133, URL https: //aclanthology.org/P18-1133/ Li Z, Chen W, Li S, et al (2022) Controllable dialogue simulation with in-context learning. In: Findings of the Association for Computa- tional Linguistics: EMNLP 2022. ACL, p 4330–4347, https://doi.org/ 10.18653/v1/2022.findings-emnlp.318, URL https://aclanthology.org/2022. findings-emnlp.318/ Liang W, Tian Y, Chen C, et al (2020) MOSS: end-to-end dialog system frame- work with modular supervision. In: The Thirty-Fourth AAAI Conference Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling51 on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Appli- cations of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020. AAAI Press, p 8327–8335, https://doi.org/10.1609/AAAI.V34I05.6349, URL https://doi.org/10.1609/ aaai.v34i05.6349 Lin Z, Madotto A, Winata GI, et al (2020) Mintl: Minimalist transfer learn- ing for task-oriented dialogue systems. In: Webber B, Cohn T, He Y, et al (eds) Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing, EMNLP 2020, Online, November 16-20, 2020. Association for Computational Linguistics, p 3391–3405, https://doi.org/ 10.18653/V1/2020.EMNLP-MAIN.273, URL https://doi.org/10.18653/v1/ 2020.emnlp-main.273 Lin Z, Madotto A, Winata GI, et al (2021) Bitod: A bilingual multi- domain dataset for task-oriented dialogue modeling. In: Vanschoren J, Yeung S (eds) Proceedings of the Neural Information Pro- cessing Systems Track on Datasets and Benchmarks 1, NeurIPS DatasetsandBenchmarks2021,December2021,virtual,URL https://datasets-benchmarks-proceedings.neurips.c/paper/2021/hash/ 6364d3f0f495b6ab9dcf8d3b5c6e0b01-Abstract-round1.html Liu Q, Yu L, Rimell L, et al (2021) Pretraining the noisy chan- nel model for task-oriented dialogue. Trans Assoc Comput Linguistics 9:657–674. https://doi.org/10.1162/TACL A00390, URL https://doi.org/ 10.1162/tacla00390 Liu X, Eshghi A, Swietojanski P, et al (2019) Benchmarking natural lan- guage understanding services for building conversational agents. In: Marchi E, Siniscalchi SM, Cumani S, et al (eds) Increasing Naturalness and Flex- ibility in Spoken Dialogue Interaction - 10th International Workshop on Spoken Dialogue Systems, IWSDS 2019, Syracuse, Sicily, Italy, 24-26 April 2019, Lecture Notes in Electrical Engineering, vol 714. Springer, p 165– 183, https://doi.org/10.1007/978-981-15-9323-915, URL https://doi.org/ 10.1007/978-981-15-9323-915 Loshchilov I, Hutter F (2019) Decoupled weight decay regularization. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, URL https: //openreview.net/forum?id=Bkg6RiCqY7 Mehri S, Eric M (2021) Example-driven intent prediction with observers. In: Toutanova K, Rumshisky A, Zettlemoyer L, et al (eds) Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT Springer Nature 2021 L A T E X template 52A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling 2021, Online, June 6-11, 2021. Association for Computational Linguistics, p 2979–2992, https://doi.org/10.18653/V1/2021.NAACL-MAIN.237, URL https://doi.org/10.18653/v1/2021.naacl-main.237 Mehri S, Srinivasan T, Esk ́enazi M (2019) Structured fusion networks for dialog. In: Nakamura S, Gasic M, Zuckerman I, et al (eds) Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, SIGdial 2019, Stockholm, Sweden, September 11-13, 2019. Association for Compu- tational Linguistics, p 165–177, https://doi.org/10.18653/V1/W19-5921, URL https://doi.org/10.18653/v1/W19-5921 Mehri S, Eric M, Hakkani-T ̈ur D (2020) Dialoglue: A natural language under- standing benchmark for task-oriented dialogue. CoRR abs/2009.13570. URL https://arxiv.org/abs/2009.13570, https://arxiv.org/abs/2009.13570 Mrksic N, S ́eaghdha D ́ O, Wen T, et al (2017) Neural belief tracker: Data- driven dialogue state tracking. In: Barzilay R, Kan M (eds) Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers. Association for Computational Linguistics, p 1777–1788, https://doi.org/ 10.18653/V1/P17-1163, URL https://doi.org/10.18653/v1/P17-1163 Nguyen Q, Nguyen Q, Pham H, et al (2025) Spec-tod: A specialized instruction-tuned LLM framework for efficient task-oriented dialogue systems. CoRR abs/2507.04841. https://doi.org/10.48550/ARXIV.2507. 04841, URL https://doi.org/10.48550/arXiv.2507.04841, https://arxiv.org/ abs/2507.04841 Niu C, Wang X, Cheng X, et al (2024) Enhancing dialogue state tracking mod- els through LLM-backed user-agents simulation. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ACL, p 8724–8741, https://doi.org/10.18653/v1/2024. acl-long.473, URL https://aclanthology.org/2024.acl-long.473/ Ouyang L, Wu J, Jiang X, et al (2022) Training language models to follow instructions with human feedback. In: Koyejo S, Mohamed S, Agarwal A, et al (eds) Advances in Neural Information Process- ing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, URL http://papers.nips.c/paper files/paper/2022/ hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html Papineni K, Roukos S, Ward T, et al (2002) Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA. ACL, p 311–318, https://doi.org/10.3115/1073083. 1073135, URL https://aclanthology.org/P02-1040/ Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling53 Paszke A, Gross S, Massa F, et al (2019) Pytorch: An imperative style, high-performance deep learning library. In: Wallach HM, Larochelle H, Beygelzimer A, et al (eds) Advances in Neural Information Process- ing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, p 8024–8035, URL https://proceedings.neurips.c/paper/2019/ hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html Peng B, Li C, Li J, et al (2021) SOLOIST: building task bots at scale with transfer learning and machine teaching. Trans Assoc Comput Linguistics 9:907–824. https://doi.org/10.1162/TACL A00399, URL https://doi.org/ 10.1162/tacla00399 Qin L, Che W, Li Y, et al (2019) A stack-propagation framework with token-level intent detection for spoken language understanding. In: Inui K, Jiang J, Ng V, et al (eds) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019. Association for Computa- tional Linguistics, p 2078–2087, https://doi.org/10.18653/V1/D19-1214, URL https://doi.org/10.18653/v1/D19-1214 Qin L, Liu T, Che W, et al (2021a) A co-interactive transformer for joint slot filling and intent detection. In: IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021, Toronto, ON, Canada, June 6- 11, 2021. IEEE, p 8193–8197, https://doi.org/10.1109/ICASSP39728.2021. 9414110, URL https://doi.org/10.1109/ICASSP39728.2021.9414110 Qin L, Xie T, Che W, et al (2021b) A survey on spoken language under- standing: Recent advances and new frontiers. In: Zhou Z (ed) Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021. ijcai.org, p 4577–4584, https://doi.org/10.24963/IJCAI.2021/622, URL https://doi. org/10.24963/ijcai.2021/622 Radford A, Wu J, Child R, et al (2019) Language models are unsupervised multitask learners. OpenAI blog 1(8):9 Raffel C, Shazeer N, Roberts A, et al (2020) Exploring the limits of transfer learning with a unified text-to-text transformer. J Mach Learn Res 21:140:1– 140:67. URL http://jmlr.org/papers/v21/20-074.html Rasley J, Rajbhandari S, Ruwase O, et al (2020) Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. In: Gupta R, Liu Y, Tang J, et al (eds) KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020. ACM, p 3505–3506, https://doi.org/10.1145/ Springer Nature 2021 L A T E X template 54A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling 3394486.3406703, URL https://doi.org/10.1145/3394486.3406703 Rastogi A, Zang X, Sunkara S, et al (2020) Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In: The Thirty- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty- Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artifi- cial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020. AAAI Press, p 8689–8696, https://doi.org/10.1609/AAAI.V34I05.6394, URL https://doi.org/10.1609/aaai.v34i05.6394 Roller S, Dinan E, Goyal N, et al (2021) Recipes for building an open-domain chatbot. In: Merlo P, Tiedemann J, Tsarfaty R (eds) Proceedings of the 16th Conference of the European Chapter of the Association for Computa- tional Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021. Association for Computational Linguistics, p 300–325, https://doi.org/10. 18653/V1/2021.EACL-MAIN.24, URL https://doi.org/10.18653/v1/2021. eacl-main.24 Shu L, Molino P, Namazifar M, et al (2019) Flexibly-structured model for task-oriented dialogues. In: Nakamura S, Gasic M, Zuckerman I, et al (eds) Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dia- logue, SIGdial 2019, Stockholm, Sweden, September 11-13, 2019. Association for Computational Linguistics, p 178–187, https://doi.org/10.18653/V1/ W19-5922, URL https://doi.org/10.18653/v1/W19-5922 Su Y, Shu L, Mansimov E, et al (2022) Multi-task pre-training for plug-and- play task-oriented dialogue system. In: Muresan S, Nakov P, Villavicencio A (eds) Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022. Association for Computational Linguistics, p 4661–4676, https://doi.org/10.18653/V1/2022.ACL-LONG.319, URL https://doi.org/ 10.18653/v1/2022.acl-long.319 Teng D, Lu C, Xu X, et al (2024) Pro-han: A heterogeneous graph atten- tion network for profile-based spoken language understanding. CoRR abs/2402.03900. https://doi.org/10.48550/ARXIV.2402.03900, URL https: //doi.org/10.48550/arXiv.2402.03900, https://arxiv.org/abs/2402.03900 Touvron H, Martin L, Stone K, et al (2023) Llama 2: Open founda- tion and fine-tuned chat models. CoRR abs/2307.09288. https://doi.org/ 10.48550/ARXIV.2307.09288, URL https://doi.org/10.48550/arXiv.2307. 09288, https://arxiv.org/abs/2307.09288 Tur G, De Mori R (2011) Spoken language understanding: Systems for extracting semantic information from speech. John Wiley & Sons Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling55 Vaswani A, Shazeer N, Parmar N, et al (2017) Attention is all you need. In: Guyon I, von Luxburg U, Bengio S, et al (eds) Advances in Neu- ral Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, p 5998–6008, URL https://proceedings.neurips.c/paper/2017/ hash/3f5e243547dee91fbd053c1c4a845a-Abstract.html Vulic I, Su P, Coope S, et al (2021) Convfit: Conversational fine-tuning of pretrained language models. In: Moens M, Huang X, Specia L, et al (eds) Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021. Association for Computational Linguistics, p 1151–1168, https://doi.org/10.18653/V1/2021.EMNLP-MAIN.88, URL https://doi.org/10.18653/v1/2021.emnlp-main.88 Wang K, Tian J, Wang R, et al (2020) Multi-domain dialogue acts and response co-generation. In: Jurafsky D, Chai J, Schluter N, et al (eds) Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020. Association for Computational Lin- guistics, p 7125–7134, https://doi.org/10.18653/V1/2020.ACL-MAIN.638, URL https://doi.org/10.18653/v1/2020.acl-main.638 Wei J, Bosma M, Zhao VY, et al (2022) Finetuned language models are zero-shot learners. In: The Tenth International Conference on Learning Rep- resentations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, URL https://openreview.net/forum?id=gEZrGCozdqR Wen T, Vandyke D, Mrksic N, et al (2017) A network-based end-to-end train- able task-oriented dialogue system. In: Lapata M, Blunsom P, Koller A (eds) Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume 1: Long Papers. Association for Computational Lin- guistics, p 438–449, https://doi.org/10.18653/V1/E17-1042, URL https: //doi.org/10.18653/v1/e17-1042 Wolf T, Debut L, Sanh V, et al (2020) Transformers: State-of-the-art nat- ural language processing. In: Liu Q, Schlangen D (eds) Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations, EMNLP 2020 - Demos, Online, November 16-20, 2020. Association for Computational Linguistics, p 38–45, https: //doi.org/10.18653/V1/2020.EMNLP-DEMOS.6, URL https://doi.org/10. 18653/v1/2020.emnlp-demos.6 Wu C, Madotto A, Hosseini-Asl E, et al (2019) Transferable multi-domain state generator for task-oriented dialogue systems. In: Korhonen A, Traum DR, M`arquez L (eds) Proceedings of the 57th Conference of the Association Springer Nature 2021 L A T E X template 56A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers. Association for Computational Linguistics, p 808–819, https://doi.org/10.18653/V1/P19-1078, URL https://doi.org/ 10.18653/v1/p19-1078 Wu C, Hoi SCH, Socher R, et al (2020) TOD-BERT: pre-trained natural lan- guage understanding for task-oriented dialogue. In: Webber B, Cohn T, He Y, et al (eds) Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020. Association for Computational Linguistics, p 917–929, https://doi. org/10.18653/V1/2020.EMNLP-MAIN.66, URL https://doi.org/10.18653/ v1/2020.emnlp-main.66 Xu H, Mao X, Yang P, et al (2024) Rethinking task-oriented dialogue systems: From complex modularity to zero-shot autonomous agent. In: Ku L, Martins A, Srikumar V (eds) Proceedings of the 62nd Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024. Association for Computational Lin- guistics, p 2748–2763, https://doi.org/10.18653/V1/2024.ACL-LONG.152, URL https://doi.org/10.18653/v1/2024.acl-long.152 Yang A, Yang B, Zhang B, et al (2024) Qwen2.5 technical report. CoRR abs/2412.15115. https://doi.org/10.48550/ARXIV.2412.15115, URL https: //doi.org/10.48550/arXiv.2412.15115, https://arxiv.org/abs/2412.15115 Yang M, Ng S, Fu J (2023) Omnidialog: An omnipotent pre-training model for task-oriented dialogue system. CoRR abs/2312.16864. https://doi.org/ 10.48550/ARXIV.2312.16864, URL https://doi.org/10.48550/arXiv.2312. 16864, https://arxiv.org/abs/2312.16864 Yang Y, Li Y, Quan X (2021) UBAR: towards fully end-to-end task-oriented dialog system with GPT-2. In: Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applica- tions of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021. AAAI Press, p 14230–14238, https://doi.org/10.1609/ AAAI.V35I16.17674, URL https://doi.org/10.1609/aaai.v35i16.17674 Yang Z, Dai Z, Yang Y, et al (2019) Xlnet: Generalized autoregressive pretraining for language understanding. In: Wallach HM, Larochelle H, Beygelzimer A, et al (eds) Advances in Neural Information Process- ing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, p 5754–5764, URL https://proceedings.neurips.c/paper/2019/ hash/dc6a7e655d7e5840e66733e9e67c69-Abstract.html Springer Nature 2021 L A T E X template A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling57 Young SJ, Gasic M, Thomson B, et al (2013) Pomdp-based statistical spoken dialog systems: A review. Proc IEEE 101(5):1160–1179. https://doi.org/ 10.1109/JPROC.2012.2225812, URL https://doi.org/10.1109/JPROC.2012. 2225812 Zeng A, Liu X, Du Z, et al (2023a) GLM-130B: an open bilingual pre-trained model. In: The Eleventh International Conference on Learning Representa- tions, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, URL https://openreview.net/pdf?id=-Aw0rrrPUF Zeng W, He K, Wang Y, et al (2023b) Futuretod: Teaching future knowledge to pre-trained language model for task-oriented dialogue. In: Rogers A, Boyd- Graber JL, Okazaki N (eds) Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023. Association for Computational Linguistics, p 6532–6546, https://doi.org/10.18653/V1/2023.ACL-LONG. 360, URL https://doi.org/10.18653/v1/2023.acl-long.360 Zhang J, Hashimoto K, Wu C, et al (2020a) Find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking. In: Gurevych I, Apidianaki M, Faruqui M (eds) Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, *SEM@COLING 2020, Barcelona, Spain (Online), December 12-13, 2020. Association for Computational Linguistics, p 154–167, URL https://aclanthology.org/2020.starsem-1.17/ Zhang X, Peng B, Li K, et al (2023) SGP-TOD: building task bots effortlessly via schema-guided LLM prompting. In: Bouamor H, Pino J, Bali K (eds) Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023. Association for Computational Linguistics, p 13348–13369, https://doi.org/10.18653/V1/2023.FINDINGS-EMNLP. 891, URL https://doi.org/10.18653/v1/2023.findings-emnlp.891 Zhang Y, Ou Z, Hu M, et al (2020b) A probabilistic end-to-end task-oriented dialog model with latent belief states towards semi-supervised learning. In: Webber B, Cohn T, He Y, et al (eds) Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020. Association for Computational Linguistics, p 9207– 9219, https://doi.org/10.18653/V1/2020.EMNLP-MAIN.740, URL https:// doi.org/10.18653/v1/2020.emnlp-main.740 Zhang Y, Ou Z, Yu Z (2020c) Task-oriented dialog systems that consider mul- tiple appropriate responses under the same context. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020. AAAI Press, p Springer Nature 2021 L A T E X template 58A Unified End-to-End Schema-Aware Task-Oriented Dialog Modeling 9604–9611, https://doi.org/10.1609/AAAI.V34I05.6507, URL https://doi. org/10.1609/aaai.v34i05.6507 Zhang Y, Sun S, Galley M, et al (2020d) DIALOGPT : Large-scale gener- ative pre-training for conversational response generation. In: Celikyilmaz A, Wen T (eds) Proceedings of the 58th Annual Meeting of the Associ- ation for Computational Linguistics: System Demonstrations, ACL 2020, Online, July 5-10, 2020. Association for Computational Linguistics, p 270–278, https://doi.org/10.18653/V1/2020.ACL-DEMOS.30, URL https: //doi.org/10.18653/v1/2020.acl-demos.30 Zhang Z, Guo T, Chen M (2021) Dialoguebert: A self-supervised learning based dialogue pre-training encoder. In: Demartini G, Zuccon G, Culpep- per JS, et al (eds) CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Aus- tralia, November 1 - 5, 2021. ACM, p 3647–3651, https://doi.org/10.1145/ 3459637.3482085, URL https://doi.org/10.1145/3459637.3482085 Zhou J, Wu H, Lin Z, et al (2021) Dialogue state tracking with multi-level fusion of predicted dialogue states and conversations. In: Li H, Levow G, Yu Z, et al (eds) Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGdial 2021, Singapore and Online, July 29-31, 2021. Association for Computational Linguistics, p 228–238, URL https://aclanthology.org/2021.sigdial-1.24 Zhou L, Small K (2019) Multi-domain dialogue state tracking as dynamic knowledge graph enhanced question answering. CoRR abs/1911.06192. URL http://arxiv.org/abs/1911.06192, https://arxiv.org/abs/1911.06192