Paper deep dive
Candidate Attended Dialogue State Tracking Using BERT
Junyuan Zheng, Onkar Salvi, John Chan
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/20/2026, 4:08:47 AM
Summary
This paper proposes a scalable framework for multi-domain Dialogue State Tracking (DST) using a Candidate Attended approach based on the BERT model. The system leverages BERT to achieve zero-shot generalization, allowing it to adapt to new domains with little or no training data by utilizing natural language descriptions of intents and slots. The architecture includes a Slot Tagger for NLU and a Dialogue State Tracker comprising an Intent Classifier, Requested Slot Detector, and Candidate Classifier. Evaluated on the Schema-Guided Dialogue (SGD) dataset, the model demonstrates significant improvements in intent classification, requested slot detection, and joint goal accuracy compared to previous baselines, particularly in handling unseen domains.
Entities (11)
Relation Signals (8)
Candidate Attended Dialogue State Tracking → uses → BERT
confidence 98% · The proposed system leverages the pretrained BERT model to achieve zero-shot generalization
Candidate Attended Dialogue State Tracking → achieves → Zero-Shot Generalization
confidence 95% · The proposed system leverages the pretrained BERT model to achieve zero-shot generalization
Candidate Attended Dialogue State Tracking → evaluatedon → SGD Dataset
confidence 95% · The performance of our model is evaluated on recently released schema-based dialogue (SGD) dataset
Candidate Attended Dialogue State Tracking → includescomponent → Candidate Classifier
confidence 92% · The candidate classifier predicts the dialogue state updates by calculating the distribution over the candidate set.
Candidate Attended Dialogue State Tracking → includescomponent → Slot Tagger
confidence 92% · Our dialogue system consists of two major components: Natural Language Understanding (NLU) and Dialogue State Tracking (DST)... Slot tagger is the only model in the NLU component
Candidate Attended Dialogue State Tracking → includescomponent → Intent Classifier
confidence 92% · In DST module, intent classifier, requested slot detector and candidate classifier are trained jointly.
BERT → pretrainedon → large corpura
confidence 90% · these models are pretrained on large corpura
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Dialogue state tracking (DST) is one of the core components in task-oriented dialogue systems. At each turn in a conversation, DST estimates the user belief or dialogue state, which is used as input for downstream modules to predict system actions and generate responses. The increasingly popular dialogue system applications like Google Assistant, Siri and Alexa need to support a large number of services and APIs, resulting in growing attention to the scalability of such systems. Especially for some domains with little or no training data, the capability of transferring existing knowledge of other domains is highly desired. In this paper, we present a novel scalable framework for multi-domain dialogue state tracking. The proposed system leverages the pretrained BERT model to achieve zero-shot generalization, making it easy to quickly adapt to new domains without additional training. The performance of our model is evaluated on recently released schema-based dialogue (SGD) dataset, showing significant improvement compared to previous baseline.
Tags
Links
- Source: https://arxiv.org/abs/2607.16021v1
- Canonical: https://arxiv.org/abs/2607.16021v1
Trouble viewing inline? Open PDF directly →
Full Text
30,104 characters extracted from source content.
Expand or collapse full text
Candidate Attended Dialogue State Tracking Using BERT Junyuan Zheng, Onkar Salvi, John Chan OneConnect US Research Institute, New York, NY, 10019 zhengjunyuan616, onkar.salvi7, jchan.jc@gmail.com Abstract Dialogue state tracking (DST) is one of the core components in task-oriented dialogue systems. At each turn in a conver- sation, DST estimates the user belief or dialogue state, which is used as input for downstream modules to predict system actions and generate responses. The increasingly popular di- alogue system applications like Google Assistant, Siri and Alexa need to support a large number of services and APIs, resulting in growing attention to the scalability of such sys- tems. Especially for some domains with little or no train- ing data, the capability of transferring existing knowledge of other domains is highly desired. In this paper, we present a novel scalable framework for multi-domain dialogue state tracking. The proposed system leverages the pretrained BERT model to achieve zero-shot generalization, making it easy to quickly adapt to new domains without additional training. The performance of our model is evaluated on recently re- leased schema-based dialogue (SGD) dataset, showing sig- nificant improvement compared to previous baseline. 1Introduction Task-oriented dialogue systems are playing an important role in facilitating our daily life in the form of personal vir- tual assistants, customer service agents, website chat sys- tems, etc. These applications, by providing a conversational interface to different backends, help users easily accomplish tasks in a wide range of domains, such as restaurant reser- vation, flight booking, and music recommendation. While sequence-to-sequence learning (Sutskever, Vinyals, and Le 2014) has inspired lots of studies in building non-task- oriented dialogue systems, task-oriented systems still heav- ily rely on modularized pipeline. Typically, a task-oriented dialogue system has four components: a Natural Language Understanding (NLU) module semantically to parse the user utterance, a Dialogue State Tracking (DST) module to main- tain the dialogue states throughout the conversation, a Dia- logue Policy module to predict the next system action based on the current dialogue state, and a Natural Language Gen- eration module to convert system actions into natural lan- guage responses (Rudnicky et al. 1999; Zue et al. 2000; Chen et al. 2017). In recent years, the research community is paying signifi- cant attention to enhancing the scalability of dialogue sys- tem in order to quickly and easily add more functionali- ties. For popular commercial applications such as Google Assistant, Siri, or Alexa, the generalization capability is de- sired when developers want to integrate support for addi- tional services or APIs. Given limited data in an unseen service or a new domain, the dialogue systems are ex- pected to perform robustly without laborious data collec- tion and annotation. Motivated by this expectation, several studies have focused on zero-shot language understanding and scalable dialogue state tracking (Bapna et al. 2017; Rastogi, Hakkani-T ̈ ur, and Heck 2017; Ren et al. 2018; Shah et al. 2019; Chao and Lane 2019). Recently in the 8th Dialog System Technology Chal- lenges (Seokhwan Kim 2019), Rastogi et al. (2019) intro- duce the Schema-Guided Dialogue (SGD) dataset, which exceeds existing task-oriented corpura like MultiWOZ (Budzianowski et al. 2018) in scale. In the proposed shcema- guided paradigm, a service’s schema is defined by a set of in- tents and slots, where a natural language description is given for each intent and slot (as shown in Figure 1). Meanwhile, the emergence of pretrained models like BERT (Devlin et al. 2018) and XLNet (Yang et al. 2019) make it easier to utilize the natural language elements as inputs to achieve zero-shot generalization, since these models are pretrained on large corpura. Thanks to the advances in natural language pro- cessing, the path towards scalable dialogue system modeling becomes clearer. In this work, we propose a dialogue system that is capa- ble of dealing with a previously unseen domain via param- eter sharing. The system focuses on NLU and DST compo- nents, both of which leverage BERT model to encode the natural language descriptions of intents and slots predefined in schema. In each dialogue turn, a slot tagger identifies non-categorical slot values mentioned in the user utterance, and a state tracker sequentially predicts the active intent, re- quested slots, and slot values of the user belief. Here, we frame DST as a natural language inference problem (Bow- man et al. 2015). Given a user utterance as premise, we use a BERT model to identify the entailment relationships among a set of hypotheses. Varying from task to task, the hypothesis sentences could be: (1) intent descriptions for intent classifi- cation; (2) slot descriptions for requested slot detection; (3) Figure 1: An example of dialogue for a taxi service. On the left demonstrates the schema of the service. For each intent and slot, a natural language description is given, along with other fields such as required slots for a intent or possible values for a categorical slot. The dialogue states of each user turn is displayed on the right panel, in the format of ”Intent[requestedslots] (slotvalues)”. slot description and value pairs for candidate classification. By fine-tuning BERT weights, we expect the summary em- bedding (i.e. embedding of the[CLS]token) of the premise- hypothesis pair will capture enough information about the logic relationship, so as to facilitate DST (e.g. helping select the right candidate for a slot). The rest of the paper is organized as follows: Section 2 describes related work. In Section 3, we detail our model architecture, followed by Section 4 presenting evaluation re- sults and ablation studies. Finally, we discuss and conclude in Section 5 and 6. 2 Related Work DST aims to maintain dialogue states in a complicated con- versation. Dialogue state is usually represented as a set of slot-value pairs, which serves as parameters to inter- act with backend APIs and as inputs for downstream Di- alogue Policy module to predict next system action. Tra- ditional methods (Henderson, Thomson, and Young 2014; Mrk ˇ si ́ c et al. 2015; Wen et al. 2016) predict the slot-value pairs by performing classification over a predefined candi- date list. These models rely on delexicalization (i.e. replac- ing the slot types and values with generic tokens), which requires hand-crafted ontology with domain-specific lexi- cons. In practice, the ontology is not always available and the semantic dictionaries could be dynamically changing or hard to exhaust, limiting the generalization of these mod- els. To this end, approaches have been proposed that score over a fixed (Mrk ˇ si ́ c et al. 2016; Zhong, Xiong, and Socher 2018) or dynamic (Rastogi, Hakkani-T ̈ ur, and Heck 2017; Rastogi, Gupta, and Hakkani-Tur 2018; Ren et al. 2018) candidate list, extending the flexibility and scalablility of DST to rarely observed or unseen values in the training data. Other systems try to predict dialogue states directly from the dialogue context to handle previously unseen values, by ap- plying pointer network (Xu and Hu 2018) or copy mecha- nism (Wu et al. 2019). These approaches are based on the assumption that the values are available in the conversation history, which may not be always true. To this end, some models combine the candidate-list-based and span-based ap- proaches to take advantages from both sides (Goel, Paul, and Hakkani-T ̈ ur 2019; Zhang et al. 2019). On the other hand, pretrained models such as BERT are gaining increasing attention in research community be- cause of their promising performances on diverse down- stream tasks. Many studies are seeking to leverage BERT in DST. Particularly, inspired by the success of BERT in reading comprehension tasks, most of the studies leverag- ing BERT in DST choose the span-based approach (Lee, Lee, and Kim 2019; Gao et al. 2019; Chao and Lane 2019; Zhang et al. 2019). In our study, we formulate DST as a natu- ral language inference problem and utilize BERT to achieve zero-shot generalization. 3 Models Our dialogue system consists of two major components: Natural Language Understanding (NLU) and Dialogue State Tracking (DST). These two components work successively to predict the user belief, i.e. dialogue state. Let each turn in a dialogue be a user utterance and its preceding system utterance. In the first stage, the slot tagger predicts all val- ues of non-categorical slots mentioned in the user utterance. The tagged slot values are then used to update the candidate tracker. In the second stage, the user utterance is paired with intent descriptions, slot descriptions and slot-value pairs to get according embeddings from BERT. These embeddings are used as inputs of intent classifier, requested slot detector and candidate classifier respectively to get final predictions of the dialogue state. Figure 2: Slot tagger architecture. 3.1 Natural Language Understanding Slot TaggerSlot tagger is the only model in the NLU com- ponent, whose task is to identify any possible values men- tioned in the current user utterance for all non-categorical slots. We use IOB tagging schema introduced in (Sang and Buchholz 2000) to assign a label to each tokenx m in the user utterance (Figure 2), where0< m≤M,Mbeing the total number of tokens in the user utterance. The distri- bution over B, I and O tags is calculated using Equations 1–4. For a given non-categorical slot, the slot description is passed through BERT and the embedding of[CLS]token is captured as the summary embeddinge. Then the system utterance and user utterance pair is encoded by BERT to ob- tain token level representationst m . Eacht m is concatenated witheand then projected to get a representationr m through a linear layer. The sequencer m is fed into a bidirectional LSTM to get hidden states sequenceh m . r m =W token (t m ⊕e) +b token (1) h m = LSTM(r m )(2) l tag m =W tag h m +b tag (3) p tag m = softmax(l tag m )(4) Finally,h m is projected into a 3-dimension logitl tag m , and p tag m is calculated using softmax. During inference, we pre- dict the label for each tokenx m asargmax(p tag m ), and values labeled with B and I tags are added into the tagged slots. 3.2 Dialogue State Tracking Intent ClassifierActive intent in a given service denotes a specific task the user wants to complete (e.g. finding a restaurant or booking a flight ticket). The user’s active in- tent remains the same until s/he ends the conversation (i.e. ”NONE” intent) or changes to another intent. In the middle of a dialogue, the user may talk about many things that are not strongly related to the active intent. In consideration of this, we predict the intent change instead of the actual active intent. More specifically, letIbe the intent set for a given service.φandψdenote ”NONE” and ”UNCHANGED” in- tent respectively. The distribution over all intents for a given service is calculated as follows. l int i =W int r int i +b int (5) l int φ =W int φ r int +b int φ (6) l int ψ =W int ψ r int +b int ψ (7) p int = softmax(l int α ), α∈I∪φ,ψ(8) Here,r int i =e i ⊕a i is the intent related feature where e i is the summary embedding of user utterance and in- tent description pair, anda i is a one-hot encoding indicat- ing whether the specific intent is offered in preceding sys- tem acts, for intenti∈I.r int =u⊕awhereuis the user utterance embedding andais a one-hot encoding in- dicating whether any intent is offered in preceding system acts.W int ,b int ,W int φ ,b int φ ,W int ψ , andb int ψ are trainable parameters shared across different services. During infer- ence,argmax(p int )is taken as the intent label. If ”UN- CHANGED” intentψis predicted, we use the same intent from the previous turn. Requested Slot DetectorRequested slots are the slots whose values the user is asking for (e.g. what is the address of the restaurant). LetSbe the slot set of a given service. The probability of a given slots∈Sis calculated as below. l req s =W req e s +b req (9) p req s = sigmoid(l req s )(10) Here,e s is the summary embedding of user utterance and slot description pair for slots.W req ,b req are trainable pa- rameters. At inference time, we predict a slot as requested if the probabilityp req s >0.5. Candidate TrackerFor each slotsin a given service, we keep track of a candidate setC s . A special candidateδde- noting ”dontcare” is always in the setC s . For categorical slots, the setC s =δ∪V s will be constant, whereV s is the set of possible values. For non-categorical slots, apart from candidateδ, the most recentKcandidates are maintained in the setC s , whereKis an arbitrary hyper-parameter. The candidate tracker is also used to calculate candidate related source featuresa src s,c . Figure 3: Architecture of Dialogue State Tracker. Candidate ClassifierGiven a candidate setC s for slots, the candidate classifier predicts the dialogue state updates by calculating the distribution over the candidate set. We define a dialogue state update for a slotsas the difference between the dialogue state of current turn and that of preceding turn. If the system asks for confirmation and the user confirms the slot, the state is also considered updated, although the value in the current state may not be different from the preceding turn. When a slotsis not updated, a special candidate label φis assigned, denoting the ”none” state. To predict the distribution over the candidate set, we use an architecture inspired by the attention mechanism (Graves, Wayne, and Danihelka 2014; Luong, Pham, and Manning 2015). First, for a given slots, a slot related query vector q s is calculated by Equation 11. Here,r s =u⊕a s , where a s is the one-hot encoding for slot related system acts. For each candidatec, the key vector is calculated by Equation 12.r s,c =e s,c ⊕a src s,c , wheree s,c is the summary embed- ding of user utterance and slot-value pair.W query ,b query , W key φ ,b key φ ,W key , andb key are trainable parameter. Con- sequentially, slot related query vectorq s attends over candi- date related vectorsk s,c via Equation 13 to get logitsl cand s,c . Trainable parametersW cand ∈R d×d ,b cand here function similarly to general attention mechanism introduced in (Lu- ong, Pham, and Manning 2015). q s =W query r s +b query (11) k s,c = W key φ e s +b key φ ifc=φ W key r s,c +b key forc∈C s (12) l cand s,c =q T s W cand k s,c +b cand (13) p cand s = softmax(l cand s,c )(14) The logitsl cand s,c are normalized by softmax to get distri- bution over all candidate values. During inference time, the candidate with maximal probability is predicted. If a slot is Active Int AccReq Slot F1Average GAJoint GASlot Tagging F1 DevTestDevTestDevTestDevTestDevTest SGD baseline0.908-0.973-0.740-0.411--- CA-DST 0.9450.9230.9890.9820.9490.8950.8350.6950.9770.959 CA-DST+joint loss0.9440.9230.9920.9880.9560.8710.8500.6530.9770.959 (-pair encoding)0.9430.9120.9730.9610.8840.8400.6740.5780.9770.959 (-attention) 0.9510.9320.9910.9860.8820.7830.6730.4680.9770.959 Table 1: Evaluation results of ourCandidateAttendedDialogueStateTracking (CA-DST) system. For Ablation study, all models are training with joint goal loss described in Section 4.1. predicted to be ”none”, its value is set the same as in preced- ing turn. 4 Evaluation 4.1 Experiments We evaluate our system on SGD dataset. The sample dia- logue and schema are described in Figure 1. In our system, NLU module and DST module are trained separately. For slot tagger, we use a large cased BERT model in a feature- based approach, without updating any of its parameters. For state tracker, we fine-tune a base uncased BERT model while the embedding layer and first 6 layers are fixed. This is a compromise due to the hardware limits. To facilitate con- vergence, we first fine-tune the BERT models on masked language modeling task using dialogues from SGD dataset before training each component. In DST module, intent clas- sifier, requested slot detector and candidate classifier are trained jointly. Additionally, concerning all slots are treated independently, we add a joint goal loss penalizing incorrect prediction on what slots appear in the current turn. The joint goal loss is calculated as cross entropy of ”none” logits for all slots. Figure 4: Joint Goal Accuracy on All services, services seen in training data, services not seen in training data. We use the same metrics described in Rastogi et al. (2019) for evaluation: Active Intent Accuracy, Requested Slot F1, Average Goal Accuracy, Joint Goal Accuracy, Slot Tagging F1. The results are showed in Table 1. We also report here the results of the baseline model (Rastogi et al. 2019) as comparison. As showed in Table 1, our system shows significant im- provement in intent classification, requested slot detection and joint goal prediction compared to baseline, doubling the joint goal accuracy. However, the performance of our system exhibits a remarkable deterioration on the test set, indicat- ing the models may suffer from over-fitting. Figure 4 shows the Joint Goal Accuracy in all, seen and unseen services. In those services that appear in training set, the model performs similarly, while in unseen services the Joint Goal Accuracy on test set is notably less than on dev set. 4.2 Ablation Studies We perform ablation studies over two of the most impor- tant architecture designs of our system. The results are also shown in Table 1. The first one is that we pair each natural language element (intent description, slot description, or slot description plus value) with user utterance before entering BERT model. In this way the model can leverage the power of self-attention in BERT. To study the effectiveness of this pair encoding approach, we singly encode natural language elements and fuse the embeddings with user utterance em- bedding using the same projection layer described in base- line model (Rastogi et al. 2019). While in easier tasks like intent classification and requested slot prediction the perfor- mances are similar, the model without pair encoding shows a notable decline in Joint Goal Accuracy. Second, we replace the attention-based bi-linear layer in the candidate classifier with feed forward layers. Model without attention performs significantly worse in Joint Goal Accuracy. 5 Discussion The challenges towards scalable multi-domain DST come from three folds. First, during inference time, there are many unseen services which include several slots very different from those in training data. The only clues to identify those unseen slots are the natural language descriptions. Second, in multi-domain dialogues, the system is expected to infer certain slot values from other services when user changes a topic. Third, the expression diversity of natural language in dialogues further complicates the DST tasks. In this section we discuss the limitation and possible improvement direc- tion based on these aspects. In Table 2, we list Average Goal Accuracy and Joint Goal Accuracy per services in the dev set and test set. By examin- ing those poorly performing services, we identify some chal- Service Average GAJoint GA DevTestDevTest Alarm1*0.9290.9390.9270.909 Banks2*0.982-0.952- Buses10.973-0.919- Buses3*-0.896-0.622 Events10.981-0.913- Events3*-0.895-0.681 Flights3*0.836-0.470- Flights 4*-0.681-0.322 Homes10.987-0.938- Homes 2*-0.956-0.563 Hotels10.949-0.884- Hotels 2-0.928-0.859 Hotels4*0.9360.9390.8440.859 Media2*0.911-0.761- Media 3*-0.791-0.594 Messaging1*-0.769-0.683 Movie1-0.957-0.819 Movie2*0.911-0.761- Media3*-0.683-0.700 Music10.936-0.823- Music3*-0.754-0.299 Payment1*-0.846-0.608 RentalCars1-0.947-0.772 RentalCars3*-0.897-0.486 Restaurants2*0.9470.9490.7470.730 RideSharing10.958-0.909- RideSharing2-0.944-0.840 Services1-0.987-0.960 Services4*0.9890.9860.8920.876 Trains1*-0.895-0.666 Travel10.9650.8620.9300.778 Weather1*0.9680.9660.9290.916 Table 2: Model performance per service. Services masked with ”*” denote that they are unseen in the training set. lenging cases. For example, in service ”Movies2” there is a slot ”starring” with the description ”Name of actor starring in movie”. Similarly service ”Movies3” has a slot ”cast” described as ”Actors in the movie”. Both unseen services include two slots of the person name type: one for direc- tor and one for actor. However, in training set, there is only one slot for director slot, which demands for higher level of generalization. The system not only needs to understand the actor slots refer to person names, but also has to understand that the person should be the actor instead of the director. This requires building a semantic correlation between word ”actor” and others such as ”act”, ”starring” or ”plays”. The BERT pretrained on large-scale corpura should have some capability to resolve such semantic ambiguity, but it seems to fail. Moreover, some cases are even difficult for human. For example, in the utterance ”Do you have any movies with Mike Colter?”, one needs to know that Mike Colter is an ac- tor to correctly identify this value as an actor instead of a director. Possibly, the system may leverage external knowl- edge to deal with such problems. Another challenge is the slot transfer in multi-domain di- alogues. For instance, when a user just checked the weather in Toronto and asks to book a hotel, the system is expected to assume the city slot in hotel domain is Toronto. Because there are unseen domains, it is difficult to come up with an exhausted slot-to-slot mapping list across different domains. Currently, our system collects all possible slot values from previous domain as the candidates, and relies on the seman- tic validity provided by BERT to solve the slot transferring problem (e.g. the probability of ”The location of the hotel is Toronto” is much higher than that of ”The location of the ho- tel is today”). This approach is not robust enough especially in certain domains like Flights, where it is hard to tell a city appeared in the dialogue history would be transferred to des- tination or origin. Additionally, in a more complicated situa- tion, this slot-to-slot mapping could be dynamic, depending on the dialogue history. Take Flights and Hotels as example. A user just booked a flight from NY to LA, and then asks for a hotel on Friday. Practically, if the flight date is Thurs- day, we assume the user is looking for a hotel in LA. Yet, if the flight starts on Saturday, one should expect the user wants the hotel to be in NY. To distinguish such nuances, a dialogue system needs to incorporate adequate level of com- mon sense reasoning. Finally, the diversified expressions in natural language sometimes make it hard to predict dialogue states. For cat- egorical slots, possible slot values are given in advance, where the task of DST is to map the words or phrases in user utterance to one of the candidate values. In most cases the mapping is trivial (e.g. ”three” to ”3”, ”checking” to ”check- ing account”). However, sometimes the mapping could be logically difficult for machine if not for human. For exam- ple, ”two years ago” for categorical slot ”year” is mapped to ”2017” given this year is 2019, which involves elementary level of mathematical calculation. This mapping may even change as time goes. As for non-categorical slots, the Joint Goal Accuracy is highly depending on the outputs of slot tagger. One major drawback of separating slot tagger and di- alogue state tracker is the error propagation. If a slot is failed to be tagged by the NLU module, the downstream DST com- ponent would not select the correct candidate. Nevertheless, the pipeline approach on the other hand make it more flex- ible to plug in additional modules. By incorporating open- domain Named Entity Resolution (NER) tools or domain- specific slot tagger, the system is easy to fix some poorly performed domains without significant architecture changes. 6 Conclusions To address the scalability challenge in task-oriented dia- logue state tracking, we frame the problem as a natural language inference problem and propose a multi-domain schema-based dialog system. The system leverages pre- trained BERT model to achieve zero-shot generalization via parameter sharing. The experiments on the newly released Schema-Guided Dialogue dataset show the effectiveness of the proposed system. Acknowledgement We would like to thank Surya Kasturi, Xin Chen, and anony- mous reviewers for their constructive discussions and feed- back. References Bapna, A.; Tur, G.; Hakkani-Tur, D.; and Heck, L. 2017. To- wards zero-shot frame semantic parsing for domain scaling. arXiv preprint arXiv:1707.02363. Bowman, S. R.; Angeli, G.; Potts, C.; and Manning, C. D. 2015. A large annotated corpus for learning natural language inference.arXiv preprint arXiv:1508.05326. Budzianowski, P.; Wen, T.-H.; Tseng, B.-H.; Casanueva, I.; Ultes, S.; Ramadan, O.; and Ga ˇ si ́ c, M.2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling.arXiv preprint arXiv:1810.00278. Chao, G.-L., and Lane, I.2019.Bert-dst: Scalable end-to-end dialogue state tracking with bidirectional en- coder representations from transformer.arXiv preprint arXiv:1907.03040. Chen, H.; Liu, X.; Yin, D.; and Tang, J. 2017. A survey on dialogue systems: Recent advances and new frontiers.Acm Sigkdd Explorations Newsletter19(2):25–35. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding.arXiv preprint arXiv:1810.04805. Gao, S.; Sethi, A.; Aggarwal, S.; Chung, T.; and Hakkani- Tur, D. 2019. Dialog state tracking: A neural reading com- prehension approach.arXiv preprint arXiv:1908.01946. Goel, R.; Paul, S.; and Hakkani-T ̈ ur, D. 2019. Hyst: A hy- brid approach for flexible and accurate dialogue state track- ing.arXiv preprint arXiv:1907.00883. Graves, A.; Wayne, G.; and Danihelka, I. 2014. Neural turing machines.arXiv preprint arXiv:1410.5401. Henderson, M.; Thomson, B.; and Young, S. 2014. Word- based dialog state tracking with recurrent neural networks. InProceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), 292– 299. Lee, H.; Lee, J.; and Kim, T.-Y. 2019. Sumbt: Slot-utterance matching for universal and scalable belief tracking.arXiv preprint arXiv:1907.07421. Luong, M.-T.; Pham, H.; and Manning, C. D. 2015. Effec- tive approaches to attention-based neural machine transla- tion.arXiv preprint arXiv:1508.04025. Mrk ˇ si ́ c, N.; S ́ eaghdha, D. O.; Thomson, B.; Ga ˇ si ́ c, M.; Su, P.-H.; Vandyke, D.; Wen, T.-H.; and Young, S. 2015. Multi- domain dialog state tracking using recurrent neural net- works.arXiv preprint arXiv:1506.07190. Mrk ˇ si ́ c, N.; S ́ eaghdha, D. O.; Wen, T.-H.; Thomson, B.; and Young, S. 2016. Neural belief tracker: Data-driven dialogue state tracking.arXiv preprint arXiv:1606.03777. Rastogi, A.; Zang, X.; Sunkara, S.; Gupta, R.; and Khai- tan, P. 2019. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset.arXiv preprint arXiv:1909.05855. Rastogi, A.; Gupta, R.; and Hakkani-Tur, D. 2018. Multi- task learning for joint language understanding and dialogue state tracking.arXiv preprint arXiv:1811.05408. Rastogi, A.; Hakkani-T ̈ ur, D.; and Heck, L. 2017. Scalable multi-domain dialogue state tracking. In2017 IEEE Au- tomatic Speech Recognition and Understanding Workshop (ASRU), 561–568. IEEE. Ren, L.; Xie, K.; Chen, L.; and Yu, K.2018.To- wards universal dialogue state tracking.arXiv preprint arXiv:1810.09587. Rudnicky, A. I.; Thayer, E.; Constantinides, P.; Tchou, C.; Shern, R.; Lenzo, K.; Xu, W.; and Oh, A. 1999. Creating nat- ural dialogs in the carnegie mellon communicator system. In Sixth European Conference on Speech Communication and Technology. Sang, E. F., and Buchholz, S. 2000. Introduction to the conll- 2000 shared task: Chunking.arXiv preprint cs/0009008. Seokhwan Kim, Michel Galley, C. G. S. L. A. A. B. P. H. S. J. G. J. L. M. A. M. H. L. L. J. K. K. W. S. L. C. H. A. C. T. K. M. A. R. X. Z. S. S. R. G. 2019. The eighth dialog system technology challenge.arXiv preprint. Shah, D. J.; Gupta, R.; Fayazi, A. A.; and Hakkani-Tur, D. 2019. Robust zero-shot cross-domain slot filling with exam- ple values.arXiv preprint arXiv:1906.06870. Sutskever, I.; Vinyals, O.; and Le, Q. 2014. Sequence to sequence learning with neural networks.Advances in NIPS. Wen, T.-H.; Vandyke, D.; Mrksic, N.; Gasic, M.; Rojas- Barahona, L. M.; Su, P.-H.; Ultes, S.; and Young, S. 2016. A network-based end-to-end trainable task-oriented dialogue system.arXiv preprint arXiv:1604.04562. Wu, C.-S.; Madotto, A.; Hosseini-Asl, E.; Xiong, C.; Socher, R.; and Fung, P. 2019. Transferable multi-domain state generator for task-oriented dialogue systems.arXiv preprint arXiv:1905.08743. Xu, P., and Hu, Q. 2018. An end-to-end approach for han- dling unknown slot values in dialogue state tracking.arXiv preprint arXiv:1805.01555. Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R.; and Le, Q. V. 2019. Xlnet: Generalized autoregres- sive pretraining for language understanding.arXiv preprint arXiv:1906.08237. Zhang, J.-G.; Hashimoto, K.; Wu, C.-S.; Wan, Y.; Yu, P. S.; Socher, R.; and Xiong, C. 2019. Find or classify? dual strat- egy for slot-value predictions on multi-domain dialog state tracking.arXiv preprint arXiv:1910.03544. Zhong, V.; Xiong, C.; and Socher, R. 2018. Global-locally self-attentive encoder for dialogue state tracking. InPro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1458– 1467. Zue, V.; Seneff, S.; Glass, J. R.; Polifroni, J.; Pao, C.; Hazen, T. J.; and Hetherington, L. 2000. Juplter: a telephone-based conversational interface for weather information.IEEE Transactions on speech and audio processing8(1):85–96.