Dialog State Tracking Challenge
Jump to navigation
Jump to search
A Dialog State Tracking Challenge is a human-computer spoken dialog tasks that is a benchmark task.
- …
- Example(s):
- See: Spoken Dialog System, Spoken Language Understanding, Dialog State Tracking.
References
2016
- https://www.microsoft.com/en-us/research/event/dialog-state-tracking-challenge/#
- QUOTE:
- DSTC1 used human-computer dialogs in the bus timetable domain. Results were presented in a special session at SIGDIAL 2013. DSTC1 was organized by Jason D. Williams, Alan Black, Deepak Ramachandran, Antoine Raux.
- DSTC2/3 used human-computer dialogs in the restaurant information domain. Results were presented in special sessions at SIGDIAL 2014 and IEEE SLT 2014. DSTC2 and 3 were organized by Matthew Henderson, Blaise Thomson, and Jason D. Williams.
- DSTC4 used human-human dialogs in the tourist information domain. Results were presented at IWSDS 2015. DSTC4 was organized by Seokhwan Kim, Luis F. D’Haro, Rafael E Banchs, Matthew Henderson, and Jason D. Williams.
- DSTC5 used human-human dialogs in the tourist information domain, where training dialogs were provided in one language, and test dialogs were in a different language. Results will be presented in a special session at IEEE SLT 2016. DSTC5 was organized by Seokhwan Kim, Luis F. D’Haro, Rafael E Banchs, Matthew Henderson, Jason D. Williams, and Koichiro Yoshino.
- QUOTE:
2014a
- (Henderson et al., 2014) ⇒ Matthew Henderson, Blaise Thomson, and Jason D. Williams. (2014). “The Third Dialog State Tracking Challenge.” In: Spoken Language Technology Workshop (SLT), 2014 IEEE, pp. 324-329 . IEEE, doi:10.1109/SLT.2014.7078595
- ABSTRACT: In spoken dialog systems, dialog state tracking refers to the task of correctly inferring the user's goal at a given turn, given all of the dialog history up to that turn. This task is challenging because of speech recognition and language understanding errors, yet good dialog state tracking is crucial to the performance of spoken dialog systems. This paper presents results from the third Dialog State Tracking Challenge, a research community challenge task based on a corpus of annotated logs of human-computer dialogs, with a blind test set evaluation. The main new feature of this challenge is that it studied the ability of trackers to generalize to new entities - i.e. new slots and values not present in the training data. This challenge received 28 entries from 7 research teams. About half the teams substantially exceeded the performance of a competitive rule-based baseline, illustrating not only the merits of statistical methods for dialog state tracking but also the difficulty of the problem.
2014b
- (Henderson et al., 2014) ⇒ Matthew Henderson, Blaise Thomson, and Jason Williams. (2014). “The Second Dialog State Tracking Challenge.” In: 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue, vol. 263.
- ABSTRACT: A spoken dialog system, while communicating with a user, must keep track of what the user wants from the system at each step. This process, termed dialog state tracking, is essential for a successful dialog system as it directly informs the system’s actions. The first Dialog State Tracking Challenge allowed for evaluation of different dialog state tracking techniques, providing common testbeds and evaluation suites. This paper presents a second challenge, which continues this tradition and introduces some additional features – a new domain, changing user goals and a richer dialog state. The challenge received 31 entries from 9 research groups.
The results suggest that while large improvements on a competitive baseline are possible, trackers are still prone to degradation in mismatched conditions. An investigation into ensemble learning demonstrates the most accurate tracking can be achieved by combining multiple trackers.
- ABSTRACT: A spoken dialog system, while communicating with a user, must keep track of what the user wants from the system at each step. This process, termed dialog state tracking, is essential for a successful dialog system as it directly informs the system’s actions. The first Dialog State Tracking Challenge allowed for evaluation of different dialog state tracking techniques, providing common testbeds and evaluation suites. This paper presents a second challenge, which continues this tradition and introduces some additional features – a new domain, changing user goals and a richer dialog state. The challenge received 31 entries from 9 research groups.
2013
- (Williams et al., 2013) ⇒ Jason D. Williams, Antoine Raux, Deepak Ramachandran, and Alan W. Black. (2013). “The Dialog State Tracking Challenge.” In: SIGDIAL Conference, pp. 404-413.
- ABSTRACT: In a spoken dialog system, dialog state tracking deduces information about the user’s goal as the dialog progresses, synthesizing evidence such as dialog acts over multiple turns with external data sources. Recent approaches have been shown to overcome ASR and SLU errors in some applications. However, there are currently no common testbeds or evaluation measures for this task, hampering progress. The dialog state tracking challenge seeks to address this by providing a heterogeneous corpus of 15K human-computer dialogs in a standard format, along with a suite of 11 evaluation metrics. The challenge received a total of 27 entries from 9 research groups. The results show that the suite of performance metrics cluster into 4 natural groups. Moreover, the dialog systems that benefit most from dialog state tracking are those with less discriminative speech recognition confidence scores. Finally, generalization is a key problem: in 2 of the 4 test sets, fewer than half of the entries out-performed simple baselines.