Using Natural Language Processing and Discourse Features to Identify Understanding Errors in a Spoken Dialogue System

While it has recently become possible to build spoken dialogue systems that interact with users in real-time in a range of domains, systems that support conversational natural language are still subject to a large number of spoken language understanding (SLU) errors. Endowing such systems with the ability to reliably distinguish SLU errors from correctly understood utterances might allow them to correct some errors automatically or to interact with users to repair them, thereby improving the system’s overall performance. We report experiments on learning to automatically distinguish SLU errors in 11,787 spoken utterances collected in a field trial of AT&T’s How May I Help You system interacting with live customer traffic. We apply the automatic classifier RIPPER (Cohen 96) to train an SLU classifier using features that are automatically obtainable in real-time. The classifer achieves 86 % accuracy on this task, an improvement of 23 % over the majority class baseline. We show that the most important features are those that the natural language understanding module can compute, suggesting that integrating the trained classifier into the NLU module of the How May I Help You system should be straightforward. 1.