Gaining Insights into Unrecognized User Utterances in Task-Oriented Dialog Systems

Ella Rabinovich; Matan Vetzler; David Boaz; Vineet Kumar; Gaurav Pandey; Ateret Anaby-Tavor

doi:10.18653/v1/2022.emnlp-industry.22

Gaining Insights into Unrecognized User Utterances in Task-Oriented Dialog Systems

Ella Rabinovich
, Matan Vetzler
, David Boaz
, Vineet Kumar
, Gaurav Pandey
, Ateret Anaby-Tavor

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

The rapidly growing market demand for automatic dialogue agents capable of goal-oriented behavior has caused many tech-industry leaders to invest considerable efforts into task-oriented dialog systems. The success of these systems is highly dependent on the accuracy of their intent identification – the process of deducing the goal or meaning of the user’s request and mapping it to one of the known intents for further processing. Gaining insights into unrecognized utterances – user requests the systems fail to attribute to a known intent – is therefore a key process in continuous improvement of goal-oriented dialog systems. We present an end-to-end pipeline for processing unrecognized user utterances, deployed in a real-world, commercial task-oriented dialog system, including a specifically-tailored clustering algorithm, a novel approach to cluster representative extraction, and cluster naming. We evaluated the proposed components, demonstrating their benefits in the analysis of unrecognized user requests.

Original language	English
Title of host publication	EMNLP 2022 - Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Subtitle of host publication	Industry Track
Publisher	Association for Computational Linguistics (ACL)
Pages	228-235
Number of pages	8
ISBN (Electronic)	9781952148255
DOIs	https://doi.org/10.18653/v1/2022.emnlp-industry.22
State	Published - 2022
Externally published	Yes
Event	2022 Conference on Empirical Methods in Natural Language Processing: Industry Track , EMNLP 2022 - Abu Dhabi, United Arab Emirates Duration: 7 Dec 2022 → 11 Dec 2022

Publication series

Name	EMNLP 2022 - Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track

Conference

Conference	2022 Conference on Empirical Methods in Natural Language Processing: Industry Track , EMNLP 2022
Country/Territory	United Arab Emirates
City	Abu Dhabi
Period	7/12/22 → 11/12/22

Bibliographical note

Publisher Copyright:
© 2022 Association for Computational Linguistics.

ASJC Scopus subject areas

Computational Theory and Mathematics
Information Systems
Computer Science Applications

Access to Document

10.18653/v1/2022.emnlp-industry.22

Cite this

Rabinovich, E., Vetzler, M., Boaz, D., Kumar, V., Pandey, G., & Anaby-Tavor, A. (2022). Gaining Insights into Unrecognized User Utterances in Task-Oriented Dialog Systems. In EMNLP 2022 - Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track (pp. 228-235). (EMNLP 2022 - Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2022.emnlp-industry.22

Rabinovich, Ella ; Vetzler, Matan ; Boaz, David et al. / Gaining Insights into Unrecognized User Utterances in Task-Oriented Dialog Systems. EMNLP 2022 - Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track. Association for Computational Linguistics (ACL), 2022. pp. 228-235 (EMNLP 2022 - Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track).

@inproceedings{a985e65dae774703a06f15b58491ad65,

title = "Gaining Insights into Unrecognized User Utterances in Task-Oriented Dialog Systems",

abstract = "The rapidly growing market demand for automatic dialogue agents capable of goal-oriented behavior has caused many tech-industry leaders to invest considerable efforts into task-oriented dialog systems. The success of these systems is highly dependent on the accuracy of their intent identification {\textendash} the process of deducing the goal or meaning of the user{\textquoteright}s request and mapping it to one of the known intents for further processing. Gaining insights into unrecognized utterances {\textendash} user requests the systems fail to attribute to a known intent {\textendash} is therefore a key process in continuous improvement of goal-oriented dialog systems. We present an end-to-end pipeline for processing unrecognized user utterances, deployed in a real-world, commercial task-oriented dialog system, including a specifically-tailored clustering algorithm, a novel approach to cluster representative extraction, and cluster naming. We evaluated the proposed components, demonstrating their benefits in the analysis of unrecognized user requests.",

author = "Ella Rabinovich and Matan Vetzler and David Boaz and Vineet Kumar and Gaurav Pandey and Ateret Anaby-Tavor",

note = "Publisher Copyright: {\textcopyright} 2022 Association for Computational Linguistics.; 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track , EMNLP 2022 ; Conference date: 07-12-2022 Through 11-12-2022",

year = "2022",

doi = "10.18653/v1/2022.emnlp-industry.22",

language = "English",

series = "EMNLP 2022 - Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track",

publisher = "Association for Computational Linguistics (ACL)",

pages = "228--235",

booktitle = "EMNLP 2022 - Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",

address = "United States",

}

Rabinovich, E, Vetzler, M, Boaz, D, Kumar, V, Pandey, G & Anaby-Tavor, A 2022, Gaining Insights into Unrecognized User Utterances in Task-Oriented Dialog Systems. in EMNLP 2022 - Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track. EMNLP 2022 - Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, Association for Computational Linguistics (ACL), pp. 228-235, 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track , EMNLP 2022, Abu Dhabi, United Arab Emirates, 7/12/22. https://doi.org/10.18653/v1/2022.emnlp-industry.22

Gaining Insights into Unrecognized User Utterances in Task-Oriented Dialog Systems. / Rabinovich, Ella; Vetzler, Matan; Boaz, David et al.
EMNLP 2022 - Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track. Association for Computational Linguistics (ACL), 2022. p. 228-235 (EMNLP 2022 - Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Gaining Insights into Unrecognized User Utterances in Task-Oriented Dialog Systems

AU - Rabinovich, Ella

AU - Vetzler, Matan

AU - Boaz, David

AU - Kumar, Vineet

AU - Pandey, Gaurav

AU - Anaby-Tavor, Ateret

PY - 2022

Y1 - 2022

N2 - The rapidly growing market demand for automatic dialogue agents capable of goal-oriented behavior has caused many tech-industry leaders to invest considerable efforts into task-oriented dialog systems. The success of these systems is highly dependent on the accuracy of their intent identification – the process of deducing the goal or meaning of the user’s request and mapping it to one of the known intents for further processing. Gaining insights into unrecognized utterances – user requests the systems fail to attribute to a known intent – is therefore a key process in continuous improvement of goal-oriented dialog systems. We present an end-to-end pipeline for processing unrecognized user utterances, deployed in a real-world, commercial task-oriented dialog system, including a specifically-tailored clustering algorithm, a novel approach to cluster representative extraction, and cluster naming. We evaluated the proposed components, demonstrating their benefits in the analysis of unrecognized user requests.

AB - The rapidly growing market demand for automatic dialogue agents capable of goal-oriented behavior has caused many tech-industry leaders to invest considerable efforts into task-oriented dialog systems. The success of these systems is highly dependent on the accuracy of their intent identification – the process of deducing the goal or meaning of the user’s request and mapping it to one of the known intents for further processing. Gaining insights into unrecognized utterances – user requests the systems fail to attribute to a known intent – is therefore a key process in continuous improvement of goal-oriented dialog systems. We present an end-to-end pipeline for processing unrecognized user utterances, deployed in a real-world, commercial task-oriented dialog system, including a specifically-tailored clustering algorithm, a novel approach to cluster representative extraction, and cluster naming. We evaluated the proposed components, demonstrating their benefits in the analysis of unrecognized user requests.

UR - https://www.scopus.com/pages/publications/85152953657

U2 - 10.18653/v1/2022.emnlp-industry.22

DO - 10.18653/v1/2022.emnlp-industry.22

M3 - Conference contribution

AN - SCOPUS:85152953657

T3 - EMNLP 2022 - Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track

SP - 228

EP - 235

BT - EMNLP 2022 - Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing

PB - Association for Computational Linguistics (ACL)

T2 - 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track , EMNLP 2022

Y2 - 7 December 2022 through 11 December 2022

ER -

Rabinovich E, Vetzler M, Boaz D, Kumar V, Pandey G, Anaby-Tavor A. Gaining Insights into Unrecognized User Utterances in Task-Oriented Dialog Systems. In EMNLP 2022 - Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track. Association for Computational Linguistics (ACL). 2022. p. 228-235. (EMNLP 2022 - Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track). doi: 10.18653/v1/2022.emnlp-industry.22

Gaining Insights into Unrecognized User Utterances in Task-Oriented Dialog Systems

Abstract

Publication series

Conference

Bibliographical note

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this