Results of WMT23 Metrics Shared Task: Metrics might be Guilty but References are not Innocent

Markus Freitag; Nitika Mathur; Chi Kiu Lo; Eleftherios Avramidis; Ricardo Rei; Brian Thompson; Tom Kocmi; Frédéric Blain; Daniel Deutsch; Craig Stewart; Chrysoula Zerva; Sheila Castilho; Alon Lavie; George Foster

Results of WMT23 Metrics Shared Task: Metrics might be Guilty but References are not Innocent

Markus Freitag, Nitika Mathur, Chi Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frédéric Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, George Foster

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

This paper presents the results of the WMT23 Metrics Shared Task. Participants submitting automatic MT evaluation metrics were asked to score the outputs of the translation systems competing in the WMT23 News Translation Task. All metrics were evaluated on how well they correlate with human ratings at the system and segment level. Similar to last year, we acquired our own human ratings based on expert-based human evaluation via Multidimensional Quality Metrics (MQM). Following last year's success, we also included a challenge set subtask, where participants had to create contrastive test suites for evaluating metrics' ability to capture and penalise specific types of translation errors. Furthermore, we improved our meta-evaluation procedure by considering fewer tasks and calculating a global score by weighted averaging across the various tasks.

Original language	English
Title of host publication	Proceedings of the 8th Conference on Machine Translation, WMT 2023
Publisher	Association for Computational Linguistics
Pages	576-626
Number of pages	51
ISBN (Electronic)	9798891760417
State	Published - 2023
Externally published	Yes
Event	8th Conference on Machine Translation, WMT 2023 - Singapore, Singapore Duration: 6 Dec 2023 → 7 Dec 2023

Publication series

Name	Conference on Machine Translation - Proceedings
ISSN (Electronic)	2768-0983

Conference

Conference	8th Conference on Machine Translation, WMT 2023
Country/Territory	Singapore
City	Singapore
Period	6/12/23 → 7/12/23

Bibliographical note

Publisher Copyright:
© 2023 Association for Computational Linguistics.

ASJC Scopus subject areas

Language and Linguistics
Human-Computer Interaction
Software

Cite this

Freitag, M., Mathur, N., Lo, C. K., Avramidis, E., Rei, R., Thompson, B., Kocmi, T., Blain, F., Deutsch, D., Stewart, C., Zerva, C., Castilho, S., Lavie, A., & Foster, G. (2023). Results of WMT23 Metrics Shared Task: Metrics might be Guilty but References are not Innocent. In Proceedings of the 8th Conference on Machine Translation, WMT 2023 (pp. 576-626). (Conference on Machine Translation - Proceedings). Association for Computational Linguistics.

@inproceedings{bae38eeef5a341cab48f687b37fa85d9,

title = "Results of WMT23 Metrics Shared Task: Metrics might be Guilty but References are not Innocent",

abstract = "This paper presents the results of the WMT23 Metrics Shared Task. Participants submitting automatic MT evaluation metrics were asked to score the outputs of the translation systems competing in the WMT23 News Translation Task. All metrics were evaluated on how well they correlate with human ratings at the system and segment level. Similar to last year, we acquired our own human ratings based on expert-based human evaluation via Multidimensional Quality Metrics (MQM). Following last year's success, we also included a challenge set subtask, where participants had to create contrastive test suites for evaluating metrics' ability to capture and penalise specific types of translation errors. Furthermore, we improved our meta-evaluation procedure by considering fewer tasks and calculating a global score by weighted averaging across the various tasks.",

author = "Markus Freitag and Nitika Mathur and Lo, {Chi Kiu} and Eleftherios Avramidis and Ricardo Rei and Brian Thompson and Tom Kocmi and Fr{\'e}d{\'e}ric Blain and Daniel Deutsch and Craig Stewart and Chrysoula Zerva and Sheila Castilho and Alon Lavie and George Foster",

note = "Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.; 8th Conference on Machine Translation, WMT 2023 ; Conference date: 06-12-2023 Through 07-12-2023",

year = "2023",

language = "English",

series = "Conference on Machine Translation - Proceedings",

publisher = "Association for Computational Linguistics",

pages = "576--626",

booktitle = "Proceedings of the 8th Conference on Machine Translation, WMT 2023",

address = "United States",

}

Freitag, M, Mathur, N, Lo, CK, Avramidis, E, Rei, R, Thompson, B, Kocmi, T, Blain, F, Deutsch, D, Stewart, C, Zerva, C, Castilho, S, Lavie, A & Foster, G 2023, Results of WMT23 Metrics Shared Task: Metrics might be Guilty but References are not Innocent. in Proceedings of the 8th Conference on Machine Translation, WMT 2023. Conference on Machine Translation - Proceedings, Association for Computational Linguistics, pp. 576-626, 8th Conference on Machine Translation, WMT 2023, Singapore, Singapore, 6/12/23.

Results of WMT23 Metrics Shared Task: Metrics might be Guilty but References are not Innocent. / Freitag, Markus; Mathur, Nitika; Lo, Chi Kiu et al.
Proceedings of the 8th Conference on Machine Translation, WMT 2023. Association for Computational Linguistics, 2023. p. 576-626 (Conference on Machine Translation - Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Results of WMT23 Metrics Shared Task

T2 - 8th Conference on Machine Translation, WMT 2023

AU - Freitag, Markus

AU - Mathur, Nitika

AU - Lo, Chi Kiu

AU - Avramidis, Eleftherios

AU - Rei, Ricardo

AU - Thompson, Brian

AU - Kocmi, Tom

AU - Blain, Frédéric

AU - Deutsch, Daniel

AU - Stewart, Craig

AU - Zerva, Chrysoula

AU - Castilho, Sheila

AU - Lavie, Alon

AU - Foster, George

PY - 2023

Y1 - 2023

N2 - This paper presents the results of the WMT23 Metrics Shared Task. Participants submitting automatic MT evaluation metrics were asked to score the outputs of the translation systems competing in the WMT23 News Translation Task. All metrics were evaluated on how well they correlate with human ratings at the system and segment level. Similar to last year, we acquired our own human ratings based on expert-based human evaluation via Multidimensional Quality Metrics (MQM). Following last year's success, we also included a challenge set subtask, where participants had to create contrastive test suites for evaluating metrics' ability to capture and penalise specific types of translation errors. Furthermore, we improved our meta-evaluation procedure by considering fewer tasks and calculating a global score by weighted averaging across the various tasks.

AB - This paper presents the results of the WMT23 Metrics Shared Task. Participants submitting automatic MT evaluation metrics were asked to score the outputs of the translation systems competing in the WMT23 News Translation Task. All metrics were evaluated on how well they correlate with human ratings at the system and segment level. Similar to last year, we acquired our own human ratings based on expert-based human evaluation via Multidimensional Quality Metrics (MQM). Following last year's success, we also included a challenge set subtask, where participants had to create contrastive test suites for evaluating metrics' ability to capture and penalise specific types of translation errors. Furthermore, we improved our meta-evaluation procedure by considering fewer tasks and calculating a global score by weighted averaging across the various tasks.

UR - http://www.scopus.com/inward/record.url?scp=85179122139&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85179122139

T3 - Conference on Machine Translation - Proceedings

SP - 576

EP - 626

BT - Proceedings of the 8th Conference on Machine Translation, WMT 2023

PB - Association for Computational Linguistics

Y2 - 6 December 2023 through 7 December 2023

ER -

Results of WMT23 Metrics Shared Task: Metrics might be Guilty but References are not Innocent

Abstract

Publication series

Conference

Bibliographical note

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this