GEMv2: Multilingual NLG Benchmarking in a Single Line of Code

Sebastian Gehrmann; Abhik Bhattacharjee; Abinaya Mahendiran; Alex Wang; Alexandros Papangelis; Aman Madaan; Angelina McMillan-Major; Anna Shvets; Ashish Upadhyay; Bernd Bohnet; Bingsheng Yao; Bryan Wilie; Chandra Bhagavatula; Chaobin You; Craig Thomson; Cristina Garbacea; Dakuo Wang; Daniel Deutsch; Deyi Xiong; Di Jin; Dimitra Gkatzia; Dragomir Radev; Elizabeth Clark; Esin Durmus; Faisal Ladhak; Filip Ginter; Genta Indra Winata; Hendrik Strobelt; Hiroaki Hayashi; Jekaterina Novikova; Jenna Kanerva; Jenny Chim; Jiawei Zhou; Jordan Clive; Joshua Maynez; João Sedoc; Juraj Juraska; Kaustubh Dhole; Khyathi Raghavi Chandu; Laura Perez-Beltrachini; Leonardo F.R. Ribeiro; Lewis Tunstall; Li Zhang; Mahima Pushkarna; Mathias Creutz; Michael White; Mihir Sanjay Kale; Moussa Kamal Eddine; Nico Daheim; Nishant Subramani; Ondrej Dusek; Paul Pu Liang; Pawan Sasanka Ammanamanchi; Qi Zhu; Ratish Puduppully; Reno Kriz; Rifat Shahriyar; Ronald Cardenas; Saad Mahamood; Salomey Osei; Samuel Cahyawijaya; Sanja Štajner; Sebastien Montella; Shailza Jolly; Simon Mille; Tahmid Hasan; Tianhao Shen; Tosin Adewumi; Vikas Raunak; Vipul Raheja; Vitaly Nikolaev; Vivian Tsai; Yacine Jernite; Ying Xu; Yisi Sang; Yixin Liu; Yufang Hou

doi:10.18653/v1/2022.emnlp-demos.27

GEMv2: Multilingual NLG Benchmarking in a Single Line of Code

Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina McMillan-Major, Anna Shvets, Ashish Upadhyay, Bernd Bohnet, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di JinDimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Laura Perez-Beltrachini, Leonardo F.R. Ribeiro, Lewis Tunstall, Li Zhang, Mahima Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. The compatibility, often facilitated through leaderboards, thus leads to outdated but standardized evaluation practices. We pose that the standardization is taking place in the wrong spot. Evaluation infrastructure should enable researchers to use the latest methods and what should be standardized instead is how to incorporate these new evaluation advances. We introduce GEMv2, the new version of the Generation, Evaluation, and Metrics Benchmark which uses a modular infrastructure for dataset, model, and metric developers to benefit from each other's work. GEMv2 supports 40 documented datasets in 51 languages, ongoing online evaluation for all datasets, and our interactive tools make it easier to add new datasets to the living benchmark.

Original language	English
Title of host publication	EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Demonstrations Session
Editors	Wanxiang Che, Ekaterina Shutova
Publisher	Association for Computational Linguistics (ACL)
Pages	266-281
Number of pages	16
ISBN (Electronic)	9781959429418
DOIs	https://doi.org/10.18653/v1/2022.emnlp-demos.27
State	Published - 2022
Externally published	Yes
Event	2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , EMNLP 2022 - Abu Dhabi, United Arab Emirates Duration: 7 Dec 2022 → 11 Dec 2022

Publication series

Name	EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Demonstrations Session

Conference

Conference	2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , EMNLP 2022
Country/Territory	United Arab Emirates
City	Abu Dhabi
Period	7/12/22 → 11/12/22

Bibliographical note

Publisher Copyright:
© 2022 Association for Computational Linguistics.

ASJC Scopus subject areas

Computational Theory and Mathematics
Computer Science Applications
Information Systems

Access to Document

10.18653/v1/2022.emnlp-demos.27

Cite this

Gehrmann, S., Bhattacharjee, A., Mahendiran, A., Wang, A., Papangelis, A., Madaan, A., McMillan-Major, A., Shvets, A., Upadhyay, A., Bohnet, B., Yao, B., Wilie, B., Bhagavatula, C., You, C., Thomson, C., Garbacea, C., Wang, D., Deutsch, D., Xiong, D., ... Hou, Y. (2022). GEMv2: Multilingual NLG Benchmarking in a Single Line of Code. In W. Che, & E. Shutova (Eds.), EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Demonstrations Session (pp. 266-281). (EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Demonstrations Session). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2022.emnlp-demos.27

Gehrmann, Sebastian ; Bhattacharjee, Abhik ; Mahendiran, Abinaya et al. / GEMv2 : Multilingual NLG Benchmarking in a Single Line of Code. EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Demonstrations Session. editor / Wanxiang Che ; Ekaterina Shutova. Association for Computational Linguistics (ACL), 2022. pp. 266-281 (EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Demonstrations Session).

@inproceedings{bee796a7d6ad436f8a7080a844d90b0b,

title = "GEMv2: Multilingual NLG Benchmarking in a Single Line of Code",

abstract = "Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. The compatibility, often facilitated through leaderboards, thus leads to outdated but standardized evaluation practices. We pose that the standardization is taking place in the wrong spot. Evaluation infrastructure should enable researchers to use the latest methods and what should be standardized instead is how to incorporate these new evaluation advances. We introduce GEMv2, the new version of the Generation, Evaluation, and Metrics Benchmark which uses a modular infrastructure for dataset, model, and metric developers to benefit from each other's work. GEMv2 supports 40 documented datasets in 51 languages, ongoing online evaluation for all datasets, and our interactive tools make it easier to add new datasets to the living benchmark.",

author = "Sebastian Gehrmann and Abhik Bhattacharjee and Abinaya Mahendiran and Alex Wang and Alexandros Papangelis and Aman Madaan and Angelina McMillan-Major and Anna Shvets and Ashish Upadhyay and Bernd Bohnet and Bingsheng Yao and Bryan Wilie and Chandra Bhagavatula and Chaobin You and Craig Thomson and Cristina Garbacea and Dakuo Wang and Daniel Deutsch and Deyi Xiong and Di Jin and Dimitra Gkatzia and Dragomir Radev and Elizabeth Clark and Esin Durmus and Faisal Ladhak and Filip Ginter and Winata, {Genta Indra} and Hendrik Strobelt and Hiroaki Hayashi and Jekaterina Novikova and Jenna Kanerva and Jenny Chim and Jiawei Zhou and Jordan Clive and Joshua Maynez and Jo{\~a}o Sedoc and Juraj Juraska and Kaustubh Dhole and Chandu, {Khyathi Raghavi} and Laura Perez-Beltrachini and Ribeiro, {Leonardo F.R.} and Lewis Tunstall and Li Zhang and Mahima Pushkarna and Mathias Creutz and Michael White and Kale, {Mihir Sanjay} and Eddine, {Moussa Kamal} and Nico Daheim and Nishant Subramani and Ondrej Dusek and Liang, {Paul Pu} and Ammanamanchi, {Pawan Sasanka} and Qi Zhu and Ratish Puduppully and Reno Kriz and Rifat Shahriyar and Ronald Cardenas and Saad Mahamood and Salomey Osei and Samuel Cahyawijaya and Sanja {\v S}tajner and Sebastien Montella and Shailza Jolly and Simon Mille and Tahmid Hasan and Tianhao Shen and Tosin Adewumi and Vikas Raunak and Vipul Raheja and Vitaly Nikolaev and Vivian Tsai and Yacine Jernite and Ying Xu and Yisi Sang and Yixin Liu and Yufang Hou",

note = "Publisher Copyright: {\textcopyright} 2022 Association for Computational Linguistics.; 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , EMNLP 2022 ; Conference date: 07-12-2022 Through 11-12-2022",

year = "2022",

doi = "10.18653/v1/2022.emnlp-demos.27",

language = "English",

series = "EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Demonstrations Session",

publisher = "Association for Computational Linguistics (ACL)",

pages = "266--281",

editor = "Wanxiang Che and Ekaterina Shutova",

booktitle = "EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Demonstrations Session",

address = "United States",

}

Gehrmann, S, Bhattacharjee, A, Mahendiran, A, Wang, A, Papangelis, A, Madaan, A, McMillan-Major, A, Shvets, A, Upadhyay, A, Bohnet, B, Yao, B, Wilie, B, Bhagavatula, C, You, C, Thomson, C, Garbacea, C, Wang, D, Deutsch, D, Xiong, D, Jin, D, Gkatzia, D, Radev, D, Clark, E, Durmus, E, Ladhak, F, Ginter, F, Winata, GI, Strobelt, H, Hayashi, H, Novikova, J, Kanerva, J, Chim, J, Zhou, J, Clive, J, Maynez, J, Sedoc, J, Juraska, J, Dhole, K, Chandu, KR, Perez-Beltrachini, L, Ribeiro, LFR, Tunstall, L, Zhang, L, Pushkarna, M, Creutz, M, White, M, Kale, MS, Eddine, MK, Daheim, N, Subramani, N, Dusek, O, Liang, PP, Ammanamanchi, PS, Zhu, Q, Puduppully, R, Kriz, R, Shahriyar, R, Cardenas, R, Mahamood, S, Osei, S, Cahyawijaya, S, Štajner, S, Montella, S, Jolly, S, Mille, S, Hasan, T, Shen, T, Adewumi, T, Raunak, V, Raheja, V, Nikolaev, V, Tsai, V, Jernite, Y, Xu, Y, Sang, Y, Liu, Y & Hou, Y 2022, GEMv2: Multilingual NLG Benchmarking in a Single Line of Code. in W Che & E Shutova (eds), EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Demonstrations Session. EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Demonstrations Session, Association for Computational Linguistics (ACL), pp. 266-281, 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , EMNLP 2022, Abu Dhabi, United Arab Emirates, 7/12/22. https://doi.org/10.18653/v1/2022.emnlp-demos.27

GEMv2: Multilingual NLG Benchmarking in a Single Line of Code. / Gehrmann, Sebastian; Bhattacharjee, Abhik; Mahendiran, Abinaya et al.
EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Demonstrations Session. ed. / Wanxiang Che; Ekaterina Shutova. Association for Computational Linguistics (ACL), 2022. p. 266-281 (EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Demonstrations Session).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - GEMv2

T2 - 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , EMNLP 2022

AU - Gehrmann, Sebastian

AU - Bhattacharjee, Abhik

AU - Mahendiran, Abinaya

AU - Wang, Alex

AU - Papangelis, Alexandros

AU - Madaan, Aman

AU - McMillan-Major, Angelina

AU - Shvets, Anna

AU - Upadhyay, Ashish

AU - Bohnet, Bernd

AU - Yao, Bingsheng

AU - Wilie, Bryan

AU - Bhagavatula, Chandra

AU - You, Chaobin

AU - Thomson, Craig

AU - Garbacea, Cristina

AU - Wang, Dakuo

AU - Deutsch, Daniel

AU - Xiong, Deyi

AU - Jin, Di

AU - Gkatzia, Dimitra

AU - Radev, Dragomir

AU - Clark, Elizabeth

AU - Durmus, Esin

AU - Ladhak, Faisal

AU - Ginter, Filip

AU - Winata, Genta Indra

AU - Strobelt, Hendrik

AU - Hayashi, Hiroaki

AU - Novikova, Jekaterina

AU - Kanerva, Jenna

AU - Chim, Jenny

AU - Zhou, Jiawei

AU - Clive, Jordan

AU - Maynez, Joshua

AU - Sedoc, João

AU - Juraska, Juraj

AU - Dhole, Kaustubh

AU - Chandu, Khyathi Raghavi

AU - Perez-Beltrachini, Laura

AU - Ribeiro, Leonardo F.R.

AU - Tunstall, Lewis

AU - Zhang, Li

AU - Pushkarna, Mahima

AU - Creutz, Mathias

AU - White, Michael

AU - Kale, Mihir Sanjay

AU - Eddine, Moussa Kamal

AU - Daheim, Nico

AU - Subramani, Nishant

AU - Dusek, Ondrej

AU - Liang, Paul Pu

AU - Ammanamanchi, Pawan Sasanka

AU - Zhu, Qi

AU - Puduppully, Ratish

AU - Kriz, Reno

AU - Shahriyar, Rifat

AU - Cardenas, Ronald

AU - Mahamood, Saad

AU - Osei, Salomey

AU - Cahyawijaya, Samuel

AU - Štajner, Sanja

AU - Montella, Sebastien

AU - Jolly, Shailza

AU - Mille, Simon

AU - Hasan, Tahmid

AU - Shen, Tianhao

AU - Adewumi, Tosin

AU - Raunak, Vikas

AU - Raheja, Vipul

AU - Nikolaev, Vitaly

AU - Tsai, Vivian

AU - Jernite, Yacine

AU - Xu, Ying

AU - Sang, Yisi

AU - Liu, Yixin

AU - Hou, Yufang

PY - 2022

Y1 - 2022

N2 - Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. The compatibility, often facilitated through leaderboards, thus leads to outdated but standardized evaluation practices. We pose that the standardization is taking place in the wrong spot. Evaluation infrastructure should enable researchers to use the latest methods and what should be standardized instead is how to incorporate these new evaluation advances. We introduce GEMv2, the new version of the Generation, Evaluation, and Metrics Benchmark which uses a modular infrastructure for dataset, model, and metric developers to benefit from each other's work. GEMv2 supports 40 documented datasets in 51 languages, ongoing online evaluation for all datasets, and our interactive tools make it easier to add new datasets to the living benchmark.

AB - Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. The compatibility, often facilitated through leaderboards, thus leads to outdated but standardized evaluation practices. We pose that the standardization is taking place in the wrong spot. Evaluation infrastructure should enable researchers to use the latest methods and what should be standardized instead is how to incorporate these new evaluation advances. We introduce GEMv2, the new version of the Generation, Evaluation, and Metrics Benchmark which uses a modular infrastructure for dataset, model, and metric developers to benefit from each other's work. GEMv2 supports 40 documented datasets in 51 languages, ongoing online evaluation for all datasets, and our interactive tools make it easier to add new datasets to the living benchmark.

UR - http://www.scopus.com/inward/record.url?scp=85149671144&partnerID=8YFLogxK

U2 - 10.18653/v1/2022.emnlp-demos.27

DO - 10.18653/v1/2022.emnlp-demos.27

M3 - Conference contribution

AN - SCOPUS:85149671144

T3 - EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Demonstrations Session

SP - 266

EP - 281

BT - EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Demonstrations Session

A2 - Che, Wanxiang

A2 - Shutova, Ekaterina

PB - Association for Computational Linguistics (ACL)

Y2 - 7 December 2022 through 11 December 2022

ER -

Gehrmann S, Bhattacharjee A, Mahendiran A, Wang A, Papangelis A, Madaan A et al. GEMv2: Multilingual NLG Benchmarking in a Single Line of Code. In Che W, Shutova E, editors, EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Demonstrations Session. Association for Computational Linguistics (ACL). 2022. p. 266-281. (EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Demonstrations Session). doi: 10.18653/v1/2022.emnlp-demos.27

GEMv2: Multilingual NLG Benchmarking in a Single Line of Code

Abstract

Publication series

Conference

Bibliographical note

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this