A Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization

Lining Zhang; Simon Mille; Yufang Hou; Daniel Deutsch; Elizabeth Clark; Yixin Liu; Saad Mahamood; Sebastian Gehrmann; Miruna Clinciu; Khyathi Chandu; João Sedoc

A Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization

Lining Zhang, Simon Mille, Yufang Hou, Daniel Deutsch, Elizabeth Clark, Yixin Liu, Saad Mahamood, Sebastian Gehrmann, Miruna Clinciu, Khyathi Chandu, João Sedoc

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

To prevent the costly and inefficient use of resources on low-quality annotations, we want a method for creating a pool of dependable annotators who can effectively complete difficult tasks, such as evaluating automatic summarization. Thus, we investigate the recruitment of high-quality Amazon Mechanical Turk workers via a two-step pipeline. We show that we can successfully filter out subpar workers before they carry out the evaluations and obtain high-agreement annotations with similar constraints on resources. Although our workers demonstrate a strong consensus among themselves and CloudResearch workers, their alignment with expert judgments on a subset of the data is not as expected and needs further training in correctness. This paper still serves as a best practice for the recruitment of qualified annotators in other challenging annotation tasks.

Original language	English
Title of host publication	Long Papers
Publisher	Association for Computational Linguistics (ACL)
Pages	14944-14982
Number of pages	39
ISBN (Electronic)	9781959429722
State	Published - 2023
Externally published	Yes
Event	61st Annual Meeting of the Association for Computational Linguistics, ACL 2023 - Toronto, Canada Duration: 9 Jul 2023 → 14 Jul 2023

Publication series

Name	Proceedings of the Annual Meeting of the Association for Computational Linguistics
Volume	1
ISSN (Print)	0736-587X

Conference

Conference	61st Annual Meeting of the Association for Computational Linguistics, ACL 2023
Country/Territory	Canada
City	Toronto
Period	9/07/23 → 14/07/23

Bibliographical note

Publisher Copyright:
© 2023 Association for Computational Linguistics.

ASJC Scopus subject areas

Computer Science Applications
Linguistics and Language
Language and Linguistics

Cite this

Zhang, L., Mille, S., Hou, Y., Deutsch, D., Clark, E., Liu, Y., Mahamood, S., Gehrmann, S., Clinciu, M., Chandu, K., & Sedoc, J. (2023). A Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization. In Long Papers (pp. 14944-14982). (Proceedings of the Annual Meeting of the Association for Computational Linguistics; Vol. 1). Association for Computational Linguistics (ACL).

@inproceedings{0744bdd688fe40b7b96694b40a0a135c,

title = "A Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization",

abstract = "To prevent the costly and inefficient use of resources on low-quality annotations, we want a method for creating a pool of dependable annotators who can effectively complete difficult tasks, such as evaluating automatic summarization. Thus, we investigate the recruitment of high-quality Amazon Mechanical Turk workers via a two-step pipeline. We show that we can successfully filter out subpar workers before they carry out the evaluations and obtain high-agreement annotations with similar constraints on resources. Although our workers demonstrate a strong consensus among themselves and CloudResearch workers, their alignment with expert judgments on a subset of the data is not as expected and needs further training in correctness. This paper still serves as a best practice for the recruitment of qualified annotators in other challenging annotation tasks.",

author = "Lining Zhang and Simon Mille and Yufang Hou and Daniel Deutsch and Elizabeth Clark and Yixin Liu and Saad Mahamood and Sebastian Gehrmann and Miruna Clinciu and Khyathi Chandu and Jo{\~a}o Sedoc",

note = "Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.; 61st Annual Meeting of the Association for Computational Linguistics, ACL 2023 ; Conference date: 09-07-2023 Through 14-07-2023",

year = "2023",

language = "English",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics (ACL)",

pages = "14944--14982",

booktitle = "Long Papers",

address = "United States",

}

Zhang, L, Mille, S, Hou, Y, Deutsch, D, Clark, E, Liu, Y, Mahamood, S, Gehrmann, S, Clinciu, M, Chandu, K & Sedoc, J 2023, A Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization. in Long Papers. Proceedings of the Annual Meeting of the Association for Computational Linguistics, vol. 1, Association for Computational Linguistics (ACL), pp. 14944-14982, 61st Annual Meeting of the Association for Computational Linguistics, ACL 2023, Toronto, Canada, 9/07/23.

A Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization. / Zhang, Lining; Mille, Simon; Hou, Yufang et al.
Long Papers. Association for Computational Linguistics (ACL), 2023. p. 14944-14982 (Proceedings of the Annual Meeting of the Association for Computational Linguistics; Vol. 1).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A Needle in a Haystack

T2 - 61st Annual Meeting of the Association for Computational Linguistics, ACL 2023

AU - Zhang, Lining

AU - Mille, Simon

AU - Hou, Yufang

AU - Deutsch, Daniel

AU - Clark, Elizabeth

AU - Liu, Yixin

AU - Mahamood, Saad

AU - Gehrmann, Sebastian

AU - Clinciu, Miruna

AU - Chandu, Khyathi

AU - Sedoc, João

PY - 2023

Y1 - 2023

N2 - To prevent the costly and inefficient use of resources on low-quality annotations, we want a method for creating a pool of dependable annotators who can effectively complete difficult tasks, such as evaluating automatic summarization. Thus, we investigate the recruitment of high-quality Amazon Mechanical Turk workers via a two-step pipeline. We show that we can successfully filter out subpar workers before they carry out the evaluations and obtain high-agreement annotations with similar constraints on resources. Although our workers demonstrate a strong consensus among themselves and CloudResearch workers, their alignment with expert judgments on a subset of the data is not as expected and needs further training in correctness. This paper still serves as a best practice for the recruitment of qualified annotators in other challenging annotation tasks.

AB - To prevent the costly and inefficient use of resources on low-quality annotations, we want a method for creating a pool of dependable annotators who can effectively complete difficult tasks, such as evaluating automatic summarization. Thus, we investigate the recruitment of high-quality Amazon Mechanical Turk workers via a two-step pipeline. We show that we can successfully filter out subpar workers before they carry out the evaluations and obtain high-agreement annotations with similar constraints on resources. Although our workers demonstrate a strong consensus among themselves and CloudResearch workers, their alignment with expert judgments on a subset of the data is not as expected and needs further training in correctness. This paper still serves as a best practice for the recruitment of qualified annotators in other challenging annotation tasks.

UR - http://www.scopus.com/inward/record.url?scp=85174401681&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85174401681

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 14944

EP - 14982

BT - Long Papers

PB - Association for Computational Linguistics (ACL)

Y2 - 9 July 2023 through 14 July 2023

ER -

A Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization

Abstract

Publication series

Conference

Bibliographical note

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this