We Need to Consider Disagreement in Evaluation

Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, Alexandra Uma

Research output: Conference Article in Proceeding or Book/Report chapterArticle in proceedingsResearchpeer-review


Evaluation is of paramount importance in data- driven research fields such as Natural Language Processing (NLP) and Computer Vision (CV). But current evaluation practice in NLP, except for end-to-end tasks such as machine translation, spoken dialogue systems, or NLG, largely hinges on the existence of a single “ground truth” against which we can meaning- fully compare the prediction of a model. However, this assumption is flawed for two reasons. 1) In many cases, more than one answer is correct. 2) Even where there is a single answer, disagreement among annotators is ubiquitous, making it difficult to decide on a gold standard. We discuss three sources of disagreement: from the annotator, the data, and the con- text, and show how this affects even seemingly objective tasks. Current methods of adjudication, agreement, and evaluation ought to be re- considered at the light of this evidence. Some researchers now propose to address this issue by minimizing disagreement, creating cleaner datasets. We argue that such a simplification is likely to result in oversimplified models just as much as it would do for end-to-end tasks such as machine translation. Instead, we suggest that we need to improve today’s evaluation practice to better capture such disagreement. Datasets with multiple annotations are becoming more common, as are methods to integrate disagreement into modeling. The logical next step is to extend this to evaluation.
Original languageEnglish
Title of host publicationACL-IJCNLP2021 Workshop on Benchmarking: Past, Present and Future
PublisherAssociation for Computational Linguistics
Publication date2021
Publication statusPublished - 2021


Dive into the research topics of 'We Need to Consider Disagreement in Evaluation'. Together they form a unique fingerprint.

Cite this