Maintaining quality in FEVER annotation

Henri Schulte, Julie Binau, Leon Derczynski

Research output: Conference Article in Proceeding or Book/Report chapterArticle in proceedingsResearchpeer-review


We propose two measures for measuring the quality of constructed claims in the FEVER task. Annotating data for this task involves the creation of supporting and refuting claims over a set of evidence. Automatic annotation processes often leave superficial patterns in data, which learning systems can detect instead of performing the underlying task. Humans also can leave these superficial patterns, either voluntarily or involuntarily (due to e.g. fatigue). The two measures introduced attempt to detect the impact of these superficial patterns. One is a new information-theoretic and distributionality based measure, DCI; and the other an extension of neural probing work over the ARCT task, utility. We demonstrate these measures over a recent major dataset, that from the English FEVER task in 2019.
Original languageEnglish
Title of host publicationProceedings of the Third Workshop on Fact Extraction and VERification (FEVER) : Association for Computational Linguistics
PublisherAssociation for Computational Linguistics
Publication date9 Jul 2020
ISBN (Electronic)978-1-952148-10-1
Publication statusPublished - 9 Jul 2020


  • Claim Quality Metrics
  • FEVER Task
  • Data Annotation
  • Superficial Patterns
  • Information-Theoretic Measure


Dive into the research topics of 'Maintaining quality in FEVER annotation'. Together they form a unique fingerprint.

Cite this