Maintaining quality in FEVER annotation

Henri Schulte; Julie Binau; Leon Derczynski

Maintaining quality in FEVER annotation

Henri Schulte
, Julie Binau
, Leon Derczynski

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

Abstract

We propose two measures for measuring the quality of constructed claims in the FEVER task. Annotating data for this task involves the creation of supporting and refuting claims over a set of evidence. Automatic annotation processes often leave superficial patterns in data, which learning systems can detect instead of performing the underlying task. Humans also can leave these superficial patterns, either voluntarily or involuntarily (due to e.g. fatigue). The two measures introduced attempt to detect the impact of these superficial patterns. One is a new information-theoretic and distributionality based measure, DCI; and the other an extension of neural probing work over the ARCT task, utility. We demonstrate these measures over a recent major dataset, that from the English FEVER task in 2019.

Original language	English
Title of host publication	Proceedings of the Third Workshop on Fact Extraction and VERification (FEVER) : Association for Computational Linguistics
Publisher	Association for Computational Linguistics
Publication date	9 Jul 2020
Pages	42-46
ISBN (Electronic)	978-1-952148-10-1
Publication status	Published - 9 Jul 2020
Event	Workshop on Fact Extraction and VERification - Online Duration: 9 Jul 2020 → 9 Jul 2020 Conference number: 3rd

Workshop

Workshop	Workshop on Fact Extraction and VERification
Number	3rd
Location	Online
Period	09/07/2020 → 09/07/2020

Keywords

Claim Quality Metrics
FEVER Task
Data Annotation
Superficial Patterns
Information-Theoretic Measure

Access to Document

2020.fever-1.6Final published version, 225 KBLicence: CC BY

Cite this

@inproceedings{24f3e9859622405ab47db4ee4e341717,

title = "Maintaining quality in FEVER annotation",

abstract = "We propose two measures for measuring the quality of constructed claims in the FEVER task. Annotating data for this task involves the creation of supporting and refuting claims over a set of evidence. Automatic annotation processes often leave superficial patterns in data, which learning systems can detect instead of performing the underlying task. Humans also can leave these superficial patterns, either voluntarily or involuntarily (due to e.g. fatigue). The two measures introduced attempt to detect the impact of these superficial patterns. One is a new information-theoretic and distributionality based measure, DCI; and the other an extension of neural probing work over the ARCT task, utility. We demonstrate these measures over a recent major dataset, that from the English FEVER task in 2019.",

keywords = "Claim Quality Metrics, FEVER Task, Data Annotation, Superficial Patterns, Information-Theoretic Measure, Claim Quality Metrics, FEVER Task, Data Annotation, Superficial Patterns, Information-Theoretic Measure",

author = "Henri Schulte and Julie Binau and Leon Derczynski",

year = "2020",

month = jul,

day = "9",

language = "English",

pages = "42--46",

booktitle = "Proceedings of the Third Workshop on Fact Extraction and VERification (FEVER)",

publisher = "Association for Computational Linguistics",

address = "United States",

note = "Workshop on Fact Extraction and VERification ; Conference date: 09-07-2020 Through 09-07-2020",

}

Maintaining quality in FEVER annotation. / Schulte, Henri; Binau, Julie; Derczynski, Leon.
Proceedings of the Third Workshop on Fact Extraction and VERification (FEVER): Association for Computational Linguistics. Association for Computational Linguistics, 2020. p. 42-46.

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

TY - GEN

T1 - Maintaining quality in FEVER annotation

AU - Schulte, Henri

AU - Binau, Julie

AU - Derczynski, Leon

N1 - Conference code: 3rd

PY - 2020/7/9

Y1 - 2020/7/9

N2 - We propose two measures for measuring the quality of constructed claims in the FEVER task. Annotating data for this task involves the creation of supporting and refuting claims over a set of evidence. Automatic annotation processes often leave superficial patterns in data, which learning systems can detect instead of performing the underlying task. Humans also can leave these superficial patterns, either voluntarily or involuntarily (due to e.g. fatigue). The two measures introduced attempt to detect the impact of these superficial patterns. One is a new information-theoretic and distributionality based measure, DCI; and the other an extension of neural probing work over the ARCT task, utility. We demonstrate these measures over a recent major dataset, that from the English FEVER task in 2019.

AB - We propose two measures for measuring the quality of constructed claims in the FEVER task. Annotating data for this task involves the creation of supporting and refuting claims over a set of evidence. Automatic annotation processes often leave superficial patterns in data, which learning systems can detect instead of performing the underlying task. Humans also can leave these superficial patterns, either voluntarily or involuntarily (due to e.g. fatigue). The two measures introduced attempt to detect the impact of these superficial patterns. One is a new information-theoretic and distributionality based measure, DCI; and the other an extension of neural probing work over the ARCT task, utility. We demonstrate these measures over a recent major dataset, that from the English FEVER task in 2019.

KW - Claim Quality Metrics

KW - FEVER Task

KW - Data Annotation

KW - Superficial Patterns

KW - Information-Theoretic Measure

KW - Claim Quality Metrics

KW - FEVER Task

KW - Data Annotation

KW - Superficial Patterns

KW - Information-Theoretic Measure

M3 - Article in proceedings

SP - 42

EP - 46

BT - Proceedings of the Third Workshop on Fact Extraction and VERification (FEVER)

PB - Association for Computational Linguistics

T2 - Workshop on Fact Extraction and VERification

Y2 - 9 July 2020 through 9 July 2020

ER -

Maintaining quality in FEVER annotation

Abstract

Workshop

Keywords

Access to Document

Fingerprint

Cite this