Automatic reference-based evaluation of pronoun translation misses the point

Liane Guillou; Christian Hardmeier

doi:10.18653/v1/D18-1513

Automatic reference-based evaluation of pronoun translation misses the point

Liane Guillou
, Christian Hardmeier

University of Edinburgh
Uppsala University

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

Abstract

We compare the performance of the APT and AutoPRF metrics for pronoun translation against a manually annotated dataset comprising human judgements as to the correctness of translations of the PROTEST test suite. Although there is some correlation with the human judgements, a range of issues limit the performance of the automated metrics. Instead, we recommend the use of semiautomatic metrics and test suites in place of fully automatic metrics.

Original language	English
Title of host publication	Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018
Publication date	2018
ISBN (Print)	9781948087841
DOIs	https://doi.org/10.18653/v1/D18-1513
Publication status	Published - 2018
Externally published	Yes

Keywords

pronoun translation evaluation
APT
AutoPRF
PROTEST test suite
semiautomatic metrics

Access to Document

10.18653/v1/D18-1513

Cite this

@inproceedings{67eb82a9b6d8434ab43dc74207318247,

title = "Automatic reference-based evaluation of pronoun translation misses the point",

abstract = "We compare the performance of the APT and AutoPRF metrics for pronoun translation against a manually annotated dataset comprising human judgements as to the correctness of translations of the PROTEST test suite. Although there is some correlation with the human judgements, a range of issues limit the performance of the automated metrics. Instead, we recommend the use of semiautomatic metrics and test suites in place of fully automatic metrics.",

keywords = "pronoun translation evaluation, APT, AutoPRF, PROTEST test suite, semiautomatic metrics",

author = "Liane Guillou and Christian Hardmeier",

year = "2018",

doi = "10.18653/v1/D18-1513",

language = "English",

isbn = "9781948087841",

booktitle = "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018",

}

TY - GEN

T1 - Automatic reference-based evaluation of pronoun translation misses the point

AU - Guillou, Liane

AU - Hardmeier, Christian

PY - 2018

Y1 - 2018

N2 - We compare the performance of the APT and AutoPRF metrics for pronoun translation against a manually annotated dataset comprising human judgements as to the correctness of translations of the PROTEST test suite. Although there is some correlation with the human judgements, a range of issues limit the performance of the automated metrics. Instead, we recommend the use of semiautomatic metrics and test suites in place of fully automatic metrics.

AB - We compare the performance of the APT and AutoPRF metrics for pronoun translation against a manually annotated dataset comprising human judgements as to the correctness of translations of the PROTEST test suite. Although there is some correlation with the human judgements, a range of issues limit the performance of the automated metrics. Instead, we recommend the use of semiautomatic metrics and test suites in place of fully automatic metrics.

KW - pronoun translation evaluation

KW - APT

KW - AutoPRF

KW - PROTEST test suite

KW - semiautomatic metrics

UR - https://www.scopus.com/pages/publications/85081746897

U2 - 10.18653/v1/D18-1513

DO - 10.18653/v1/D18-1513

M3 - Article in proceedings

SN - 9781948087841

BT - Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018

ER -

Automatic reference-based evaluation of pronoun translation misses the point

Abstract

Keywords

Access to Document

Fingerprint

Cite this