Automatic reference-based evaluation of pronoun translation misses the point

Liane Guillou; Christian Hardmeier

doi:10.18653/v1/D18-1513

Automatic reference-based evaluation of pronoun translation misses the point

Liane Guillou
, Christian Hardmeier

Publikation: Konference artikel i Proceeding eller bog/rapport kapitel › Konferencebidrag i proceedings › Forskning › peer review

Abstract

We compare the performance of the APT and AutoPRF metrics for pronoun translation against a manually annotated dataset comprising human judgements as to the correctness of translations of the PROTEST test suite. Although there is some correlation with the human judgements, a range of issues limit the performance of the automated metrics. Instead, we recommend the use of semiautomatic metrics and test suites in place of fully automatic metrics.

Originalsprog	Engelsk
Titel	Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018
Publikationsdato	2018
ISBN (Trykt)	9781948087841
DOI	https://doi.org/10.18653/v1/D18-1513
Status	Udgivet - 2018
Udgivet eksternt	Ja

Adgang til dokumentet

10.18653/v1/D18-1513

Citationsformater

@inproceedings{67eb82a9b6d8434ab43dc74207318247,

title = "Automatic reference-based evaluation of pronoun translation misses the point",

abstract = "We compare the performance of the APT and AutoPRF metrics for pronoun translation against a manually annotated dataset comprising human judgements as to the correctness of translations of the PROTEST test suite. Although there is some correlation with the human judgements, a range of issues limit the performance of the automated metrics. Instead, we recommend the use of semiautomatic metrics and test suites in place of fully automatic metrics.",

keywords = "pronoun translation evaluation, APT, AutoPRF, PROTEST test suite, semiautomatic metrics",

author = "Liane Guillou and Christian Hardmeier",

year = "2018",

doi = "10.18653/v1/D18-1513",

language = "English",

isbn = "9781948087841",

booktitle = "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018",

}

TY - GEN

T1 - Automatic reference-based evaluation of pronoun translation misses the point

AU - Guillou, Liane

AU - Hardmeier, Christian

PY - 2018

Y1 - 2018

N2 - We compare the performance of the APT and AutoPRF metrics for pronoun translation against a manually annotated dataset comprising human judgements as to the correctness of translations of the PROTEST test suite. Although there is some correlation with the human judgements, a range of issues limit the performance of the automated metrics. Instead, we recommend the use of semiautomatic metrics and test suites in place of fully automatic metrics.

AB - We compare the performance of the APT and AutoPRF metrics for pronoun translation against a manually annotated dataset comprising human judgements as to the correctness of translations of the PROTEST test suite. Although there is some correlation with the human judgements, a range of issues limit the performance of the automated metrics. Instead, we recommend the use of semiautomatic metrics and test suites in place of fully automatic metrics.

KW - pronoun translation evaluation

KW - APT

KW - AutoPRF

KW - PROTEST test suite

KW - semiautomatic metrics

UR - https://www.scopus.com/pages/publications/85081746897

U2 - 10.18653/v1/D18-1513

DO - 10.18653/v1/D18-1513

M3 - Article in proceedings

SN - 9781948087841

BT - Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018

ER -

Automatic reference-based evaluation of pronoun translation misses the point

Abstract

Adgang til dokumentet

Fingeraftryk

Citationsformater