Recall is the Proper Evaluation Metric for Word Segmentation

Yan Shao; Christian Hardmeier; Joakim Nivre

Recall is the Proper Evaluation Metric for Word Segmentation

Yan Shao
, Christian Hardmeier
, Joakim Nivre

Uppsala University

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

Abstract

We extensively analyse the correlations and drawbacks of conventionally employed evaluation metrics for word segmentation. Unlike in standard information retrieval, precision favours under-splitting systems and therefore can be misleading in word segmentation. Overall, based on both theoretical and experimental analysis, we propose that precision should be excluded from the standard evaluation metrics and that the evaluation score obtained by using only recall is sufficient and better correlated with the performance of word segmentation systems.

Original language	English
Title of host publication	Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)
Number of pages	5
Publication date	1 Dec 2017
Pages	86–90
ISBN (Print)	978-1-948087-01-8
Publication status	Published - 1 Dec 2017
Externally published	Yes

Keywords

Word segmentation
Evaluation metrics
Recall-based evaluation
Under-splitting
Information retrieval

Access to Document

https://aclanthology.org/I17-2015.pdf

Cite this

@inproceedings{4c76e6b3764f4d09ad0a3bff57f52158,

title = "Recall is the Proper Evaluation Metric for Word Segmentation",

abstract = "We extensively analyse the correlations and drawbacks of conventionally employed evaluation metrics for word segmentation. Unlike in standard information retrieval, precision favours under-splitting systems and therefore can be misleading in word segmentation. Overall, based on both theoretical and experimental analysis, we propose that precision should be excluded from the standard evaluation metrics and that the evaluation score obtained by using only recall is sufficient and better correlated with the performance of word segmentation systems.",

keywords = "Word segmentation, Evaluation metrics, Recall-based evaluation, Under-splitting, Information retrieval",

author = "Yan Shao and Christian Hardmeier and Joakim Nivre",

year = "2017",

month = dec,

day = "1",

language = "English",

isbn = "978-1-948087-01-8 ",

pages = "86–90",

booktitle = "Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)",

}

Recall is the Proper Evaluation Metric for Word Segmentation. / Shao, Yan; Hardmeier, Christian; Nivre, Joakim.
Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers). 2017. p. 86–90.

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

TY - GEN

T1 - Recall is the Proper Evaluation Metric for Word Segmentation

AU - Shao, Yan

AU - Hardmeier, Christian

AU - Nivre, Joakim

PY - 2017/12/1

Y1 - 2017/12/1

N2 - We extensively analyse the correlations and drawbacks of conventionally employed evaluation metrics for word segmentation. Unlike in standard information retrieval, precision favours under-splitting systems and therefore can be misleading in word segmentation. Overall, based on both theoretical and experimental analysis, we propose that precision should be excluded from the standard evaluation metrics and that the evaluation score obtained by using only recall is sufficient and better correlated with the performance of word segmentation systems.

AB - We extensively analyse the correlations and drawbacks of conventionally employed evaluation metrics for word segmentation. Unlike in standard information retrieval, precision favours under-splitting systems and therefore can be misleading in word segmentation. Overall, based on both theoretical and experimental analysis, we propose that precision should be excluded from the standard evaluation metrics and that the evaluation score obtained by using only recall is sufficient and better correlated with the performance of word segmentation systems.

KW - Word segmentation

KW - Evaluation metrics

KW - Recall-based evaluation

KW - Under-splitting

KW - Information retrieval

M3 - Article in proceedings

SN - 978-1-948087-01-8

SP - 86

EP - 90

BT - Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)

ER -

Recall is the Proper Evaluation Metric for Word Segmentation

Abstract

Keywords

Access to Document

Fingerprint

Cite this