A taxonomy and review of generalization research in NLP

Dieuwke Hupkes; Mario Giulianelli; Verna Dankers; Mikel Artetxe; Yanai Elazar; Tiago Pimentel; Christos Christodoulopoulos; Karim Lasri; Naomi Saphra; Arabella Sinclair; Dennis Thomas Ulmer; Florian Schottmann; Khuyagbaatar Batsuren; Kaiser Sun; Koustouv Sinha; Leila Khalatbari; Maria Ryskina; Ryan Cotterell; Zhijing Jin

A taxonomy and review of generalization research in NLP

Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Thomas Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustouv Sinha, Leila Khalatbari, Maria Ryskina, Ryan Cotterell, Zhijing Jin

Publikation: Konferencebidrag - EJ publiceret i proceeding eller tidsskrift › Paper › Forskning › peer review

Abstract

The ability to generalize well is one of the primary desiderata for models of natural language processing (NLP), but what ‘good generalization’ entails and how it should be evaluated is not well understood. In this Analysis we present a taxonomy for characterizing and understanding generalization research in NLP. The proposed taxonomy is based on an extensive literature review and contains five axes along which generalization studies can differ: their main motivation, the type of generalization they aim to solve, the type of data shift they consider, the source by which this data shift originated, and the locus of the shift within the NLP modelling pipeline. We use our taxonomy to classify over 700 experiments, and we use the results to present an in-depth analysis that maps out the current state of generalization research in NLP and make recommendations for which areas deserve attention in the future.

Originalsprog	Engelsk
Publikationsdato	1 okt. 2023
Status	Udgivet - 1 okt. 2023

Emneord

Generalization in NLP
Taxonomy of Generalization
Data Shift Origins
Natural Language Processing Evaluation
Generalization Research Analysis

Adgang til dokumentet

s42256-023-00729-yForlagets udgivne version, 2,06 MB

https://www.nature.com/articles/s42256-023-00729-y

Citationsformater

Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., Ulmer, D. T., Schottmann, F., Batsuren, K., Sun, K., Sinha, K., Khalatbari, L., Ryskina, M., Cotterell, R., & Jin, Z. (2023). A taxonomy and review of generalization research in NLP. https://www.nature.com/articles/s42256-023-00729-y

@conference{a0dbba8b3f2a4b45b0fb8812244ede95,

title = "A taxonomy and review of generalization research in NLP",

abstract = "The ability to generalize well is one of the primary desiderata for models of natural language processing (NLP), but what {\textquoteleft}good generalization{\textquoteright} entails and how it should be evaluated is not well understood. In this Analysis we present a taxonomy for characterizing and understanding generalization research in NLP. The proposed taxonomy is based on an extensive literature review and contains five axes along which generalization studies can differ: their main motivation, the type of generalization they aim to solve, the type of data shift they consider, the source by which this data shift originated, and the locus of the shift within the NLP modelling pipeline. We use our taxonomy to classify over 700 experiments, and we use the results to present an in-depth analysis that maps out the current state of generalization research in NLP and make recommendations for which areas deserve attention in the future.",

keywords = "Generalization in NLP, Taxonomy of Generalization, Data Shift Origins, Natural Language Processing Evaluation, Generalization Research Analysis, Generalization in NLP, Taxonomy of Generalization, Data Shift Origins, Natural Language Processing Evaluation, Generalization Research Analysis",

author = "Dieuwke Hupkes and Mario Giulianelli and Verna Dankers and Mikel Artetxe and Yanai Elazar and Tiago Pimentel and Christos Christodoulopoulos and Karim Lasri and Naomi Saphra and Arabella Sinclair and Ulmer, {Dennis Thomas} and Florian Schottmann and Khuyagbaatar Batsuren and Kaiser Sun and Koustouv Sinha and Leila Khalatbari and Maria Ryskina and Ryan Cotterell and Zhijing Jin",

year = "2023",

month = oct,

day = "1",

language = "English",

}

TY - CONF

T1 - A taxonomy and review of generalization research in NLP

AU - Hupkes, Dieuwke

AU - Giulianelli, Mario

AU - Dankers, Verna

AU - Artetxe, Mikel

AU - Elazar, Yanai

AU - Pimentel, Tiago

AU - Christodoulopoulos, Christos

AU - Lasri, Karim

AU - Saphra, Naomi

AU - Sinclair, Arabella

AU - Ulmer, Dennis Thomas

AU - Schottmann, Florian

AU - Batsuren, Khuyagbaatar

AU - Sun, Kaiser

AU - Sinha, Koustouv

AU - Khalatbari, Leila

AU - Ryskina, Maria

AU - Cotterell, Ryan

AU - Jin, Zhijing

PY - 2023/10/1

Y1 - 2023/10/1

N2 - The ability to generalize well is one of the primary desiderata for models of natural language processing (NLP), but what ‘good generalization’ entails and how it should be evaluated is not well understood. In this Analysis we present a taxonomy for characterizing and understanding generalization research in NLP. The proposed taxonomy is based on an extensive literature review and contains five axes along which generalization studies can differ: their main motivation, the type of generalization they aim to solve, the type of data shift they consider, the source by which this data shift originated, and the locus of the shift within the NLP modelling pipeline. We use our taxonomy to classify over 700 experiments, and we use the results to present an in-depth analysis that maps out the current state of generalization research in NLP and make recommendations for which areas deserve attention in the future.

AB - The ability to generalize well is one of the primary desiderata for models of natural language processing (NLP), but what ‘good generalization’ entails and how it should be evaluated is not well understood. In this Analysis we present a taxonomy for characterizing and understanding generalization research in NLP. The proposed taxonomy is based on an extensive literature review and contains five axes along which generalization studies can differ: their main motivation, the type of generalization they aim to solve, the type of data shift they consider, the source by which this data shift originated, and the locus of the shift within the NLP modelling pipeline. We use our taxonomy to classify over 700 experiments, and we use the results to present an in-depth analysis that maps out the current state of generalization research in NLP and make recommendations for which areas deserve attention in the future.

KW - Generalization in NLP

KW - Taxonomy of Generalization

KW - Data Shift Origins

KW - Natural Language Processing Evaluation

KW - Generalization Research Analysis

KW - Generalization in NLP

KW - Taxonomy of Generalization

KW - Data Shift Origins

KW - Natural Language Processing Evaluation

KW - Generalization Research Analysis

M3 - Paper

ER -

A taxonomy and review of generalization research in NLP

Abstract

Emneord

Adgang til dokumentet

Fingeraftryk

Citationsformater