We Need to Talk About train-dev-test Splits

Publikation: Konference artikel i Proceeding eller bog/rapport kapitelKonferencebidrag i proceedingsForskningpeer review

Abstrakt

Standard train-dev-test splits used to benchmark multiple models against each other are ubiquitously used in Natural Language Processing (NLP). In this setup, the train data is used for training the model, the development set for evaluating different versions of the proposed model(s) during development, and the test set to confirm the answers to the main research question(s). However, the introduction of neural networks in NLP has led to a different use of these standard splits; the development set is now often used for model selection during the training procedure. Because of this, comparing multiple versions of the same model during development leads to overestimation on the development data. As an effect, people have started to compare an increasing amount of models on the test data, leading to faster overfitting and ``expiration'' of our test sets. We propose to use a tune-set when developing neural network methods, which can be used for model picking so that comparing the different versions of a new model can safely be done on the development data.
OriginalsprogEngelsk
TitelProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Antal sider9
ForlagAssociation for Computational Linguistics
Publikationsdatookt. 2021
Sider4485
StatusUdgivet - okt. 2021
Begivenhedthe 2021 Conference on Empirical Methods in Natural Language Processing -
Varighed: 7 nov. 202111 nov. 2021

Konference

Konferencethe 2021 Conference on Empirical Methods in Natural Language Processing
Periode07/11/202111/11/2021

Fingeraftryk

Dyk ned i forskningsemnerne om 'We Need to Talk About train-dev-test Splits'. Sammen danner de et unikt fingeraftryk.

Citationsformater