We Need to Talk About train-dev-test Splits

Publikation: Konference artikel i Proceeding eller bog/rapport kapitelKonferencebidrag i proceedingsForskningpeer review


Standard train-dev-test splits used to benchmark multiple models against each other are ubiquitously used in Natural Language Processing (NLP). In this setup, the train data is used for training the model, the development set for evaluating different versions of the proposed model(s) during development, and the test set to confirm the answers to the main research question(s). However, the introduction of neural networks in NLP has led to a different use of these standard splits; the development set is now often used for model selection during the training procedure. Because of this, comparing multiple versions of the same model during development leads to overestimation on the development data. As an effect, people have started to compare an increasing amount of models on the test data, leading to faster overfitting and ``expiration'' of our test sets. We propose to use a tune-set when developing neural network methods, which can be used for model picking so that comparing the different versions of a new model can safely be done on the development data.
TitelProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Antal sider9
ForlagAssociation for Computational Linguistics
Publikationsdatookt. 2021
StatusUdgivet - okt. 2021
BegivenhedThe 2021 Conference on Empirical Methods in Natural Language Processing - Punta Cana, Dominikanske Republik, Den
Varighed: 7 nov. 202112 nov. 2021


KonferenceThe 2021 Conference on Empirical Methods in Natural Language Processing
Land/OmrådeDominikanske Republik, Den
ByPunta Cana


Dyk ned i forskningsemnerne om 'We Need to Talk About train-dev-test Splits'. Sammen danner de et unikt fingeraftryk.