When data permutations are pathological: the case of neural natural language inference

Natalie Schluter, Daniel Varab

Research output: Conference Article in Proceeding or Book/Report chapterArticle in proceedingsResearchpeer-review


Consider two competitive machine learning models, one of which was considered state-of-the art, and the other a competitive baseline. Suppose that by just permuting the examples of the training set, say by reversing the original order, by shuffling, or by mini-batching, you could report substantially better/worst performance for the system of your choice, by multiple percentage points. In this paper, we illustrate this scenario for a trending NLP task: Natural Language Inference (NLI). We show that for the two central NLI corpora today, the learning process of neural systems is far too sensitive to permutations of the
data. In doing so we reopen the question of how to judge a good neural architecture for NLI, given the available dataset and perhaps, further, the soundness of the NLI task itself in its current state.
Original languageEnglish
Title of host publicationProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Number of pages5
PublisherAssociation for Computational Linguistics
Publication date31 Oct 2018
ISBN (Electronic)978-1-948087-84-1
Publication statusPublished - 31 Oct 2018


Dive into the research topics of 'When data permutations are pathological: the case of neural natural language inference'. Together they form a unique fingerprint.

Cite this