Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics

Prajjwal Bhargava; Aleksandr Drozd; Anna Rogers

Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics

Prajjwal Bhargava
, Aleksandr Drozd
, Anna Rogers

University of Texas at Austin
RIKEN Center for Computational Science
Tokyo Institute of Technology

Publikation: Konference artikel i Proceeding eller bog/rapport kapitel › Konferencebidrag i proceedings › Forskning › peer review

Abstract

Much of recent progress in NLU was shown to be due to models' learning dataset-specific heuristics. We conduct a case study of generalization in NLI (from MNLI to the adversarially constructed HANS dataset) in a range of BERT-based architectures (adapters, Siamese Transformers, HEX debiasing), as well as with subsampling the data and increasing the model size. We report 2 successful and 3 unsuccessful strategies, all providing insights into how Transformer-based models learn to generalize.

Originalsprog	Engelsk
Titel	Proceedings of the Second Workshop on Insights from Negative Results in NLP
Antal sider	11
Udgivelsessted	Online and Punta Cana, Dominican Republic
Forlag	Association for Computational Linguistics
Publikationsdato	1 nov. 2021
Sider	125-135
Status	Udgivet - 1 nov. 2021
Begivenhed	Insights from Negative Results in NLP - VIRTUAL, Dominikanske Republik, Den Varighed: 1 nov. 2021 → … Konferencens nummer: 2

Konference

Konference	Insights from Negative Results in NLP
Nummer	2
Land/Område	Dominikanske Republik, Den
By	VIRTUAL
Periode	01/11/2021 → …

Emneord

Natural Language Understanding
Generalization
BERT-based architectures
Adversarial robustness
Transformer models

Adgang til dokumentet

https://aclanthology.org/2021.insights-1.18

Citationsformater

@inproceedings{40cdd8350fe947eba133d72a41aacf01,

title = "Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics",

abstract = "Much of recent progress in NLU was shown to be due to models' learning dataset-specific heuristics. We conduct a case study of generalization in NLI (from MNLI to the adversarially constructed HANS dataset) in a range of BERT-based architectures (adapters, Siamese Transformers, HEX debiasing), as well as with subsampling the data and increasing the model size. We report 2 successful and 3 unsuccessful strategies, all providing insights into how Transformer-based models learn to generalize.",

keywords = "Natural Language Understanding, Generalization, BERT-based architectures, Adversarial robustness, Transformer models, Natural Language Understanding, Generalization, BERT-based architectures, Adversarial robustness, Transformer models",

author = "Prajjwal Bhargava and Aleksandr Drozd and Anna Rogers",

year = "2021",

month = nov,

day = "1",

language = "English",

pages = "125--135",

booktitle = "Proceedings of the Second Workshop on Insights from Negative Results in NLP",

publisher = "Association for Computational Linguistics",

address = "United States",

note = "Insights from Negative Results in NLP ; Conference date: 01-11-2021",

}

Bhargava, P, Drozd, A & Rogers, A 2021, Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics. i Proceedings of the Second Workshop on Insights from Negative Results in NLP. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, s. 125-135, Insights from Negative Results in NLP, VIRTUAL, Dominikanske Republik, Den, 01/11/2021. <https://aclanthology.org/2021.insights-1.18>

Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics. / Bhargava, Prajjwal; Drozd, Aleksandr; Rogers, Anna.
Proceedings of the Second Workshop on Insights from Negative Results in NLP. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, 2021. s. 125-135.

Publikation: Konference artikel i Proceeding eller bog/rapport kapitel › Konferencebidrag i proceedings › Forskning › peer review

TY - GEN

T1 - Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics

AU - Bhargava, Prajjwal

AU - Drozd, Aleksandr

AU - Rogers, Anna

N1 - Conference code: 2

PY - 2021/11/1

Y1 - 2021/11/1

N2 - Much of recent progress in NLU was shown to be due to models' learning dataset-specific heuristics. We conduct a case study of generalization in NLI (from MNLI to the adversarially constructed HANS dataset) in a range of BERT-based architectures (adapters, Siamese Transformers, HEX debiasing), as well as with subsampling the data and increasing the model size. We report 2 successful and 3 unsuccessful strategies, all providing insights into how Transformer-based models learn to generalize.

AB - Much of recent progress in NLU was shown to be due to models' learning dataset-specific heuristics. We conduct a case study of generalization in NLI (from MNLI to the adversarially constructed HANS dataset) in a range of BERT-based architectures (adapters, Siamese Transformers, HEX debiasing), as well as with subsampling the data and increasing the model size. We report 2 successful and 3 unsuccessful strategies, all providing insights into how Transformer-based models learn to generalize.

KW - Natural Language Understanding

KW - Generalization

KW - BERT-based architectures

KW - Adversarial robustness

KW - Transformer models

KW - Natural Language Understanding

KW - Generalization

KW - BERT-based architectures

KW - Adversarial robustness

KW - Transformer models

M3 - Article in proceedings

SP - 125

EP - 135

BT - Proceedings of the Second Workshop on Insights from Negative Results in NLP

PB - Association for Computational Linguistics

CY - Online and Punta Cana, Dominican Republic

T2 - Insights from Negative Results in NLP

Y2 - 1 November 2021

ER -

Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics

Abstract

Konference

Emneord

Adgang til dokumentet

Fingeraftryk

Citationsformater