Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics

Prajjwal Bhargava; Aleksandr Drozd; Anna Rogers

Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics

Prajjwal Bhargava, Aleksandr Drozd, Anna Rogers

Publikation: Konference artikel i Proceeding eller bog/rapport kapitel › Konferencebidrag i proceedings › Forskning › peer review

Abstract

Much of recent progress in NLU was shown to be due to models' learning dataset-specific heuristics. We conduct a case study of generalization in NLI (from MNLI to the adversarially constructed HANS dataset) in a range of BERT-based architectures (adapters, Siamese Transformers, HEX debiasing), as well as with subsampling the data and increasing the model size. We report 2 successful and 3 unsuccessful strategies, all providing insights into how Transformer-based models learn to generalize.

Originalsprog	Engelsk
Titel	Proceedings of the Second Workshop on Insights from Negative Results in NLP
Antal sider	11
Udgivelsessted	Online and Punta Cana, Dominican Republic
Forlag	Association for Computational Linguistics
Publikationsdato	1 nov. 2021
Sider	125-135
Status	Udgivet - 1 nov. 2021

Emneord

Natural Language Understanding
Generalization
BERT-based architectures
Adversarial robustness
Transformer models

Adgang til dokumentet

https://aclanthology.org/2021.insights-1.18

Citationsformater

@inproceedings{40cdd8350fe947eba133d72a41aacf01,

title = "Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics",

abstract = "Much of recent progress in NLU was shown to be due to models' learning dataset-specific heuristics. We conduct a case study of generalization in NLI (from MNLI to the adversarially constructed HANS dataset) in a range of BERT-based architectures (adapters, Siamese Transformers, HEX debiasing), as well as with subsampling the data and increasing the model size. We report 2 successful and 3 unsuccessful strategies, all providing insights into how Transformer-based models learn to generalize.",

keywords = "Natural Language Understanding, Generalization, BERT-based architectures, Adversarial robustness, Transformer models, Natural Language Understanding, Generalization, BERT-based architectures, Adversarial robustness, Transformer models",

author = "Prajjwal Bhargava and Aleksandr Drozd and Anna Rogers",

year = "2021",

month = nov,

day = "1",

language = "English",

pages = "125--135",

booktitle = "Proceedings of the Second Workshop on Insights from Negative Results in NLP",

publisher = "Association for Computational Linguistics",

address = "United States",

}

Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics. / Bhargava, Prajjwal; Drozd, Aleksandr; Rogers, Anna.
Proceedings of the Second Workshop on Insights from Negative Results in NLP. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, 2021. s. 125-135.

Publikation: Konference artikel i Proceeding eller bog/rapport kapitel › Konferencebidrag i proceedings › Forskning › peer review

TY - GEN

T1 - Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics

AU - Bhargava, Prajjwal

AU - Drozd, Aleksandr

AU - Rogers, Anna

PY - 2021/11/1

Y1 - 2021/11/1

N2 - Much of recent progress in NLU was shown to be due to models' learning dataset-specific heuristics. We conduct a case study of generalization in NLI (from MNLI to the adversarially constructed HANS dataset) in a range of BERT-based architectures (adapters, Siamese Transformers, HEX debiasing), as well as with subsampling the data and increasing the model size. We report 2 successful and 3 unsuccessful strategies, all providing insights into how Transformer-based models learn to generalize.

AB - Much of recent progress in NLU was shown to be due to models' learning dataset-specific heuristics. We conduct a case study of generalization in NLI (from MNLI to the adversarially constructed HANS dataset) in a range of BERT-based architectures (adapters, Siamese Transformers, HEX debiasing), as well as with subsampling the data and increasing the model size. We report 2 successful and 3 unsuccessful strategies, all providing insights into how Transformer-based models learn to generalize.

KW - Natural Language Understanding

KW - Generalization

KW - BERT-based architectures

KW - Adversarial robustness

KW - Transformer models

KW - Natural Language Understanding

KW - Generalization

KW - BERT-based architectures

KW - Adversarial robustness

KW - Transformer models

M3 - Article in proceedings

SP - 125

EP - 135

BT - Proceedings of the Second Workshop on Insights from Negative Results in NLP

PB - Association for Computational Linguistics

CY - Online and Punta Cana, Dominican Republic

ER -

Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics

Abstract

Emneord

Adgang til dokumentet

Fingeraftryk

Citationsformater