BERT Busters: Outlier Dimensions That Disrupt Transformers

Olga Kovaleva; Saurabh Kulshreshtha; Anna Rogers; Anna Rumshisky

BERT Busters: Outlier Dimensions That Disrupt Transformers

Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, Anna Rumshisky

University of Massachusetts

Publikation: Konference artikel i Proceeding eller bog/rapport kapitel › Konferencebidrag i proceedings › Forskning › peer review

Abstract

Multiple studies have shown that Transformers are remarkably robust to pruning. Contrary to this received wisdom, we demonstrate that pre-trained Transformer encoders are surprisingly fragile to the removal of a very small number of features in the layer outputs (

Originalsprog	Engelsk
Titel	Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021
Antal sider	14
Udgivelsessted	Online
Forlag	Association for Computational Linguistics
Publikationsdato	1 aug. 2021
Sider	3392-3405
Status	Udgivet - 1 aug. 2021

Emneord

Transformers robustness
Pre-trained models
Layer outputs
Feature pruning
Model fragility

Adgang til dokumentet

https://aclanthology.org/2021.findings-acl.300

Citationsformater

@inproceedings{3ec033a5b6aa42688fcaad93977d6249,

title = "BERT Busters: Outlier Dimensions That Disrupt Transformers",

abstract = "Multiple studies have shown that Transformers are remarkably robust to pruning. Contrary to this received wisdom, we demonstrate that pre-trained Transformer encoders are surprisingly fragile to the removal of a very small number of features in the layer outputs (",

keywords = "Transformers robustness, Pre-trained models, Layer outputs, Feature pruning, Model fragility, Transformers robustness, Pre-trained models, Layer outputs, Feature pruning, Model fragility",

author = "Olga Kovaleva and Saurabh Kulshreshtha and Anna Rogers and Anna Rumshisky",

year = "2021",

month = aug,

day = "1",

language = "English",

pages = "3392--3405",

booktitle = "Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021",

publisher = "Association for Computational Linguistics",

address = "United States",

}

BERT Busters: Outlier Dimensions That Disrupt Transformers. / Kovaleva, Olga; Kulshreshtha, Saurabh; Rogers, Anna et al.
Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. Online: Association for Computational Linguistics, 2021. s. 3392-3405.

Publikation: Konference artikel i Proceeding eller bog/rapport kapitel › Konferencebidrag i proceedings › Forskning › peer review

TY - GEN

T1 - BERT Busters: Outlier Dimensions That Disrupt Transformers

AU - Kovaleva, Olga

AU - Kulshreshtha, Saurabh

AU - Rogers, Anna

AU - Rumshisky, Anna

PY - 2021/8/1

Y1 - 2021/8/1

N2 - Multiple studies have shown that Transformers are remarkably robust to pruning. Contrary to this received wisdom, we demonstrate that pre-trained Transformer encoders are surprisingly fragile to the removal of a very small number of features in the layer outputs (

AB - Multiple studies have shown that Transformers are remarkably robust to pruning. Contrary to this received wisdom, we demonstrate that pre-trained Transformer encoders are surprisingly fragile to the removal of a very small number of features in the layer outputs (

KW - Transformers robustness

KW - Pre-trained models

KW - Layer outputs

KW - Feature pruning

KW - Model fragility

KW - Transformers robustness

KW - Pre-trained models

KW - Layer outputs

KW - Feature pruning

KW - Model fragility

M3 - Article in proceedings

SP - 3392

EP - 3405

BT - Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

PB - Association for Computational Linguistics

CY - Online

ER -

BERT Busters: Outlier Dimensions That Disrupt Transformers

Abstract

Emneord

Adgang til dokumentet

Fingeraftryk

Citationsformater