BERT Busters: Outlier Dimensions That Disrupt Transformers

Olga Kovaleva; Saurabh Kulshreshtha; Anna Rogers; Anna Rumshisky

BERT Busters: Outlier Dimensions That Disrupt Transformers

Olga Kovaleva
, Saurabh Kulshreshtha
, Anna Rogers
, Anna Rumshisky

University of Massachusetts

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

Abstract

Multiple studies have shown that Transformers are remarkably robust to pruning. Contrary to this received wisdom, we demonstrate that pre-trained Transformer encoders are surprisingly fragile to the removal of a very small number of features in the layer outputs (

Original language	English
Title of host publication	Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021
Number of pages	14
Place of Publication	Online
Publisher	Association for Computational Linguistics
Publication date	1 Aug 2021
Pages	3392-3405
Publication status	Published - 1 Aug 2021

Keywords

Transformers robustness
Pre-trained models
Layer outputs
Feature pruning
Model fragility

Access to Document

https://aclanthology.org/2021.findings-acl.300

Cite this

@inproceedings{3ec033a5b6aa42688fcaad93977d6249,

title = "BERT Busters: Outlier Dimensions That Disrupt Transformers",

abstract = "Multiple studies have shown that Transformers are remarkably robust to pruning. Contrary to this received wisdom, we demonstrate that pre-trained Transformer encoders are surprisingly fragile to the removal of a very small number of features in the layer outputs (",

keywords = "Transformers robustness, Pre-trained models, Layer outputs, Feature pruning, Model fragility, Transformers robustness, Pre-trained models, Layer outputs, Feature pruning, Model fragility",

author = "Olga Kovaleva and Saurabh Kulshreshtha and Anna Rogers and Anna Rumshisky",

year = "2021",

month = aug,

day = "1",

language = "English",

pages = "3392--3405",

booktitle = "Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021",

publisher = "Association for Computational Linguistics",

address = "United States",

}

BERT Busters: Outlier Dimensions That Disrupt Transformers. / Kovaleva, Olga; Kulshreshtha, Saurabh; Rogers, Anna et al.
Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. Online: Association for Computational Linguistics, 2021. p. 3392-3405.

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

TY - GEN

T1 - BERT Busters: Outlier Dimensions That Disrupt Transformers

AU - Kovaleva, Olga

AU - Kulshreshtha, Saurabh

AU - Rogers, Anna

AU - Rumshisky, Anna

PY - 2021/8/1

Y1 - 2021/8/1

N2 - Multiple studies have shown that Transformers are remarkably robust to pruning. Contrary to this received wisdom, we demonstrate that pre-trained Transformer encoders are surprisingly fragile to the removal of a very small number of features in the layer outputs (

AB - Multiple studies have shown that Transformers are remarkably robust to pruning. Contrary to this received wisdom, we demonstrate that pre-trained Transformer encoders are surprisingly fragile to the removal of a very small number of features in the layer outputs (

KW - Transformers robustness

KW - Pre-trained models

KW - Layer outputs

KW - Feature pruning

KW - Model fragility

KW - Transformers robustness

KW - Pre-trained models

KW - Layer outputs

KW - Feature pruning

KW - Model fragility

M3 - Article in proceedings

SP - 3392

EP - 3405

BT - Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

PB - Association for Computational Linguistics

CY - Online

ER -

BERT Busters: Outlier Dimensions That Disrupt Transformers

Abstract

Keywords

Access to Document

Fingerprint

Cite this