A Primer in BERTology: What We Know About How BERT Works

Anna Rogers; Olga Kovaleva; Anna Rumshisky

A Primer in BERTology: What We Know About How BERT Works

Anna Rogers, Olga Kovaleva, Anna Rumshisky

University of Massachusetts

Research output: Journal Article or Conference Article in Journal › Journal article › Research › peer-review

Abstract

Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue, and approaches to compression. We then outline directions for future research.

Original language	English
Journal	Transactions of the Association for Computational Linguistics
Volume	8
Pages (from-to)	842-866
Number of pages	25
ISSN	2307-387X
Publication status	Published - 1 Dec 2020

Keywords

Transformer-based models
BERT model analysis
Neural network overparameterization
Training objective modifications
Model compression techniques

Access to Document

https://doi.org/10.1162/tacl_a_00349

Cite this

@article{40be91bf0fd1449e9d0dac2961bdd112,

title = "A Primer in BERTology: What We Know About How BERT Works",

abstract = "Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue, and approaches to compression. We then outline directions for future research.",

keywords = "Transformer-based models, BERT model analysis, Neural network overparameterization, Training objective modifications, Model compression techniques, Transformer-based models, BERT model analysis, Neural network overparameterization, Training objective modifications, Model compression techniques",

author = "Anna Rogers and Olga Kovaleva and Anna Rumshisky",

year = "2020",

month = dec,

day = "1",

language = "English",

volume = "8",

pages = "842--866",

journal = "Transactions of the Association for Computational Linguistics",

issn = "2307-387X",

publisher = "MIT Press",

}

TY - JOUR

T1 - A Primer in BERTology: What We Know About How BERT Works

AU - Rogers, Anna

AU - Kovaleva, Olga

AU - Rumshisky, Anna

PY - 2020/12/1

Y1 - 2020/12/1

N2 - Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue, and approaches to compression. We then outline directions for future research.

AB - Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue, and approaches to compression. We then outline directions for future research.

KW - Transformer-based models

KW - BERT model analysis

KW - Neural network overparameterization

KW - Training objective modifications

KW - Model compression techniques

KW - Transformer-based models

KW - BERT model analysis

KW - Neural network overparameterization

KW - Training objective modifications

KW - Model compression techniques

M3 - Journal article

SN - 2307-387X

VL - 8

SP - 842

EP - 866

JO - Transactions of the Association for Computational Linguistics

JF - Transactions of the Association for Computational Linguistics

ER -

A Primer in BERTology: What We Know About How BERT Works

Abstract

Keywords

Access to Document

Fingerprint

Cite this