Experimental Standards for Deep Learning in Natural Language Processing Research

Dennis Thomas Ulmer; Elisa Bassignana; Max Müller-Eberstein; Daniel Varab; Mike Zhang; Rob van der Goot; Christian Hardmeier; Barbara Plank

doi:10.48550/arXiv.2204.06251

Experimental Standards for Deep Learning in Natural Language Processing Research

Dennis Thomas Ulmer
, Elisa Bassignana
, Max Müller-Eberstein
, Daniel Varab
, Mike Zhang
, Rob van der Goot
, Christian Hardmeier
, Barbara Plank

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

Abstract

The field of Deep Learning (DL) has undergone explosive growth during the last decade, with a substantial impact on Natural Language Processing (NLP) as well. Yet, compared to more established disciplines, a lack of common experimental standards remains an open challenge to the field at large. Starting from fundamental scientific principles, we distill ongoing discussions on experimental standards in NLP into a single, widely-applicable methodology. Following these best practices is crucial to strengthen experimental evidence, improve reproducibility and support scientific progress. These standards are further collected in a public repository to help them transparently adapt to future needs.

Original language	English
Title of host publication	Findings of 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Number of pages	20
Publication date	7 Dec 2022
DOIs	https://doi.org/10.48550/arXiv.2204.06251
Publication status	Published - 7 Dec 2022
Event	Empirical Methods in Natural Language Processing - Abu Dhabi National Exhibition Center (ADNEC), Abu Dhabi, United Arab Emirates Duration: 7 Dec 2022 → 11 Dec 2022 https://2022.emnlp.org/

Conference

Conference	Empirical Methods in Natural Language Processing
Location	Abu Dhabi National Exhibition Center (ADNEC)
Country/Territory	United Arab Emirates
City	Abu Dhabi
Period	07/12/2022 → 11/12/2022
Internet address	https://2022.emnlp.org/

Keywords

Deep Learning
Natural Language Processing
Experimental Standards
Reproducibility
Scientific Methodology

Access to Document

10.48550/arXiv.2204.06251Licence: CC BY

Cite this

@inproceedings{b05d6e0861c749ecad5de9ce8f0f531e,

title = "Experimental Standards for Deep Learning in Natural Language Processing Research",

abstract = "The field of Deep Learning (DL) has undergone explosive growth during the last decade, with a substantial impact on Natural Language Processing (NLP) as well. Yet, compared to more established disciplines, a lack of common experimental standards remains an open challenge to the field at large. Starting from fundamental scientific principles, we distill ongoing discussions on experimental standards in NLP into a single, widely-applicable methodology. Following these best practices is crucial to strengthen experimental evidence, improve reproducibility and support scientific progress. These standards are further collected in a public repository to help them transparently adapt to future needs. ",

keywords = "Deep Learning, Natural Language Processing, Experimental Standards, Reproducibility, Scientific Methodology, Deep Learning, Natural Language Processing, Experimental Standards, Reproducibility, Scientific Methodology",

author = "Ulmer, \{Dennis Thomas\} and Elisa Bassignana and Max M{\"u}ller-Eberstein and Daniel Varab and Mike Zhang and \{van der Goot\}, Rob and Christian Hardmeier and Barbara Plank",

year = "2022",

month = dec,

day = "7",

doi = "10.48550/arXiv.2204.06251",

language = "English",

booktitle = "Findings of 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)",

note = "Empirical Methods in Natural Language Processing ; Conference date: 07-12-2022 Through 11-12-2022",

url = "https://2022.emnlp.org/",

}

Ulmer, DT, Bassignana, E , Müller-Eberstein, M, Varab, D, Zhang, M, van der Goot, R , Hardmeier, C & Plank, B 2022, Experimental Standards for Deep Learning in Natural Language Processing Research. in Findings of 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, 07/12/2022. https://doi.org/10.48550/arXiv.2204.06251

Experimental Standards for Deep Learning in Natural Language Processing Research. / Ulmer, Dennis Thomas; Bassignana, Elisa ; Müller-Eberstein, Max et al.
Findings of 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2022.

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

TY - GEN

T1 - Experimental Standards for Deep Learning in Natural Language Processing Research

AU - Ulmer, Dennis Thomas

AU - Bassignana, Elisa

AU - Müller-Eberstein, Max

AU - Varab, Daniel

AU - Zhang, Mike

AU - van der Goot, Rob

AU - Hardmeier, Christian

AU - Plank, Barbara

PY - 2022/12/7

Y1 - 2022/12/7

N2 - The field of Deep Learning (DL) has undergone explosive growth during the last decade, with a substantial impact on Natural Language Processing (NLP) as well. Yet, compared to more established disciplines, a lack of common experimental standards remains an open challenge to the field at large. Starting from fundamental scientific principles, we distill ongoing discussions on experimental standards in NLP into a single, widely-applicable methodology. Following these best practices is crucial to strengthen experimental evidence, improve reproducibility and support scientific progress. These standards are further collected in a public repository to help them transparently adapt to future needs.

AB - The field of Deep Learning (DL) has undergone explosive growth during the last decade, with a substantial impact on Natural Language Processing (NLP) as well. Yet, compared to more established disciplines, a lack of common experimental standards remains an open challenge to the field at large. Starting from fundamental scientific principles, we distill ongoing discussions on experimental standards in NLP into a single, widely-applicable methodology. Following these best practices is crucial to strengthen experimental evidence, improve reproducibility and support scientific progress. These standards are further collected in a public repository to help them transparently adapt to future needs.

KW - Deep Learning

KW - Natural Language Processing

KW - Experimental Standards

KW - Reproducibility

KW - Scientific Methodology

KW - Deep Learning

KW - Natural Language Processing

KW - Experimental Standards

KW - Reproducibility

KW - Scientific Methodology

U2 - 10.48550/arXiv.2204.06251

DO - 10.48550/arXiv.2204.06251

M3 - Article in proceedings

BT - Findings of 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)

T2 - Empirical Methods in Natural Language Processing

Y2 - 7 December 2022 through 11 December 2022

ER -

Experimental Standards for Deep Learning in Natural Language Processing Research

Abstract

Conference

Keywords

Access to Document

Fingerprint

Cite this