Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity

Dennis Thomas Ulmer; Jes Frellsen; Christian Hardmeier

doi:10.18653/v1/2022.findings-emnlp.198

Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity

Dennis Thomas Ulmer, Jes Frellsen, Christian Hardmeier

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

Abstract

We investigate the problem of determining the predictive confidence (or, conversely, uncertainty) of a neural classifier through the lens of low-resource languages. By training models on sub-sampled datasets in three different languages, we assess the quality of estimates from a wide array of approaches and their dependence on the amount of available data. We find that while approaches based on pre-trained models and ensembles achieve the best results overall, the quality of uncertainty estimates can surprisingly suffer with more data. We also perform a qualitative analysis of uncertainties on sequences, discovering that a model's total uncertainty seems to be influenced to a large degree by its data uncertainty, not model uncertainty. All model implementations are open-sourced in a software package.

Original language	English
Title of host publication	Findings of 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Publication date	7 Dec 2022
DOIs	https://doi.org/10.18653/v1/2022.findings-emnlp.198
Publication status	Published - 7 Dec 2022
Event	Empirical Methods in Natural Language Processing - Abu Dhabi National Exhibition Center (ADNEC), Abu Dhabi, United Arab Emirates Duration: 7 Dec 2022 → 11 Dec 2022 https://2022.emnlp.org/

Conference

Conference	Empirical Methods in Natural Language Processing
Location	Abu Dhabi National Exhibition Center (ADNEC)
Country/Territory	United Arab Emirates
City	Abu Dhabi
Period	07/12/2022 → 11/12/2022
Internet address	https://2022.emnlp.org/

Keywords

Predictive confidence
Neural classifier
Low-resource languages
Uncertainty estimation
Pre-trained models
Ensembles
Data uncertainty
Model uncertainty
Sequence analysis
Open-source software

Access to Document

10.18653/v1/2022.findings-emnlp.198

Cite this

@inproceedings{cc5a86d5a1f645bebd290c4319271ed7,

title = "Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity",

abstract = "We investigate the problem of determining the predictive confidence (or, conversely, uncertainty) of a neural classifier through the lens of low-resource languages. By training models on sub-sampled datasets in three different languages, we assess the quality of estimates from a wide array of approaches and their dependence on the amount of available data. We find that while approaches based on pre-trained models and ensembles achieve the best results overall, the quality of uncertainty estimates can surprisingly suffer with more data. We also perform a qualitative analysis of uncertainties on sequences, discovering that a model's total uncertainty seems to be influenced to a large degree by its data uncertainty, not model uncertainty. All model implementations are open-sourced in a software package.",

keywords = "Predictive confidence, Neural classifier, Low-resource languages, Uncertainty estimation, Pre-trained models, Ensembles, Data uncertainty, Model uncertainty, Sequence analysis, Open-source software, Predictive confidence, Neural classifier, Low-resource languages, Uncertainty estimation, Pre-trained models, Ensembles, Data uncertainty, Model uncertainty, Sequence analysis, Open-source software",

author = "Ulmer, {Dennis Thomas} and Jes Frellsen and Christian Hardmeier",

year = "2022",

month = dec,

day = "7",

doi = "10.18653/v1/2022.findings-emnlp.198",

language = "English",

booktitle = "Findings of 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)",

note = "Empirical Methods in Natural Language Processing ; Conference date: 07-12-2022 Through 11-12-2022",

url = "https://2022.emnlp.org/",

}

Ulmer, DT, Frellsen, J & Hardmeier, C 2022, Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity. in Findings of 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, 07/12/2022. https://doi.org/10.18653/v1/2022.findings-emnlp.198

Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity. / Ulmer, Dennis Thomas; Frellsen, Jes; Hardmeier, Christian.
Findings of 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2022.

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

TY - GEN

T1 - Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity

AU - Ulmer, Dennis Thomas

AU - Frellsen, Jes

AU - Hardmeier, Christian

PY - 2022/12/7

Y1 - 2022/12/7

N2 - We investigate the problem of determining the predictive confidence (or, conversely, uncertainty) of a neural classifier through the lens of low-resource languages. By training models on sub-sampled datasets in three different languages, we assess the quality of estimates from a wide array of approaches and their dependence on the amount of available data. We find that while approaches based on pre-trained models and ensembles achieve the best results overall, the quality of uncertainty estimates can surprisingly suffer with more data. We also perform a qualitative analysis of uncertainties on sequences, discovering that a model's total uncertainty seems to be influenced to a large degree by its data uncertainty, not model uncertainty. All model implementations are open-sourced in a software package.

AB - We investigate the problem of determining the predictive confidence (or, conversely, uncertainty) of a neural classifier through the lens of low-resource languages. By training models on sub-sampled datasets in three different languages, we assess the quality of estimates from a wide array of approaches and their dependence on the amount of available data. We find that while approaches based on pre-trained models and ensembles achieve the best results overall, the quality of uncertainty estimates can surprisingly suffer with more data. We also perform a qualitative analysis of uncertainties on sequences, discovering that a model's total uncertainty seems to be influenced to a large degree by its data uncertainty, not model uncertainty. All model implementations are open-sourced in a software package.

KW - Predictive confidence

KW - Neural classifier

KW - Low-resource languages

KW - Uncertainty estimation

KW - Pre-trained models

KW - Ensembles

KW - Data uncertainty

KW - Model uncertainty

KW - Sequence analysis

KW - Open-source software

KW - Predictive confidence

KW - Neural classifier

KW - Low-resource languages

KW - Uncertainty estimation

KW - Pre-trained models

KW - Ensembles

KW - Data uncertainty

KW - Model uncertainty

KW - Sequence analysis

KW - Open-source software

U2 - 10.18653/v1/2022.findings-emnlp.198

DO - 10.18653/v1/2022.findings-emnlp.198

M3 - Article in proceedings

BT - Findings of 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)

T2 - Empirical Methods in Natural Language Processing

Y2 - 7 December 2022 through 11 December 2022

ER -

Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity

Abstract

Conference

Keywords

Access to Document

Fingerprint

Cite this