What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability

Mario Giulianelli; Joris Baan; Wilker Aziz; Raquel Fernandez; Barbara Plank

doi:10.48550/arXiv.2305.11707

What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability

Mario Giulianelli, Joris Baan, Wilker Aziz, Raquel Fernandez, Barbara Plank

NLPnorth

University of Amsterdam

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

Abstract

In Natural Language Generation (NLG) tasks,
for any input, multiple communicative goals are
plausible, and any goal can be put into words,
or produced, in multiple ways. We characterise
the extent to which human production varies
lexically, syntactically, and semantically across
four NLG tasks, connecting human production
variability to aleatoric or data uncertainty. We
then inspect the space of output strings shaped
by a generation system’s predicted probability
distribution and decoding algorithm to probe
its uncertainty. For each test input, we measure
the generator’s calibration to human production
variability. Following this instance-level ap-
proach, we analyse NLG models and decoding
strategies, demonstrating that probing a genera-
tor with multiple samples and, when possible,
multiple references, provides the level of detail
necessary to gain understanding of a model’s
representation of uncertainty.

Original language	English
Title of host publication	Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
Number of pages	22
Place of Publication	Singapore
Publisher	Association for Computational Linguistics
Publication date	Dec 2023
Pages	14349–14371
DOIs	https://doi.org/10.48550/arXiv.2305.11707
Publication status	Published - Dec 2023

Keywords

Natural Language Generation
Communicative goals
Human production variability
Aleatoric uncertainty
Decoding algorithm
Lexical variability
Syntactic variability
Semantic variability
Model calibration
Uncertainty representation

Access to Document

10.48550/arXiv.2305.11707Licence: Unspecified

https://aclanthology.org/2023.emnlp-main.887.pdf
https://arxiv.org/abs/2305.11707Licence: Unspecified
https://arxiv.org/pdf/2305.11707.pdfLicence: Unspecified

Replication data from: What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability
Giulianelli, M. (Creator), Baan, J. (Creator), Aziz, W. (Creator), Fernández, R. (Creator) & Plank, B. (Creator), ZENODO, 20 Oct 2023
DOI: 10.5281/zenodo.10025272, https://zenodo.org/records/10025272
Dataset

Cite this

Giulianelli, M., Baan, J., Aziz, W., Fernandez, R., & Plank, B. (2023). What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 14349–14371). Association for Computational Linguistics. https://doi.org/10.48550/arXiv.2305.11707

@inproceedings{4f25c1ddd9f74b5fb30f16b061954562,

title = "What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability",

abstract = "In Natural Language Generation (NLG) tasks,for any input, multiple communicative goals areplausible, and any goal can be put into words,or produced, in multiple ways. We characterisethe extent to which human production varieslexically, syntactically, and semantically acrossfour NLG tasks, connecting human productionvariability to aleatoric or data uncertainty. Wethen inspect the space of output strings shapedby a generation system{\textquoteright}s predicted probabilitydistribution and decoding algorithm to probeits uncertainty. For each test input, we measurethe generator{\textquoteright}s calibration to human productionvariability. Following this instance-level ap-proach, we analyse NLG models and decodingstrategies, demonstrating that probing a genera-tor with multiple samples and, when possible,multiple references, provides the level of detailnecessary to gain understanding of a model{\textquoteright}srepresentation of uncertainty.",

keywords = "Natural Language Generation, Communicative goals, Human production variability, Aleatoric uncertainty, Decoding algorithm, Lexical variability, Syntactic variability, Semantic variability, Model calibration, Uncertainty representation, Natural Language Generation, Communicative goals, Human production variability, Aleatoric uncertainty, Decoding algorithm, Lexical variability, Syntactic variability, Semantic variability, Model calibration, Uncertainty representation",

author = "Mario Giulianelli and Joris Baan and Wilker Aziz and Raquel Fernandez and Barbara Plank",

year = "2023",

month = dec,

doi = "10.48550/arXiv.2305.11707",

language = "English",

pages = " 14349–14371",

booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",

publisher = "Association for Computational Linguistics",

address = "United States",

}

Giulianelli, M, Baan, J, Aziz, W, Fernandez, R & Plank, B 2023, What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability. in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Singapore, pp. 14349–14371. https://doi.org/10.48550/arXiv.2305.11707

What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability. / Giulianelli, Mario; Baan, Joris; Aziz, Wilker et al.
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguistics, 2023. p. 14349–14371.

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

TY - GEN

T1 - What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability

AU - Giulianelli, Mario

AU - Baan, Joris

AU - Aziz, Wilker

AU - Fernandez, Raquel

AU - Plank, Barbara

PY - 2023/12

Y1 - 2023/12

N2 - In Natural Language Generation (NLG) tasks,for any input, multiple communicative goals areplausible, and any goal can be put into words,or produced, in multiple ways. We characterisethe extent to which human production varieslexically, syntactically, and semantically acrossfour NLG tasks, connecting human productionvariability to aleatoric or data uncertainty. Wethen inspect the space of output strings shapedby a generation system’s predicted probabilitydistribution and decoding algorithm to probeits uncertainty. For each test input, we measurethe generator’s calibration to human productionvariability. Following this instance-level ap-proach, we analyse NLG models and decodingstrategies, demonstrating that probing a genera-tor with multiple samples and, when possible,multiple references, provides the level of detailnecessary to gain understanding of a model’srepresentation of uncertainty.

AB - In Natural Language Generation (NLG) tasks,for any input, multiple communicative goals areplausible, and any goal can be put into words,or produced, in multiple ways. We characterisethe extent to which human production varieslexically, syntactically, and semantically acrossfour NLG tasks, connecting human productionvariability to aleatoric or data uncertainty. Wethen inspect the space of output strings shapedby a generation system’s predicted probabilitydistribution and decoding algorithm to probeits uncertainty. For each test input, we measurethe generator’s calibration to human productionvariability. Following this instance-level ap-proach, we analyse NLG models and decodingstrategies, demonstrating that probing a genera-tor with multiple samples and, when possible,multiple references, provides the level of detailnecessary to gain understanding of a model’srepresentation of uncertainty.

KW - Natural Language Generation

KW - Communicative goals

KW - Human production variability

KW - Aleatoric uncertainty

KW - Decoding algorithm

KW - Lexical variability

KW - Syntactic variability

KW - Semantic variability

KW - Model calibration

KW - Uncertainty representation

KW - Natural Language Generation

KW - Communicative goals

KW - Human production variability

KW - Aleatoric uncertainty

KW - Decoding algorithm

KW - Lexical variability

KW - Syntactic variability

KW - Semantic variability

KW - Model calibration

KW - Uncertainty representation

U2 - 10.48550/arXiv.2305.11707

DO - 10.48550/arXiv.2305.11707

M3 - Article in proceedings

SP - 14349

EP - 14371

BT - Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

PB - Association for Computational Linguistics

CY - Singapore

ER -

What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability

Abstract

Keywords

Access to Document

Fingerprint

Datasets

Replication data from: What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability

Cite this