What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability

Mario Giulianelli, Joris Baan, Wilker Aziz, Raquel Fernandez, Barbara Plank

Publikation: Konference artikel i Proceeding eller bog/rapport kapitelKonferencebidrag i proceedingsForskningpeer review

Abstract

In Natural Language Generation (NLG) tasks,
for any input, multiple communicative goals are
plausible, and any goal can be put into words,
or produced, in multiple ways. We characterise
the extent to which human production varies
lexically, syntactically, and semantically across
four NLG tasks, connecting human production
variability to aleatoric or data uncertainty. We
then inspect the space of output strings shaped
by a generation system’s predicted probability
distribution and decoding algorithm to probe
its uncertainty. For each test input, we measure
the generator’s calibration to human production
variability. Following this instance-level ap-
proach, we analyse NLG models and decoding
strategies, demonstrating that probing a genera-
tor with multiple samples and, when possible,
multiple references, provides the level of detail
necessary to gain understanding of a model’s
representation of uncertainty.
OriginalsprogEngelsk
TitelProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
Antal sider22
UdgivelsesstedSingapore
ForlagAssociation for Computational Linguistics
Publikationsdatodec. 2023
Sider 14349–14371
DOI
StatusUdgivet - dec. 2023

Fingeraftryk

Dyk ned i forskningsemnerne om 'What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability'. Sammen danner de et unikt fingeraftryk.

Citationsformater