Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning

Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, Massimo Poesio

    Publikation: Konference artikel i Proceeding eller bog/rapport kapitelKonferencebidrag i proceedingsForskningpeer review

    Abstract

    Supervised learning assumes that a ground truth label exists. However, the reliability of this ground truth depends on human annotators, who often disagree. Prior work has shown that this disagreement can be helpful in training models. We propose a novel method to incorporate this disagreement as information: in addition to the standard error computation, we use soft labels (i.e., probability distributions over the annotator labels) as an auxiliary task in a multi-task neural network. We measure the divergence between the predictions and the target soft labels with several loss-functions and evaluate the models on various NLP tasks. We find that the soft-label pre- diction auxiliary task reduces the penalty for errors on ambiguous entities and thereby mitigates overfitting. It significantly improves performance across tasks beyond the standard approach and prior work.
    OriginalsprogEngelsk
    TitelProceedings of NAACL
    ForlagAssociation for Computational Linguistics
    Publikationsdato2021
    Sider2591–2597
    StatusUdgivet - 2021

    Emneord

    • Supervised Learning
    • Ground Truth Reliability
    • Human Annotator Disagreement
    • Soft Labels
    • Multi-task Neural Networks

    Fingeraftryk

    Dyk ned i forskningsemnerne om 'Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning'. Sammen danner de et unikt fingeraftryk.

    Citationsformater