From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding

Rob van der Goot; Ibrahim Sharaf; Aizhan  Imankulova; Ahmet Üstün; Marija Stepanovic; Alan Ramponi; Siti Oryza Khairunnisa; Mamoru Komachi; Barbara Plank

From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding

Rob van der Goot
, Ibrahim Sharaf
, Aizhan Imankulova
, Ahmet Üstün
, Marija Stepanovic
, Alan Ramponi
, Siti Oryza Khairunnisa
, Mamoru Komachi
, Barbara Plank

Tokyo Metropolitan University
Factmata
University of Groningen
Università degli Studi di Trento

Publikation: Konference artikel i Proceeding eller bog/rapport kapitel › Konferencebidrag i proceedings › Forskning › peer review

Abstract

The lack of publicly available evaluation data for low-resource languages limits progress in Spoken Language Understanding (SLU). As key tasks like intent classification and slot filling require abundant training data, it is desirable to reuse existing data in high-resource languages to develop models for low-resource scenarios. We introduce XSID, a new benchmark for cross-lingual (X) Slot and Intent Detection in 13 languages from 6 language families, including a very low-resource dialect.
To tackle the challenge, we propose a joint learning approach, with English SLU training data and non-English auxiliary tasks from raw text, syntax and translation for transfer. We study two setups which differ by type and language coverage of the pre-trained embeddings. Our results show that jointly learning the main
tasks with masked language modeling is effective for slots, while machine translation works best for intent classification.

Originalsprog	Engelsk
Titel	Proceedings of NAACL
Forlag	Association for Computational Linguistics
Publikationsdato	2021
Status	Udgivet - 2021
Begivenhed	Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - VIRTUAL Varighed: 6 jun. 2021 → 11 jun. 2021

Konference

Konference	Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
By	VIRTUAL
Periode	06/06/2021 → 11/06/2021

Emneord

Spoken Language Understanding
Low-resource languages
Cross-lingual benchmarks
Intent classification
Slot filling

Adgang til dokumentet

1050_PaperAccepteret manuskript, 403 KB
2021.naacl-main.197Accepteret manuskript, 549 KB

Multi-Task Sequence Labeling Under Adverse Conditions
Plank, B. (PI) & van der Goot, R. (CoI)
Amazon.com, Inc.
01/04/2019 → 31/08/2020
Projekter: Projekt › Andet

Citationsformater

@inproceedings{d2d5d0529bdc4831838b2b7215d6dc52,

title = "From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding",

abstract = "The lack of publicly available evaluation data for low-resource languages limits progress in Spoken Language Understanding (SLU). As key tasks like intent classification and slot filling require abundant training data, it is desirable to reuse existing data in high-resource languages to develop models for low-resource scenarios. We introduce XSID, a new benchmark for cross-lingual (X) Slot and Intent Detection in 13 languages from 6 language families, including a very low-resource dialect. To tackle the challenge, we propose a joint learning approach, with English SLU training data and non-English auxiliary tasks from raw text, syntax and translation for transfer. We study two setups which differ by type and language coverage of the pre-trained embeddings. Our results show that jointly learning the main tasks with masked language modeling is effective for slots, while machine translation works best for intent classification.",

keywords = "Spoken Language Understanding, Low-resource languages, Cross-lingual benchmarks, Intent classification, Slot filling, Spoken Language Understanding, Low-resource languages, Cross-lingual benchmarks, Intent classification, Slot filling",

author = "\{van der Goot\}, Rob and Ibrahim Sharaf and Aizhan Imankulova and Ahmet {\"U}st{\"u}n and Marija Stepanovic and Alan Ramponi and Khairunnisa, \{Siti Oryza\} and Mamoru Komachi and Barbara Plank",

year = "2021",

language = "English",

booktitle = "Proceedings of NAACL",

publisher = "Association for Computational Linguistics",

address = "United States",

note = "Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ; Conference date: 06-06-2021 Through 11-06-2021",

}

van der Goot, R, Sharaf, I, Imankulova, A, Üstün, A, Stepanovic, M, Ramponi, A, Khairunnisa, SO, Komachi, M & Plank, B 2021, From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding. i Proceedings of NAACL. Association for Computational Linguistics, Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, VIRTUAL, 06/06/2021.

From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding. / van der Goot, Rob; Sharaf, Ibrahim; Imankulova, Aizhan et al.
Proceedings of NAACL. Association for Computational Linguistics, 2021.

Publikation: Konference artikel i Proceeding eller bog/rapport kapitel › Konferencebidrag i proceedings › Forskning › peer review

TY - GEN

T1 - From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding

AU - van der Goot, Rob

AU - Sharaf, Ibrahim

AU - Imankulova, Aizhan

AU - Üstün, Ahmet

AU - Stepanovic, Marija

AU - Ramponi, Alan

AU - Khairunnisa, Siti Oryza

AU - Komachi, Mamoru

AU - Plank, Barbara

PY - 2021

Y1 - 2021

N2 - The lack of publicly available evaluation data for low-resource languages limits progress in Spoken Language Understanding (SLU). As key tasks like intent classification and slot filling require abundant training data, it is desirable to reuse existing data in high-resource languages to develop models for low-resource scenarios. We introduce XSID, a new benchmark for cross-lingual (X) Slot and Intent Detection in 13 languages from 6 language families, including a very low-resource dialect. To tackle the challenge, we propose a joint learning approach, with English SLU training data and non-English auxiliary tasks from raw text, syntax and translation for transfer. We study two setups which differ by type and language coverage of the pre-trained embeddings. Our results show that jointly learning the main tasks with masked language modeling is effective for slots, while machine translation works best for intent classification.

AB - The lack of publicly available evaluation data for low-resource languages limits progress in Spoken Language Understanding (SLU). As key tasks like intent classification and slot filling require abundant training data, it is desirable to reuse existing data in high-resource languages to develop models for low-resource scenarios. We introduce XSID, a new benchmark for cross-lingual (X) Slot and Intent Detection in 13 languages from 6 language families, including a very low-resource dialect. To tackle the challenge, we propose a joint learning approach, with English SLU training data and non-English auxiliary tasks from raw text, syntax and translation for transfer. We study two setups which differ by type and language coverage of the pre-trained embeddings. Our results show that jointly learning the main tasks with masked language modeling is effective for slots, while machine translation works best for intent classification.

KW - Spoken Language Understanding

KW - Low-resource languages

KW - Cross-lingual benchmarks

KW - Intent classification

KW - Slot filling

KW - Spoken Language Understanding

KW - Low-resource languages

KW - Cross-lingual benchmarks

KW - Intent classification

KW - Slot filling

M3 - Article in proceedings

BT - Proceedings of NAACL

PB - Association for Computational Linguistics

T2 - Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

Y2 - 6 June 2021 through 11 June 2021

ER -

From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding

Abstract

Konference

Emneord

Adgang til dokumentet

Fingeraftryk

Projekter

Multi-Task Sequence Labeling Under Adverse Conditions

Citationsformater