From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding

Rob van der Goot; Ibrahim Sharaf; Aizhan  Imankulova; Ahmet Üstün; Marija Stepanovic; Alan Ramponi; Siti Oryza Khairunnisa; Mamoru Komachi; Barbara Plank

From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding

Rob van der Goot
, Ibrahim Sharaf
, Aizhan Imankulova
, Ahmet Üstün
, Marija Stepanovic
, Alan Ramponi
, Siti Oryza Khairunnisa
, Mamoru Komachi
, Barbara Plank

Tokyo Metropolitan University
Factmata
University of Groningen
University of Trento

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

Abstract

The lack of publicly available evaluation data for low-resource languages limits progress in Spoken Language Understanding (SLU). As key tasks like intent classification and slot filling require abundant training data, it is desirable to reuse existing data in high-resource languages to develop models for low-resource scenarios. We introduce XSID, a new benchmark for cross-lingual (X) Slot and Intent Detection in 13 languages from 6 language families, including a very low-resource dialect.
To tackle the challenge, we propose a joint learning approach, with English SLU training data and non-English auxiliary tasks from raw text, syntax and translation for transfer. We study two setups which differ by type and language coverage of the pre-trained embeddings. Our results show that jointly learning the main
tasks with masked language modeling is effective for slots, while machine translation works best for intent classification.

Original language	English
Title of host publication	Proceedings of NAACL
Publisher	Association for Computational Linguistics
Publication date	2021
Publication status	Published - 2021
Event	Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - VIRTUAL Duration: 6 Jun 2021 → 11 Jun 2021

Conference

Conference	Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
City	VIRTUAL
Period	06/06/2021 → 11/06/2021

Keywords

Spoken Language Understanding
Low-resource languages
Cross-lingual benchmarks
Intent classification
Slot filling

Access to Document

1050_PaperAccepted author manuscript, 403 KB
2021.naacl-main.197Accepted author manuscript, 549 KB

Multi-Task Sequence Labeling Under Adverse Conditions
Plank, B. (PI) & van der Goot, R. (CoI)
Amazon.com, Inc.
01/04/2019 → 31/08/2020
Project: Other

Cite this

@inproceedings{d2d5d0529bdc4831838b2b7215d6dc52,

title = "From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding",

abstract = "The lack of publicly available evaluation data for low-resource languages limits progress in Spoken Language Understanding (SLU). As key tasks like intent classification and slot filling require abundant training data, it is desirable to reuse existing data in high-resource languages to develop models for low-resource scenarios. We introduce XSID, a new benchmark for cross-lingual (X) Slot and Intent Detection in 13 languages from 6 language families, including a very low-resource dialect. To tackle the challenge, we propose a joint learning approach, with English SLU training data and non-English auxiliary tasks from raw text, syntax and translation for transfer. We study two setups which differ by type and language coverage of the pre-trained embeddings. Our results show that jointly learning the main tasks with masked language modeling is effective for slots, while machine translation works best for intent classification.",

keywords = "Spoken Language Understanding, Low-resource languages, Cross-lingual benchmarks, Intent classification, Slot filling, Spoken Language Understanding, Low-resource languages, Cross-lingual benchmarks, Intent classification, Slot filling",

author = "\{van der Goot\}, Rob and Ibrahim Sharaf and Aizhan Imankulova and Ahmet {\"U}st{\"u}n and Marija Stepanovic and Alan Ramponi and Khairunnisa, \{Siti Oryza\} and Mamoru Komachi and Barbara Plank",

year = "2021",

language = "English",

booktitle = "Proceedings of NAACL",

publisher = "Association for Computational Linguistics",

address = "United States",

note = "Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ; Conference date: 06-06-2021 Through 11-06-2021",

}

van der Goot, R, Sharaf, I, Imankulova, A, Üstün, A, Stepanovic, M, Ramponi, A, Khairunnisa, SO, Komachi, M & Plank, B 2021, From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding. in Proceedings of NAACL. Association for Computational Linguistics, Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, VIRTUAL, 06/06/2021.

From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding. / van der Goot, Rob; Sharaf, Ibrahim; Imankulova, Aizhan et al.
Proceedings of NAACL. Association for Computational Linguistics, 2021.

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

TY - GEN

T1 - From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding

AU - van der Goot, Rob

AU - Sharaf, Ibrahim

AU - Imankulova, Aizhan

AU - Üstün, Ahmet

AU - Stepanovic, Marija

AU - Ramponi, Alan

AU - Khairunnisa, Siti Oryza

AU - Komachi, Mamoru

AU - Plank, Barbara

PY - 2021

Y1 - 2021

N2 - The lack of publicly available evaluation data for low-resource languages limits progress in Spoken Language Understanding (SLU). As key tasks like intent classification and slot filling require abundant training data, it is desirable to reuse existing data in high-resource languages to develop models for low-resource scenarios. We introduce XSID, a new benchmark for cross-lingual (X) Slot and Intent Detection in 13 languages from 6 language families, including a very low-resource dialect. To tackle the challenge, we propose a joint learning approach, with English SLU training data and non-English auxiliary tasks from raw text, syntax and translation for transfer. We study two setups which differ by type and language coverage of the pre-trained embeddings. Our results show that jointly learning the main tasks with masked language modeling is effective for slots, while machine translation works best for intent classification.

AB - The lack of publicly available evaluation data for low-resource languages limits progress in Spoken Language Understanding (SLU). As key tasks like intent classification and slot filling require abundant training data, it is desirable to reuse existing data in high-resource languages to develop models for low-resource scenarios. We introduce XSID, a new benchmark for cross-lingual (X) Slot and Intent Detection in 13 languages from 6 language families, including a very low-resource dialect. To tackle the challenge, we propose a joint learning approach, with English SLU training data and non-English auxiliary tasks from raw text, syntax and translation for transfer. We study two setups which differ by type and language coverage of the pre-trained embeddings. Our results show that jointly learning the main tasks with masked language modeling is effective for slots, while machine translation works best for intent classification.

KW - Spoken Language Understanding

KW - Low-resource languages

KW - Cross-lingual benchmarks

KW - Intent classification

KW - Slot filling

KW - Spoken Language Understanding

KW - Low-resource languages

KW - Cross-lingual benchmarks

KW - Intent classification

KW - Slot filling

M3 - Article in proceedings

BT - Proceedings of NAACL

PB - Association for Computational Linguistics

T2 - Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

Y2 - 6 June 2021 through 11 June 2021

ER -

From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding

Abstract

Conference

Keywords

Access to Document

Fingerprint

Projects

Multi-Task Sequence Labeling Under Adverse Conditions

Cite this