IndicBART: A Pre-trained Model for Indic Natural Language Generation.

Raj Dabre; Himani Shrotriya; Anoop Kunchukuttan; Ratish Puduppully; Mitesh M. Khapra; Pratyush Kumar

doi:10.18653/V1/2022.FINDINGS-ACL.145

IndicBART: A Pre-trained Model for Indic Natural Language Generation.

Raj Dabre
, Himani Shrotriya
, Anoop Kunchukuttan
, Ratish Puduppully
, Mitesh M. Khapra
, Pratyush Kumar

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

Abstract

In this paper, we study pre-trained sequence-to-sequence models for a group of related languages, with a focus on Indic languages. We present IndicBART, a multilingual, sequence-to-sequence pre-trained model focusing on 11 Indic languages and English. IndicBART utilizes the orthographic similarity between Indic scripts to improve transfer learning between similar Indic languages. We evaluate IndicBART on two NLG tasks: Neural Machine Translation (NMT) and extreme summarization. Our experiments on NMT and extreme summarization show that a model specific to related languages like IndicBART is competitive with large pre-trained models like mBART50 despite being significantly smaller. It also performs well on very low-resource translation scenarios where languages are not included in pre-training or fine-tuning. Script sharing, multilingual training, and better utilization of limited model capacity contribute to the good performance of the compact IndicBART model.

Original language	English
Title of host publication	Findings of the Association for Computational Linguistics: ACL 2022
Publisher	Association for Computational Linguistics
Publication date	2022
Pages	1849-1863
DOIs	https://doi.org/10.18653/V1/2022.FINDINGS-ACL.145
Publication status	Published - 2022
Externally published	Yes
Event	Conference on the Association for Computational Linguistics - Dublin, Ireland Duration: 22 May 2022 → 27 May 2022 Conference number: 60

Conference

Conference	Conference on the Association for Computational Linguistics
Number	60
Country/Territory	Ireland
City	Dublin
Period	22/05/2022 → 27/05/2022

Keywords

IndicBART
Multilingual sequence-to-sequence models
Orthographic similarity
Neural machine translation
Extreme summarization

Access to Document

10.18653/V1/2022.FINDINGS-ACL.145

https://dblp.org/rec/conf/acl/DabreSKPKK22

Cite this

@inproceedings{63b9343b294b423c8927ed6a6d5ba725,

title = "IndicBART: A Pre-trained Model for Indic Natural Language Generation.",

abstract = "In this paper, we study pre-trained sequence-to-sequence models for a group of related languages, with a focus on Indic languages. We present IndicBART, a multilingual, sequence-to-sequence pre-trained model focusing on 11 Indic languages and English. IndicBART utilizes the orthographic similarity between Indic scripts to improve transfer learning between similar Indic languages. We evaluate IndicBART on two NLG tasks: Neural Machine Translation (NMT) and extreme summarization. Our experiments on NMT and extreme summarization show that a model specific to related languages like IndicBART is competitive with large pre-trained models like mBART50 despite being significantly smaller. It also performs well on very low-resource translation scenarios where languages are not included in pre-training or fine-tuning. Script sharing, multilingual training, and better utilization of limited model capacity contribute to the good performance of the compact IndicBART model.",

keywords = "IndicBART, Multilingual sequence-to-sequence models, Orthographic similarity, Neural machine translation, Extreme summarization",

author = "Raj Dabre and Himani Shrotriya and Anoop Kunchukuttan and Ratish Puduppully and Khapra, \{Mitesh M.\} and Pratyush Kumar",

note = "DBLP's bibliographic metadata records provided through http://dblp.org/search/publ/api are distributed under a Creative Commons CC0 1.0 Universal Public Domain Dedication. Although the bibliographic metadata records are provided consistent with CC0 1.0 Dedication, the content described by the metadata records is not. Content may be subject to copyright, rights of privacy, rights of publicity and other restrictions.; Conference on the Association for Computational Linguistics, ACL ; Conference date: 22-05-2022 Through 27-05-2022",

year = "2022",

doi = "10.18653/V1/2022.FINDINGS-ACL.145",

language = "English",

pages = "1849--1863",

booktitle = "Findings of the Association for Computational Linguistics: ACL 2022",

publisher = "Association for Computational Linguistics",

address = "United States",

}

Dabre, R, Shrotriya, H, Kunchukuttan, A, Puduppully, R, Khapra, MM & Kumar, P 2022, IndicBART: A Pre-trained Model for Indic Natural Language Generation. in Findings of the Association for Computational Linguistics: ACL 2022. Association for Computational Linguistics, pp. 1849-1863, Conference on the Association for Computational Linguistics, Dublin, Ireland, 22/05/2022. https://doi.org/10.18653/V1/2022.FINDINGS-ACL.145

IndicBART: A Pre-trained Model for Indic Natural Language Generation. / Dabre, Raj; Shrotriya, Himani; Kunchukuttan, Anoop et al.
Findings of the Association for Computational Linguistics: ACL 2022. Association for Computational Linguistics, 2022. p. 1849-1863.

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

TY - GEN

T1 - IndicBART: A Pre-trained Model for Indic Natural Language Generation.

AU - Dabre, Raj

AU - Shrotriya, Himani

AU - Kunchukuttan, Anoop

AU - Puduppully, Ratish

AU - Khapra, Mitesh M.

AU - Kumar, Pratyush

N1 - Conference code: 60

PY - 2022

Y1 - 2022

N2 - In this paper, we study pre-trained sequence-to-sequence models for a group of related languages, with a focus on Indic languages. We present IndicBART, a multilingual, sequence-to-sequence pre-trained model focusing on 11 Indic languages and English. IndicBART utilizes the orthographic similarity between Indic scripts to improve transfer learning between similar Indic languages. We evaluate IndicBART on two NLG tasks: Neural Machine Translation (NMT) and extreme summarization. Our experiments on NMT and extreme summarization show that a model specific to related languages like IndicBART is competitive with large pre-trained models like mBART50 despite being significantly smaller. It also performs well on very low-resource translation scenarios where languages are not included in pre-training or fine-tuning. Script sharing, multilingual training, and better utilization of limited model capacity contribute to the good performance of the compact IndicBART model.

AB - In this paper, we study pre-trained sequence-to-sequence models for a group of related languages, with a focus on Indic languages. We present IndicBART, a multilingual, sequence-to-sequence pre-trained model focusing on 11 Indic languages and English. IndicBART utilizes the orthographic similarity between Indic scripts to improve transfer learning between similar Indic languages. We evaluate IndicBART on two NLG tasks: Neural Machine Translation (NMT) and extreme summarization. Our experiments on NMT and extreme summarization show that a model specific to related languages like IndicBART is competitive with large pre-trained models like mBART50 despite being significantly smaller. It also performs well on very low-resource translation scenarios where languages are not included in pre-training or fine-tuning. Script sharing, multilingual training, and better utilization of limited model capacity contribute to the good performance of the compact IndicBART model.

KW - IndicBART

KW - Multilingual sequence-to-sequence models

KW - Orthographic similarity

KW - Neural machine translation

KW - Extreme summarization

U2 - 10.18653/V1/2022.FINDINGS-ACL.145

DO - 10.18653/V1/2022.FINDINGS-ACL.145

M3 - Article in proceedings

SP - 1849

EP - 1863

BT - Findings of the Association for Computational Linguistics: ACL 2022

PB - Association for Computational Linguistics

T2 - Conference on the Association for Computational Linguistics

Y2 - 22 May 2022 through 27 May 2022

ER -

IndicBART: A Pre-trained Model for Indic Natural Language Generation.

Abstract

Conference

Keywords

Access to Document

Fingerprint

Cite this