The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset

BigScience; Hugo Laurençon; Lucile Saulnier; Thomas Wang; Christopher Akiki; Albert Villanova del Moral; Teven Le Scao; Leandro von Werra; Chenghao Mou; Eduardo González Ponferrada; Huu Nguyen; Jörg Frohberg; Mario Šaško; Quentin Lhoest; Angelina Mcmillan-Major; Gérard Dupont; Stella Biderman; Anna Rogers; Loubna Ben Allal; Francesco de Toni; Giada Pistilli; Olivier Nguyen; Somaieh Nikpoor; Maraim Masoud; Pierre Colombo; Javier de la Rosa; Paulo Villegas; Tristan Thrush; Shayne Longpre; Sebastian Nagel; Leon Weber; Manuel Romero Muñoz; Jian Zhu; Daniel van Strien; Zaid Alyafeai; Khalid Almubarak; Vu Minh Chien; Itziar Gonzalez-Dios; Aitor Soroa; Kyle Lo; Manan Dey; Pedro Ortiz Suarez; Aaron Gokaslan; Shamik Bose; David Ifeoluwa Adelani; Long Phan; Hieu Tran; Ian Yu; Suhas Pai; Jenny Chim; Violette Lepercq

The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset

BigScience, Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina Mcmillan-Major, Gérard Dupont, Stella Biderman, Anna Rogers, Loubna Ben Allal, Francesco de ToniGiada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Romero Muñoz, Jian Zhu, Daniel van Strien, Zaid Alyafeai, Khalid Almubarak, Vu Minh Chien, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Ifeoluwa Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq

Publikation: Konference artikel i Proceeding eller bog/rapport kapitel › Konferencebidrag i proceedings › Forskning › peer review

Abstract

As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.

Originalsprog	Engelsk
Titel	Thirty-Sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track
Udgivelsessted	New Orleans, United States
Publikationsdato	1 nov. 2022
Status	Udgivet - 1 nov. 2022

Adgang til dokumentet

Citationsformater

BigScience, Laurençon, H., Saulnier, L., Wang, T., Akiki, C., Villanova del Moral, A., Le Scao, T., von Werra, L., Mou, C., González Ponferrada, E., Nguyen, H., Frohberg, J., Šaško, M., Lhoest, Q., Mcmillan-Major, A., Dupont, G., Biderman, S., Rogers, A., Ben Allal, L., ... Lepercq, V. (2022). The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset. I Thirty-Sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track https://openreview.net/pdf?id=UoEw6KigkUn

@inproceedings{bff59fa62f4d4330ac01bbb626b928a3,

title = "The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset",

abstract = "As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.",

keywords = "Multilingual Datasets, Language Models, Ethics in AI, Data Curation, Open Science",

author = "BigScience and Hugo Lauren{\c c}on and Lucile Saulnier and Thomas Wang and Christopher Akiki and {Villanova del Moral}, Albert and {Le Scao}, Teven and {von Werra}, Leandro and Chenghao Mou and {Gonz{\'a}lez Ponferrada}, Eduardo and Huu Nguyen and J{\"o}rg Frohberg and Mario {\v S}a{\v s}ko and Quentin Lhoest and Angelina Mcmillan-Major and G{\'e}rard Dupont and Stella Biderman and Anna Rogers and {Ben Allal}, Loubna and {de Toni}, Francesco and Giada Pistilli and Olivier Nguyen and Somaieh Nikpoor and Maraim Masoud and Pierre Colombo and {de la Rosa}, Javier and Paulo Villegas and Tristan Thrush and Shayne Longpre and Sebastian Nagel and Leon Weber and {Romero Mu{\~n}oz}, Manuel and Jian Zhu and {van Strien}, Daniel and Zaid Alyafeai and Khalid Almubarak and Chien, {Vu Minh} and Itziar Gonzalez-Dios and Aitor Soroa and Kyle Lo and Manan Dey and {Ortiz Suarez}, Pedro and Aaron Gokaslan and Shamik Bose and Adelani, {David Ifeoluwa} and Long Phan and Hieu Tran and Ian Yu and Suhas Pai and Jenny Chim and Violette Lepercq",

year = "2022",

month = nov,

day = "1",

language = "English",

booktitle = "Thirty-Sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track",

}

BigScience, Laurençon, H, Saulnier, L, Wang, T, Akiki, C, Villanova del Moral, A, Le Scao, T, von Werra, L, Mou, C, González Ponferrada, E, Nguyen, H, Frohberg, J, Šaško, M, Lhoest, Q, Mcmillan-Major, A, Dupont, G, Biderman, S, Rogers, A, Ben Allal, L, de Toni, F, Pistilli, G, Nguyen, O, Nikpoor, S, Masoud, M, Colombo, P, de la Rosa, J, Villegas, P, Thrush, T, Longpre, S, Nagel, S, Weber, L, Romero Muñoz, M, Zhu, J, van Strien, D, Alyafeai, Z, Almubarak, K, Chien, VM, Gonzalez-Dios, I, Soroa, A, Lo, K, Dey, M, Ortiz Suarez, P, Gokaslan, A, Bose, S, Adelani, DI, Phan, L, Tran, H, Yu, I, Pai, S, Chim, J & Lepercq, V 2022, The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset. i Thirty-Sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. New Orleans, United States. <https://openreview.net/pdf?id=UoEw6KigkUn>

The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset. / BigScience; Laurençon, Hugo; Saulnier, Lucile et al.
Thirty-Sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. New Orleans, United States, 2022.

Publikation: Konference artikel i Proceeding eller bog/rapport kapitel › Konferencebidrag i proceedings › Forskning › peer review

TY - GEN

T1 - The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset

AU - BigScience

AU - Laurençon, Hugo

AU - Saulnier, Lucile

AU - Wang, Thomas

AU - Akiki, Christopher

AU - Villanova del Moral, Albert

AU - Le Scao, Teven

AU - von Werra, Leandro

AU - Mou, Chenghao

AU - González Ponferrada, Eduardo

AU - Nguyen, Huu

AU - Frohberg, Jörg

AU - Šaško, Mario

AU - Lhoest, Quentin

AU - Mcmillan-Major, Angelina

AU - Dupont, Gérard

AU - Biderman, Stella

AU - Rogers, Anna

AU - Ben Allal, Loubna

AU - de Toni, Francesco

AU - Pistilli, Giada

AU - Nguyen, Olivier

AU - Nikpoor, Somaieh

AU - Masoud, Maraim

AU - Colombo, Pierre

AU - de la Rosa, Javier

AU - Villegas, Paulo

AU - Thrush, Tristan

AU - Longpre, Shayne

AU - Nagel, Sebastian

AU - Weber, Leon

AU - Romero Muñoz, Manuel

AU - Zhu, Jian

AU - van Strien, Daniel

AU - Alyafeai, Zaid

AU - Almubarak, Khalid

AU - Chien, Vu Minh

AU - Gonzalez-Dios, Itziar

AU - Soroa, Aitor

AU - Lo, Kyle

AU - Dey, Manan

AU - Ortiz Suarez, Pedro

AU - Gokaslan, Aaron

AU - Bose, Shamik

AU - Adelani, David Ifeoluwa

AU - Phan, Long

AU - Tran, Hieu

AU - Yu, Ian

AU - Pai, Suhas

AU - Chim, Jenny

AU - Lepercq, Violette

PY - 2022/11/1

Y1 - 2022/11/1

N2 - As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.

AB - As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.

KW - Multilingual Datasets

KW - Language Models

KW - Ethics in AI

KW - Data Curation

KW - Open Science

M3 - Article in proceedings

BT - Thirty-Sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track

CY - New Orleans, United States

ER -

The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset

Abstract

Adgang til dokumentet

Fingeraftryk

Citationsformater