Previous work on multi-task learning in Natural Language Processing (NLP) oftenincorporated carefully selected tasks as well as carefully tuning ofarchitectures to share information across tasks. Recently, it has shown thatfor autoregressive language models, a multi-task second pre-training step on awide variety of NLP tasks leads to a set of parameters that more easily adaptfor other NLP tasks. In this paper, we examine whether a similar setup can beused in autoencoder language models using a restricted set of semanticallyoriented NLP tasks, namely all SemEval 2022 tasks that are annotated at theword, sentence or paragraph level. We first evaluate a multi-task model trainedon all SemEval 2022 tasks that contain annotation on the word, sentence orparagraph level (7 tasks, 11 sub-tasks), and then evaluate whetherre-finetuning the resulting model for each task specificially leads to furtherimprovements. Our results show that our mono-task baseline, our multi-taskmodel and our re-finetuned multi-task model each outperform the other modelsfor a subset of the tasks. Overall, huge gains can be observed by doingmulti-task learning: for three tasks we observe an error reduction of more than40%.
|Title of host publication||Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)|
|Publisher||Association for Computational Linguistics|
|Publication status||Published - 2022|
|Event||Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022) - |
Duration: 6 Jul 2022 → 13 Jul 2022
|Conference||Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)|
|Period||06/07/2022 → 13/07/2022|