Abstract
Word embeddings have undoubtedly revolutionized NLP. However, pre-trained embeddings do not always work for a specific
task (or set of tasks), particularly in limited resource setups. We introduce a simple
yet effective, self-supervised post-processing
method that constructs task-specialized word
representations by picking from a menu of
reconstructing transformations to yield improved end-task performance (MORTY). The
method is complementary to recent state-of-the-art approaches to inductive transfer via
fine-tuning, and forgoes costly model architectures and annotation. We evaluate MORTY
on a broad range of setups, including different
word embedding methods, corpus sizes and
end-task semantics. Finally, we provide a surprisingly simple recipe to obtain specialized
embeddings that better fit end-tasks
task (or set of tasks), particularly in limited resource setups. We introduce a simple
yet effective, self-supervised post-processing
method that constructs task-specialized word
representations by picking from a menu of
reconstructing transformations to yield improved end-task performance (MORTY). The
method is complementary to recent state-of-the-art approaches to inductive transfer via
fine-tuning, and forgoes costly model architectures and annotation. We evaluate MORTY
on a broad range of setups, including different
word embedding methods, corpus sizes and
end-task semantics. Finally, we provide a surprisingly simple recipe to obtain specialized
embeddings that better fit end-tasks
| Originalsprog | Engelsk |
|---|---|
| Titel | Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)) : RepL4NLP-2019 |
| Udgivelsessted | Florence |
| Forlag | Association for Computational Linguistics |
| Publikationsdato | 2019 |
| Sider | 49-54 |
| ISBN (Elektronisk) | 978-1-950737-35-2 |
| Status | Udgivet - 2019 |