Spectral Probing

Max Müller-Eberstein, Rob van der Goot, Barbara Plank

Publikation: Konference artikel i Proceeding eller bog/rapport kapitelKonferencebidrag i proceedingsForskningpeer review

Abstract

Linguistic information is encoded at varying timescales (subwords, phrases, etc.) and communicative levels, such as syntax and semantics. Contextualized embeddings have analogously been found to capture these phenomena at distinctive layers and frequencies. Leveraging these findings, we develop a fully learnable frequency filter to identify spectral profiles for any given task. It enables vastly more granular analyses than prior handcrafted filters, and improves on efficiency. After demonstrating the informativeness of spectral probing over manual filters in a monolingual setting, we investigate its multilingual characteristics across seven diverse NLP tasks in six languages. Our analyses identify distinctive spectral profiles which quantify cross-task similarity in a linguistically intuitive manner, while remaining consistent across languages—highlighting their potential as robust, lightweight task descriptors.
OriginalsprogEngelsk
TitelProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
UdgivelsesstedAbu Dhabi, United Arab Emirates
ForlagAssociation for Computational Linguistics
Publikationsdatodec. 2022
Sider7730-7741
StatusUdgivet - dec. 2022

Fingeraftryk

Dyk ned i forskningsemnerne om 'Spectral Probing'. Sammen danner de et unikt fingeraftryk.

Citationsformater