Iterative Structured Knowledge Distillation: Optimizing Language Models Through Layer-by-Layer Distillation

Malthe Have Musaeus; Rob van der Goot

Iterative Structured Knowledge Distillation: Optimizing Language Models Through Layer-by-Layer Distillation

Malthe Have Musaeus
, Rob van der Goot

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

Abstract

Traditional language model compression techniques, like knowledge distillation, require a fixed architecture, limiting flexibility, while structured pruning methods often fail to preserve performance. This paper introduces Iterative Structured Knowledge Distillation (ISKD), which integrates knowledge distillation and structured pruning by progressively replacing transformer blocks with smaller, efficient versions during training. This study validates ISKD on two transformer-based language models: GPT-2 and Phi-1. ISKD outperforms L1 pruning and achieves similar performance to knowledge distillation while offering greater flexibility. ISKD reduces model parameters - 30.68% for GPT-2 and 30.16% for Phi-1 - while maintaining at least four-fifths of performance on both language modeling and commonsense reasoning tasks. These findings suggest that this method offers a promising balance between model efficiency and accuracy.

Original language	English
Title of host publication	Proceedings of the 31st International Conference on Computational Linguistics
Editors	Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, Steven Schockaert
Number of pages	6
Place of Publication	Abu Dhabi, UAE
Publisher	Association for Computational Linguistics
Publication date	1 Jan 2025
Pages	6601-6606
Publication status	Published - 1 Jan 2025
Event	International Conference on Computational Linguistics - Abu Dhabi, United Arab Emirates Duration: 19 Jan 2025 → 24 Jan 2025 Conference number: 31 https://coling2025.org/ https://coling2025.org/calls/main_conference_papers/

Conference

Conference	International Conference on Computational Linguistics
Number	31
Country/Territory	United Arab Emirates
City	Abu Dhabi
Period	19/01/2025 → 24/01/2025
Internet address	https://coling2025.org/ https://coling2025.org/calls/main_conference_papers/

Keywords

Iterative Structured Knowledge Distillation
Structured pruning
Transformer block replacement
Language model compression
Parameter-efficient NLP

Access to Document

https://aclanthology.org/2025.coling-main.440.pdfLicence: CC BY

Cite this

Musaeus, M. H., & van der Goot, R. (2025). Iterative Structured Knowledge Distillation: Optimizing Language Models Through Layer-by-Layer Distillation. In O. Rambow, L. Wanner, M. Apidianaki, H. Al-Khalifa, B. D. Eugenio, & S. Schockaert (Eds.), Proceedings of the 31st International Conference on Computational Linguistics (pp. 6601-6606). Association for Computational Linguistics. https://aclanthology.org/2025.coling-main.440.pdf

Musaeus, Malthe Have ; van der Goot, Rob. / Iterative Structured Knowledge Distillation: Optimizing Language Models Through Layer-by-Layer Distillation. Proceedings of the 31st International Conference on Computational Linguistics. editor / Owen Rambow ; Leo Wanner ; Marianna Apidianaki ; Hend Al-Khalifa ; Barbara Di Eugenio ; Steven Schockaert. Abu Dhabi, UAE : Association for Computational Linguistics, 2025. pp. 6601-6606

@inproceedings{16af6681d76f4a8f8ee8d9f111f6c0fe,

title = "Iterative Structured Knowledge Distillation: Optimizing Language Models Through Layer-by-Layer Distillation",

abstract = "Traditional language model compression techniques, like knowledge distillation, require a fixed architecture, limiting flexibility, while structured pruning methods often fail to preserve performance. This paper introduces Iterative Structured Knowledge Distillation (ISKD), which integrates knowledge distillation and structured pruning by progressively replacing transformer blocks with smaller, efficient versions during training. This study validates ISKD on two transformer-based language models: GPT-2 and Phi-1. ISKD outperforms L1 pruning and achieves similar performance to knowledge distillation while offering greater flexibility. ISKD reduces model parameters - 30.68\% for GPT-2 and 30.16\% for Phi-1 - while maintaining at least four-fifths of performance on both language modeling and commonsense reasoning tasks. These findings suggest that this method offers a promising balance between model efficiency and accuracy.",

keywords = "Iterative Structured Knowledge Distillation, Structured pruning, Transformer block replacement, Language model compression, Parameter-efficient NLP",

author = "Musaeus, \{Malthe Have\} and \{van der Goot\}, Rob",

note = "ACL materials are Copyright {\textcopyright} 1963–2025 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License.; International Conference on Computational Linguistics, COLING ; Conference date: 19-01-2025 Through 24-01-2025",

year = "2025",

month = jan,

day = "1",

language = "English",

pages = "6601--6606",

editor = "Owen Rambow and Leo Wanner and Marianna Apidianaki and Hend Al-Khalifa and Eugenio, \{Barbara Di\} and Steven Schockaert",

booktitle = "Proceedings of the 31st International Conference on Computational Linguistics",

publisher = "Association for Computational Linguistics",

address = "United States",

url = "https://coling2025.org/, https://coling2025.org/calls/main\_conference\_papers/",

}

Musaeus, MH & van der Goot, R 2025, Iterative Structured Knowledge Distillation: Optimizing Language Models Through Layer-by-Layer Distillation. in O Rambow, L Wanner, M Apidianaki, H Al-Khalifa, BD Eugenio & S Schockaert (eds), Proceedings of the 31st International Conference on Computational Linguistics. Association for Computational Linguistics, Abu Dhabi, UAE, pp. 6601-6606, International Conference on Computational Linguistics, Abu Dhabi, United Arab Emirates, 19/01/2025. <https://aclanthology.org/2025.coling-main.440.pdf>

Iterative Structured Knowledge Distillation: Optimizing Language Models Through Layer-by-Layer Distillation. / Musaeus, Malthe Have; van der Goot, Rob.
Proceedings of the 31st International Conference on Computational Linguistics. ed. / Owen Rambow; Leo Wanner; Marianna Apidianaki; Hend Al-Khalifa; Barbara Di Eugenio; Steven Schockaert. Abu Dhabi, UAE: Association for Computational Linguistics, 2025. p. 6601-6606.

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

TY - GEN

T1 - Iterative Structured Knowledge Distillation: Optimizing Language Models Through Layer-by-Layer Distillation

AU - Musaeus, Malthe Have

AU - van der Goot, Rob

N1 - Conference code: 31

PY - 2025/1/1

Y1 - 2025/1/1

N2 - Traditional language model compression techniques, like knowledge distillation, require a fixed architecture, limiting flexibility, while structured pruning methods often fail to preserve performance. This paper introduces Iterative Structured Knowledge Distillation (ISKD), which integrates knowledge distillation and structured pruning by progressively replacing transformer blocks with smaller, efficient versions during training. This study validates ISKD on two transformer-based language models: GPT-2 and Phi-1. ISKD outperforms L1 pruning and achieves similar performance to knowledge distillation while offering greater flexibility. ISKD reduces model parameters - 30.68% for GPT-2 and 30.16% for Phi-1 - while maintaining at least four-fifths of performance on both language modeling and commonsense reasoning tasks. These findings suggest that this method offers a promising balance between model efficiency and accuracy.

AB - Traditional language model compression techniques, like knowledge distillation, require a fixed architecture, limiting flexibility, while structured pruning methods often fail to preserve performance. This paper introduces Iterative Structured Knowledge Distillation (ISKD), which integrates knowledge distillation and structured pruning by progressively replacing transformer blocks with smaller, efficient versions during training. This study validates ISKD on two transformer-based language models: GPT-2 and Phi-1. ISKD outperforms L1 pruning and achieves similar performance to knowledge distillation while offering greater flexibility. ISKD reduces model parameters - 30.68% for GPT-2 and 30.16% for Phi-1 - while maintaining at least four-fifths of performance on both language modeling and commonsense reasoning tasks. These findings suggest that this method offers a promising balance between model efficiency and accuracy.

KW - Iterative Structured Knowledge Distillation

KW - Structured pruning

KW - Transformer block replacement

KW - Language model compression

KW - Parameter-efficient NLP

M3 - Article in proceedings

SP - 6601

EP - 6606

BT - Proceedings of the 31st International Conference on Computational Linguistics

A2 - Rambow, Owen

A2 - Wanner, Leo

A2 - Apidianaki, Marianna

A2 - Al-Khalifa, Hend

A2 - Eugenio, Barbara Di

A2 - Schockaert, Steven

PB - Association for Computational Linguistics

CY - Abu Dhabi, UAE

T2 - International Conference on Computational Linguistics

Y2 - 19 January 2025 through 24 January 2025

ER -

Musaeus MH, van der Goot R. Iterative Structured Knowledge Distillation: Optimizing Language Models Through Layer-by-Layer Distillation. In Rambow O, Wanner L, Apidianaki M, Al-Khalifa H, Eugenio BD, Schockaert S, editors, Proceedings of the 31st International Conference on Computational Linguistics. Abu Dhabi, UAE: Association for Computational Linguistics. 2025. p. 6601-6606

Iterative Structured Knowledge Distillation: Optimizing Language Models Through Layer-by-Layer Distillation

Abstract

Conference

Keywords

Access to Document

Fingerprint

Cite this