Language Models Learn Universal Representations of Numbers and Here’s Why You Should Care

Michal Štefánik; Timothee Mickus; Marek  Kadlčík; Bertram  Højer; Michael Spiegel; Raúl Vázquez; Aman Sinha; Josef Kuchar; Philipp Mondorf; Pontus Stenetorp

doi:10.48550/arXiv.2510.26285

Language Models Learn Universal Representations of Numbers and Here’s Why You Should Care

Michal Štefánik
, Timothee Mickus
, Marek Kadlčík
, Bertram Højer
, Michael Spiegel
, Raúl Vázquez
, Aman Sinha
, Josef Kuchar
, Philipp Mondorf
, Pontus Stenetorp

University of Helsinki
Masaryk University
Inria, CNRS, Universite de Lorraine
Ludwig Maximilian University of Munich
University College London

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

Abstract

Prior work has shown that large language models (LLMs) often converge to accurate input embedding for numbers, based on sinusoidal representations. In this work, we demonstrate that these representations are in fact strikingly systematic, to the point of being almost perfectly universal: different LLM families develop equivalent sinusoidal structures, and number representations are broadly interchangeable in a large swathe of experimental setups. We show that properly factoring in this characteristic is crucial when it comes to assessing how accurately LLMs encode numeric and other ordinal information, and that mechanistically enhancing this sinusoidality can also lead to reductions of LLMs' arithmetic errors.

Original language	English
Title of host publication	Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Editors	Maria Liakata, Viviane P. Moreira, Jiajun Zhang, David Jurgens
Number of pages	19
Volume	1
Publisher	Association for Computational Linguistics
Publication date	1 Jul 2026
Pages	30663–30681
DOIs	https://doi.org/10.48550/arXiv.2510.26285
Publication status	Published - 1 Jul 2026

Series	Proceedings of the Annual Meeting of the Association for Computational Linguistics

Access to Document

10.48550/arXiv.2510.26285Licence: CC BY

https://aclanthology.org/2026.acl-long.1415.pdfLicence: CC BY

Cite this

Štefánik, M., Mickus, T., Kadlčík, M., Højer, B., Spiegel, M., Vázquez, R., Sinha, A., Kuchar, J., Mondorf, P., & Stenetorp, P. (2026). Language Models Learn Universal Representations of Numbers and Here’s Why You Should Care. In M. Liakata, V. P. Moreira, J. Zhang, & D. Jurgens (Eds.), Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Vol. 1, pp. 30663–30681). Association for Computational Linguistics. https://doi.org/10.48550/arXiv.2510.26285

Štefánik, Michal ; Mickus, Timothee ; Kadlčík, Marek et al. / Language Models Learn Universal Representations of Numbers and Here’s Why You Should Care. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). editor / Maria Liakata ; Viviane P. Moreira ; Jiajun Zhang ; David Jurgens. Vol. 1 Association for Computational Linguistics, 2026. pp. 30663–30681 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

@inproceedings{91fa53ab801c4ebebf0b58d27ada0592,

title = "Language Models Learn Universal Representations of Numbers and Here{\textquoteright}s Why You Should Care",

abstract = "Prior work has shown that large language models (LLMs) often converge to accurate input embedding for numbers, based on sinusoidal representations. In this work, we demonstrate that these representations are in fact strikingly systematic, to the point of being almost perfectly universal: different LLM families develop equivalent sinusoidal structures, and number representations are broadly interchangeable in a large swathe of experimental setups. We show that properly factoring in this characteristic is crucial when it comes to assessing how accurately LLMs encode numeric and other ordinal information, and that mechanistically enhancing this sinusoidality can also lead to reductions of LLMs' arithmetic errors.",

author = "Michal {\v S}tef{\'a}nik and Timothee Mickus and Marek Kadl{\v c}{\'i}k and Bertram H{\o}jer and Michael Spiegel and Ra{\'u}l V{\'a}zquez and Aman Sinha and Josef Kuchar and Philipp Mondorf and Pontus Stenetorp",

year = "2026",

month = jul,

day = "1",

doi = "10.48550/arXiv.2510.26285",

language = "English",

volume = "1",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

pages = "30663–30681",

editor = "Maria Liakata and \{P. Moreira\}, Viviane and Jiajun Zhang and David Jurgens",

booktitle = "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",

publisher = "Association for Computational Linguistics",

address = "United States",

}

Štefánik, M, Mickus, T, Kadlčík, M, Højer, B, Spiegel, M, Vázquez, R, Sinha, A, Kuchar, J, Mondorf, P & Stenetorp, P 2026, Language Models Learn Universal Representations of Numbers and Here’s Why You Should Care. in M Liakata, V P. Moreira, J Zhang & D Jurgens (eds), Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). vol. 1, Association for Computational Linguistics, Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 30663–30681. https://doi.org/10.48550/arXiv.2510.26285

Language Models Learn Universal Representations of Numbers and Here’s Why You Should Care. / Štefánik, Michal; Mickus, Timothee; Kadlčík, Marek et al.
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ed. / Maria Liakata; Viviane P. Moreira; Jiajun Zhang; David Jurgens. Vol. 1 Association for Computational Linguistics, 2026. p. 30663–30681 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

TY - GEN

T1 - Language Models Learn Universal Representations of Numbers and Here’s Why You Should Care

AU - Štefánik, Michal

AU - Mickus, Timothee

AU - Kadlčík, Marek

AU - Højer, Bertram

AU - Spiegel, Michael

AU - Vázquez, Raúl

AU - Sinha, Aman

AU - Kuchar, Josef

AU - Mondorf, Philipp

AU - Stenetorp, Pontus

PY - 2026/7/1

Y1 - 2026/7/1

N2 - Prior work has shown that large language models (LLMs) often converge to accurate input embedding for numbers, based on sinusoidal representations. In this work, we demonstrate that these representations are in fact strikingly systematic, to the point of being almost perfectly universal: different LLM families develop equivalent sinusoidal structures, and number representations are broadly interchangeable in a large swathe of experimental setups. We show that properly factoring in this characteristic is crucial when it comes to assessing how accurately LLMs encode numeric and other ordinal information, and that mechanistically enhancing this sinusoidality can also lead to reductions of LLMs' arithmetic errors.

AB - Prior work has shown that large language models (LLMs) often converge to accurate input embedding for numbers, based on sinusoidal representations. In this work, we demonstrate that these representations are in fact strikingly systematic, to the point of being almost perfectly universal: different LLM families develop equivalent sinusoidal structures, and number representations are broadly interchangeable in a large swathe of experimental setups. We show that properly factoring in this characteristic is crucial when it comes to assessing how accurately LLMs encode numeric and other ordinal information, and that mechanistically enhancing this sinusoidality can also lead to reductions of LLMs' arithmetic errors.

U2 - 10.48550/arXiv.2510.26285

DO - 10.48550/arXiv.2510.26285

M3 - Article in proceedings

VL - 1

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 30663

EP - 30681

BT - Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

A2 - Liakata, Maria

A2 - P. Moreira, Viviane

A2 - Zhang, Jiajun

A2 - Jurgens, David

PB - Association for Computational Linguistics

ER -

Štefánik M, Mickus T, Kadlčík M, Højer B, Spiegel M, Vázquez R et al. Language Models Learn Universal Representations of Numbers and Here’s Why You Should Care. In Liakata M, P. Moreira V, Zhang J, Jurgens D, editors, Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vol. 1. Association for Computational Linguistics. 2026. p. 30663–30681. (Proceedings of the Annual Meeting of the Association for Computational Linguistics). doi: 10.48550/arXiv.2510.26285

Language Models Learn Universal Representations of Numbers and Here’s Why You Should Care

Abstract

Access to Document

Fingerprint

Cite this