The Curious Case of High-Dimensional Indexing as a File Structure: A Case Study of eCP-FS

Omar Shahbaz Khan; Gylfi Þór Guðmundsson; Björn Þór Jónsson

doi:10.1007/978-3-032-06069-3_24

The Curious Case of High-Dimensional Indexing as a File Structure: A Case Study of eCP-FS

Omar Shahbaz Khan
, Gylfi Þór Guðmundsson
, Björn Þór Jónsson

Reykjavík University

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

Abstract

While approximate nearest-neighbor (ANN) search is an integral part of the modern multimedia analytics pipeline, the ever-hungry AI-models may frequently starve the ANN structures of resources, particularly memory. We present a novel white-box implementation of the disk-based hierarchical eCP index, called eCP-FS, that extends the disk-based strategy beyond merely storing data on disk, and instead implements the entire structure as an overlay file system using Zarr. This maps the (normally complex) index structure intuitively to a familiar hierarchical folder structure, which in turn makes the index much easier to visualize and analyse than the typical in-memory black-box structures of other algorithms. We furthermore implement incremental retrieval over eCP-FS, which benefits even more from file-system caching. Using an experimental benchmark inspired by live retrieval competitions, we show that despite trading raw speed for reduced memory footprint, eCP-FS is still a competitive option in the modern day analytics pipeline.

Original language	English
Title of host publication	Similarity Search and Applications
Editors	Giuseppe Amato, Vladimir Mic, Agma Traina, Nicola Messina, Laurent Amsaleg, Gylfi Þór Guðmundsson, Björn Þór Jónsson, Lucia Vadicamo
Number of pages	9
Place of Publication	Cham
Publisher	Springer Nature Switzerland
Publication date	8 Oct 2025
Pages	303-311
ISBN (Print)	978-3-032-06068-6
ISBN (Electronic)	978-3-032-06069-3
DOIs	https://doi.org/10.1007/978-3-032-06069-3_24
Publication status	Published - 8 Oct 2025
Externally published	Yes
Event	Similarity Search and Applications - Bologna, Italy Duration: 5 Oct 2022 → 7 Oct 2022 Conference number: 15th

Conference

Conference	Similarity Search and Applications
Number	15th
Country/Territory	Italy
City	Bologna
Period	05/10/2022 → 07/10/2022

Series	LNCS
Volume	16134

Keywords

High-dimensional indexing
Resource Constrained Search
Incremental Retrieval
Disk-based ANN

Access to Document

10.1007/978-3-032-06069-3_24Licence: Unspecified

Cite this

Khan, O. S., Guðmundsson, G. Þ., & Jónsson, B. Þ. (2025). The Curious Case of High-Dimensional Indexing as a File Structure: A Case Study of eCP-FS. In G. Amato, V. Mic, A. Traina, N. Messina, L. Amsaleg, G. Þ. Guðmundsson, B. Þ. Jónsson, & L. Vadicamo (Eds.), Similarity Search and Applications (pp. 303-311). Springer Nature Switzerland. https://doi.org/10.1007/978-3-032-06069-3_24

Khan, Omar Shahbaz ; Guðmundsson, Gylfi Þór ; Jónsson, Björn Þór. / The Curious Case of High-Dimensional Indexing as a File Structure: A Case Study of eCP-FS. Similarity Search and Applications. editor / Giuseppe Amato ; Vladimir Mic ; Agma Traina ; Nicola Messina ; Laurent Amsaleg ; Gylfi Þór Guðmundsson ; Björn Þór Jónsson ; Lucia Vadicamo. Cham : Springer Nature Switzerland, 2025. pp. 303-311 (LNCS, Vol. 16134).

@inproceedings{36144fab83044ab991d834d77a4d63b5,

title = "The Curious Case of High-Dimensional Indexing as a File Structure: A Case Study of eCP-FS",

abstract = "While approximate nearest-neighbor (ANN) search is an integral part of the modern multimedia analytics pipeline, the ever-hungry AI-models may frequently starve the ANN structures of resources, particularly memory. We present a novel white-box implementation of the disk-based hierarchical eCP index, called eCP-FS, that extends the disk-based strategy beyond merely storing data on disk, and instead implements the entire structure as an overlay file system using Zarr. This maps the (normally complex) index structure intuitively to a familiar hierarchical folder structure, which in turn makes the index much easier to visualize and analyse than the typical in-memory black-box structures of other algorithms. We furthermore implement incremental retrieval over eCP-FS, which benefits even more from file-system caching. Using an experimental benchmark inspired by live retrieval competitions, we show that despite trading raw speed for reduced memory footprint, eCP-FS is still a competitive option in the modern day analytics pipeline.",

keywords = "High-dimensional indexing, Resource Constrained Search, Incremental Retrieval, Disk-based ANN",

author = "Khan, \{Omar Shahbaz\} and Gu{\dh}mundsson, \{Gylfi {\TH}{\'o}r\} and J{\'o}nsson, \{Bj{\"o}rn {\TH}{\'o}r\}",

year = "2025",

month = oct,

day = "8",

doi = "10.1007/978-3-032-06069-3\_24",

language = "English",

isbn = "978-3-032-06068-6",

series = "LNCS",

pages = "303--311",

editor = "Giuseppe Amato and Vladimir Mic and Agma Traina and Nicola Messina and Laurent Amsaleg and Gu{\dh}mundsson, \{Gylfi {\TH}{\'o}r\} and J{\'o}nsson, \{Bj{\"o}rn {\TH}{\'o}r\} and Lucia Vadicamo",

booktitle = "Similarity Search and Applications",

publisher = "Springer Nature Switzerland",

note = "Similarity Search and Applications ; Conference date: 05-10-2022 Through 07-10-2022",

}

Khan, OS, Guðmundsson, GÞ & Jónsson, BÞ 2025, The Curious Case of High-Dimensional Indexing as a File Structure: A Case Study of eCP-FS. in G Amato, V Mic, A Traina, N Messina, L Amsaleg, GÞ Guðmundsson, BÞ Jónsson & L Vadicamo (eds), Similarity Search and Applications. Springer Nature Switzerland, Cham, LNCS, vol. 16134, pp. 303-311, Similarity Search and Applications, Bologna, Italy, 05/10/2022. https://doi.org/10.1007/978-3-032-06069-3_24

The Curious Case of High-Dimensional Indexing as a File Structure: A Case Study of eCP-FS. / Khan, Omar Shahbaz; Guðmundsson, Gylfi Þór; Jónsson, Björn Þór.
Similarity Search and Applications. ed. / Giuseppe Amato; Vladimir Mic; Agma Traina; Nicola Messina; Laurent Amsaleg; Gylfi Þór Guðmundsson; Björn Þór Jónsson; Lucia Vadicamo. Cham: Springer Nature Switzerland, 2025. p. 303-311 (LNCS, Vol. 16134).

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

TY - GEN

T1 - The Curious Case of High-Dimensional Indexing as a File Structure: A Case Study of eCP-FS

AU - Khan, Omar Shahbaz

AU - Guðmundsson, Gylfi Þór

AU - Jónsson, Björn Þór

N1 - Conference code: 15th

PY - 2025/10/8

Y1 - 2025/10/8

N2 - While approximate nearest-neighbor (ANN) search is an integral part of the modern multimedia analytics pipeline, the ever-hungry AI-models may frequently starve the ANN structures of resources, particularly memory. We present a novel white-box implementation of the disk-based hierarchical eCP index, called eCP-FS, that extends the disk-based strategy beyond merely storing data on disk, and instead implements the entire structure as an overlay file system using Zarr. This maps the (normally complex) index structure intuitively to a familiar hierarchical folder structure, which in turn makes the index much easier to visualize and analyse than the typical in-memory black-box structures of other algorithms. We furthermore implement incremental retrieval over eCP-FS, which benefits even more from file-system caching. Using an experimental benchmark inspired by live retrieval competitions, we show that despite trading raw speed for reduced memory footprint, eCP-FS is still a competitive option in the modern day analytics pipeline.

AB - While approximate nearest-neighbor (ANN) search is an integral part of the modern multimedia analytics pipeline, the ever-hungry AI-models may frequently starve the ANN structures of resources, particularly memory. We present a novel white-box implementation of the disk-based hierarchical eCP index, called eCP-FS, that extends the disk-based strategy beyond merely storing data on disk, and instead implements the entire structure as an overlay file system using Zarr. This maps the (normally complex) index structure intuitively to a familiar hierarchical folder structure, which in turn makes the index much easier to visualize and analyse than the typical in-memory black-box structures of other algorithms. We furthermore implement incremental retrieval over eCP-FS, which benefits even more from file-system caching. Using an experimental benchmark inspired by live retrieval competitions, we show that despite trading raw speed for reduced memory footprint, eCP-FS is still a competitive option in the modern day analytics pipeline.

KW - High-dimensional indexing

KW - Resource Constrained Search

KW - Incremental Retrieval

KW - Disk-based ANN

U2 - 10.1007/978-3-032-06069-3_24

DO - 10.1007/978-3-032-06069-3_24

M3 - Article in proceedings

SN - 978-3-032-06068-6

T3 - LNCS

SP - 303

EP - 311

BT - Similarity Search and Applications

A2 - Amato, Giuseppe

A2 - Mic, Vladimir

A2 - Traina, Agma

A2 - Messina, Nicola

A2 - Amsaleg, Laurent

A2 - Guðmundsson, Gylfi Þór

A2 - Jónsson, Björn Þór

A2 - Vadicamo, Lucia

PB - Springer Nature Switzerland

CY - Cham

T2 - Similarity Search and Applications

Y2 - 5 October 2022 through 7 October 2022

ER -