Running Experiments with Confidence and Sanity

Martin Aumüller; Matteo Ceccarello

doi:10.1007/978-3-030-60936-8_31

Running Experiments with Confidence and Sanity

Martin Aumüller
, Matteo Ceccarello

Algorithms

University of Padova

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

Abstract

Analyzing data from large experimental suites is a daily task for anyone doing experimental algorithmics. In this paper we report on several approaches we tried for this seemingly mundane task in a similarity search setting, reflecting on the challenges it poses.

We conclude by proposing a workflow, which can be implemented using several tools, that allows to analyze experimental data with confidence.

The extended version of this paper and the support code are provided at https://github.com/Cecca/running-experiments.

Original language	English
Title of host publication	International Conference on Similarity Search and Applications
Publisher	Springer
Publication date	2020
Pages	18-32
DOIs	https://doi.org/10.1007/978-3-030-60936-8_31
Publication status	Published - 2020
Event	International Conference on Similarity Search and Applications - Reykjavik, Iceland Duration: 1 Oct 2025 → 3 Oct 2025 Conference number: 18 https://www.sisap.org/2025

Conference

Conference	International Conference on Similarity Search and Applications
Number	18
Country/Territory	Iceland
City	Reykjavik
Period	01/10/2025 → 03/10/2025
Internet address	https://www.sisap.org/2025

Keywords

Experimental algorithmics
Data analysis
Similarity search
Workflow development
Support code

Access to Document

10.1007/978-3-030-60936-8_31

running-experimentsAccepted author manuscript, 592 KB

Cite this

@inproceedings{df94bc0cec7f4976b35ef89c5dc3a69a,

title = "Running Experiments with Confidence and Sanity",

abstract = "Analyzing data from large experimental suites is a daily task for anyone doing experimental algorithmics. In this paper we report on several approaches we tried for this seemingly mundane task in a similarity search setting, reflecting on the challenges it poses. We conclude by proposing a workflow, which can be implemented using several tools, that allows to analyze experimental data with confidence. The extended version of this paper and the support code are provided at https://github.com/Cecca/running-experiments.",

keywords = "Experimental algorithmics, Data analysis, Similarity search, Workflow development, Support code, Experimental algorithmics, Data analysis, Similarity search, Workflow development, Support code",

author = "Martin Aum{\"u}ller and Matteo Ceccarello",

year = "2020",

doi = "10.1007/978-3-030-60936-8\_31",

language = "English",

pages = "18--32",

booktitle = "International Conference on Similarity Search and Applications",

publisher = "Springer",

address = "Germany",

note = "International Conference on Similarity Search and Applications, SISAP ; Conference date: 01-10-2025 Through 03-10-2025",

url = "https://www.sisap.org/2025",

}

TY - GEN

T1 - Running Experiments with Confidence and Sanity

AU - Aumüller, Martin

AU - Ceccarello, Matteo

N1 - Conference code: 18

PY - 2020

Y1 - 2020

N2 - Analyzing data from large experimental suites is a daily task for anyone doing experimental algorithmics. In this paper we report on several approaches we tried for this seemingly mundane task in a similarity search setting, reflecting on the challenges it poses. We conclude by proposing a workflow, which can be implemented using several tools, that allows to analyze experimental data with confidence. The extended version of this paper and the support code are provided at https://github.com/Cecca/running-experiments.

AB - Analyzing data from large experimental suites is a daily task for anyone doing experimental algorithmics. In this paper we report on several approaches we tried for this seemingly mundane task in a similarity search setting, reflecting on the challenges it poses. We conclude by proposing a workflow, which can be implemented using several tools, that allows to analyze experimental data with confidence. The extended version of this paper and the support code are provided at https://github.com/Cecca/running-experiments.

KW - Experimental algorithmics

KW - Data analysis

KW - Similarity search

KW - Workflow development

KW - Support code

KW - Experimental algorithmics

KW - Data analysis

KW - Similarity search

KW - Workflow development

KW - Support code

U2 - 10.1007/978-3-030-60936-8_31

DO - 10.1007/978-3-030-60936-8_31

M3 - Article in proceedings

SP - 18

EP - 32

BT - International Conference on Similarity Search and Applications

PB - Springer

T2 - International Conference on Similarity Search and Applications

Y2 - 1 October 2025 through 3 October 2025

ER -

Running Experiments with Confidence and Sanity

Abstract

Conference

Keywords

Access to Document

Fingerprint

Cite this