DAPHNE: An Open and Extensible System Infrastructure for Integrated Data Analysis Pipelines

Patrick Damme; Marius Birkenbach; Constantinos Bitsakos; Matthias Boehm; Philippe Bonnet; Florina Ciorba; Mark Dokter; Pawl Dowgiallo; Ahmed Eleliemy; Christian Faerber; Georgios Goumas; Dirk Habich; Niclas Hedam; Marlies Hofer; Wenjun Huang; Kevin Innerebner; Vasileios Karakostas; Roman Kern; Tomaž Kosar; Alexander Krause; Daniel Krems; Andreas Laber; Wolfgang Lehner; Eric Mier; Marcus Paradies; Bernhard Peischl; Gabrielle Poerwawinata; Stratos Psomadakis; Tilmann Rabl; Piotr Ratuszniak; Pedro Silva; Nikolai Skuppin; Andreas Starzacher; Benjamin Steinwender; Ilin Tolovski; Pinar Tözün; Wojciech Ulatowski; Yuanyuan Wang; Izajasz Wrosz; Aleš Zamuda; Ce Zhang; Xiao Xiang Zhu

DAPHNE: An Open and Extensible System Infrastructure for Integrated Data Analysis Pipelines

Patrick Damme, Marius Birkenbach, Constantinos Bitsakos, Matthias Boehm, Philippe Bonnet, Florina Ciorba, Mark Dokter, Pawl Dowgiallo, Ahmed Eleliemy, Christian Faerber, Georgios Goumas, Dirk Habich, Niclas Hedam, Marlies Hofer, Wenjun Huang, Kevin Innerebner, Vasileios Karakostas, Roman Kern, Tomaž Kosar, Alexander KrauseDaniel Krems, Andreas Laber, Wolfgang Lehner, Eric Mier, Marcus Paradies, Bernhard Peischl, Gabrielle Poerwawinata, Stratos Psomadakis, Tilmann Rabl, Piotr Ratuszniak, Pedro Silva, Nikolai Skuppin, Andreas Starzacher, Benjamin Steinwender, Ilin Tolovski, Pinar Tözün, Wojciech Ulatowski, Yuanyuan Wang, Izajasz Wrosz, Aleš Zamuda, Ce Zhang, Xiao Xiang Zhu

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

Abstract

Integrated data analysis (IDA) pipelines---that combine data management (DM) and query processing, high-performance computing (HPC), and machine learning (ML) training and scoring---become increasingly common in practice. Interestingly, systems of these areas share many compilation and runtime techniques, and the used---increasingly heterogeneous---hardware infrastructure converges as well. Yet, the programming paradigms, cluster resource management, data formats and representations, as well as execution strategies differ substantially. DAPHNE is an open and extensible system infrastructure for such IDA pipelines, including language abstractions, compilation and runtime techniques, multi-level scheduling, hardware (HW) accelerators, and computational storage for increasing productivity and eliminating unnecessary overheads. In this paper, we make a case for IDA pipelines, describe the overall DAPHNE system architecture, its key components, and the design of a vectorized execution engine for computational storage, HW accelerators, as well as local and distributed operations. Preliminary experiments that compare DAPHNE with MonetDB, Pandas, DuckDB, and TensorFlow show promising results.

Original language	English
Title of host publication	Conference on Innovative Data Systems Research
Place of Publication	Santa Cruz, California, USA
Publication date	9 Jan 2022
Publication status	Published - 9 Jan 2022

Keywords

Integrated Data Analysis
High-Performance Computing
Machine Learning Pipelines
DAPHNE System
Vectorized Execution Engine

Access to Document

DAPHNE: An Open and Extensible System Infrastructure for Integrated Data Analysis PipelinesFinal published version, 1.3 MBLicence: CC BY

http://cidrdb.org/cidr2022/papers/p4-damme.pdfLicence: CC BY

Cite this

Damme, P., Birkenbach, M., Bitsakos, C., Boehm, M., Bonnet, P., Ciorba, F., Dokter, M., Dowgiallo, P., Eleliemy, A., Faerber, C., Goumas, G., Habich, D., Hedam, N., Hofer, M., Huang, W., Innerebner, K., Karakostas, V., Kern, R., Kosar, T., ... Zhu, X. X. (2022). DAPHNE: An Open and Extensible System Infrastructure for Integrated Data Analysis Pipelines. In Conference on Innovative Data Systems Research http://cidrdb.org/cidr2022/papers/p4-damme.pdf

@inproceedings{33e1f3af607e45ad821be2e22f3113bc,

title = "DAPHNE: An Open and Extensible System Infrastructure for Integrated Data Analysis Pipelines",

abstract = "Integrated data analysis (IDA) pipelines---that combine data management (DM) and query processing, high-performance computing (HPC), and machine learning (ML) training and scoring---become increasingly common in practice. Interestingly, systems of these areas share many compilation and runtime techniques, and the used---increasingly heterogeneous---hardware infrastructure converges as well. Yet, the programming paradigms, cluster resource management, data formats and representations, as well as execution strategies differ substantially. DAPHNE is an open and extensible system infrastructure for such IDA pipelines, including language abstractions, compilation and runtime techniques, multi-level scheduling, hardware (HW) accelerators, and computational storage for increasing productivity and eliminating unnecessary overheads. In this paper, we make a case for IDA pipelines, describe the overall DAPHNE system architecture, its key components, and the design of a vectorized execution engine for computational storage, HW accelerators, as well as local and distributed operations. Preliminary experiments that compare DAPHNE with MonetDB, Pandas, DuckDB, and TensorFlow show promising results.",

keywords = "Integrated Data Analysis, High-Performance Computing, Machine Learning Pipelines, DAPHNE System, Vectorized Execution Engine, Integrated Data Analysis, High-Performance Computing, Machine Learning Pipelines, DAPHNE System, Vectorized Execution Engine",

author = "Patrick Damme and Marius Birkenbach and Constantinos Bitsakos and Matthias Boehm and Philippe Bonnet and Florina Ciorba and Mark Dokter and Pawl Dowgiallo and Ahmed Eleliemy and Christian Faerber and Georgios Goumas and Dirk Habich and Niclas Hedam and Marlies Hofer and Wenjun Huang and Kevin Innerebner and Vasileios Karakostas and Roman Kern and Toma{\v z} Kosar and Alexander Krause and Daniel Krems and Andreas Laber and Wolfgang Lehner and Eric Mier and Marcus Paradies and Bernhard Peischl and Gabrielle Poerwawinata and Stratos Psomadakis and Tilmann Rabl and Piotr Ratuszniak and Pedro Silva and Nikolai Skuppin and Andreas Starzacher and Benjamin Steinwender and Ilin Tolovski and Pinar T{\"o}z{\"u}n and Wojciech Ulatowski and Yuanyuan Wang and Izajasz Wrosz and Ale{\v s} Zamuda and Ce Zhang and Zhu, {Xiao Xiang}",

note = "No publisher listed. (jcg: 14/02/2022) In the call for papers it is stated that {"}Final versions of accepted submissions will be published in the electronic proceedings of the CIDR conference.{"}. Please insert these proceedings as the place of publication (jcg: 22/02/2022) ",

year = "2022",

month = jan,

day = "9",

language = "English",

booktitle = "Conference on Innovative Data Systems Research",

}

Damme, P, Birkenbach, M, Bitsakos, C, Boehm, M, Bonnet, P, Ciorba, F, Dokter, M, Dowgiallo, P, Eleliemy, A, Faerber, C, Goumas, G, Habich, D, Hedam, N, Hofer, M, Huang, W, Innerebner, K, Karakostas, V, Kern, R, Kosar, T, Krause, A, Krems, D, Laber, A, Lehner, W, Mier, E, Paradies, M, Peischl, B, Poerwawinata, G, Psomadakis, S, Rabl, T, Ratuszniak, P, Silva, P, Skuppin, N, Starzacher, A, Steinwender, B, Tolovski, I, Tözün, P, Ulatowski, W, Wang, Y, Wrosz, I, Zamuda, A, Zhang, C & Zhu, XX 2022, DAPHNE: An Open and Extensible System Infrastructure for Integrated Data Analysis Pipelines. in Conference on Innovative Data Systems Research. Santa Cruz, California, USA. <http://cidrdb.org/cidr2022/papers/p4-damme.pdf>

DAPHNE: An Open and Extensible System Infrastructure for Integrated Data Analysis Pipelines. / Damme, Patrick; Birkenbach, Marius; Bitsakos, Constantinos et al.
Conference on Innovative Data Systems Research. Santa Cruz, California, USA, 2022.

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

TY - GEN

T1 - DAPHNE: An Open and Extensible System Infrastructure for Integrated Data Analysis Pipelines

AU - Damme, Patrick

AU - Birkenbach, Marius

AU - Bitsakos, Constantinos

AU - Boehm, Matthias

AU - Bonnet, Philippe

AU - Ciorba, Florina

AU - Dokter, Mark

AU - Dowgiallo, Pawl

AU - Eleliemy, Ahmed

AU - Faerber, Christian

AU - Goumas, Georgios

AU - Habich, Dirk

AU - Hedam, Niclas

AU - Hofer, Marlies

AU - Huang, Wenjun

AU - Innerebner, Kevin

AU - Karakostas, Vasileios

AU - Kern, Roman

AU - Kosar, Tomaž

AU - Krause, Alexander

AU - Krems, Daniel

AU - Laber, Andreas

AU - Lehner, Wolfgang

AU - Mier, Eric

AU - Paradies, Marcus

AU - Peischl, Bernhard

AU - Poerwawinata, Gabrielle

AU - Psomadakis, Stratos

AU - Rabl, Tilmann

AU - Ratuszniak, Piotr

AU - Silva, Pedro

AU - Skuppin, Nikolai

AU - Starzacher, Andreas

AU - Steinwender, Benjamin

AU - Tolovski, Ilin

AU - Tözün, Pinar

AU - Ulatowski, Wojciech

AU - Wang, Yuanyuan

AU - Wrosz, Izajasz

AU - Zamuda, Aleš

AU - Zhang, Ce

AU - Zhu, Xiao Xiang

N1 - No publisher listed. (jcg: 14/02/2022) In the call for papers it is stated that "Final versions of accepted submissions will be published in the electronic proceedings of the CIDR conference.". Please insert these proceedings as the place of publication (jcg: 22/02/2022)

PY - 2022/1/9

Y1 - 2022/1/9

N2 - Integrated data analysis (IDA) pipelines---that combine data management (DM) and query processing, high-performance computing (HPC), and machine learning (ML) training and scoring---become increasingly common in practice. Interestingly, systems of these areas share many compilation and runtime techniques, and the used---increasingly heterogeneous---hardware infrastructure converges as well. Yet, the programming paradigms, cluster resource management, data formats and representations, as well as execution strategies differ substantially. DAPHNE is an open and extensible system infrastructure for such IDA pipelines, including language abstractions, compilation and runtime techniques, multi-level scheduling, hardware (HW) accelerators, and computational storage for increasing productivity and eliminating unnecessary overheads. In this paper, we make a case for IDA pipelines, describe the overall DAPHNE system architecture, its key components, and the design of a vectorized execution engine for computational storage, HW accelerators, as well as local and distributed operations. Preliminary experiments that compare DAPHNE with MonetDB, Pandas, DuckDB, and TensorFlow show promising results.

AB - Integrated data analysis (IDA) pipelines---that combine data management (DM) and query processing, high-performance computing (HPC), and machine learning (ML) training and scoring---become increasingly common in practice. Interestingly, systems of these areas share many compilation and runtime techniques, and the used---increasingly heterogeneous---hardware infrastructure converges as well. Yet, the programming paradigms, cluster resource management, data formats and representations, as well as execution strategies differ substantially. DAPHNE is an open and extensible system infrastructure for such IDA pipelines, including language abstractions, compilation and runtime techniques, multi-level scheduling, hardware (HW) accelerators, and computational storage for increasing productivity and eliminating unnecessary overheads. In this paper, we make a case for IDA pipelines, describe the overall DAPHNE system architecture, its key components, and the design of a vectorized execution engine for computational storage, HW accelerators, as well as local and distributed operations. Preliminary experiments that compare DAPHNE with MonetDB, Pandas, DuckDB, and TensorFlow show promising results.

KW - Integrated Data Analysis

KW - High-Performance Computing

KW - Machine Learning Pipelines

KW - DAPHNE System

KW - Vectorized Execution Engine

KW - Integrated Data Analysis

KW - High-Performance Computing

KW - Machine Learning Pipelines

KW - DAPHNE System

KW - Vectorized Execution Engine

M3 - Article in proceedings

BT - Conference on Innovative Data Systems Research

CY - Santa Cruz, California, USA

ER -

DAPHNE: An Open and Extensible System Infrastructure for Integrated Data Analysis Pipelines

Abstract

Keywords

Access to Document

Fingerprint

Cite this