A Machine Learning Approach for Predicting Execution Time of Spark Jobs

Sara Moustafa, Iman Elghandour, Mohamed Ismail

Research output: Journal Article or Conference Article in JournalJournal articleResearchpeer-review

Abstract

Spark has gained growing attention in the past couple of years as an in-memory cloud computing platform. It supports execution of various types of workloads such as SQL queries and machine learning applications. Currently, many enterprises use Spark to exploit its fast in-memory processing of large scale data. Additionally, speeding up the execution in Spark is an important problem for many real-time applications. This can be achieved by improving the scheduling approaches employed by Spark, optimizing the execution plans generated by Spark for various applications, and selecting the best cluster configuration to run an input workload. A first step for all these optimization approaches is to predict the execution time of an input Spark application. In this paper, we present a new platform that predicts with high accuracy the execution time of SQL queries and machine learning applications executed by Spark. We evaluate our proposed platform by measuring the accuracy of predicting execution time of various types of Spark jobs including TPC-H queries and machine learning classification/clustering applications. The evaluation experiments show that we are able to predict the execution time of Spark jobs using our proposed platform with accuracy greater than 90% for SQL queries and greater than 75% for machine learning jobs.

Original languageEnglish
JournalAlexandria Engineering Journal
ISSN1110-0168
DOIs
Publication statusPublished - 22 Nov 2018
Externally publishedYes

Fingerprint

Dive into the research topics of 'A Machine Learning Approach for Predicting Execution Time of Spark Jobs'. Together they form a unique fingerprint.

Cite this