Reconfigurable and Adaptive Spark Applications

Mohamad Jaber, Mohamed Nassar, Wael Al Rahal Al Orabi, Bilal Abi Farraj, Mohamad Omar Kayali, Chadi Helwe

2017

Abstract

The contribution of this paper is two-fold. First, we propose a Domain Specific Language (DSL) to easily reconfigure and compose Spark applications. For each Spark application we define its input and output interfaces. Then, given a set of connections that map outputs of some Spark applications to free inputs of other Spark applications, we automatically embed Spark applications with the required synchronization and communication to properly run them according to the user-defined mapping. Second, we present an adaptive quality management/selection method for Spark applications. The method takes as input a pipeline of parameterized Spark applications, where the execution time of each Spark application is an unknown increasing function of quality level parameters. The method builds a controller that automatically computes adequate quality for each Spark application to meet a user-defined deadline. Consequently, users can submit a pipeline of Spark applications and a deadline, our method automatically runs all the Spark applications with the maximum quality while respecting the deadline specified by the user. We present experimental results showing the effectiveness of our method.

Paper Citation

in Harvard Style

Jaber M., Nassar M., Al Rahal Al Orabi W., Abi Farraj B., Kayali M. and Helwe C. (2017). Reconfigurable and Adaptive Spark Applications . In Proceedings of the 7th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER, ISBN 978-989-758-243-1, pages 112-119. DOI: 10.5220/0006289901120119

in Bibtex Style

@conference{closer17,
author={Mohamad Jaber and Mohamed Nassar and Wael Al Rahal Al Orabi and Bilal Abi Farraj and Mohamad Omar Kayali and Chadi Helwe},
title={Reconfigurable and Adaptive Spark Applications},
booktitle={Proceedings of the 7th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER,},
year={2017},
pages={112-119},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006289901120119},
isbn={978-989-758-243-1},
}

in EndNote Style

TY - CONF
JO - Proceedings of the 7th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER,
TI - Reconfigurable and Adaptive Spark Applications
SN - 978-989-758-243-1
AU - Jaber M.
AU - Nassar M.
AU - Al Rahal Al Orabi W.
AU - Abi Farraj B.
AU - Kayali M.
AU - Helwe C.
PY - 2017
SP - 112
EP - 119
DO - 10.5220/0006289901120119