A Hadoop based Framework to Process Geo-distributed Big Data

Marco Cavallo, Lorenzo Cusma', Giuseppe Di Modica, Carmelo Polito, Orazio Tomarchio

2016

Abstract

In many application fields such as social networks, e-commerce and content delivery networks there is a constant production of big amounts of data in geographically distributed sites that need to be timely elaborated. Distributed computing frameworks such as Hadoop (based on the MapReduce paradigm) have been used to process big data by exploiting the computing power of many cluster nodes interconnected through high speed links. Unfortunately, Hadoop was proved to perform very poorly in the just mentioned scenario. We designed and developed a Hadoop framework that is capable of scheduling and distributing hadoop tasks among geographically distant sites in a way that optimizes the overall job performance. We propose a hierarchical approach where a top-level entity, by exploiting the information concerning the data location, is capable of producing a smart schedule of low-level, independent MapReduce sub-jobs. A software prototype of the framework was developed. Tests run on the prototype showed that the job scheduler makes good forecasts of the expected job’s execution time.

Paper Citation

in Harvard Style

Cavallo M., Cusma' L., Di Modica G., Polito C. and Tomarchio O. (2016). A Hadoop based Framework to Process Geo-distributed Big Data . In Proceedings of the 6th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER, ISBN 978-989-758-182-3, pages 178-185. DOI: 10.5220/0005806101780185

in Bibtex Style

@conference{closer16,
author={Marco Cavallo and Lorenzo Cusma' and Giuseppe Di Modica and Carmelo Polito and Orazio Tomarchio},
title={A Hadoop based Framework to Process Geo-distributed Big Data},
booktitle={Proceedings of the 6th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER,},
year={2016},
pages={178-185},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005806101780185},
isbn={978-989-758-182-3},
}

in EndNote Style

TY - CONF
JO - Proceedings of the 6th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER,
TI - A Hadoop based Framework to Process Geo-distributed Big Data
SN - 978-989-758-182-3
AU - Cavallo M.
AU - Cusma' L.
AU - Di Modica G.
AU - Polito C.
AU - Tomarchio O.
PY - 2016
SP - 178
EP - 185
DO - 10.5220/0005806101780185