Design of an RDMA Communication Middleware for Asynchronous Shuffling in Analytical Processing

Rui C. Gonçalves, José Pereira, Ricardo Jimenez-Peris

2016

Abstract

A key component in a distributed parallel analytical processing engine is shuffling, the distribution of data to multiple nodes such that the computation can be done in parallel. In this paper we describe the initial design of a communication middleware to support asynchronous shuffling of data among multiple processes on a distributed memory environment. The proposed middleware relies on RDMA (Remote Direct Memory Access) operations to transfer data, and provides basic operations to send and queue data on remote machines, and to retrieve this queued data. Preliminary results show that the RDMA-based middleware can provide a 75% reduction on communication costs, when compared with a traditional sockets implementation.

Paper Citation

in Harvard Style

Gonçalves R., Pereira J. and Jimenez-Peris R. (2016). Design of an RDMA Communication Middleware for Asynchronous Shuffling in Analytical Processing . In Proceedings of the 6th International Conference on Cloud Computing and Services Science - Volume 1: DataDiversityConvergence, (CLOSER 2016) ISBN 978-989-758-182-3, pages 348-351. DOI: 10.5220/0005923703480351

in Bibtex Style

@conference{datadiversityconvergence16,
author={Rui C. Gonçalves and José Pereira and Ricardo Jimenez-Peris},
title={Design of an RDMA Communication Middleware for Asynchronous Shuffling in Analytical Processing},
booktitle={Proceedings of the 6th International Conference on Cloud Computing and Services Science - Volume 1: DataDiversityConvergence, (CLOSER 2016)},
year={2016},
pages={348-351},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005923703480351},
isbn={978-989-758-182-3},
}

in EndNote Style

TY - CONF
JO - Proceedings of the 6th International Conference on Cloud Computing and Services Science - Volume 1: DataDiversityConvergence, (CLOSER 2016)
TI - Design of an RDMA Communication Middleware for Asynchronous Shuffling in Analytical Processing
SN - 978-989-758-182-3
AU - Gonçalves R.
AU - Pereira J.
AU - Jimenez-Peris R.
PY - 2016
SP - 348
EP - 351
DO - 10.5220/0005923703480351