Efficient Evidence Accumulation Clustering for Large Datasets

Diogo Silva, Helena Aidos, Ana Fred

2016

Abstract

The unprecedented collection and storage of data in electronic format has given rise to an interest in automated analysis for generation of knowledge and new insights. Cluster analysis is a good candidate since it makes as few assumptions about the data as possible. A vast body of work on clustering methods exist, yet, typically, no single method is able to respond to the specificities of all kinds of data. Evidence Accumulation Clustering (EAC) is a robust state of the art ensemble algorithm that has shown good results. However, this robustness comes with higher computational cost. Currently, its application is slow or restricted to small datasets. The objective of the present work is to scale EAC, allowing its applicability to big datasets, with technology available at a typical workstation. Three approaches for different parts of EAC are presented: a parallel GPU K-Means implementation, a novel strategy to build a sparse CSR matrix specialized to EAC and Single-Link based on Minimum Spanning Trees using an external memory sorting algorithm. Combining these approaches, the application of EAC to much larger datasets than before was accomplished.

Paper Citation

in Harvard Style

Silva D., Aidos H. and Fred A. (2016). Efficient Evidence Accumulation Clustering for Large Datasets . In Proceedings of the 5th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM, ISBN 978-989-758-173-1, pages 367-374. DOI: 10.5220/0005770803670374

in Bibtex Style

@conference{icpram16,
author={Diogo Silva and Helena Aidos and Ana Fred},
title={Efficient Evidence Accumulation Clustering for Large Datasets},
booktitle={Proceedings of the 5th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,},
year={2016},
pages={367-374},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005770803670374},
isbn={978-989-758-173-1},
}

in EndNote Style

TY - CONF
JO - Proceedings of the 5th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,
TI - Efficient Evidence Accumulation Clustering for Large Datasets
SN - 978-989-758-173-1
AU - Silva D.
AU - Aidos H.
AU - Fred A.
PY - 2016
SP - 367
EP - 374
DO - 10.5220/0005770803670374