Fair Comparison of Population-based Heuristic Approaches - The Evils of Competitive Testing

Anikó Csébfalvi, György Csébfalvi

2012

Abstract

17 years ago, Hooker (1995) presented a pioneering work with the following title: "Testing Heuristics: We Have It All Wrong". If we ask the question now: "Do we have it all wrong?" the answer will be undoubtedly yes. The problem of the fair comparison remained essentially the same in the heuristic community. When we use stochastic methods in the optimization (namely heuristics or metaheuristics with several tunable parameters and starting seeds) then the usual presentation practice: "one problem - one result" is extremely far from the fair comparison. From statistical point of view, the minimal requirement is a so-called "small-sample" which is a set of results generated by independent runs and an appropriate "small-sample-test" according to the theory of the experimental design and evaluation and the protocol used for example, in the drug development processes. The viability and efficiency of the proposed statistically correct "bias-free" nonparametric methodology is demonstrated using a well-known nonlinear structural optimization example on the set of state-of-the-art heuristics. In the motivating example we used the presented solutions as a small-sample generated by a "hyperheuristic" and we test its quality against ANGEL, where the "supernatural" hybrid metaheuristic ANGEL combines ant colony optimization (AN), genetic algorithm (GE) and a gradient-based local search (L) strategy. ANGEL is an "essence of the different but at the same time similar heuristic approaches". The extremely simple and practically tuning-free ANGEL presents a number of interesting aspects such as extremely good adaptability and the ability to cope with totally different large real applications from the highly nonlinear structural optimization to the long-term optimization of the geothermal energy utilization.

Paper Citation

in Harvard Style

Csébfalvi A. and Csébfalvi G. (2012). Fair Comparison of Population-based Heuristic Approaches - The Evils of Competitive Testing . In Proceedings of the 4th International Joint Conference on Computational Intelligence - Volume 1: ECTA, (IJCCI 2012) ISBN 978-989-8565-33-4, pages 306-309. DOI: 10.5220/0004168403060309

in Bibtex Style

@conference{ecta12,
author={Anikó Csébfalvi and György Csébfalvi},
title={Fair Comparison of Population-based Heuristic Approaches - The Evils of Competitive Testing},
booktitle={Proceedings of the 4th International Joint Conference on Computational Intelligence - Volume 1: ECTA, (IJCCI 2012)},
year={2012},
pages={306-309},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0004168403060309},
isbn={978-989-8565-33-4},
}

in EndNote Style

TY - CONF
JO - Proceedings of the 4th International Joint Conference on Computational Intelligence - Volume 1: ECTA, (IJCCI 2012)
TI - Fair Comparison of Population-based Heuristic Approaches - The Evils of Competitive Testing
SN - 978-989-8565-33-4
AU - Csébfalvi A.
AU - Csébfalvi G.
PY - 2012
SP - 306
EP - 309
DO - 10.5220/0004168403060309