In the perspective of analyzing the stochastic algorithms, a concept of presenting a single solution per problem type is typically incorrect and far from a fair comparison. Such algorithms shall run for n number of trials before initiating its respective perfomance. As a benchmark to a known standard such as CEC 2017 that requires 25 number of trials with 20000*D maximum number of function evaluations for constrained real parameter optimization (D as the number of dimensions), whereas in multimodal multiobjective problems CEC 2020 requires 21 runs for performance comparison. The main reason is due to its stochastic nature that may resulted in a spectrum of n solutions by executing the algorithm in n number of trials. Comparing several solutions with respect to the number of algorithms and problem types lead the analyst to focus on the statistical method that able to characterize the algorithm's efficiency and effectiveness towards finding the optimum solution. Besides its importance, a correct statistical method and comprehensive analysis is also highly recommended to avoid any judgmental error that lead into wrong conclusion. In general, the analysis of algorithm performance in the scope of efficiency and effectiveness can be viewed in two clusters: the group difference and trends. The group difference is described as the comparison of algorithm performance such as the converged fitness or computation time required after reached the maximum number of evaluations. The most appropriate method for analyzing the group difference is via two-sample or multiple sample comparison. The trend analysis is related to the dynamic progress of each compared algorithm towards finding the optimum solution. Typical measures for observing the trend between algorithm is based on the run-time analysis and convergence. These measures can be characterized by comparing the cumulative distribution and ordered alternatives method.
This book reviews the recommended basic concept and detail statistical analysis that been carried out in numerous analyses of metaheuristic algorithm, which cover both descriptive and inferential statistics. In addition to each sub-topic, the book also discusses several basic applications and examples related to the parametric and non-parametric analysis. This book also discusses several Bayesian statistic that been proposed in the literatures for evaluating the algorithm performance.
