Category Archives: Benchmarking

GECCO Challenge 2021 is Online

The GECCO Industrial Challenge is a yearly competition for CI researchers and practitioners taking place at the GECCO. It poses difficult real-world problems provided by industry partners from various fields.

Simulation models are valuable tools for resource usage estimation and capacity planning. The simulator, BaBSim.Hospital, explicitly covers difficulties for hospitals caused by the COVID-19 pandemic. The simulator can handle many aspects of resource planning in hospitals, such as ICU beds, ventilators or personal, while taking into consideration several cohorts as age or current health status.

The task represents an instance of an expensive, high-dimensional computer simulation-based optimization problem. The simulations will be executed through an interface and hosted on one of our servers (similar to our last year’s challenge).

Your goal is to find an optimal parameter configuration for the BaBSim.Hospital simulator with a very limited budget of objective function evaluations. The participants will be free to apply one or multiple optimization algorithms of their choice.

More information can be found here.

Free Preprint: “Expected Improvement versus Predicted Value in Surrogate-Based #Optimization” Available on Cologne Open Science

The publication “Expected Improvement versus Predicted Value in Surrogate-Based Optimization”, written by Frederik Rehbach, Martin Zaefferer, Boris Naujoks, and Thomas Bartz-Beielstein, deals with the correct parameterization for model-based optimization algorithms. The publication is available on Cologne Open Science.

Continue reading

New on arXiv: “Benchmarking in Optimization: Best Practice and Open Issues”

The most recent version of the article “Benchmarking in Optimization: Best Practice and Open Issues”, which was written by Thomas Bartz-Beielstein, Carola Doerr, Jakob Bossek, Sowmya Chandrasekaran, Tome Eftimov, Andreas Fischbach, Pascal Kerschke, Manuel Lopez-Ibanez, Katherine M. Malan, Jason H. Moore, Boris Naujoks, Patryk Orzechowski, Vanessa Volz, Markus Wagner, and Thomas Weise can be downloaded from arXiv

Survey “Benchmarking in Optimization: Best Practice and Open Issues” available

This survey compiles ideas and recommendations from more than a dozen
researchers with different backgrounds and from different institutes around the
world. Promoting best practice in benchmarking is its main goal. The article
discusses eight essential topics in benchmarking: clearly stated goals, well-
specified problems, suitable algorithms, adequate performance measures,
thoughtful analysis, effective and efficient designs, comprehensible
presentations, and guaranteed reproducibility. The final goal is to provide
well-accepted guidelines (rules) that might be useful for authors and
reviewers. As benchmarking in optimization is an active and evolving field of
research this manuscript is meant to co-evolve over time by means of periodic

The PDF version of this survey is available here and will be published on arXiv soon.