Chat record from “Benchmarking and experimentation: pitfalls and best practices” at CEC 2021

Really enjoyed the tutorial today – unfortunately, time was limited (and too short). Further materials (slides) will be available soon.

Here ist the chat record (thanks to Nikola Ivković, University of Zagreb):

From CC 5 to Everyone: 02:05 PM
We can start in 10 mints
From CC 5 to Everyone: 02:14 PM
We can start now
From Carola Doerr to Everyone: 02:15 PM
you can mention the benchmarking network and the workshop 😉
woho!!
https://sites.google.com/view/benchmarking-network/home/activities/cec-2021-workshop
From Carola Doerr to Everyone: 02:34 PM
I am pretty much convinced that we need both, theory AND experiments. They just serve a different purpose
unfortunately, not too many people know how to get the best out of “the other side”
We can discuss it in the workshop. It’s an important discussion. 🙂 I see not much value in doing theory for stuff that can be done experimentally. what we really need is to leverage both approaches
From 86156 to Everyone: 02:35 PM
In a good work theory AND experiment should be supported by each other
From Marcos Galindo to Everyone: 02:50 PM
Hello! Can we have the presentation file?
From Thomas Bartz-Beielstein to Everyone: 02:50 PM
https://arxiv.org/abs/2007.03488
From Carola Doerr to Everyone: 02:50 PM
Here’s the survey mentioned by Boris: https://arxiv.org/abs/2007.03488
From Thomas Bartz-Beielstein to Everyone: 02:51 PM
Material will be also on https://www.spotseven.de
From Carola Doerr to Everyone: 03:09 PM
Nevergrad is open–source, available at https://github.com/facebookresearch/nevergrad
From 86156 to Everyone: 03:10 PM
Thanks for sharing
From Carola Doerr to Everyone: 03:12 PM
For those interested:
Newest version of Shiwa is called NGOpt in Nevergrad.
Shiwa: https://arxiv.org/abs/2004.14014
update: https://arxiv.org/abs/2010.04542
(super hot topic 🙂 )
From Carola Doerr to Everyone: 03:21 PM
what about quasi-random sampling? (DOEs)
From Mike Preuss to Everyone: 03:22 PM
I think it is a question of dimensions. the advantages of DOEs shrink if you have 10 dimensions already… there is no exact point where one is better of course…
From Carola Doerr to Everyone: 03:28 PM
yes, but for non-adaptive sampling a good DOE should be better than random (and of course better than grid). But, yes, depending on the relationship between number of samples and dimension, it can be hard to beat random search
What Thomas is saying right now also shows that it is super important to present algorithms in standardized ways (no metaphors etc). Join our discussion in the benchmarking workshop just after this tutorial, where CHristian will present why this is sooooooo important!
From Carola Doerr to Everyone: 04:05 PM
https://dlnext.acm.org/journal/telo
reproducibility paper: https://arxiv.org/abs/2102.03380
Have to run. Thanks for the great tutorial!
Link to our CEC workshop on benchmarking
Starting 16:15 🙂
https://us02web.zoom.us/j/83540739084
Hope to see you all there!
From Simone Ludwig to Everyone: 04:06 PM
What do you think about AutoML?
From Thomas Bartz-Beielstein to Everyone: 04:06 PM
https://www.spotseven.de
From joaoba2 to Everyone: 04:08 PM
Thanks for your interesting presentation. In your experience does the industry is getting interested in EAs and if you know any practical example? Do they rely on the methods even if in benchmarks they perform well? . Thanks. Joao Soares, Polytechnic of Porto.
From Simone Ludwig to Everyone: 04:08 PM
Thank you
From joaoba2 to Everyone: 04:11 PM
Thank you for pointing out that.
Thanks.
From CC 5 to Everyone: 04:12 PM
Thank you
From Heder Bernardino to Everyone: 04:12 PM
Thank you for your presentation
From Me to Everyone: 04:13 PM
Ctr+C works
Ctrl+V
OK