ICON Challenge on Algorithm Selection

Challenge results

The challenge is now closed. We received a total of 8 submissions. The final ranking of submissions is:

Rank Submission
1 zilla
2 autofolio
3 zillafolio
4 ASAP_RF
5 ASAP_kNN
6 flexfolio-schedules
7 sunny-presolv
8 sunny

Congratulations to the winner zilla and ASAP_RF, which receives an honourable mention. More detailed results are available in this PDF. Thanks to all participants for taking the time to prepare submissions.

The description of the results is also available on arXiv.

All data, code and results are available for download here.

Challenge description

The algorithm selection problem is to select the best algorithm for solving a given problem. It is relevant where algorithm portfolios are employed and instead of tackling a set of problem instances with just a single algorithm, a set of them is used with a subset, which may be of size 1, being selected for each instance.

A common approach to algorithm selection in practice is to characterise the problem instance to be solved through sets of features that can be extracted in a computationally efficient manner. These features, along with ground truth data of algorithm performance on some problem instances, are then used to induce performance models of the portfolio and its constituent algorithms. Usually, machine learning is used to induce such models. To solve a given new problem instance, the learned model makes a prediction as to the most suitable algorithm(s).

ASlib release 1.0 (https://github.com/coseal/aslib_data/releases) comprises 13 algorithm selection scenarios with different numbers of algorithms, problem instances, and features. We leverage this diverse data set for the ICON challenge on algorithm selection.

Challenge

Your mission, should you choose to accept it, is to build algorithm selection systems based on the ASlib data sets. Your systems must accept input in the ASlib data format to build the performance model(s) and to make predictions for new problem instances (determined by their features).

You will not be required to run any algorithms or compute any features yourself -- all of this information is contained in the ASlib data. Your task is to make the best use of this information.

Please read the problem description for more details on data formats and evaluation.

Download the challenge flyer.