Automated Parameter Tuning and Algorithm Configuration

This is the temporary website for the Freiburg university seminar Automated Parameter Tuning and Algorithm Configuration. I just started my new junior research group Optimization, Learning, and Automated Algorithm Design) in Freiburg and once my webspace is ready this website will move there.

The language of the seminar will be English to help participants practice it.

Background

Many algorithms have free parameters that can be modified to improve performance in practice (e.g., numerical thresholds or discrete algorithm flags). The setting of these parameters often has a drastic impact, and thus algorithm designers optimize their parameters in practice, often manually (a tedious and time-consuming task). In this seminar, we will study an automated approach for this algorithm configuration problem. We will discuss both foundations of algorithmic procedures for solving the general algorithm configuration problem and their applications to optimize algorithm parameters in practice. 

Foundations include concepts from statistics (experimental design, statistical tests), machine learning (regression, stochastic processes), optimization (mixed continuous/discrete optimization, stochastic optimization, global optimization), and artificial intelligence (local search, population-based methods).

Applications span algorithms in most areas of computer science, including, e.g.: formal verification of hardware and software, AI planning, computer graphics, machine learning, compiler optimization, database query optimization, numerical optimization, and algorithm engineering. They also include algorithms in various related areas, ranging from engineering to operations research.

Likely, at some point (e.g., during your MSc thesis) you will need to optimize the parameters of an algorithm you developed. In this seminar, you will learn how to do so in a scientific, convenient, and effective manner. 
Algorithm configuration is also a young and dynamic field of research, full of interesting questions for practica, semester projects, MSc theses, and so on. (If you're interested in any of these during/after the course, let me know.)

Dynamics

We will meet weekly (Friday 12:00-14:00 c.t., in room S1-018) to discuss research papers that participants select from the list of available papers (additional suggestions for application papers are possible, too). For each paper, one participant will have the lead and give a presentation. All participants will read the paper and submit a summary and 3 questions about it before the presentation. After the presentation, all participants will discuss the paper, and its merits and limitations. This discussion will in part be guided by the questions submitted by the participants. 

Towards the end of term, each participant will hand in a (relatively brief) written report (on a presented paper or a related subject). Final marks will be formed using a combination of marks for the oral presentation and written report, quality of summaries and questions submitted, and class participation.

Overall, next to the seminar topic, you will learn several soft-skills necessary in academia: to read and understand research papers and assess their strengths and weaknesses; to orally present in front of your peers; to discuss with your peers; and to summarize on a high level a piece of research you are not intimately familiar with.

Resources

What to put into the final report?
In a nutshell, I think of this report as a detailed summary of the paper you presented that also covers points that would come up in a research discussion about the paper. (I say "paper" here even though technically, you are not restricted to write about the paper you present.)
E.g., some questions that should be discussed at some point in the report next to a detailed summary are the following:

• What is the paper's main contribution and why is it important? 
• How does it relate to other techniques in the literature?
• What are strong and what are weak points about the paper?
• What would be interesting follow-up work? Any possible improvements in the methods? Any further interesting applications?
• Is the code/data available online? If not, is it available on demand (let me know and I'm happy to send an email asking about it)? Does it run off-the-shelf? If not, what problems are there with running it? (I'd put a limited amount of time into this; not longer than a full work day.)

Formatting and length of the final report
Final reports have to be typeset in LaTeX (sorry, but you were warned). We will use the formatting guidelines and electronic templates from the AI conference IJCAI: http://ijcai13.org/files/ijcai13.zip. Reports don't have to be long (you already wrote all the paper summaries); 2 pages in IJCAI style are appropriate. (It's OK if it's a bit longer, but please don't go beyond 4 pages - you might not be able to include everything you would like to include, but such is life in academic writing.) 

For questions, please send me email: seminar@fhutter.de