Automated Parameter Tuning and Algorithm Configuration

Seminar at Freiburg University, Summer term (SS) 2013

Home | Papers to be discussed | Schedule

List of papers to select from

Note: Please select your favourite papers by April 29; we'll assign papers on April 30.

Foundations (papers introducing methods for parameter tuning / algorithm configuration)

Bayesian optimization for continuous blackbox functions: EGO. Efficient Global Optimization of Expensive Black-Box Functions. Journal of Global Optimization, 1998. A classic.
(Improved) Bayesian optimization for continuous blackbox functions: GPGO (code). Gaussian Processes for Global Optimization. Proceedings of LION 2009: Intl. Conference on Learning and Intelligent Optimization. Various improvements over EGO.
Evolutionary strategies for continuous blackbox functions: CMA-ES (code). Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation. Proceedings of CEC 1996: Intl. Conference on Evolutionary Computation. Relatively theoretical, basis for a very successful algorithm.
Local search for discrete algorithm configuration: ParamILS (code). Automatic Algorithm Configuration based on Local Search. Proceedings of AAAI 2007: Twentysecond Conference on Artificial Intelligence. First approach for algorithm configuration with many discrete parameters.
Genetic algorithms for algorithm configuration: GGA (code). A Gender-Based Genetic Algorithm for the Automatic Configuration of Algorithms. Proceedings of CP 2009: Principles and Practice of Constraint Programming. Exploits some parallelism.
Bayesian optimization for algorithm configuration: SMAC (code). Sequential Model-Based Optimization for General Algorithm Configuration. Proceedings of LION 2011: Intl. Conference on Learning and Intelligent Optimization. Improvement & generalization of EGO for algorithm configuration.
Racing strategies for algorithm configuration: Iterated F-Race (code). F-Race and Iterated F-Race: an overview. Technical report, then book chapter in Experimental methods for the analysis of optimization algorithms. Focusses on selecting the best approach under noise, not so much on the search.
New: Random search for hyper-parameter optimization: Random Search for Hyperparameter Optimization. Journal of Machine Learning Research (JMLR), 2012.
New: EGO and TPE for hyper-parameter optimization (code). Algorithms for Hyper-Parameter Optimization. Proceedings of NIPS 2011: Neural Information Processing Systems.
New: Bayesian optimization under "environmental variables". Sequential Design of Computer Experiments to Minimize Integrated Response Functions. Statistica Sinica, 2000.
New: Bayesian optimization for Multi-Objective Optimization. Active Learning for Multi-Objective Optimization. Proceedings of ICML 2013: International Conference on Machine Learning.
New: Parallel Bayesian Optimization. Parallelizing Exploration Exploitation Tradeoffs with Gaussian Process Bandit Optimization, Proceedings of ICML 2012: International Conference on Machine Learning.
New: Bayesian optimization in High Dimensions. Bayesian Optimization in High Dimensions via Random Embeddings, Proceedings of IJCAI 2013: International Joint Conference on AI.
New: Portfolio of Bayesian optimization methods. Portfolio Allocation for Bayesian Optimization, Proceedings of UAI 2011: International Conference on Uncertainty in AI.
New: Parallel Algorithm Configuration. Parallel Algorithm Configuration, Proceedings of LION 2013: Intl. Conference on Learning and Intelligent Optimization

Applications (papers applying parameter tuning / algorithm configuration)

Planning 1: FD-Autotune: Domain-Specific Configuration using Fast Downward, Proceedings of ICAPS-PAL 2011: ICAPS Workshop on Planning and Learning.
Planning 2: Learning Portfolios of Automatically Tuned Planners, Proceedings of ICAPS 2012: International Conference on Planning and Scheduling.
Formal verification: Boosting Verification by Automatic Tuning of Decision Procedures, Proceedings of FMCAD 2007: Formal Methods in Computer-Aided Design.
Robotics 1: Automated Gait Optimization using Gaussian Process Regression, Proceedings of IJCAI 2007: International Joint Conference on Artificial Intelligence.
Robotics 2: Adapting Control Policies for Expensive Systems to Changing Environments, Proceedings of IROS 2011: International Conference on Intelligent Robots and Systems.
Machine Learning 1: Practical Bayesian Optimization of Machine Learning Algorithms, Proceedings of NIPS 2012: Neural Information Processing Systems.
Machine Learning 2: Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms, ArXiv preprint, 2012.
Computer Vision: Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures, Proceedings of ICML 2013: International Conference on Machine Learning.
Graphics: A Bayesian Interactive Optimization Approach to Procedural Animation Design, ACM SIGGRAPH/Eurographics Symposium on Computer Animation, 2010.
Parallel Computing 1: High-level optimization via automated statistical modeling, Proceedings of PPOPP 1995: Principles and Practice of Parallel Programming.
Parallel Computing 2: Automated Empirical Optimization of Software and the ATLAS project, Parallel Computing Journal, 2001.
Algorithm Engineering: Optimizing Sorting with Genetic Algorithms, Proceedings of CGO 1995: International Symposium on Code Generation and Optimization.
High-Performance Computing: Automatic tuning of inlining heuristics. In Proceedings of SC 2005: ACM/IEEE Conference on Supercomputing.