Project Goals
HAL"> SMAC in HAL
HAL through Java/MA"> Port SMAC to HAL through Java/MATLAB interface (January)
- Implement:
ROAR as instantiation of SMBO framework (FH: 8h)
- Implement:
Get Matlab interface going (FH: 4h + 3h for Mav. to read up)
- Implement:
EIC optimization (using models to select parameter configurations)
-
Simple random sampling: (FH: 4h)
-
Local search: (FH: 8h) (b/c interface with HAL)
- Reproduce all experiments from SMAC paper within HAL through API
- Reproduce some experiments from SMAC paper through web GUI, to make sure it works
Pure Java implementation of SMAC (January -> April)
- Implement Random Forests in Java
- Implement PCA in Java
HAL"> Gathering runtime data from HAL (January -> February)
- e.g. CPLEX, Spear, SAT data sets
HAL (February - Marc"> Analysis in HAL (February -> March)
- Bootstrap & permutation tests in R and Java (1d/wk)
Prioritized Task List
- HAL UI page for SMAC
-
Code clean-up / code review for SMAC --- 3 hours Thursday March 10
-
Make inputs to ExperimentRunner actual inputs to the jar --- 1 day
-
Enable users to just run SMAC (similar to ParamILS), with SMAC still using HAL under the hood
- Parameterize SMAC (--- 3 hrs)
- Random Forests in Java
- PCA in Java
- Edit wrapper scripts (Which ones?)
- Feature: start SMAC from previous run-logs
- Implement capping in ROAR (need CN and FH)
- Fake configuration scenarios (need CN advice and FH)
Weekly Goals/Milestones
May 30 - Jun 3
- GPGO
- Partial results -> do postprocessing
- Compare with CMA-ES and assess which is 'better'.
- Run gpgo
- ParamILS comparison to smac.
May 23 - May 27
- Start experiments with Surrogate Algorithms
May 2 - May 6
Apr 26 - Apr 29
- Continued from last week.
- SMAC UI
- Implement Surrogate Algorithm
- Gathering data:
- Collect existing runtime matrices ii. Generate new runtime matrices from Westgrid
Apr 18 - Apr 22
- Surrogate Configuration Scenarios
- Algorithm runs on Westgrid Random configurations + "good configurations" (See Frank's thesis: 10.2.1)
- Implement surrogate algorithm
- SMAC UI with CF
-
Use FeatureInstanceMetricMetaProblemInsatnce in SMAC code.
Apr 11 - Apr 15
- Surrogate Configuration Scenarios:
- Read up on SATLIB, TSPLIB, MIPLIB, ORLIB
- BBOB
- interface with test problems?
- Debugging
- Being able to start from a previous state for SMAC
Apr 4 - Apr 8
- Look into the #CplexBug
- Debugging SMAC
-
make RunHistory serializable. (What else needs to be serialized?)
- "Fake Configuration Scenarios" - Surrogate Algorithms
-
Results for analysis plugin (in progress...):
- Look into BCI bug in RStatistics
-
Compare search steps instead of runtime
-
Sanity check with different wp/noise parameter.
Mar 28 - Apr 1
- Goal: Results for analysis by Friday, April 8th
-
Implemented additional outputs (from SCD) and confidence intervals for each output
-
Fixed bug with termination
- Comparison for Walksat
-
Wrappers for ubcsat and original (Technically, I don't need ParamILS wrappers because I'll be using HAL...)
-
Bug: wrappers aren't terminated.
- Bug: null pointer exception in RStatistics.
- Experimental runs
-
Compile jar for (old) ActiveConfigurator
- (Old) Paper results:
- /ubc/cs/research/arrow-raid1/hutter/LION5/k_pils-mult-cat/
- .../k_pils-single-cat
- .../N_SMBO_mult for ROAR and SMAC
- -c in the latter ones means continuous
- .../N_SMBO_single for the single instance scenarios
-
Debugging Bootstrap CI plugin
- Debugging SMAC
-
Log whether a selection was random, ei.random, ei.local search.
-
Keep statistics on above.
- Output callstring to smac.jar into the runlog file.
- Be able to start smac from a runlog
- "Cplex bug"
- Test performance for configurations found by ParamILS (when run through HAL) doesn't match results from paper.
-
I need CPLEX 10.1
- Features for runner:
- Start from list of instances instead of a directory.
- (First implementation done, but how to handle relative paths, etc?)
- Implement UI page for SMAC (with features).
Mar 21 - Mar 25
- Implement features for Dave:
-
loading from ParamILS files
-
trajectory file
- instance lists
- bugfixing
Mar 14 - Mar 18
-
Commandline runner that works with Frank's scripts.
Mar 7 - Mar 11
-
SMAC logging - log into the same directory as the results. (Estimate: 1 hr)
- Fix bug in new SMAC version--?
Feb 28 - Mar 4
- Random Forests (Estimate: 1 week?)
-
Feature loading in HAL (Estimate: 2 days)
-
testing with fake feature extractors on dummy instances
-
More extensive testing with "real" feature extractors on real instances (Est: 1 day)
-
Use "K-way comparison" to run all feature extractors (not necessarily faster?)
-
Matlab output/graphs (Estimate: 3 days or when runs finish without exceptions)
- (Tues) EARG presentation
- (Thurs) CO-OP Site Visit
Feb 21 - Feb 25
- Preparation for Mar 1 EARG
- Random Forests (Estimate: 2 weeks)
Feb 14 - Feb 18
Feb 7 - Feb 11
-
Investigate MCR Licenses
-
Rewrite "output" code for runner
-
BUGFIX: Matlab model fails with mixed Integer/Double parameter values (CN)
-
Re-write Experiment Runner to poll DB instead of storing runs.
-
Delay: experimentrunner exceptions when running on cluster
-
Ask DB for instance distributions
Jan 31 - Feb 4
-
Experiment Script:
-
Incorporate HAL changes (immutable meta-problem instances)
- Unit Tests:
-
Write Unit test that fails for NeighbourhoodBuilder (handling Numerical Domain sampling)
- Unit tests for
Selector, Model, Neighbourhood Builder
Jan 24 - Jan 28
- Start BootstrapCI/PermutationTest plugin for HAL
- Westgrid
-
Westgrid JavaBuilder Hello World (lattice, orcinus)
-
Matlab Model (dependence on FH)
-
Move code for reading instance features / building instance list from MatlabRfModel into AbstractModel
-
Drop instances for which features can't be read.
-
Ask DB for instance distributions
- Build smac.jar plugin for HAL (Goal: Jan 26 / dependence on FH)
- Wiki
- HAL WebUI Guide
- ExperimentRunner Guide
- Unit test for model building.
Jan 17 - Jan 21
- Note: when blocking, send email to Frank and move onto next item. Not expected to finish all tasks this week.
- Priority 1: Implement:
model and selector for SMAC via interface with Matlab code cont. from last week
- Generate smac.jar plugin for website (have James/ChrisF test), dependence on FH
- Priority 2: Experiments and experiment setup (from last week)
-
Write script to run experiments from smac paper with FH
- Wiki for steps to run experiments (new item)
- Run CPLEX, SPEAR, SAT instances with GGA, ParamILS, ROAR
-
output results (configuration, performance) to filesystem
- Also, format for matlab plotting
- Priority 3: HAL
- WebUI Guide and reorganize tutorials (this week)
-
Enable better bugfixing by having HAL output the command used to run wrapper (feature is in newest HAL)
- For convenience of users: output line for calling wrapper with final incumbent (with James)
- Priority 4:
Java/Matlab "Hello World" on Westgrid
- Priority 5: Be able to get runtime data from HAL in the same format as webpage below (1000 random configurations with #instances runs each) refer to Hutter 2009 p.61, 225
- Priority 5: Be able to write/read parameter configurations, runtimes per instance indexed by configuration from Java using HAL parameter settings (see cfg file) (this week or next week)
- Priority 5: Edit (.rb) wrapper scripts to allow HAL to terminate it early and delete temporary files; and return exit code for failure (Ask CF)
- (Jan 17) Working on matlab model code in Java.
- (Jan 18) Debugging Java code (model, selector) -- now runs locally on single instance.
- (Jan 18) model: Started dealing with instance features. Started .csv reader.
- (Jan 19) Reviewed and sent pdfscan. Finished csv reader. Started reworking SMBORunner to take CLI args.
- (Jan 20) Reading instance features implemented.
- (Jan 21) Debugging model, selector, runner. Need to change paths in "wrapper" scripts.
Jan 10 - Jan 14
- ROAR Experiments and experiment setup (Goal: Jan 12)
-
Set up
- Note: Unpredicted overhead for debugging : 1 day
- Write script to run experiments from HAL paper with FH- Production experiments (moved to next week)
- Small test run with single instance, uncollected results (
locally Jan 11, beta Jan 12, westgrid Jan 13)
- Run CPLEX, SPEAR, SAT instances with GGA, ParamILS, ROAR and
- output results (configuration, performance) to filesystem (in progress)
-
Register for Westgrid access
(done Jan 13)
-
Implement: model and selector for SMAC via interface with Matlab code (Goal: next week / in progress)
- Done: implemented EI-Optimizing selector without local search (Jan. 14)
Jan 4 - Jan 7
- Getting started.
-
Set up Eclipse and git
-
password-less ssh
- Reading the HAL paper: done first pass
-
Reading the SMAC paper: done
-
Reading the Algorithm Configuration chapter outstanding: pdfscan typos (Jan 19)
-
Refactor ROAR to SMBO framework with FH
-
Interfaces for Selector, Model
-
Test runs with dummy algorithm on local machine, arrow, and beta
BBOB
CPLEX bug infodump
- Use the correct CPLEX version and wrapper: CPLEX 10.1 CPLEX 10 HAL Wrapper (altuning/algorithms/cplex/hal_cplex10_wrapper)
- SMAC paper results
/ubc/cs/project/arrow/hutter/ParamILS/cplex/CATS100-first1000-strat-det_algo/paramils-out
You only need to be concerned about these files:
focused-dec29-runs2000-runobjruntime-overallobjmean10-bm2-time5.0-tunerTime18000.0-algocplex-*
For example, test performance for the first run is in
focused-dec29-runs2000-runobjruntime-overallobjmean10-bm2-time5.0-tunerTime18000.0-algocplex-test_0.txt
The corresponding configuration is at the end of
focused-dec29-runs2000-runobjruntime-overallobjmean10-bm2-time5.0-tunerTime18000.0-algocplex-log_0.txt
Paper results for ParamILS
/ubc/cs/research/arrow-raid1/hutter/LION5/l_pils-mult-cat/CPLEX-regions100-train_CPLEX-regions100-test_false_PAR1halPostFindBugs_10-08-1851.jar/
Update
- It seems like the issue was with using CPLEX12 instead of CPLEX10, causing the validated performances to worsen.
Permutation test notes
- GENERAL PROCEDURE FOR PERMUTATION TESTS To carry out a permutation test based on a statistic that measures the size of an effect of interest:
- Compute the statistic for the original data.
- Choose permutation resamples from the data without replacement in a way that is consistent with the null hypothesis of the test and with the study design. Construct the permutation distribution of the statistic from its values in a large number of resamples.
- Find the P-value by locating the original statistic on the permutation distribution.
- Permutation test for matched pairs. The key step in the general procedure for permutation tests is to form permutation resamples in a way that is consistent with the study design and with the null hypothesis. Our examples to this point have concerned two-sample settings. How must we modify our procedure for a matched pairs design?
Ongoing Tasks
-- Main.mavc - 05 Jan 2011
Topic revision: r101 - 2011-05-30
- mavc