SMACTaskList < BETA

%TOC%


---++Project Goals
---+++SMAC in !HAL
---++++Port SMAC to !HAL through Java/MATLAB interface (January)
   * Implement: <s>ROAR as instantiation of SMBO framework</s> (FH: 8h)
   * Implement: <s>Get Matlab interface going (FH: 4h + 3h for Mav. to read up)</s>
   * Implement: <s>EIC optimization (using models to select parameter configurations)</s>
      * <s>Simple random sampling:</s> (FH: 4h)
      * <s>Local search:</s> (FH: 8h) (b/c interface with HAL)
   * Reproduce all experiments from SMAC paper within HAL through API
   * Reproduce some experiments from SMAC paper through web GUI, to make sure it works

---++++Pure Java implementation of SMAC (January -> April)
   * Implement Random Forests in Java
   * Implement PCA in Java

---+++Gathering runtime data from !HAL (January -> February)
   * e.g. CPLEX, Spear, SAT data sets

---+++Analysis in !HAL (February -> March)
   * Bootstrap & permutation tests in R and Java (1d/wk)

---++ Prioritized Task List

   * !HAL UI page for SMAC
   * <s>Code clean-up / code review for SMAC --- 3 hours Thursday March 10 </s>
   * <s>Make inputs to !ExperimentRunner actual inputs to the jar --- 1 day </s>
   * <s>Enable users to just run SMAC (similar to !ParamILS), with SMAC still using !HAL under the hood </s>

   * Parameterize SMAC (--- 3 hrs)
   * Random Forests in Java
   * PCA in Java

   * Edit wrapper scripts (Which ones?)

   * Feature: start SMAC from previous run-logs

   * Implement capping in ROAR (need CN and FH)
   * Fake configuration scenarios (need CN advice and FH)

---++Weekly Goals/Milestones

---+++ May 30 - Jun 3
   * GPGO
      * Partial results -> do postprocessing
      * Compare with CMA-ES and assess which is 'better'.
      * Run gpgo
   * ParamILS comparison to smac.

---+++ May 23 - May 27
   * Start experiments with Surrogate Algorithms

---+++ May 2 - May 6
   * Continued from last week.
   * BBOB [[http://coco.gforge.inria.fr/doku.php?id=bbob-2010-downloads]]
   * Call !HAL from matlab in order to do target runs. (Look at paramils code) 

---+++ Apr 26 - Apr 29
   * Continued from last week.
   1. SMAC UI
   2. Implement Surrogate Algorithm
   3. Gathering data:
      i. Collect existing runtime matrices
      ii. Generate new runtime matrices from Westgrid

---+++ Apr 18 - Apr 22
   A. Surrogate Configuration Scenarios
      i. Algorithm runs on Westgrid
         Random configurations + "good configurations" (See Frank's thesis: 10.2.1)
      i. Implement surrogate algorithm
   A. SMAC UI with CF
      i. <s>Use !FeatureInstanceMetricMetaProblemInsatnce in SMAC code.</s>

---+++ Apr 11 - Apr 15
   * Surrogate Configuration Scenarios:
      * Read up on SATLIB, TSPLIB, MIPLIB, ORLIB
      * BBOB
         * interface with test problems?

   * Debugging
      * Being able to start from a previous state for SMAC

---+++ Apr 4 - Apr 8
   * Look into the [[#CplexBug]]
   * Debugging SMAC
      * <s>make !RunHistory serializable.</s> (What else needs to be serialized?)
   * "Fake Configuration Scenarios" - Surrogate Algorithms
   * <s>Results for analysis plugin (in progress...):</s>
      0 Look into BCI bug in RStatistics
      1 <s>Compare search steps instead of runtime</s>
      2 <s>Sanity check with different wp/noise parameter.</s>

---+++ Mar 28 - Apr 1
   * Goal: Results for analysis by <b>Friday, April 8th</b>
      * <s>Implemented additional outputs (from SCD) and confidence intervals for each output</s>
      * <s>Fixed bug with termination</s>
      * Comparison for Walksat
         * <s>Wrappers for ubcsat and original</s> (Technically, I don't need !ParamILS wrappers because I'll be using !HAL...)
         * <s>Bug: wrappers aren't terminated.</s>
         * Bug: null pointer exception in RStatistics.
         * Experimental runs
   * <s>Compile jar for (old) !ActiveConfigurator</s>
   * (Old) Paper results:
      * /ubc/cs/research/arrow-raid1/hutter/LION5/k_pils-mult-cat/
      * .../k_pils-single-cat
      * .../N_SMBO_mult for ROAR and SMAC
      * -c in the latter ones means continuous
      * .../N_SMBO_single for the single instance scenarios
   * <s>Debugging Bootstrap CI plugin</s>
   * Debugging SMAC
      * <s>Log whether a selection was random, ei.random, ei.local search.</s>
      * <s>Keep statistics on above.</s>
      * Output callstring to smac.jar into the runlog file.
      * Be able to start smac from a runlog
   * "Cplex bug"
      * Test performance for configurations found by !ParamILS (when run through HAL) doesn't match results from paper.
      * <s>I need CPLEX 10.1</s>
   * Features for runner:
      * Start from list of instances instead of a directory.
         * (First implementation done, but how to handle relative paths, etc?)
   * Implement UI page for SMAC (with features).

---+++ Mar 21 - Mar 25
   * Implement features for Dave:
      * <s>loading from !ParamILS files</s>
      * <s>trajectory file</s>
      * instance lists
      * bugfixing

---+++ Mar 14 - Mar 18
   * <s>Commandline runner that works with Frank's scripts.</s>

---+++ Mar 7 - Mar 11
   * <s>SMAC logging - log into the same directory as the results.</s> (Estimate: 1 hr)
   * Fix bug in new SMAC version--?

---+++ Feb 28 - Mar 4

   * Random Forests (Estimate: 1 week?)
   * <s>Feature loading in !HAL</s> (Estimate: 2 days)
      * <s>testing with fake feature extractors on dummy instances</s>
      * <s>More extensive testing with "real" feature extractors on real instances</s> (Est: 1 day)
      * <s>Use "K-way comparison" to run all feature extractors</s> (not necessarily faster?)
   * <s>Matlab output/graphs</s> (Estimate: 3 days or when runs finish without exceptions)
   * (Tues) EARG presentation
   * (Thurs) CO-OP Site Visit

---+++ Feb 21 - Feb 25
   * Preparation for Mar 1 EARG
   * Random Forests (Estimate: 2 weeks)

---+++ Feb 14 - Feb 18
   * <s>SMAC Logging</s>

---+++ Feb 7 - Feb 11
   * <s>Investigate MCR Licenses</s>
   * <s>Rewrite "output" code for runner</s>
   * <s>BUGFIX: Matlab model fails with mixed Integer/Double parameter values (CN)</s>
   * <s>Re-write Experiment Runner to poll DB instead of storing runs.</s>
   * <s>Delay: experimentrunner exceptions when running on cluster</s>
   * <s>Ask DB for instance distributions</s> 

---+++ Jan 31 - Feb 4
   * <s>Experiment Script:</s>
      * <s>Incorporate !HAL changes (immutable meta-problem instances)</s>
   * Unit Tests:
      * <s>Write Unit test that fails for !NeighbourhoodBuilder (handling Numerical Domain sampling)</s>
      * Unit tests for <s>Selector</s>, Model, <s>Neighbourhood Builder</s>

---+++ Jan 24 - Jan 28
   * Start !BootstrapCI/PermutationTest plugin for !HAL
   * Westgrid
      * <s>Westgrid !JavaBuilder Hello World (lattice, orcinus)</s>
   * <s>Matlab Model</s> (dependence on FH)
      * <s>Move code for reading instance features / building instance list from !MatlabRfModel into !AbstractModel</s>
      * <s>Drop instances for which features can't be read.</s>
   * <s>Ask DB for instance distributions</s>
   * Build smac.jar plugin for !HAL (Goal: Jan 26 / dependence on FH)
   * Wiki
      * !HAL !WebUI Guide
      * !ExperimentRunner Guide
   * Unit test for model building.

---+++ Jan 17 - Jan 21
   * _Note: when blocking, send email to Frank and move onto next item. Not expected to finish all tasks this week._ 
   * *Priority 1:* Implement: <s>model and selector for SMAC via interface with Matlab</s> code cont. from last week
      * Generate smac.jar plugin for website (have James/ChrisF test), dependence on FH      
   * *Priority 2:* Experiments and experiment setup (from last week)
      * <s>Write script to run experiments from !smac paper</s> with FH 
         * Wiki for steps to run experiments (new item)
         * Run CPLEX, SPEAR, SAT instances with GGA, !ParamILS, ROAR
         * <s>output results (configuration, performance) to filesystem</s>
         * Also, format for matlab plotting
   * *Priority 3:* !HAL
      * !WebUI Guide and reorganize tutorials (this week)
      * <s>Enable better bugfixing by having !HAL output the command used to run wrapper</s> (feature is in newest !HAL)
      * For convenience of users: output line for calling wrapper with final incumbent (with James)
   * *Priority 4:* <s>Java/Matlab "Hello World" on Westgrid</s>
   * *Priority 5:* Be able to get runtime data from !HAL in the same format as webpage below
                  (1000 random configurations with #instances runs each) refer to Hutter 2009 p.61, 225 
      * Read Ch.4 in FH thesis and [[http://www.cs.ubc.ca/labs/beta/Projects/AAC/empirical_analysis/index.html]]
   * *Priority 5:* Be able to write/read parameter configurations, runtimes per instance indexed by configuration from Java
                   using !HAL parameter settings (see cfg file) (this week or next week)
   * *Priority 5:* Edit (.rb) wrapper scripts to allow !HAL to terminate it early and delete temporary files; and return exit code for failure (Ask CF)
   * *(Jan 17)* Working on matlab model code in Java.
   * *(Jan 18)* Debugging Java code (model, selector) -- now runs locally on single instance.
   * *(Jan 18)* model: Started dealing with instance features. Started .csv reader.
   * *(Jan 19)* Reviewed and sent pdfscan. Finished csv reader. Started reworking SMBORunner to take CLI args.
   * *(Jan 20)* Reading instance features implemented.
   * *(Jan 21)* Debugging model, selector, runner. Need to change paths in "wrapper" scripts.

---+++ Jan 10 - Jan 14
   * ROAR Experiments and experiment setup (Goal: Jan 12)
      * <s>Set up</s>
      * _Note: Unpredicted overhead for debugging : 1 day_
      * Write script to run experiments from !HAL paper with FH- Production experiments (moved to next week)
         * Small test run with single instance, uncollected results (<s>locally</s> Jan 11, <s>beta</s> Jan 12, <s>westgrid</s> Jan 13)
         * Run CPLEX, SPEAR, SAT instances with GGA, !ParamILS, ROAR and
         * output results (configuration, performance) to filesystem (in progress)
   * <s>Register for [[http://www.westgrid.ca/support/accounts/CCDB][Westgrid access]]</s> (done Jan 13)
      * <s>Lattice access?</s>
   * <s>Implement: model</s> and selector for SMAC via interface with Matlab code (Goal: next week / in progress)
      * Done: implemented EI-Optimizing selector without local search (Jan. 14)

---+++ Jan 4 - Jan 7
   * Getting started.
      * <s>Set up Eclipse and git</s>
      * <s>password-less ssh</s>
      * Reading the !HAL paper: done first pass
      * <s>Reading the SMAC paper</s>: done
      * <s>Reading the Algorithm Configuration chapter</s> <s>outstanding: pdfscan typos (Jan 19)</s>
   * <s>Refactor ROAR to SMBO framework</s> with FH
      * <s>Interfaces for Selector, Model</s>
   * <s>Test runs with dummy algorithm on local machine, arrow, and beta</s>

---++ BBOB
   * BBOB Code: [[http://coco.gforge.inria.fr/doku.php?id=bbob-2010-downloads]]
   * /ubc/cs/project/arrow/hutter/bbob/bbob.v10.2/matlab

#CplexBug
---++ CPLEX bug infodump

   * Use the correct CPLEX version and wrapper: CPLEX 10.1 CPLEX 10 !HAL Wrapper (altuning/algorithms/cplex/hal_cplex10_wrapper)
   * SMAC paper results

   * Step 1: validation runs

/ubc/cs/project/arrow/hutter/ParamILS/cplex/CATS100-first1000-strat-det_algo/paramils-out

You only need to be concerned about these files:
focused-dec29-runs2000-runobjruntime-overallobjmean10-bm2-time5.0-tunerTime18000.0-algocplex-*

For example, test performance for the first run is in
focused-dec29-runs2000-runobjruntime-overallobjmean10-bm2-time5.0-tunerTime18000.0-algocplex-test_0.txt

The corresponding configuration is at the end of
focused-dec29-runs2000-runobjruntime-overallobjmean10-bm2-time5.0-tunerTime18000.0-algocplex-log_0.txt

---+++ Paper results for !ParamILS
/ubc/cs/research/arrow-raid1/hutter/LION5/l_pils-mult-cat/CPLEX-regions100-train_CPLEX-regions100-test_false_PAR1halPostFindBugs_10-08-1851.jar/

---+++ Update
   * It seems like the issue was with using CPLEX12 instead of CPLEX10, causing the validated performances to worsen.

---++ Permutation test notes
   * GENERAL PROCEDURE FOR PERMUTATION TESTS
     To carry out a permutation test based on a statistic that measures the
     size of an effect of interest:
   1 Compute the statistic for the original data.
   1 Choose permutation resamples from the data without replacement
     in a way that is consistent with the null hypothesis of the test and with
     the study design. Construct the permutation distribution of the statistic
     from its values in a large number of resamples.
   1 Find the P-value by locating the original statistic on the permutation
     distribution.

   * Permutation test for matched pairs. The key step in the general procedure
     for permutation tests is to form permutation resamples in a way that is
     consistent with the study design and with the null hypothesis. Our examples
     to this point have concerned two-sample settings. How must we modify our
     procedure for a matched pairs design?

---++Ongoing Tasks
   * split: HowToGuides HALWebTutorial
   * Wiki


-- Main.mavc - 05 Jan 2011
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
pdf	moore14.pdf	r1	manage	515.1 K	2011-04-07 - 00:14	UnknownUser	Resampling Notes
This topic: BETA > SMACTaskList
Topic revision: r101 - 2011-05-30 - mavc