• probably require support for unstructured broadcast communications via data mgr.

• possibly fix by having dedicated per-machine run controller instance? Possibly related to EDACC-style execution? Also related to removal of shared FS.

Versioning supporting in serialization/deserialization

• e.g., the refactor of metrics implied a change in the serialization format, which broke old exports. It is not clear from the resultant error that this breakage was due to a version change.

• ParamILS through HAL vs. standalone

HAL 1.1+ Projects

Memory Efficiency

• redundant instantiation of immutable or largely immutable objects
• inefficient serialized representations of hierarchical/composite objects (currently, data manager manually removes this redundancy for a limited set of objects)
• needed to scale to very large instance sets, deeply nested meta-algorithm instances, etc.

Profiling/Optimizations

• general performance and developer experience improvements

Detection of and Cleanup After Edge Case or Unexpected Behaviour

• e.g. network outages, crashed/hanged/disappeared runs, etc.

Removal of RPC Reliance/Data Manager-centric Communications

• used to implement some AlgorithmRun methods for remote runs (getCpuTime, etc.)
• fragile, and causes noticeable delays in UI

SSH Execution Manager improvements

• many limitations when running targets over SSH (See devel guide)
• possibly fix by having dedicated per-machine run controller instance? Possibly related to EDACC-style execution?

Removal of Shared Filesystem Requirement

• transfer of algs, instances, etc.
• monitoring output for remote SSH and Cluster execution managers

Cluster batching support

• needed for lattice support
• useful in general when distributing many short runs over cluster.

Problem-specific, procedure-agnostic UI

• replace/augment "instances" with space-like introspectable representation
• meta-algorithm instance constraints (single target, etc.) should be explicit instead of via tags
• additional UI elements for specific plugins can be bundled in the plugin

Features needing testing

• Overall unit test coverage
• queueRun support for meta-algs via SubrunRunner.
• Reused runs, duplicate experiments w/different metrics, etc.; esp. edge cases w.r.t. runs that (barely) exceed cutoff time

New Feature: subsidiary analyses

• User can specify e.g. new metrics for already-performed analyses
• HAL uses an Environment w/new target XM that ONLY uses DB-sourced target runs to quickly compute these modified expts
• Only one modified expt per target expt at a time
• (HH) Think about how to let the user specify plots/etc. at an even finer level of granularity, in more appropriate ways

For LION

must-haves

• cloning experiments
• run k-times for configuration
• plot data point labels in plot interface (eventually in the UI as well)
• viewing/cloning algorithms, etc.
• refactoring experiment specification pages
• star-Zilla -- demo with Cplex+gurobi? .. using pre-collected data.
• Torque support given arrow-like architecture assumptions
• rudimentary deletion of runs, etc., and a better plan
• screencast of basic HAL usage

medium term

• result data exporting (general run data)
• Westgrid support

• (2d) generic "selector" which, given a trained model, extracts features, performs inference, and selects/runs a solver
  • (update: fundamentals done; needs refinement though)

• (1d) "internal" target algorithms (to implement trained selector with; note not a meta-algorithmic procedure per LION def'n)
• (2d) generic "selector which, given a trained model, extracts features, performs inference, and selects/runs a solver

• (~2d) Integration of pieces into complete SATzilla pkg

• (~2d) testing

• (~1d) Expose more ParamILS features (BasicILS, SQ tuning, etc etc)

• (~3d??) Problems as predefined semantics; useful mostly for target problems, e.g. planning

HAL-related tasks not c"> Parallel HAL-related tasks not considered in above lists (see below for some details on some of these)

• (ongoing) release mgmt

• • Allow the user to specify the location of missing files; for now we just warn that they aren't present and import anyway?

*zilla+HAL

HAL prep, mostly CN

• (1d) K-way analysis for data collection (no tests yet)
• (1d) Feature extractor as a wrapped external target algorithm
• (1d) "internal" target algorithms (to implement trained selector with; note not a meta-algorithmic procedure per LION def'n)

*zilla, w/LX

• (~2d) High-level control in Java
• (~2d) Model building and inference using Matlab from Java
• (~2d?) UI elements (w CF?)

Hydra+HAL

• plan once *zilla is further along

ActiveConfigurator+HAL

w/FH, ~1-week tasks:

• (2d) Fake configuration scenarios (i.e. fake target algorithms)
• (1d) Update AC project w.r.t. recent HAL changes (mostly FH)
• SMAC
  • (1d) high-level design
  • (2d) impl. using interface to Matlab

• (~1.5d) Viewing/editing/copying of HAL top-level objects

• (~1d) Outstanding import/export backend issues (exec. environments, object descriptions, etc.)

• • incumbent qual plot - extend to current time

• (~2d) decouple toSpec() and getHash(); serialize everything incl. metadata; fix exporting re: this. much less fragile than current situation of an object's hash code depending directly on its serialized representation.
• (~10d?? w/CF?) Finish designing Problem class, integration with rest of system, and implement. Needed for generic UI templates, and for target problems with problem-specific semantics/formats; e.g. planning problems.

• • (2d) MxN (algorithms x instances) data collection meta-alg
  • (3d) Instance feature population using outputs of external algorithms
  • (??) MatlabBuilder/J integration for SATzilla

• • (1d) assess existing SGE manager on TORQUE, assuming assumptions like common file system are satisfied

• • (??) MatlabBuilder/J integration for SMBO

• (~1d) test target run distribution w/SGE Exec Mgr (may be working already..)

• (~1d) Outstanding import/export backend issues (exec. environments, object descriptions, etc.)

• • Allow the user to specify the location of missing files, or maybe just warn that they aren't present in the path and import anyway?

• • meta-algorithm selected second-but last (just before exec environment)
    • Can't do this. Each meta-algorithm can require different input fields.

• • Single alg analysis: refer to alg as "name + (target algorithm)" in monitor/results page
  • Pairwise analysis: refer to alg 1, alg 2 as "name (algorithm 1/2)" in monitor/results page

• (~0.5d) Database deletion of HAL top-level objects, from the landing page

• (~2d) Algorithm specification screen changes, as per previous HAL meeting.

• • Ready to launch runs

• • Ready to launch runs

• (~1d) backend support for user-space textual object annotations.
• (~1d) Outstanding import/export backend issues (exec. environments, object descriptions, etc.)

• • • On change from partial run to actual run, need to make sure descriptions get transferred.

• (~0.5d) Graying out and showing alg spec. steps or something so that users can see how much they have left.

• (~1d) Landing page run status divs should maintain individual state, not have pieces disappear on refresh, and should not appear under more than one heading (or disappear entirely) when the run status changes.

• • "add output capture" below info for last string added
  • examples for params (call string), output capture

• • Algorithm
  • Instance set
  • Experiment
  • Execution environment
  • Configuration

• (~0.5d) Database deletion of HAL top-level objects, from the landing page
  • Algorithm
  • Instance set
  • Experiments
  • Execution environment
  • Configuration

• (~0.5d) Database deletion of HAL top-level objects, from a separate screen
  • Algorithm
  • Instance set
  • Experiments
  • Execution environment
  • Configuration

• (~0.5d) Modify domain transformation interface to accomodate linear transformations / multiple transformations as soon as the backend infrastructure is present.

• (~2d) deeply lazy database algorithm run objects (mostly, lazy meta problem instances)

• (~1d) test target run distribution w/SGE Exec Mgr (may be working already..)
• (~1d) backend support for user-space textual object annotations.

• (~1d) Notes/annotation for 'top-level' objects.

HAL 1.0 Task List

Backend Tasks

4-week plan (CN)

• (~1d) Outstanding import/export backend issues (exec. environments, etc)
• (~3d) job health monitoring (check outstanding jobs e.g. on cluster; detect hangs/problems and alter/cleanup appropriately).
• (~1d) backend support for user-space textual object annotations.
• (~1d) test target run distribution w/SGE Exec Mgr (may be working already..)
• (~1d) sporadic missing finishtime bug

Unprioritized

• (~3d) pause/resume for meta-algorithms [estimate doesn't include pausing target algorithms mid-run, just halting dispatch of new runs]
• (~1d combined) Plotting:
  • incumbent qual plot - extend to current time
  • different colours/linestyles for the two medians in dual SCD plot
  • show cutoff time in plots (dashed lines)
  • different linestyles / labels?? for vertical lines in overlaid SCD plots

HAL-related tasks not c"> Parallel HAL-related tasks not considered in above lists:

• (~2d) setup of new DB server
• (???) documentation
• (??? w/Lin) portfolio-based selection (SATzilla)
• (??? w/James) Improved cluster support (incl. Torque)
• (~1w w/Frank) "virtual" target algorithms (based on Matlab models)
• (??? w/Frank) SMBO (Active Configurator)

Frontend tasks

4-week plan (CF)

• (~1d) Landing page run status divs should maintain individual state, not have pieces disappear on refresh, and should not appear under more than one heading (or disappear entirely) when the run status changes.

• (~2d) Algorithm specification screen changes, as per previous HAL meeting.
  • "add output capture" below info for last string added
  • examples for params (call string), output capture

• (~1d) Rearranging meta-algorithm specification pages, as previously discussed.
  • default name
  • ? Submit -> launch
  • after completing exp page, offer choice to
    • launch
    • save as 'ready for launch'
    • ? save as 'in preparation'
  • Single alg analysis: refer to alg as "name + (target algorithm)" in monitor/results page
  • target algorithm -> algorithm to be analysed
  • hide meta-alg slc (since there is only one choice right now)
  • Pairwise analysis: refer to alg 1, alg 2 as "name (algorithm 1/2)" in monitor/results page
  • target algorithm 1/2 -> algorithm 1/2 to be analysed
  • meta-algorithm selected second-but last (just before exec environment)
  • keep options collapsed by default wherever possible

• (~1d) Various minor previously-reported monitor page changes.
  • auto-reload of monitor pages, interval configurable in HAL settings - local to client/user
  • report HAL overhead also as percentage of overall running time
  • include percentage complete, predicted completion time for experiment
  • single/pairwise analysis: report # / fraction of time-outs
    • adjustment of param names (r) and value (l); 2 columns? scrollable box?
  • Incumbent Design Information: add time when current incumbent was found
    • name for current incumbent / final result? (how can user specify this?)
  • contrast incumbent param settings with default settings (display latter as well)

• (~0.5d) Sub-run information table in the monitor pages, sortable by column headers, not loaded until requested.

• (~0.5d) Database deletion of HAL top-level objects, both from the landing page if applicable and a separate screen.

• (~1.5d) Viewing/editing/copying of HAL top-level objects

Unprioritized

• (~0.5d after backend functionality exists) UI elements for user-specified annotations of HAL objects (eg, descriptions of instance sets, etc.)

• Define New Inst Distr
  • (~0.25d) hide type (until we have > 1 types)

• Define New Alg:
  • (~0.5d) also allow creation of new configs etc. from Create New Alg?
  • ??? easy way to add param domain decl to default (to be copied from there later)
  • ??? alternative ways of specifying param config spaces: text file/text field + parse button?

• ??? number of indep, par runs + auto select best on training. i.e. a new meta-algorithm.

• Alg config results/status:
  • ??? support exporting from inspection pages (for exp, alg, inst sets)

• Export
  • (~0.1d) any -> all?
  • (~0.1d) some labels need to be more intuitive (Filename for Object Export, ...)
  • (~0.1d) size of package should be displayed
  • (0.5d) export instance sets without taking everything in respective directory (checkable option in instance set spec, default = don't take whole dir)

• Import
  • ??? allow user to select within package

• everwhere
  • (mostly covered in 4wk plan) hide (and make expandable) everything that has decent default values most users may not want to touch
  • (~0.25d) capitalisation: do not capitalise 'short words' (yet, not, ...), certain messages ('Not Yet Finished' -> 'not yet finished'), best solution: put all in properties file, let me (hh) edit them.
  • (~0.1d) time & date displays (everywhere): dd-mm-yyyy or yyyy/mm/dd, 24h-times
  • (~0.1d) instance distribution -> instance set
  • (covered mostly in 4wk plan) selection lists with only one choice: make the one choice the default

Old Tasks

-- ChrisFawcett - 04 Nov 2010