| | Feature Milestones
target: September, 2010 |
| | Functionality for meta-algorithm developers
- Ability to interact with the parameter space of an algorithm (examine domains, conditionalities, etc.) done
- Ability to transform algorithm parameter spaces: log transforms, discretization done
|
|
<
< |
- Ability to run arbitrary algorithms, including other meta-algorithms, in identical fashion done in redesign
|
>
> |
- Ability to run arbitrary algorithms, including other meta-algorithms, in identical fashion done
|
| |
- Ability to monitor the trajectories of all output variables of an executed algorithm run, in real time done
|
|
<
< |
- Ability to query database of previous runs directly done in redesign
- Ability to access instance features in prog. in refactor
- Pre-defined metrics for aggregating performance across runs done in redesign
|
>
> |
- Ability to query database of previous runs directly done
- Ability to access instance features done
- Pre-defined metrics for aggregating performance across runs done
|
| |
Backend functionality exposed in above
- Ability to execute algorithms locally done
- Ability to execute algorithms on a remote host via SSH needs update re: API changes
- Ability to execute algorithms on a SGE cluster needs update re: object API changes
- Ability to actively monitor remotely running algorithms via RPC needs update re: object API changes
|
|
<
< |
- MySQL database storing records of all algorithms, instances, runs, etc. being redesigned now
- SQLite database fallback if MySQL unavailable as above
|
>
> |
- MySQL database storing records of all algorithms, instances, runs, etc. done
- SQLite database fallback if MySQL unavailable done
|
| |
- R interface for performing statistical tests, etc. done
Meta-Algorithms Included |
|
<
< |
- Configuration procedure: ParamILS (external) done; will need minor updates to work with backend redesign
|
>
> |
- Configuration procedure: ParamILS (external) in progress
|
| |
- Configuration procedure: ROAR (internal) done; will need minor updates to work with backend redesign
|
|
<
< |
- Analysis procedure: Paired algorithm comparison done, updated
- Analysis procedure: Single-algorithm analysis done; will need minor updates to work with backend redesign
|
>
> |
- Analysis procedure: Paired algorithm comparison in progress
- Analysis procedure: Single-algorithm analysis in progress
|
| |
Distribution Issues |
| |
- Support for "bag-of-machines" execution manager
Meta-Algorithms Included |
|
<
< | |
>
> | |
| |
- Multi-algorithm comparison
- SATzilla-like portfolio builder
- Parallelized AC
|
| |
- (CF) Selection of performance metric before selecting the configurator to use. What is the exact problem specification for configuration?
- (CN) convenience methods in MetaAlgorithm hiding next(), hasNext(), report() from the 3rd-party developer; instead providing an interface like AlgorithmRun fetchRun(Algorithm a), with no InterruptedException; implies an AlgorithmRun class that can adaptively switch between a "queued" and a "running" implementation for before and after the true environment fetchRun(...) call is made/returns.
- (HH) Service-oriented volunteer computing. See, e.g., "Service-Oriented Volunteer Computing for Massively Parallel Constraint Solving Using Portfolios", Zeynep Kiziltan and Jacopo Mauro, in CPAIOR-2010 proceedings.
|
|
<
< | |
>
> |
- (KLB) Handle network issues (e.g. loss of connection to datamanager, etc.) robustly. Restart runs, etc., as required to ensure that the originally-requested job ultimately completes correctly with as little babysitting by the user as possible.
- (FH) Normalization transform, in addition to existing log transform
|
| |
Active work items
Frontend |
| |
Backend
Release-critical |
|
<
< |
- CN Split algorithms and configuration spaces, allowing run reuse for common-binary configuration spaces. Both DB and Java object model; requires Algorithm refactor below. (CN: done for Java objects; in progress for DB)
- CN Explicit representation of problems/encodings, compatability of algs and instances via problem (encodings). done in Java objects; to do in data management
- CN Refactor code to align class hierarchy with terminology of paper (CN: done for all but configurator implementations)
|
>
> |
- CN Split algorithms and configuration spaces, allowing run reuse for common-binary configuration spaces. Both DB and Java object model; requires Algorithm refactor below. done
- CN Explicit representation of problems/encodings, compatability of algs and instances via problem (encodings). done
- CN Refactor code to align class hierarchy with terminology of paper (CN: done for all but meta-algorithm implementations, which are in progress)
|
| |
- CN Refactor Algorithm/ParameterSpace/Parameter/Domain structure to allow above done
|
|
<
< |
- CN Database schema -- speed-related refactor (CN: in progress)
|
>
> |
- CN Database schema -- speed-related refactor done (may want further tuning)
|
| |
- CN Refactor SSH & RPC execution managers to work under refactor
Important |
|
<
< |
- CN Connection pooling done (contingent on rest of DataManager refactor, above)
- Caching analysis results
- CN Query optimization (CN: in progress)
|
>
> |
- CN Connection pooling done
- Caching analysis results (CN: in progress as part of meta-alg changes above)
- CN Query optimization done (may want more depending on real-world observations)
|
| |
- Selective limitation of run-level archiving (dynamic based on runtime?)
- add incumbentname semantic input to (design) procedures
- instance features
|
| |
- CN DataManager API refinement (in progress as part of DataManager refactor)
- CF N-way performance comparison
- Stale connection issue; incl. robustness to general network issues
|
|
<
< |
- CN Read-only DataManager connection for use by individual MA procedures done (as part of DataManager refactor)
|
>
> |
- CN Read-only DataManager connection for use by individual MA procedures done
|
| |
- Allowing relationships (incl. possible run-reuse) between different-binary "builds" of algorithms, including due to bugfixes, additional exposed parameters, etc. Also for different "versions" (without reuse) corresponding to added funcitonality.
|
|
<
< |
- Ability to quantify membership of configurations to different design spaces
|
>
> |
- Ability to quantify membership of configurations to different design spaces done
|
| |
Release Critical |
|
<
< |
- VC ROAR in HAL done
- VC Calling Matlab from Java done
|
>
> |
- VC ROAR in Java in testing
- VC Calling Matlab from Java in testing
|
| |
- CN parameter transformations (log, discretization, etc.) done
- VC SMBO, calling Matlab for model building/evaluation (VC: implemented, in testing)
- Adapt Weka RF implementation for regression
|
| |
Support/QA/Misc.
Release Critical |
|
<
< |
- JX unit testing: parameters (domains) (in progress)
- unit testing: parameter spaces
|
>
> |
- unit testing: parameters (domains) OK
- unit testing: parameter spaces OK
|
| |
- unit testing: algorithms
- unit testing: execution managers (local, SSH, cluster)
- unit testing: data managers (SQLite, MySQL)
|
| |
Important |
|
<
< |
- CN Order+configure new DB server (CN: ordered; waiting for shipment)
|
>
> |
- CN Order+configure new DB server (CN: waiting for Dave B to make final changeover)
|
| |
- user-facing documentation (help)
|
|
<
< |
- CN Better logging/error-reporting (to console/within HAL). eg: log4j (in progress)
|
>
> |
- CN Better logging/error-reporting (to console/within HAL). eg:*done* (for most cases; exceptions are auto-logged)
|
| |
- CN JX VC Basic Windows support done, in testing
- Better handling of overhead runtime vs. target algorithm runtime
Nice-to-have |
|
<
< |
- JX developer-facing documentation (javadocs) (in progress in parallel with unit testing)
|
>
> |
- developer-facing documentation (javadocs) (in progress in parallel with other work)
|
| |
Bug Reports |
Copyright © 2008-2025 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.