Merge pull request #255 from automl/development

Development v 1.2.0

Author: shukon

.gitignore

@@ -3,20 +3,12 @@
 data/
 output/
 results/
-vespy/
 _run_number.txt
-logo.py
-traj2csv.py
-rh2csv.py
 
 # Specific examples
 data/
 data_original/
 results/
-examples/spear_qcp_large/
-examples/spear_qcp_small/example_output_1/validated_rh.json
-examples/spear_qcp_small/example_output_2/validated_rh.json
-examples/spear_qcp_small/example_output_3/validated_rh.json
 
 # Old versions
 *.old
@@ -96,6 +88,7 @@ target/
 
 # pyenv
 .python-version
+.idea
 
 # celery beat schedule file
 celerybeat-schedule

MANIFEST.in

@@ -1,3 +1,4 @@
 include requirements.txt
 recursive-include cave/html/web_files *.css *.js *.png *.json *.svg *.eot *.ttf *.woff*
 include cave/plot/mpl_style
+include cave/utils/options/*.ini

README.md

@@ -2,46 +2,32 @@ Status for master branch / development branch:
 
 [![Build Status](https://travis-ci.org/automl/CAVE.svg?branch=master)](https://travis-ci.org/automl/CAVE) / [![Build Status](https://travis-ci.org/automl/CAVE.svg?branch=development)](https://travis-ci.org/automl/CAVE)
 
-# CAVE 
+# CAVE
 CAVE is a versatile analysis tool for automatic algorithm configurators. It generates comprehensive reports (e.g. http://ml.informatik.uni-freiburg.de/~biedenka/cave.html) that
 give you insights into the configured algorithm, the used instance set and also the configuration tool itself.
-The current version works out-of-the-box with [SMAC3](https://github.com/automl/SMAC3), but can be easily adapted to other configurators, as long as they use the same output-structure.
+The current version works out-of-the-box with [BOHB](https://github.com/automl/HpBandSter) and [SMAC3](https://github.com/automl/SMAC3), but can be easily adapted to other configurators, either by adding a custom reader or by [using the CSV-Reader](https://automl.github.io/CAVE/stable/manualdoc/fileformats.html#csv) integrated in CAVE.
 You can also find a [talk on CAVE](https://drive.google.com/file/d/1lNu6sZGB3lcr6fYI1tzLOJzILISO9WE1/view) online.
 
-# LICENSE 
-Please refer to [LICENSE](https://github.com/automl/CAVE/blob/master/LICENSE)
-
-If you use out tool, please cite us:
-
-```bibtex
-@InProceedings{biedenkapp-lion18a,
-            author = {A. Biedenkapp and J. Marben and M. Lindauer and F. Hutter},
-            title = {{CAVE}: Configuration Assessment, Visualization and Evaluation},
-            booktitle = {Proceedings of the International Conference on Learning and Intelligent Optimization (LION'18)},
-            year = {2018}}
-
-@journal{
-    title   = {BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameters},
-    author  = {M. Lindauer and K. Eggensperger and M. Feurer and A. Biedenkapp and J. Marben and P. Müller and F. Hutter},
-    journal = {arXiv:1908.06756 {[cs.LG]}},
-    date    = {2019},
-}
-```
-
+If you use this tool, please [cite us](#license).
 
+If you have feature requests or encounter bugs, feel free to contact us via the issue-tracker.
 
 # OVERVIEW 
 CAVE is an analysis tool.
-It is written in Python 3.5 and uses [SMAC3](https://github.com/automl/SMAC3), [pimp](https://github.com/automl/ParameterImportance) and [ConfigSpace](https://github.com/automl/ConfigSpace).
-CAVE generates performance-values (e.g. PAR10), scatter- and cdf-plots to compare the default and the optimized incumbent and provides further inside into the optimization process by quantifying the parameter- and feature-importance.
+It is written in Python 3.5 and uses [SMAC3](https://github.com/automl/SMAC3), [pimp](https://github.com/automl/ParameterImportance),  and [ConfigSpace](https://github.com/automl/ConfigSpace).  
+CAVE generates performance-values (e.g. PAR10), scatter- and cdf-plots to compare the default and the optimized incumbent and provides further inside into the optimization process by quantifying the parameter- and feature-importance.  
+CAVE also generates configurator footprints to get a grip on the search behaviour of the configurator and many budget-based analyses.  
+CAVE integrates seamlessly with [jupyter-notebooks](https://automl.github.io/CAVE/stable/manualdoc/jupyternotebook.html).
 
 # REQUIREMENTS
 - Python 3.5
 - SMAC3 and all its dependencies
 - ParameterImportance and all its dependencies
+- HpBandSter and all its dependencies
 - everything specified in requirements.txt
-Some of the plots in the report are generated using [bokeh](https://bokeh.pydata.org/en/latest/). To automagically export them as `.png`s, you need to also install [phantomjs-prebuilt](https://www.npmjs.com/package/phantomjs-prebuilt). CAVE will run without it, but you will need to manually export the plots if you wish to use them.
-- phatomjs-prebuilt
+
+Some of the plots in the report are generated using [bokeh](https://bokeh.pydata.org/en/latest/). To automagically export them as `.png`s, you need to also install [phantomjs-prebuilt](https://www.npmjs.com/package/phantomjs-prebuilt). CAVE will run without it, but you will need to manually export the plots if you wish to use them (which is easily done through a button in the report).
+
 
 # INSTALLATION
 You can install CAVE from pip:
@@ -59,16 +45,19 @@ npm install phantomjs-prebuilt
 ```
 
 # USAGE
-We are currently working on the [documentation](https://automl.github.io/CAVE/stable/) of CAVE. Here a little Quickstart-Guide.
+Have a look into the [documentation](https://automl.github.io/CAVE/stable/) of CAVE. Here a little Quickstart-Guide.
 
-You can analyze multiple folders (that are generated with the same scenario) for the analysis, simply provide the paths to all the individual results in `--folders`.
+You can analyze results of an optimizer in one or multiple folders (that are generated with the same scenario, i.e. parallel runs).
+Provide paths to all the individual parallel results using `--folders`.
 
-Commandline arguments:
+Some helpful commandline arguments:
 - `--folders`: path(s) to folder(s) containing the configurator-output (works with `output/run_*`)
 
+**NOTE:** *the keyword `--folders` is optional, CAVE interprets positional arguments in the commandline as folders of parallel runs*
+
 Optional:
 - `--output`: where to save the CAVE-output
-- `--file_format`: of results to be analyzed, choose from [SMAC3](https://github.com/automl/SMAC3), [SMAC2](https://www.cs.ubc.ca/labs/beta/Projects/SMAC), [CSV](https://automl.github.io/CAVE/stable/quickstart.html#csv) or [BOHB](https://github.com/automl/HpBandSter)
+- `--file_format`: if the automatic file-detection fails for some reason, choose from [SMAC3](https://github.com/automl/SMAC3), [SMAC2](https://www.cs.ubc.ca/labs/beta/Projects/SMAC), [CSV](https://automl.github.io/CAVE/stable/quickstart.html#csv) or [BOHB](https://github.com/automl/HpBandSter)
 - `--validation_format`: of (optional) validation data (to enhance epm-quality where appropriate), choose from [SMAC3](https://github.com/automl/SMAC3), [SMAC2](https://www.cs.ubc.ca/labs/beta/Projects/SMAC), [CSV](https://automl.github.io/CAVE/stable/quickstart.html#csv) or NONE
 - `--ta_exec_dir`: target algorithm execution directories, this should be one or multiple path(s) to
   the directories from which the configurator was run initially. not necessary for all configurators (BOHB doesn't need it). used to find instance-files and
@@ -79,7 +68,7 @@ Optional:
 - `--feature_analysis`: analysis features is expensive, so you can specify which
   algorithm to run: `box_violin`, `clustering`, `importance` and/or `feature_cdf`.
   either provide a combination of those or use `all` or `none`
-- `--no_tabular_analysis`: toggles the tabular analysis
+- `--no_performance_table`: toggles the tabular analysis
 - `--no_ecdf`, `--no_scatter_plots`: toggle ecdf- and scatter-plots
 - `--no_cost_over_time`: toggles the cost-over-time plot
 - `--no_parallel_coordinates`: toggles the parallel-coordinates plot
@@ -90,32 +79,50 @@ Optional:
 - `--cot_inc_traj`: how the incumbent trajectory for the cost-over-time plot will be generated if the optimizer is BOHB (from [`racing`, `minimum`, `prefer_higher_budget`])
 
 For further information on  to use CAVE, see:
-`python scripts/cave.py -h
+`cave -h`
 
 # EXAMPLE
+## SMAC3
 You can run the spear-qcp example like this:
 ```
-cave --folders examples/smac3/example_output/* --ta_exec examples/smac3/ --output CAVE_results/
+cave examples/smac3/example_output/* --ta_exec examples/smac3/ --output output/smac3_example
 ```
 This will analyze the results located in `examples/smac3` in the dirs `example_output/run_1` and `example_output/run_2`.
 The report is located in `CAVE_results/report.html`.
 `--ta_exec` corresponds to the folder from which the optimizer was originally executed (used to find the necessary files for loading the `scenario`).
 For other formats, e.g.:
 ```
-cave --folders examples/smac2/ --ta_exec_dir examples/smac2/smac-output/aclib/state-run1/ --file_format smac2 --no_algorithm_footprint
-cave --folders examples/csv_allinone/ --ta_exec_dir examples/csv_allinone/ --file_format csv
-
+cave examples/smac2/ --ta_exec_dir examples/smac2/smac-output/aclib/state-run1/ --output output/smac2_example
+cave examples/csv_allinone/ --ta_exec_dir examples/csv_allinone/ --output output/csv_example
 ```
 
-# USAGE WITH BOHB
-You can also use cave with Configurators that use budgets to estimate a quality of a certain algorithm (e.g. epochs in
-neural networks), a good example for this behaviour is [BOHB](https://github.com/automl/HpBandSter). To interpret BOHB's
-results, you need to install an additional dependency:
-```
-pip install hpbandster
+## BOHB
+You can also use cave with configurators that use budgets to estimate a quality of a certain algorithm (e.g. epochs in
+neural networks), a good example for this behaviour is [BOHB](https://github.com/automl/HpBandSter).
 ```
-and then you can use CAVE as usual, specifying the file_format as BOHB:
-```
-cave --folders examples/bohb --file_format BOHB --output CAVE_BOHB_results
+cave examples/bohb --output output/bohb_example
 ```
+
 There is an [example jupyter-notebook](https://github.com/automl/HpBandSter/blob/add_docu/hpbandster/examples/Workflow.ipynb) on how to use CAVE with BOHB.
+# LICENSE 
+Please refer to [LICENSE](https://github.com/automl/CAVE/blob/master/LICENSE)
+
+If you use out tool, please cite us:
+
+```bibtex
+@InProceedings{biedenkapp-lion18a,
+            author = {A. Biedenkapp and J. Marben and M. Lindauer and F. Hutter},
+            title = {{CAVE}: Configuration Assessment, Visualization and Evaluation},
+            booktitle = {Proceedings of the International Conference on Learning and Intelligent Optimization (LION'18)},
+            year = {2018}}
+
+@journal{
+    title   = {BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameters},
+    author  = {M. Lindauer and K. Eggensperger and M. Feurer and A. Biedenkapp and J. Marben and P. Müller and F. Hutter},
+    journal = {arXiv:1908.06756 {[cs.LG]}},
+    date    = {2019},
+}
+```
+
+
+

cave/__version__.py

@@ -1 +1 @@
-__version__ = "1.1.8"
+__version__ = "1.2.0"

cave/analyzer/algorithm_footprint.py

@@ -1,37 +1,75 @@
 import os
-from collections import OrderedDict
-import logging
 
-from bokeh.plotting import show
-from bokeh.io import output_notebook
 from bokeh.embed import components
+from bokeh.io import output_notebook
+from bokeh.plotting import show
 
 from cave.analyzer.base_analyzer import BaseAnalyzer
 from cave.plot.algorithm_footprint import AlgorithmFootprintPlotter
+from cave.utils.helpers import NotApplicable
 from cave.utils.timing import timing
 
+
 class AlgorithmFootprint(BaseAnalyzer):
+    """
+    The instance features are projected into a two/three dimensional space using principal component analysis (PCA)
+    and the footprint of each algorithm is plotted, i.e., on which instances the default or the optimized
+    configuration performs well. In contrast to the other analysis methods in this section, these plots allow
+    insights into which of the two configurations performs well on specific types or clusters of instances. Inspired
+    by Smith-Miles.
+    """
 
     @timing
-    def __init__(self, algorithms, epm_rh, train, test, features,
-                 cutoff, output_dir, rng, density=200, purity=0.95):
-        self.logger = logging.getLogger(self.__module__ + '.' + self.__class__.__name__)
+    def __init__(self,
+                 runscontainer,
+                 density=200,
+                 purity=0.95):
+        """
+        Parameters
+        ----------
+        runscontainer: RunsContainer
+            contains all important information about the configurator runs
+        """
+        super().__init__(runscontainer)
+
+        # Aggregated run over current runscontainer
+        self.logger.info("Note: Algorithm Footprint does not support budgets / fidelities yet.")
+        agg_run = self.runscontainer.get_aggregated(False, False)[0]
+
+        algorithms = [(agg_run.default, "default"),
+                      (agg_run.incumbent, "incumbent")]
+        epm_rh = agg_run.epm_runhistory
+        train = agg_run.scenario.train_insts
+        test = agg_run.scenario.test_insts
+        features = agg_run.scenario.feature_dict
+        cutoff = agg_run.scenario.cutoff
+        output_dir = agg_run.output_dir
+        rng = agg_run.rng
 
         # filter instance features
+        self.logger.debug("Features: " + str(features))
         train_feats = {k: v for k, v in features.items() if k in train}
         test_feats = {k: v for k, v in features.items() if k in test}
         if not (train_feats or test_feats):
             self.logger.warning("No instance-features could be detected. "
                                 "No algorithm footprints available.")
-            raise ValueError("Could not detect any instances.")
+            raise NotApplicable("Could not detect any instances.")
 
         self.logger.info("... algorithm footprints for: {}".format(",".join([a[1] for a in algorithms])))
-        self.footprint = AlgorithmFootprintPlotter(epm_rh,
-                                              train_feats, test_feats,
-                                              algorithms,
-                                              cutoff,
-                                              output_dir,
-                                              rng=rng)
+
+        try:
+            self.footprint = AlgorithmFootprintPlotter(epm_rh,
+                                                   train_feats, test_feats,
+                                                   algorithms,
+                                                   cutoff,
+                                                   output_dir,
+                                                   rng=rng)
+        except ValueError as err:
+            self.logger.debug(err, exc_info=1)
+            self.error = str(err)
+
+    def get_name(self):
+        return "Algorithm Footprint"
 
     def _plot(self):
         # Plot footprints
@@ -45,18 +83,22 @@ def get_jupyter(self):
         show(bokeh_plot)
 
     def get_html(self, d=None, tooltip=None):
-        script, div = components(self._plot())
-        bokeh_components = script, div
-        if d is not None:
-            d["tooltip"] = tooltip
-            # Interactive bokeh-plot
-            d["Interactive Algorithm Footprint"] = {"bokeh" : (script, div)}
-            for plots in self.plots3d:
-                header = os.path.splitext(os.path.split(plots[0])[1])[0][10:-2]
-                header = header[0].upper() + header[1:].replace('_', ' ')
-                d[header] = {"figure_x2": plots}
-
-        return bokeh_components
+
+        if self.error:
+            if d is not None:
+                d["Algorithm Footprint"] = {"else": self.error}
+            return self.error
+        else:
+            bokeh_components = components(self._plot())
+            if d is not None:
+                d["Algorithm Footprint"] = {"tooltip" : self.__doc__}
+                # Interactive bokeh-plot
+                d["Algorithm Footprint"]["Interactive Algorithm Footprint"] = {"bokeh" : bokeh_components}
+                for plots in self.plots3d:
+                    header = os.path.splitext(os.path.split(plots[0])[1])[0][10:-2]
+                    header = header[0].upper() + header[1:].replace('_', ' ')
+                    d["Algorithm Footprint"][header] = {"figure_x2": plots}
+            return bokeh_components
 
     def get_plots(self):
         all_plots = []