What's in store for Auto-Sklearn? -- From the Developers

[eddiebergman]

What's going on?

Auto-Sklearn has recently been under-maintained, we appreciate that this has caused many users to face dependency issues as pinned dependencies slowly start going out of data. While we support this project primarily through academic means, we are still proud of the community that has formed around it and are dedicated to push it forward.

Will Auto-Sklearn still be maintained?

Yes, auto-sklearn will be maintained and updated moving forward! We initially tried some of these updates, e.g. #1611, #1618 but there were larger issues at play. To alleviate this, we are currently working on a major refactor of the tool, introducing more flexibility and long-wanted features, including pipeline export, flexible pipelines, and a modular design. We expect the first prototype will be available within the next 1-2 months.

Why the refactor?

Auto-Sklearn was initially built during Python 2 and during the eariler days of scikit-learn. Machine learning libraries and their eco-system were still developing and a lot has changed since then. There were also a lot of lessons learned which while easy in concept, truly difficult to integrate into the current design.

Doing research with Auto-Sklearn has also become harder. By becoming a robust and well-performing tool, this has made performing novel research with Auto-Sklearn more difficult.

What to expect?

... Not that much, it's a refactor to get back to where we were but with the goal to make it more extensible.

We will still maintain the front facing AutoSklearnClassifier and AutoSklearnRegressor, to act primarily as it did before and staying very scikit-learn like with it's simple interface.

This refactor will allow us to solve some long standing issues that have arose. We looked through all the issues and tried to categorize what this new refactor will enable. Not all of these issues will be solved upon release but they will provide a tangible rode towards these.

• We will have a new flexible scheduling system, allowing users to hook into events as they happen, hopefully handling issues like:
  • Error running "fit" with many cores. #1236
  • [Question] Modify Stopping Criterion to Accuracy #1624
  • callback function error! #1569
  • Dask has stopped support for 3.7 #1522
  • S3 support for auto-sklearn to store and load models and configurations for each run #986
  • Making auto-sklearn fit methods be interactively stoppable and still give a fitted model #397
• A more flexible pipeline definition, allowing you to create your own or just modify the default, solving:
  • When adding NoPreprocessing component to auto-sklearn, the lassoregression can run successfully, while the abess regression crashed #1661
  • Feature Request: AutoSklearnOutlierDetector #578
  • How to apply a custom preprocessor to only specified features #1110
  • [Question] Is it possible to change the hyperparameter space of an algorithm? #1587
  • [Question] How can I make sure AutoSklearn is always using StandardScaler for feature preprocessing? #1548
  • Can Autosklearn handle Multi-Class/Multi-Label Classification and which classifiers will it use? #1429
  • [Question] Are there any alternatives to One-hot encoding? #1268
  • Is that possible set the initial value of hyperparams when use auto sklearn to search #577
  • Enhancement: Make the Ordinal Encoder a encoder choice #1150
  • Custom pipelines #379
• Auto-Sklearn will allow you to optimize your own custom sklearn pipelines and try it's darn best to return you pure functioning sklearn pipelines (no auto-sklearn custom parts attached). This means you will be able to run any library that supports sklearn pipelines. This should allow great strides towards:
  • convert to scikit learn code. #388
  • Allow autosklearn to export ONNX model #1006
  • [Question] How to know the data and feature preprocessing used in the ensemble? #1633
  • [Question] Are their any methods to get all models, not only used in ensemble?  #1667
  • [Question] Rebuilding Auto Sklearn pipelines with the parameter dictionary returned by .cv_results_ #1663
  • [Question] Is to_sklearn() available now? #1641
  • Can Autosklearn be used with SHAP? #1272
  • [Question] Integration with sklearn-evaluation #1640
  • [Question] How to get values of categorical variables from a fit model? #1634
  • Third Party Components not shared with spawned child processes when n_jobs > 1 #1607
  • [Question] Is it possible to use Scikit-learn version >=1.1.1 #1597
  • [Question] Viewable Preprocessors and Regressors (internal mechanisms)? #1600
  • How can I get/export a production model from a trained model (after refit) with autosklearn? #1467
  • How to see the selected features when ensemble size = 1? #1102
  • [Question] Is there any straight forward way to retrieve the solution and prediction vector during CV? #1448
  • Is it possible to integrate a metric of imblearn as a scorer? #786
• By refactoring, we can also use newer features of sklearn, that previously we tried to bolt in, but was never a first class citizen at the time of Auto-Sklearn's conception.
  • How to give sample weights? #288
  • [Maint] Specify encoded_missing_value to OrdinalEncoder #1615
  • [Research] Use grouping of infrequent categories in OneHotEncoder #1614
  • [Research] Use parameter quantile in HistGradientBoostingRegressor #1613
  • How to weight a given class? [class balancing] #1596
  • Transformer should accept y argument in the transform method #1494
  • Update SGD regressor loss values for scikit learn 1.0 #1334

What can I do?

Please let us know what you think and what you'd like to see from this rebuild!

[AmirAlavi]

I think that such a refactor would be very beneficial, and make auto-sklearn even more useful. Are there opportunities to contribute to this refactor?

[eddiebergman]

Hi @AmirAlavi, unfortunately not at this time but once we publish there will be definitely be a lot of opportunities to contribute! Thanks very much for offering, we appreciate it :)

[manuelinfosec]

Hi, @eddiebergman. I and my team are really looking forward to AutoML integration to ONNX formats. We've been really excited working with AutoML do far, and can wait to take interoperability to another level.

[eddiebergman]

Hello, brief update:

We're in the process of doing some benchmarking, namely to ensure we can still handle most datasets. Progress after this will be to re-enable some more performance features of the original auto-sklearn, namely iterative fitting, meta-learning and AutoSklearn2.

@manuelinfosec I am unfamiliar with ONNX, would you mind be able to help me with a few questions?

• Does ONNX work with full sklearn pipelines that are trained on dataframes and output dataframes?
  • Notably the input given to pipelines will now be pandas and we use pipeline.set_output("pandas") which enables pandas for intermediate and final output. The ONNX tutorial seems to suggest numerical dtypes as input, however any encoders (e.g. OrdinalEncoder) would expect a "str" or "categorical" dtype as they were trained on. Is this still possible or does ONNX have some strict restrictions on the setup of how models must be trained?
• I see VotingClassifier and VotingRegressor is supported, that's how the final ensemble is returned to the user (new and old). However we have to use a hack to make this work, namely setting the private estimators_ attribute Add prefit to VotingClassifier scikit-learn/scikit-learn#7382 and some other necessities. Would this cause an issue with ONNX?
• I see not all estimators/transformers are supported. Does this selection also extend to estimator parameters? Is there any automatic way within ONNX to automatically know if a model is supported?

[eddiebergman]

Update on ONNX:

I spent some time with sklearn-onnx to convert some of the pure-sklearn pipelines that auto-sklearn will output (no autosklearn code in there). Most of these pipeline will essentially be a VotingClassifier or VotingRegressor with several Pipeline's inside them. This is how we can handle ensembles and CV fold models.

However this doesn't seem to be supported by sklearn-onnx as documented in onnx/sklearn-onnx#1016. As the VotingX with multiple datatypes is our main component for combining estimators, I am not sure how we can make ONNX compatible pipelines.

If anyone has more information on something I could be missing, please let me know either in the above mentioned issue or here!

[AmirAlavi]

@eddiebergman something I'd like to see from this refactor is the ability to specify an optional max_success_trials parameter (maximum number of successful candidate algorithms tried) to the constructors for AutoSklearn estimators for early stopping. I don't think this is currently possible from the get_trials_callback parameter, since the result: RunValue argument of that callback only lists num_run in the additional_info, which I think is the counter of total candidates tried, whether they succeeded or not.

Or rather than adding that special param to the constructor, giving the aforementioned callback hook access to the runhistory?

[AmirAlavi]

@eddiebergman something else we noticed was the extremely long refit time for AutoSklearn models. For example, if we set the time_left_for_this_task to be just 1hr, and so the per_run_time_limit is just 6minutes, we noticed that the refit could somehow take hours in some cases.

We didn't investigate it thoroughly, but I think I recall discovering that the HistGradientBoosting models would take a very long time to fit. I know ensembling adds another layer to this, but I think we had observed this even for ensemble_size=1.

I'm wondering if you had also noticed any performance issues with that, and if the new updates address it? (perhaps the upgrade to newer sklearn takes care of it)

[eddiebergman]

Note: If you appreciated this longer form insight into the progress and new design, please give an emoji response, otherwise we can just stick to short form responses :) Please feel free to ask about other topics and I can write up a response of the new underlying API and how things will work.

Hi @AmirAlavi,

---

Regarding point 1. Callbacks are how the new autosklearn is mostly built, i.e. that's how most of the control flow now works (with some additional extra features ;) ). However to keep things simple for the end user, the AutoSklearnEstimator is following the sklearn api, where most of the functionality happens in fit(...). There is no exposure to these callbacks and customization that doesn't fit naively into init args are mostly avoided. Worth restating, the estimator API will remain mostly identical, along with it's feature set

To this end, we have a very simple argument n_trials= to the Estimator classes which stops after a certain number of trials are done (not necessarily successfully).

However this fit(...) call creates an AutoSklearn object which is more traditional in it's setup and usage. If you'd like full control, then you'd have to create one of these AutoSklearn objects, following the Estimator as a guide for how to do so.

Tasks and Plugins

To give you some idea of how this works and how we tried to increase the surface area of interaction, we now follow a more "server" like control flow, i.e. event-driven. The bit I'll share for today is namely the notion of a Scheduler, Task and a TaskPlugin.

• Scheduler: A event-loop based controller using async/await to prevent the pit-falls of while True event loop. You do not need to know Python's async to use it, but the functionality is there for advanced use cases (i.e. as a server). Can use anything that looks like a python Executor to submit compute and return a Future.
  • ProcessPoolExecutor (python's native local processes)
  • Dask
  • Loky
• Task: A unit of compute to run on some Scheduler with associated events. Can be extended for more context-dependent events.
  • Evaluating Trials
  • Ensembling
  • Pruning non-performant trials
• TaskPlugin: Extends/modifies the runtime of a Task (parameters/setup/etc...)
  • Pynisher to limit resource usage
  • ThreadpoolCtl limit oversubscription to threads in sklearn.
  • CallLimiter control if a task gets submitted by call count, concurrent count or conflicting tasks (e.g. don't prune models from disk while ensembling)

With this context, here is 2 examples of how you could accomplish your goal of stopping after a number of successful trials, the first focusing on the notion of a Task, while the second on TaskPlugins, let me know if you have feedback!

askl = AutoSklearn(...)
task = askl.trial_task

# Option 1

@task.on_success(when=lambda: task.on_success.count >= 10)  # optional `when=`
def stop_autosklearn(report: Trial.Report) -> None:
    askl.stop()


# Option 2
task.on_success(
    stop_autosklearn,
    when=lambda: task.on_success.count >= 10,
)


# This `.count` property doesn't exist but I'll add it, thanks for the illuminating question!
# has other events such as `on_{cancelled/failed/crashed/memout/timeout/...}

askl.run(...)

There are also "plugins" which modify behaviors of Tasks, the use-case agnostic unit of compute. This example goes a bit deeper, trying to show both the events system, using the Emitter and also the usage plugin

from typing import ParamSpec, TypeVar
from datetime import datetime
from ... import TaskPlugin, Task, Emitter, Event

P = ParamSpec("P")
R = TypeVar("R")

class MyPlugin(Emitter, TaskPlugin):
    """A TaskPlugin interacts with a submission of a Task before
    it hits the Scheduler. This can be used to modify the function or its
    arguments, as well as getting the highest priority in terms of responding
    to events emitted from the task.
    """
    
    name = "my-plugin"
    """Name of the plugin for logging purposes"""
    
    SUCCESS_LIMIT_REACHED: Event[str] = Event("an-event-name")
    """Will emit the current time when the limit is reached"""
    
    def __init__(self, n: int) -> None:
    	super().__init__()
        self.n = n
        self.count = 0
        
        # These are how the callbacks are created
        # by using the `Event` defined above, it enables type-safety
        # for anyone using the `.on_reached` attribute to register
        # callbacks
        self.on_reached = self.subscriber(self.SUCCESS_LIMIT_REACHED)
                
	    # The typing is like a `Callable` if familiar (not required)
        self.task: Task[..., Trial.Report] | None = None
    
    def pre_submit(
        self,
        fn: Callable[P, R],
        *args: P.args,
        **kwargs: P.kwargs,
    ) -> tuple[Callable[P, R], tuple, dict] | None:
        # TaskPlugin can modify the function and args before
        # it's submitted to the Scheduler
        if self.count >= self.n:
            return None  # Scheduler will not submit anything
            
        return fn, args, kwargs  # Submit as normal


    def attach_task(self, task: Task[..., Trial.Report]) -> None:
        self.task = task
        task.on_returned(self._check_to_stop) # This `.on_returned` created the same way as above
        
    def _check_to_stop(self, report: Trial.Report):
        assert self.task is not None

        if report.status is Trial.Status.Success:
        	# Or just if report.status == "success"          
			self.count += 1
			if self.count >= self.n:
			     time_stamp = datetime.now().isoformat()
			     self.on_reached.emit(time_stamp)

myplugin = MyPlugin()


# Only now are we really in the context of AutoSklearn
askl = AutoSklearn(..., plugins=[myplugin])

@myplugin.on_reached
def stop_askl(timestamp: str) -> None:
    askl.stop()
    print(f"askl_stopped at {timestamp} after {myplugin.n} successes")

These plugins are how a lot of the additional, optional, functionality is given to autosklearn tasks, such as memory limiting and call limiting.

Sorry for the long response to what's a rather simple question, I wanted to share a little bit of how the internals work so that you can give any feedback or raise any other questions. I unfortunately can still not share any source code.

---

Regarding question 2. I don't think we ever noticed this so thanks for bringing it to our attention. Could you raise a separate issue so that we have a note to investigate this? Unfortunately it's a little more complicated than just an ensemble. AutoSklearn works by ensembling all fold models from cv-folds and then also creates a weighted ensemble on top this. For example, with 5 fold CV and an ensemble which contains 3 models, this will be the weighted probabilities of 3 ensembles of 5 models, i.e. all 15 train sklearn models will be used. This still wouldn't fully explain the fitting times but just for some extra info.

[AmirAlavi]

@eddiebergman Thank you for the longform response! It was very useful and informative.

I like the new API from what it sounds like (though I don't think i have my head fully wrapped around it, since i'm not familiar with python's concurrent package). To confirm/summarize:

• If I really like the built-in, opinionated pipeline that AutoSklearn uses, and I really like the SMAC algorithm and greedy ensemble building, I'll still be able to create an AutoSklearnEstimator as before
• If I want to dig in more, I can follow the example of AutoSklearnEstimator (or probably some tutorial/user guide) to interact with AutoSklearn directly. So now, for full control, I won't have to create my own Estimator classes and copy a lot of boilerplate.

Will we be able to specify the bayesian optimization algorithm as well? for example, what if rather than SMAC (which I believe uses a regression tree as the posterior probability model, and also flips a coin every time to determine whether to listen to the posterior model or totally randomly explore), I'd like to try another algorithm? Will the new API have a "registry" of such algorithms already defined? or maybe define the API under which we can add our own?

[AmirAlavi]

@eddiebergman Regarding point two, sure I'll make an issue so there's a record of it, but not sure i'll get to a thorough investigation myself.

And thank you! I wasn't aware of that detail of using all models from the CVs in the ensembling, and that could certainly partially explain why even when setting ensemble size=1, the refit would still take long (since i was using 10-fold CV)! That's really good to know going forward.