AutoML Fails due to FastTree GetProcessInformation UWP

[Jenscaasen]

Issue

• What did you do?
  I used AutoML for a Forecast problem in an UWP App. newest version of ML.NET. Letting the training run for more than 2 Minutes. Even when trying to not include trainers named "Tree" in the "RegressionExperimentSettings" it still arises. No idea why it would try to get ProcessInformation. Can you please remove that "GetProcessInfos" from FastTree?

• What happened?
  This Exception arises:

 at System.Diagnostics.NtProcessInfoHelper.GetProcessInfos(Predicate`1 processIdFilter)
  at System.Diagnostics.ProcessManager.GetProcessInfo(Int32 processId, String machineName)
  at System.Diagnostics.Process.EnsureState(State state)
  at Microsoft.ML.Trainers.FastTree.FastTreeTrainerBase`3.PrintMemoryStats(IChannel ch)
  at Microsoft.ML.Trainers.FastTree.FastTreeTrainerBase`3.TrainCore(IChannel ch)
  at Microsoft.ML.Trainers.FastTree.FastForestRegressionTrainer.TrainModelCore(TrainContext context)
  at Microsoft.ML.Trainers.TrainerEstimatorBase`2.TrainTransformer(IDataView trainSet, IDataView validationSet, IPredictor initPredictor)
  at Microsoft.ML.AutoML.SmacSweeper.FitModel(IEnumerable`1 previousRuns)
  at Microsoft.ML.AutoML.SmacSweeper.ProposeSweeps(Int32 maxSweeps, IEnumerable`1 previousRuns)
  at Microsoft.ML.AutoML.PipelineSuggester.SampleHyperparameters(MLContext context, SuggestedTrainer trainer, IEnumerable`1 history, Boolean isMaximizingMetric)
  at Microsoft.ML.AutoML.PipelineSuggester.GetNextInferredPipeline(MLContext context, IEnumerable`1 history, DatasetColumnInfo[] columns, TaskKind task, Boolean isMaximizingMetric, CacheBeforeTrainer cacheBeforeTrainer, IEnumerable`1 trainerWhitelist)
  at Microsoft.ML.AutoML.Experiment`2.Execute()
  at Microsoft.ML.AutoML.ExperimentBase`2.Execute(ColumnInformation columnInfo, DatasetColumnInfo[] columns, IEstimator`1 preFeaturizer, IProgress`1 progressHandler, IRunner`1 runner)
  at Microsoft.ML.AutoML.ExperimentBase`2.ExecuteCrossValSummary(IDataView[] trainDatasets, ColumnInformation columnInfo, IDataView[] validationDatasets, IEstimator`1 preFeaturizer, IProgress`1 progressHandler)
  at Microsoft.ML.AutoML.ExperimentBase`2.Execute(IDataView trainData, ColumnInformation columnInformation, IEstimator`1 preFeaturizer, IProgress`1 progressHandler)
  at Microsoft.ML.AutoML.ExperimentBase`2.Execute(IDataView trainData, String labelColumnName, String samplingKeyColumn, IEstimator`1 preFeaturizer, IProgress`1 progressHandler)
  at MLAtHome.Helpers.AutoMLHelper.<BuildTrainEvaluateAndSaveModel>d__8.MoveNext() in C:\Users\caasen\source\repos\MLAtHome\MLAtHome\Helpers\AutoMLHelper.cs:line 70
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[...]

Source code / logs

//Excluding "FastTree" here in the settings
     RegressionExperiment experiment=  mlContext.Auto().CreateRegressionExperiment(experimentSettings);
            
            ExperimentResult<RegressionMetrics> experimentResult = experiment.Execute(myview, Statics.CurrentModel.LabelName, progressHandler: progressHandler);

[justinormont]

This is an error in the SMAC sweeper, which uses a tree in the background for predicting which hyperparameters sets to try next.

Hence even when the trees are disabled in the RegressionExperimentSettings, there are trees used in the AutoML experiment for the hyperparameter optimization.

I'm not sure exactly what in FastTree is throwing, but it's within PrintMemoryStats():

machinelearning/src/Microsoft.ML.FastTree/FastTree.cs

Lines 437 to 462 in d0b3f86

	private protected virtual void PrintMemoryStats(IChannel ch)
	{
	Contracts.AssertValue(ch);
	ch.Trace("Training {0}", GetDatasetStatistics(TrainSet));
	if (ValidSet != null)
	ch.Trace("Validation {0}", GetDatasetStatistics(ValidSet));
	if (TestSets != null)
	{
	for (int i = 0; i < TestSets.Length; ++i)
	ch.Trace("ComputeTests[{1}] {0}",
	GetDatasetStatistics(TestSets[i]), i);
	}
	if (AllowGC)
	ch.Trace("GC Total Memory = {0} MB", GC.GetTotalMemory(true) / 1024 / 1024);
	Process currentProcess = Process.GetCurrentProcess();
	ch.Trace("Working Set = {0} MB", currentProcess.WorkingSet64 / 1024 / 1024);
	ch.Trace("Virtual Memory = {0} MB",
	currentProcess.VirtualMemorySize64 / 1024 / 1024);
	ch.Trace("Private Memory = {0} MB",
	currentProcess.PrivateMemorySize64 / 1024 / 1024);
	ch.Trace("Peak Working Set = {0} MB", currentProcess.PeakWorkingSet64 / 1024 / 1024);
	ch.Trace("Peak Virtual Memory = {0} MB",
	currentProcess.PeakVirtualMemorySize64 / 1024 / 1024);
	}

I'm guessing this is a duplicate of: #2444 (FastTree does not work in UWP)

@eerhardt, @codemzs: Do you see any quick solutions? Perhaps the try/catch recommended by danmosemsft.

[justinormont]

@Jenscaasen: For a short term workaround, you can stop AutoML at 68 iterations (I may be off by a couple). For iteration 9 to ~67, AutoML is warming up the hyperparameter space using random exploration. Beginning at iteration ~68, SMAC is used for hyperparameter optimization.

For this custom early stopping after iteration 67, you would add a RegressionExperimentProgressHandler() to monitor the current iteration number and stop just before SMAC begins.

[eerhardt]

Unfortunately the FastTree issue isn't the only one you will encounter. I wouldn't recommend using ML.NET in UWP at the moment, because there are other UWP bugs that are going to block you.

Have you tried building your app in Release mode?

See the discussion here #2252 (comment).

Instead, I would recommend using a WPF app on .NET Core 3.0. You can still submit these apps to the App Store.

[Jenscaasen]

Thank you for your replys!
As for try catching and stopping the search prematurely: that would not result in a let's say 1 hour trained model, right? In the case i am working on the user can choose the training time, limiting him to something between 1 second and 1 minute does not sound good (but I still appreciate the suggestions).

Is there an estimation when UWP will be supported by ML.net? I like the platform with its design principles and the easy-to-market option by using the Microsoft store.

As i see in the code dump: the get process Infos only gets executed with "AllowGC". Can i make that be false?

Also I will look into whether WPF is a good replacement for UWP

[eerhardt]

Closing as a duplicate of #2444.