Wrong order of golden features creation

[pplonski]

When running training:

import numpy as np
import pandas as pd
from supervised import AutoML

df = pd.read_csv("~/Downloads/Flight_Datasets/Data_Train.csv")
X = df[df.columns[:-1]]
y = df[df.columns[-1]]

automl = AutoML(mode="Compete")
automl.fit(X, y

I got:

AutoML directory: AutoML_18
The task is regression with evaluation metric rmse
AutoML will use algorithms: ['Linear', 'Decision Tree', 'Random Forest', 'Extra Trees', 'LightGBM', 'Xgboost', 'CatBoost', 'Neural Network', 'Nearest Neighbors']
AutoML will stack models
AutoML will ensemble availabe models
AutoML steps: ['simple_algorithms', 'default_algorithms', 'not_so_random', 'golden_features', 'insert_random_feature', 'features_selection', 'hill_climbing_1', 'hill_climbing_2', 'ensemble', 'stack', 'ensemble_stacked']
* Step simple_algorithms will try to check up to 3 models
1_DecisionTree rmse 3020.469038 trained in 3.2 seconds
2_DecisionTree rmse 2604.862023 trained in 3.03 seconds
3_Linear rmse 2686.658748 trained in 4.16 seconds
* Step default_algorithms will try to check up to 7 models
4_Default_RandomForest rmse 2480.499445 trained in 16.28 seconds
5_Default_ExtraTrees rmse 2652.152774 trained in 13.31 seconds
6_Default_Xgboost rmse 1487.010901 trained in 26.76 seconds
7_Default_LightGBM rmse 1644.944661 trained in 14.49 seconds
8_Default_CatBoost rmse 1517.840591 trained in 80.6 seconds
9_Default_NeuralNetwork rmse 2234.376146 trained in 201.85 seconds
10_Default_NearestNeighbors rmse 3071.047241 trained in 6.48 seconds
* Step not_so_random will try to check up to 58 models
11_Xgboost rmse 1856.97224 trained in 207.25 seconds
12_Xgboost rmse 1709.244007 trained in 52.0 seconds
13_Xgboost rmse 1663.65119 trained in 25.43 seconds
14_Xgboost rmse 2023.345354 trained in 35.66 seconds
15_Xgboost rmse 1889.214983 trained in 26.1 seconds
16_Xgboost rmse 2355.023108 trained in 55.27 seconds
17_Xgboost rmse 2557.776798 trained in 40.92 seconds
18_Xgboost rmse 1945.344602 trained in 38.25 seconds
19_Xgboost rmse 1978.908622 trained in 71.05 seconds
20_LightGBM rmse 1768.722936 trained in 30.38 seconds
21_LightGBM rmse 1726.683587 trained in 29.91 seconds
22_LightGBM rmse 1650.713309 trained in 16.75 seconds
23_LightGBM rmse 1612.480231 trained in 34.68 seconds
24_LightGBM rmse 1690.154324 trained in 19.85 seconds
25_LightGBM rmse 1662.705707 trained in 38.87 seconds
26_LightGBM rmse 1976.329237 trained in 20.56 seconds
27_LightGBM rmse 1645.461174 trained in 23.17 seconds
28_LightGBM rmse 1963.690856 trained in 22.23 seconds
29_CatBoost rmse 1720.143318 trained in 46.92 seconds
30_CatBoost rmse 1531.53048 trained in 46.96 seconds
31_CatBoost rmse 1553.254717 trained in 45.58 seconds
32_CatBoost rmse 1652.194219 trained in 62.21 seconds
33_CatBoost rmse 1555.6532 trained in 38.76 seconds
34_CatBoost rmse 1686.310668 trained in 33.34 seconds
* Step golden_features will try to check up to 3 models
8_Default_CatBoost_GoldenFeatures rmse 1517.840591 trained in 105.71 seconds
23_LightGBM_GoldenFeatures rmse 1612.480231 trained in 45.69 seconds
6_Default_Xgboost_GoldenFeatures rmse 1487.010901 trained in 80.44 seconds
* Step insert_random_feature will try to check up to 1 model
6_Default_Xgboost_GoldenFeatures_RandomFeature rmse 1554.147216 trained in 67.32 seconds
Drop features ['Total_Stops_2 stops', 'Source_Kolkata', 'Airline_Jet Airways Business', 'Source_Delhi', 'Destination_Delhi', 'random_feature', 'Airline_Multiple carriers Premium economy', 'Total_Stops_3 stops', 'Additional_Info_No check-in baggage included', 'Additional_Info_1 Long layover', 'Airline_Vistara Premium economy', 'Additional_Info_Business class', 'Additional_Info_2 Long layover', 'Additional_Info_1 Short layover', 'Additional_Info_No Info', 'Total_Stops_4 stops', 'Source_Banglore', 'Source_Chennai', 'Additional_Info_Red-eye flight', 'Destination_Banglore', 'Destination_Cochin', 'Destination_Hyderabad', 'Destination_Kolkata', 'Airline_Trujet', 'Additional_Info_Change airports']
* Step features_selection will try to check up to 6 models
8_Default_CatBoost_SelectedFeatures rmse 1517.840591 trained in 84.99 seconds
5_Default_ExtraTrees_SelectedFeatures rmse 2690.949907 trained in 13.18 seconds
23_LightGBM_SelectedFeatures rmse 1601.555996 trained in 69.53 seconds
9_Default_NeuralNetwork_SelectedFeatures rmse 2327.928432 trained in 271.67 seconds
4_Default_RandomForest_SelectedFeatures rmse 2507.381756 trained in 13.45 seconds
6_Default_Xgboost_SelectedFeatures rmse 1594.607861 trained in 30.4 seconds
* Step hill_climbing_1 will try to check up to 27 models
35_CatBoost rmse 1503.743702 trained in 78.91 seconds
36_CatBoost rmse 1551.392214 trained in 80.52 seconds
37_CatBoost_GoldenFeatures rmse 1503.743702 trained in 87.42 seconds
38_CatBoost_GoldenFeatures rmse 1551.392214 trained in 95.08 seconds
Skip hill_climbing_2 because of the time limit.
* Step ensemble will try to check up to 1 model
Ensemble rmse 1405.844508 trained in 8.05 seconds
* Step stack will try to check up to 30 models
6_Default_Xgboost_Stacked rmse 1447.586878 trained in 15.21 seconds
Add Golden Feature: 2_DecisionTree_prediction_ratio_1_DecisionTree_prediction
Add Golden Feature: 1_DecisionTree_prediction_ratio_2_DecisionTree_prediction
Add Golden Feature: 2_DecisionTree_prediction_diff_1_DecisionTree_prediction
Add Golden Feature: Ensemble_prediction_ratio_1_DecisionTree_prediction
Add Golden Feature: 1_DecisionTree_prediction_ratio_Ensemble_prediction
Created 5 Golden Features in 1.67 seconds.
6_Default_Xgboost_GoldenFeatures_Stacked rmse 1443.021739 trained in 12.69 seconds
37_CatBoost_GoldenFeatures_Stacked rmse 1480.39579 trained in 21.73 seconds
35_CatBoost_Stacked rmse 1467.320995 trained in 19.69 seconds
8_Default_CatBoost_Stacked rmse 1433.532618 trained in 21.21 seconds
8_Default_CatBoost_SelectedFeatures_Stacked rmse 1433.532618 trained in 20.45 seconds
8_Default_CatBoost_GoldenFeatures_Stacked rmse 1481.492691 trained in 19.62 seconds
30_CatBoost_Stacked rmse 1467.093997 trained in 12.07 seconds
36_CatBoost_Stacked rmse 1456.002184 trained in 22.06 seconds
38_CatBoost_GoldenFeatures_Stacked rmse 1495.237039 trained in 20.72 seconds
31_CatBoost_Stacked rmse 1453.445886 trained in 14.21 seconds
6_Default_Xgboost_GoldenFeatures_RandomFeature_Stacked rmse 1436.269761 trained in 24.35 seconds
33_CatBoost_Stacked rmse 1484.243631 trained in 10.08 seconds
6_Default_Xgboost_SelectedFeatures_Stacked rmse 1495.962609 trained in 59.58 seconds
23_LightGBM_SelectedFeatures_Stacked rmse 1492.117447 trained in 12.43 seconds
23_LightGBM_Stacked rmse 1499.029953 trained in 10.33 seconds
23_LightGBM_GoldenFeatures_Stacked rmse 1488.859107 trained in 13.16 seconds
7_Default_LightGBM_Stacked rmse 1482.642943 trained in 7.0 seconds
27_LightGBM_Stacked rmse 1507.619072 trained in 10.41 seconds
22_LightGBM_Stacked rmse 1536.804184 trained in 6.38 seconds
25_LightGBM_Stacked rmse 1554.44995 trained in 11.12 seconds
13_Xgboost_Stacked rmse 1465.883147 trained in 7.6 seconds
24_LightGBM_Stacked rmse 1486.977392 trained in 7.98 seconds
12_Xgboost_Stacked rmse 1513.513172 trained in 16.72 seconds
21_LightGBM_Stacked rmse 1525.964054 trained in 11.93 seconds
20_LightGBM_Stacked rmse 1575.585287 trained in 10.27 seconds
11_Xgboost_Stacked rmse 1510.667912 trained in 34.14 seconds
15_Xgboost_Stacked rmse 1547.047907 trained in 9.73 seconds
18_Xgboost_Stacked rmse 1533.803019 trained in 15.97 seconds
19_Xgboost_Stacked rmse 1516.763391 trained in 34.06 seconds
* Step ensemble_stacked will try to check up to 1 model
Ensemble_Stacked rmse 1385.755878 trained in 22.86 seconds
AutoML fit time: 3134.1 seconds

From which it looks that Golden Features are trained much later than they should.

[pplonski]

The problem was because there were no continuous features in the data. I've added an exception for such situations.