Repository navigation
Feature fraction gives different column selections when using a custom objective #6053
Description
Description
I ran into this because I got different column subsampling behaviour when using a built-in objective like MSE and when using a custom objective. I thought either: the reported seed in the booster.txt was wrong, or that columns are subsampled twice each iteration when using a custom objective, but only once when using a built-in objective (via perhaps an extra call to 'ResetByTree').
Reproducible example
import lightgbm as lgb
import numpy as np
import pandas as pd
np.random.seed(1)
X = np.random.randn(300, 300)
y = np.random.randn(300)
data = lgb.Dataset(X, label=y, init_score=np.zeros_like(y))
params = {'max_depth': 1, 'learning_rate': 0.001, 'verbose': -1, 'objective': 'l2', 'feature_fraction': 1.0, 'seed': 0}
model = lgb.train(params, data, num_boost_round=1000)
model_dict = model.dump_model(num_iteration=3)['tree_info']
def fobj(predictions, dataset):
return predictions - dataset.get_label(), np.ones_like(predictions)
model_custom = lgb.train(params, data, num_boost_round=1000, fobj=fobj)
model_custom_dict = model_custom.dump_model(num_iteration=3)['tree_info']
model_custom_dict == model_dictThis works when feature fraction is equal to 1.0 only. It breaks when feature_fraction = 0.5 for instance.
I can hack around the seeds to make it work:
import lightgbm as lgb
import numpy as np
import pandas as pd
np.random.seed(1)
X = np.random.randn(300, 300)
y = np.random.randn(300)
data = lgb.Dataset(X, label=y, init_score=np.zeros_like(y))
params = {'max_depth': 1, 'learning_rate': 0.001, 'verbose': -1, 'objective': 'l2', 'feature_fraction': 0.5, 'seed': 0, 'feature_fraction_seed': 974891790}
model = lgb.train(params, data, num_boost_round=1000)
model_dict = model.dump_model(num_iteration=3)['tree_info']
def fobj(predictions, dataset):
return predictions - dataset.get_label(), np.ones_like(predictions)
params['feature_fraction_seed'] = 2
model_custom = lgb.train(params, data, num_boost_round=1000, fobj=fobj)
model_custom_dict = model_custom.dump_model(num_iteration=3)['tree_info']
model_custom_dict == model_dictEnvironment info
LightGBM version or commit hash: 8d01d648942a427f6bb4962dc3f4330e005fa495
Command(s) you used to install LightGBM
pip install lightgbm
Activity
jameslamb commentedon Sep 6, 2023
Thanks for using LightGBM and for your thorough report.
What is commit
8d01d648942a427f6bb4962dc3f4330e005fa495? I don't see that commit in LightGBM's history: 8d01d64.I strongly suspect that that is from some version of LightGBM prior to v4.0.0. The parameter
fobjwas removed fromlgb.train()in #5052, so your example code yields the following error on latestmaster(8203306).I ran the code from your report today on that latest commit, with the following modifications:
fobjthroughparams(the new encouraged pattern, see [python-package] Where has the "fobj" parameter gone in lightgbm.cv() in v4.0.0? #6072 for example)"deterministic": Trueand"force_row_wise": True", to eliminate more possible sources of randomness"verbose": 1num_boost_roundtoo3code (click me)
I saw that the first splits were identical
But then they started to diverge starting with the second split.
I'm not sure exactly what's happening, but that narrows it down a bit.
I did not understand this comment and example. Do you just mean that you experimented with different values of seeds until you found a combination of 2 different ones led to the models being identical?
Shanxce commentedon Feb 26, 2025
If we set objective in python, in python code will call reset_parameter, which will call col_sampler.SetConfig, this function will call ResetByTree and randomly select feature columns. Thus, in custom objective, it will call one more random feature selection at first, result in all random selection is one time older than default pattern. @jameslamb @jrvmalik
jameslamb commentedon Apr 4, 2025
Good investigation @Shanxce ! I haven't looked closely at the links you shared, but that sounds like a plausible explanation to me.
Are you interested in submitting a PR to try to fix this? I'm not quite sure what the best approach is, but I do think that this behavior from LightGBM is surprising and that how you provide the loss funciton should not change which columns are chosen when sampling columns.