Adaptive learning rate for linear regression

[jacowp357]

Is there a way to adapt the learning rate of the linear regression model over time?

Like a exponential decay of sorts?

[kulbachcedric]

Hi @jacowp357,

we (especially @lucasczz ) played around with adapting the learning rate through a custom optimizer for torch in deep-river e.g. when a concept drift occurs one could increase the learning rate.
Maybe you could exchange ideas and experiences.

Best
Cedric

[jacowp357]

There seems to be a "scheduler" and "InverseScaling" function, but I'm really struggling to get this to work. See my pipeline below:

regressor_model = compose.Pipeline(
    ('features', compose.TransformerUnion(
            ('ordinal_date', compose.FuncTransformer(get_ordinal_date)),
            ('15min', compose.FuncTransformer(get_15min_interval_distances)),
            ('hour', compose.FuncTransformer(get_hour_distances)),
            ('dayOfWeek', compose.FuncTransformer(get_day_of_week_distances)),
            ('dayOfMonth', compose.FuncTransformer(get_day_of_month_distances)),
            ('holiday', compose.FuncTransformer(get_holiday)), 
    )),
    ('scale', preprocessing.StandardScaler()),
    ('lin_reg', linear_model.LinearRegression(
        optimizer=optim.SGD(lr=optim.schedulers.InverseScaling(learning_rate=0.0005)),
        intercept_lr=0,
    ))
)

Or is there a way to manually set the learning rate during each step?

[lucasczz]

Your code looks fine to me. A similar setup ran without issues on my end:

from river.linear_model import LinearRegression
from river.optim import SGD, schedulers
from river.datasets import Elec2
from river.compose import Pipeline

optimizer= SGD(lr=schedulers.InverseScaling(learning_rate=1e-3))
model = Pipeline(LinearRegression(optimizer))

for x, y in Elec2().take(5000):
    pred = model.predict_one(x)
    model.learn_one(x, y)

It might not perform very well though, since the learning rate shrinks pretty rapidly with the InverseScaling scheduler.