[Question] Parallelizing Across Multiple 'learn_one' calls

[gcverissimo]

Hi, and thanks for the great library!

I’m wondering whether it’s possible to call model.learn_one() in parallel from multiple workers or threads in order to speed up the processing of incoming data. Specifically, I’d like to know:

1. Is it safe or supported to call model.learn_one() concurrently on the same model instance?
2. If not, is there a recommended approach for handling data updates in parallel? (e.g., merging models)
3. My target components are:

• preprocessing.RobustScaler
• Regression models such as HoeffdingTreeRegressor, etc.

My use case involves multiple producers generating data samples simultaneously, and I’d like each to update the model as those samples arrive.

[kulbachcedric]

Hi @gcverissimo,
thanks for your kind words!
These a really interesting questions.
In my opinion it it makes the task of measuring the performance pretty difficult as additionally to the question of the model performance over the evolving data stream you are opening the question, when would be the best point to merge the models within the data stream.
However, merging two model states is actually not supported as it highly depends on the chosen model (but would be very interesting for some models in my opinion).

At the beginning of river I remember some experiments of running ensembles in parallel, at least they had not the performance benefits that I expected.

It's maybe not the answer you expected, but does this help?