Initialize proto model with initial tensors before recovery #1373
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cloudnoize
requested review from
MasterSkepticista,
kminhta,
psfoley and
teoparvanov
as code owners
cloudnoize
force-pushed
the
elerer/1367_restart_on_save_model
branch
from
8b77859 to
645d29b
Compare
Collaborator
|
Thanks for fixing this @cloudnoize . I have successfully tested these changes and provided the run details on the Issue page itself. |
noopurintel
approved these changes
Feb 12, 2025
Collaborator
noopurintel
left a comment
noopurintel
left a comment
There was a problem hiding this comment.
Minor review comment provided. Changes look good otherwise. Tested successfully.
cloudnoize
force-pushed
the
elerer/1367_restart_on_save_model
branch
from
645d29b to
24089d5
Compare
teoparvanov
approved these changes
Feb 12, 2025
rahulga1
approved these changes
Feb 13, 2025
Signed-off-by: Lerer, Eran <[email protected]>
cloudnoize
force-pushed
the
elerer/1367_restart_on_save_model
branch
from
24089d5 to
75ebfc2
Compare
noopurintel
merged commit c679d64
into
securefederatedai:develop
27 of 28 checks passed
yuliasherman
pushed a commit
to yuliasherman/openfl
that referenced
this pull request
Mar 18, 2025
…deratedai#1373) Signed-off-by: Lerer, Eran <[email protected]> Co-authored-by: Noopur <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Address #1367
if the Aggregator crashes after collecting all data needed to finish round 0 but before it saved the model, on restart the model proto is not constructed by the recovery process, leading to a crash.
The PR initialize the proto model with the initial tensors that can be overridden by the recovery process if needed.
For safety, I moved other initializations before the recovery process as well.