Warn instead of raise when user-provided data_files yields a subset#8215
Open
adityasingh2400 wants to merge 2 commits into
Open
Conversation
NonMatchingSplitsSizesError fires whenever the loaded split size differs from the expected size, including when the user has explicitly passed data_files for a known subset of the dataset. The user-side workaround is verification_mode='no_checks', which silences ALL checks rather than just the split-size one. Downgrade the split-size mismatch to a UserWarning when data_files was explicitly provided by the caller. Other mismatch paths (corrupted download, wrong size for full download) remain hard errors. Fixes huggingface#7867
Member
|
I think we can simply not raise or warn anything at all when the user provides the data_files. The number of examples is expected to be likely different |
Contributor
Author
|
Makes sense, done in 2162efc. verify_splits now returns early when the user passed data_files, so there is no raise and no warning at all in that case. The early return also covers the split name checks, since custom data_files can define a different set of splits. I removed the warning path and updated the tests accordingly. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #7867.
NonMatchingSplitsSizesError currently fires whenever the loaded split size differs from the expected size, including when the user explicitly passed data_files for a known subset of the dataset. The only user-side workaround is verification_mode='no_checks', which silences ALL checks rather than just the split-size one.
This PR downgrades the split-size mismatch to a UserWarning when data_files was explicitly provided by the caller. Other mismatch paths (corrupted download, wrong size for full download) remain hard errors.