Make coupled-dataset path handling work for cloud and local stores#1363
Draft
jpdunc23 wants to merge 1 commit into
Draft
Make coupled-dataset path handling work for cloud and local stores#1363jpdunc23 wants to merge 1 commit into
jpdunc23 wants to merge 1 commit into
Conversation
Path-existence decisions in create_coupled_datasets.py, the xpartition output writer, get_stats.py, and merge_stats.py now resolve the filesystem from the URL scheme via a shared fsspec helper (fs_utils.py), so gs:// inputs/outputs and local paths share one code path. Adds an end-to-end test of write_datasets_and_stats on synthetic zarrs covering staged outputs, resume-on-existing, and debug mode.
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The coupled-dataset creation script assumed local filesystem paths: existence checks on output zarrs and stats directories, and the output writer's overwrite guard, used
os.path, which silently misbehaves ongs://URLs. This meant multi-terabyte production stores had to be mirrored to a local volume before coupled processing could run. Path-existence decisions now resolve the filesystem from the URL scheme via a shared fsspec helper, so cloud and local stores share one code path; the staged compute-or-read-existing behavior, debug mode, and subsample mode are unchanged on local paths.Changes:
scripts/data_process/fs_utils.py(new):path_exists,is_dir,makedirs,is_localhelpers resolving the filesystem from the URL scheme viafsspec.core.url_to_fsscripts/data_process/create_coupled_datasets.py: all path-existence checks (output stores, stats dirs, merged-stats dirs, ensemble category dirs) go throughpath_existsscripts/data_process/writer_utils.py:OutputWriterConfig.write's overwrite/initialize guard uses the scheme-awareis_dir, acceptinggs://output storesscripts/data_process/get_stats.py,scripts/data_process/merge_stats.py: hardcodedgs:prefix checks and local-onlyos.makedirsreplaced with the shared helpersNew end-to-end test running
CoupledDatasetsConfig.write_datasets_and_statson tiny synthetic zarrs: staged outputs and merged stats produced, re-runs resume on existing outputs without recomputing, debug mode writes nothingTests added
If dependencies changed, "deps only" image rebuilt and "latest_deps_only_image.txt" file updated