Point CM4 coupled configs at the OM4 GCS production stores#1365
Draft
jpdunc23 wants to merge 3 commits into
Draft
Point CM4 coupled configs at the OM4 GCS production stores#1365jpdunc23 wants to merge 3 commits into
jpdunc23 wants to merge 3 commits into
Conversation
Path-existence decisions in create_coupled_datasets.py, the xpartition output writer, get_stats.py, and merge_stats.py now resolve the filesystem from the URL scheme via a shared fsspec helper (fs_utils.py), so gs:// inputs/outputs and local paths share one code path. Adds an end-to-end test of write_datasets_and_stats on synthetic zarrs covering staged outputs, resume-on-existing, and debug mode.
…led-production-configs
The CM4 piControl and 1pctCO2 coupled configs now read the 2026 OM4 1-degree production stores directly on GCS and write coupled outputs and stats to gs://vcm-ml-intermediate. The sea ice fractions come from the ocean store's native 5-daily fields: no separate sea ice input, no sea-ice window average, and use_atmosphere_sea_ice_fraction_fallback is false so a misconfiguration fails loudly instead of silently using the atmosphere's own field. Time chunk sizes align with the stores' shards (atmosphere 360, ocean 365). The surface-flux window timestamps are right-labeled at the ocean stores' interval-end instants, and the 1pctCO2 timestamps follow that store's model years (0001-0141). The 1pctCO2 config's precomputed sea ice mask points at the piControl coupled ocean output, which must exist before the 1pctCO2 run.
Member
Author
|
Stacked on #1364 (ocean-sourced sea ice), which is itself stacked on #1363 (cloud path handling); the base is set to |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The OM4 production ocean stores and the CM4 atmosphere stores now live on GCS, so the CM4 coupled configs should read them there directly instead of assuming a local
/climate-defaultmirror. This updatesCM4-piControl-200yr-coupled.yamlandCM4-1pctCO2-140yr-coupled.yamlin place to read the 2026 OM4 1-degree production stores on GCS and write coupled outputs and stats togs://vcm-ml-intermediate, with sea ice sourced from the ocean store's native 5-daily fields: no separate sea ice input, no sea-ice window average, anduse_atmosphere_sea_ice_fraction_fallback: falseso a misconfiguration fails loudly instead of silently using the atmosphere's own field.Time chunk sizes align with the stores' shards (atmosphere 360, ocean 365). The surface-flux window timestamps are right-labeled at the ocean stores' interval-end instants (the previous stores used midpoint-style labels), and the 1pctCO2 timestamps follow that store's model years (0001–0141). The 1pctCO2 config's precomputed sea ice mask points at the piControl coupled ocean output, which must exist before the 1pctCO2 run.
Changes:
configs/CM4-piControl-200yr-coupled.yaml,configs/CM4-1pctCO2-140yr-coupled.yaml: repointed at the GCS production stores with ocean-sourced sea ice; no compatibility with their previous/climate-defaultcontentsTests added
If dependencies changed, "deps only" image rebuilt and "latest_deps_only_image.txt" file updated