Feat: add top-level open_virtual_datatree #838

maxrjones · 2025-11-14T17:36:49Z

This PR adds open_virtual_datatree for virtualizing higher-level hierarchical data structures.

Builds on #837

TODO:

Unit tests
Test with other backends (e.g., HDF)

Checklist:

Closes Support for groups #84
Tests added
Tests passing
Full type hint coverage
Changes are documented in docs/releases.rst
New functions/methods are listed in api.rst
New functionality has documentation

commit 113c712 Author: Ilan Gold <[email protected]> Date: Tue Nov 11 17:25:49 2025 +0100 Update virtualizarr/tests/test_parsers/test_hdf/test_hdf.py Co-authored-by: Chuck Daniels <[email protected]> commit 5693649 Author: Ilan Gold <[email protected]> Date: Tue Nov 11 17:25:10 2025 +0100 refactor: `get_deepest_group_or_array` type commit 849d567 Merge: 916c3e1 7a13261 Author: Ilan Gold <[email protected]> Date: Fri Nov 7 14:18:51 2025 +0100 Merge branch 'main' into ig/nested_h5_group commit 916c3e1 Merge: d8dc369 a2b65c1 Author: Ilan Gold <[email protected]> Date: Wed Sep 10 13:11:01 2025 +0200 Merge branch 'main' into ig/nested_h5_group commit d8dc369 Author: Ilan Gold <[email protected]> Date: Sun Sep 7 19:53:08 2025 +0200 fix: update release note commit a3a2615 Author: ilan-gold <[email protected]> Date: Sun Sep 7 16:45:22 2025 +0200 chore: clean up error handling for non-metadata group requests + tests commit fc0e999 Merge: 3036de8 1facb10 Author: ilan-gold <[email protected]> Date: Sun Sep 7 16:29:59 2025 +0200 Merge branch 'ig/nested_h5_group' of https://github.com/ilan-gold/VirtualiZarr into ig/nested_h5_group commit 3036de8 Author: ilan-gold <[email protected]> Date: Sun Sep 7 16:29:44 2025 +0200 fix: remove subgroup check commit 1facb10 Author: Ilan Gold <[email protected]> Date: Sun Sep 7 16:17:43 2025 +0200 fix: wrong place! commit 33e517e Merge: afcaaed c1db925 Author: Ilan Gold <[email protected]> Date: Sun Sep 7 16:17:21 2025 +0200 Merge branch 'main' into ig/nested_h5_group commit afcaaed Author: ilan-gold <[email protected]> Date: Sun Sep 7 16:15:26 2025 +0200 chore: consolidate tests commit 253d184 Author: ilan-gold <[email protected]> Date: Sun Sep 7 15:57:18 2025 +0200 fix: disallow nested xarray handling commit 57551b8 Author: ilan-gold <[email protected]> Date: Fri Aug 29 08:52:28 2025 +0200 chore: relnote commit 6fe1dbd Author: ilan-gold <[email protected]> Date: Fri Aug 29 08:45:57 2025 +0200 fix: typing commit 85f2930 Author: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Thu Aug 28 16:27:24 2025 +0000 [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci commit e92ba8c Author: ilan-gold <[email protected]> Date: Thu Aug 28 18:19:46 2025 +0200 fix: simplify node logic commit 323be33 Author: ilan-gold <[email protected]> Date: Thu Aug 28 18:02:04 2025 +0200 feat: nested groups

commit 128138b Author: Max Jones <[email protected]> Date: Thu Nov 13 18:39:24 2025 -0500 Apply suggestion from code review commit 9c0e803 Author: Max Jones <[email protected]> Date: Thu Nov 13 18:25:39 2025 -0500 Apply suggestions from code review Co-authored-by: Chuck Daniels <[email protected]> commit d1595b4 Author: Max Jones <[email protected]> Date: Thu Nov 13 18:25:10 2025 -0500 Update virtualizarr/manifests/store.py Co-authored-by: Chuck Daniels <[email protected]> commit 1573992 Author: Max Jones <[email protected]> Date: Thu Nov 13 18:17:32 2025 -0500 Add test for listing array in subgroup commit dcf80bb Author: Max Jones <[email protected]> Date: Thu Nov 13 15:25:10 2025 -0500 Fix typing commit 2448c5d Author: Max Jones <[email protected]> Date: Thu Nov 13 15:08:33 2025 -0500 Improve ManifestStore.list_dir for arrays and nested groups

codecov · 2025-11-14T17:38:47Z

Codecov Report

❌ Patch coverage is 92.85714% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.17%. Comparing base (c67dcc1) to head (9b91ad3).

Files with missing lines	Patch %	Lines
virtualizarr/manifests/group.py	91.66%	1 Missing ⚠️
virtualizarr/xarray.py	91.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #838      +/-   ##
==========================================
- Coverage   89.22%   89.17%   -0.06%     
==========================================
  Files          34       34              
  Lines        1987     2013      +26     
==========================================
+ Hits         1773     1795      +22     
- Misses        214      218       +4

Files with missing lines	Coverage Δ
virtualizarr/__init__.py	`75.00% <100.00%> (ø)`
virtualizarr/manifests/store.py	`89.20% <100.00%> (+0.23%)`	⬆️
virtualizarr/manifests/group.py	`93.10% <91.66%> (-0.38%)`	⬇️
virtualizarr/xarray.py	`85.96% <91.66%> (+0.52%)`	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

chuckwondo

@maxrjones, this looks great. Approved!

Feel free to ignore any or all of my suggested changes, which are only suggestions for some simplification, which are certainly arguable, and thus may not appear simpler to others. I leave them all to your discretion.

virtualizarr/manifests/group.py

virtualizarr/tests/test_xarray.py

virtualizarr/xarray.py

Co-authored-by: Chuck Daniels <[email protected]>

TomNicholas · 2026-01-09T20:45:24Z

virtualizarr/xarray.py

+    drop_variables
+        Variables in the data source to drop before returning.
+    loadable_variables
+        Variables in the data source to load as Dask/NumPy arrays instead of as virtual arrays.


This seems under-specified. Can I load/drop specific variables from one group? If so how? Does it understand a path-like string?

I added more details and examples in 9b91ad3 (#838)

I should ask, do you approve of the behavior that's now described regarding loadable/droppable variables @TomNicholas? we should avoid breaking changes obviously, so this is an important component to get right

I'm not a big fan of the current map-over-groups behaviour, because IMO it only makes sense for overviews-style-datatrees. Accepting path-like strings is more effort but should work. Unfortunately this whole API design problem is just a little bit messy 😞

I think maybe this is an argument for implementing something like vdt["var"].load()...

Ok, so what if we only implement the default behavior in this PR and punt on adding loadable/drop_variables? It's the end of the project increment for me, so I'd like to get something in before I get pulled onto other work and splitting PRs is good practice anyways.

We'd need to decide on the default behavior between:

Same as xarray (load coordinates)

Don't load anything

I'm okay with that, because then generalizing later wouldn't be a breaking change.

We'd need to decide on the default behavior

Maybe you could keep both these options by keeping the kwargs, but raising a NotImplementedError if the user attempts to pass an a non-empty list.

maxrjones added 6 commits November 11, 2025 12:50

Return None for Zarr V2/consolidated metadata requests

439accd

Merge branch 'main' into open-virtual-datatree

da2f601

Feat: add top-level open_virtual_datatree

437a5d8

Remove unused function

5c35e1b

maxrjones temporarily deployed to test-release November 14, 2025 17:37 — with GitHub Actions Inactive

maxrjones self-assigned this Nov 14, 2025

Fix cross-reference

e563aa0

maxrjones temporarily deployed to test-release November 14, 2025 17:51 — with GitHub Actions Inactive

Fixup docstrings

0f45437

maxrjones temporarily deployed to test-release November 14, 2025 17:54 — with GitHub Actions Inactive

Merge branch 'main' into open-virtual-datatree

0da38c8

maxrjones temporarily deployed to test-release November 14, 2025 18:45 — with GitHub Actions Inactive

Merge branch 'main' into open-virtual-datatree

e1b5009

maxrjones temporarily deployed to test-release January 7, 2026 18:41 — with GitHub Actions Inactive

Add a couple tests

a03f0fc

maxrjones temporarily deployed to test-release January 7, 2026 20:31 — with GitHub Actions Inactive

maxrjones marked this pull request as ready for review January 7, 2026 20:39

maxrjones added 2 commits January 7, 2026 16:23

Add test for loadable vars

62b0e99

Add to docs

7fc4669

maxrjones temporarily deployed to test-release January 7, 2026 21:24 — with GitHub Actions Inactive

maxrjones requested a review from a team January 7, 2026 21:43

chuckwondo approved these changes Jan 9, 2026

View reviewed changes

Apply suggestions from code review

3132005

Co-authored-by: Chuck Daniels <[email protected]>

maxrjones temporarily deployed to test-release January 9, 2026 19:47 — with GitHub Actions Inactive

Update virtualizarr/xarray.py

59e61e7

Co-authored-by: Chuck Daniels <[email protected]>

maxrjones temporarily deployed to test-release January 9, 2026 19:48 — with GitHub Actions Inactive

TomNicholas reviewed Jan 9, 2026

View reviewed changes

TomNicholas mentioned this pull request Jan 10, 2026

Kerchunk / VirtualiZarr way to open radar files openradar/xradar#187

Open

maxrjones added 2 commits January 10, 2026 11:37

Add test for drop_vars

0e10381

Add more details and examples

9b91ad3

maxrjones deployed to test-release January 10, 2026 16:55 — with GitHub Actions View deployment

maxrjones mentioned this pull request Jan 12, 2026

ODD PI 26.1 Objective 3: 🤖 Support virtualization of additional data products NASA-IMPACT/veda-odd#246

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feat: add top-level open_virtual_datatree #838

Feat: add top-level open_virtual_datatree #838

Uh oh!

maxrjones commented Nov 14, 2025 •

edited

Loading

Uh oh!

codecov bot commented Nov 14, 2025 •

edited

Loading

Uh oh!

chuckwondo left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TomNicholas Jan 9, 2026

Uh oh!

maxrjones Jan 10, 2026

Uh oh!

maxrjones Jan 12, 2026

Uh oh!

TomNicholas Jan 12, 2026

Uh oh!

maxrjones Jan 12, 2026

Uh oh!

TomNicholas Jan 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Feat: add top-level open_virtual_datatree #838

Are you sure you want to change the base?

Feat: add top-level open_virtual_datatree #838

Uh oh!

Conversation

maxrjones commented Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

chuckwondo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TomNicholas Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

maxrjones Jan 10, 2026

Choose a reason for hiding this comment

Uh oh!

maxrjones Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

TomNicholas Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

maxrjones Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

TomNicholas Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

maxrjones commented Nov 14, 2025 •

edited

Loading

codecov bot commented Nov 14, 2025 •

edited

Loading