Version 1.0 milestones

[jjhelmus]

This issue will be used to track work towards the 1.0 milestone where pyfive can read all HDF5 files which are supported by hdf5 using a similar API. This issue will be updates as new issues are opened.

HDF5 features not supported by pyfive

• compression, implemented in Add support for gzip compression and shuffle filter #9
• shuffling, implemented in Add support for gzip compression and shuffle filter #9
• dimension scale, Dimension scales #14
• complex attributes, implemented in Support for complex attribute datatypes #26
• fletcher32 checksum, implemented in Support for reading and verifting fletcher32 checksums #28
• unlimited dimensions, tests added in TST: unit tests for resizable datasets #33
• multiple unlimited dimensions, tests added in TST: unit tests for resizable datasets #33
• version 1.10 files, implemented in Support v3 superblocks, minor maintenance #39
• scale offset filter
• various datatypes

h5py API features not implemented in pyfive

• compression Dataset attribute, implemented in Add support for gzip compression and shuffle filter #9
• compression_opts Dataset attribute, implemented in Add support for gzip compression and shuffle filter #9
• shuffle Dataset attribute, implemented in Add support for gzip compression and shuffle filter #9
• chunks Dataset attribute, implemented in Add chunks attribute to pyfive.Dataset class #11
• fletcher32 Dataset attribute, implemented in Support for reading and verifting fletcher32 checksums #28
• fillvalue Dataset attribute, implemented in support for reading dataset fillvalues #29
• visit methods, implemented in Add visit and visititems methods to Group class #31
• visititems methods, implemented in Add visit and visititems methods to Group class #31
• astype Dataset method, implemented in ENH: Dataset.astype method #36
• read_direct Dataset method, implemented in ENH: Dataset.read_direct method #37
• scaleoffset Dataset attribute.

[bnlawrence]

Given that the discussion on #89 suggests that it will be quite a while before we implement that, and given that with the eventual completion of #114 close, my thought is that we are ready for V1.0 now, and that the issues around support for scale-offset (and anything else) should be pushed into "post V1.0". I think the most active folks with an opinion on this would be @bmaranville, @kmuehlbauer, @davidhassell and @valeriupredoi. What say you all? Remove scaleoffset from here, and aim for a V1.0 with #95 ?

[kmuehlbauer]

@bnlawrence I tend to agree. It might not be the best indicator, but almost all tests for xarray->h5netcdf->pyfive are passing for latest pyfive main branch.

Nevertheless we should come up with a list of missing bits and pieces and reference it in the docs.

[valeriupredoi]

one thing I reckon we can still work on before v1.0 is test coverage; @kmuehlbauer would you reckon pre-commit/ruff support would be something we want before v1.0 also? 🍻

[kmuehlbauer]

@valeriupredoi Yes, if we transform the codebase using the latest linting and formatting, this would indeed be wanted for a version 1.

I found another glitch with REFERENCE type over at h5netcdf, which would be great to sort out before V1.0. I'll open an issue in the next day describing the problem. Maybe there is a quick solution to this.

[valeriupredoi]

@valeriupredoi Yes, if we transform the codebase using the latest linting and formatting, this would indeed be wanted for a version 1.

I found another glitch with REFERENCE type over at h5netcdf, which would be great to sort out before V1.0. I'll open an issue in the next day describing the problem. Maybe there is a quick solution to this.

eagle eye 🦅
will start the motions for modern linting with pre-commit+ruff then 🍺

[bnlawrence]

I've moved the ruff support to V2.0 (for now, and by which I mean it can go into 1.X). If we get it done now, great, otherwise, I'd rather we released V1.0 sooner rather than later, as I think the paper should be associated with V1.0. If folks think this is a bad decision, please reopen this issue!

Given we then have scale-offset filters, which we all agree is hard, and is covered by #89 and "various datatype" left, and the latter is somewhat vague. I think we can close this issue since we're ready for V1.0.