0.22.0

What's Changed

• Remove skipna=False from the Xarray examples by @tomwhite in #706
• Spark executor by @songhan89 in #704
• Update executor docs with new experimental executors by @tomwhite in #705
• Icechunk example by @tomwhite in #708 and #709
• Remove new_axis from map_overlap kwargs by @tomwhite in #710
• Make mypy ignore vendored libraries by @tomwhite in #716
• Fix doc build by @tomwhite in #717
• Drop Python 3.10 by @tomwhite in #718
• Add scan by @dcherian in #531
• Change list of output arrays in BlockwiseSpec to a map by @tomwhite in #724
• Simplify resume implementation by taking out of runtime by @tomwhite in #725
• Mem utilization tests fail on zarr-python v3 by @tomwhite in #697
• Minor improvements to rechunk_new by @tomwhite in #732
• Use general_blockwise for scan, rather than map_direct by @tomwhite in #726
• Run array API tests on Zarr Python 2 and 3 by @tomwhite in #733
• Add MemoryVisualizationCallback by @tomwhite in #734
• Ruff by @tomwhite in #735
• Explicitly skip test_resume with tensorstore by @tomwhite in #736
• Allow use of obstore as a Zarr store by @tomwhite in #723
• Fix cardinality of output_dtypes in apply_gufunc by @tomwhite in #738
• Fix chunking where indexing an array returns an empty array by @tomwhite in #739
• Add a blocks property to Cubed arrays to allow block-level indexing by @tomwhite in #737
• Migrate to Icechunk 1.0 ForkSessions by @tomwhite in #742
• Use new rechunk implementation by default by @tomwhite in #741
• Remove zarray and zarray_maybe_lazy properties from Cubed Array by @tomwhite in #743
• Move index to indexing.py module by @tomwhite in #744
• Don't measure coverage for new executors by @tomwhite in #745
• Remove Spec from getting started doc by @tomwhite in #746
• Fix successors_unordered by @tomwhite in #747
• Exclude array API test that attempts in-place mutation by @tomwhite in #754
• Only run Zarr v2 CI tests on default local executors by @tomwhite in #759
• Update lithops aws image by @tomwhite in #755
• Fix indexing zero-sized dimension by @tomwhite in #761
• Run lithops CI on ubuntu-22.04 by @tomwhite in #762

New Contributors

• @songhan89 made their first contribution in #704
• @dcherian made their first contribution in #531

Full Changelog: 0.21.0...0.22.0

0.21.0

What's Changed

• Icechunk store by @tomwhite in #633
• Move retry logic to lithops by @tomwhite in #673
• Coverage improvements by @tomwhite in #674
• Add scalar support to where by @tomwhite in #679
• Add PendingDeprecationWarning for map_direct by @tomwhite in #682
• Add a rechunking example by @tomwhite in #681
• Remove Lithops retries tests since this is now tested upstream by @tomwhite in #683
• Using default_dtypes instead of hard-coding dtypes by @alxmrs in #666
• Array API status update for draft 2024 spec by @tomwhite in #684
• Refactor asyncio executors to share common code by @tomwhite in #685
• Add scalar support to binary element-wise ops by @tomwhite in #686
• Fix icechunk tests to opt-in to pickling of Session by @tomwhite in #688
• Move non-standardized functions to separate package by @tomwhite in #690
• Add Ray executor by @tomwhite in #687
• Allow nan functions to work with xarray by @tomwhite in #670
• Make callbacks context managers by @tomwhite in #691
• Adding pre-commit (for linting) to test dependencies by @alxmrs in #671
• Support float32 in random number generation by @tomwhite in #693
• Fix for Zarr 3.0.3 change to OrthogonalIndexer internals from Zarr #2784 by @tomwhite in #696
• Modal 1.0 changes by @tomwhite in #695
• Show nchunks on visualization by @tomwhite in #698
• Always set region for Modal executor by @tomwhite in #699
• Improve memory model by explicitly modelling buffer copies by @tomwhite in #701
• Multistage rechunking by @tomwhite in #700
• Remove duplicate code from new rechunk implementation by @tomwhite in #702
• Use extra buffer copies for cloud stores like S3 by @tomwhite in #703

Full Changelog: 0.20.0...0.21.0

0.20.0

What's Changed

• Include task result in callbacks by @tomwhite in #632
• Pin dask and distributed to < 2024.12.0 by @tomwhite in #639
• Fix Modal deprecation (show progress) and pip package by @tomwhite in #634
• Remove test 'extras' and run lithops tests separately by @tomwhite in #640
• Array API tests fixes by @tomwhite in #636
• New tagline by @tomwhite in #641
• Add tile by @tomwhite in #644
• Add searchsorted by @tomwhite in #647
• Declare Array API 2023.12 support by @tomwhite in #651
• Change chunk_shape back to chunks for Zarr v3 by @tomwhite in #656
• Document use of cubed.config by @tomwhite in #654
• Make async executors work in Jupyter notebooks by @tomwhite in #661
• Add Zarr and Xarray examples to docs by @tomwhite in #655
• Fix tooltips for SVGs generated by visualize() in Jupyter notebooks by @tomwhite in #663
• Add note to contributing guide for developers to have graphviz installed by @alxmrs in #665
• Incorrect dtypes in map_selection by @tomwhite in #669

Full Changelog: 0.19.0...0.20.0

0.19.0

What's Changed

• Support block_id for general_blockwise functions by @tomwhite in #593
• Run tests on Python 3.13 by @tomwhite in #503
• Fix dtypes for QR by @tomwhite in #594
• Change usages of map_direct in QR to general_blockwise by @tomwhite in #597
• Implement var and std using a numerically stable parallel algorithm by @tomwhite in #596
• Add map_selection primitive to replace map_direct by @tomwhite in #595
• Use unstack from array-api-compat by @tomwhite in #602
• Increase allowed_mem default to 2GB by @tomwhite in #599
• Implement structured arrays as Zarr group of arrays by @tomwhite in #603
• Implement int array indexing using map_selection by @tomwhite in #604
• Implement merge_chunks using map_selection by @tomwhite in #605
• Free memory earlier by changing function scope by @tomwhite in #606
• Fix groupby tests running on tensorstore by @tomwhite in #608
• Implement concat using general_blockwise by @tomwhite in #607
• Speed up test collection by @tomwhite in #609
• Update GH actions to macos-13 test runner and upload-artifact@v4 by @tomwhite in #611
• Implement repeat by @tomwhite in #610
• Map nested improvements by @tomwhite in #613
• SVD by @tomwhite in #598
• Unify chunks for concat, and check preconditions by @tomwhite in #616
• Defer to merge_chunks in special cases of rechunk by @tomwhite in #612
• Document map_selection core operation by @tomwhite in #617
• Change default max_total_num_input_blocks to 10 by @tomwhite in #615
• Update slides - November 2024 by @tomwhite in #620
• Change use_backups default to True only on cloud stores by @tomwhite in #619
• Various minor doc improvements by @tomwhite in #625
• Make threads the default executor by @tomwhite in #621
• Rewrite Cubed intro docs by @tomwhite in #622
• More doc improvements: simplify getting started page by @tomwhite in #629
• Add examples to docs by @tomwhite in #630

Full Changelog: 0.18.0...0.19.0

0.18.0

What's Changed

• Multiple outputs by @tomwhite in #419
• Support use of multiprocessing start methods other than "spawn" (e.g. "dragon") by @applio in #554
• Tensorstore compression fix by @tomwhite in #574
• Implement unstack using multiple outputs by @tomwhite in #575
• Handle more memray allocation types by @tomwhite in #578
• Drop Python 3.9 by @tomwhite in #576
• QR decomposition by @tomwhite in #577
• Expose the linalg namespace and include in status page by @tomwhite in #581
• Fix outer result dtype by @tomwhite in #582
• Add new 2023.12 elemwise functions by @tomwhite in #583
• Bring vecdot implementation in line with the one in array-api-compat by @tomwhite in #402
• Zarr v3 now requires at least Python 3.11 by @tomwhite in #585
• Debug failing straggler test by @tomwhite in #580
• Set object codec for object arrays by @tomwhite in #573
• Fix for flaky straggler test by @tomwhite in #587
• Use general_blockwise in case of chunk-aligned selections in index by @tomwhite in #586
• Remove deprecated code by @tomwhite in #588
• Documentation small fixes by @tomwhite in #591
• Add page on optimization to user guide by @tomwhite in #397
• Remove old reduction implementation by @tomwhite in #589
• Add array API inspection utilities by @tomwhite in #592

New Contributors

• @applio made their first contribution in #554

Full Changelog: 0.17.0...0.18.0

0.17.0

What's Changed

• Skip fewer array API tests by @tomwhite in #547
• Rechunk where dict has missing axes by @tomwhite in #546
• Adds logging [INFO] to TimelineVisualizationCallback and HistoryCallback by @norlandrhagen in #548
• More lenient dtype support by @tomwhite in #550
• Cast inputs to Cubed arrays in apply_ufunc by @tomwhite in #551
• Fix bug where newaxis with full slices doesn't add new axes by @tomwhite in #559
• Memray integration by @tomwhite in #558
• Add chunkmem to VirtualFullArray, which was missed in #447 by @tomwhite in #560
• Fix Array API Tests by @tomwhite in #562
• Memray callback by @tomwhite in #561
• Introduce FinalizedPlan by @tomwhite in #563
• Materialize arrays optimizer bugfix by @tomwhite in #564
• Align tqdm progress bars by @tomwhite in #567
• Coerce args to map_blocks to arrays by @tomwhite in #566
• Fail during planning if map_blocks drop_axis is for a chunked dimension by @tomwhite in #569
• Fix Rich progress bar flickering by limiting refreshes by @tomwhite in #571
• Allow Zarr compression to be set for intermediate files by @tomwhite in #572

Full Changelog: 0.16.0...0.17.0

0.16.0

What's Changed

• Improve documentation around Array API support by @tomwhite in #477
• Speed up tests by @JosepSampe in #478
• Add normalize_shape since it is not in Zarr's public API by @tomwhite in #479
• Implement virtual array indexing using ndindex by @tomwhite in #441
• Ndindex indexing by @tomwhite in #481
• Pluggable backend storage by @tomwhite in #480
• Ensure integer arrays used for indexing are NumPy arrays by @tomwhite in #485
• Indicate that we support both 2021.12 and 2022.12 versions of the Array API by @tomwhite in #487
• TensorStore storage by @tomwhite in #483
• Update slides - June 2024 by @tomwhite in #489
• Fail with NotImplementedError if backend Zarr is missing nchunks_initialized by @tomwhite in #491
• Zarr v3 by @tomwhite in #484
• Remove print for local executors by @tomwhite in #506
• document what needs to be installed to run climatology anomaly example by @rbavery in #501
• Document how to run examples on local machine using processes by @tomwhite in #507
• Check allowed_mem does not exceed total on machine for processes executor by @tomwhite in #517
• Warn if peak mem exceeds allowed_mem by @tomwhite in #516
• Remove max_tasks_per_child=1 limitation from processes executor by @tomwhite in #515
• Zarr Python v3 updates by @tomwhite in #523
• Modal progress fix by @tomwhite in #522
• Delete local context directories on exit by @tomwhite in #524
• Update demo notebooks by @tomwhite in #525
• Update CUBED_CONFIG path in example README.md by @thodson-usgs in #530
• Implement flip by @tomwhite in #528
• Add better test for local paths in delete_on_exit by @tomwhite in #532
• Deprecate cubed.extensions and move to cubed.diagnostics by @tomwhite in #533
• Turning on JAX tests for all PRs. by @alxmrs in #537
• Improve set-up instructions for examples by @tomwhite in #534
• Configuration documentation by @tomwhite in #535
• Adding compile_function as execute option. by @alxmrs in #536
• Diagnostics documentation by @tomwhite in #540

New Contributors

• @JosepSampe made their first contribution in #478
• @rbavery made their first contribution in #501
• @thodson-usgs made their first contribution in #530
• @alxmrs made their first contribution in #537

Full Changelog: 0.15.0...0.16.0

0.15.0

What's Changed

• Test on Python 3.12 by @tomwhite in #453
• Test NumPy 2 by @tomwhite in #454
• Expose Lithops wait_dur_sec to speed up tests by @tomwhite in #456
• Simplify executor naming by @tomwhite in #457
• Treat warnings as errors when running tests by @tomwhite in #458
• Re-interpret 'fusable' to mean with predecessor operations by @tomwhite in #460
• Don't include hidden arrays when counting max_total_source_arrays by @tomwhite in #463
• Limited implementation of map_overlap by @tomwhite in #462
• Limited implementation of pad by @tomwhite in #461
• Implement roll by @tomwhite in #465
• Fix map_overlap in case of single chunk in overlap dimension by @tomwhite in #466
• Fix case of Lithops 3.4.0 hanging by @tomwhite in #471
• Fix unify_chunks to return regular chunks in all cases by @tomwhite in #470
• Improve test coverage by @tomwhite in #472
• Change development status to beta by @tomwhite in #474
• Top level array namespace by @tomwhite in #473
• Add notebooks for Pangeo examples by @tomwhite in #476

Full Changelog: 0.14.3...0.15.0

0.14.3

What's Changed

• Take broadcast trick into account for array chunk memory by @tomwhite in #447
• Fix Documentation link by @norlandrhagen in #451
• Include j2 templates in the distribution by @tomwhite in #450
• Add links to blog posts by @TomNicholas in #452
• Add a groupby_blockwise function for use in Flox by @tomwhite in #448

New Contributors

• @norlandrhagen made their first contribution in #451

Full Changelog: 0.14.2...0.14.3

0.14.2

What's Changed

• Allow networkx 3.3 now that pydot issue has been fixed by @tomwhite in #445
• Change drop_axis to default to None rather than an empty list in map_blocks by @tomwhite in #446
• Add a groupby_reduction function for use in Flox by @tomwhite in #442

Full Changelog: 0.14.1...0.14.2