Skip to content

Advanced indexing #172

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 68 commits into from
Nov 13, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
69fde3a
ellipsis handling; resolves #93
alimanfoo Oct 25, 2017
30fae46
minor conciseness
alimanfoo Oct 26, 2017
d47492f
initial pass on orthogonal advanced indexing
alimanfoo Oct 29, 2017
f45dc6a
add 2d test
alimanfoo Oct 29, 2017
4ef758c
renaming
alimanfoo Oct 29, 2017
2b0aaaa
WIP boolean indexing
alimanfoo Oct 30, 2017
b9cda15
some refactoring
alimanfoo Oct 30, 2017
5819a84
WIP integer selection
alimanfoo Oct 30, 2017
9d1ea39
refactoring
alimanfoo Oct 30, 2017
d0abae0
doco
alimanfoo Oct 30, 2017
4d0e7ae
initial implementation of integer indexing
alimanfoo Oct 30, 2017
6c10c6e
refactor for simplicity
alimanfoo Oct 30, 2017
7c12181
only do np.ix_ if absolutely necessary
alimanfoo Oct 30, 2017
3edf96c
refactor getitem setitem together
alimanfoo Oct 30, 2017
5699983
tests for advanced setting
alimanfoo Oct 30, 2017
53efd48
test and fix error handling
alimanfoo Oct 30, 2017
95fb592
handle 3d set cases
alimanfoo Oct 30, 2017
733ef08
benchmark advanced indexing
alimanfoo Oct 31, 2017
1420ce2
flake8
alimanfoo Oct 31, 2017
db69c0d
refactor and extend tests to handle indexing edge cases
alimanfoo Oct 31, 2017
0e9acaf
deal with filter interaction bug
alimanfoo Oct 31, 2017
c7e07da
WIP refactoring to split orthogonal and basic indexing
alimanfoo Nov 1, 2017
2561945
WIP refactor oindex
alimanfoo Nov 2, 2017
8f0895d
refactored to use oindex
alimanfoo Nov 2, 2017
04f263d
add tests for slice with step
alimanfoo Nov 2, 2017
33c2023
WIP coordinate indexing
alimanfoo Nov 3, 2017
d0d50ff
WIP refactor indexing
alimanfoo Nov 4, 2017
fe28594
WIP refactor indexing
alimanfoo Nov 4, 2017
d6cf587
WIP refactor indexing
alimanfoo Nov 4, 2017
af6f65f
tests passing
alimanfoo Nov 4, 2017
3f0a98b
WIP optimise coordinate indexing
alimanfoo Nov 5, 2017
050ad79
rework indexing for performance
alimanfoo Nov 5, 2017
b94fa67
tidy up
alimanfoo Nov 6, 2017
197c6b6
rework indexing tests
alimanfoo Nov 6, 2017
ec64c08
tidy notebook
alimanfoo Nov 6, 2017
983355d
WIP index with fields
alimanfoo Nov 6, 2017
c814c97
add support for fields with selection, resolves #112
alimanfoo Nov 7, 2017
e0aeb9b
nd coordinate indexing
alimanfoo Nov 7, 2017
b4b272e
tidy up, improve tests
alimanfoo Nov 7, 2017
c9304a0
added basic selection tests
alimanfoo Nov 7, 2017
3b0966a
benchmarking, optimisation
alimanfoo Nov 7, 2017
92a0b5c
add fields selection examples
alimanfoo Nov 7, 2017
8655a77
improve test coverage
alimanfoo Nov 8, 2017
7a57458
optimise slice with step
alimanfoo Nov 8, 2017
fad4db1
rework slices; refactor tests; increase coverage
alimanfoo Nov 8, 2017
069c670
fix syntax and pep8
alimanfoo Nov 8, 2017
7d75460
WIP documentation
alimanfoo Nov 8, 2017
cbee2fc
WIP documentation
alimanfoo Nov 8, 2017
0fbfe0b
docstrings
alimanfoo Nov 9, 2017
9115ffe
documentation
alimanfoo Nov 9, 2017
0bac9f3
review tutorial, add section on advanced indexing
alimanfoo Nov 9, 2017
af2b505
improve errors
alimanfoo Nov 10, 2017
600aa93
review errors
alimanfoo Nov 10, 2017
4e19759
rebase; resolve issues with structured arrays
alimanfoo Nov 11, 2017
c7654c6
spike appveyor
alimanfoo Nov 11, 2017
c459730
spike appveyor 2
alimanfoo Nov 11, 2017
7347c4d
spike appveyor 3
alimanfoo Nov 11, 2017
e776977
spike appveyor 4
alimanfoo Nov 11, 2017
d1d8bbe
spike appveyor 5
alimanfoo Nov 11, 2017
5283bd9
spike appveyor 6
alimanfoo Nov 11, 2017
c1d2c5f
spike appveyor 7
alimanfoo Nov 11, 2017
fbb29a9
fix for windows unicode issue
alimanfoo Nov 13, 2017
23f7e96
doh
alimanfoo Nov 13, 2017
8b41414
comments
alimanfoo Nov 13, 2017
2234c33
Merge branch 'advanced-indexing-20171028' of github.com:alimanfoo/zar…
alimanfoo Nov 13, 2017
19de333
py2 compat
alimanfoo Nov 13, 2017
64db65c
cover py27
alimanfoo Nov 13, 2017
d08189c
pragma
alimanfoo Nov 13, 2017
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 15 additions & 20 deletions docs/api/codecs.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,27 +2,22 @@ Compressors and filters (``zarr.codecs``)
=========================================
.. module:: zarr.codecs

This module contains compressor and filter classes for use with Zarr.
This module contains compressor and filter classes for use with Zarr. Please note that this module
is provided for backwards compatibility with previous versions of Zarr. From Zarr version 2.2
onwards, all codec classes have been moved to a separate package called Numcodecs_. The two
packages (Zarr and Numcodecs_) are designed to be used together. For example, a Numcodecs_ codec
class can be used as a compressor for a Zarr array::

Other codecs can be registered dynamically with Zarr. All that is required
is to implement a class that provides the same interface as the classes listed
below, and then to add the class to the ``codec_registry``. See the source
code of this module for details.
>>> import zarr
>>> from numcodecs import Blosc
>>> z = zarr.zeros(1000000, compressor=Blosc(cname='zstd', clevel=1, shuffle=Blosc.SHUFFLE)

.. autoclass:: Codec
Codec classes can also be used as filters. See the tutorial section on :ref:`tutorial_filters`
for more information.

.. automethod:: encode
.. automethod:: decode
.. automethod:: get_config
.. automethod:: from_config
Please note that it is also relatively straightforward to define and register custom codec
classes. See the Numcodecs `codec API <http://numcodecs.readthedocs.io/en/latest/abc.html>`_ and
`codec registry <http://numcodecs.readthedocs.io/en/latest/registry.html>`_ documentation for more
information.

.. autoclass:: Blosc
.. autoclass:: Zlib
.. autoclass:: BZ2
.. autoclass:: LZMA
.. autoclass:: Delta
.. autoclass:: AsType
.. autoclass:: FixedScaleOffset
.. autoclass:: Quantize
.. autoclass:: PackBits
.. autoclass:: Categorize
.. _Numcodecs: http://numcodecs.readthedocs.io/
8 changes: 8 additions & 0 deletions docs/api/core.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,14 @@ The Array class (``zarr.core``)

.. automethod:: __getitem__
.. automethod:: __setitem__
.. automethod:: get_basic_selection
.. automethod:: set_basic_selection
.. automethod:: get_mask_selection
.. automethod:: set_mask_selection
.. automethod:: get_coordinate_selection
.. automethod:: set_coordinate_selection
.. automethod:: get_orthogonal_selection
.. automethod:: set_orthogonal_selection
.. automethod:: resize
.. automethod:: append
.. automethod:: view
Expand Down
43 changes: 20 additions & 23 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,27 +12,25 @@ Highlights

* Create N-dimensional arrays with any NumPy dtype.
* Chunk arrays along any dimension.
* Compress chunks using the fast Blosc_ meta-compressor or alternatively using zlib, BZ2 or LZMA.
* Compress and/or filter chunks using any numcodecs_ codec.
* Store arrays in memory, on disk, inside a Zip file, on S3, ...
* Read an array concurrently from multiple threads or processes.
* Write to an array concurrently from multiple threads or processes.
* Organize arrays into hierarchies via groups.
* Use filters to preprocess data and improve compression.

Status
------

Zarr is still in an early phase of development. Feedback and bug
reports are very welcome, please get in touch via the `GitHub issue
tracker <https://github.com/alimanfoo/zarr/issues>`_.
Zarr is still a young project. Feedback and bug reports are very welcome, please get in touch via
the `GitHub issue tracker <https://github.com/alimanfoo/zarr/issues>`_.

Installation
------------

Zarr depends on NumPy. It is generally best to `install NumPy
<http://docs.scipy.org/doc/numpy/user/install.html>`_ first using
whatever method is most appropriate for you operating system and
Python distribution.
<http://docs.scipy.org/doc/numpy/user/install.html>`_ first using whatever method is most
appropriate for you operating system and Python distribution. Other dependencies should be
installed automatically if using one of the installation methods below.

Install Zarr from PyPI::

Expand All @@ -41,26 +39,18 @@ Install Zarr from PyPI::
Alternatively, install Zarr via conda::

$ conda install -c conda-forge zarr

Zarr includes a C extension providing integration with the Blosc_
library. Installing via conda will install a pre-compiled binary distribution.
However, if you have a newer CPU that supports the AVX2 instruction set (e.g.,
Intel Haswell, Broadwell or Skylake) then installing via pip is preferable,
because this will compile the Blosc library from source with optimisations
for AVX2.


To work with Zarr source code in development, install from GitHub::

$ git clone --recursive https://github.com/alimanfoo/zarr.git
$ cd zarr
$ python setup.py install

To verify that Zarr has been fully installed (including the Blosc
extension) run the test suite::
To verify that Zarr has been fully installed, run the test suite::

$ pip install nose
$ python -m nose -v zarr

Contents
--------

Expand All @@ -75,13 +65,20 @@ Contents
Acknowledgments
---------------

Zarr bundles the `c-blosc <https://github.com/Blosc/c-blosc>`_
library and uses it as the default compressor.
The following people have contributed to the development of Zarr, by contributing code and/or
providing ideas, feedback and advice:

* `Francesc Alted <https://github.com/FrancescAlted>`_
* `Stephan Hoyer <https://github.com/shoyer>`_
* `John Kirkham <https://github.com/jakirkham>`_
* `Alistair Miles <https://github.com/alimanfoo>`_
* `Matthew Rocklin <https://github.com/mrocklin>`_
* `Vincent Schut <https://github.com/vincentschut>`_

Zarr is inspired by `HDF5 <https://www.hdfgroup.org/HDF5/>`_, `h5py
<http://www.h5py.org/>`_ and `bcolz <http://bcolz.blosc.org/>`_.

Development of this package is supported by the
Development of Zarr is supported by the
`MRC Centre for Genomics and Global Health <http://www.cggh.org>`_.

Indices and tables
Expand All @@ -91,4 +88,4 @@ Indices and tables
* :ref:`modindex`
* :ref:`search`

.. _Blosc: http://www.blosc.org/
.. _numcodecs: http://numcodecs.readthedocs.io/
Loading