Merge remote-tracking branch 'upstream/master' into load_nonindex_coords

Author: gimperiale

ci/requirements-py36.yml

@@ -16,6 +16,7 @@ dependencies:
   - seaborn
   - toolz
   - rasterio
+  - bottleneck
   - pip:
     - coveralls
     - pytest-cov

doc/Makefile

@@ -2,10 +2,11 @@
 #
 
 # You can set these variables from the command line.
-SPHINXOPTS    =
-SPHINXBUILD   = sphinx-build
-PAPER         =
-BUILDDIR      = _build
+SPHINXOPTS      =
+SPHINXBUILD     = sphinx-build
+SPHINXATUOBUILD = sphinx-autobuild
+PAPER           =
+BUILDDIR        = _build
 
 # Internal variables.
 PAPEROPT_a4     = -D latex_paper_size=a4
@@ -18,6 +19,7 @@ I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
 help:
 	@echo "Please use \`make <target>' where <target> is one of"
 	@echo "  html       to make standalone HTML files"
+	@echo "  livehtml   Make standalone HTML files and rebuild the documentation when a change is detected. Also includes a livereload enabled web server"
 	@echo "  dirhtml    to make HTML files named index.html in directories"
 	@echo "  singlehtml to make a single large HTML file"
 	@echo "  pickle     to make pickle files"
@@ -56,6 +58,11 @@ html:
 	@echo
 	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
 
+.PHONY: livehtml
+livehtml:
+	# @echo "$(SPHINXATUOBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html"
+	$(SPHINXATUOBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
+
 .PHONY: dirhtml
 dirhtml:
 	$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml

doc/dask.rst

@@ -13,7 +13,7 @@ dependency in a future version of xarray.
 For a full example of how to use xarray's dask integration, read the
 `blog post introducing xarray and dask`_.
 
-.. _blog post introducing xarray and dask: http://continuum.io/blog/xray-dask
+.. _blog post introducing xarray and dask: https://www.anaconda.com/blog/developer-blog/xray-dask-out-core-labeled-arrays-python/
 
 What is a dask array?
 ---------------------

doc/installing.rst

@@ -30,7 +30,7 @@ For accelerating xarray
 
 - `bottleneck <https://github.com/kwgoodman/bottleneck>`__: speeds up
   NaN-skipping and rolling window aggregations by a large factor
-  (1.0 or later)
+  (1.1 or later)
 - `cyordereddict <https://github.com/shoyer/cyordereddict>`__: speeds up most
   internal operations with xarray data structures (for python versions < 3.5)
 

doc/io.rst

@@ -38,18 +38,22 @@ module:
 
     pickle.loads(pkl)
 
-Pickle support is important because it doesn't require any external libraries
+Pickling is important because it doesn't require any external libraries
 and lets you use xarray objects with Python modules like
-:py:mod:`multiprocessing`. However, there are two important caveats:
+:py:mod:`multiprocessing` or :ref:`Dask <dask>`. However, pickling is
+**not recommended for long-term storage**.
 
-1. To simplify serialization, xarray's support for pickle currently loads all
-   array values into memory before dumping an object. This means it is not
-   suitable for serializing datasets too big to load into memory (e.g., from
-   netCDF or OPeNDAP).
-2. Pickle will only work as long as the internal data structure of xarray objects
-   remains unchanged. Because the internal design of xarray is still being
-   refined, we make no guarantees (at this point) that objects pickled with
-   this version of xarray will work in future versions.
+Restoring a pickle requires that the internal structure of the types for the
+pickled data remain unchanged. Because the internal design of xarray is still
+being refined, we make no guarantees (at this point) that objects pickled with
+this version of xarray will work in future versions.
+
+.. note::
+
+  When pickling an object opened from a NetCDF file, the pickle file will
+  contain a reference to the file on disk. If you want to store the actual
+  array values, load it into memory first with :py:meth:`~xarray.Dataset.load`
+  or :py:meth:`~xarray.Dataset.compute`.
 
 .. _dictionary io:
 
@@ -384,6 +388,38 @@ over the network until we look at particular values:
 
 .. image:: _static/opendap-prism-tmax.png
 
+Some servers require authentication before we can access the data. For this
+purpose we can explicitly create a :py:class:`~xarray.backends.PydapDataStore`
+and pass in a `Requests`__ session object. For example for 
+HTTP Basic authentication::
+
+    import xarray as xr
+    import requests
+
+    session = requests.Session()
+    session.auth = ('username', 'password')
+
+    store = xr.backends.PydapDataStore.open('http://example.com/data',
+                                            session=session)
+    ds = xr.open_dataset(store)
+
+`Pydap's cas module`__ has functions that generate custom sessions for 
+servers that use CAS single sign-on. For example, to connect to servers
+that require NASA's URS authentication::
+
+  import xarray as xr
+  from pydata.cas.urs import setup_session
+
+  ds_url = 'https://gpm1.gesdisc.eosdis.nasa.gov/opendap/hyrax/example.nc'
+
+  session = setup_session('username', 'password', check_url=ds_url)
+  store = xr.backends.PydapDataStore.open(ds_url, session=session)
+
+  ds = xr.open_dataset(store)
+
+__ http://docs.python-requests.org
+__ http://pydap.readthedocs.io/en/latest/client.html#authentication
+
 .. _io.rasterio:
 
 Rasterio

doc/whats-new.rst

@@ -23,14 +23,23 @@ Breaking changes
 
 - Supplying ``coords`` as a dictionary to the ``DataArray`` constructor without
   also supplying an explicit ``dims`` argument is no longer supported. This
-  behavior was deprecated in version 0.9 but is now an error (:issue:`727`).
+  behavior was deprecated in version 0.9 but will now raise an error
+  (:issue:`727`).
   By `Joe Hamman <https://github.com/jhamman>`_.
 
+- ``repr`` and the Jupyter Notebook won't automatically compute dask variables.
+  Datasets loaded with ``open_dataset`` won't automatically read coords from
+  disk when calling ``repr`` (:issue:`1522`).
+  By `Guido Imperiale <https://github.com/crusaderky>`_.
+
 Backward Incompatible Changes
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 - Old numpy < 1.11 and pandas < 0.18 are no longer supported (:issue:`1512`).
   By `Keisuke Fujii <https://github.com/fujiisoup>`_.
+- The minimum supported version bottleneck has increased to 1.1
+  (:issue:`1279`).
+  By `Joe Hamman <https://github.com/jhamman>`_.
 
 Enhancements
 ~~~~~~~~~~~~
@@ -58,7 +67,7 @@ Enhancements
 
 - More attributes available in :py:attr:`~xarray.Dataset.attrs` dictionary when
   raster files are opened with :py:func:`~xarray.open_rasterio`.
-  By `Greg Brener <https://github.com/gbrener>`_
+  By `Greg Brener <https://github.com/gbrener>`_.
 
 - Support for NetCDF files using an ``_Unsigned`` attribute to indicate that a
   a signed integer data type should be interpreted as unsigned bytes
@@ -97,23 +106,41 @@ Enhancements
   other means (:issue:`1459`).
   By `Ryan May <https://github.com/dopplershift>`_.
 
- - Support passing keyword arguments to ``load``, ``compute``, and ``persist``
-   methods. Any keyword arguments supplied to these methods are passed on to
-   the corresponding dask function (:issue:`1523`).
-   By `Joe Hamman <https://github.com/jhamman>`_.
+- Support passing keyword arguments to ``load``, ``compute``, and ``persist``
+  methods. Any keyword arguments supplied to these methods are passed on to
+  the corresponding dask function (:issue:`1523`).
+  By `Joe Hamman <https://github.com/jhamman>`_.
+
 - Encoding attributes are now preserved when xarray objects are concatenated.
   The encoding is copied from the first object  (:issue:`1297`).
   By `Joe Hamman <https://github.com/jhamman>`_ and
-  `Gerrit Holl <https://github.com/gerritholl`_.
+  `Gerrit Holl <https://github.com/gerritholl>`_.
+
+- Changed :py:class:`~xarray.backends.PydapDataStore` to take a Pydap dataset.
+  This permits opening Opendap datasets that require authentication, by
+  instantiating a Pydap dataset with a session object. Also added
+  :py:meth:`xarray.backends.PydapDataStore.open` which takes a url and session
+  object (:issue:`1068`).
+  By `Philip Graae <https://github.com/mrpgraae>`_.
+
+- Support applying rolling window operations using bottleneck's moving window
+  functions on data stored as dask arrays (:issue:`1279`).
+  By `Joe Hamman <https://github.com/jhamman>`_.
 
 Bug fixes
 ~~~~~~~~~
 
+- Unsigned int support for reduce methods with skipna=True
+  (:issue:1562).
+  By `Keisuke Fujii` <https://github.com/fujiisoup>`_.
+
 - Fixes to ensure xarray works properly with the upcoming pandas 0.21 release:
+
   - Fix :py:meth:`~xarray.DataArray.isnull` method (:issue:`1549`).
   - :py:meth:`~xarray.DataArray.to_series` and
     :py:meth:`~xarray.Dataset.to_dataframe` should not return a ``pandas.MultiIndex``
     for 1D data (:issue:`1548`).
+  By `Stephan Hoyer <https://github.com/shoyer>`_.
 
 - :py:func:`~xarray.open_rasterio` method now shifts the rasterio
   coordinates so that they are centered in each pixel.
@@ -141,6 +168,9 @@ Bug fixes
   objects with data stored as ``dask`` arrays (:issue:`1529`).
   By `Joe Hamman <https://github.com/jhamman>`_.
 
+- ``:py:meth:`~xarray.Dataset.__init__` raises a ``MergeError`` if an
+  coordinate shares a name with a dimension but is comprised of arbitrary
+  dimensions(:issue:`1120`).
 - :py:func:`~xarray.open_rasterio` method now skips rasterio.crs -attribute if
   it is none.
   By `Leevi Annala <https://github.com/leevei>`_.
@@ -149,6 +179,10 @@ Bug fixes
   provided (:issue:`1410`).
   By `Joe Hamman <https://github.com/jhamman>`_.
 
+- Fix :py:func:`xarray.save_mfdataset` to properly raise an informative error
+  when objects other than  ``Dataset`` are provided (:issue:`1555`).
+  By `Joe Hamman <https://github.com/jhamman>`_.
+
 .. _whats-new.0.9.6:
 
 v0.9.6 (8 June 2017)
@@ -1433,7 +1467,7 @@ a collection of netCDF (using dask) as a single ``xray.Dataset`` object. For
 more on dask, read the `blog post introducing xray + dask`_ and the new
 documentation section :doc:`dask`.
 
-.. _blog post introducing xray + dask: http://continuum.io/blog/xray-dask
+.. _blog post introducing xray + dask: https://www.anaconda.com/blog/developer-blog/xray-dask-out-core-labeled-arrays-python/
 
 Dask makes it possible to harness parallelism and manipulate gigantic datasets
 with xray. It is currently an optional dependency, but it may become required

xarray/backends/api.py

@@ -2,15 +2,14 @@
 from __future__ import division
 from __future__ import print_function
 import os.path
-from distutils.version import LooseVersion
 from glob import glob
 from io import BytesIO
 from numbers import Number
 
 
 import numpy as np
 
-from .. import backends, conventions
+from .. import backends, conventions, Dataset
 from .common import ArrayWriter, GLOBAL_LOCK
 from ..core import indexing
 from ..core.combine import auto_combine
@@ -289,7 +288,7 @@ def maybe_decode_store(store, lock=False):
             store = backends.ScipyDataStore(filename_or_obj,
                                             autoclose=autoclose)
         elif engine == 'pydap':
-            store = backends.PydapDataStore(filename_or_obj)
+            store = backends.PydapDataStore.open(filename_or_obj)
         elif engine == 'h5netcdf':
             store = backends.H5NetCDFStore(filename_or_obj, group=group,
                                            autoclose=autoclose)
@@ -656,6 +655,11 @@ def save_mfdataset(datasets, paths, mode='w', format=None, groups=None,
         raise ValueError("cannot use mode='w' when writing multiple "
                          'datasets to the same path')
 
+    for obj in datasets:
+        if not isinstance(obj, Dataset):
+            raise TypeError('save_mfdataset only supports writing Dataset '
+                            'objects, recieved type %s' % type(obj))
+
     if groups is None:
         groups = [None] * len(datasets)
 

xarray/backends/pydap_.py

@@ -60,9 +60,19 @@ class PydapDataStore(AbstractDataStore):
     This store provides an alternative way to access OpenDAP datasets that may
     be useful if the netCDF4 library is not available.
     """
-    def __init__(self, url):
+    def __init__(self, ds):
+        """
+        Parameters
+        ----------
+        ds : pydap DatasetType
+        """
+        self.ds = ds
+
+    @classmethod
+    def open(cls, url, session=None):
         import pydap.client
-        self.ds = pydap.client.open_url(url)
+        ds = pydap.client.open_url(url, session=session)
+        return cls(ds)
 
     def open_store_variable(self, var):
         data = indexing.LazilyIndexedArray(PydapArrayWrapper(var))

xarray/core/common.py

@@ -474,6 +474,33 @@ def rolling(self, min_periods=None, center=False, **windows):
         Returns
         -------
         rolling : type of input argument
+        
+        Examples
+        --------
+        Create rolling seasonal average of monthly data e.g. DJF, JFM, ..., SON:
+        
+        >>> da = xr.DataArray(np.linspace(0,11,num=12),
+        ...                   coords=[pd.date_range('15/12/1999',
+        ...                           periods=12, freq=pd.DateOffset(months=1))], 
+        ...                   dims='time')
+        >>> da
+        <xarray.DataArray (time: 12)>
+        array([  0.,   1.,   2.,   3.,   4.,   5.,   6.,   7., 8.,   9.,  10.,  11.])
+        Coordinates:
+          * time     (time) datetime64[ns] 1999-12-15 2000-01-15 2000-02-15 ...
+        >>> da.rolling(time=3).mean()
+        <xarray.DataArray (time: 12)>
+        array([ nan,  nan,   1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.])
+        Coordinates:
+          * time     (time) datetime64[ns] 1999-12-15 2000-01-15 2000-02-15 ...
+
+        Remove the NaNs using ``dropna()``:
+        
+        >>> da.rolling(time=3).mean().dropna('time')
+        <xarray.DataArray (time: 10)>
+        array([  1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.])
+        Coordinates:
+          * time     (time) datetime64[ns] 2000-02-15 2000-03-15 2000-04-15 ...
         """
 
         return self._rolling_cls(self, min_periods=min_periods,

xarray/core/dask_array_ops.py

@@ -0,0 +1,26 @@
+"""Define core operations for xarray objects.
+"""
+import numpy as np
+
+try:
+    import dask.array as da
+except ImportError:
+    pass
+
+
+def dask_rolling_wrapper(moving_func, a, window, min_count=None, axis=-1):
+    '''wrapper to apply bottleneck moving window funcs on dask arrays'''
+    # inputs for ghost
+    if axis < 0:
+        axis = a.ndim + axis
+    depth = {d: 0 for d in range(a.ndim)}
+    depth[axis] = window - 1
+    boundary = {d: np.nan for d in range(a.ndim)}
+    # create ghosted arrays
+    ag = da.ghost.ghost(a, depth=depth, boundary=boundary)
+    # apply rolling func
+    out = ag.map_blocks(moving_func, window, min_count=min_count,
+                        axis=axis, dtype=a.dtype)
+    # trim array
+    result = da.ghost.trim_internal(out, depth)
+    return result

xarray/core/dataarray.py

@@ -447,8 +447,8 @@ def _level_coords(self):
         """
         level_coords = OrderedDict()
         for cname, var in self._coords.items():
-            if var.ndim == 1:
-                level_names = var.to_index_variable().level_names
+            if var.ndim == 1 and isinstance(var, IndexVariable):
+                level_names = var.level_names
                 if level_names is not None:
                     dim, = var.dims
                     level_coords.update({lname: dim for lname in level_names})