Performance degradation: `DataArray` with `dtype=object` of `DataArray` gets very slow indexing

[burnpanck]

I did not follow the code deeply, but there clearly seems to be a huge overhead when indexing such arrays. In particular, in the following code

import xarray as xr
import numpy as np

a = xr.DataArray([None for k in range(100)],dims='c')
for k in range(a.c.size):
    a[k] = xr.DataArray(np.random.randn(1000,5),dims=['a','b'])

%prun a[0]

the indexing operation takes about 1 second on my machine, or 2 seconds when running under the profiler. The profiler output shows lots of functions with a recursive call count exceeding 100'000 (most likely iterating through each row of the contained sub-arrays). However, there is really no reason to iterate through the nested elements.

[shoyer]

Nesting xarray objects was not an intended use case, but I agree, this is ridiculously slow.

Looks the culprit is xarray.core.utils.to_0d_object_array:

xarray/xarray/core/utils.py

Lines 198 to 212 in d66f673

	def to_0d_object_array(value):
	"""Given a value, wrap it in a 0-D numpy.ndarray with dtype=object."""
	result = np.empty((1,), dtype=object)
	result[:] = [value]
	result.shape = ()
	return result
	def to_0d_array(value):
	"""Given a value, wrap it in a 0-D numpy.ndarray."""
	if np.isscalar(value) or (isinstance(value, np.ndarray)
	and value.ndim == 0):
	return np.array(value)
	else:
	return to_0d_object_array(value)

Specifically, when value is an xarray.DataArray, these lines appear to trigger a iterating over every element of the DataArray, one by one:

result = np.empty((1,), dtype=object)
result[:] = [value]

I'm definitely open to fixes here. Creating a NumPy object array can be very fussy -- NumPy tries way too hard to coerce dtypes.

[burnpanck]

Well, xarrays are way too useful not to nest them, even if that involves the scary dtype=object :-). Thanks for pointing me in the right direction, I'll try to find a fix.

[burnpanck]

I think I found it (#1122). I guess whenever a non-scalar assignment is made (as in result[:] = [value]), something like np.asanyarray is performed on the new value. Luckily, numpy is perfectly happy with indexing of a 0d array with a 0d index (i.e. an empty tuple).
Thinking about it, result[0] = value would probably have worked too.