Numexpr 2.2.2: "could not operate with block values <...>" error

[aldanor]

Under numexpr 2.2.2 (and for some reason that's the latest version available in the anaconda distribution at the moment), I get the following kind of errors:

>>> data.shape
(2, 219, 104, 6)
>>> data.dtype
dtype('float64')
>>> df = pd.Panel(data[0]).to_frame().reset_index()
>>> df[0].dtype
dtype('float64')
>>> df[0].count()
711
>>> df[0] + 1
TypeError: Could not operate 1 with block values using `oa_ndim == 0` when `op_axes` is NULL is 
deprecated. Use `oa_ndim == -1` or the MultiNew iterator for NumPy <1.8 compatibility

Looks like the same as here and here.

Upgrading to numexpr 2.3 solves it. Should v2.3 be then strictly required by pandas?

[jreback]

hmm; their was some issues with 2.2, so 2.2.1 was released, but that was for an argumenet incompatiblity with pytables < 3.0.0.

afaict numexpr 2.2 was only released for 6 days ubntil 2.2.1 superseded. this was back in sept 2013 iirc. when did you install?

this is with numpy 1.8?

numexpr < 2 is prohibited

maybe just a note in the install.rst....care to do PR?

[jreback]

sorry....i meant 2.2.2....can you post ci/print_versions.py

[aldanor]

Versions: pandas==0.13.0, numpy==1.8.0, numexpr==2.2.2.

So, I've double-checked and it looks like the error appears only if numpy >= 1.8 and numexpr <= 2.2.2. Works fine with numpy==1.7.1 and numexpr==2.2.2 (but numpy < 1.8 currently causes other sorts of bugs as you may know). Also works fine with numpy==1.8.0 and numexpr==2.3. So what exactly would you put in install.rst? (note also I'm using anaconda distribution and they currently ship numpy 1.7.1, numexpr 2.2.2 and pandas 0.13.0 for some reason)

Btw: numexpr 2.3 was released just yesterday.

[jreback]

This does not raise in master,
can you give a try...maybe something else ahhpening (or maybe tell me where the error appear)

e.g. do %pdb on in ipython...then show me a bigger traceback, also what platform?

In [5]: df[0].dtype
Out[5]: dtype('float64')

In [6]: df[0].count()
Out[6]: 624

In [7]: df[0]+1
Out[7]: 
0     0.184474
1     0.247715
2     2.766676
3     0.669518
4     2.737060
5     0.185374
6     0.188507
7     0.862162
8    -0.538982
9     1.319512
10    0.734816
11    0.934172
12    0.469460
13   -1.052177
14   -1.257805
...
609    1.222683
610    0.163969
611    0.490085
612   -0.081300
613   -0.396591
614    0.009749
615   -0.075669
616    1.020612
617    1.110930
618    1.832211
619    2.219956
620    0.507602
621   -0.916604
622    0.571296
623    1.614161
Name: 0, Length: 624, dtype: float64

In [8]: import numexpr as ne

In [9]: import numpy as np

In [10]: np.__version__
Out[10]: '1.8.0'

In [11]: ne.__version__
Out[11]: '2.2.2'

[jreback]

@aldanor can you try this with master to see if seomthing else is going on?

thanks

[aldanor]

@jreback So, I installed v2.2.2 back and it's crashing again 100% of times, but in a very strange way that I can't seem to reproduce. If right before the crash I pickle.dump / numpy.save / numpy.getbuffer the data (whether it's the panel, the dataframe or the underlying numpy array) and later try to reload it and run the same code -- it doesn't crash. If I np.save/np.load the numpy array while within the %pdb, it also works fine and doesn't crash. Gotta be something corrupting the np.array then? That's mighty weird.

[jreback]

its possible that since you are taking a view of the 4-d numpy array that its getting corrupted. I'll bet if you copy it before you stick it in the panel it will work.

run it in a separate python shell, put try: except: around the line that crashes and post the traceback

[aldanor]

Here's the traceback

Traceback (most recent call last):
  File "export_snapshots.py", line 323, in retrieve_snapshots
    df['bid'] += 1.  # make it crash
  File ".../pandas/core/ops.py", line 504, in wrapper
    return left._constructor(wrap_results(na_op(lvalues, rvalues)),
  File ".../pandas/core/ops.py", line 443, in na_op
    raise_on_error=True, **eval_kwargs)
  File ".../pandas/computation/expressions.py", line 175, in evaluate
    **eval_kwargs)
  File ".../pandas/computation/expressions.py", line 103, in _evaluate_numexpr
    **eval_kwargs)
  File ".../numexpr/necompiler.py", line 742, in evaluate
    return compiled_ex(*arguments, **kwargs)
DeprecationWarning: using `oa_ndim == 0` when `op_axes` is NULL is 
deprecated. Use `oa_ndim == -1` or the MultiNew iterator for NumPy <1.8 compatibility

[aldanor]

Looks like copying the np array doesn't help either, that's what I tried first thing.

[aldanor]

Another observation: if you use .to_frame(True), it works fine, however .to_frame(False) makes it crash.

[jreback]

ahh....that has to do with duplicate observations....

[jreback]

can you post exactyl what you are contructing from the beginning?

starting with the numpy array

[aldanor]

Yea it has stuff like that for the most part, could it matter?

0     0     NaN NaN NaN NaN NaN NaN
      1     NaN NaN NaN NaN NaN NaN
      2     NaN NaN NaN NaN NaN NaN
      3     NaN NaN NaN NaN NaN NaN
      4     NaN NaN NaN NaN NaN NaN
      5     NaN NaN NaN NaN NaN NaN

[jreback]

I need your exactly construction code

[aldanor]

Ok looks like I found it, the problem is in np.swapaxes -- very weird though since I'm making an np.copy.

This fails:

pd.Panel(np.copy(np.swapaxes(snapshots.data[0], 0, 2))).to_frame(False)[0] + 1

while this doesn't (in this case np.copy is not even needed actually):

pd.Panel(np.copy(snapshots.data[0])).to_frame(False)[0] + 1

Here snapshots.data is a 4-D np.float64 array with lots of NaNs.