pd.read_hdf bug

[romquentin]

Seems related to #18252
Link to the hdf5 file to reproduce: https://www.dropbox.com/s/kwwv6hqtdpswkf9/behavior.hdf5?dl=0

Code Sample

These 2 codes:

from pandas import read_hdf  
events = read_hdf('./behavior.hdf5')

from h5io import read_hdf5
events = read_hdf5('./behavior.hdf5')

give me this similar error:

Error

/data/quentinra/conda/envs/mne/lib/python3.6/site-packages/pandas/io/pytables.py in read_hdf(path_or_buf, key, mode, **kwargs)
    392                                      'contains multiple datasets.')
    393             key = candidate_only_group._v_pathname
--> 394         return store.select(key, auto_close=auto_close, **kwargs)
    395     except:
    396         # if there is an error, close the store

/data/quentinra/conda/envs/mne/lib/python3.6/site-packages/pandas/io/pytables.py in select(self, key, where, start, stop, columns, iterator, chunksize, auto_close, **kwargs)
    739                            chunksize=chunksize, auto_close=auto_close)
    740 
--> 741         return it.get_result()
    742 
    743     def select_as_coordinates(

/data/quentinra/conda/envs/mne/lib/python3.6/site-packages/pandas/io/pytables.py in get_result(self, coordinates)
   1481 
   1482         # directly return the result
-> 1483         results = self.func(self.start, self.stop, where)
   1484         self.close()
   1485         return results

/data/quentinra/conda/envs/mne/lib/python3.6/site-packages/pandas/io/pytables.py in func(_start, _stop, _where)
    732             return s.read(start=_start, stop=_stop,
    733                           where=_where,
--> 734                           columns=columns)
    735 
    736         # create the iterator

/data/quentinra/conda/envs/mne/lib/python3.6/site-packages/pandas/io/pytables.py in read(self, start, stop, **kwargs)
   2926 
   2927             _start, _stop = (start, stop) if i == select_axis else (None, None)
-> 2928             ax = self.read_index('axis%d' % i, start=_start, stop=_stop)
   2929             axes.append(ax)
   2930 

/data/quentinra/conda/envs/mne/lib/python3.6/site-packages/pandas/io/pytables.py in read_index(self, key, **kwargs)
   2521             return self.read_sparse_intindex(key, **kwargs)
   2522         elif variety == u('regular'):
-> 2523             _, index = self.read_index_node(getattr(self.group, key), **kwargs)
   2524             return index
   2525         else:  # pragma: no cover

/data/quentinra/conda/envs/mne/lib/python3.6/site-packages/pandas/io/pytables.py in read_index_node(self, node, start, stop)
   2651             index = factory(_unconvert_index(data, kind,
   2652                                              encoding=self.encoding,
-> 2653                                              errors=self.errors), **kwargs)
   2654 
   2655         index.name = name

/data/quentinra/conda/envs/mne/lib/python3.6/site-packages/pandas/io/pytables.py in _unconvert_index(data, kind, encoding, errors)
   4561     elif kind in (u('string')):
   4562         index = _unconvert_string_array(data, nan_rep=None, encoding=encoding,
-> 4563                                         errors=errors)
   4564     elif kind == u('object'):
   4565         index = np.asarray(data[0])

/data/quentinra/conda/envs/mne/lib/python3.6/site-packages/pandas/io/pytables.py in _unconvert_string_array(data, nan_rep, encoding, errors)
   4654         nan_rep = 'nan'
   4655 
-> 4656     data = libwriters.string_array_replace_from_nan_rep(data, nan_rep)
   4657     return data.reshape(shape)
   4658 

pandas/_libs/writers.pyx in pandas._libs.writers.string_array_replace_from_nan_rep()

ValueError: Buffer dtype mismatch, expected 'Python object' but got 'double'

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]
INSTALLED VERSIONS

commit: None
python: 3.6.6.final.0
python-bits: 64
OS: Linux
OS-release: 3.10.0-862.14.4.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.4
pytest: 4.0.0
pip: 18.1
setuptools: 40.6.2
Cython: 0.29
numpy: 1.15.4
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 7.1.1
sphinx: 1.8.2
patsy: 0.5.1
dateutil: 2.7.5
pytz: 2018.7
blosc: None
bottleneck: None
tables: 3.4.4
numexpr: 2.6.8
feather: None
matplotlib: 3.0.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[TomAugspurger]

Is it possible to provide code that generates a small DataFrame that creates the error when round-tripped through HDF5? We prefer that to dropbox links with files, since it'll be easier to convert to a unit test, and the dropbox link may go away before the issue is fixed.

[romquentin]

I am not able to reproduce the error by writing a new hdf5 file from a pandaFrame. The hdf5 file that I am using has been saved several months ago under python2 and earlier version of pandas, maybe this is the reason. I can rewrite it. Thanks :-)