Error when augmenting dataset (single machine)

[gnarayan]

Hi Kyle,

I'm running your avocado code as part of putting together the PLAsTiCC validation paper, and I'm running into an error generating the augmented dataset, with pandas claiming a file isn't open.

(avocado) gnarayan@dhcp194|~/work/plasticc> avocado_augment plasticc_train plasticc_augment
Loading augmentor...
Processing the dataset in 100 chunks...
Chunk:   0%|                                                                                                                                                                                                                                                                                                                                        | 0/100 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/Users/gnarayan/miniconda3/envs/avocado/bin/avocado_augment", line 4, in <module>
    __import__('pkg_resources').run_script('avocado==0.1', 'avocado_augment')
  File "/Users/gnarayan/miniconda3/envs/avocado/lib/python3.7/site-packages/pkg_resources/__init__.py", line 666, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/Users/gnarayan/miniconda3/envs/avocado/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1453, in run_script
    exec(code, namespace, namespace)
  File "/Users/gnarayan/miniconda3/envs/avocado/lib/python3.7/site-packages/avocado-0.1-py3.7.egg/EGG-INFO/scripts/avocado_augment", line 84, in <module>
    process_chunk(augmentor, chunk, args, verbose=False)
  File "/Users/gnarayan/miniconda3/envs/avocado/lib/python3.7/site-packages/avocado-0.1-py3.7.egg/EGG-INFO/scripts/avocado_augment", line 16, in process_chunk
    num_chunks=args.num_chunks,
  File "/Users/gnarayan/miniconda3/envs/avocado/lib/python3.7/site-packages/avocado-0.1-py3.7.egg/avocado/dataset.py", line 182, in load
    num_chunks=num_chunks, **kwargs)
  File "/Users/gnarayan/miniconda3/envs/avocado/lib/python3.7/site-packages/avocado-0.1-py3.7.egg/avocado/utils.py", line 167, in read_dataframes
    key_store = store.get_storer(key)
  File "/Users/gnarayan/miniconda3/envs/avocado/lib/python3.7/site-packages/pandas/io/pytables.py", line 1249, in get_storer
    group = self.get_node(key)
  File "/Users/gnarayan/miniconda3/envs/avocado/lib/python3.7/site-packages/pandas/io/pytables.py", line 1239, in get_node
    self._check_if_open()
  File "/Users/gnarayan/miniconda3/envs/avocado/lib/python3.7/site-packages/pandas/io/pytables.py", line 1360, in _check_if_open
    raise ClosedFileError("{0} file is not open!".format(self._path))
pandas.io.pytables.ClosedFileError: ./data/plasticc_train.h5 file is not open!

Now the data very much does exist, and the files are about the right size:

(avocado) gnarayan@dhcp194|~/work/plasticc> ls -lh data/
total 57899032
drwxr-xr-x  19 gnarayan  staff   608B Sep  5 11:49 plasticc_raw
-rw-r--r--   1 gnarayan  staff    28G Sep  5 12:17 plasticc_test.h5
-rw-r--r--   1 gnarayan  staff    88M Sep  5 11:50 plasticc_train.h5

I don't believe this is an environment issue, but attaching the conda env below.

avocado.txt

Have you run into this before? Alternately, if you are willing to provide a copy of the augmented dataset you used, that'd be fine as well. We're just trying to get a sense of classification performance vs original training sample, augmented sample, and an effectively infinitely large simulated sample.

Best,
-Gautham

[kboone]

Hi Gautham,

Do you maybe have that file open in another Jupyter notebook or python environment? pandas sometimes uses a lock on HDF5 files, and seems to fail with cryptic errors when it fails to acquire that lock. I think that you might be running into something like this... I'll see if I can reproduce it and generate a better error message if that is the case.

As a stopgap, I put all of the files produced by avocado for the PLAsTiCC dataset up here:

http://supernova.lbl.gov/avocado_plasticc/

Kyle

[gnarayan]

Nope - no other open instances of python, so not sure what's up there, but stopgap will work - wgetting now - many thanks!

[kboone]

I ran into this bug myself, and tracked it down to a bug in new versions of pandas (v0.25 only) where calls to read_hdf will close files on you unexpectedly. I fixed this with a workaround in 3c23b9f.

I also raised in issue with pandas (pandas-dev/pandas#28699) to address the underlying issue.

Regardless, everything should be fine in the latest version of avocado so I'm closing this.

[gnarayan]

Thanks!