np.nan rows being dropped from HDFStore frame_tables using append

[gravesee]

When storing a dataframe using append, np.nan rows are dropped from the stored frame_tables:

test = pd.Series(range(401125), dtype=np.float64)
idx = np.random.permutation(test.index)[:50000]
test[idx] = np.nan

# check that there are indeed np.nan in the series
test.isnull().value_counts()
False    351125
True      50000

# create a store
tmp = pd.HDFStore('tmp.h5', 'w')

# add as a DataFrame using put
tmp.put('df_put', pd.DataFrame(test))

tmp.df_put.squeeze().isnull().value_counts()
False    351125
True      50000

# add as a DataFrame using append
tmp.append('df_append', pd.DataFrame(test))

tmp.df_append.squeeze().isnull().value_counts()
False    351125

[jreback]

see notes & caveats http://pandas.pydata.org/pandas-docs/dev/io.html#notes-caveats, 2nd one down

hard to fix and usually not not a bug deal

[gravesee]

Gah, sorry man. I even referenced the docs before posting this issue. I'm really trying to get a viable workflow going and I'm very close. As for how to handle the np.nans being dropped, what do you think about storing the index in a StoreManager class. When an object is requested form the store, it is reindexed using the original, complete index. Something like this StoreManager class is what I'm working on. Thanks again for your help and sorry for the noob questions.

[jreback]

For your workflow that is probably approriate, n.b. I only drop all-nan rows, which is normally not a problem in a frame (as every field has to be a nan in a particular row). I originally did this for panels. Its not hard to fix, but prob won't get to it till 0.12. Maybe can incorporate your StoreManager for that purpose

[jreback]

closed by #4625