BUG: #17778 - DataFrame.to_pickle() fails for .zip format on MacOS and pandas 0.20.3 #2
Conversation
Codecov Report
@@ Coverage Diff @@
## master #2 +/- ##
=========================================
Coverage ? 91.36%
=========================================
Files ? 164
Lines ? 49891
Branches ? 0
=========================================
Hits ? 45582
Misses ? 4309
Partials ? 0
Continue to review full report at Codecov.
|
ghost
force-pushed
the
#17778
branch
5 times, most recently
from
2813bb9 to
9a80160
Compare
|
There are |
ghost
force-pushed
the
#17778
branch
3 times, most recently
from
52eab51 to
df5233c
Compare
GH17778: add 'zip' format to unittests. Added entry in doc/source/whatsnew/v0.22.0.txt file to Bug Fixes section.
pandas/io/pickle.py
Outdated
| @@ -42,7 +42,17 @@ def to_pickle(obj, path, compression='infer', protocol=pkl.HIGHEST_PROTOCOL): | |||
| if protocol < 0: | |||
| protocol = pkl.HIGHEST_PROTOCOL | |||
| try: | |||
| pkl.dump(obj, f, protocol=protocol) | |||
| import zipfile | |||
There was a problem hiding this comment.
Update docstring as now it also uses 'zip' compression
Moved imports to top.
| @@ -42,7 +46,14 @@ def to_pickle(obj, path, compression='infer', protocol=pkl.HIGHEST_PROTOCOL): | |||
| if protocol < 0: | |||
| protocol = pkl.HIGHEST_PROTOCOL | |||
| try: | |||
| pkl.dump(obj, f, protocol=protocol) | |||
| if isinstance(f, zipfile.ZipFile): | |||
There was a problem hiding this comment.
I really don't like it - I don't expect that _get_handle returns something other than file-like. Creating temp file is unacceptable - some buffered solution can be introduced.Unfortunately support for buffered zip writing is in Python 3.6, but we can backport it and place in: https://github.com/pandas-dev/pandas/blob/master/pandas/compat/
Python 3.6 supports mode='w' in Zipfile.open (https://github.com/python/cpython/blob/master/Lib/zipfile.py#L1312) so this whole commit for Python 3.6 could look like:
if mode == 'wb':
f = zipfile.open('data.bin', mode='w')
else:
# find filename and open for read
It should be really easy to backport for Python 3,x - backporting for Python 2.7 can be trickier
git diff upstream/master -u -- "*.py" | flake8 --diff