BUG: AttributeError when read_sas used with GCS #33070
Conversation
With GCSFS (and possibly other connectors), the output from they file read is `bytes`, not a Unicode `str`. The `encode` call is unnecessary in this case.
tswast
force-pushed
the
issue33069-read_sas-gcsfs
branch
from
8f760d2 to
f31b6b2
Compare
|
Not sure why "20.00% of diff hit (target 50.00%)", since I added a test and it's only one additional line of code. Maybe needs another test in text-mode? Edit: None of the available XPT data files can be opened as a text file. It seems the |
|
@bashtage if you'd have a look. |
|
@tswast can you merge upstream master to resolve conflicts. |
| @@ -265,10 +265,6 @@ def __init__( | |||
| else: | |||
| # Copy to BytesIO, and ensure no encoding | |||
| contents = filepath_or_buffer.read() | |||
| try: | |||
| contents = contents.encode(self._encoding) | |||
There was a problem hiding this comment.
Is the encoding every applied to contents after this change? Does this matter?
There was a problem hiding this comment.
The encoding was only to get from text-mode to binary mode. As far as I can tell, SAS XPORT files don't have a text-mode, so the encoding is unnecessary in Python 3.
Later on, bytes corresponding to text columns are decoded into strings.
pandas/io/sas/sas_xport.py
Outdated
| try: | ||
| contents = contents.encode(self._encoding) | ||
| except UnicodeEncodeError: | ||
| pass | ||
| self.filepath_or_buffer = BytesIO(contents) |
There was a problem hiding this comment.
Is it possible to determine if it is a binary file handle already, so that one can avoid copying the file to a BytesIO and the having to reread this?
There was a problem hiding this comment.
I think the conversion to BytesIO is always unnecessary on Python 3. Removed this conversion and added a comment with my thoughts.
There was a problem hiding this comment.
Per https://www.loc.gov/preservation/digital/formats/fdd/fdd000464.shtml, rows are padded with Null bytes, so I think reading an XPORT file in text-mode on Python 3 will always fail.
|
Linting |
| @@ -265,10 +265,6 @@ def __init__( | |||
| else: | |||
| # Copy to BytesIO, and ensure no encoding | |||
| contents = filepath_or_buffer.read() | |||
| try: | |||
| contents = contents.encode(self._encoding) | |||
|
Thanks @tswast |
With GCSFS (and when a binary file buffer is passed in), the output from
the file read is
bytes, not a Unicodestr. Theencodecall is unnecessaryin this case.
black pandasgit diff upstream/master -u -- "*.py" | flake8 --diff