Encoding error : `UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2: invalid continuation byte`

[ghsama]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): ubuntu 18.04
• **Modin installed from : pip install modin[ray]
• Modin version: 0.6.3
• Python version: 3.7.3

Describe the problem

Hello,
i'm trying to use modin to reduce the memory peak due the volum of the data, so i change the pandas with modin.pandas, i try to do a simple read of a file but encoded in 'latin-1' (french) . With pandas all goes smoothly but using modin i got an error of encoding as follow :
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2: invalid continuation byte

the script used (which works fine on pandas but not in modin ) :
caract = pd.read_csv(path, sep="\t", encoding = "ISO-8859-1")

ps :: i tried other encoding and the same remark : works on pandas and not on modin (backed by ray) : ISO-8859-1, ISO-8859-9, latin-1

any solution ??

thanks

Source code / logs

`RayTaskError: ray_worker (pid=10815, host=ubuntu)
File "pandas/_libs/parsers.pyx", line 1297, in pandas._libs.parsers.TextReader._string_convert
File "pandas/_libs/parsers.pyx", line 1520, in pandas._libs.parsers._string_box_utf8
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2: invalid continuation byte

During handling of the above exception, another exception occurred:

ray_worker (pid=10815, host=ubuntu)
File "/home/lasngd/.conda/envs/pytorch/lib/python3.7/site-packages/modin/engines/ray/task_wrapper.py", line 8, in deploy_ray_func
return func(**args)
File "/home/lasngd/.conda/envs/pytorch/lib/python3.7/site-packages/modin/backends/pandas/parsers.py", line 69, in parse
pandas_df = pandas.read_csv(BytesIO(to_read), **kwargs)
File "/home/lasngd/.conda/envs/pytorch/lib/python3.7/site-packages/pandas/io/parsers.py", line 685, in parser_f
return _read(filepath_or_buffer, kwds)
File "/home/lasngd/.conda/envs/pytorch/lib/python3.7/site-packages/pandas/io/parsers.py", line 463, in _read
data = parser.read(nrows)
File "/home/lasngd/.conda/envs/pytorch/lib/python3.7/site-packages/pandas/io/parsers.py", line 1154, in read
ret = self._engine.read(nrows)
File "/home/lasngd/.conda/envs/pytorch/lib/python3.7/site-packages/pandas/io/parsers.py", line 2059, in read
data = self._reader.read(nrows)
File "pandas/_libs/parsers.pyx", line 881, in pandas._libs.parsers.TextReader.read
File "pandas/_libs/parsers.pyx", line 896, in pandas._libs.parsers.TextReader._read_low_memory
File "pandas/_libs/parsers.pyx", line 973, in pandas._libs.parsers.TextReader._read_rows
File "pandas/_libs/parsers.pyx", line 1105, in pandas._libs.parsers.TextReader._convert_column_data
File "pandas/_libs/parsers.pyx", line 1158, in pandas._libs.parsers.TextReader._convert_tokens
File "pandas/_libs/parsers.pyx", line 1281, in pandas._libs.parsers.TextReader._convert_with_dtype
File "pandas/_libs/parsers.pyx", line 1297, in pandas._libs.parsers.TextReader._string_convert
File "pandas/_libs/parsers.pyx", line 1520, in pandas._libs.parsers._string_box_utf8
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2: invalid continuation byte`

[devin-petersohn]

Hi @ghsama, thanks for posting!

Do you mind to share a couple of lines of synthetic or real data so I can reproduce this locally? Locally and in our test suite, we test with various forms of encoding, but I haven't been able to reproduce. If you can share some data that reproduces the issue for you, I will be able to diagnose and fix the issue. Thanks!

[hvardhan20]

System information
OS Platform Windows 10 Home
**Modin installed from : pip install modin[dask]
Modin version: 0.6.3
Python version: 3.7.3

Hi @devin-petersohn,

I am facing a similar issue as @ghsama on windows with modin using dask engine. With vanilla pandas this works just fine:
pd.read_csv('fires_50k.csv', encoding = "ISO-8859-1")
However, while reading CSV with modin.pandas like this:
mpd.read_csv('fires_50k.csv', encoding = "ISO-8859-1")
throws this UnicodeDecodeError:

Traceback (most recent call last):
File "", line 1, in
File "D:\Users\hvard\Anaconda3\lib\site-packages\modin\pandas\io.py", line 97, in parser_func
return _read(**kwargs)
File "D:\Users\hvard\Anaconda3\lib\site-packages\modin\pandas\io.py", line 110, in _read
pd_obj = BaseFactory.read_csv(**kwargs)
File "D:\Users\hvard\Anaconda3\lib\site-packages\modin\data_management\factories.py", line 52, in read_csv
return cls._determine_engine()._read_csv(**kwargs)
File "D:\Users\hvard\Anaconda3\lib\site-packages\modin\data_management\factories.py", line 56, in _read_csv
return cls.io_cls.read_csv(**kwargs)
File "D:\Users\hvard\Anaconda3\lib\site-packages\modin\engines\base\io\text\csv_reader.py", line 197, in read
row_lengths = cls.materialize(index_ids)
File "D:\Users\hvard\Anaconda3\lib\site-packages\modin\engines\dask\task_wrapper.py", line 20, in materialize
return client.gather(future)
File "D:\Users\hvard\Anaconda3\lib\site-packages\distributed\client.py", line 1876, in gather
asynchronous=asynchronous,
File "D:\Users\hvard\Anaconda3\lib\site-packages\distributed\client.py", line 771, in sync
self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
File "D:\Users\hvard\Anaconda3\lib\site-packages\distributed\utils.py", line 334, in sync
raise exc.with_traceback(tb)
File "D:\Users\hvard\Anaconda3\lib\site-packages\distributed\utils.py", line 318, in f
result[0] = yield future
File "D:\Users\hvard\Anaconda3\lib\site-packages\tornado\gen.py", line 1133, in run
value = future.result()
File "D:\Users\hvard\Anaconda3\lib\site-packages\distributed\client.py", line 1732, in _gather
raise exception.with_traceback(traceback)
File "D:\Users\hvard\Anaconda3\lib\site-packages\modin\backends\pandas\parsers.py", line 69, in parse
pandas_df = pandas.read_csv(BytesIO(to_read), **kwargs)
File "D:\Users\hvard\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 685, in parser_f
return _read(filepath_or_buffer, kwds)
File "D:\Users\hvard\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 463, in _read
data = parser.read(nrows)
File "D:\Users\hvard\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 1154, in read
ret = self._engine.read(nrows)
File "D:\Users\hvard\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 2059, in read
data = self._reader.read(nrows)
File "pandas/_libs/parsers.pyx", line 881, in pandas._libs.parsers.TextReader.read
File "pandas/_libs/parsers.pyx", line 896, in pandas._libs.parsers.TextReader._read_low_memory
File "pandas/_libs/parsers.pyx", line 973, in pandas._libs.parsers.TextReader._read_rows
File "pandas/_libs/parsers.pyx", line 1105, in pandas._libs.parsers.TextReader._convert_column_data
File "pandas/_libs/parsers.pyx", line 1158, in pandas._libs.parsers.TextReader._convert_tokens
File "pandas/_libs/parsers.pyx", line 1281, in pandas._libs.parsers.TextReader._convert_with_dtype
File "pandas/_libs/parsers.pyx", line 1297, in pandas._libs.parsers.TextReader._string_convert
File "pandas/_libs/parsers.pyx", line 1520, in pandas._libs.parsers._string_box_utf8
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 8: invalid start byte

This is the CSV I am trying to read. This is a part(50K) of a large 1.88M rows dataset. I think you should be able to reproduce the issue with this data. Please do let me know otherwise.

Thanks!

[ghsama]

Hi @devin-petersohn ,
here is a part of the csv file that causes the bug : csvFile
Thanks

[devin-petersohn]

Thanks @ghsama and @hvardhan20! I was able to reproduce the issue with your files.

In short, this line was the culprit and was excluding the csv reader from seeing the encoding:

https://github.com/modin-project/modin/blob/master/modin/backends/pandas/parsers.py#L79

We should add more testing for a wider variety of encodings because this was missed by our tests.

I will get this resolved, thanks so much for reporting!

[sector119]

MacOS Big Sur 11.2

python 3.8.6

ray = https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-1.2.0.dev0-cp38-cp38-macosx_10_13_x86_64.whl
modin from git = https://github.com/modin-project/modin

Got the same error with windows-1251 encoding. CSV file and traceback are attached.

file.txt

Traceback (most recent call last):
File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/bin/unity", line 5, in
app()
File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/typer/main.py", line 214, in call
return get_command(self)(*args, **kwargs)
File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/typer/main.py", line 497, in wrapper
return callback(**use_params) # type: ignore
File "/Users/sector119/PycharmProjects/epsilon_unity/unity/lib/cmd/commands/db/commands/read.py", line 18, in read
adapter.read(cache)
File "/Users/sector119/PycharmProjects/epsilon/epsilon/lib/profiling.py", line 9, in timed
output = func(*args, **kw)
File "/Users/sector119/PycharmProjects/epsilon_unity/unity/lib/parser/adapter/plugins/base.py", line 87, in read
self.dataframe = source_type.dataframe()
File "/Users/sector119/PycharmProjects/epsilon_unity/unity/lib/parser/type/plugins/csv.py", line 10, in dataframe
df = pd.read_csv(self.filepath,
File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/modin/pandas/io.py", line 115, in parser_func
return _read(**kwargs)
File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/modin/pandas/io.py", line 133, in _read
pd_obj = EngineDispatcher.read_csv(**kwargs)
File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/modin/data_management/factories/dispatcher.py", line 104, in read_csv
return cls.__engine._read_csv(**kwargs)
File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/modin/data_management/factories/factories.py", line 87, in _read_csv
return cls.io_cls.read_csv(**kwargs)
File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/modin/engines/base/io/file_dispatcher.py", line 29, in read
query_compiler = cls._read(*args, **kwargs)
File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/modin/engines/base/io/text/csv_dispatcher.py", line 74, in _read
empty_pd_df = pandas.read_csv(filepath_or_buffer, nrows=0)
File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/pandas/io/parsers.py", line 688, in read_csv
return _read(filepath_or_buffer, kwds)
File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/pandas/io/parsers.py", line 454, in _read
parser = TextFileReader(fp_or_buf, **kwds)
File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/pandas/io/parsers.py", line 948, in init
self._make_engine(self.engine)
File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/pandas/io/parsers.py", line 1180, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/pandas/io/parsers.py", line 2010, in init
self._reader = parsers.TextReader(src, **kwds)
File "pandas/_libs/parsers.pyx", line 537, in pandas._libs.parsers.TextReader.cinit
File "pandas/_libs/parsers.pyx", line 740, in pandas._libs.parsers.TextReader._get_header
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcc in position 32: invalid continuation byte

[Garra1980]

сс @anmyachev

[anmyachev]

@sector119 thanks for posting. I was able to reproduce the error, but only with using names parameter in read_csv call.

Can you share your read_csv call? Maybe there are another details.