pip issues UnicodeDecodeError on Windows 10 for Russian language

[xtipacko]

• Pip version: 9.0.1
• Python version: 3.6.0
• Operating system: Microsoft Windows 10 Home Edition [Version 10.0.10586] for Russian language

Description:

pip issues UnicodeDecodeError on byte 0x8d in Windows 10 for Russian language.
It is not a problem for Windows 7 Ultimate SP1 for English language.
Probably has something to do with default CMD encoding, please fix it.

What I've run:

C:\WINDOWS\system32>pip install pyyaml
Collecting pyyaml
  Using cached PyYAML-3.12.tar.gz
Building wheels for collected packages: pyyaml
  Running setup.py bdist_wheel for pyyaml ... error
  Failed building wheel for pyyaml
  Running setup.py clean for pyyaml
Failed to build pyyaml
Installing collected packages: pyyaml
  Running setup.py install for pyyaml ... error
Exception:
Traceback (most recent call last):
  File "c:\program files (x86)\python36-32\lib\site-packages\pip\compat\__init__.py", line 73, in console_to_str
    return s.decode(sys.__stdout__.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8d in position 68: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\program files (x86)\python36-32\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "c:\program files (x86)\python36-32\lib\site-packages\pip\commands\install.py", line 342, in run
    prefix=options.prefix_path,
  File "c:\program files (x86)\python36-32\lib\site-packages\pip\req\req_set.py", line 784, in install
    **kwargs
  File "c:\program files (x86)\python36-32\lib\site-packages\pip\req\req_install.py", line 878, in install
    spinner=spinner,
  File "c:\program files (x86)\python36-32\lib\site-packages\pip\utils\__init__.py", line 676, in call_subprocess
    line = console_to_str(proc.stdout.readline())
  File "c:\program files (x86)\python36-32\lib\site-packages\pip\compat\__init__.py", line 75, in console_to_str
    return s.decode('utf_8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8d in position 68: invalid start byte

[pfmoore]

This is likely due to the fact that on Windows Python 3.6 switched to using UTF-8 for console IO. The code is running a subprocess, and then guessing the encoding of the subprocess output as being the same as the encoding of sys.stdout - which was true in Python <3.6 (arguably more by luck than anything else) but is no longer true in 3.6+

The simplest fix is probably to use locale.getpreferredencoding(False) for the encoding, as that's the default encoding used in io.TextIOWrapper and for subprocess when universal_newlines is True.

[xtipacko]

I thought pip is suposed to be easy for users, is it possible to hide this problems from us ?:)

[pfmoore]

Encodings are not easy for anyone :-) It's certainly possible to deal with this as I said. Just the first time it's come up (it's a Python 3.6 change).

[xtipacko]

I've easily reproduced the problem on my VM with Windows 6.1.7601 (win7 SP1 Russian),
cmd utility: chcp shows up 866,
chcp 65001(UTF-8) - doesn't help

[robinxb]

Add a solution here:
run a new cmd.exe console
chcp
it will show the system default code, for example 936.
open Lib/site-package/pip/compat/__init__.py
around 75 line, change return s.decode('utf_8') to return s.decode('cp936')

It's just a workaround. I think pip might need solve this issue asap, it's not easy to find solution.

This may have a general solution using cdll.
Not sure if this is the best solution on windows but I still made a PR for this issue.

[xtipacko]

actually it was easier to use easy_install as workaround...

[dstufft]

Closing as a duplicate of #4110.