problems with unicode characters in pytester stdout

[Sup3rGeo]

Normal case

I have this very simple test which is just logging some unicode character:

test_log.py

import logging

logger = logging.getLogger(__name__)

def test():
    logger.info("Normal")
    logger.info("\u251c")
    logger.info("Also normal")

And when I run it using --log-cli-level=INFO then I get this terminal output, which is fine:

============================= test session starts =============================
platform win32 -- Python 3.6.3, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- c:\users\victor\desktop\pytest unicode bug\venv\scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\Victor\Desktop\pytest unicode bug, inifile:
collected 1 item                                                               

test_log.py::test
-------------------------------- live log call --------------------------------
test_log.py                  6 INFO     Normal
test_log.py                  7 INFO     ├
test_log.py                  8 INFO     Also normal
PASSED                                                                   [100%]

========================== 1 passed in 0.02 seconds ===========================

stdout using pytester plugins

Another problem is when I try to run the same test in a subprocess using pytester plugin. Here is my test (I know files might not get properly closed but just as a quick snippet):

test_log_with_pytester.py

with open("test_log.py", "r") as f:
    test = f.read()

f_stderr = open("stderr", "w")
f_stdout = open("stdout", "w")

def test_with_logs(testdir):
    testdir.makepyfile(test_reference=test)
    result = testdir.runpytest_subprocess("--log-cli-level=INFO")

    f_stdout.write(result.stdout.str())
    f_stdout.close()
    f_stderr.write(result.stderr.str())
    f_stderr.close()

pytester stdout saved to file:

============================= test session starts =============================
platform win32 -- Python 3.6.3, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- c:\users\victor\desktop\pytest unicode bug\venv\scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\Victor\AppData\Local\Temp\pytest-of-Victor\pytest-77\test_with_logs0, inifile:
collecting ... collected 1 item

test_reference.py::test 
-------------------------------- live log call --------------------------------
test_reference.py            6 INFO     Normal
test_reference.py            7 INFO     \u251c
test_reference.py            8 INFO     Also normal
PASSED                                                                   [100%]

========================== 1 passed in 0.03 seconds ===========================

In this case, instead of the unicode character, I get literally the code point in the output.

Environment details

• pytest 3.6.2
• Windows 10

[pytestbot]

GitMate.io thinks possibly related issues are #3630 (problems with unicode characters when saving log to file), #1131 (Experiencing unicode error inside pytester in Hypothesis test suite), #531 (Problems with fixture parametrization), #2258 (Unicode issues with pytest.fail), and #412 (crash on bad stdout).

[marcelotrevisani]

Is this related to the #3630?
Did your MR fix that already? #3641

[Sup3rGeo]

Is this related to the #3630?

No I don't think it is. I investigated the problem a little bit more and I believe this is something related to the encoding settings of the terminal used to run pytest. Turns out that setting the environment variable PYTHONIOENCODING=utf-8 makes it work just fine here on my machine.

Not sure if this is the best solution though.

I am closing this issue as this does not appear to be something related to pytest specifically.