enable ANSI color (instead of ANSI color codes) in the Windows terminal #4393 #4403

akshaysharmajs · 2020-03-04T20:38:38Z

modified scrapy/utils/display.py to stop ANSI color sequences in the Windows terminal (issue #4393)

)

…Windows terminal (issue scrapy#4393)

codecov · 2020-03-04T20:55:43Z

Codecov Report

Merging #4403 into master will increase coverage by 0.34%.
The diff coverage is 86.66%.

@@            Coverage Diff             @@
##           master    #4403      +/-   ##
==========================================
+ Coverage   84.90%   85.25%   +0.34%     
==========================================
  Files         162      162              
  Lines        9748     9760      +12     
  Branches     1437     1439       +2     
==========================================
+ Hits         8277     8321      +44     
+ Misses       1211     1178      -33     
- Partials      260      261       +1

Impacted Files	Coverage Δ
scrapy/utils/display.py	`92.85% <86.66%> (+61.60%)`	⬆️
scrapy/core/downloader/contextfactory.py	`90.00% <0.00%> (-6.67%)`	⬇️
scrapy/utils/trackref.py	`82.85% <0.00%> (-2.86%)`	⬇️
scrapy/utils/spider.py	`77.77% <0.00%> (+11.11%)`	⬆️
scrapy/robotstxt.py	`97.53% <0.00%> (+22.22%)`	⬆️
scrapy/utils/py36.py	`100.00% <0.00%> (+80.00%)`	⬆️

scrapy/utils/display.py

wRAR · 2020-03-05T06:38:12Z

Ideally the colorization should stay enabled on the terminals that actually support it, such as the ones from WSL or MSYS, but I suspect we need to check terminfo for that which is a lot of work for no significant gain.

OTOH this PR doesn't allow enabling the colorization explicitly on Windows, maybe the check should be earlier.

akshaysharmajs · 2020-03-05T21:34:01Z

Ideally the colorization should stay enabled on the terminals that actually support it, such as the ones from WSL or MSYS, but I suspect we need to check terminfo for that which is a lot of work for no significant gain.
I completely agree with this.

I just made a read for the problem and found out that Windows 10's console recently began interpreting escapes, but only with ENABLE_VIRTUAL_TERMINAL_PROCESSING which is off by default. So i tried enabling it and i think it works just fine.

wRAR · 2020-03-11T18:22:02Z

I wonder if this new code works on pre-Windows 10 terminals? And again, what about non-cmd.exe terminals?

And I'd want some error handling there.

akshaysharmajs · 2020-03-11T20:29:24Z

I wonder if this new code works on pre-Windows 10 terminals?

I just added a check for the windows 10 release and for specific version 10.0.14393 as the ANSI color codes problem only persist after this release of windows. So, there ll' be no problem for pre-windows 10 terminals. Also, ENABLE_VIRTUAL_TERMINAL_PROCESSING flag is only available in windows 10 and up.

akshaysharmajs · 2020-03-11T20:32:55Z

what about non-cmd.exe terminals

For unix terminals running on windows its working fine according to me, @wRAR Please check it once.
WINDOWS CMD.EXE

CYGWIN TERMINAL (RUNNING ON WINDOWS)

Gallaecio · 2020-03-12T10:39:40Z

Just for reference, Django’s implementation: https://github.com/django/django/blob/master/django/core/management/color.py#L12

akshaysharmajs · 2020-03-23T20:45:34Z

@wRAR Is there anything that concerns you about the changes that I have made?
or what further changes can be done?

Gallaecio · 2020-03-24T07:07:55Z

Django’s implementation seems less prone to issues to me, as it does not rely on specific Windows versions. Could you check if it also works for us?

wRAR · 2020-03-24T09:59:50Z

@akshaysharmajs my point about error handling is still valid.

akshaysharmajs · 2020-03-24T21:50:59Z

@wRAR @Gallaecio thanks for reply. I have added some error handling (used django's implementation for reference).
windows versions check is still there as windows has no real support for ANSI escape sequences.

akshaysharmajs · 2020-06-06T15:26:28Z

@Gallaecio @elacuesta Please go through the changes. I have used only normal terminal formatter for all platforms and added a separate test for windows by mocking ctypes.

akshaysharmajs · 2020-06-06T15:27:53Z

I figured out that GetStdhandle() is giving wrong handle during test I don't know why?. I tried passing actual handle to SetConsoleMode() then test is passing fine.

This test is passing fine as GetStdHandle() giving valid handle.

Gallaecio · 2020-06-09T19:43:30Z

Could you work on 100% coverage of your changes? (see https://codecov.io/gh/scrapy/scrapy/compare/c86a1035dd9b8b10acaf8f9e8bdb1b5494a287e2...91bf06c48724a844886d1514e9c49b0ff3bb5a68/diff)

akshaysharmajs · 2020-06-10T18:05:54Z

Okay, I am working on it.

akshaysharmajs · 2020-06-12T09:59:30Z

scrapy/utils/display.py

+    if sys.platform == "win32" and parse_version(version()) >= parse_version("10.0.14393"):
+        try:
+            import ctypes
+            kernel32 = ctypes.windll.kernel32
+            handle = kernel32.GetStdHandle(-11)
+            # set `ENABLE_VIRTUAL_TERMINAL_PROCESSING` flag
+            if not kernel32.SetConsoleMode(handle, 7):
+                raise ValueError
+        except ValueError:
+            return text


@Gallaecio please suggest how should I cover this part. I tried mocking windows platform but I think parse_version is still failing.

Could you please push the test code that fails, so that I can have a look?

I have pushed the test code.Please have a look, windows mocking part is failing giving this error AttributeError: <module 'ctypes' from '/usr/lib/python3.6/ctypes/__init__.py'> does not have the attribute 'windll'

Gallaecio · 2020-06-16T19:04:51Z

Please, have a look at akshaysharmajs#1

It’s a refactoring proposal, where separate methods are used in the color-handling implementation for easier testing and better code readability.

Note that I removed the curses stuff, because I still don’t understand why it is needed, and fear it may result in terminals supporting colors not displaying them just because curses is not installed. But I may be wrong here.

Refactor output color handling

akshaysharmajs · 2020-06-16T19:55:24Z

@Gallaecio thanks for the changes, looks perfect and precise to me. I see removing curses is completely fine and also helping in testing.
I don't know if we are fine with windows versions < 10.0.14393 with no colors?

Gallaecio · 2020-06-17T14:41:52Z

I don't know if we are fine with windows versions < 10.0.14393 with no colors?

I thought those Windows versions would not support color.

Is that what curses was for? Do older Windows versions correctly parse ANSI color codes if curses is installed?

If so, maybe it makes sense to add back the curses code (but only for those old Windows versions?).

—

On a separate note, my refactoring broke the Flake8 checks, as well as Python 3.5 tests through the use of f-strings (not the first time I forgot Scrapy cannot use them yet 🤦 ).

akshaysharmajs · 2020-06-17T19:43:14Z

curses was only for different formatters for different platform. but now we are only using normal formatter so there is no use of it.

Is that what curses was for? Do older Windows versions correctly parse ANSI color codes if curses is installed?

Older versions of windows also interpret ansi escape codes as terminal processing is enabled by default and in newer versions we have to enable it separately.

akshaysharmajs · 2020-06-17T19:44:24Z

On a separate note, my refactoring broke the Flake8 checks, as well as Python 3.5 tests through the use of f-strings (not the first time I forgot Scrapy cannot use them yet 🤦 ).

😄😄 no problem I will fix it.

Gallaecio · 2020-06-18T09:54:04Z

Older versions of windows also interpret ansi escape codes as terminal processing is enabled by default and in newer versions we have to enable it separately.

Oh, I had no idea. I see you’ve already fixed the logic accordingly, so 👍.

akshaysharmajs · 2020-06-18T10:23:07Z

Yes I have fixed it accordingly. 😃
Thanks @Gallaecio

akshaysharmajs · 2020-06-29T20:26:15Z

Please suggest, what else can be done here??

…o Scrapychanges

wRAR · 2020-07-20T12:23:31Z

Thank you!

akshaysharmajs and others added 8 commits February 16, 2020 23:47

changed ie. -> i.e.(spelling error) on lines 667, 763 (issue scrapy#4332

58dce1a

)

updated all text files for issue scrapy#4332 (ie. -> i.e.)

7000eaf

Apply ie. → i.e. in source comments

c9b0871

ie → e.g.

aaa1c93

Merge branch 'master' of https://github.com/AKSHAYSHARMAJS/scrapy

e06bf3e

modified scrapy/utils/display.py to stop ANSI color sequences in the …

5f2ad45

…Windows terminal (issue scrapy#4393)

Merge branch 'master' of https://github.com/scrapy/scrapy

5f7c980

modified scrapy/utils/display.py to stop ANSI color sequences in the …

4d5646b

…Windows terminal (issue scrapy#4393)

wRAR reviewed Mar 5, 2020

View reviewed changes

scrapy/utils/display.py Outdated Show resolved Hide resolved

enabled virtual terminal processing (pr scrapy#4403)

0415173

akshaysharmajs changed the title ~~stop ANSI color sequences in the Windows terminal #4393~~ enable ANSI color sequences in the Windows terminal #4393 Mar 8, 2020

akshaysharmajs changed the title ~~enable ANSI color sequences in the Windows terminal #4393~~ enable ANSI color (instead of ANSI color codes) in the Windows terminal #4393 Mar 8, 2020

check for specific windows 10 version (pr scrapy#4403)

1cc1b52

fixing flake-8 test (pr scrapy#4403)

a87fc0e

added error handling for terminal info (pr scrapy#4403)

2d416d7

akshaysharmajs added 4 commits March 25, 2020 03:28

corrected stderr (pr scrapy#4403)

34f16f0

changed orientation, removed unwanted spaces (pr scrapy#4403)

825210f

no need for style variable (pr scrapy#4403)

8798172

fixing trailing whitespaces

75abbb3

akshaysharmajs added 2 commits June 5, 2020 16:48

all curses under try block

fd4933d

added global TestStr

91bf06c

more test added

3de0517

akshaysharmajs commented Jun 12, 2020

View reviewed changes

akshaysharmajs and others added 5 commits June 12, 2020 19:42

small fix

980e79a

covering exceptions

3cf06e6

windows test failing

038e46f

Refactor output color handling

4cffd1d

Fix pprint test

c17287c

Merge pull request #1 from Gallaecio/windows-colors

662f454

Refactor output color handling

fix flake8

a3051fb

Gallaecio approved these changes Jun 18, 2020

View reviewed changes

akshaysharmajs requested a review from elacuesta June 19, 2020 16:18

Merge branch 'master' of https://github.com/AKSHAYSHARMAJS/scrapy int…

4e291d1

…o Scrapychanges

wRAR approved these changes Jul 20, 2020

View reviewed changes

wRAR merged commit de297a3 into scrapy:master Jul 20, 2020

akshaysharmajs deleted the Scrapychanges branch July 24, 2020 15:48

enable ANSI color (instead of ANSI color codes) in the Windows terminal #4393 #4403

enable ANSI color (instead of ANSI color codes) in the Windows terminal #4393 #4403

Uh oh!

Conversation

akshaysharmajs commented Mar 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Mar 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

wRAR commented Mar 5, 2020

Uh oh!

akshaysharmajs commented Mar 5, 2020

Uh oh!

wRAR commented Mar 11, 2020

Uh oh!

akshaysharmajs commented Mar 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akshaysharmajs commented Mar 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Gallaecio commented Mar 12, 2020

Uh oh!

akshaysharmajs commented Mar 23, 2020

Uh oh!

Gallaecio commented Mar 24, 2020

Uh oh!

wRAR commented Mar 24, 2020

Uh oh!

akshaysharmajs commented Mar 24, 2020

Uh oh!

akshaysharmajs commented Jun 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akshaysharmajs commented Jun 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Gallaecio commented Jun 9, 2020

Uh oh!

akshaysharmajs commented Jun 10, 2020

Uh oh!

akshaysharmajs Jun 12, 2020

Choose a reason for hiding this comment

Uh oh!

Gallaecio Jun 14, 2020

Choose a reason for hiding this comment

Uh oh!

akshaysharmajs Jun 15, 2020

Choose a reason for hiding this comment

Uh oh!

Gallaecio commented Jun 16, 2020

Uh oh!

akshaysharmajs commented Jun 16, 2020

Uh oh!

Gallaecio commented Jun 17, 2020

Uh oh!

akshaysharmajs commented Jun 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akshaysharmajs commented Jun 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Gallaecio commented Jun 18, 2020

Uh oh!

akshaysharmajs commented Jun 18, 2020

Uh oh!

akshaysharmajs commented Jun 29, 2020

Uh oh!

wRAR commented Jul 20, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

akshaysharmajs commented Mar 4, 2020 •

edited

Loading

codecov bot commented Mar 4, 2020 •

edited

Loading

akshaysharmajs commented Mar 11, 2020 •

edited

Loading

akshaysharmajs commented Mar 11, 2020 •

edited

Loading

akshaysharmajs commented Jun 6, 2020 •

edited

Loading

akshaysharmajs commented Jun 6, 2020 •

edited

Loading

akshaysharmajs commented Jun 17, 2020 •

edited

Loading

akshaysharmajs commented Jun 17, 2020 •

edited

Loading