Fix the display of utf8 characters in the code editor

[AltGr]

Work done by @baloone during his internship at OCamlPro. This fixes the display with invalid syntax (display in strings and comments was already OK), but it shouldn't get scrambled either way.

NOTE: should be rebased and squashed once the 4.12 PR is merged

[AltGr]

Rebased

[erikmd]

Thanks a lot @AltGr ! I was precisely about to ask this to you 🙂

[erikmd]

(Let's just wait for the CI though :)

[erikmd]

Hi again!
so the CI if fine, the code is tidy, and to experiment with some UTF-8 chars a bit (given this display issue was definitely something my students were stumbling on) I've Docker-built the PR branch (§) to get this toy example:

(§) docker build -t test-412 . && docker run --rm -it -p 8080:8080 -v "$PWD/demo-repository:/repository" test-412

so thanks a lot for this enhancement, @baloone and @AltGr 👍

(As an aside, I've did more tests with a fresh opam switch as well, and noticed a small issue for which I've found a workaround after investigation, but which happens to be orthogonal to this PR. So, merging — and opening another ticket subsequently.)

[baloone]

Hello @erikmd, Thanks for the merge!

I think using smiley inside string has always worked fine, this fixes the display of Unicode characters outside of comments or string i.e. when the ocaml lexer throws an "illegal character" error.