UTF-16 console support for Windows #15051

retep998 · 2014-06-20T07:05:24Z

This implementation does have the minor issue of not handling things correctly when a codepoint is split across multiple writes or reads, but its better than not having unicode support at all.

Adds a Windows specific struct WindowsTTY in libnative and make tty_open create that struct on Windows. Adds needed functions and constants to c_win32.rs.

Libuv still needs to be updated before #15028 can be closed.

alexcrichton · 2014-06-20T16:25:19Z

src/liblibc/lib.rs

+                                     lpBuffer: LPCVOID,
+                                     nNumberOfCharsToWrite: DWORD,
+                                     lpNumberOfCharsWritten: LPDWORD,
+                                     lpReserved: LPVOID) -> BOOL;


We're trying to keep things out of liblibc nowadays, could you add these to c_win32.rs instead?

(inside of libnative)

alexcrichton · 2014-06-20T16:33:21Z

It would be nice to update libuv at the same time. If you open a PR against the rust-lang/libuv repo with the relevant libuv commit I'll push it so you can update the submodule.

alexcrichton · 2014-06-20T16:36:09Z

src/libnative/io/mod.rs

+    fn tty_open(&mut self, fd: c_int, _readable: bool)
+                -> IoResult<Box<rtio::RtioTTY + Send>> {
+        use ERROR = libc::ERROR_INVALID_HANDLE;
+        if unsafe { libc::isatty(fd) } != 0 {


If this is a windows-specific implementation, it should probably use the windows-specific function GetConsoleMode

GetConsoleMode merely tells us the set of flags that we've set for the console, not whether it is or isn't a TTY. isatty does the job fine at telling whether we're attached to a TTY or just redirected to a file.

It looks like it suffices for libuv, and I tend to not trust posix functions with file descriptors on windows, but if it's working well I guess it's working well.

retep998 · 2014-06-20T23:59:08Z

No, setting the code page to UTF-8 does not work. I have already tested that option extensively in C++. The Windows console only supports narrow byte encodings and UTF-16.

retep998 · 2014-06-21T01:10:36Z

Ah, so output with UTF-8 does indeed work, but I've used both SetConsoleCP and SetConsoleOutputCP and input with UTF-8 simply does not work. It just ignores the entire line of text I entered if there's any unicode in it.

retep998 · 2014-06-21T01:38:42Z

Even still, when using UTF-8, Windows will often act buggy. For some examples:

http://social.msdn.microsoft.com/Forums/vstudio/en-US/e4b91f49-6f60-4ffe-887a-e18e39250905/possible-bugs-in-writefile-and-crt-unicode-issues?forum=vcgeneral

It is probably safer to simply use UTF-16 because that is what Windows is built around.

retep998 · 2014-06-21T02:28:54Z

Added another commit to take into account comments by @alexcrichton

alexcrichton · 2014-06-22T14:04:30Z

src/libnative/io/mod.rs

+        } else {
+            // If its not a tty, then it was redirected to a file
+            // So just roll with it instead of erroring
+            Ok(box file::FileDesc::new(fd, true) as Box<rtio::RtioTTY + Send>)


This should not happen here. It's the responsibility of the caller to fall back to a file descriptor. This should have the same error path as the unix equivalent.

alexcrichton · 2014-06-30T00:36:29Z

Oh dear, sorry for being so late to this! Feel free to ping a PR whenever you update this because sadly github doesn't send out notifications when that happens.

retep998 · 2014-06-30T08:10:27Z

The bors build failed due to the following error:

note: i686-pc-mingw32\rt\uv.lib(core.o): In function `uv_process_reqs':
c:\bot\slave\auto-win-32-opt\build\src\libuv/src/win/req-inl.h:203: undefined reference to `uv_process_getnameinfo_req'

A fix for this is already upstream in libuv: joyent/libuv@6de622c

I have created a pull request that updates rust's libuv: rust-lang-deprecated/libuv#3

Also, rebased and squashed my changes onto the latest rust master and verified that it works.

Make sure you update the libuv repo before doing an r+ or the travis build will fail as it is now.

alexcrichton · 2014-06-30T14:20:44Z

Thanks again!

retep998 · 2014-06-30T16:51:56Z

This is the point where I break down and cry because the test that failed on the windows botbuild passes just fine for me, so I can't even replicate the error in an attempt to fix it.

EDIT: And Travis doesn't feel like working either because building librustc takes more than 10 minutes.

retep998 · 2014-06-30T21:14:53Z

After doing a fresh clone of my fork and rebuilding, I still cannot reproduce the error on my end.
test io::net::tcp::test::fast_rebind::green ... ok
For all I know this problem could be specific to certain versions of Windows. I'm using Windows 8.1 x64. If other people could report back on their OS and whether or not my changes cause any test failures, that would be great.

~~However, I do have some other failures, but they appear in both my fork and a clean version of rust, so I'm not sure if they are related.~~ Nevermind, these failures were due to me not having ping in my path.

failures:
    io::process::tests::forget::green
    io::process::tests::test_add_to_env::green
    io::process::tests::test_change_working_directory::green
    io::process::tests::test_exists::green
    io::process::tests::test_inherit_env::green
    io::process::tests::test_keep_current_working_dir::green
    io::process::tests::test_kill::green
    io::process::tests::test_zero::green
    io::process::tests::wait_timeout2::green
    io::process::tests::wait_timeout::green

retep998 · 2014-07-01T10:40:12Z

I updated the uv_req_type enums in uvll.rs. It fixes the test failure I encountered on linux. After fixing my path, the windows tests also pass for me. If someone with windows 7 could run make check-fast on my changes to determine whether the fast_rebind::green test still fails, that would be wonderful.

alexcrichton · 2014-07-01T13:28:10Z

I updated the uv_req_type enums in uvll.rs

Sneaky! Nice catch!

This implementation does have the minor issue of not handling things correctly when a codepoint is split across multiple writes or reads, but its better than not having unicode support at all. Adds a Windows specific struct `WindowsTTY` in `libnative` and make `tty_open` create that struct on Windows. Adds needed functions and constants to `c_win32.rs`. Also updates the libuv submodule. This fixes #15028 for both libnative and libgreen.

retep998 · 2014-07-03T12:53:42Z

One potential reason for why that test is failing is because SO_REUSEADDR isn't set for tcp listeners on windows. joyent/libuv#1352

Adds a WindowsTTY for libnative that converts between UTF-8 and UTF-16. Signed-off-by: Peter Atashian <[email protected]>

This implementation does have the minor issue of not handling things correctly when a codepoint is split across multiple writes or reads, but its better than not having unicode support at all. Adds a Windows specific struct `WindowsTTY` in `libnative` and make `tty_open` create that struct on Windows. Adds needed functions and constants to `c_win32.rs`. Libuv still needs to be updated before #15028 can be closed.

retep998 mentioned this pull request Jun 20, 2014

Unicode input on windows command line does not work #15028

Closed

alexcrichton reviewed Jun 20, 2014
View reviewed changes

retep998 changed the title ~~Crude implementation of Unicode console support for Windows.~~ Implementation of UTF-16 console support for Windows Jun 21, 2014

alexcrichton reviewed Jun 22, 2014
View reviewed changes

retep998 changed the title ~~Implementation of UTF-16 console support for Windows~~ UTF-16 console support for Windows Jun 22, 2014

windows: Unicode console support.

a34fd5a

Adds a WindowsTTY for libnative that converts between UTF-8 and UTF-16. Signed-off-by: Peter Atashian <[email protected]>

bors closed this Jul 4, 2014

bors merged commit a34fd5a into rust-lang:master Jul 4, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UTF-16 console support for Windows #15051

UTF-16 console support for Windows #15051

retep998 commented Jun 20, 2014

alexcrichton Jun 20, 2014

alexcrichton Jun 20, 2014

alexcrichton commented Jun 20, 2014

alexcrichton Jun 20, 2014

retep998 Jun 21, 2014

alexcrichton Jun 21, 2014

retep998 commented Jun 20, 2014

retep998 commented Jun 21, 2014

retep998 commented Jun 21, 2014

retep998 commented Jun 21, 2014

alexcrichton Jun 22, 2014

alexcrichton commented Jun 30, 2014

retep998 commented Jun 30, 2014

alexcrichton commented Jun 30, 2014

retep998 commented Jun 30, 2014

retep998 commented Jun 30, 2014

retep998 commented Jul 1, 2014

alexcrichton commented Jul 1, 2014

retep998 commented Jul 3, 2014

UTF-16 console support for Windows #15051

UTF-16 console support for Windows #15051

Conversation

retep998 commented Jun 20, 2014

alexcrichton Jun 20, 2014

Choose a reason for hiding this comment

alexcrichton Jun 20, 2014

Choose a reason for hiding this comment

alexcrichton commented Jun 20, 2014

alexcrichton Jun 20, 2014

Choose a reason for hiding this comment

retep998 Jun 21, 2014

Choose a reason for hiding this comment

alexcrichton Jun 21, 2014

Choose a reason for hiding this comment

retep998 commented Jun 20, 2014

retep998 commented Jun 21, 2014

retep998 commented Jun 21, 2014

retep998 commented Jun 21, 2014

alexcrichton Jun 22, 2014

Choose a reason for hiding this comment

alexcrichton commented Jun 30, 2014

retep998 commented Jun 30, 2014

alexcrichton commented Jun 30, 2014

retep998 commented Jun 30, 2014

retep998 commented Jun 30, 2014

retep998 commented Jul 1, 2014

alexcrichton commented Jul 1, 2014

retep998 commented Jul 3, 2014