Make std::os::unix::ffi::OsStrExt generally available

[mitsuhiko]

Is there a specific reason why std::os::unix::ffi::OsStrExt is unavailable on Windows? It's equally useful there.

[Diggsey]

OsStrings are not byte arrays on windows.

[kennytm]

The Windows one is like

pub trait OsStringExt {
    fn from_wide(wide: &[u16]) -> Self;
}

pub trait OsStrExt {
    fn encode_wide(&self) -> EncodeWide;
}

[mitsuhiko]

@kennytm i am aware but the "i have a byte array here and i want an os-str" operation is also common on windows because there are lots of utf-8 based APIs from unix that are also used on windows. Not everything is a windows str style api.

@Diggsey how so? It's a Wtf8 buffer which is just a fancy wrapper around a u8 vec/slice.

[kennytm]

@mitsuhiko

If the byte array is UTF-8 you could convert it to a String / &str as an intermediate step.

OsStr::new(std::str::from_utf8(stuff).unwrap())

[Diggsey]

@mitsuhiko Wtf8 has additional constraints beyond being a u8 slice, but more importantly, it's a private implementation detail of the standard library. The Wtf8 encoding was never intended to be exposed.

If you need to convert to a utf8 buffer, go via &str. If you need to serialize it, go via encode_wide.

[mitsuhiko]

@Diggsey in that case I assume what is missing is an API that bridges the differences better.

In my case I'm dealing with a binary file format that contains paths in "bytes" without specifics about what format it uses. So I ended up writing something like this:

use std::ffi::{CStr, OsStr};
#[cfg(unix)] use std::os::unix::ffi::OsStrExt;

#[cfg(unix)]
fn cstr_to_path(s: &CStr) -> Option<&Path> {
    Path::new(OsStr::from_bytes(s.to_bytes()))
}

#[cfg(windows)]
fn cstr_to_path(s: &CStr) -> Option<&Path> {
    s.map(|s| Path::new(OsStr::new(s)))
}

The reason I cannot go through str on unix is that there is no guarantee that the path is utf-8 an forcing it through a str will destroy it. However on windows that's the best I can do.

[kennytm]

This is better handled via a new cross-platform method e.g.

impl OsStr {
    fn from_bytes_checked(_: &[u8]) -> Result<&OsStr, Utf8Error>;
}

If you port OsStrExt::from_bytes to Windows, you need to consider what to do when the input contains non-UTF-8 bytes like b"\xff\xff\xff", and the current signature -> &Self means "panic" is the only possible solution, and it is a bad solution for a library.

I think this should be written as an RFC.

[Diggsey]

In that case the best course of action is going to depend very heavily on context, such as how the file format is used: there's no universal solution that the stdlib can provide.

For example, if this is an existing file format, there may already be a convention for how to convert between a byte array and a platform-specific path. In that case you should try to emulate the existing behaviour as closely as possible.

Alternatively, if a primary function of this file format is to be shared, it might make more sense to place a hard restriction on paths to be utf8 - that way taking the file from one system and transferring it to another will work more often.

Finally, if the file is rarely shared, you might be better off tagging the data with the format, and storing [u8] for unix, and [u16] for windows, and determining what to do when it comes to load the file how to handle the differences.

[mitsuhiko]

The stdlib is already making a lot of decisions and providing convenience methods. I do not quite see why this should not be one of them. This does not only come up with file formats (though it is very common there) but really any API that decides to use utf-8 as path names. For instance anything based on glib will uses utf-8 for filename encoding on windows.

This is particularly relevant going forward as std::filesystem::u8path means utf-8 paths appear on windows now.

[Diggsey]

@mitsuhiko The documentation for std::filesystem::u8path says "Constructs a path p from a UTF-8 encoded sequence of chars" - we already have methods to do that!

This is not about whether a convenience method should be provided or not, because there isn't an unambiguous way to do this conversion - it's inherently lossy and/or non-portable.

Paths on unix are [u8]
Paths on windows are [u16]

If you need to convert all paths to a [u8], you need to know more context, or you will likely do the wrong thing.

[retep998]

This is particularly relevant going forward as std::filesystem::u8path means utf-8 paths appear on windows now.

Just because C++ finally exposes a sane way to create a path from UTF-8 does not mean that Windows internally uses UTF-8. Also, this isn't going to change the miles of crappy C and C++ code out there that uses narrow C/C++ standard library functions to work with files, which use some arbitrary narrow encoding which is never UTF-8 on Windows.

If your bytes are UTF-8, great, you can already trivially create paths from UTF-8 Rust strings. If your bytes are not UTF-8 then you need to know what format the bytes are in and convert them to UTF-8. If you don't know what format they are then there is nothing sane you can do, because Windows does not support arbitrary byte sequences for path. The narrow versions of Windows functions such as CreateFileA are always set to some arbitrary encoding which is never UTF-8, and changes based on what locale your system is set to, so the odds of it matching up with your binary file format's string encoding is very slim.

A lot of work went into the current path API, and while it still has some flaws, the inability to specify arbitrary bytes as a path on Windows is fully intentional.

[mitsuhiko]

Just because C++ finally exposes a sane way to create a path from UTF-8 does not mean that Windows internally uses UTF-8.

I never implied that this is the case. However utf-8 filesystem paths show up on windows just as much as on other platforms.

If your bytes are UTF-8, great, you can already trivially create paths from UTF-8 Rust strings. If your bytes are not UTF-8 then you need to know what format the bytes are in and convert them to UTF-8.

This is where you are not completely correct because on Linux there is absolutely no chance to assume an FS encoding. Which is why at least for many situations on Linux you do want to assume arbitrary bytes rather than utf-8 strings. Which is why I end up making different code paths for Windows and Unix around the unicode path issue currently. Since I keep doing this in different applications over and over I raised the issue why there is no api to help with this.

says "Constructs a path p from a UTF-8 encoded sequence of chars" - we already have methods to do that!

I am not disputing this, I am however pointing out that the current API is insufficient for handling the common case for Unix and Windows alike. Effectively it means that code that wants to support the platform as best as possible currently needs to take different paths on Unix and Windows in common situations.

Here is my understanding of the situation:

• paths on UNIX are best effort UTF-8 but essentially bytes
• paths on Windows are "a form of unicode"
• common behavior on Unix is to pass filenames as byte strings that might or might not be utf-8
• a lot of file formats us bytes to represent filenames the encoding of which is usually undefined or assumed to be utf-8

Rust currently assumes these conversions take place:

• on Windows: UCS2 -> Wtf8 (inside OsStr; cannot fail) -> UCS2
• on Unix: bytes -> bytes (with an utf-8 view; cannot fail) inside OsStr -> bytes

I generally agree that there is no reason to ever look at strings as UCS2 on Unix. There are no common APIs that deal with this. However bytes do come up on Windows. So what I am looking for myself is an API that provides this:

• on Windows: bytes -> Wtf8 (if it's utf-8 compatible; can fail) -> UCS2
• on Unix: bytes -> bytes (with an utf-8 view; cannot fail) inside OsStr -> bytes

The situation comes up in a wide range of situations:

• informal filenames (for instance file uploads in web applications)
• filenames inside binary file formats (DWARF files, etc.)

Given the fact that byte strings for filenames are generally iffy on Windows the best path to success there is to attempt to decode them from utf-8 or if that does not work, fail. This is a method that generally has been used where filenames are transported in environments that cannot guarantee an encoding. Common systems that have this issue are file uploads in web applications for instance as well as cross platform binary file formats and more.

In Windows obviously if the OsStr type internally cannot deal with arbitrary malformed utf-8 (or should not) a way that can work is to restrict it to utf-8 as mentioned. That's effectively what I am doing currently. However for non windows platforms I do want to support paths as potentially random bytes.

[retep998]

Existing issues: #27657 #27390 #26730 rust-lang/rfcs#1255