Allow surrogate codepoints when escaped with \u

[mnemnion]

Update: the first post (which you're reading) is imprecise in some ways, especially the use of the word "valid". Please skip ahead to this comment for a better take on the aims and goals here.

In generating some Unicode test data, I discovered that Zig doesn't allow surrogates, even when encoded with \u.

Minimal reproduction:

const invalid_str = "\u{d800}";

The error (Zig 0.12.0): "error: unicode escape does not correspond to a valid codepoint".

The error message is not correct, the UTF-16 surrogates are valid codepoints, in category Other: surrogate. Here's a property page for U+D800.

It makes sense to me that the parser finding random surrogates in the string should reject that as not well-formed, just like it would balk on control character, or a bare \xff. Such random garbage is most likely a mistake. The same applies to overlong encodings, you could argue that they're "well encoded" in that they match the necessary pattern for a UTF-8 encoded point, but the standard specifically forbids them. Unlike the surrogates, these are not codepoints.

But Zig does not demand that string data is well formed UTF-8, the \x byte encoding can represent arbitrary bytes within a string. In fact, "\u{01}" is valid, when it would not be if embedded raw in the string.

It doesn't make sense to me that a codepoint specifically encoded as e.g. \u{d800} would create a compile error. That's the author affirmatively requesting that said codepoint be decoded into UTF-8 and added to the sequence, it's not the sort of thing which happens by accident. It has an exact interpretation, .{0xed, 0xao, 0x80}, which is validly-encoded UTF-8. I'll contrast this with \u{80000000}, which can't be turned into bytes: this is genuinely invalid. .{0xc0, 0xaf} is also invalid, despite having the superficial qualities of a UTF-8 codepoint, since it's overlong. There's no way to represent those in U+ notation, so this is a bit off topic, the point is that I could stuff it into a string with \xco\xaf and that would be fine with the compiler.

There's no reason for the compiler to contain extra logic to prevent surrogates from being represented using \u notation, when it's specifically requested. In my case, it means that if I want test data which covers the entire three-byte range of Unicode, I must detect the surrogate ranges and special-case encode them as \x sequences. Or perhaps someone might be writing a fixup tool which detects an invalid encoding of surrogate pairs and produces correct UTF-8. There, too, the test data will have surrogates in it, and this behavior is an arbitrary limitation to work around. TL;DR, the compiler should accept \u notation for all codepoints in Unicode, surrogates included.

[squeek502]

My current understanding is that there is no canonical UTF-8 representation of surrogate code points, hence WTF-8.

From the latest Unicode Standard:

3.8 Surrogates
D71 High-surrogate code point: A Unicode code point in the range U+D800 to U+DBFF.
D72 High-surrogate code unit: A 16-bit code unit in the range D800 16 to DBFF 16, used in UTF-16 as the leading code unit of a surrogate pair.
D73 Low-surrogate code point: A Unicode code point in the range U+DC00 to U+DFFF.
D74 Low-surrogate code unit: A 16-bit code unit in the range DC00 16 to DFFF 16, used in UTF-16 as the trailing code unit of a surrogate pair.
• High-surrogate and low-surrogate code points are designated only for that use.
• High-surrogate and low-surrogate code units are used only in the context of the UTF-16 character encoding form.

3.9 Unicode Encoding Forms
D76 Unicode scalar value: Any Unicode code point except high-surrogate and low-surrogate code points.

UTF-8
• Because surrogate code points are not Unicode scalar values, any UTF-8 byte sequence that would otherwise map to code points U+D800..U+DFFF is ill-formed.

So, the most precise error for current Zig would be unicode escape does not correspond to a valid unicode scalar value, and perhaps this proposal should instead be advocating for \u to encode the code points as WTF-8?

[mnemnion]

Yes, this gets at the distinction between a Unicode character and a Unicode codepoint. The surrogates are not valid Unicode characters, nor the noncharacter codepoints. They are codepoints, however: they have an assigned value, character properties, and a valid (let's say: specific, or: singular) encoding in every Unicode encoding. Scalar value is the modern way the standard uses to refer to characters, because what civilians think of as a character includes Farmer Bob here 👨🏻‍🌾, which is made up of several Unicode scalar values.

The standard Unicode uses for codepoints is U+abcdef, which is reflected in the use of \u style escape sequences in most programming languages. When the standard says "surrogate pairs have no interpretation in UTF-8", that's in contrast to UTF-16, when a pair of surrogates stands for a scalar value from a higher plane.

But they do have an encoding in UTF-8. What the standard is getting it is that it's forbidden to interpret surrogate pairs in UTF-8 as the abstract character / scalar value which they would represent if found in UTF-16. This is invalid for the same reason that overlong encodings are invalid: it serves no purpose, complicates software, and has security implications which are decidedly negative.

All of these are good reason to reject bare surrogates in string data! But they aren't good reasons to treat \u{d800} as a syntax error. If you look at the property page I linked to, it shows the UTF-8 encoding for that codepoint. The page isn't canonical, but it's generated from canonical data. Both the codepoint and its encoding are official parts of Unicode, and UTF-8, respectively.

A string containing these codepoints is ill-formed UTF-8. WTF-8 doesn't have to be brought into the picture, because once turned into binary data, Zig strings aren't WTF-8 either, they allow all sorts of byte sequences which aren't valid in either encoding.

As a sort of Zen koan to convey my meaning: if the surrogates didn't have an encoding in UTF-8, then software wouldn't be able to reject those byte sequences as invalid. That's also the most pragmatic reason I'm advocating for their inclusion in the \u notation: representing invalid data is an important thing to be able to do.

It's invalid to have the surrogate codepoints anywhere in a UTF-8 string, just as it's invalid to have only one in a UT-16 string, or a low followed by a high, but in both cases is is a codepoint and it does have an encoding.

There are all sorts of ways for UTF-8 to be ill-formed, all of them are supported through the \x sequences in Zig strings. Some of those ill-formed sequences represent Unicode codepoints: my case is that we should be able to encode those codepoints using the \u sequence, which exists for codepoints, and which, stripped of the extraneous requirement for well-formedness (which does not exist in Zig strings), would have a one-to-one correspondence to Unicode codepoints.

So we all agree: if the Zig parser is chugging along through a string, and sees these three bytes: 0xed, 0xa0, 0x80, that should of course be a syntax error. But if it sees \u{d800}, it should do what Julia (for an example I'm familiar with) does with \ud800: generate those three bytes for the in-memory representation of the string.

[squeek502]

It would be helpful if you could link to the relevant parts of the Unicode Standard and/or UCD. I'm unable to find anything that specifically talks about/declares a canonical encoding of surrogate code points for UTF-8.

From what I can tell, U+D800 is treated roughly the same as e.g. code point U+110000. Both are technically representable using <= 4 bytes with the normal UTF-8 encoding algorithm (see RFC2279 where encoding up to 7FFF FFFF was allowed), but are now defined by the standard to be unrepresentable/invalid for historical UTF-16 related reasons. It's unclear to me why \u{d800} would be allowed but \u{110000} wouldn't be on that basis (in contrast to defining \u to encode the code point as WTF-8, which would make a clear distinction).

As for my position on this proposal:

• Zig enforcing valid unicode scalar values for \u seems like a reasonable/defensible choice (and the error message should be updated to be more accurate if this behavior is kept)
• Zig allowing surrogate code points in \u escapes and encoding them as WTF-8 seems like a reasonable/defensible choice, but I'm not sure it's defensible using the Unicode Standard, and instead the argument would be about the practicality/use cases (e.g. that this test case and other WTF-8 test cases would be more understandable, that being able to use \u with surrogate code points would make working with WTF-8 paths using string literals nicer, etc)

EDIT: To be clear: I'm largely in favor of this proposal. My only reservation would be the potential for users to accidentally/unexpectedly create a sequence of ill-formed UTF-8, but the risk of that does seem pretty minor.

---

EDIT#2: Some clearer statements:

RFC3629:

The definition of UTF-8 prohibits encoding character numbers between U+D800 and U+DFFF, which are reserved for use with the UTF-16 encoding form (as surrogate pairs) and do not directly represent characters.

Unicode Standard Ch 2.4:

Restricted Interchange. Code points that are not assigned to abstract characters are subject to restrictions in interchange.
• Surrogate code points cannot be conformantly interchanged using Unicode encoding forms. They do not correspond to Unicode scalar values and thus do not have well-formed representations in any Unicode encoding form.

[zzo38]

UTF-8 can also be treated as an encoding of 31-bit numbers rather than of Unicode characters (although if it is a valid Unicode code point number then it is a valid encoding of a Unicode character (or at least a part of one)); and in that interpretation, \u{d800} makes sense as producing the UTF-8 encoding of the number 55926, and this encoding is well-defined (even though there is no Unicode character with code point number 55926). (Similarly, \u{7fffffff} is also valid if it would produce the UTF-8 encoding of the number 2147483647, although of course it should be an error if you tried to include such a code in a UTF-16 string.)

My only reservation would be the potential for users to accidentally/unexpectedly create a sequence of ill-formed UTF-8, but the risk of that does seem pretty minor.

If you are concerned about that, then a (suppressable) warning message might make sense.

[squeek502]

the UTF-8 encoding of the number 55926
the UTF-8 encoding of the number 2147483647

My reading of UTF-8 as defined by RFC3629 and the Unicode Standard is that there are no such encodings, i.e. U+D800-U+DFFF and > U+10FFFF are explicitly disallowed from being encoded.

Hence why I think this proposal should be focused on defining \u as being WTF-8 encoded instead.

(side note: 7FFFFFFF, if legal, would actually be encoded as 6 bytes as detailed in the obsoleted RFC2279; the highest possible value encoded by 4 bytes is 1FFFFF)

(suppressable) warning message

Zig doesn't have warnings

[mnemnion]

It's possible to describe the proposal without reference to WTF-8. I think that's generally valuable, because the goal isn't the same as that of WTF-8. WTF-8 defines a more permissive encoding, whereas Zig strings don't define an encoding at all.

I'm going to use bold and italics here to reference terms defined by, or used in, the standard, and try to draw out some precise language, which will satisfy the desire for the proposal to be a) expressed according to the language of the standard and b) without reference to the separate WTF-8 standard.

The proposal relates to codepoints. Unicode defines U+0 through U+10ffff as codepoints. In Zig, with the exception we're discussing, these may all be represented as \u{0} through \u{10ffff}.

Currently, without said exception, Zig turns these escape sequences into what the standard calls code unit sequences, specifically those of UTF-8. In UTF-8, the surrogate codepoints are not valid code unit sequences, because they don't correspond to a valid abstract character or, equivalently, scalar value. In UTF-16, this is true of mismatched surrogates: low-high, or one or the other encountered without a pair.

Section C8 reads:

When a process interprets a code unit sequence which purports to be in a Unicode character encoding form, it shall interpret that code unit sequence according to the corresponding code point sequence.

So now we introduce code point sequences as well. These are a specific byte pattern representing any codepoint in any encoding. Corollary: every codepoint, without exception, has a code point sequence in all Unicode encodings.

Are we sure that surrogates are codepoints? We are:

D10a Code point type: Any of the seven fundamental classes of code points in the standard: Graphic, Format, Control, Private-Use, Surrogate, Noncharacter, Reserved.

Also, D15:

Surrogate code points and noncharacters are considered assigned code points, but not assigned characters.

And we have this:

D84 Ill-formed: A Unicode code unit sequence that purports to be in a Unicode encoding form is called ill-formed if and only if it does not follow the specification of that Unicode encoding form.

Any code unit sequence that would correspond to a code point outside the defined range of Unicode scalar values would, for example, be ill-formed.

Defining a scalar value thus:

D76 Unicode scalar value: Any Unicode code point except high-surrogate and low-surrogate code points.

As a result of this definition, the set of Unicode scalar values consists of the ranges 0 to D7FF₁₆ and E000₁₆ to 10FFFF₁₆, inclusive.

Referring to UTF-8 specifically, the standard dismisses surrogates as follows:

Because surrogate code points are not Unicode scalar values, any UTF-8 byte sequence that would otherwise map to code points U+D800..U+DFFF is ill-formed.

So here's the synthesis: in order to reject ill-formed values as defined by D84, a UTF-8 decoder first recognizes that code point sequence, as it is directed to do by C8, then rejects it as an invalid code unit sequence.

The proposal

Current behavior: Zig interprets a sequence \u{xxxxxx} not (as the documentation says) as a "hexadecimal Unicode code point UTF-8 encoded (1 or more digits)", but rather as a hexadecimal Unicode scalar value, and translates escape sequences for which that interpretation is valid into UTF-8 code unit sequences.

Proposed behavior: Zig interprets a sequence \u{xxxxx} as corresponding to a Unicode codepoint, and generates the corresponding UTF-8 code point sequence, without reference to its validity as UTF-8.

I believe that here we have a succinct description of the proposal, which uses terminology in the way in which the Unicode standard uses it.

[mnemnion]

Small addendum: this proposal, as expressed above, rejects sequences such as \u{110000} and higher values.

While one can follow the rules elucidated in the FSS UTF-8 standard of 1993, and turn that into bytes, U+110000 is not a codepoint, and those bytes are therefore not a code point sequence. So those remain invalid.

[squeek502]

My problems with that proposal:

• You're justifying it using spec language around parsing instead of encoding. Currently, the Zig language reference says that \u will be UTF-8 encoded (and your intention is to keep it that way), but my understanding is still that there is no specified way to UTF-8 encode U+D800-U+DFFF.
• I don't think this is ultimately very relevant to making a decision about this proposal and only serves to obscure the effects. I think biting the bullet and proposing to define \u to encode as WTF-8 (and therefore accept surrogate code points) makes for a clearer/better proposal in terms of the tradeoffs being made.

Here's my attempt at this proposal:

Status quo

Currently, Zig defines \u{NNNNNN} as

hexadecimal Unicode scalar value UTF-8 encoded (1 or more digits)

This means that:

• All code points except the surrogate code points D800-DFFF are allowed
• By using only \u escapes, there is no possible way to end up with ill-formed UTF-8.

However, with Zig now commonly returning and accepting WTF-8, being able to use \u for surrogate code points has practical use cases.

The proposal

This proposal is to instead define \u{NNNNNN} as

hexadecimal Unicode code point WTF-8 encoded (1 or more digits)

With potentially a clarifying note:

WTF-8 is a superset of UTF-8 that allows the codepoints U+D800 to U+DFFF (surrogate codepoints) to be encoded using the normal UTF-8 encoding algorithm

Use cases

The main use case here would be avoiding the need to manually specify the bytes for the WTF-8 encoding of surrogate code points in string literals. For example,

zig/lib/std/unicode.zig

Lines 2039 to 2047 in d9bd34f

	try testRoundtripWtf8("\xed\x9f\xbf"); // not a surrogate half
	try testRoundtripWtf8("\xed\xa0\xbd"); // high surrogate
	try testRoundtripWtf8("\xed\xb2\xa9"); // low surrogate
	try testRoundtripWtf8("\xed\xa0\xbd \xed\xb2\xa9"); // <high surrogate><space><low surrogate>
	try testRoundtripWtf8("\xed\xa0\x80\xed\xaf\xbf"); // <high surrogate><high surrogate>
	try testRoundtripWtf8("\xed\xa0\x80\xee\x80\x80"); // <high surrogate><not surrogate>
	try testRoundtripWtf8("\xed\x9f\xbf\xed\xb0\x80"); // <not surrogate><low surrogate>
	try testRoundtripWtf8("a\xed\xb0\x80"); // <not surrogate><low surrogate>
	try testRoundtripWtf8("\xf0\x9f\x92\xa9"); // U+1F4A9, encoded as a surrogate pair in WTF-16

Could instead look like:

    try testRoundtripWtf8("\u{D7FF}"); // not a surrogate half
    try testRoundtripWtf8("\u{D83D}"); // high surrogate
    try testRoundtripWtf8("\u{DCA9}"); // low surrogate
    try testRoundtripWtf8("\u{D83D} \u{DCA9}"); // <high surrogate><space><low surrogate>
    try testRoundtripWtf8("\u{D800}\u{DBFF}"); // <high surrogate><high surrogate>
    try testRoundtripWtf8("\u{D800}\u{E000}"); // <high surrogate><not surrogate>
    try testRoundtripWtf8("\u{D7FF}\u{DC00}"); // <not surrogate><low surrogate>
    try testRoundtripWtf8("a\u{DC00}"); // <not surrogate><low surrogate>
    try testRoundtripWtf8("\u{1F4A9}"); // U+1F4A9, encoded as a surrogate pair in WTF-16

(this test [and others in std.unicode] was somewhat of a pain to write initially due to having to specify the WTF-8 encoding as bytes)

In the event that you're specifying a path/environment variable that happens to contain an unpaired surrogate, then not needing to manually specify it as WTF-8 encoded bytes would make things much nicer:

const path = "some/wtf8path/with\u{d83d}surrogates\u{dc00}";
var file = try std.fs.cwd().openFile(path, .{});
defer file.close();

const expected_val = "some wtf8 value with \u{d83d} surrogates \u{dc00}";
const actual_val = try std.process.getEnvVarOwned(allocator, "FOO");
defer allocator.free(actual_val);

if (std.mem.eql(u8, expected_val, actual_val)) {
    // ...
}

With status quo, you'd have to look up/calculate the WTF-8 encoded form of the surrogate code points and specify them as bytes.

Potential drawbacks

The biggest drawback would be that \u escapes would lose the property of always resulting in well-formed UTF-8, which may be surprising/unexpected.

However, using \u with surrogate code points seems somewhat hard to do by accident. The most "realistic" accidental usage I can think of would come from doing something like looping through all values 0 through 10FFFF and generating Zig code with string literals that have \u escapes with that value.

WTF-8 can be ill-formed, too

Another thing to keep in mind is that WTF-8 is intended to only encode unpaired surrogates. Doing something like:

"\u{D83D}\u{DCA9}" // <high surrogate><low surrogate>

would result in ill-formed WTF-8, since the encoded surrogate code points form a surrogate pair.

• Ill-formed WTF-8 will not roundtrip when converting to WTF-16 and back to WTF-8 (the surrogate pair, when converted to WTF-16, will be indistinguishable from any other surrogate pair, so when converting back to WTF-8, it will end up as the UTF-8 encoding of the code point that is encoded by the surrogate pair)
• This is not necessarily a huge deal, since most of the time WTF-8 is just used as an interchange format and will be converted to WTF-16 before being used in syscalls. It is a problem if you're e.g. comparing two WTF-8 strings, though.
• Note also that the std.unicode WTF-8 functions do not attempt to detect/enforce WTF-8 well-formedness; it is up to the user to ensure well-formedness if they have reason to care about it (see the notes in Fix handling of Windows (WTF-16) and WASI (UTF-8) paths, etc #19005)

[mnemnion]

WTF-8 can be ill-formed, too

This is one of a few reasons why I don't believe that bringing WTF-8 into the picture adds much to the discussion. UTF-8 is an encoding, and it's the one which Zig source code uses, but Zig isn't a language which validates the encoding property of strings.

I believe I've demonstrated adequately that the standard has a concept of a code point encoding, which the proposal for \u{xxxxxx} escape sequences precisely matches.

The standard devotes a great deal of energy to making sure that no one gets the idea that invalid use of codepoints is in any sense valid Unicode. It doesn't spend a lot of energy discussing the ins and outs of invalid sequences. Nonetheless, we get things like the official Unicode name for overlong encodings, a "non-shortest form".

I draw your attention again to this paragraph:

Because surrogate code points are not Unicode scalar values, any UTF-8 byte sequence that would otherwise map to code points U+D800..U+DFFF is ill-formed.

So I'm not sure what we're supposed to do with this claim:

but my understanding is still that there is no specified way to UTF-8 encode U+D800-U+DFFF.

There is a well-understood way to emit a code point sequence corresponding to those characters. It's given in table 3.6 of the standard. No one, at any point, in discussing this proposal, has ever had the misapprehension that the result would be well-formed UTF-8.

As a result, there's no concrete difference being discussed here.

Proposed behavior: Zig interprets a sequence \u{xxxxx} as corresponding to a Unicode codepoint, and generates the corresponding UTF-8 code point sequence, without reference to its validity as UTF-8.

This is accurate, and actionable, and uses the language of the standard.

What are you trying to accomplish by rephrasing it? I could tweak it a bit, actually, it has a bit of the flavor of standardese, given what I was reading right before I wrote it:

Proposed behavior: Zig interprets a sequence \u{xxxxx} as corresponding to a Unicode codepoint, and generates the corresponding UTF-8 code point sequence, whether or not this is a valid UTF-8 scalar value.

"the UTF-8 code point sequence corresponding to a Unicode codepoint" is, as I took pains to illustrate, a well-defined concept according to the standard. There's no need to bring a second standard along for the ride.