Allow non-ascii identifiers

[thejoshwolfe]

Here is a concrete proposal for #3947 (comment) .

Background

All Zig code is always encoded in UTF-8, and this proposal does not change that.

This proposal does not change the interpretation of ASCII codepoints anywhere in Zig code.

The only non-ascii codepoints with special handling in Zig before this proposal are: U+0085 (NEL), U+2028 (LS), U+2029 (PS). This proposal does not change the interpretation of these codepoints; they are not allowed in identifiers.

Proposal

Zig's current lexical rule for identifiers is:

IDENTIFIER
    <- !keyword ("c" !["\\] / [A-Zabd-z_]) [A-Za-z0-9_]* skip
     / "@\"" string_char* "\""                            skip

This proposal adds the codepoints listed in the table below to both the ranges [A-Zabd-z_] and [A-Za-z0-9_] in the above rule.

00A0
00A8
00AA
00AD
00AF
00B2..00B5
00B7..00BA
00BC..00BE
00C0..00D6
00D8..00F6
00F8..200D
202A..202F
203F..2040
2054
205F..218F
2460..24FF
2776..2793
2C00..2DFF
2E80..3000
3004..3007
3021..302F
3031..D7FF
F900..FD3D
FD40..FDCF
FDF0..FE44
FE47..FFFD
10000..1FFFD
20000..2FFFD
30000..3FFFD
40000..4FFFD
50000..5FFFD
60000..6FFFD
70000..7FFFD
80000..8FFFD
90000..9FFFD
A0000..AFFFD
B0000..BFFFD
C0000..CFFFD
D0000..DFFFD
E0000..EFFFD

Explanation

This set of codepoints was determined by following the recommendation here: https://unicode.org/reports/tr31/#Immutable_Identifier_Syntax . Specifically, this is the set of all characters except characters meeting any of these criteria:

• Pattern_White_Space=True
• Pattern_Syntax=True
• General_Category=Private_Use, Surrogate, or Control
• Noncharacter_Code_Point=True

Unicode Character Data version 5.2.0 was used to generate this list, but this list can remain stable forever despite future versions to Unicode Character Data, as per the recommendation and discussion in tr31 linked above. (EDIT: @daurnimator pointed out that this is many major versions behind, but even using the latest version 12.1.0, the list of codepoints in this proposal is identical.)

The code I used to generate the above set of codepoints can be found here: https://github.com/ziglang/zig/blob/6f8e2fad94fde6c9a8c4ca52d964d0616690ee4c/tools/gen_id_char_table.py

[daurnimator]

Why generated with 5.2.0 rather than the latest? Could you check to see if it's different for the purposes of this proposal?

[thejoshwolfe]

Thanks for pointing that out. I misunderstood the versioning scheme. I updated my script to use data version 12.1.0, and resulting table is identical. I guess that says something about the stability of this proposal.

[daurnimator]

I do think this is the best way to get Unicode identifiers in zig, but I'm still afraid of bugs caused by differing normalisation. Perhaps we could have zig fmt emit a warning when an identifer is not in NFKC form?

[Serentty]

@daurnimator That seems like a decent solution to me. It wouldn't be futureproof due to the addition of new characters over time, but since this is just a warning and not an error there are no compatibility issues introduced from that.

[mikdusan]

U+00A0 is NO-BREAK SPACE:

const hello var = "hello";
std.debug.warn("{}\n", .{hello var});

U+00B2 is SUPERSCRIPT TWO:

const ²name = "joe"
std.debug.warn("{}\n", .{²name});

If anyone is interested in a comparison, Swift language specs out their identifiers codepoints pretty concisely.

[Serentty]

I'm quite happy with this proposal. I think it's the simplest possible proposal that's still well-rounded. There are more complex solutions, and they're good too, but this is a very good compromise between features and utility.

[andrewrk]

The script looks for "Pattern_White_Space" but it should probably also exclude "White_Space"

[thejoshwolfe]

I don't claim to be an expert on Unicode, but unicode.org does, and they (indirectly) suggest allowing U+00A0 (NO-BREAK SPACE) in identifiers. To quote the experts:

The drawback of this method is that it allows “nonsense” to be part of identifiers because the concerns of lexical classification and of human intelligibility are separated. Human intelligibility can, however, be addressed by other means, such as usage guidelines that encourage a restriction to meaningful terms for identifiers.

If we wanted the compiler to try to prevent abuse in identifier naming, we would need something far more sophisticated than this proposal. See the discussion in #3947.

This proposal requires that programmers are trying to be responsible with identifier naming. We're opening the door to nasty code obfuscation, if programmers were trying to write obfuscated code. Are we worried about that?

I don't think we need to worry about U+00A0 (NO-BREAK SPACE). Maybe some language out there would really like to use it for their identifiers. It has no place in an English codebase, but this proposal is not for English speakers.

[jakwings]

This proposal deserves a more accurate title, like: Allow (only) UTF-8/UTF-16/...-encoded identifiers?

--

If one doesn't care about U+00A0 then why not ignore U+0085 (NEL), U+2028 (LS), U+2029 (PS)? Still complicated anyway. When would you type those invisible characters in source code besides in raw strings?

[Serentty]

I can't imagine the NBSP character being useful for identifiers in any language. It would be terribly confusing given that it's visually identical to a normal space except for its line breaking rules. Even in languages where it is used, it's mostly just to keep punctuation on the same line and stuff like that. An underscore should work just fine.

We're opening the door to nasty code obfuscation, if programmers were trying to write obfuscated code. Are we worried about that?

I think Zig already allows for all sorts of horribly obfuscated code, but this can't be considered a bug unless it makes it easy to do so accidentally. You could use comptime to generate all sorts of opaque stuff, but other than for the fun of it, why would you? So I would apply the same perspective here.

[thejoshwolfe]

@iology see #663 regarding UTF-16 and the characters you listed.

[jakwings]

see #663 regarding UTF-16 and the characters you listed.

Thanks. Now I see

1. UTF-8 (endianness unrelated) is enforced

2. The line endings NEL, LS and PS are forbidden in comments purely for their visual effect.

So if whitespace like U+00A0 (NBSP) and U+200B (ZERO WIDTH SPACE) are allowed in identifier names, then I question restriction (2). Even further, I think all valid code points outside 00-7F can be allowed if they will never affect the functionality of compilers and editors. Of course this opinion only holds when extreme tiny handling for Unicode is required.

Regarding the attributes like White_Space, will there be any newly assigned code point attribute in the future? (ah, got it: "small set" + "compromise")

Another observation: U+1680 (OGHAM SPACE MARK) appears as a dash symbol in the font Noto Sans Ogham (is it important?).

[kavika13]

@thejoshwolfe
Not sure if it's entirely clear in the first post why this proposal. I think you summed up the solution just fine. But the problem statement is a bit fuzzy unless you read the whole thread from the original proposal.

The problem statement - From a comment Andrew made on the Unicode identifier thread:

One benefit of status quo that I am reluctant to give up is that the definition of zig tokenization is finished. Aside from language changes before 1.0.0, tokenization is immortal and unchanging; already in its final form. 100% stable.

A dependency on Unicode tables means more than just the chore of implementing and maintaining support. It means the zig language itself depends on a third party standard that changes independently of Zig, without warning, and without any decision-making power from the Zig community.

I would be interested to explore what it might look like to support non-ascii identifiers without any knowledge of Unicode. For example, the naive approach of allowing any sequence of (non ASCII) bytes as an identifier. Some downsides I can think of:

The proposed solution - From Unicode 12.0.0 standard, annex 31: Immutable Identifier Syntax:

... Instead of defining the set of code points that are allowed, define a small, fixed set of code points that are reserved for syntactic use and allow everything else (including unassigned code points) as part of an identifier. All parsers written to this specification would behave the same way for all versions of the Unicode Standard, because the classification of code points is fixed forever ...

You summed this up pretty well, but maybe quoting this part of the unicode thing you linked directly could be valuable too?

Not sure how much more discussion needs to happen though, so this could be just make-work.