gh-97556: Raise null bytes syntax error upon null in multiline string

[lysnikolaou]

• Issue: Null characters in strings cause a C SystemError #97556

[pablogsal]

LGTM

Maybe we should add sone asserts here and there that tok->done is not error to prevent this issue from happening in other places

[lysnikolaou]

Maybe we should add sone asserts here and there that tok->done is not error to prevent this issue from happening in other places

That's a good idea. Where would you add these asserts though, since most of the time we shouldn't be asserting, but rather check for it?

One of the items I have in my to-do list is to add comments and assert in various places in the tokenizer to make the logic a bit easier to follow. Would it be okay to merge this as-is and then do that in a separate PR?

[pablogsal]

Would it be okay to merge this as-is and then do that in a separate PR?

Absolutely, let's merge this as is 🚀

Great job as always man!

[vstinner]

@lysnikolaou: Do you consider backporting your change to Python 3.11?

[bskinn]

See discussion linked in the associated issue, there was a decision made not to backport the initial change that worked for single quoted strings, to avoid breaking compatibility in a patch version release.

[bskinn]

See #96670 (comment)

[lysnikolaou]

FYI I removed the label until it's more clear what should be backported.

[lysnikolaou]

GH-104195 includes a backport of this commit to 3.11.