Syntax feature request: subpattern recursion

[CAD97]

(Note: this is purely about regex-syntax being able to parse the syntax, and not about regex supporting executing them in any manner.)

Specifically, I'd like to see regex-syntax be able to parse the PCRE syntax of (?&name) and/or (?P>name).

Concrete benefits:

• regex can recognize the syntax and say something along the lines of "subpattern recursion is not supported" and link the docs for why
• Users of regex-syntax can implement ?(DEFINE)-like functionality, providing "subpatterns as subroutines" for more legible regexes, just replacing the recursion reference with a copy of the subpattern regex. (We're looking into doing this for logos, and I'd much prefer to avoid regex preprocessing.)

I'm happy to do the implementation work if this is something desirable; I'd just need to know where in the Ast enum it'd be best to store this node kind.

[BurntSushi]

I don't think this is a good idea, sorry. I don't think regex-syntax should evolve to support syntax that isn't being used by regex. Its primary purpose is to be an implementation detail of regex.

My main concern with adding syntax like this is that it becomes a huge maintenance burden in three different dimensions.

Firstly, in the past, parsing has been a source of bugs. I've mostly fixed that through more rigorous testing. There are so many tests that compiling and running tests for regex-syntax takes quite a bit of time. Adding even more syntax features is only going to make that worse.

Secondly, this is exactly the kind of feature that begets more features. If regex-syntax supports recursion even though regex doesn't use it, then what about look-around? And backreferences? And who knows what other things PCRE offers.

Thirdly, it creates a compatibility hazard. Is adding recursion like this intended to match the syntax of PCRE exactly? If so, there's real maintenance work and effort required to make sure both implementations of the syntax are correct.

Other than maintenance hazards, there are other issues. For example, if recursion were added to the AST, would it also get added to the HIR? If not, then would it still be useful to you? (Are you using the AST directly or the HIR?)

regex can recognize the syntax and say something along the lines of "subpattern recursion is not supported" and link the docs for why

The parser today actually does this for look-around and backreferences, simply because they are so common. But it's just a best effort recognition of a syntax that is already invalid, and is meant to improve failure modes. I'd be okay doing this for recursion too if it's a easy to do, which would get us this benefit without having to add it to the AST. (But I've never heard anyone complain about missing recursion, unlike look-around or backreferences.)

I'm sorry to turn you down. I realize that this would be a really big benefit from your perspective, since regex-syntax already does so much else. I guess one path forward for you would be to just maintain a patched vendored copy of regex-syntax in your project. It's not a great answer, but as of now, I'm hopeful that regex-syntax will not see any more major changes in its core parsing code. Off the top of my head, these are the changes I can think of that should be happening Real Soon Now:

• Completely re-work literal extraction.
• Possibly add a new HirKind::Fail variant.
• Relax the escaping rules, e.g., so that \/ is just a / even though / isn't a regex meta character.

I think only the last one is a change to the parser, but it should be a pretty simple one. So in that sense, I do feel like maintaining a patched version of regex-syntax is actually feasible if unpleasant.

[CAD97]

(Wow, I got a convenient issue number for this 😄)

That's definitely an understandable position to take. As much as it's maybe not "nice," there shouldn't be issues with us doing textual recognition of (?&name) specifically because it's such a simple syntax.

I'd be okay doing [best effort error diagnostics] for recursion too if it's a easy to do, which would get us this benefit without having to add it to the AST.

That should be fairly simple to do, actually: when parsing flags, have special diagnostics for a flag of R, P, or &. (Similarly, seeing digits in the flags could be a special error for recursion as well.) Or all of (?R, (?P> and (?& could be handled as an extra branch to this condition.

[CAD97]

Relax the escaping rules, e.g., so that \/ is just a / even though / isn't a regex meta character.

Just one other note that just popped into my head: you probably still want to "just" add / to the list of characters that are valid to escape. Allowing any character to be redundantly escaped would allow patterns like \g to pass the parser but not have the PCRE behavior.

A quick scan of regex101 shows the following escape characters as having meaning: 0123456789\ABcDEfgHkLnPQrStUvWxz

Also according to regex101, the delimiter characters that might be escaped even though they don't have meta meaning in the regex: /~@;%`"'

My gut says that adding the latter set to the escape whitelist would get 99% the benefit of truly relaxing the escaping rules further than just that set.

[ethanpailes]

@CAD97 only allowing escape sequences that actually have meaning is a pretty long standing part of regex's design with some pretty good reasons behind it:

• It allows the creation of new escape sequences in a backwards compatible way, because adding a new escape sequence is just turning a regex that was previously invalid into a valid regex. No one will have an existing regex that gets broken due to a behavior change.
• It makes it much easier to keep all the things that need to or can be escaped in your head if it is a small set. This one is more a matter of taste, but at least for me personally I find that the fewer ways to do something there are in a given language, the easier it is to remember the whole language.

I doubt that this will change.

[RReverser]

@CAD97 It sounds like you almost want completely different syntax than what regex-syntax can parse (with some overlap due to both being regexes).

Have you considered making a custom parser instead? It shouldn't be too hard to do via e.g. nom and you would have full control over which features are allowed and how they should be represented.

[CAD97]

@RReverser It's not a completely different syntax, but rather a superset that lies somewhere between regex-syntax and PCRE. And I've written probably too many parsers, so I definitely know how to write my own parser 😛

[BurntSushi]

I'm going to close this out since I don't see the core request here being granted, given the reasons I already stated. I believe there is already another open issue for making escapes a touch more liberal than they are today.

As for diagnostics, I think if someone cared enough and it was a simple patch, then I'd accept a PR. But otherwise, I don't feel moved to do it myself or even track it. Recursion is a really niche regex feature that is rarely used in my experience.