API to get size of compiled regex

[bentheiii]

As an extension of the RegexBuilder::size_limit function, it might be useful to know the actual size taken by a compiled regular expression. A possible use for this is for when you want to process a number of untrusted regular expressions, and be assured that collectively they don't exceed a limit.

I propose that the size of the compiled program (the same one used when checking against the size_limit of the builder) be stored inside a regular expression struct, and be accessible through a new function Regex::approximate_size.

[BurntSushi]

A possible use for this is for when you want to process a number of untrusted regular expressions, and be assured that collectively they don't exceed a limit.

Could you elaborate on this? If you're processing untrusted regexes, then wouldn't RegexBuilder::size_limit prevents ones that are too big from compiling in the first place?

[bentheiii]

@BurntSushi I'm referring to a case where an untrusted user submits a number of patterns individually, and you want to make sure they do not exceed a "budget" for all of them.

For example if you wanted to set a limit of 10,000 bytes and to allow the users to submit over 100 patterns, you'd need to set the per-pattern limit to 100 bytes, which is not ideal.

[BurntSushi]

I can probably make this happen once #656 lands. Although, the precise relationship between the size limit and the bytes reported by this function may be tricky to get right. It probably does make sense from a certain perspective, but the problem is in how the limits are enforced. The limit tends to be enforced on a per-internal-NFA-graph basis, where as the natural thing to do for an "approximate memory usage" function would be to report the sum of everything using heap memory. Then there is the mutable scratch space to think about as well.

Now we could just say, "no the approximate memory usage function should be specifically scoped to precisely what RegexBuilder::size_limit is applied to." But then it might become quite a misleading function where folks might just assume it reports the total memory usage. Maybe a better name could help eliminate this confusion. I'm not sure.

[bentheiii]

Perhaps a clearer name would be built_size or compiled_size?

[fuchsnj]

This API would have been very helpful for me. In my case, I had a sample of a few thousand regex patterns that would be used, and I needed to determine what size limit should be used. To do that, I wanted to know how large all of these sample patterns were to see how close to the limit they were. I did it with a binary search of the builder (to see if it compiled or not) but that obviously wasn't ideal.

[BurntSushi]

@fuchsnj I don't think it will ever be possible for an API like the one you want to exist in such a granular way. The size limit really is just an approximation. And the binary search thing that you've done is exactly what I do in such cases as well. (Although I tend to just sent a very high size limit if I'm trying thousands of patterns. You're likely in for a pretty bad time in that case no matter what you do.)

[typetetris]

@BurntSushi Thanks for all your work in the rust ecosystem!

Now we could just say, "no the approximate memory usage function should be specifically scoped to precisely what RegexBuilder::size_limit is applied to." But then it might become quite a misleading function where folks might just assume it reports the total memory usage. Maybe a better name could help eliminate this confusion. I'm not sure.

To be able to do some approximate budgeting of a users resource consumption to prevent overuse, wouldn't it be better to have a function reporting the sum of everything using heap space? Why would one scope it to the same thing that RegexBuilder::size_limit is applied to?

[BurntSushi]

It's confusing to have a size limit restriction and a function that reports size but where these two things are not related.