Skip to content

Add Clickbench inspired REGEXP_REPLACE benchmark - #28

Open
Dandandan wants to merge 1 commit into
BurntSushi:masterfrom
Dandandan:add/clickbench-q28-host
Open

Dandandan wants to merge 1 commit into
BurntSushi:masterfrom
Dandandan:add/clickbench-q28-host

Conversation

@Dandandan

@Dandandan Dandandan commented May 19, 2026 •

Copy link
Copy Markdown

A new grep-captures benchmark modeling ClickBench Q28 (analytic SQL benchmark), which extracts the URL host from HTTP Referer values:

REGEXP_REPLACE(\"Referer\", '^https?://(?:www\.)?([^/]+)/.*\$', '\1')

Currently rust/regex is about 6.5x slower than javascript/v8 on this benchmark.

results

  Engine                          |   Median |  vs slowest |  vs rust/regex
  --------------------------------+----------+-------------+----------------
  javascript/v8 23.10.0           |  2.48 ms |       21.7x |          2.4x
  python/re 3.13.7                | 12.63 ms |        4.3x |         12.4x
  perl 5.34.1                     | 15.31 ms |        3.5x |         15.0x
  python/regex 2026.5.9           | 16.46 ms |        3.3x |         16.1x
  rust/regex 1.7.3 (no patch)     | 16.43 ms |        3.3x |         16.1x
  rust/regex (patched).           |  1.02 ms |       52.7x |          1.0
  go/regexp 1.21.5                | 18.25 ms |        2.9x |         17.9x
  rust/regex/lite 0.1.8           | 53.73 ms |        1.0x |         52.7x

A `grep-captures` benchmark modeling ClickBench Q28, which extracts the
URL host out of HTTP `Referer` values via

    REGEXP_REPLACE("Referer", '^https?://(?:www\.)?([^/]+)/.*$', '\1')

The regex is anchored at both ends, the prefix reduces to a finite set
of literal byte alternatives, the capture is a single `[^/]+`, and the
tail is `.*$`. Patterns of this shape are common for "extract a field
bounded by a known delimiter" tasks — host extraction here, but also
log fields, single-line CSV-ish parsing, path components, and similar.

Haystack: 20,000 newline-separated synthetic Referer URLs (~800 KiB),
deterministically generated with ~80% matching the pattern. Expected
capture count: 32,000 (16,000 matching lines × {group 0, group 1}).

Also adds a small `compile` benchmark for the same regex.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@Dandandan
Dandandan force-pushed the add/clickbench-q28-host branch from 08b232a to 76ea776 Compare May 19, 2026 08:21
@Dandandan
Dandandan marked this pull request as ready for review May 19, 2026 08:22
@Dandandan Dandandan changed the title add wild/clickbench-q28-host benchmark Add Clickbench inspired REGEXP_REPLACE benchmark May 19, 2026
@BurntSushi

Copy link
Copy Markdown
Owner

This is amazing work. The write-up here is especially lovely. Thank you for that.

I see your PR on regex too and I've only had a chance to look at it broadly. So far I don't see any red flags. I love the general idea of looking for load bearing regexes and optimizing them like you've done here. That was my intent with the strategy infrastructure anyway.

With that said... I don't have a good sense of how well the strategy infrastructure scales. This isn't totally fair to pin on you, but could you run the rust/regex compile benchmarks (not Rust compilation, but regex compilation) before and after your change? I just want to check whether the additional strategy checking comes with a regression here. And if so, how big.

Thank you. :-) <3

@Dandandan

Dandandan commented May 19, 2026 •

Copy link
Copy Markdown
Author

This is amazing work. The write-up here is especially lovely. Thank you for that.

I see your PR on regex too and I've only had a chance to look at it broadly. So far I don't see any red flags. I love the general idea of looking for load bearing regexes and optimizing them like you've done here. That was my intent with the strategy infrastructure anyway.

With that said... I don't have a good sense of how well the strategy infrastructure scales. This isn't totally fair to pin on you, but could you run the rust/regex compile benchmarks (not Rust compilation, but regex compilation) before and after your change? I just want to check whether the additional strategy checking comes with a regression here. And if so, how big.

Thank you. :-) <3

I must admit this was mostly done by Claude (with some bit of steering / cleaning up of course) and I am not familiar with the code base so feel free to let it "cool down" a bit ;) or to let it inspire another implementation

The compilation time is within noise (earlier comment said it was slower (~16 vs ~70 us), but that was compared to rust/regex 1.7.3 which seems to have some lower overhead). 1.12.2 has similar overhead:

  Build                          | extract-host | compile
  -------------------------------+--------------+---------
  upstream master (no patch)     |     14.16 ms | 70.58 us
  my fork (with patch)           |      1.02 ms | 71.62 us
  -------------------------------+--------------+---------

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants