[charliermarsh]

Summary

We scan ASCII identifier continuations in sixteen-byte batches using a byte lookup table, avoiding per-character decoding. Keyword lookup filters by length and initial byte, then uses a compact hash table with a full-word comparison to reject collisions. Non-ASCII suffixes retain the existing Unicode path.

[astral-sh-bot]

Memory usage report

Memory usage unchanged ✅

[astral-sh-bot]

ecosystem-analyzer results

No diagnostic changes detected ✅

Full report with detailed diff (timing results)

[astral-sh-bot]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

[codspeed]

Merging this PR will improve performance by 4.7%

⚡ 3 improved benchmarks
✅ 241 untouched benchmarks
⏩ 10 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	lexer[pydantic/types.py]	682.8 µs	650 µs	+5.04%
⚡	Simulation	lexer[numpy/globals.py]	41.6 µs	39.6 µs	+4.94%
⚡	Simulation	lexer[numpy/ctypeslib.py]	302.1 µs	290.1 µs	+4.13%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing charlie/ruff-ascii-identifiers (5410284) with main (34266aa)²}

Footnotes

1. 10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. No successful run was found on main (355cda3) during the generation of this report, so 34266aa was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[MichaReiser]

I haven't looked at this in detail but it feels tricky to maintain (at least as it is now). I think we need some more documentation with guidance on how it works and what needs changing and how when adding/removing/changing keywords). Better isolation by e.g. encapsulating some into its own function might also help

[MichaReiser]

why?

[charliermarsh]

I got the vibe from your comments that you didn't want to proceed with it haha! Do you think it's salvageable? I'm happy to clean it up, I just thought you were left feeling like the tradeoffs weren't right.

[MichaReiser]

I think it's a worthwhile change. I just think it requires some more explanation of what happens here and how to update that weird constant when adding/removing kewyords 😆

[charliermarsh]

Haha ok. I will clean it up. I'll ask Codex to make it even weirder and more confusing.