Update Rust crate tokenizers to v0.21.4 - autoclosed

[renovate]

This PR contains the following updates:

Package	Type	Update	Change
tokenizers	dependencies	patch	0.21.0 -> 0.21.4

---

Release Notes

huggingface/tokenizers (tokenizers)

v0.21.4

Compare Source

Full Changelog: huggingface/tokenizers@v0.21.3...v0.21.4

No change, the 0.21.3 release failed, this is just a re-release.

https://github.com/huggingface/tokenizers/releases/tag/v0.21.3

v0.21.2

Compare Source

What's Changed

This release if focused around some performance optimization, enabling broader python no gil support, and fixing some onig issues!

• Update the release builds following 0.21.1. by @​Narsil in https://github.com/huggingface/tokenizers/pull/1746
• replace lazy_static with stabilized std::sync::LazyLock in 1.80 by @​sftse in https://github.com/huggingface/tokenizers/pull/1739
• Fix no-onig no-wasm builds by @​414owen in https://github.com/huggingface/tokenizers/pull/1772
• Fix typos in strings and comments by @​co63oc in https://github.com/huggingface/tokenizers/pull/1770
• Fix type notation of merges in BPE Python binding by @​Coqueue in https://github.com/huggingface/tokenizers/pull/1766
• Bump http-proxy-middleware from 2.0.6 to 2.0.9 in /tokenizers/examples/unstable_wasm/www by @​dependabot in https://github.com/huggingface/tokenizers/pull/1762
• Fix data path in test_continuing_prefix_trainer_mismatch by @​GaetanLepage in https://github.com/huggingface/tokenizers/pull/1747
• clippy by @​ArthurZucker in https://github.com/huggingface/tokenizers/pull/1781
• Update pyo3 and rust-numpy depends for no-gil/free-threading compat by @​Qubitium in https://github.com/huggingface/tokenizers/pull/1774
• Use ApiBuilder::from_env() in from_pretrained function by @​BenLocal in https://github.com/huggingface/tokenizers/pull/1737
• Upgrade onig, to get it compiling with GCC 15 by @​414owen in https://github.com/huggingface/tokenizers/pull/1771
• Itertools upgrade by @​sftse in https://github.com/huggingface/tokenizers/pull/1756
• Bump webpack-dev-server from 4.10.0 to 5.2.1 in /tokenizers/examples/unstable_wasm/www by @​dependabot in https://github.com/huggingface/tokenizers/pull/1792
• Bump brace-expansion from 1.1.11 to 1.1.12 in /bindings/node by @​dependabot in https://github.com/huggingface/tokenizers/pull/1796
• Fix features blending into a paragraph by @​bionicles in https://github.com/huggingface/tokenizers/pull/1798
• Adding throughput to benches to have a more consistent measure across by @​Narsil in https://github.com/huggingface/tokenizers/pull/1800
• Upgrading dependencies. by @​Narsil in https://github.com/huggingface/tokenizers/pull/1801
• [docs] Whitespace by @​stevhliu in https://github.com/huggingface/tokenizers/pull/1785
• Hotfixing the stub. by @​Narsil in https://github.com/huggingface/tokenizers/pull/1802
• Bpe clones by @​sftse in https://github.com/huggingface/tokenizers/pull/1707
• Fixed Length Pre-Tokenizer by @​jonvet in https://github.com/huggingface/tokenizers/pull/1713
• Consolidated optimization ahash dary compact str by @​Narsil in https://github.com/huggingface/tokenizers/pull/1799
• 🚨 breaking: Fix training with special tokens by @​ArthurZucker in https://github.com/huggingface/tokenizers/pull/1617

New Contributors

• @​414owen made their first contribution in https://github.com/huggingface/tokenizers/pull/1772
• @​co63oc made their first contribution in https://github.com/huggingface/tokenizers/pull/1770
• @​Coqueue made their first contribution in https://github.com/huggingface/tokenizers/pull/1766
• @​GaetanLepage made their first contribution in https://github.com/huggingface/tokenizers/pull/1747
• @​Qubitium made their first contribution in https://github.com/huggingface/tokenizers/pull/1774
• @​BenLocal made their first contribution in https://github.com/huggingface/tokenizers/pull/1737
• @​bionicles made their first contribution in https://github.com/huggingface/tokenizers/pull/1798
• @​stevhliu made their first contribution in https://github.com/huggingface/tokenizers/pull/1785
• @​jonvet made their first contribution in https://github.com/huggingface/tokenizers/pull/1713

Full Changelog: huggingface/tokenizers@v0.21.1...v0.21.2rc0

v0.21.1

Compare Source

What's Changed

• Update dev version and pyproject.toml by @​ArthurZucker in https://github.com/huggingface/tokenizers/pull/1693
• Add feature flag hint to README.md, fixes #​1633 by @​sftse in https://github.com/huggingface/tokenizers/pull/1709
• Upgrade to PyO3 0.23 by @​Narsil in https://github.com/huggingface/tokenizers/pull/1708
• Fixing the README. by @​Narsil in https://github.com/huggingface/tokenizers/pull/1714
• Fix typo in Split docstrings by @​Dylan-Harden3 in https://github.com/huggingface/tokenizers/pull/1701
• Fix typos by @​tinyboxvk in https://github.com/huggingface/tokenizers/pull/1715
• Update documentation of Rust feature by @​sondalex in https://github.com/huggingface/tokenizers/pull/1711
• Fix panic in DecodeStream::step due to incorrect index usage by @​n0gu-furiosa in https://github.com/huggingface/tokenizers/pull/1699
• Fixing the stream by removing the read_index altogether. by @​Narsil in https://github.com/huggingface/tokenizers/pull/1716
• Fixing NormalizedString append when normalized is empty. by @​Narsil in https://github.com/huggingface/tokenizers/pull/1717
• 🚨 Support updating template processors by @​ArthurZucker in https://github.com/huggingface/tokenizers/pull/1652. Removed in this release to keep backware compatibility temporarily.
• Update metadata as Python3.7 and Python3.8 support was dropped by @​earlytobed in https://github.com/huggingface/tokenizers/pull/1724
• Add rustls-tls feature by @​torymur in https://github.com/huggingface/tokenizers/pull/1732

New Contributors

• @​Dylan-Harden3 made their first contribution in https://github.com/huggingface/tokenizers/pull/1701
• @​sondalex made their first contribution in https://github.com/huggingface/tokenizers/pull/1711
• @​n0gu-furiosa made their first contribution in https://github.com/huggingface/tokenizers/pull/1699
• @​earlytobed made their first contribution in https://github.com/huggingface/tokenizers/pull/1724
• @​torymur made their first contribution in https://github.com/huggingface/tokenizers/pull/1732

Full Changelog: huggingface/tokenizers@v0.21.0...v0.21.1

---

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.