Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ keywords = ["llm", "genai", "inference", "nvidia", "distributed"]
dynamo-runtime = { path = "lib/runtime", version = "1.4.0" }
dynamo-llm = { path = "lib/llm", version = "1.4.0" }
dynamo-rl = { path = "lib/rl", version = "1.4.0" }
dynamo-tokenizers = { version = "=1.7.0" }
dynamo-tokenizers = { version = "=1.8.0" }
Comment thread
jthomson04 marked this conversation as resolved.
Resolved
dynamo-renderer = { version = "=5.0.0" }
dynamo-tokens = { path = "lib/tokens", version = "1.4.0" }
dynamo-truthy = { path = "lib/truthy", version = "1.4.0" }
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,13 @@ It supports features in `tokenizer.json` files (normalizers, pre-tokenizers, pos

The `fastokens` backend uses the [`fastokens`](https://github.com/Atero-ai/fastokens) crate, a purpose-built encoder optimized for throughput on supported BPE `tokenizer.json` models.
It is a _hybrid_ backend: encoding uses `fastokens` while decoding falls back to HuggingFace so that incremental detokenization, byte-fallback, and special-token handling work correctly.
It supports segmented encoding so renderers can distinguish trusted control tokens from ordinary content.
Comment thread
jthomson04 marked this conversation as resolved.
Resolved

Use this backend when tokenization is a measurable bottleneck, for example on high-concurrency prefill-heavy workloads.

#### `basetenkenizer` Native Encoder and Decoder

The `basetenkenizer` backend uses the Baseten Tokenizer implementation exposed by `dynamo-tokenizers`, a high-performance Rust BPE implementation for inference. Unlike the hybrid `fastokens` path, it performs both encoding and decoding natively and supports segmented encoding for renderers that must preserve trusted control-token boundaries.
The `basetenkenizer` backend uses the Baseten Tokenizer implementation exposed by `dynamo-tokenizers`, a high-performance Rust BPE implementation for inference. It performs both encoding and decoding natively and supports segmented encoding for renderers that must preserve trusted control-token boundaries.

Use this backend for supported `tokenizer.json` models when you need Baseten Tokenizer behavior, including token-compatible Kimi tokenizer artifacts.

Expand Down
11 changes: 7 additions & 4 deletions lib/bindings/kvbm/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 7 additions & 4 deletions lib/bindings/python/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading