Skip to content

[Feature] Add env-configurable bounded grammar cache for constrained … - #22196

Closed
YzXiao101 wants to merge 2 commits into
sgl-project:mainfrom
YzXiao101:feat/bounded_grammar_cache
Closed

YzXiao101 wants to merge 2 commits into
sgl-project:mainfrom
YzXiao101:feat/bounded_grammar_cache

Conversation

@YzXiao101

Copy link
Copy Markdown
Contributor

Motivation

BaseGrammarBackend currently keeps compiled constrained-decoding grammars in an unbounded in-memory cache. Long-running structured-output workloads can accumulate many distinct schemas over time, so giving users a simple way to cap this cache makes SGLang easier to operate.

This PR adds a user-configurable bounded cache while preserving the current default. The goal is not to change behavior for existing users, but to provide a practical memory-control knob that is already available in other serving engines like vLLM.

Modifications

  • add SGLANG_GRAMMAR_CACHE_MAX_ENTRIES with default -1
  • use cachetools.LRUCache in BaseGrammarBackend when the value is positive
  • keep the existing unbounded cache behavior for values less than or equal to 0
  • add focused unit coverage for eviction and reset behavior
  • document the new environment variable in the env-var reference and structured-output docs
  • add cachetools to all relevant manifests

Accuracy Tests

Basic server test on Qwen/Qwen3-0.6B with xgrammar + JSON schema. This is only a smoke test to confirm constrained decoding still works under both default and bounded cache settings. If we need more comprehensive accuracy tests or benchmarks, I'll provide them in a follow-up.

Default / unbounded cache:

(sglang) root@ba5687f0db35:~/sglang# CUDA_VISIBLE_DEVICES=0 sglang serve --model-path Qwen/Qwen3-0.6B --host 127.0.0.1 --port 30000 --grammar-backend xgrammar --disable-cuda-graph
(sglang) root@ba5687f0db35:~/sglang# curl -sf http://127.0.0.1:30000/health
(sglang) root@ba5687f0db35:~/sglang# curl -s http://127.0.0.1:30000/generate -H 'Content-Type: application/json' -d @/tmp/sglang_structured_output_req.json
{"text":"{\"city\": \"Paris\"}","output_ids":[4913,8926,788,330,59604,9207,151643],"meta_info":{"id":"7fe23ef7fde84e55805f2224076a2972","finish_reason":{"type":"stop","matched":151643},"prompt_tokens":12,"completion_tokens":7}}

Bounded cache:

(sglang) root@ba5687f0db35:~/sglang# CUDA_VISIBLE_DEVICES=0 SGLANG_GRAMMAR_CACHE_MAX_ENTRIES=1 sglang serve --model-path Qwen/Qwen3-0.6B --host 127.0.0.1 --port 30000 --grammar-backend xgrammar --disable-cuda-graph
(sglang) root@ba5687f0db35:~/sglang# curl -s http://127.0.0.1:30000/generate -H 'Content-Type: application/json' -d @/tmp/sglang_structured_output_req.json
{"text":"{\"city\": \"Paris\" }","output_ids":[4913,8926,788,330,59604,1,335,151643],"meta_info":{"id":"b6d92939a5294602b029fd7db7d46ae1","finish_reason":{"type":"stop","matched":151643},"prompt_tokens":29,"completion_tokens":8}}
(sglang) root@ba5687f0db35:~/sglang# curl -s http://127.0.0.1:30000/generate -H 'Content-Type: application/json' -d @/tmp/sglang_structured_output_req.json
{"text":"{\"city\": \"Paris\" }","output_ids":[4913,8926,788,330,59604,1,335,151643],"meta_info":{"id":"c514644065484616af9e6b28840dc0f7","finish_reason":{"type":"stop","matched":151643},"prompt_tokens":29,"completion_tokens":8}}

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

@github-actions github-actions Bot added documentation Improvements or additions to documentation dependencies Pull requests that update a dependency file npu labels Apr 6, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces an LRU cache for compiled grammar objects in constrained decoding, allowing users to cap the in-memory cache size via the SGLANG_GRAMMAR_CACHE_MAX_ENTRIES environment variable. The changes include adding cachetools as a dependency across various pyproject.toml files, updating documentation, and implementing the cache logic in BaseGrammarBackend. Feedback was provided regarding the thread-safety of LRUCache, suggesting the use of a lock to prevent race conditions during concurrent access in a multi-threaded environment.

@@ -17,10 +17,12 @@
import time

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The threading module is required to implement a lock for the cache, as cachetools.LRUCache is not thread-safe.

Suggested change
import time
import threading
import time

Comment on lines +135 to +139
self.cache: MutableMapping[Tuple[str, str], BaseGrammarObject] = (
LRUCache(maxsize=cache_max_entries)
if (cache_max_entries := envs.SGLANG_GRAMMAR_CACHE_MAX_ENTRIES.get()) > 0
else {}
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The cachetools.LRUCache is not thread-safe. Since BaseGrammarBackend is used in a multi-threaded environment (as indicated by the ThreadPoolExecutor and the server's architecture), concurrent access to self.cache can lead to race conditions and internal state corruption. Note that even get operations on an LRUCache are mutations because they update the LRU order.

You should protect all access to self.cache (specifically in get_cached_or_future_value, set_cache, and reset) with a lock.

Suggested change
self.cache: MutableMapping[Tuple[str, str], BaseGrammarObject] = (
LRUCache(maxsize=cache_max_entries)
if (cache_max_entries := envs.SGLANG_GRAMMAR_CACHE_MAX_ENTRIES.get()) > 0
else {}
)
self.cache_lock = threading.Lock()
self.cache: MutableMapping[Tuple[str, str], BaseGrammarObject] = (
LRUCache(maxsize=cache_max_entries)
if (cache_max_entries := envs.SGLANG_GRAMMAR_CACHE_MAX_ENTRIES.get()) > 0
else {}
)

@YzXiao101

Copy link
Copy Markdown
Contributor Author

@DarkSharpness @hnyls2002 Hi, could you help review this PR? Regarding the gemini bot's CR, I don't think the grammar backend cache is actually shared across threads. If there are any other issues, please let me know!

@github-actions

Copy link
Copy Markdown
Contributor

Thanks @YzXiao101. Closing this because it has had no updates in 141 days.

Reopen it if the work is still relevant.

Some directories moved recently, so an older branch may need retargeting:
sgl-kernel/ -> python/sglang/kernels/aot/, python/sglang/jit_kernel/
-> python/sglang/kernels/jit/, docs/ -> docs/docs/ (.mdx),
bench_serving.py -> benchmark/serving.py, test/srt/ -> test/registered/.

@github-actions github-actions Bot closed this Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file documentation Improvements or additions to documentation npu

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant