Repository navigation
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces an LRU cache for compiled grammar objects in constrained decoding, allowing users to cap the in-memory cache size via the SGLANG_GRAMMAR_CACHE_MAX_ENTRIES environment variable. The changes include adding cachetools as a dependency across various pyproject.toml files, updating documentation, and implementing the cache logic in BaseGrammarBackend. Feedback was provided regarding the thread-safety of LRUCache, suggesting the use of a lock to prevent race conditions during concurrent access in a multi-threaded environment.
| @@ -17,10 +17,12 @@ | |||
| import time | |||
| self.cache: MutableMapping[Tuple[str, str], BaseGrammarObject] = ( | ||
| LRUCache(maxsize=cache_max_entries) | ||
| if (cache_max_entries := envs.SGLANG_GRAMMAR_CACHE_MAX_ENTRIES.get()) > 0 | ||
| else {} | ||
| ) |
There was a problem hiding this comment.
The cachetools.LRUCache is not thread-safe. Since BaseGrammarBackend is used in a multi-threaded environment (as indicated by the ThreadPoolExecutor and the server's architecture), concurrent access to self.cache can lead to race conditions and internal state corruption. Note that even get operations on an LRUCache are mutations because they update the LRU order.
You should protect all access to self.cache (specifically in get_cached_or_future_value, set_cache, and reset) with a lock.
| self.cache: MutableMapping[Tuple[str, str], BaseGrammarObject] = ( | |
| LRUCache(maxsize=cache_max_entries) | |
| if (cache_max_entries := envs.SGLANG_GRAMMAR_CACHE_MAX_ENTRIES.get()) > 0 | |
| else {} | |
| ) | |
| self.cache_lock = threading.Lock() | |
| self.cache: MutableMapping[Tuple[str, str], BaseGrammarObject] = ( | |
| LRUCache(maxsize=cache_max_entries) | |
| if (cache_max_entries := envs.SGLANG_GRAMMAR_CACHE_MAX_ENTRIES.get()) > 0 | |
| else {} | |
| ) |
|
@DarkSharpness @hnyls2002 Hi, could you help review this PR? Regarding the gemini bot's CR, I don't think the grammar backend cache is actually shared across threads. If there are any other issues, please let me know! |
|
Thanks @YzXiao101. Closing this because it has had no updates in 141 days. Reopen it if the work is still relevant. Some directories moved recently, so an older branch may need retargeting: |
Motivation
BaseGrammarBackendcurrently keeps compiled constrained-decoding grammars in an unbounded in-memory cache. Long-running structured-output workloads can accumulate many distinct schemas over time, so giving users a simple way to cap this cache makes SGLang easier to operate.This PR adds a user-configurable bounded cache while preserving the current default. The goal is not to change behavior for existing users, but to provide a practical memory-control knob that is already available in other serving engines like vLLM.
Modifications
SGLANG_GRAMMAR_CACHE_MAX_ENTRIESwith default-1cachetools.LRUCacheinBaseGrammarBackendwhen the value is positive0cachetoolsto all relevant manifestsAccuracy Tests
Basic server test on
Qwen/Qwen3-0.6Bwithxgrammar+ JSON schema. This is only a smoke test to confirm constrained decoding still works under both default and bounded cache settings. If we need more comprehensive accuracy tests or benchmarks, I'll provide them in a follow-up.Default / unbounded cache:
Bounded cache:
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ci