Correctly abort the failed grammar requests & Improve the handling of abort

[merrymercy]

• Abort the requests in the grammar queue correctly
• Return errors immediately for invalid grammar requests instead of silently ignoring the grammar
• Simplify the redundant code for aborted requests

[gemini-code-assist]

Hello @merrymercy, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

Summary of Changes

Hello! Gemini or gemini-code-assist here, providing a summary of this pull request. This PR primarily focuses on improving the handling of failed grammar requests and standardizing the process for aborting requests within the system. It introduces a dedicated object (INVALID_GRAMMAR_OBJ) to represent invalid grammar states, ensuring these failures are explicitly tracked and cached. Requests encountering invalid grammars or other validation errors are now consistently marked for abortion using a new helper method (set_finish_with_abort). Furthermore, the PR adds the capability to abort requests that are currently waiting in the grammar processing queue and includes metrics to track aborted requests. Minor adjustments are also made to CUDA graph capture batch sizes, memory pool size, logging, and test utilities.

Highlights

• Standardized Request Abortion: Introduced a new method set_finish_with_abort on the Req class to provide a consistent way to mark requests for abortion with a specific error message and status code. This replaces scattered logic for handling various validation failures.
• Improved Invalid Grammar Handling: Created a dedicated INVALID_GRAMMAR_OBJ to represent grammar compilation failures. Grammar backends now return this object instead of None on failure. The scheduler checks for this object (including in the cache) and aborts the corresponding request.
• Grammar Queue Abort Support: Implemented the ability to abort requests that are currently waiting in the grammar processing queue, ensuring that AbortReq signals are handled promptly for these requests.
• Metrics for Aborted Requests: Added a new Prometheus counter (sglang:num_aborted_requests) to track the total number of requests that are aborted.

Changelog

Click here to see the changelog

• python/sglang/srt/constrained/base_grammar_backend.py
  • Added a default copy implementation returning self (L63).
  • Introduced INVALID_GRAMMAR_OBJ (L102).
  • Changed CacheEntry.value type hint from Optional[BaseGrammarObject] to BaseGrammarObject (L107).
• python/sglang/srt/constrained/llguidance_backend.py
  • Imported INVALID_GRAMMAR_OBJ (L31).
  • Changed error handling in _from_serialized, dispatch_json, dispatch_ebnf, and dispatch_structural_tag to return INVALID_GRAMMAR_OBJ instead of None on failure (L131, L143, L156, L174).
  • Changed logging level from warning to error for grammar compilation failures (L130, L142, L155, L173).
• python/sglang/srt/constrained/outlines_backend.py
  • Imported INVALID_GRAMMAR_OBJ (L27).
  • Changed error handling in _compile_regex and dispatch_json to return INVALID_GRAMMAR_OBJ instead of None on failure (L156, L175).
  • Changed logging level from warning to error for JSON schema failures (L174).
• python/sglang/srt/constrained/xgrammar_backend.py
  • Imported INVALID_GRAMMAR_OBJ (L31).
  • Wrapped tokenizer info initialization in if True (L156).
  • Changed error handling in dispatch_json, dispatch_ebnf, dispatch_regex, and dispatch_structural_tag to return INVALID_GRAMMAR_OBJ instead of None on failure (L186, L194, L202, L221).
  • Changed logging level from warning to error for grammar compilation failures (L185, L193, L201, L220).
  • Removed conditional check for self.grammar_compiler in reset (L225).
• python/sglang/srt/entrypoints/http_server.py
  • Added [http_server] prefix to error logs in stream_results and generate (L259, L277).
• python/sglang/srt/managers/schedule_batch.py
  • Imported HTTPStatus (L40) and get_tensor_model_parallel_rank (L55).
  • Removed import of get_compiler_backend (L65).
  • Added set_finish_with_abort method to the Req class for standardized abortion (L776-L784).
• python/sglang/srt/managers/scheduler.py
  • Imported INVALID_GRAMMAR_OBJ (L39).
  • Updated handle_generate_request to use set_finish_with_abort for disaggregation errors (L960), multimodal prompt length errors (L991), input length errors (L1007), and logprob_start_len errors (L1021).
  • Added check in handle_generate_request to abort requests hitting a cached INVALID_GRAMMAR_OBJ (L1059-L1061).
  • Updated handle_embedding_request to use set_finish_with_abort for multimodal prompt length errors (L1109).
  • Updated move_ready_grammar_requests to handle requests already finished (aborted) (L1783-L1785).
  • Updated move_ready_grammar_requests to handle INVALID_GRAMMAR_OBJ from grammar backend results and abort requests (L1788-L1791, L1822-L1825).
  • Renamed num_abort_reqs to num_timeout_reqs in move_ready_grammar_requests (L1780, L1810, L1814).
  • Updated timeout handling in move_ready_grammar_requests to use set_finish_with_abort and cache INVALID_GRAMMAR_OBJ (L1834-L1835).
  • Removed TODO comment about aborting grammar queue requests (L2027).
  • Added logic to abort_request to cancel and abort requests found in the grammar_queue (L2056-L2061).
• python/sglang/srt/managers/tokenizer_manager.py
  • Initialized mm_processor to None if multimodal is not enabled (L224).
  • Updated _tokenize_one_request to only process multimodal data if mm_processor is available (L465).
  • Ensured image_inputs is None if multimodal processing is skipped (L475).
  • Modified _send_one_request to return the created ReqState (L635).
  • Modified _wait_one_response to accept ReqState as an argument instead of looking it up (L640).
  • Updated calls to _wait_one_response in _handle_batch_request to pass the ReqState (L714, L722, L748, L757).
  • Added metrics observation for aborted requests in abort_request (L793-L794).
• python/sglang/srt/managers/utils.py
  • Removed manual setting of finished_reason and logging in validate_input_length (L38-L41), as this is now handled by set_finish_with_abort in the scheduler.
• python/sglang/srt/metrics/collector.py
  • Added num_aborted_requests_total counter (L405-L408).
  • Added observe_one_aborted_request method to increment the aborted requests counter (L543-L544).
• python/sglang/srt/model_executor/cuda_graph_runner.py
  • Removed unused import two_batch_overlap (L27).
  • Adjusted default CUDA graph capture batch sizes, especially for speculative decoding and larger GPUs (L135, L144-L145, L150).
  • Removed logic adding size - 1 to capture sizes (L155-L157).
  • Removed filtering of batch sizes based on enable_two_batch_overlap (L159-L160).
  • Added assertion check for capture_bs (L165).
• python/sglang/srt/model_executor/model_runner.py
  • Changed ReqToTokenPool size from max_num_reqs + 1 to max_num_reqs (L921).
• python/sglang/srt/utils.py
  • Added is_blackwell helper function (L2058-L2061).
• python/sglang/test/send_one.py
  • Added check for non-200 HTTP status code and print response before returning (L130-L132).
• python/sglang/test/test_utils.py
  • Changed STDERR_FILENAME and STDOUT_FILENAME to use /tmp/ directory (L884-L885).
  • Reduced sleep time in read_output from 1 second to 0.01 seconds (L891).
  • Added try-except block for FileNotFoundError in read_output (L897-L901).
• scripts/killall_sglang.sh
  • Added conditional logic to skip nvidia-smi calls and GPU process killing if the first argument is 'rocm' (L3-L8, L10-L33).

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

---

Grammar fails, a syntax error,
No longer None, but a clear terror.
Requests now abort,
A cleaner report,
No more hidden, silent bearer.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request significantly improves the handling of failed and aborted grammar requests. Key improvements include:

• Correct Abort Handling: Requests in the grammar queue are now correctly aborted, and a new set_finish_with_abort method in the Req class centralizes the logic for aborting requests, which simplifies the codebase.
• Clearer Error Signaling: Invalid grammar requests now consistently return an INVALID_GRAMMAR_OBJ and log errors instead of warnings, making issues more visible and easier to debug.
• Code Simplification: Redundant code for handling aborted requests has been refactored using the new set_finish_with_abort method.

The changes are well-aligned with the PR's objectives and enhance the robustness and clarity of the error handling mechanisms. There are a few minor areas for improvement noted below.

Summary of Findings

• Improved Error Handling for Grammar Requests: The PR successfully standardizes error handling for invalid grammar requests by introducing INVALID_GRAMMAR_OBJ and consistently using logger.error.
• Centralized Request Abort Logic: The new Req.set_finish_with_abort() method effectively centralizes and simplifies the logic for aborting requests, improving code maintainability.
• Correct Abort of Grammar Queue Requests: The abort_request method in the scheduler now correctly handles aborting requests that are pending in the grammar queue.
• Redundant Code Removal: A redundant if True: block was identified in xgrammar_backend.py.
• Logging Consistency: A minor inconsistency in logging levels (logger.warning vs logger.error) was noted in outlines_backend.py for similar error conditions.

Merge Readiness

This pull request introduces valuable improvements to error handling and request abortion logic, significantly enhancing the robustness and maintainability of the grammar processing components. The changes are well-structured and address the stated objectives effectively.

There are a few minor points (a redundant if True:, a logging consistency question, and a query about enable_two_batch_overlap's CUDA graph implications) that would be good to address or clarify. Once these are considered, the PR should be in excellent shape for merging.

I am unable to approve pull requests, so please ensure other reviewers take a look and approve before merging.

[gemini-code-assist]

In _compile_regex, a logger.warning is used for invalid regex schemas, while other similar error handling spots in this PR (e.g., dispatch_json in this file, and other backend files) have been updated to use logger.error and return INVALID_GRAMMAR_OBJ.

Is there a specific reason to keep logger.warning here? For consistency with the goal of returning errors immediately for invalid grammar requests, perhaps logger.error would be more appropriate, even if this is an internal helper method.

[gemini-code-assist]

This if True: block appears to be redundant. Could it be removed to simplify the code?

        tokenizer_info = TokenizerInfo.from_huggingface(
            tokenizer, vocab_size=vocab_size
        )
        override_stop_tokens = None

[gemini-code-assist]

The if self.grammar_compiler: check was removed from the reset method. This implies self.grammar_compiler is guaranteed to be initialized. While this seems to be the case from the constructor, could you confirm if there are any scenarios where self.grammar_compiler might not be set before reset is called? If it's always set, this change is fine.

[gemini-code-assist]

The line self.origin_input_ids = [0] with the comment "set it to one token to skip the long prefill" is a bit of an implicit contract. While the comment explains the intent, it might be worth considering if there's a more explicit way to signal to the prefill logic that the request is aborted and should be skipped or handled minimally, rather than relying on a specific input ID pattern. However, if this is a well-established pattern in the codebase, it's acceptable.

[gemini-code-assist]

The logic for enable_two_batch_overlap (previously filtering capture_bs for bs >= 2) has been removed. Does this mean enable_two_batch_overlap is no longer supported, or is its handling for CUDA graph capture managed elsewhere? If the feature is still intended to work, it might be good to clarify how this change affects it.