Repository navigation
[Perf] request_logger: skip per-token watermark redaction walk - #44692
Merged
hanming-lu merged 1 commit intoOct 23, 2026
Merged
hanming-lu merged 1 commit into
hanming-lu merged 1 commit into
Conversation
Collaborator
Author
|
/rerun-failed-ci |
JustinTong0323
force-pushed
the
xinyuan/watermark-redact-log-perf
branch
2 times, most recently
from
October 21, 2026 04:00
5f5bd41 to
0a09eba
Compare
JustinTong0323
requested review from
Ying1123,
hnyls2002,
merrymercy and
xiezhq-hermann
as code owners
October 21, 2026 07:49
The watermark redaction added to the request-log formatters redacted the full request object before truncation: the dataclass/dict branches of _dataclass_to_string_truncated and _transform_data_for_logging recursed into every element of input_ids/output_ids (~0.5us/token, tens of ms for long prompts) on the tokenizer-manager event loop, twice per request. Redact by field/key name instead: watermark secrets only live under the WATERMARK_SECRET_FIELD_NAMES / WATERMARK_CONTAINER_FIELD_NAMES sets, now exported from watermarking/core.py instead of being re-listed per branch. Everything else passes through untouched, except the string fallback, which still redacts so a msgspec Struct carrying watermark fields under an ordinary key stays redacted exactly as on main. redact_watermark_secrets gets a scalar fast-path: a C-level issuperset(map(type, value)) check returns plain-scalar sequences after a single scan (lists are copied, since the crash-dump path pickles the result on a worker thread; immutable tuples may alias), and the mixed walk skips scalar items inline. This bounds the remaining whole-object callers (dump_requests / crash dump, server info) and list-of-lists batched input_ids. _transform_data_for_logging's truncated-list branch now transforms each printed item instead of returning them raw, so a truncated list of dataclasses/Structs no longer breaks json.dumps. Formatting an 80k-token request at --log-requests-level 2 drops from ~62ms to ~0.3ms (text) / ~1.1ms (json). Log output is byte-identical to main across levels 0-3 (22 payload shapes x both formatters), except that Struct/dataclass items in truncated json lists now print as redacted strings instead of crashing json.dumps.
JustinTong0323
force-pushed
the
xinyuan/watermark-redact-log-perf
branch
2 times, most recently
from
October 21, 2026 08:10
5abaa98 to
8564450
Compare
Collaborator
|
This port clears our bar on gate✓ lint✓ build✓ A✓ B✓ C✓ other 6✓✂1 · accel ✗amd,npu @luccafong @merrymercy @JustinTong0323 — could one of you take a look and merge it if you are happy? Nothing else is waiting on our side. |
5 tasks done
3 tasks done
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
#37577 (Aaronson-Gumbel text watermarking) added
redact_watermark_secretsto the request-log formatters. Both_dataclass_to_string_truncatedand_transform_data_for_loggingredact the whole request object before truncation and beforeskip_namesfiltering, so every--log-requestscall walks the fullinput_idselement by element (~0.5us/token). With long prompts this costs tens of ms per log call on the tokenizer-manager event loop, on receive and again on finish, and shows up directly in TTFT.On main, formatting an 80k-token
GenerateReqInputat--log-requests-level 2costs ~62 ms (text) / ~60 ms (json); with this PR it is ~0.3 ms / ~1.1 ms.Modifications
python/sglang/srt/utils/request_logger.py:_redact_field: redact by field/key name. Fields namedwatermark_key/watermark_key_b/watermark_configbecome<redacted>;watermark/sampling_params/preferred_sampling_paramsare redacted recursively; everything else passes through untouched.skip_namesfields are dropped before redaction.str(...)fallback in both formatters still redacts first, so a msgspec Struct carrying watermark fields under an ordinary key stays redacted, exactly as on main._transform_data_for_loggingnow transforms each printed item instead of returning raw items, so a truncated list of dataclasses/Structs no longer breaksjson.dumps.python/sglang/srt/sampling/watermarking/core.py:WATERMARK_SECRET_FIELD_NAMES/WATERMARK_CONTAINER_FIELD_NAMES, shared by all redact branches and the logger instead of being re-listed in five places._SCALAR_TYPES.issuperset(map(type, value))scans a sequence in C, and a plain-scalar sequence (e.g. token ids) can never hold a secret, since secrets only sit under dict keys or struct/dataclass fields. Mutable lists come back as copies (the crash-dump path pickles the result on a worker thread); immutable tuples may alias. The mixed-container walk skips scalar items inline. This bounds the otherredact_watermark_secretscallers too —dump_requests/ crash dump intokenizer_manager.py, server info — and list-of-lists batchedinput_ids.Log output is byte-identical to main at every
--log-requests-level, with one intentional exception: Struct/dataclass items inside truncated json lists now print as redacted strings instead of crashingjson.dumps.Accuracy Tests
Logging-only change; no model path touched.
New
test/registered/unit/utils/test_request_logger_redaction.py(CPU,base-a-test-cpu):GenerateReqInput.watermarkfield.watermark_keyfield under an ordinary key stays redacted (regression test for the string fallback).json.dumps-serializable.(max_length, skip_names)— this test fails on main.Output equivalence vs main was checked with both formatters over 22 payload shapes (levels 0-3 metadata, nested secrets, msgspec Structs under ordinary fields, long secret-bearing lists, batched
input_ids, finishoutdicts, tuples, None secrets, unicode text): byte-identical in all 44 runs, no secret string in any output.Speed Tests and Profiling
CPU, single thread:
input_ids, level 2 textoutput_ids)Level 3 and batched requests remain dominated by
str()of the ids they print; the redaction on top is now a C-level scan.Related: #44683 (closed) fixed the same logger path with a different split (it skipped the walk only for
skip_namesfields and long non-watermark lists). This PR redacts strictly by field name and additionally bounds the otherredact_watermark_secretscallers via the core fast-path.Checklist
CI States
Latest PR Test (Base): ✅ Run #39567354719
Latest PR Test (Extra): ❌ Run #39567354781
Latest PR Test (AMD ROCm 10): ❌ Run #39567354833