Skip to content

[Perf] request_logger: skip per-token watermark redaction walk - #44692

Merged
hanming-lu merged 1 commit into
sgl-project:mainfrom
JustinTong0323:xinyuan/watermark-redact-log-perf
Oct 23, 2026
Merged

hanming-lu merged 1 commit into
sgl-project:mainfrom
JustinTong0323:xinyuan/watermark-redact-log-perf

Conversation

@JustinTong0323

@JustinTong0323 JustinTong0323 commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

#37577 (Aaronson-Gumbel text watermarking) added redact_watermark_secrets to the request-log formatters. Both _dataclass_to_string_truncated and _transform_data_for_logging redact the whole request object before truncation and before skip_names filtering, so every --log-requests call walks the full input_ids element by element (~0.5us/token). With long prompts this costs tens of ms per log call on the tokenizer-manager event loop, on receive and again on finish, and shows up directly in TTFT.

On main, formatting an 80k-token GenerateReqInput at --log-requests-level 2 costs ~62 ms (text) / ~60 ms (json); with this PR it is ~0.3 ms / ~1.1 ms.

Modifications

python/sglang/srt/utils/request_logger.py:

  • New _redact_field: redact by field/key name. Fields named watermark_key / watermark_key_b / watermark_config become <redacted>; watermark / sampling_params / preferred_sampling_params are redacted recursively; everything else passes through untouched. skip_names fields are dropped before redaction.
  • The str(...) fallback in both formatters still redacts first, so a msgspec Struct carrying watermark fields under an ordinary key stays redacted, exactly as on main.
  • The truncated-list branch of _transform_data_for_logging now transforms each printed item instead of returning raw items, so a truncated list of dataclasses/Structs no longer breaks json.dumps.

python/sglang/srt/sampling/watermarking/core.py:

  • The watermark field-name sets are now module-level WATERMARK_SECRET_FIELD_NAMES / WATERMARK_CONTAINER_FIELD_NAMES, shared by all redact branches and the logger instead of being re-listed in five places.
  • New scalar fast-path: _SCALAR_TYPES.issuperset(map(type, value)) scans a sequence in C, and a plain-scalar sequence (e.g. token ids) can never hold a secret, since secrets only sit under dict keys or struct/dataclass fields. Mutable lists come back as copies (the crash-dump path pickles the result on a worker thread); immutable tuples may alias. The mixed-container walk skips scalar items inline. This bounds the other redact_watermark_secrets callers too — dump_requests / crash dump in tokenizer_manager.py, server info — and list-of-lists batched input_ids.

Log output is byte-identical to main at every --log-requests-level, with one intentional exception: Struct/dataclass items inside truncated json lists now print as redacted strings instead of crashing json.dumps.

Accuracy Tests

Logging-only change; no model path touched.

New test/registered/unit/utils/test_request_logger_redaction.py (CPU, base-a-test-cpu):

  • Secrets are redacted in both formatters, for 16 and 100k input ids, including the top-level GenerateReqInput.watermark field.
  • A msgspec Struct with a watermark_key field under an ordinary key stays redacted (regression test for the string fallback).
  • Truncated lists keep their printed halves redacted; a truncated json list of Structs stays json.dumps-serializable.
  • The redaction walk is bounded at each level's real (max_length, skip_names) — this test fails on main.
  • Scalar sequences skip the walk; mutable lists come back as copies while tuples may alias; containers are still redacted.

Output equivalence vs main was checked with both formatters over 22 payload shapes (levels 0-3 metadata, nested secrets, msgspec Structs under ordinary fields, long secret-bearing lists, batched input_ids, finish out dicts, tuples, None secrets, unicode text): byte-identical in all 44 runs, no secret string in any output.

Speed Tests and Profiling

CPU, single thread:

case main (ms) PR (ms)
level 2, text, 80k input_ids 61.6 0.3
level 2, json, 80k input_ids 59.6 1.1
scalar list redact, 100k ids 74.4 2.3
mixed 100k scalars + 1 dict 74.2 5.8
batched 32x4k input_ids, level 2 text 204.6 14.0
level 3, text, 80k input_ids (prints every id) 127.0 9.2
crash-dump-shaped redact (80k prompt + 20k output_ids) 14.7 0.5

Level 3 and batched requests remain dominated by str() of the ids they print; the redaction on top is now a C-level scan.

Related: #44683 (closed) fixed the same logger path with a different split (it skipped the walk only for skip_names fields and long non-watermark lists). This PR redacts strictly by field name and additionally bounds the other redact_watermark_secrets callers via the core fast-path.

Checklist


CI States

Latest PR Test (Base): ✅ Run #39567354719
Latest PR Test (Extra): ❌ Run #39567354781
Latest PR Test (AMD ROCm 10): ❌ Run #39567354833

@JustinTong0323 JustinTong0323 added the run-ci CI: run the baseline test suite on this PR label Oct 16, 2026
@JustinTong0323 JustinTong0323 changed the title [Perf] Fix watermark redaction walking every token id in request logging [Perf] request_logger: skip per-token watermark redaction walk Oct 20, 2026
@JustinTong0323

Copy link
Copy Markdown
Collaborator Author

/rerun-failed-ci

@JustinTong0323
JustinTong0323 force-pushed the xinyuan/watermark-redact-log-perf branch 2 times, most recently from 5f5bd41 to 0a09eba Compare October 21, 2026 04:00
The watermark redaction added to the request-log formatters redacted the
full request object before truncation: the dataclass/dict branches of
_dataclass_to_string_truncated and _transform_data_for_logging recursed
into every element of input_ids/output_ids (~0.5us/token, tens of ms for
long prompts) on the tokenizer-manager event loop, twice per request.

Redact by field/key name instead: watermark secrets only live under the
WATERMARK_SECRET_FIELD_NAMES / WATERMARK_CONTAINER_FIELD_NAMES sets, now
exported from watermarking/core.py instead of being re-listed per branch.
Everything else passes through untouched, except the string fallback,
which still redacts so a msgspec Struct carrying watermark fields under
an ordinary key stays redacted exactly as on main.

redact_watermark_secrets gets a scalar fast-path: a C-level
issuperset(map(type, value)) check returns plain-scalar sequences after
a single scan (lists are copied, since the crash-dump path pickles the
result on a worker thread; immutable tuples may alias), and the mixed
walk skips scalar items inline. This bounds the remaining whole-object
callers (dump_requests / crash dump, server info) and list-of-lists
batched input_ids.

_transform_data_for_logging's truncated-list branch now transforms each
printed item instead of returning them raw, so a truncated list of
dataclasses/Structs no longer breaks json.dumps.

Formatting an 80k-token request at --log-requests-level 2 drops from
~62ms to ~0.3ms (text) / ~1.1ms (json). Log output is byte-identical to
main across levels 0-3 (22 payload shapes x both formatters), except
that Struct/dataclass items in truncated json lists now print as
redacted strings instead of crashing json.dumps.
@JustinTong0323
JustinTong0323 force-pushed the xinyuan/watermark-redact-log-perf branch 2 times, most recently from 5abaa98 to 8564450 Compare October 21, 2026 08:10
@metamergebot

Copy link
Copy Markdown
Collaborator

This port clears our bar on 85644502aaba7cf3cd16f6b4203eb5f0306d3acc: every base-a and base-b job is green.
base-c and the accelerator suites are reported but not gated, because they run
on scarce hardware and upstream merges through them routinely.

gate✓ lint✓ build✓ A✓ B✓ C✓ other 6✓✂1 · accel ✗amd,npu

@luccafong @merrymercy @JustinTong0323 — could one of you take a look and merge it if you are happy? Nothing else is waiting on our side.

@hanming-lu
hanming-lu merged commit 92e57d5 into sgl-project:main Oct 23, 2026
192 of 206 checks passed
@JustinTong0323
JustinTong0323 deleted the xinyuan/watermark-redact-log-perf branch November 5, 2026 08:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants