Honor a configurable format depth in RabbitMQ's log formatters#16992
Open
lukebakken wants to merge 8 commits into
Open
Honor a configurable format depth in RabbitMQ's log formatters#16992lukebakken wants to merge 8 commits into
lukebakken wants to merge 8 commits into
Conversation
|
Tick the box to add this pull request to the merge queue (same as
|
`rabbit_logger_fmt_helpers:format_msg/3` renders `{Format, Args}`
messages with a bare `io_lib:format/2`, which ignores any configured
format depth. As a result crash reports with large process states or
mailboxes are logged in full, unlike OTP's `logger_formatter` which
honors a `depth` parameter.
Add a Common Test suite that pins the intended behavior: a `{Format,
Args}` message formatted with `depth => 3` must truncate deep sub-terms
to `...`, while `depth => unlimited` must render the full term. The
truncation case fails against the current implementation.
`format_msg1/3` renders `{Format, Args}` messages with a bare
`io_lib:format/2`, ignoring any configured format depth. Crash reports
carrying large process states or mailboxes are therefore logged in
full, which can drive a large peak memory footprint.
Read `depth` from the formatter config (defaulting to `unlimited`, the
previous behavior) and apply it the same way OTP's `logger_formatter`
does: rewrite the ~p/~w control sequences to ~P/~W with the depth
appended so deep sub-terms are truncated to `...`.
This turns the truncation test added previously green while leaving the
`unlimited` path byte-for-byte unchanged.
The arity-2 `report_cb' clause in `format_report/3' invokes the callback with an empty `Extra' map, so report callbacks never receive the configured format depth. `proc_lib:report_cb/2' therefore falls back to its `unlimited' default and logs crash reports in full, including large mailboxes and process states. Pass the depth-limiting keys (`depth', `chars_limit', `single_line') through to the callback, the same way OTP's `logger_formatter' does. Report callbacks can now truncate their output as intended. Add tests covering an arity-2 report callback that honors the `depth' from its `Extra' argument: truncated at a finite depth, full when `unlimited'.
Derive a `depth' value in the generic formatter config from the `log.*.formatter.depth' cuttlefish variable. When the key is unset or `unlimited', fall back to the deprecated `error_logger_format_depth' kernel variable, matching OTP's `logger_formatter:get_depth/1' (which also floors an explicit depth at 5). This lets the existing kernel setting finally take effect through RabbitMQ's own formatters. Export `translate_generic_conf/2' for testing alongside the other internal-testing helpers, and add a suite covering the explicit, `unlimited', and unset cases. The schema mapping that exposes `log.*.formatter.depth' in `rabbitmq.conf' is added separately.
Expose the formatter depth in `rabbitmq.conf` as
`log.{console,file,exchange,syslog}.formatter.depth', mirroring the
existing `formatter.single_line' mappings. The value accepts a positive
integer or `unlimited' (the default), and is routed into the translated
formatter config where `translate_generic_conf/2' resolves it.
Without this mapping cuttlefish rejects the key with `unknown_variable'.
Add a config-schema snippet asserting the schema accepts the key. The
harness strips the `rabbit.log' subtree before comparison, so the
snippet cannot check the resulting value; a comment records this and
points to `rabbit_prelaunch_early_logging_SUITE', which covers the
resolved depth value.
The JSON formatter renders the `msg' field through the shared
`format_msg/3', so it already honors the configured format depth,
including crash reports (logged as `{report, _}' messages). Lock this
in with a test suite covering the `{Format, Args}' path, the
`unlimited' case, and the report-callback path, each also asserting the
produced line is still valid JSON.
No formatter change is needed; the behavior comes from the earlier
`format_msg/3' depth support.
`translate_generic_conf/2` returns a map carrying a `depth` key, which the plaintext and JSON formatter translators merge into their results. The `formatter_plaintext_conf()` and `formatter_json_conf()` types do not declare `depth`, so dialyzer infers a return that violates the `translate_*_formatter_conf` contracts and reports "has no local return". Add `depth := unlimited | pos_integer()` to both types, matching the field already present on `formatter_generic_conf()`.
lukebakken
force-pushed
the
feature/logger-formatter-honor-format-depth
branch
from
July 23, 2026 15:31
2a72712 to
05af8b8
Compare
The `log.*.formatter.depth` mappings accept any integer, so a misconfiguration such as `depth = 0` or a negative value passes schema validation and is then silently clamped to 5. This is inconsistent with the sibling `log.error_logger_format_depth` mapping, which rejects non-positive values. Add a `formatter_depth` validator that accepts `unlimited` or a positive integer, and apply it to all four formatter depth mappings (console, exchange, syslog, file). Document in `translate_generic_conf/2` that an explicit depth is floored at 5, matching OTP's `logger_formatter:get_depth/1`, so the clamp is no longer undocumented.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Follow-up to #14349 and the work that shipped in #14523 / #14555.
Those changes shrink the queue process state via
format_state/1before it is logged, which stops the largest offender (the backing-queue state) from being dumped on a crash. They do not, however, cap the depth of everything else a crash report prints: theLast message, the exitReason, and the fullcrasher:SASL report fromproc_lib. A crash in a process with a large mailbox or large arguments (for example agen_batch_serversuch asra_log_wal, as @the-mikedavis noted in the discussion) can still produce a multi-hundred-KB to multi-MB log entry containing message bodies, and in the worst case OOM the node.OTP's own
logger_formatteralready solves this with itsdepthconfig value, which rewrites~p/~wto~P/~Wso deep sub-terms are truncated to.... RabbitMQ ships its own formatters (rabbit_logger_text_fmtandrabbit_logger_json_fmt), which did not honor any depth, so the deprecatedkernel.error_logger_format_depthsetting mentioned in #14349 was not reflected in RabbitMQ's formatted output.What
Make RabbitMQ's formatters honor a format depth, exposed as a per-output formatter setting:
Behavior:
logger_formatterdoes:~p/~wcontrol sequences are rewritten to~P/~Wwith the depth appended, truncating deeper sub-terms to.... This covers both{Format, Args}messages and reports formatted through an arity-2report_cb(which is howproc_libcrash reports are logged), so the crash report itself is truncated, not just the server state.unlimited, and RabbitMQ falls back to the deprecatederror_logger_format_depthkernel variable viaerror_logger:get_format_depth/0, exactly as OTP'slogger_formatter:get_depth/1does. This keeps the deprecated setting working for anyone relying on it today, with no behavior change for users who set neither.logger_formattersemantics.This is complementary to #14523 / #14555, not a replacement:
format_state/1removes the bulk of the state, and this bounds whatever remains across all crash-report fields.Testing
deps/rabbitmq_prelaunch: new CT suitesrabbit_logger_fmt_helpers_SUITE(text/report paths),rabbit_logger_json_fmt_SUITE(JSON path, asserting output remains valid JSON), andrabbit_prelaunch_early_logging_SUITE(config resolution and theerror_logger_format_depthfallback). All green.deps/rabbit:config_schema_SUITEcovers the new mappings; green.make dialyzeandmake xrefpass ondeps/rabbitmq_prelaunch;make dialyzepasses ondeps/rabbit.Related
format_state/1inrabbit_priority_queue#14555 / Correctly implementformat_state/1inrabbit_priority_queue(backport #14555) #14558 (format_state/1forrabbit_priority_queue)