Skip to content

yaml: apply a node's tag in the parser instead of baking it into the next scanned scalar - #38147

Closed
robobun wants to merge 3 commits into
mainfrom
farm/c2b0b142/yaml-tag-leak-next-key
Closed

robobun wants to merge 3 commits into
mainfrom
farm/c2b0b142/yaml-tag-leak-next-key

Conversation

@robobun

@robobun robobun commented Aug 13, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • A tag on one node changes how an unrelated plain scalar resolves:
    • Bun.YAML.parse("a: !!str b\n0x10: x") gives { a: "b", "0x10": "x" }; the key should be 16, as it is without the tag.
    • Bun.YAML.parse("a: !!str b\n~: x") gives { a: "b", "~": "x" }; the key should be null.
    • Bun.YAML.parse("x: !!map\n 0x10: a") gives { x: { "0x10": "a" } }; the key should be 16.
  • The first two shapes leak for every tag kind (!!str, !, !local, !<verbatim>, and !!int/!!float/!!bool/!!null, which block keys of any other kind), after a tagged value of any scalar style (plain, quoted, block), and wherever the next key lives (same mapping, ? key, : value, compact mapping in a sequence item, or an outer mapping reached by dedenting). js-yaml and the yaml package resolve the key independently in all of these.
  • Cause: the scanner resolves a plain scalar under ScanOptions.tag at scan time (src/parsers/yaml.rs, ScalarResolverCtx::resolve / try_resolve_number). The parser passed the tag to more scans than the one that produces the tagged node's content: parse_node's Scalar arm passed it to the scan that steps past the tagged scalar (whose result is the next key), and the Tag arm / parse_block_indented pass it to the scan after a tag even when that token turns out to be a block mapping's first key rather than the tagged node. parse_block_indented already carried a rewind-and-rescan workaround for a third instance (a: !!str\n0x10: x).

Fix

  • The scanner no longer takes a tag. A plain scalar token now carries both its text and, when the text has one, its core-schema value (TokenScalar { text, resolved, style }).
  • parse_node's Scalar arm is the only consumer of scalar tokens; it applies the node's tag there via TokenScalar::to_expr, using the same keep-or-string table the scanner used (NodeTag::keeps). For an implicit key it only applies a tag that sits on the key's line; a tag on an earlier line belongs to the block mapping (YAML [200]), which is the rule take_implicit_key_anchors already applies to anchors.
  • Correct because a tag belongs to exactly one node, and the parser is the only place that knows which node that is; resolving at scan time required guessing, and every scan that received the tag for a token that was not that node's content produced one of the bugs above. Resolution of the tagged scalar itself is unchanged: keeps is the table that was in resolve/try_resolve_number, and the new table test pins it.
  • Deleted along the way: ScanOptions.tag, the rewind in parse_block_indented, and ScalarStyle::Plain { multiline } (only the rewind read it). Token grows from 56 to 72 bytes; the parser holds one token at a time, so this is a per-token copy, not a buffer.
  • One other visible change, on invalid input only: a flow-sequence pair whose tag sits on the line before the key, [!!str\n0x10: a], now gives [{ 16: "a" }] instead of [{ "0x10": "a" }], because the tag goes where an anchor in that position already went ([&a\n0x10: 1, *a] aliases the pair mapping, before and after). js-yaml and the yaml package reject that input; the valid [!!str 0x10: a] is unchanged. Pinned by the "flow-sequence pair" test.
  • Verified:
    • test/js/bun/yaml/yaml.test.ts: six new tests under "anchor/tag on empty node" (next key after a tagged value, per tag kind, per scalar style, per key position including dedent, the [200] first-key case, and the flow-pair rule above) fail on the current release and pass with this change; "explicit tag on a plain scalar keeps the core-schema value only if it is of the tag's kind" pins the unchanged resolution table (passes before and after).
    • yaml.test.ts, yaml-test-suite.test.ts, yaml-block-scalar-matrix.test.ts (2134 tests), resolve/yaml, bundler yaml, and the pnpm-lock migration tests pass with the debug build; cargo clippy -p bun_parsers and cargo fmt are clean.

Background

  • Core schema: an untagged plain scalar resolves by its text (0x10 is 16, ~ is null, true is a boolean, anything else is a string). This applies to keys and values alike.
  • Tags: !!str, !!int, !foo, ... annotate the single node they precede. In this parser a tag of a different kind does not validate the text, it only switches the core-schema resolution off (!!str 0x10 is the string "0x10", !!int foo is the string "foo"); that behavior is unchanged here.
  • Properties before a block mapping ([200]): in !!map\n0x10: a or &m\n0x10: a, the property on the earlier line annotates the mapping, while in !!str 0x10: a it annotates the key. The parser cannot tell which until it has scanned the key and seen the :, which is why the decision has to live in the parser rather than the scanner.
  • Parser structure: scan() produces one token at a time into self.token; parse_node collects a node's properties (anchor, tag), dispatches on the content token, and for a scalar scans once more to see whether a : follows (implicit key) before returning. parse_block_indented is the shared property loop for the token after :, ?, or -; when the token after the properties belongs to the parent, the properties annotate an empty node.

…next scanned scalar

The scanner resolved a plain scalar under whatever tag the parser had
most recently seen (ScanOptions.tag). The tag was passed to more scans
than the one producing the tagged node's content, so it leaked into
unrelated plain scalars: after a tagged block-mapping value, into the
next entry's key (`a: !!str b` followed by `0x10: x` gave the key
"0x10" instead of 16, `~: x` gave "~" instead of null); after a tag on
the line before a block mapping, into that mapping's first key.
parse_block_indented worked around one instance of this by rewinding
and re-scanning an abandoned sibling.

The scanner now records both the core-schema value and the text of a
plain scalar, and parse_node's Scalar arm picks one once it knows which
node the collected properties belong to (a tag on an earlier line than
an implicit key is the block mapping's, mirroring the anchor rule).
This removes ScanOptions.tag, the rewind, and ScalarStyle::Plain's
multiline flag.
@coderabbitai

coderabbitai Bot commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@robobun, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 19 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 7c0dcb8f-450c-42d2-a77b-ba29abe15f55

📥 Commits

Reviewing files that changed from the base of the PR and between b7a0431 and 7678137.

📒 Files selected for processing (2)
  • src/parsers/yaml.rs
  • test/js/bun/yaml/yaml.test.ts

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Aug 13, 2026 •

Copy link
Copy Markdown
Collaborator Author

Reproduced on bun 1.4.0 with Bun.YAML.parse("a: !!str b\n0x10: x") (key "0x10", expected 16), "a: !!str b\n~: x" (key "~", expected null) and "x: !!map\n 0x10: a" (key "0x10", expected 16); js-yaml and the yaml package give the expected values for all three.

Fix: #38147 (this PR). Tests: test/js/bun/yaml/yaml.test.ts, six new tests fail before and pass after; yaml-test-suite and the block scalar matrix stay green.

Comment thread src/parsers/yaml.rs Resolved
Comment thread src/parsers/yaml.rs Resolved
Comment thread src/parsers/yaml.rs Resolved
Comment thread src/parsers/yaml.rs Resolved
Comment thread src/parsers/yaml.rs Resolved
Comment thread src/parsers/yaml.rs Resolved
Comment thread src/parsers/yaml.rs Resolved
@robobun

robobun commented Aug 13, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 4:45 AM PT - Aug 13th, 2026

✅ @robobun, your commit 76781377c01710e761e2144e00f78f934446cfe6 passed in Build #94681! 🎉


🧪   To try this PR locally:

bunx bun-pr 38147

That installs a local version of the PR into your bun-38147 executable, so you can run:

bun-38147 --bun

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I re-reviewed after f00e679 and 7678137 and didn't find any bugs — the flow-sequence-pair nit from my earlier pass is now pinned by a test and explained in the description, and the comment-cop feedback is addressed. Because this restructures the scanner/parser boundary (TokenScalar now carries text+resolved, ScanOptions.tag and the parse_block_indented rewind are gone), a human look is still worthwhile.

What was reviewed:

  • NodeTag::keeps is the same table that was inlined in ScalarResolverCtx::resolve/try_resolve_number; the new resolution-table test pins it.
  • YamlString::to_expr is a straight move of the old NodeScalar::String arm, arena-copy of List bytes preserved.
  • All TokenScalar/ScalarInit construction sites (plain, block, both quoted paths) updated; block/quoted set resolved: None so to_expr falls through to text as before.
  • ScalarStyle::Plain { multiline } had no reader left after the rewind deletion.
Extended reasoning...

Overview

The PR moves YAML tag application from scan-time to parse-time in src/parsers/yaml.rs. TokenScalar is restructured from { data: NodeScalar<Enc>, style } to { text: YamlString<Enc>, resolved: Option<NodeScalar>, style }; NodeScalar drops its String variant and its Enc parameter; ScanOptions.tag, ScalarResolverCtx.{tag,multiline}, ScalarStyle::Plain { multiline }, and the rewind-and-rescan block in parse_block_indented are deleted. parse_node's Scalar arm now applies the tag via TokenScalar::to_expr → NodeTag::keeps, with a line-based split for the [200] implicit-key case mirroring take_implicit_key_anchors. Roughly 200 lines of Rust net, plus ~100 lines of new tests covering the variant matrix (tag kind × scalar style × key position × dedent) and a table test pinning the unchanged resolution behavior.

Security risks

None. This is pure parser logic on already-tokenized input; no new untrusted-length arithmetic, no new allocation sizing, no FFI boundary changes. The one unsafe block (from_raw_parts in YamlString::to_expr) is moved verbatim from the old NodeScalar::to_expr String arm with the same SAFETY invariant (Enc::KIND == Utf16 ⇒ Enc::Unit == u16).

Level of scrutiny

Moderate-to-high. Bun.YAML.parse feeds the bundler's config loading and the pnpm-lock migration path, so a silent misparse would propagate. The change is architecturally the right fix per REVIEW.md ("fix at the layer that owns the invariant" — only the parser knows which node a tag belongs to), and the author reports yaml-test-suite.test.ts, yaml-block-scalar-matrix.test.ts, resolve/yaml, bundler yaml, and pnpm-lock migration tests all pass. But the data-type restructure touches every scalar-token construction and consumption site, which is more than a localized bugfix.

Other factors

My earlier nit (the key_tag line-split reaching the flow-sequence-pair path and changing [!!str\n0x10: a] on invalid input) was answered with a coherent rationale — tags now follow the same rule anchors already followed there, and the pair mapping is what begins at the property so it is the natural target — and pinned by the new "property on the line before a flow-sequence pair's key" test. The comment-cop bot's six long-comment flags were resolved by 7678137 (comments now one line each). All inline threads are resolved. Given the scope of the restructure I'm deferring rather than approving so a human can confirm the yaml-test-suite CI result and sanity-check the TokenScalar shape change.

@robobun

robobun commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

Note for whoever lands second: #44630 adds a guard to the radix branch of try_resolve_number that reads ScalarResolverCtx.tag (matches!(self.tag, NodeTag::None)), so that an untagged -0x1f stays a string and !!int -0x1f stays -31. This PR removes that field. src/parsers/yaml.rs merges without a text conflict, but the result does not compile. The port is small: the guard becomes unconditional in the scanner, and !!int keeps a signed hex or octal where the tag is applied (NodeTag::keeps). #44630 also has a test.todo for a: !!int 1 followed by the key -0x1f, which this PR makes pass.

@robobun

robobun commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Closing as stale: this PR conflicts with main, and its last commit is from 2026-08-13.

The branch stays, so the work is not lost. To pick it up again, rebase the branch on main and reopen this PR.

@robobun robobun closed this Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant