Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions changelog.d/6929-gc-root-word-mark-rewrite-parity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
**GC hardening — the mutable-root mark path and the rewrite path now accept the same word forms (#6910).**

A GC root slot can carry a heap reference NaN-boxed (`POINTER_TAG` / `STRING_TAG` / `BIGINT_TAG`) or **bare** — the untagged user address written straight into the word. Four hand-copied predicates decided that question four ways: `try_mark_value` (shadow-slot root marking) accepted NaN-boxed tags only, while `try_rewrite_value` (mutable-root + heap-slot rewrite), `CopyingPointerSet::decode_bits` (copying minor) and `heap_word_candidate_addr` (incremental mark barrier) all accepted bare addresses too. A bare address in a shadow-stack slot was therefore a root to three of the four surfaces and invisible to the fourth.

The concrete harm is a use-after-free, not a missed rewrite: an unmarked object is never evacuated, so it never gets a forwarding address for the rewrite pass to follow. If such a slot was the only reference, mark skipped it, evacuation skipped it, rewrite found nothing to do, and **the sweep freed it**, leaving the slot pointing at reclaimed memory. The failure would also have been invisible to CI — the unit-test build defaults to `ConservativeStackScanMode::Full`, whose native-stack scan accepts bare addresses and would have rescued the object, while production resolves `Auto -> SkipDisabled` and has no such net. Green tests, production-only UAF.

Latent today: no shipped representation puts a bare address in a shadow slot (every phase deliberately chose tagged-at-rest). Fixed rather than documented because the representation-selection RFC §5.6 unboxed pointer reps (`Ptr<Shape>`, `Str`, closure envs) are designed to store exactly that form.

Fix: a single decoder, `gc/root_words.rs`. `decode_root_word` returns `RootWord::{Nanboxed, Bare}`, and `mark_mutable_root_bits`, `try_rewrite_value` and `heap_word_candidate_addr` all route through it, so adding a representation teaches all three at once. `RootWord::encode` preserves the stored form across a rewrite (bare stays bare, a NaN box keeps its tag). `mark_global_root_bits` folds into `mark_mutable_root_bits` — both slot kinds share one decoding and `MutableRootSlotKind` now selects a telemetry bucket only. Marking is address-identical for the two forms once decoded, so the hot path pays no extra lookup.

`try_rewrite_value` additionally picks up the arm64_32 (ILP32) payload guard `heap_word_candidate_addr` already carried; on watchOS it was truncating 48-bit payloads to 32-bit addresses. Inert on 64-bit targets.

Two surfaces deliberately stay outside the shared decoder, both safe because they are supersets rather than subsets: `try_mark_value_or_raw` (conservative native-stack scan, also resolves interior pointers) and `CopyingPointerSet::decode_bits` (copying minor — additionally requires 8-byte alignment and a live classification, and its mark and rewrite are the same operation so it cannot disagree with itself). The tagged-at-rest/bare contract is documented on `js_shadow_slot_set`, including the standing prohibition on *interior* pointers in a slot.

Tests: `gc::tests::root_words`, four tests, all pinning `ConservativeStackScanMode::Disabled` because the `Full` default masks exactly this bug. `bare_address_in_shadow_slot_survives_a_real_collection` and `mutable_root_mark_and_rewrite_accept_the_same_word_forms` were both verified failing on the pre-fix mark path and passing after; the latter drives `mark_mutable_root_slots` / `rewrite_mutable_root_slots` themselves over both slot kinds, so a call site abandoning the shared decoder also fails it. Green across the full GC stress matrix: `PERRY_GC_FORCE_EVACUATE=1`, `PERRY_GC_VERIFY_EVACUATION=1`, `PERRY_GEN_GC=0`, `PERRY_WRITE_BARRIERS=0` and four combinations.
35 changes: 7 additions & 28 deletions crates/perry-runtime/src/gc/barrier.rs
Original file line number Diff line number Diff line change
Expand Up @@ -854,35 +854,14 @@ pub(super) fn incremental_mark_barrier_active() -> bool {
}

#[inline]
/// The address an incremental-mark barrier must shade for a stored word,
/// NaN-boxed or bare.
///
/// Shares [`decode_root_word`] with the mark and rewrite paths (#6910): a
/// word form the barrier shades but root marking skips (or vice versa) is
/// exactly the kind of drift that module exists to prevent.
pub(super) fn heap_word_candidate_addr(bits: u64) -> Option<usize> {
let tag = bits & TAG_MASK;
if tag == POINTER_TAG || tag == STRING_TAG || tag == BIGINT_TAG {
let payload = bits & POINTER_MASK;
// arm64_32 (ILP32): a real heap pointer fits in 32 bits, so `payload as
// usize` is lossless only when the high 16 payload bits are zero. A
// mistagged/immediate value with a >32-bit payload would otherwise
// truncate to a garbage 32-bit address that the GC marks/derefs,
// corrupting unrelated heap memory. Reject it.
#[cfg(not(target_pointer_width = "64"))]
if payload > 0xFFFF_FFFF {
return None;
}
let ptr = payload as usize;
return (ptr != 0).then_some(ptr);
}
if tag >= 0x7FF8_0000_0000_0000 {
return None;
}
if (0x1000..=0x0000_FFFF_FFFF_FFFF).contains(&bits) {
// Same ILP32 guard for the raw-f64 pointer path.
#[cfg(not(target_pointer_width = "64"))]
if bits > 0xFFFF_FFFF {
return None;
}
Some(bits as usize)
} else {
None
}
decode_root_word(bits).map(RootWord::addr)
}

#[inline]
Expand Down
5 changes: 5 additions & 0 deletions crates/perry-runtime/src/gc/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,11 @@ mod malloc;
pub use malloc::*;
mod roots;
pub use roots::*;
// The one decoder shared by the mark, rewrite and incremental-barrier paths
// for words that may hold a heap reference (#6910). Declared before its
// consumers for readability only — Rust module order is irrelevant.
mod root_words;
use root_words::*;
mod layout;
pub use layout::*;
mod trace;
Expand Down
162 changes: 162 additions & 0 deletions crates/perry-runtime/src/gc/root_words.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
//! The single decoder shared by the collector's *mark* and *rewrite* paths
//! for words that may hold a heap reference (#6910).
//!
//! # The invariant
//!
//! **Anything the rewrite path will relocate, the mark path must mark.**
//!
//! A GC root slot or heap word can carry a heap reference in two forms:
//!
//! - **NaN-boxed** — `POINTER_TAG` / `STRING_TAG` / `BIGINT_TAG` in the top
//! 16 bits, user address in the low 48. This is what all shipped codegen
//! stores in a shadow-stack slot ("tagged at rest").
//! - **Bare** — the untagged user address written straight into the word.
//! Module-variable globals have always done this, runtime side tables that
//! hold `*mut T` do it, and the representation-selection RFC's unboxed
//! pointer reps (§5.6 `Ptr<Shape>`, `Str`, closure envs) are designed to.
//!
//! Before #6910 the two forms were decoded by four hand-copied predicates
//! that did not agree:
//!
//! | surface | accepted |
//! |---|---|
//! | `try_mark_value` (fed by shadow-stack root marking) | NaN-boxed only |
//! | `try_rewrite_value` (mutable-root + heap-slot rewrite) | NaN-boxed **and bare** |
//! | `CopyingPointerSet::decode_bits` (copying minor) | NaN-boxed **and bare** |
//! | `heap_word_candidate_addr` (incremental mark barrier) | NaN-boxed **and bare** |
//!
//! So a bare address in a shadow-stack slot was accepted by the rewrite,
//! copying and barrier paths but **invisible to mark-sweep root marking**.
//! If such a slot were the only reference to an object, the mark phase would
//! skip it, the sweep would free it, and the slot would be left pointing at
//! reclaimed memory — a use-after-free, not merely a missed rewrite (an
//! unmarked object is never evacuated, so it never gets a forwarding address
//! for the rewrite pass to follow).
//!
//! The failure would also have been invisible to the test suite: the unit
//! tests default to `ConservativeStackScanMode::Full`
//! (`gc::roots::conservative_stack_scan_mode`), whose native-stack scan
//! accepts bare addresses and would have rescued the object, while
//! production resolves `Auto -> SkipDisabled` and has no such safety net.
//! Green tests, prod-only UAF. `gc::tests::root_words` therefore pins the
//! invariant with the conservative scan explicitly `Disabled`.
//!
//! # Using this module
//!
//! Decode through [`decode_root_word`] and act on the result; do not re-type
//! the tag comparisons or the address-range literals at a call site. Adding a
//! representation here teaches the mark path, the rewrite path and the
//! incremental barrier at once, which is the property that keeps them from
//! drifting apart again.
//!
//! Two collector surfaces deliberately do **not** route through here, and
//! both are safe because they are *supersets* of this decoder rather than
//! subsets:
//!
//! - `try_mark_value_or_raw` — the conservative native-stack scan, which
//! additionally resolves *interior* pointers via `enclosing_object`.
//! - `CopyingPointerSet::decode_bits` — the copying minor collector, which
//! additionally requires 8-byte alignment and a live classification, and
//! whose mark and rewrite are the same operation (`visit_value_bits`), so
//! it cannot disagree with itself.

use super::*;

/// Exclusive floor of the NaN-tag space. Every bit pattern at or above this
/// is a NaN-boxed immediate (number, boolean, `undefined`/`null`, SSO
/// string, INT32, handle) or one of the three heap tags — never a bare
/// address.
const NAN_TAG_FLOOR: u64 = 0x7FF8_0000_0000_0000;

/// Inclusive floor for a bare heap address. Below the first page nothing is
/// mappable, so a smaller word is a small integer or null, not a pointer.
const BARE_ADDR_MIN: u64 = 0x1000;

/// A word that decoded to a heap reference, in the form it was stored in.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(super) enum RootWord {
/// NaN-boxed reference. `tag` is `POINTER_TAG`, `STRING_TAG` or
/// `BIGINT_TAG` and must be re-applied when the address is rewritten.
Nanboxed { addr: usize, tag: u64 },
/// Bare (untagged) user address stored directly in the word.
Bare { addr: usize },
}

impl RootWord {
/// The user address the collector must mark / relocate.
#[inline]
pub(super) fn addr(self) -> usize {
match self {
Self::Nanboxed { addr, .. } | Self::Bare { addr } => addr,
}
}

/// Re-encode `new_addr` in this word's original form, so a rewrite
/// preserves tagged-ness rather than converting between the two.
#[inline]
pub(super) fn encode(self, new_addr: usize) -> u64 {
match self {
Self::Nanboxed { tag, .. } => tag | (new_addr as u64 & POINTER_MASK),
Self::Bare { .. } => new_addr as u64,
}
}
}

/// Decode a root-slot / heap word into the heap reference it carries, if any.
///
/// This is the one predicate behind [`mark_mutable_root_bits`],
/// `try_rewrite_value` and `heap_word_candidate_addr`; see the module docs
/// for why they must not diverge.
#[inline]
pub(super) fn decode_root_word(bits: u64) -> Option<RootWord> {
let tag = bits & TAG_MASK;
if tag == POINTER_TAG || tag == STRING_TAG || tag == BIGINT_TAG {
let payload = bits & POINTER_MASK;
// arm64_32 (ILP32 — watchOS): a real heap pointer fits in 32 bits, so
// `payload as usize` is lossless only when the high 16 payload bits
// are zero. A mistagged / immediate value with a >32-bit payload
// would otherwise truncate to a garbage 32-bit address that the GC
// marks, derefs, or rewrites — corrupting unrelated heap memory.
#[cfg(not(target_pointer_width = "64"))]
if payload > 0xFFFF_FFFF {
return None;
}
let addr = payload as usize;
return (addr != 0).then_some(RootWord::Nanboxed { addr, tag });
}
// Not a heap tag. Everything else in the NaN-tag space is an immediate.
if tag >= NAN_TAG_FLOOR {
return None;
}
// Bare address: must fit the 48-bit user-address range. Plain `f64`
// payloads and raw integers land outside it and are rejected here.
if !(BARE_ADDR_MIN..=POINTER_MASK).contains(&bits) {
return None;
}
// Same ILP32 guard for the bare form.
#[cfg(not(target_pointer_width = "64"))]
if bits > 0xFFFF_FFFF {
return None;
}
Some(RootWord::Bare {
addr: bits as usize,
})
}

/// Mark the object referenced by a mutable-root slot word — shadow-stack
/// slots (`MutableRootSlotKind::ShadowStack`) and registered module-global
/// roots (`MutableRootSlotKind::GlobalRoot`) alike.
///
/// Both slot kinds are rewritten by `rewrite_mutable_root_slots` through
/// `try_rewrite_value`, so both must be marked through the same decoder
/// (#6910). Marking is address-identical for the two forms once decoded —
/// `try_mark_raw_root_addr` performs the same validation and mark that
/// `try_mark_value` does after unwrapping a NaN box — so a single call
/// covers them.
#[inline]
pub(super) fn mark_mutable_root_bits(bits: u64, valid_ptrs: &ValidPointerSet) {
let Some(word) = decode_root_word(bits) else {
return;
};
try_mark_raw_root_addr(word.addr(), valid_ptrs);
}
35 changes: 13 additions & 22 deletions crates/perry-runtime/src/gc/roots.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1442,6 +1442,13 @@ pub(super) fn atomic_store_ordering(
}
}

/// Which registry a mutable root slot came from.
///
/// The kind selects a *telemetry bucket* only — it must never select a
/// different pointer decoding. Both kinds are marked by
/// `mark_mutable_root_bits` and rewritten by `try_rewrite_value`, and both
/// therefore accept a heap reference either NaN-boxed or bare. That symmetry
/// is the #6910 invariant; see `gc::root_words`.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(super) enum MutableRootSlotKind {
ShadowStack,
Expand Down Expand Up @@ -1569,7 +1576,7 @@ pub(super) fn mark_mutable_root_slots_step(
stats.record_scan(bits);
}
if bits != 0 {
try_mark_value(bits, valid_ptrs);
mark_mutable_root_bits(bits, valid_ptrs);
}
seen += 1;
cursor.shadow_seen = seen;
Expand Down Expand Up @@ -1600,7 +1607,7 @@ pub(super) fn mark_mutable_root_slots_step(
let bits = slot.read();
record_mutable_slot_scan_source(slot, bits, valid_ptrs, &mut root_sources);
if bits != 0 {
mark_global_root_bits(bits, valid_ptrs);
mark_mutable_root_bits(bits, valid_ptrs);
}
seen += 1;
cursor.global_seen = seen;
Expand Down Expand Up @@ -1677,21 +1684,10 @@ pub(super) fn record_mutable_slot_rewrite_source(
}
}

#[inline]
pub(super) fn mark_global_root_bits(bits: u64, valid_ptrs: &ValidPointerSet) {
// First try NaN-boxed interpretation (exported globals, closures, etc.).
if try_mark_value(bits, valid_ptrs) {
return;
}
// Module variable globals store raw I64 pointers (not NaN-boxed).
// Preserve the historical direct-object-start behavior: validate
// against valid_ptrs and mark the target, without the conservative
// interior-pointer fallback used by stack scanning.
let raw_ptr = bits as usize;
try_mark_raw_root_addr(raw_ptr, valid_ptrs);
}

/// Mark mutable roots (shadow-stack slots and registered globals).
///
/// Both slot kinds go through `mark_mutable_root_bits`, which shares its
/// decoder with the rewrite path (#6910) — see `gc::root_words`.
#[allow(dead_code)]
pub(super) fn mark_mutable_root_slots(
valid_ptrs: &ValidPointerSet,
Expand All @@ -1709,12 +1705,7 @@ pub(super) fn mark_mutable_root_slots(
if bits == 0 {
return;
}
match slot.kind {
MutableRootSlotKind::ShadowStack => {
try_mark_value(bits, valid_ptrs);
}
MutableRootSlotKind::GlobalRoot => mark_global_root_bits(bits, valid_ptrs),
}
mark_mutable_root_bits(bits, valid_ptrs);
});
}

Expand Down
18 changes: 17 additions & 1 deletion crates/perry-runtime/src/gc/roots/shadow_stack.rs
Original file line number Diff line number Diff line change
Expand Up @@ -109,11 +109,27 @@ pub extern "C" fn js_shadow_frame_pop(frame_handle: u64) {
});
}

/// Update slot `idx` in the current frame with NaN-boxed `value`.
/// Update slot `idx` in the current frame with `value`.
/// Codegen emits this at safepoints for each live pointer-typed
/// local. Hot path — compiled code calls this directly or inlines
/// an equivalent sequence; Rust version exists for runtime tests
/// and debug builds.
///
/// # Slot value contract (#6910)
///
/// A slot may hold a heap reference either **NaN-boxed** (`POINTER_TAG` /
/// `STRING_TAG` / `BIGINT_TAG`, what all shipped codegen emits — "tagged at
/// rest") or **bare** (the untagged user address, what the
/// representation-selection RFC §5.6 unboxed pointer reps are designed to
/// store). Both are marked and both are rewritten, because both mark
/// (`mark_mutable_root_bits`) and rewrite (`try_rewrite_value`) decode
/// through `gc::root_words::decode_root_word`. If you ever add a third
/// in-slot representation, teach that decoder — never one path only.
///
/// What a slot must NOT hold is an *interior* pointer: root marking resolves
/// object starts, so a derived address (typed-array data pointer, `arr + 8`)
/// is only safe region-locally between safepoints and must be recomputed
/// from the rewritten header after each one.
#[no_mangle]
pub extern "C" fn js_shadow_slot_set(idx: u32, value: u64) {
SHADOW.with(|cell| unsafe {
Expand Down
1 change: 1 addition & 0 deletions crates/perry-runtime/src/gc/tests/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ mod incremental_sweep_reclaim;
mod layout_trace;
mod oldgen;
mod os_tag;
mod root_words;
mod roots;
mod runtime_roots;
mod smoke;
Expand Down
Loading
Loading