Repository navigation
Conversation
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
|
/review |
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
bc79e5f to
9a68262
Compare
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP WITH CARE
Four RISK findings affect interrupted-write isolation, replay recovery, and timing-scope validation. I inspected all 16 changed files and their directly affected callers, contracts, and tests.
This was a static, read-only review; tests weren't executed. The suggested patches need verification before adoption.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
| with path.open("ab") as handle: | ||
| fcntl.flock(handle.fileno(), fcntl.LOCK_EX) | ||
| try: | ||
| handle.write(line) |
There was a problem hiding this comment.
RISK — Isolate an interrupted append before the next producer writes
If a process dies partway through writing a record, the next producer appends its complete JSON directly to the unfinished bytes. For example, seed the file with b'{"schema_version":', then record another producer's start and finish. Its start becomes part of an invalid composite line, which the reader skips entirely. This loses a complete start from an unaffected producer sharing the capture file. The current process-death test exits only after start() returns; the truncated-tail test never appends afterward.
Could you separate an unterminated tail under the exclusive lock before appending another tool event? Preserve the damaged bytes for diagnostics. Keep this opt-in so model capture retains write-only access and its existing fsync behavior. Suggested replacement for append_jsonl:
def append_jsonl(
path: Path, line: bytes, *, durable: bool = True, recover_tail: bool = False
) -> None:
with path.open("a+b" if recover_tail else "ab") as handle:
fcntl.flock(handle.fileno(), fcntl.LOCK_EX)
try:
if recover_tail:
handle.seek(0, os.SEEK_END)
if handle.tell():
handle.seek(-1, os.SEEK_END)
if handle.read(1) != b"\n":
handle.write(b"\n")
handle.write(line)
handle.flush()
if durable:
os.fsync(handle.fileno())
finally:
fcntl.flock(handle.fileno(), fcntl.LOCK_UN)Pass recover_tail=True at the recorder append. The invariant is that one interrupted append cannot consume the next producer's complete record. This opt-in tool path requires read access to inspect its tail.
Proposed acceptance checks, not executed: seed a malformed unterminated tail, record a complete attempt, and assert recovery retains both endpoints plus an invalid-record gap. Also seed a complete JSON event without its final newline and assert it and the next event both survive. Retain the concurrent-writer sequence/count assertions in the recorder tests and the incomplete-start assertions in the truncated-tail test. Inspect that the model-store call still uses durable appends without tail-read access.
| for name in ("started_at", "completed_at", "duration_ms"): | ||
| value = values.get(name) | ||
| if value is not None and ( | ||
| type(value) not in (int, float) or not math.isfinite(value) or (name == "duration_ms" and value < 0) |
There was a problem hiding this comment.
RISK — An oversized measurement aborts recovery of the entire stream
A JSON-valid finish event with duration_ms=10**400 makes math.isfinite(value) raise OverflowError before the validation handler. Consequently, read_tool_events() aborts instead of preserving the failed attempt's identity/outcome and recovering unrelated records. This affects damaged or externally produced captures; it doesn't require malformed JSON. The sanitizer explicitly intends to salvage invalid timing.
Could you treat unrepresentable numbers as invalid timing? Suggested replacement for the numeric sanitization loop:
for name in ("started_at", "completed_at", "duration_ms"):
value = values.get(name)
if value is None:
continue
try:
invalid = (
type(value) not in (int, float)
or not math.isfinite(value)
or (name == "duration_ms" and value < 0)
)
except OverflowError:
invalid = True
if invalid:
values[name] = None
invalid_timing = TrueThe invariant is that malformed timing cannot suppress recoverable identity, explicit outcomes, or other attempts. Preserve rejection of booleans, negative durations, and nonfinite floats while retaining zero and ordinary finite integer/float measurements.
Proposed acceptance checks, not executed: parameterize the existing event fixture with 10**400 in each numeric timing field and include another valid attempt. Both replay and file-based reading should return both attempts, retain the affected attempt's explicit failed outcome, clear only the oversized measurement, and emit tool_timing_invalid. Retain the existing assertions that invalid timing preserves the failed outcome and that a zero-duration finish remains useful.
| domain = start.clock_domain if start else finish.clock_domain if finish else None | ||
| end = finish.completed_at if finish else None | ||
| duration = finish.duration_ms if finish else None | ||
| if start and finish and (start.clock_domain is None or start.clock_domain != finish.clock_domain): |
There was a problem hiding this comment.
RISK — A start without a usable timestamp discards the valid finish endpoint
Using the existing fixture, replay_tool_events([event("start", started_at=None, clock_domain=None), event("finish")]) clears the valid finish timestamp 102 and selects a null clock domain. Both records satisfy the event model. The reconciliation branch treats a start record as an endpoint even when no start timestamp exists. This also loses the finish endpoint when a damaged start timestamp is sanitized to null.
Could you retain the single usable finish endpoint under its own domain? Replace the domain selection with:
has_start_endpoint = start is not None and start.started_at is not None
domain = (
start.clock_domain
if has_start_endpoint
else finish.clock_domain if finish is not None else None
)Within the existing clock-reconciliation branch, replace unconditional end = None with:
if has_start_endpoint:
end = NoneLeave its wall-clock-duration rejection and diagnostics intact. This restores the invariant that a single usable endpoint survives without combining incomparable endpoints. Two usable endpoints with mismatched or unknown domains must still not be combined; monotonic/reported durations must survive, while wall-clock durations across incomparable domains must still be discarded.
Proposed acceptance checks, not executed: for the trigger above, assert null started_at, completed_at == 102, the finish's clock domain, and unchanged status/monotonic duration. Repeat with a nonfinite start timestamp and require tool_timing_invalid without losing the finish endpoint. With a missing start timestamp but differing domains and a wall-clock duration, retain the finish endpoint but discard that duration. Retain the existing mismatched-domain assertions for usable start timestamps and all three duration sources, plus the finish-without-start zero-duration case.
| executor_sources: tuple[str, ...] = () | ||
| process_attribution: bool = False | ||
|
|
||
| def __post_init__(self) -> None: |
There was a problem hiding this comment.
RISK — Reject scalar source declarations before checking membership
The JSON loader passes values directly into this dataclass, but __post_init__ doesn't validate either source collection. A scope with tool_paths: ["bash"] and lifecycle_sources: "not-native" is accepted. A bash measurement with timing_source: "native", duration_ms: 2.0, and duration_source: "monotonic" then passes the source check, because string membership checks substrings. This incorrectly reports authoritative timing evidence from an undeclared source. It affects conformance results loaded from malformed scope JSON, not tool execution.
Could you validate both source fields as collections of nonempty strings? Add this to ToolTimingScope.__post_init__, retaining its required-field check:
for name in ("lifecycle_sources", "executor_sources"):
sources = getattr(self, name)
if not isinstance(sources, (list, tuple)) or not all(
isinstance(source, str) and bool(source.strip()) for source in sources
):
raise ValueError(f"{name} must be a list or tuple of nonempty strings")This restores exact-name matching while preserving JSON lists, Python tuples, and empty collections meaning an unclaimed boundary. It leaves duration-only executor evidence and lifecycle/executor independence unchanged. The existing configuration test explicitly expects a loaded list, so no tuple conversion is needed.
Proposed acceptance checks, not executed: load the scalar "not-native" for each source field and assert ValueError before probe launch. With ["not-native"], assert a "native" measurement fails; with ["native"], assert it passes. Retain the existing assertions that empty executor sources are not applicable, lifecycle evidence cannot satisfy executor timing, and zero-duration executor evidence passes.
bfc9ff5 to
4935a69
Compare
Add reusable start/finish capture for harnesses that need to retain tool evidence after interruption. Adapters stage
tool_recording_files()with their existing session setup, record throughToolCallRecorder, and pass recovered observations to the common trajectory normalizer.The stream preserves attempt identity, model references, explicit outcomes and monotonic duration. Starts are flushed before execution; replay diagnoses conflicting copies and damaged tails. Tool streams support recovery after harness process death while files remain available; model capture retains its existing fsync durability. Timing conformance checks keep lifecycle, executor and process claims independent. The probe runner accepts explicit runtime/mode/tool-path expectations through
--tool-timing-scopes; suite reports retain those scopes and the timing-target inventory, including targets whose runtime probes are unavailable.This introduces shared instrumentation helpers and does not enable additional harness hooks. Standalone staged-module checks exercised concurrent writers and a process exit after start. Each harness's adapter change supplies its runtime rollout evidence.
Depends on #5507; this draft targets its branch so the review shows only the shared recorder and conformance additions.