Skip to content

feat(trajectory): add recoverable tool timing recorder - #5508

Draft
lafrancef wants to merge 5 commits into
mainfrom
flafrance/tool-timing-recorder
Draft

lafrancef wants to merge 5 commits into
mainfrom
flafrance/tool-timing-recorder

Conversation

@lafrancef

@lafrancef lafrancef commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Work on this draft is deferred. The initial tool-timing delivery will retain available evidence and report explicit observability gaps for interrupted calls. Recorder recovery and the outstanding review findings remain follow-up work.

Add reusable start/finish capture for harnesses that need to retain tool evidence after interruption. Adapters stage tool_recording_files() with their existing session setup, record through ToolCallRecorder, and pass recovered observations to the common trajectory normalizer.

The stream preserves attempt identity, model references, explicit outcomes and monotonic duration. Starts are flushed before execution; replay diagnoses conflicting copies and damaged tails. Tool streams support recovery after harness process death while files remain available; model capture retains its existing fsync durability. Timing conformance checks keep lifecycle, executor and process claims independent. The probe runner accepts explicit runtime/mode/tool-path expectations through --tool-timing-scopes; suite reports retain those scopes and the timing-target inventory, including targets whose runtime probes are unavailable.

This introduces shared instrumentation helpers and does not enable additional harness hooks. Standalone staged-module checks exercised concurrent writers and a process exit after start. Each harness's adapter change supplies its runtime rollout evidence.

Depends on #5507; this draft targets its branch so the review shows only the shared recorder and conformance additions.

Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
@lafrancef

Copy link
Copy Markdown
Contributor Author

/review

@lafrancef lafrancef added area:core Shared APIs, servers, telemetry, health, and registries feature New capabilities, enhancements, or enablement work labels Nov 3, 2026
@copy-pr-bot

copy-pr-bot Bot commented Nov 3, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
@lafrancef
lafrancef force-pushed the flafrance/tool-timing-recorder branch from bc79e5f to 9a68262 Compare November 3, 2026 15:36
@lafrancef

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP WITH CARE

Four RISK findings affect interrupted-write isolation, replay recovery, and timing-scope validation. I inspected all 16 changed files and their directly affected callers, contracts, and tests.

This was a static, read-only review; tests weren't executed. The suggested patches need verification before adoption.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

with path.open("ab") as handle:
fcntl.flock(handle.fileno(), fcntl.LOCK_EX)
try:
handle.write(line)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — Isolate an interrupted append before the next producer writes

If a process dies partway through writing a record, the next producer appends its complete JSON directly to the unfinished bytes. For example, seed the file with b'{"schema_version":', then record another producer's start and finish. Its start becomes part of an invalid composite line, which the reader skips entirely. This loses a complete start from an unaffected producer sharing the capture file. The current process-death test exits only after start() returns; the truncated-tail test never appends afterward.

Could you separate an unterminated tail under the exclusive lock before appending another tool event? Preserve the damaged bytes for diagnostics. Keep this opt-in so model capture retains write-only access and its existing fsync behavior. Suggested replacement for append_jsonl:

def append_jsonl(
    path: Path, line: bytes, *, durable: bool = True, recover_tail: bool = False
) -> None:
    with path.open("a+b" if recover_tail else "ab") as handle:
        fcntl.flock(handle.fileno(), fcntl.LOCK_EX)
        try:
            if recover_tail:
                handle.seek(0, os.SEEK_END)
                if handle.tell():
                    handle.seek(-1, os.SEEK_END)
                    if handle.read(1) != b"\n":
                        handle.write(b"\n")
            handle.write(line)
            handle.flush()
            if durable:
                os.fsync(handle.fileno())
        finally:
            fcntl.flock(handle.fileno(), fcntl.LOCK_UN)

Pass recover_tail=True at the recorder append. The invariant is that one interrupted append cannot consume the next producer's complete record. This opt-in tool path requires read access to inspect its tail.

Proposed acceptance checks, not executed: seed a malformed unterminated tail, record a complete attempt, and assert recovery retains both endpoints plus an invalid-record gap. Also seed a complete JSON event without its final newline and assert it and the next event both survive. Retain the concurrent-writer sequence/count assertions in the recorder tests and the incomplete-start assertions in the truncated-tail test. Inspect that the model-store call still uses durable appends without tail-read access.

for name in ("started_at", "completed_at", "duration_ms"):
value = values.get(name)
if value is not None and (
type(value) not in (int, float) or not math.isfinite(value) or (name == "duration_ms" and value < 0)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — An oversized measurement aborts recovery of the entire stream

A JSON-valid finish event with duration_ms=10**400 makes math.isfinite(value) raise OverflowError before the validation handler. Consequently, read_tool_events() aborts instead of preserving the failed attempt's identity/outcome and recovering unrelated records. This affects damaged or externally produced captures; it doesn't require malformed JSON. The sanitizer explicitly intends to salvage invalid timing.

Could you treat unrepresentable numbers as invalid timing? Suggested replacement for the numeric sanitization loop:

for name in ("started_at", "completed_at", "duration_ms"):
    value = values.get(name)
    if value is None:
        continue
    try:
        invalid = (
            type(value) not in (int, float)
            or not math.isfinite(value)
            or (name == "duration_ms" and value < 0)
        )
    except OverflowError:
        invalid = True
    if invalid:
        values[name] = None
        invalid_timing = True

The invariant is that malformed timing cannot suppress recoverable identity, explicit outcomes, or other attempts. Preserve rejection of booleans, negative durations, and nonfinite floats while retaining zero and ordinary finite integer/float measurements.

Proposed acceptance checks, not executed: parameterize the existing event fixture with 10**400 in each numeric timing field and include another valid attempt. Both replay and file-based reading should return both attempts, retain the affected attempt's explicit failed outcome, clear only the oversized measurement, and emit tool_timing_invalid. Retain the existing assertions that invalid timing preserves the failed outcome and that a zero-duration finish remains useful.

domain = start.clock_domain if start else finish.clock_domain if finish else None
end = finish.completed_at if finish else None
duration = finish.duration_ms if finish else None
if start and finish and (start.clock_domain is None or start.clock_domain != finish.clock_domain):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — A start without a usable timestamp discards the valid finish endpoint

Using the existing fixture, replay_tool_events([event("start", started_at=None, clock_domain=None), event("finish")]) clears the valid finish timestamp 102 and selects a null clock domain. Both records satisfy the event model. The reconciliation branch treats a start record as an endpoint even when no start timestamp exists. This also loses the finish endpoint when a damaged start timestamp is sanitized to null.

Could you retain the single usable finish endpoint under its own domain? Replace the domain selection with:

has_start_endpoint = start is not None and start.started_at is not None
domain = (
    start.clock_domain
    if has_start_endpoint
    else finish.clock_domain if finish is not None else None
)

Within the existing clock-reconciliation branch, replace unconditional end = None with:

if has_start_endpoint:
    end = None

Leave its wall-clock-duration rejection and diagnostics intact. This restores the invariant that a single usable endpoint survives without combining incomparable endpoints. Two usable endpoints with mismatched or unknown domains must still not be combined; monotonic/reported durations must survive, while wall-clock durations across incomparable domains must still be discarded.

Proposed acceptance checks, not executed: for the trigger above, assert null started_at, completed_at == 102, the finish's clock domain, and unchanged status/monotonic duration. Repeat with a nonfinite start timestamp and require tool_timing_invalid without losing the finish endpoint. With a missing start timestamp but differing domains and a wall-clock duration, retain the finish endpoint but discard that duration. Retain the existing mismatched-domain assertions for usable start timestamps and all three duration sources, plus the finish-without-start zero-duration case.

executor_sources: tuple[str, ...] = ()
process_attribution: bool = False

def __post_init__(self) -> None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — Reject scalar source declarations before checking membership

The JSON loader passes values directly into this dataclass, but __post_init__ doesn't validate either source collection. A scope with tool_paths: ["bash"] and lifecycle_sources: "not-native" is accepted. A bash measurement with timing_source: "native", duration_ms: 2.0, and duration_source: "monotonic" then passes the source check, because string membership checks substrings. This incorrectly reports authoritative timing evidence from an undeclared source. It affects conformance results loaded from malformed scope JSON, not tool execution.

Could you validate both source fields as collections of nonempty strings? Add this to ToolTimingScope.__post_init__, retaining its required-field check:

for name in ("lifecycle_sources", "executor_sources"):
    sources = getattr(self, name)
    if not isinstance(sources, (list, tuple)) or not all(
        isinstance(source, str) and bool(source.strip()) for source in sources
    ):
        raise ValueError(f"{name} must be a list or tuple of nonempty strings")

This restores exact-name matching while preserving JSON lists, Python tuples, and empty collections meaning an unclaimed boundary. It leaves duration-only executor evidence and lifecycle/executor independence unchanged. The existing configuration test explicitly expects a loaded list, so no tuple conversion is needed.

Proposed acceptance checks, not executed: load the scalar "not-native" for each source field and assert ValueError before probe launch. With ["not-native"], assert a "native" measurement fails; with ["native"], assert it passes. Retain the existing assertions that empty executor sources are not applicable, lifecycle evidence cannot satisfy executor timing, and zero-duration executor evidence passes.

@lafrancef
lafrancef force-pushed the flafrance/tool-timing-opencode-v2 branch 14 times, most recently from bfc9ff5 to 4935a69 Compare November 9, 2026 15:22
Base automatically changed from flafrance/tool-timing-opencode-v2 to main November 10, 2026 13:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core Shared APIs, servers, telemetry, health, and registries feature New capabilities, enhancements, or enablement work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants