Spec: Session traces, OTLP JSON Lines in place of Session logs #700

From the #698 grilling. The decision record is .scratch/grill-with-docs/done/698-evals/: see its Settled list and Q2.1, Q3.14–Q3.19, Q4.10–Q4.12, Q4.14, Q5.5, Q5.8–Q5.10 and Q7.2. This Spec builds what ADR 0019 decided. It is the first of #698's four Specs: Session traces, Experiments (#977), the Scoring run (#978) and Run metrics (#981).

Problem Statement

Each Harness prints its own stream of events, and thirdshift saves each session's stream as its Session log, in that Harness's own format. Six Harnesses means six formats, and nothing reads them all:

• the Console's conversation view renders only Claude's and Codex's events;
• the Waterfall guesses each tool call's time by matching its label to a Command log line, to the second;
• show re-reads the end of a Session log and interprets it again to find the latest event;
• no session records its model calls' tokens or cost, or when each model call and tool call began and ended;
• a Codex edit shows "Patch unavailable in this log.", since Codex's stream says which files changed but not how (Decide Codex patch evidence retention for Console edit diffs #854).

#698's evals need all of that. Experiments grade Trials and report their efficiency, the Scoring run samples production work, and Run metrics track tokens, cost, tool errors and killed work by Harness and Model. Each needs every session's model calls, tool calls, tokens and times, from every Harness, tied to the Command and Run that made them. No eval tool reads Session logs as they are, and you want one standard format rather than six.

Solution

Each session writes a Session trace in place of its Session log: two OpenTelemetry (OTLP) JSON Lines files where its Session log is today.

• Log records, one for each Harness event, written the moment it arrives. They keep the Console's live view and Silent working as they do now.
• Spans, one for the session and one for each model call and tool call, written as each ends. They carry times, tokens, cost and the rest in OpenTelemetry's GenAI conventions, with thirdshift.* attributes for what those lack.

A Trace is one Command's whole tree. All of a Command's sessions share its trace ID. The Command, its Runs and their stages come from the Command record, which already holds them, so nothing is written twice. The Console puts a whole Trace together, and an export command writes it out as OTLP for any other tool.

The Console reads Session traces for all six Harnesses:

• its conversation view renders them;
• the Waterfall takes real tool times from them;
• show reports the latest record;
• Silent and the heartbeat keep watching the files grow.

A span tree grows out of the Waterfall, designed first in a /prototype session. Codex's edits get their diffs from Codex's own session file. Every Session log already on a machine is converted once into a Session trace, with the same code that writes them live, and the originals stay until you've checked the result.

User Stories

Recording:

1. As the Day shift, I want every session of every Harness recorded in one standard format, so that I can read and compare any session the same way.
2. As the Day shift, I want each Harness event written to its Session trace the moment it arrives, so that the Console's live view keeps up with a running session.
3. As the Day shift, I want a session's file to keep growing during a long tool call, such as a 20-minute build, as its Session log does now, so that a working Run never looks Silent.
4. As the Day shift, I want each model call's and tool call's start and end recorded, so that I can see where a session spent its time.
5. As the Day shift, I want each model call's tokens recorded by Model, with the cache split, so that Run metrics can show what a Run used.
6. As the Day shift, I want a session's cost recorded where its Harness reports one, so that cost needs no price table.
7. As the Day shift, I want rate limits and usage-limit errors recorded, so that I can see when a session hit its plan's limit.
8. As the Day shift, I want permission denials, killed or interrupted work and sub-agents' lifecycles recorded, so that evals can count them.
9. As the Day shift, I want diffs, pushes and pull-request events recorded with the session that made them, so that a Trace shows what each session changed.
10. As the Day shift, I want the Harness's setup and the Session prompt recorded with each session, so that I can see exactly what a session was told, which no Claude Session log holds today.
11. As the Day shift, I want each record to keep the raw Harness event, so that nothing a Session log holds today is lost.
12. As the Day shift, I want the Harness's version recorded with each session, so that the Scoring run can show a drop after a Harness updates itself.
13. As the Day shift, I want the requested Harness, Model and Effort, and the Models a session was seen to use, recorded with the session, so that a Trace says what ran it.
14. As the Day shift, I want each Session trace to say which version of the GenAI conventions it follows, so that a later change to the conventions can't be misread.
15. As the Day shift, I want an interrupted session's open spans closed with an error status, so that a stopped session still reads as a whole.
16. As the Day shift, I want a hard-killed session's records already on disk, so that a power cut loses only what was in flight.
17. As the Day shift, I want each file to hold one type of data, as the OTLP file format requires, so that any tool that reads the format can read them as they are.

The Trace:

18. As the Day shift, I want all of a Command's sessions to share one trace ID, its child Runs' included, so that a Command's whole tree reads as one Trace.
19. As the Day shift, I want each session's spans to name their Run and stage as parents, using IDs computed from the Command record, so that a Trace's upper levels need no guessing and no second copy of the stage times.
20. As the Day shift, I want an export command that writes a Command's whole Trace as OTLP, its Command, Run and stage spans included, so that I can load it into the Collector, Tempo, MLflow or anything else that reads OTLP.
21. As the Day shift, I want spans also sent to an OTLP endpoint when I set one in the User config, so that a local viewer can receive them live.
22. As the Day shift, I want sending to that endpoint to be best effort, so that a viewer that is down never fails a session.
23. As the Day shift, I want a Security run's Session traces kept on the machine and never sent to an endpoint, so that ADR 0015's rule holds for traces too.

The Console and the CLI:

24. As the Day shift, I want the Console's conversation view to render sessions from all six Harnesses, so that an agy, Grok, Muse or OpenCode session reads like a Claude or Codex one.
25. As the Day shift, I want the Console's live view to stream a Session trace by byte offset, as it streams Session logs now, so that a reconnect resumes where it left off.
26. As the Day shift, I want the Waterfall to take each tool call's time from the Session trace, so that its bars are exact rather than matched to the nearest second of a Command log line.
27. As the Day shift, I want a span tree that grows out of the Waterfall, with each span's details in the Peek panel, so that I can open any model call or tool call.
28. As the Day shift, I want that span tree designed in a prototype before it's built, so that its layout is settled before code depends on it.
29. As the Day shift, I want show to report the live session's latest record, so that a quiet terminal tells me what a Run is doing.
30. As the Day shift, I want Silent and the heartbeat to watch the log records' file, so that they mean what they mean today.
31. As the Day shift, I want the Logs page and logs --json to list each session once, so that a session's two files don't count as two sessions.
32. As the Day shift, I want Session traces of a Security run shown only behind the Console's sign-in, on the machine that ran it, as Spec: the Console, a web page for each machine's Commands #776 has it for Session logs.

Codex's diffs:

33. As the Day shift, I want each Codex edit to show the diff Codex applied, so that I can see what a Codex session changed (Decide Codex patch evidence retention for Console edit diffs #854).
34. As the Day shift, I want a Codex edit made through a shell command, or one whose diff can't be found, to show as it does today, so that the Console never shows a guessed diff.
35. As the Day shift, I want a Codex diff marked as Codex's own record of the edit, so that I know where it came from.

Converting old Session logs:

36. As the Day shift, I want every Session log on a machine converted once into a Session trace, so that the Console needs one renderer and Run metrics start with history.
37. As the Day shift, I want the conversion to use the same code that writes Session traces live, so that a converted session and a new one look the same.
38. As the Day shift, I want each host to convert its own Session logs, Security runs' included, so that nothing is copied between machines.
39. As the Day shift, I want a converted record with no time of its own to carry one estimated from the Command log, marked as estimated, so that I can tell real times from estimates.
40. As the Day shift, I want old Codex sessions to get their diffs in the conversion, where Codex's session files still exist.
41. As the Day shift, I want the conversion to report a round-trip check, comparing what the Console, the Waterfall, show and Model evidence read from each original and from its Session trace, so that I know nothing I use was lost.
42. As the Day shift, I want the originals kept until I remove them after reading that report, so that a mistake in the conversion costs nothing.

Words and docs:

43. As the Day shift, I want the glossary to rename Session log to Session trace, and add Trace, when the switch ships and not before, so that factory Runs working on today's logs aren't told to call them Session traces.
44. As a factory session, I want the guide, the README and the Session prompts to say Session trace once the switch ships, so that what I'm told matches the files I find.

Evals:

45. As an Experiment, I want each Trial's session recorded as a Session trace, so that its tokens, times and tool calls can be graded and reported.
46. As the Scoring run and Run metrics, I want tokens by Model, cost, tool errors, permission denials, killed work and repeated tool calls in Session traces, so that I can compute signals without reading six Harness formats.

Maintenance:

47. As a maintainer, I want the OTLP writer to be a module with no thirdshift types, so that it can move into a crate of its own if anyone else wants it.
48. As a maintainer, I want the writer tested by round trips through OpenTelemetry's official types, so that our files read wherever the format is read.
49. As a maintainer, I want the writer to add no dependency at run time, so that the release binary doesn't grow by an SDK.
50. As a maintainer, I want every Harness's decoder to produce the same normalized events, so that recording, the Console and metrics don't depend on which Harness ran a session.
51. As a maintainer, I want fake Harnesses for all six to print their real formats in tests, so that every Harness's Session trace is checked end to end.

Implementation Decisions

Files:

• Two files per session. Each session writes <name>.logs.jsonl and <name>.spans.jsonl in its Command's logs folder, where <name> is today's Session log name without .jsonl. Each line is one OTLP export request: resourceLogs in the first, resourceSpans in the second. Each file holds one type of data, as the OTLP file format requires and ADR 0019 now says.
• Neighbours keep their names. Model evidence, <name>.models.json, and the review reports beside it keep their names and formats.
• The Command record names the log records' file where it names a session's Session log today. Silent's rule and the heartbeat watch that file's growth, so neither rule changes.
• The logs inventory and file-name decoding read a session's two files as one session. Until the conversion has run, they also still read old Session logs.

What a Session trace holds:

• A log record for each Harness event, carrying:
  • the trace ID, and the ID of the span the event belongs to;
  • the event's own time where the Harness gives one, and the time thirdshift saw it, always;
  • an event name;
  • GenAI convention attributes where they apply, and thirdshift.* attributes for what they lack;
  • the raw Harness event, as the record's body.
• A span for the session, each model call and each tool call. Each carries GenAI convention attributes: the operation, the requested and response Model, and token usage. thirdshift.* attributes cover cost, the cache split, rate limits, permission denials, interrupted or killed work, the sub-agent lifecycle, diffs, pushes and pull-request events, the Harness's setup and the Session prompt.
• Resource attributes: thirdshift's version, the Harness and its version, the requested Model and Effort, the session's kind, the repository and the Command's identity. Each file names the GenAI conventions version it follows.
• Values the conventions don't type are written as JSON strings, which every viewer keeps.
• Raw events stay. A record is trimmed only after the round-trip check shows that a trimmed record loses nothing in use, and that trimming is a later decision.

IDs and the Trace's upper levels:

• The trace ID is derived from the Command's log stamp and token. A Command record has no ID field, and child processes already inherit that pair, so every session of a Command shares the ID, its child Runs' included.
• The span IDs of the Command, each Run and each stage are computed from the Command record: the Run's identity, and the stage's name and start. A session's span names its parent without the parent being written anywhere, and the Command record stays the one place that keeps stage times.
• The Console and the export command build those upper spans from the Command record whenever they need them.

Normalized events:

• Each Harness's decoder gains one normalized event, beside the progress lines it gives today. The decoders are Claude and Grok's shared stream, Codex, agy, Muse and OpenCode. The event covers model calls and tool calls starting and ending, messages, usage, errors and killed work. The session recorder turns these events into log records and spans.
• Today's consumers keep their behavior: progress lines, Model evidence, usage-limit errors, Resume detection and the final message.
• Harness files read after exit still count. What thirdshift reads after Muse or OpenCode exits, as today, goes into the Session trace.

Recording and interruption:

• The session recorder writes each log record as its event arrives, and each span as it ends.
• On interruption, it closes the open spans with an error status before it returns. After a hard kill, the records already written stand, and a missing parent can be rebuilt from its children's parent IDs.

The Harness's version:

• Each session records its Harness's version as the Harness reports it. Today nothing captures it. Pinning versions for an Experiment's Trials belongs to Spec: Experiments, which measure a change before it reaches production #977, and showing the version in the pull request's Built-with line to Spec: the Scoring run, which scores a weekly sample of production work #978.

The OTLP writer:

• A module with no thirdshift types, built on the serde that thirdshift already uses, with no new run-time dependency.
• It follows OTLP's JSON encoding rules: IDs in hex, enums as numbers, camelCase keys and 64-bit integers as strings.
• It can become a crate of its own, such as otlp-jsonl, if anyone else wants it. Its name leaves out "opentelemetry" and "otel", which are trademarks.

The OTLP endpoint:

• A User config setting names an optional OTLP/HTTP endpoint. When one is set, each span is also sent there as it ends.
• Sending is best effort. A failed send never fails a session. Security runs' sessions are never sent.

The Console:

• The conversation view renders Session traces. It first unwraps each record's raw event and keeps its Claude and Codex rendering, then grows to all six Harnesses through the normalized attributes.
• The live view streams the log records' file by byte offset, as it streams Session logs now.
• The Waterfall takes tool times from the spans, and from the records of a tool call still running. Its Command-log matcher is removed.
• A span tree grows out of the Waterfall, with span details in the Peek panel. A /prototype session designs it first, borrowing from MLflow's trace view.
• A Codex edit shows its diff when the Session trace has one. Otherwise it keeps "Patch unavailable in this log."
• show reports the latest record of the live session's Session trace, and so does show --json's latest event.

Codex's diffs (#854):

• thirdshift reads each apply_patch edit's diff from Codex's own session file, which it finds by the thread ID it already reads from Codex's stream. It adds the diff to the Session trace as Codex's record of that edit.
• Edits are matched by order and file path, since Codex's stream and its file give an edit different IDs.
• Shell edits stay without a diff. When the file can't be read, the edit's record stays as it is. Codex doesn't publish its session-file format, so a change to it falls back to today's behavior rather than failing.

Export:

• thirdshift logs <command> --export <dir> writes the Command's whole Trace into <dir> as two OTLP JSON Lines files:
  • the spans, including the Command, Run and stage spans built from the Command record;
  • the log records.

Converting old Session logs:

• A one-time logs option converts every Session log under the machine's logs folder, Security runs' included, with the same code that writes Session traces live. It writes each Session trace beside its original.
• A record without a time of its own gets one estimated from the Command log, as the Waterfall estimates times now, and is marked as estimated.
• Old Codex sessions get their diffs where Codex's session files still exist.
• It reports a round-trip check for each session: what the conversation view, the Waterfall, show and Model evidence read from the original and from its Session trace. The originals stay until you remove them in a second step, after reading the report.
• Each host runs it on its own logs.

Words, docs and security:

• The glossary changes when the switch ships. It renames Session log to Session trace and adds Trace, and the guide, the README and the Session prompts that mention Session logs change with it.
• Command logs and Activity logs stay logs (ADR 0011).
• A Security run's Session traces stay on the machine. The Console shows them only behind its sign-in (Spec: the Console, a web page for each machine's Commands #776), and they are never sent to an endpoint (ADR 0015).

Testing Decisions

• A good test checks what a user or another tool sees: the files a session leaves, what the CLI prints and what the Console serves. It doesn't check a decoder's internals.

• Seam 1: the whole binary against fake Harnesses. This is the existing integration scenario, which runs the built binary with fake CLIs. Fakes for all six Harnesses print their real stream formats. Tests assert that:

  • each session leaves two files that decode as OTLP;
  • one trace ID runs through a Command's sessions, child Runs included;
  • every parent ID matches the one computed from the Command record;
  • an interrupted session's open spans have an error status;
  • show --json gives the latest record, and logs --json lists each session once;
  • the export command's files decode, with the Command, Run and stage spans present;
  • the conversion writes Session traces whose round-trip check passes, over a Session-log fixture for each Harness;
  • Codex's diffs reach the trace, from a fake Codex that writes its own session file under a temporary home.

  Prior art: the integration tests of Command logs, show and logs, background work, and each session kind.

• Seam 2: the OTLP writer on its own. Its tests round-trip through OpenTelemetry's official protocol types, a test-only dependency, and check the specification's JSON examples. A one-time check that the Collector's file receiver reads the files is recorded with its result.

• Seam 3: the Console's existing tests. These are the Rust tests that call its routes (the conversation view, the byte-offset stream, the heartbeat and the Waterfall) and the browser tests that append to a live log. Session-trace fixtures, one per Harness, replace today's Claude and Codex Session-log fixtures.

• Decoder tests grow only for a stream quirk that needs a case of its own.

Out of Scope

• Experiments (Spec: Experiments, which measure a change before it reaches production #977), the Scoring run (Spec: the Scoring run, which scores a weekly sample of production work #978) and Run metrics (Spec: Run metrics, the signals each Run leaves, kept apart #981), which are add evals -integrate with langfuse / oss eval tool? #698's other three Specs. This Spec gives them Session traces to read.
• The Harness version in the Built-with line (Spec: the Scoring run, which scores a weekly sample of production work #978), and pinning Harness versions for Trials (Spec: Experiments, which measure a change before it reaches production #977).
• Trimming raw events from records, until the round-trip check shows what can go.
• Another trace viewer, such as MLflow, Phoenix, or Grafana's Tempo and Loki (check relevance of grafana, prometheus, tempo, mimir, opentelemetry, Alertmanager, and or loki integration with thirdshift or formatting of thirdshift data to fit with these systems #914). That waits until you want search across Runs beyond Charts on the Console: what to chart over time #813's charts. Copying log records onto spans for viewers that read spans only waits with it.
• OTLP metrics. Session traces hold log records and spans only.
• Publishing the OTLP writer as a crate of its own, until someone wants it.
• Command logs and Activity logs, which stay as they are.

Further Notes

• Closes Decide Codex patch evidence retention for Console edit diffs #854. Part of add evals -integrate with langfuse / oss eval tool? #698.
• Order: Spec: Experiments, which measure a change before it reaches production #977 is built in parallel with this one. Spec: Run metrics, the signals each Run leaves, kept apart #981's Session-trace Tickets wait for this Spec.
• Two specifications here are still in Development: the OTLP file format and OpenTelemetry's GenAI conventions. That's why each Session trace names the conventions version it follows.
• Research: docs/research/otlp-file-writer.md on the writer, and docs/research/eval-viewers.md on viewers and the Console as one.
• Upstream: OpenTelemetry Rust's file exporter is in review as open-telemetry/opentelemetry-rust#3763. Reviewing and testing it is the way to help it land.
• Tickets come next, with to-tickets. This Spec stays needs-triage until all of them and their blocking links are published.