# feat(data): add episode analysis workflows to dataviewer - #1234
Conversation
… panels to dataviewer - add Episode Analyzer tab, motion metrics, and analysis records - add per-episode language instructions in LeRobot v3 task format - add foldable right-pane sections persisted across episodes - import analysis fields as filterable labels; add VLM labeling CLI ✨ - Generated by Copilot
🧪 - Generated by Copilot
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Snapshot WarningsEnsure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice. Scanned FilesNone |
ad5ba2c to
1bb42c9
Compare
- correct spelling and formatting findings - cover analysis label import behavior and no-op paths - restore backend coverage above the required threshold ✅ - Generated by Copilot
- cover Azure label storage factory wiring - verify nested dataset label path resolution ✅ - Generated by Copilot
…/episode-analysis
Katrien De Graeve (katriendg)
left a comment
Katrien De Graeve (katriendg)
left a comment
There was a problem hiding this comment.
Nice feature set — the Episode Analyzer flow, persisted per-episode analysis, and the switch to a synchronous worker-thread judge all read well, and the import-from-analysis endpoint is genuinely well-tested. Leaving this as a comment (non-blocking); the inline notes are suggestions plus a few known gaps.
Please add user-facing docs. The PR is marked "no documentation changes needed," but several of these flows are unclear for end users and there's nothing in docs/, the dataviewer README, or a skill covering them. A short user guide would go a long way:
- The Qwen3-VL labeling script (
vlm_label_dataset.py): document that it's a script, how to run it, what it emits (labels.jsonl/labels.csv), and — most importantly — how those results actually reach the viewer. Right now nothing bridges the script output to themeta/episode_labels.jsonthe Episode Analyzer reads, so it's not obvious how a user closes that loop. - The Episode Analyzer tab end-to-end: importing analysis fields as labels, and what each motion metric (including the new
log-scaledvsradian-basedsmoothness modes) means.
Missing tests (known — captured inline, not blocking). The lower-coverage areas are called out on the relevant lines: _compute_normalized_smoothness + smoothness_mode validation, the smoothness_mode route field, LabelPanel (lowest patch coverage at 41%), and useImportAnalysisLabels. Keeping these as inline notes so they're easy to pick up.
A couple of small non-blocking items are inline too: the "resumable JSONL" claim doesn't match open("w") truncation; setAllEpisodeLabels resets saved labels for all episodes on import; the import UI can't reach the backend's overwrite/prefix; and the frontend still carries dead 202-polling scaffolding after the judge went synchronous.
One pre-existing observation (not introduced here, worth a follow-up issue): BlobLabelStorage.load uses except (json.JSONDecodeError, Exception) and reaches into provider privates — it can silently return empty labels that a later save persists over real data.
Chris Montazer (rezatnoMsirhC)
left a comment
Chris Montazer (rezatnoMsirhC)
left a comment
There was a problem hiding this comment.
PR Soundtrack Approved 🔥
…resolution - explicitly use IPv4 for backend health check - resolve VLM_JUDGE_ENABLED and VLM_JUDGE_BACKEND from .env if not set - improve DATA_DIR resolution logic from .env 🔧 - Generated by Copilot
- add resumable VLM labeling and persisted analysis output - preserve label edits and expose safe import controls - remove stale judge polling and expand behavioral coverage 🔧 - Generated by Copilot
🎨 - Generated by Copilot
Ali Zaidi (akzaidi)
left a comment
Ali Zaidi (akzaidi)
left a comment
There was a problem hiding this comment.
Review result: changes required before merge
The branch is current with main at 6582c578, and the post-sync validation workflow passes. The following functional issues remain:
Blocking
- Partial analysis updates erase existing fields.
labels.py:505-516replaces the complete record even though every field is optional. Merge explicitly supplied fields or require a complete PUT representation. - Concurrent label and analysis writes can lose data. The endpoints use an unlocked load-modify-save cycle, while Blob storage writes with unconditional overwrite. Add an atomic storage mutation with per-dataset locking and Blob ETag retries.
- Saved per-episode judge instructions are ignored.
vlm_judge.py:147-195resolves only request or dataset metadata, breaking annotation-only batch judging and cache retrieval. Restore request → saved annotation → dataset precedence. - Synchronous VLM inference can exceed the deployed request limit.
vlm_judge.py:176-248keeps the request open while inference may take minutes; Azure Container Apps ingress limits requests to 240 seconds. Return a durable asynchronous job and poll its status. - Inference lacks single-flight deduplication and a GPU concurrency bound. Concurrent misses can duplicate model execution, exhaust GPU memory, and race on the same cache temporary path. Add keyed coordination, a bounded executor or semaphore, and unique atomic cache temporaries.
- Malformed model output is persisted as complete.
vlm_label_dataset.py:128-205accepts any JSON object, fills defaults, and lets--resumeskip it forever. Validate a strict response schema and retain schema failures as retryable errors.
Follow-ups
- Make optional
.envlookups instart.shsafe undererrexitandpipefail. - Add immutable model revision input and provenance to the labeling CLI.
- Return a nonzero CLI exit code when episode errors remain.
- Memoize trajectory-derived analysis inputs during playback.
Fresh CI has no failed or pending checks. CodeQL alert #392 is pre-existing on main from July 9 and is not attributable to this PR. GitHub does not permit the PR author account to submit a formal request-changes review, so these blockers are recorded as a comment review.
7 participants
feat(data): add episode analysis workflows to dataviewer
PR Soundrack: Juicy J & Project Pat - Dem Goats
Description
Added an Episode Analyzer workflow that combines trajectory metrics, VLM outcomes, persisted analysis, filterable labels, and language instructions in the dataviewer. The backend now stores structured per-episode analysis and supports importing selected analysis fields as labels, while the standalone labeling CLI processes LeRobot datasets with Qwen3-VL.
The VLM judge request flow now returns completed judgments synchronously from a worker thread instead of returning
202 Acceptedand requiring client polling. Dataset instruction lookup also supports LeRobot v3 task metadata.Related issues: None
Type of Change
Component(s) Affected
infrastructure/terraform/prerequisites/- Azure subscription setupinfrastructure/terraform/- Terraform infrastructureinfrastructure/setup/- OSMO control plane / Helmworkflows/- Training and evaluation workflowstraining/- Training pipelines and scriptsdocs/- Documentationdata-management/viewer/- Dataset analysis and annotationevaluation/vlm_judge/- VLM judge service contractTesting Performed
planreviewed (no unexpected changes)applytested in dev environmentsmoke_test_azure.py)npm run validate)npm run validatefromdata-management/viewer/frontend)Documentation Impact
Bug Fix Checklist
Complete this section for bug fix PRs. Skip for other contribution types.
Checklist
Changes
Analysis and labeling
Dataviewer interface
Compatibility and tests