feat(evaluation): add pi0 replay evaluation and VLA schema v1 - #1564
Open
Alexandre Gattiker (algattik) wants to merge 1 commit into
Open
Alexandre Gattiker (algattik) wants to merge 1 commit into
Alexandre Gattiker (algattik) wants to merge 1 commit into
Conversation
Alexandre Gattiker (algattik)
requested a review
from a team
as a code owner
September 18, 2026 15:08
Contributor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Snapshot WarningsEnsure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice. OpenSSF Scorecard
Scanned Files
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1564 +/- ##
==========================================
- Coverage 88.10% 86.79% -1.32%
==========================================
Files 279 280 +1
Lines 23203 23721 +518
Branches 3199 3272 +73
==========================================
+ Hits 20444 20589 +145
- Misses 2071 2421 +350
- Partials 688 711 +23
*This pull request uses carry forward flags. Click here to find out more.
🚀 New features to boost your workflow:
|
- dispatch ACT, diffusion, pi0, and pi0_fast through shared policy handling - emit strict schema-v1 metrics and failure artifacts - cover replay behavior and Azure ML train-to-evaluate lifecycles - forward Hugging Face credentials for gated pi0 evaluation assets 🤖 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Alexandre Gattiker (algattik)
force-pushed
the
algattik-issue-930-pi0-vla-dispatch
branch
from
September 21, 2026 15:57
02c7661 to
d26cd3f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Description
Added shared replay evaluation for ACT, diffusion, pi0, and pi0_fast policies. The evaluator now loads policy-specific processors, propagates required VLA task metadata, handles shared LeRobot v3 data and video files within episode bounds, and emits strict schema-v1 metrics, results, and failure-case artifacts.
Added dedicated Azure ML pi0-family submission assets backed by the frozen VLA runtime. Hub inputs require immutable revisions, and gated pi0 evaluation forwards the configured Hugging Face token without exposing it in configuration previews.
Closes #930
Type of Change
Component(s) Affected
infrastructure/terraform/prerequisites/- Azure subscription setupinfrastructure/terraform/- Terraform infrastructureinfrastructure/setup/- OSMO control plane / Helmworkflows/- Training and evaluation workflowstraining/- Training pipelines and scriptsdocs/- Documentationevaluation/- Replay evaluation, policy dispatch, and schema contractsTesting Performed
planreviewed (no unexpected changes)applytested in dev environmentsmoke_test_azure.py)Live Azure ML E2E validation
The affected lifecycle tests ran against an Azure ML deployment:
tests/e2e/test_e2e_aml_il_lifecycle.py::test_aml_il_lifecycle_e2etests/e2e/test_e2e_aml_vla_pi0_training.py::test_aml_vla_pi0_lifecycle_e2eThe pi0 lifecycle exposed missing
HF_TOKENpropagation in the first evaluation attempt. After forwarding the token to pi0/pi0_fast evaluation jobs, the complete lifecycle and artifact contract passed.Documentation Impact
Bug Fix Checklist
The Hugging Face credential propagation fix is covered by the live pi0 lifecycle validation.
Checklist