test(training): close cloud E2E runtime validation gaps #1490

Component

Cloud E2E coverage for Azure ML, OSMO, runtime dependency installation, and MLflow integration

Problem Statement

The cloud E2E suite leaves three production integration boundaries insufficiently validated:

1. The Azure ML to OSMO proxy has a CPU smoke workflow and production submission script, but no automated E2E invokes the path. Dependency or Azure ML MLflow plugin incompatibility can leave the Azure ML job apparently successful because proxy MLflow failures are non-fatal.
2. The OSMO RL tests execute training/rl/scripts/setup_isaac_runtime.sh, but do not prove that the runtime loaded the versions exported from training/rl/uv.lock rather than compatible packages already present in the container image.
3. The OSMO VLA fine-tuning test executes training/vla/scripts/groot/osmo-train-entry.sh, but does not verify MLflow results, model registration, or pinned runtime and base-model provenance.

These gaps can allow dependency, plugin, or runtime-bootstrap regressions to pass task-level success checks.

Proposed Solution

Add one CPU-only AML-to-OSMO proxy E2E lane and strengthen the existing OSMO RL and VLA tests with runtime and integration assertions. Reuse existing production entrypoints and cloud jobs rather than adding duplicate GPU lanes.

Acceptance Criteria

AML-to-OSMO proxy

• Add a tracked default E2E test for workflows/azureml/submit-osmo-proxy-job.sh, workflows/azureml/osmo-proxy-job.yaml, and workflows/osmo/smoke-test-proxy-e2e.yaml.
• Submit the proxy through Azure ML and wait for both the Azure ML job and OSMO workflow to complete.
• Verify the OSMO task ran successfully and produced the declared durable blob output.
• Verify the proxy registered the expected Azure ML data asset.
• Query the Azure ML MLflow tracking URI and verify osmo.workflow_id, terminal status, task counts, success rate, and duration.
• Fail when the Azure ML MLflow plugin is unavailable or metric logging is absent, even if the proxy process otherwise exits successfully.

OSMO RL runtime

• Extend an existing OSMO RL E2E to capture machine-readable runtime provenance after setup_isaac_runtime.sh.
• Verify the installed runtime versions match the committed training/rl/pyproject.toml and frozen lock resolution.
• Verify the assertion cannot pass solely because compatible packages are preinstalled in the base image.

OSMO VLA runtime and MLflow

• Extend test_e2e_osmo_vla_finetune rather than adding another GPU workflow.
• Verify the runtime and base-model provenance produced by training/vla/scripts/groot/osmo-train-entry.sh.
• Verify the expected Azure ML MLflow run, correlation identifiers, metrics, and model-registration result.

E2E lifecycle

• Preserve exact submission commands, Azure ML job and OSMO workflow identifiers, attempts, logs, retry classification, and terminal state in the E2E handle.
• Clean up proxy outputs, registered data assets, MLflow/model artifacts created for the test, and other test-owned cloud resources.
• Keep the proxy lane CPU-only and avoid adding duplicate RL or VLA GPU jobs.

Out of Scope

• HiL edge coverage tracked by feat(testing): cover HiL edge journey in cloud E2E #1489
• Physical robot hardware and laboratory networking
• New GPU SKUs, node pools, or infrastructure changes