Measure model loading memory and prepare GPU baseline counters (#1429)

Model construction can allocate memory before the existing perf
baseline, while missing GPU process counters leave the pre-model delta
unavailable. This ports the loading-memory and GPU-baseline changes from
gim-home/winml-cli-mirror PRs #10 and #11 onto current main.

- Add versioned load-only memory measurements from runtime/device
readiness through model compilation, excluding benchmark inputs and
inference. Keep sampled process peaks and OS lifetime high-water
diagnostics separate. Preserve null counters and signed deltas.
- Prepare absent counters with a model-free D3D12 device on the
explicitly selected GPU, retain it through final sampling, and release
it on success or failure. Connect preparation to the new process tracker
used by the combined port.
- Keep automatic Runtime outputs for older projections and explicit
output requests for newer ones. Include the initializer-shape regression
test; its implementation fix is already upstream.

Compatibility: the outer report remains schema version 2 and
`load_memory.version` is 1. Classic `memory_measurement.schema_version`
is 3 because its structure and legacy-named baseline/delta boundaries
change; documentation describes migration and limitations. Existing
upstream CGIR changes are retained.

Validation on Windows / Python 3.11:
- Affected perf CLI/GenAI/lifecycle, all session unit tests, and
initializer-shape regression: **1,449 passed, 6 existing skips**.
- Final focused memory/GPU integration rerun: **34 passed**; Runtime
compatibility suite: **40 passed**.
- Ruff across `src/` and `tests/`, changed-file formatting, license
headers, and `git diff --check` passed.
- Mypy: **no issues in 464 source files**.

Review follow-up: distinguish configured polling delay from observed
mean/max sampling intervals, including GPU-query overhead. Add native
factory/adapter/device failure-injection tests and idempotent COM
cleanup checks. The updated memory/GPU suite passes all 40 tests.

A fresh-process native smoke test on Qualcomm Adreno X1-85 successfully
created the baseline device on the selected LUID, discovered valid PDH
counters, recorded final observations, and released the retained device.
This validates baseline preparation/cleanup; full model hardware
benchmarks from the source PRs were not rerun.

---------

Co-authored-by: Qiong Wu <120818070+qiowu_microsoft@users.noreply.github.com>

Author: DingmaomaoBJTU

docs/commands/perf.md

@@ -53,6 +53,81 @@ Both runtime reports include `schema_version: 2` and a `benchmark_info.runtime`
 
 When `--memory` is enabled, both `winml-ort` and `ort-genai` reports use the same `memory` field names for shared concepts: RSS baseline, after-compile/load, after-inference, peak, model-load delta, inference/generation delta, and total delta; VRAM local/shared baseline, after-compile/load, after-inference, peak, model-load delta, inference/generation delta, and total delta.
 
+### Classic memory lifecycle
+
+#### Model loading from a user's perspective
+
+`load_memory` is the load-only view (block `version: 1`). It answers how much
+additional memory was observed while this artifact became ready in this
+runtime, and how much remained at readiness. Runtime/EP setup precedes its
+baseline; its endpoint is after session compilation but before allocating
+benchmark inputs, warmup or inference. If the selected path requires online
+compilation, that temporary cost is included because the user must get through
+it to load the model. A precompiled artifact is a different configuration.
+
+Each RSS/local/shared block records baseline, absolute peak, absolute ready
+value, `peak_extra_mb` and `ready_extra_mb`, plus the peak method and availability.
+RSS uses the Windows high-water when a **new** process high-water occurs inside
+the load window; otherwise only observed load samples/endpoints can be used.
+GPU peaks are sampled. No inference peak or larger pre-existing process peak
+is substituted into this view. Unknown baselines produce null increments.
+
+This is an **observed load footprint**, not a certified minimum device-memory
+capacity: RSS is resident memory, allocator caches can remain, sampling may
+miss short GPU allocations, and unified RAM/GPU counters may overlap. Report
+the load peak and ready increment separately; do not reduce this to parameter
+file size or sum local/shared GPU counters. Already-loaded composite children
+cannot provide this window and return an explicit unavailable status.
+
+The enclosing perf document still uses schema version 2; the optional
+`load_memory` block is explicitly versioned. Consumers must support nullable
+memory values and use the recorded scope rather than comparing older deltas
+as if they covered the same interval.
+
+For `winml-ort` and `winml-runtime`, RSS baseline is captured **before model
+loading**, including eager session construction and Runtime pipeline creation.
+GPU baseline is captured after resolving the bound EP/adapter but before
+constructing the model. Adapter discovery must not read lazy model properties.
+The added `*_after_load_mb` checkpoint separates loading from the explicit
+compile step. Input generation follows compilation/readiness. `*_after_compile_mb` and `*_after_inference_mb`
+retain their existing names.
+
+`memory_measurement` records PID, adapter LUID, lifecycle scope, units and
+per-checkpoint missing reasons. Fields with the legacy `_mb` suffix use MiB
+(bytes / 1,048,576). Deltas are signed process changes, not model-only allocation;
+negative values can reflect released buffers or working-set changes. Library
+imports, build caches and input preparation can contribute to the measured span.
+
+`process_memory` separately records sampled RSS/local/shared peaks, actual
+sample counts, interval and duration over loading, compilation and inference.
+These are sampled peaks, not an exact continuous maximum. The configured
+polling delay is not a sampling frequency: `configured_poll_delay_sec` records
+the wait after each observation. `observed_mean_interval_sec` and
+`observed_max_interval_sec` record elapsed time between completed observations
+(null with fewer than two observations). GPU discovery/retries also delay RSS
+sampling; these intervals include that overhead and scheduling delays. The legacy
+`*_checkpoint_peak_mb` remains the maximum of the three original checkpoints,
+and is `null` if any required checkpoint is unavailable.
+
+On Windows, `process_memory.os_rss_peak_before_mb` and
+`os_rss_peak_after_mb` also record the OS working-set high-water marks. These
+catch short allocations missed by polling, including native code holding the
+Python GIL. They cover the **process lifetime**, cannot be reset at baseline,
+and do not replace the benchmark-scoped sampled peak. On unsupported systems
+they are `null` with an explicit reason.
+
+Missing GPU counters produce `null`, not zero; deltas with missing endpoints
+also remain `null`. A GPU process-memory instance may not exist before model
+allocation, so valid later absolute readings do not establish a zero baseline.
+Local/shared counters can overlap on unified-memory devices and must not be
+summed. GenAI's shared GPU-memory consumer also preserves unavailable values.
+
+Composite components are already loaded when measured. Their explicit scope
+is `already_loaded_component_through_inference`; their values include the
+shared process and do not claim an independent component-load footprint.
+`--no-memory` disables this collector. Background sampling adds overhead and
+is separate from the native inference timer.
+
 With `--runtime ort-genai`, `winml perf` benchmarks the onnxruntime-genai decoder pipeline rather than a single `session.run()`. The JSON report uses a phase-based schema: `load` contains startup spans, `requests` contains one warmup or timed generation sample per request, `aggregate` summarizes timed requests only, `memory` contains optional RAM/VRAM deltas, and `hw_monitor` contains optional monitor output. The optional `memory` and `hw_monitor` top-level names match the classic `winml-ort` perf report; GenAI keeps `load`/`requests`/`aggregate` instead of classic `latency_ms`/`throughput` because generation has distinct prompt, first-token, and decode phases.
 
 For model-ID auto-builds, the selected EP/device must be supported by the model's
@@ -79,34 +154,19 @@ target validation.
 
 ### Memory measurement contract
 
-With --memory, the legacy baseline stays **after the model factory, before
-input generation and explicit session.compile()**, as on main. Existing
-baseline/load/inference/total delta fields and the maximum-of-three checkpoint
-peak keep that boundary. Eager model loading before this baseline is excluded.
-
-Single-model runs also take an earlier, separately named before_model_load
-snapshot after device resolution and before the model factory. For each of
-rss, vram_local and vram_shared, additive fields are:
-
-- *_before_model_load_mb: earlier absolute snapshot.
-- *_model_factory_delta_mb: legacy baseline minus earlier snapshot.
-- *_total_from_before_model_load_delta_mb: inference end minus earlier snapshot.
+With `--memory`, classic and Runtime runs use the lifecycle boundaries described
+above: process RSS starts before model construction, while the load-only window
+starts after runtime/device setup and ends before benchmark input allocation.
+Preloaded composite components cannot claim a load baseline; their `load_memory`
+block is unavailable and their process scope explicitly starts after loading.
 
-The new total includes model factory/build and input/session overhead. It is
-not weights-only memory or a continuous peak. Preloaded composite components
-have no observation before loading: all added fields are null with an explicit
-reason. They must not inherit the parent's aggregate baseline.
-
-Legacy *_mb fields use MiB. memory_measurement schema_version 2 retains the
-legacy baseline definition and adds the earlier boundary definition, PID,
-process creation time, selected LUID and timestamped byte/status/source records.
-Private commit is separate from RSS. The legacy checkpoint peak excludes the
-new earlier snapshot, even if that snapshot is larger. Signed deltas can be negative.
-
-Unavailable readings and dependent deltas are null, never zero. If the earlier
-GPU process instance is absent, only metrics needing that point are unavailable;
-a valid legacy baseline and its deltas remain usable. CPU GPU memory is
-not_applicable. Performance success does not certify memory.
+The outer result remains `schema_version: 2`; `load_memory.version` is 1.
+Classic `memory_measurement.schema_version` is now 3 (replacing the version 2
+byte-checkpoint structure). It describes PID, selected LUID, lifecycle scope and missing
+counter reasons. The old after-factory baseline and its additive
+`*_before_model_load_mb` fields are superseded by these explicit blocks.
+Signed deltas can be negative. Missing readings and dependent deltas stay null.
+Performance success does not certify memory requirements.
 
 GPU memory uses main's effective EP-device binding, including --device-luid and
 provider selectors resolved through the advertised device options. Unresolved
@@ -121,6 +181,28 @@ not proof of zero; enumeration errors and invalid readings are separate states.
 The CLI cannot certify that a process has never used the GPU merely from an
 absent PDH instance, so it never fabricates a zero baseline from that condition.
 
+For a Windows GPU target with a resolved LUID, if process memory instances are
+absent before model loading, the CLI creates a model-free D3D12 device on that
+exact adapter and retries counter discovery for up to two seconds. It creates
+no model, buffers or command queue and submits no GPU work. The device is kept
+alive through the final checkpoint and released on success or failure, so its
+setup footprint is present in both endpoints rather than appearing as model
+allocation. Load-only RAM is captured after this preparation too; process RSS
+still starts before generic runtime setup. Already valid counters,
+CPU/NPU targets, unresolved adapters and preloaded components do not create this
+device. Initialization errors never prevent the model's own provider from running.
+
+This is a prepared-device measurement, not cold GPU initialization. The
+process delta includes model/provider/input allocations and is
+not a minimum VRAM requirement. The load-only delta excludes inputs and inference.
+Legacy-named baseline and delta fields use the new boundaries documented above;
+they must not be compared directly with version 2 memory measurements. The optional
+memory_measurement.gpu_baseline_preparation block records method, timing,
+initial/prepared observations, selected adapter and failures. If the subsequent
+checkpoint remains unavailable, dependent deltas remain null. A later valid
+checkpoint never replaces a missing pre-model baseline. Historical results
+cannot be repaired without recollection.
+
 The hardware monitor refreshes PID/LUID memory instances every 200 ms, including
 when monitoring starts before model load. Counter registration is deduplicated,
 failed registrations retry, and missing/disappeared instances stay unknown.

src/winml/modelkit/commands/_perf_genai.py

@@ -187,14 +187,14 @@ def _round_stats(values: dict[str, float]) -> dict[str, float]:
 
 def _get_rss_mb() -> float:
     """Return current process RSS in MB."""
-    from ..session.monitor.memory_tracker import get_rss_mb
+    from ..session.monitor import get_rss_mb
 
     return get_rss_mb()
 
 
 def _get_vram_mb(adapter_luid: str | None) -> tuple[float | None, float | None]:
     """Return current process device-memory usage as local/shared MB."""
-    from ..session.monitor.memory_tracker import get_vram_mb
+    from ..session.monitor import get_vram_mb
 
     return get_vram_mb(adapter_luid)