Skip to content

Commit a03d1cd

Browse files
algattikCopilot
andauthored
fix(dataviewer): harden detection model loading and caching (#1567)
# Pull Request ## Description Restricts detection models to approved identifiers and reviewed SHA-256 digests, loads verified bytes from service-owned staging files, and provisions model weights through a read-only container mount. Replaces the unbounded result cache with a configurable TTL/LRU cache, centralizes effective confidence resolution, and returns logged, sanitized errors for invalid or unavailable checkpoints. Closes #119 ## Type of Change - [x] 🐛 Bug fix (non-breaking change fixing an issue) - [ ] ✨ New feature (non-breaking change adding functionality) - [ ] 💥 Breaking change (fix or feature causing existing functionality to change) - [x] 📚 Documentation update - [ ] 🏗️ Infrastructure change (Terraform/IaC) - [ ] ♻️ Refactoring (no functional changes) ## Component(s) Affected - [ ] `infrastructure/terraform/prerequisites/` - Azure subscription setup - [ ] `infrastructure/terraform/` - Terraform infrastructure - [ ] `infrastructure/setup/` - OSMO control plane / Helm - [ ] `workflows/` - Training and evaluation workflows - [ ] `training/` - Training pipelines and scripts - [ ] `docs/` - Documentation - [x] `data-management/viewer/` - Dataset analysis backend and frontend types ## Testing Performed - [ ] Terraform `plan` reviewed (no unexpected changes) - [ ] Terraform `apply` tested in dev environment - [ ] Training scripts tested locally with Isaac Sim - [ ] OSMO workflow submitted successfully - [ ] Smoke tests passed (`smoke_test_azure.py`) - [x] Backend focused tests passed: 105 tests - [x] Backend suite passed: 1,158 tests, 115 skipped, excluding unrelated VLM import failures - [x] Ruff, Markdown, YAML, Compose, and spelling checks passed - [x] uv lock consistency passed in CI - [x] Docker Compose image build and runtime smoke passed in CI ## Documentation Impact - [ ] No documentation changes needed - [x] Documentation updated in this PR - [ ] Documentation issue filed ## Bug Fix Checklist - [x] Linked to issue being fixed - [x] Regression test included, OR - [ ] Justification for no regression test: ## Checklist - [x] My code follows the [project conventions](copilot-instructions.md) - [x] Commit messages follow [conventional commit format](instructions/commit-message.instructions.md) - [x] I have performed a self-review - [x] Documentation impact assessed above - [x] No new linting warnings introduced --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 84128a3e-e250-43fc-b9ad-0b0f049ce73a Copilot-Session: 16904c91-fd21-4efb-9909-c008f45ca45e Copilot-Session: 92cf0356-961b-441e-8cc2-0177c8068781
1 parent e3b2b92 commit a03d1cd

18 files changed

Lines changed: 885 additions & 192 deletions

File tree

data-management/viewer/README.md

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -496,7 +496,11 @@ Without `--write-analysis`, the JSONL and CSV files remain standalone exports an
496496
### Docker Compose (local)
497497

498498
```bash
499-
# Local storage mode (mount datasets directory)
499+
# Stage reviewed model weights outside the repository.
500+
export DATAVIEWER_HOST_MODELS_DIR=/absolute/path/to/models
501+
export DETECTION_MODEL_DIGESTS='{"yolo11n":"<sha256>","yolov8s-world":"<sha256>"}'
502+
503+
# Local storage mode
500504
DATAVIEWER_HOST_DATA_DIR=/path/to/datasets docker compose up --build
501505

502506
# Azure Blob Storage mode
@@ -507,6 +511,29 @@ export AZURE_STORAGE_ANNOTATION_CONTAINER=annotations
507511
docker compose up --build
508512
```
509513

514+
The backend mounts `DATAVIEWER_HOST_MODELS_DIR` read-only at `/models`. Each active `<model-identifier>.pt` file requires a matching reviewed SHA-256 value in `DETECTION_MODEL_DIGESTS`; missing or mismatched checkpoints return HTTP 503 before deserialization.
515+
516+
Run this preflight before deployment:
517+
518+
```bash
519+
test -r "$DATAVIEWER_HOST_MODELS_DIR/yolo11n.pt"
520+
test -r "$DATAVIEWER_HOST_MODELS_DIR/yolov8s-world.pt"
521+
printf '%s %s\n' \
522+
"$(shasum -a 256 "$DATAVIEWER_HOST_MODELS_DIR/yolo11n.pt" | cut -d' ' -f1)" \
523+
"$DATAVIEWER_HOST_MODELS_DIR/yolo11n.pt"
524+
printf '%s %s\n' \
525+
"$(shasum -a 256 "$DATAVIEWER_HOST_MODELS_DIR/yolov8s-world.pt" | cut -d' ' -f1)" \
526+
"$DATAVIEWER_HOST_MODELS_DIR/yolov8s-world.pt"
527+
docker compose run --rm backend \
528+
python -c "from src.api.services.detection_service import get_detection_service; service = get_detection_service(); service._get_model('yolo11n'); service._get_model('yolov8s-world', labels=['robot'])"
529+
```
530+
531+
Rollback by restoring the previous read-only model directory and its matching `DETECTION_MODEL_DIGESTS` value, then recreate the backend:
532+
533+
```bash
534+
docker compose up -d --force-recreate backend
535+
```
536+
510537
### Azure Kubernetes Service (AKS) / Container Apps
511538

512539
For AKS with workload identity or Container Apps with managed identity, set:
@@ -517,11 +544,15 @@ AZURE_STORAGE_ACCOUNT_NAME=mystorageaccount
517544
AZURE_STORAGE_DATASET_CONTAINER=datasets
518545
BACKEND_HOST=0.0.0.0
519546
CORS_ORIGINS=https://your-frontend-url.example.com
547+
DETECTION_MODELS_DIR=/models
548+
DETECTION_MODEL_DIGESTS={"yolo11n":"<sha256>","yolov8s-world":"<sha256>"}
520549
```
521550

522551
`AZURE_STORAGE_SAS_TOKEN` is **not** needed — `DefaultAzureCredential` automatically
523552
uses the pod/container managed identity when running in Azure.
524553

554+
Mount the reviewed model directory read-only at `/models`. Update the mount and digest map together during rollout or rollback.
555+
525556
### Building Images
526557

527558
```bash

data-management/viewer/backend/.env.example

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,20 @@ DATA_DIR=../../../datasets
6969
# RATE_LIMIT_DETECT=10/minute
7070
# RATE_LIMIT_DETECTIONS=120/minute
7171

72+
# Absolute directory containing approved YOLO weight files named `<model-identifier>.pt`.
73+
# Container deployments mount this directory read-only at /models.
74+
# DETECTION_MODELS_DIR=/models
75+
76+
# JSON object mapping every active model identifier to its reviewed SHA-256 digest.
77+
# DETECTION_MODEL_DIGESTS={"yolo11n":"<64-character-sha256>","yolov8s-world":"<64-character-sha256>"}
78+
79+
# Maximum cached episode summaries and their time-to-live in seconds.
80+
# DETECTION_CACHE_MAX_SIZE=100
81+
# DETECTION_CACHE_TTL_SECONDS=3600
82+
83+
# Default confidence threshold when a detection request omits `confidence`.
84+
# DETECTION_CONFIDENCE_THRESHOLD=0.1
85+
7286
# ─────────────────────────────────────────────────────────────────────────────
7387
# CORS (set for container / cloud deployments)
7488
# ─────────────────────────────────────────────────────────────────────────────

data-management/viewer/backend/Dockerfile

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -14,12 +14,11 @@ COPY --from=ghcr.io/astral-sh/uv:0.10.9@sha256:10902f58a1606787602f303954cea0996
1414
# Copy dependency manifests
1515
COPY pyproject.toml uv.lock ./
1616

17-
# Install production + azure dependencies into /app/.venv.
18-
# The 'azure' extra includes azure-storage-blob and azure-identity for MSI support.
17+
# Install production dependencies and runtime feature extras into /app/.venv.
1918
ENV UV_COMPILE_BYTECODE=1 \
2019
UV_LINK_MODE=copy \
2120
PATH=/app/.venv/bin:$PATH
22-
RUN uv sync --frozen --no-dev --extra azure --extra analysis --extra export --extra auth
21+
RUN uv sync --frozen --no-dev --extra azure --extra analysis --extra export --extra auth --extra yolo
2322

2423
# Copy application source
2524
COPY src/ ./src/
@@ -36,7 +35,8 @@ EXPOSE 8000
3635
ENV BACKEND_HOST=0.0.0.0 \
3736
BACKEND_PORT=8000 \
3837
STORAGE_BACKEND=local \
39-
DATA_DIR=/data
38+
DATA_DIR=/data \
39+
DETECTION_MODELS_DIR=/models
4040

4141
# Health check
4242
HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \

data-management/viewer/backend/pyproject.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ dependencies = [
1515
"pyarrow==25.0.1",
1616
"Pillow==12.3.0",
1717
"av==18.1.0",
18+
"cachetools==7.1.8",
1819
]
1920

2021
[project.optional-dependencies]

data-management/viewer/backend/src/api/config.py

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,17 @@
88

99
from __future__ import annotations
1010

11+
import json
1112
import logging
1213
import os
14+
import re
15+
from collections.abc import Mapping
1316
from dataclasses import dataclass, field
1417
from pathlib import Path
18+
from types import MappingProxyType
19+
from typing import Any
20+
21+
from .detection_constants import ALLOWED_DETECTION_MODELS
1522

1623
logger = logging.getLogger(__name__)
1724

@@ -57,6 +64,21 @@ class AppConfig:
5764
episode_cache_max_mb: int = 100
5865
"""Max memory budget for the LRU cache in megabytes. 0 means count-only."""
5966

67+
detection_models_dir: str = "./models"
68+
"""Directory containing approved YOLO weight files."""
69+
70+
detection_model_digests: Mapping[str, str] = field(default_factory=lambda: MappingProxyType({}))
71+
"""Approved model identifiers mapped to reviewed SHA-256 checkpoint digests."""
72+
73+
detection_cache_max_size: int = 100
74+
"""Maximum number of episode detection summaries retained in memory."""
75+
76+
detection_cache_ttl_seconds: int = 3600
77+
"""Seconds before an episode detection summary expires."""
78+
79+
detection_confidence_threshold: float = 0.1
80+
"""Default detection confidence when the request omits an override."""
81+
6082
vlm_judge_enabled: bool = False
6183
"""Whether the VLM-as-judge router is mounted."""
6284

@@ -117,6 +139,12 @@ def load_config(env_path: Path | None = None) -> AppConfig:
117139
episode_cache_capacity = int(os.environ.get("EPISODE_CACHE_CAPACITY", "32"))
118140
episode_cache_max_mb = int(os.environ.get("EPISODE_CACHE_MAX_MB", "100"))
119141

142+
detection_models_dir = os.environ.get("DETECTION_MODELS_DIR", "./models")
143+
detection_model_digests = _detection_model_digests_env()
144+
detection_cache_max_size = _positive_int_env("DETECTION_CACHE_MAX_SIZE", 100)
145+
detection_cache_ttl_seconds = _positive_int_env("DETECTION_CACHE_TTL_SECONDS", 3600)
146+
detection_confidence_threshold = _bounded_float_env("DETECTION_CONFIDENCE_THRESHOLD", 0.1)
147+
120148
vlm_judge_enabled = os.environ.get("VLM_JUDGE_ENABLED", "false").lower() == "true"
121149
vlm_judge_backend = os.environ.get("VLM_JUDGE_BACKEND", "echo").lower()
122150
vlm_judge_model_id = os.environ.get("VLM_JUDGE_MODEL_ID", "Qwen/Qwen3-VL-4B-Instruct")
@@ -139,6 +167,11 @@ def load_config(env_path: Path | None = None) -> AppConfig:
139167
cors_origins=cors_origins,
140168
episode_cache_capacity=episode_cache_capacity,
141169
episode_cache_max_mb=episode_cache_max_mb,
170+
detection_models_dir=detection_models_dir,
171+
detection_model_digests=detection_model_digests,
172+
detection_cache_max_size=detection_cache_max_size,
173+
detection_cache_ttl_seconds=detection_cache_ttl_seconds,
174+
detection_confidence_threshold=detection_confidence_threshold,
142175
vlm_judge_enabled=vlm_judge_enabled,
143176
vlm_judge_backend=vlm_judge_backend,
144177
vlm_judge_model_id=vlm_judge_model_id,
@@ -151,6 +184,40 @@ def load_config(env_path: Path | None = None) -> AppConfig:
151184
)
152185

153186

187+
def _positive_int_env(name: str, default: int) -> int:
188+
value = int(os.environ.get(name, str(default)))
189+
if value <= 0:
190+
raise ValueError(f"{name} must be greater than zero")
191+
return value
192+
193+
194+
def _bounded_float_env(name: str, default: float) -> float:
195+
value = float(os.environ.get(name, str(default)))
196+
if not 0.0 <= value <= 1.0:
197+
raise ValueError(f"{name} must be between 0.0 and 1.0")
198+
return value
199+
200+
201+
def _detection_model_digests_env() -> Mapping[str, str]:
202+
name = "DETECTION_MODEL_DIGESTS"
203+
raw_value = os.environ.get(name, "{}")
204+
try:
205+
value: Any = json.loads(raw_value)
206+
except json.JSONDecodeError as exc:
207+
raise ValueError(f"{name} must be a JSON object") from exc
208+
if not isinstance(value, dict):
209+
raise ValueError(f"{name} must be a JSON object")
210+
211+
digests: dict[str, str] = {}
212+
for model_name, digest in value.items():
213+
if model_name not in ALLOWED_DETECTION_MODELS:
214+
raise ValueError(f"{name} contains an unapproved model identifier")
215+
if not isinstance(digest, str) or re.fullmatch(r"[0-9a-fA-F]{64}", digest) is None:
216+
raise ValueError(f"{name} values must be SHA-256 digests")
217+
digests[model_name] = digest.lower()
218+
return MappingProxyType(digests)
219+
220+
154221
def create_annotation_storage(config: AppConfig):
155222
"""
156223
Create the annotation storage adapter based on config.
Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
"""Constants for YOLO object detection."""
2+
3+
from __future__ import annotations
4+
5+
ALLOWED_DETECTION_MODELS = frozenset(
6+
{
7+
"yolo11n",
8+
"yolo11s",
9+
"yolo11m",
10+
"yolo11l",
11+
"yolo11x",
12+
"yolov8s-world",
13+
"yolov8m-world",
14+
"yolov8l-world",
15+
"yolov8x-worldv2",
16+
}
17+
)
18+
19+
COCO_CLASSES = (
20+
"person",
21+
"bicycle",
22+
"car",
23+
"motorcycle",
24+
"airplane",
25+
"bus",
26+
"train",
27+
"truck",
28+
"boat",
29+
"traffic light",
30+
"fire hydrant",
31+
"stop sign",
32+
"parking meter",
33+
"bench",
34+
"bird",
35+
"cat",
36+
"dog",
37+
"horse",
38+
"sheep",
39+
"cow",
40+
"elephant",
41+
"bear",
42+
"zebra",
43+
"giraffe",
44+
"backpack",
45+
"umbrella",
46+
"handbag",
47+
"tie",
48+
"suitcase",
49+
"frisbee",
50+
"skis",
51+
"snowboard",
52+
"sports ball",
53+
"kite",
54+
"baseball bat",
55+
"baseball glove",
56+
"skateboard",
57+
"surfboard",
58+
"tennis racket",
59+
"bottle",
60+
"wine glass",
61+
"cup",
62+
"fork",
63+
"knife",
64+
"spoon",
65+
"bowl",
66+
"banana",
67+
"apple",
68+
"sandwich",
69+
"orange",
70+
"broccoli",
71+
"carrot",
72+
"hot dog",
73+
"pizza",
74+
"donut",
75+
"cake",
76+
"chair",
77+
"couch",
78+
"potted plant",
79+
"bed",
80+
"dining table",
81+
"toilet",
82+
"tv",
83+
"laptop",
84+
"mouse",
85+
"remote",
86+
"keyboard",
87+
"cell phone",
88+
"microwave",
89+
"oven",
90+
"toaster",
91+
"sink",
92+
"refrigerator",
93+
"book",
94+
"clock",
95+
"vase",
96+
"scissors",
97+
"teddy bear",
98+
"hair drier",
99+
"toothbrush",
100+
)

data-management/viewer/backend/src/api/models/detection.py

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@
55
endpoints and match the frontend TypeScript type definitions.
66
"""
77

8+
from __future__ import annotations
9+
810
from pydantic import Field, field_validator
911

1012
from ..validation import SanitizedModel
@@ -17,14 +19,15 @@ class DetectionRequest(SanitizedModel):
1719
default=None,
1820
description="Specific frame indices to process. If None, processes all frames.",
1921
)
20-
confidence: float = Field(
21-
default=0.1,
22+
confidence: float | None = Field(
23+
default=None,
2224
ge=0.0,
2325
le=1.0,
24-
description="Minimum confidence threshold for detections.",
26+
description="Minimum confidence threshold. Uses the server-configured default when omitted.",
2527
)
2628
model: str = Field(
2729
default="yolo11n",
30+
max_length=64,
2831
description=(
2932
"YOLO model variant. Closed-vocabulary: yolo11n, yolo11s, yolo11m, yolo11l, yolo11x. "
3033
"Open-vocabulary (used when 'labels' is supplied): yolov8s-world, yolov8m-world, "

data-management/viewer/backend/src/api/routers/detection.py

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@
55
and retrieving cached results.
66
"""
77

8+
from __future__ import annotations
9+
810
import logging
911
import os
1012

@@ -14,7 +16,12 @@
1416
from ..models.detection import DetectionRequest, EpisodeDetectionSummary
1517
from ..rate_limiter import limiter
1618
from ..services.dataset_service import DatasetService, get_dataset_service
17-
from ..services.detection_service import DetectionService, get_detection_service
19+
from ..services.detection_service import (
20+
DetectionModelUnavailableError,
21+
DetectionService,
22+
InvalidDetectionModelError,
23+
get_detection_service,
24+
)
1825
from ..validation import SAFE_DATASET_ID_PATTERN, path_int_param, path_string_param
1926

2027
router = APIRouter()
@@ -56,7 +63,7 @@ async def run_detection(
5663
_sanitize_for_log(dataset_id),
5764
int(episode_idx),
5865
_sanitize_for_log(request_body.model),
59-
float(request_body.confidence),
66+
float(detection_service.effective_confidence(request_body)),
6067
)
6168

6269
# Validate episode exists
@@ -106,6 +113,14 @@ async def get_frame_image(frame_idx: int) -> bytes | None:
106113
total_frames,
107114
)
108115
return summary
116+
except InvalidDetectionModelError as exc:
117+
raise HTTPException(status_code=400, detail=str(exc))
118+
except DetectionModelUnavailableError:
119+
logger.exception(
120+
"Configured detection model is unavailable for model %s",
121+
_sanitize_for_log(request_body.model),
122+
)
123+
raise HTTPException(status_code=503, detail="Configured detection model is unavailable")
109124
except ImportError:
110125
raise HTTPException(
111126
status_code=503,

0 commit comments

Comments
 (0)