Skip to content

Fix #136: audit fan-out + legacy migration subdir + cache_path warnings#137

Merged
Brandon-Haney merged 3 commits into
StudioNirin:mainfrom
Brandon-Haney:pr/issue-136
Apr 10, 2026
Merged

Fix #136: audit fan-out + legacy migration subdir + cache_path warnings#137
Brandon-Haney merged 3 commits into
StudioNirin:mainfrom
Brandon-Haney:pr/issue-136

Conversation

@Brandon-Haney

@Brandon-Haney Brandon-Haney commented Apr 9, 2026

Copy link
Copy Markdown
Collaborator

Fixes #136.

Summary

Three commits addressing the root cause and two contributing configuration issues behind the 26-minute audit cycles reported in #136. The reporter's server (1.23M cache files) was issuing ~4.9M os.path.exists probes per audit, saturating the 5-minute WebCacheService refresh thread and keeping Unraid array disks spun up indefinitely.

1. Audit fan-out fix (perf) — primary fix

MaintenanceService.run_full_audit() previously issued 3-4 os.path.exists probes per cache file:

  • _check_plexcached_backup() — one probe on /mnt/user0/...plexcached
  • _check_array_duplicate() — one probe on /mnt/user0/...
  • A second duplicates-only pass repeating _check_array_duplicate() on every cache file
  • _get_orphaned_plexcached() separately walked every array dir

On cold Unraid arrays each probe blocks on disk spinup; even on a warm array the aggregate wall time on a 1.23M-file library was ~26 minutes per audit.

Fix: walk the array disks exactly once inside _get_orphaned_plexcached, build array_files_set and plexcached_set during that same walk, and answer all backup/duplicate questions via O(1) set lookups. The duplicates pass is collapsed into the main cache-file loop so cache files are iterated once instead of twice. Zero extra array I/O vs. the walk that was already happening.

Also adds observability so this regression can't silently creep back:

  • run_full_audit logs total wall time at INFO and per-phase timings (scan / array_walk / main_loop) at DEBUG.
  • WebCacheService.refresh_all() logs a WARNING when a single refresh cycle exceeds REFRESH_INTERVAL_SECONDS, surfacing the exact symptom from Dashboard Stuck Loading - Unraid Docker Version #136.

2. Legacy→path_mappings migration subdir bug

The reporter's migrated config had:

{
  "real_path": "/mnt/user/Media/",
  "cache_path": "/mnt/cache/"        // ← should be /mnt/cache/Media/
}

Because cache_path pointed at the cache drive root, the audit walked the entire SSD (appdata, docker, system, every container) — 1.23M files. The migration now mirrors the real_path subdirectory into cache_path. Ships with a read-only detect_path_mapping_health_issues check and dashboard banner so existing bad configs get flagged loudly on startup without auto-rewriting user state.

3. cache_path validation (warn, don't block)

PathMapping.cache_path values like /mnt/cache (bare root) or /mnt/user/... (FUSE) cause audit fan-out or make _check_array_duplicate silently wrong. The final approach is warn-only, never blocking — some configs legitimately use /mnt/cache/ as a flat root or /mnt/user/... in containers where /mnt/cache isn't mounted.

  • New SettingsService.warn_cache_path() helper returns a human-readable warning for risky values, or None otherwise. Exempts /mnt/user0/ and custom mount points.
  • Save endpoints (POST /settings/paths, PUT /settings/paths/{index}) call the warner, log a WARNING, and always persist the user's value.
  • auto_fill_mapping() refuses to trust settings['cache_dir'] when it starts with /mnt/user/ — falls back to /mnt/cache to prevent stale legacy values from propagating into newly auto-generated mappings.
  • Dashboard health banner (shared with Fix hardcoded config path to use script location #2) persistently surfaces the warning until the user fixes it or decides to keep the value.

Commits

  • Fix legacy migration cache_path subdir and add health detector (#136)
  • Warn (non-blocking) on risky cache_path values and tighten auto-fill (#136)
  • Fix audit fan-out: single array walk with O(1) set lookups (#136)

Expected impact

  • Reporter's 1.23M-file library: ~26 min → a few seconds per audit cycle (single O(n) walk instead of ~4.9M probes).
  • Dashboard / maintenance tabs stop stalling behind the background refresh thread.
  • Unraid array disks no longer stay spun up by the refresh loop.

Test plan

  • New unit tests for the legacy migration (tests/test_config_migration.py, 12 tests) covering the exact real_path=/mnt/user/Media/cache_path=/mnt/cache/Media/ shape.
  • New unit tests for cache_path warnings + auto_fill_mapping tightening (tests/test_libraries_settings.py, 28 tests).
  • New end-to-end TestRunFullAuditFanOut tests (tests/test_maintenance_service.py): one verifies unprotected/duplicate/orphaned detection is preserved against real tmp_path files, the other patches os.path.exists and asserts it is not called per individual cache file — guards against reintroducing the fan-out.
  • Existing _check_plexcached_backup / _check_array_duplicate helper tests still pass (helpers are untouched; only the audit loop bypasses them with set lookups).
  • Full suite: 88/88 maintenance + migration + libraries tests pass on the PR branch.
  • Verification on the reporter's 1.23M-file production library — compare os.path.exists call count and wall time before/after (reporter willing).

Notes for reviewer

  • _get_orphaned_plexcached() signature changed to return a 4-tuple (orphaned, extensionless, array_files_set, plexcached_set) and accepts an optional cache_files arg so run_full_audit doesn't re-scan. All five in-repo callers updated.
  • The cache_path warnings are intentionally non-blocking per team discussion — blocking the save path would break legitimate flat-root and container configurations.
  • No API changes for callers of run_full_audit().

…oNirin#136)

The legacy→path_mappings migration copied cache_dir verbatim into the new
mapping's cache_path. For users whose legacy cache_dir was the bare cache
drive root ('/mnt/cache/') and real_source was a subdirectory
('/mnt/user/Media/'), this produced a mapping that made run_full_audit()
walk the entire cache drive — appdata, docker, every other share.

Fixes:
- core/config.py: new _derive_migrated_cache_path() helper mirrors the
  real_source subpath under /mnt/cache/ when cache_dir is the bare root.
  migrate_path_settings() now uses the derived value. Configs that already
  have a specific cache_dir are preserved.
- web/services/settings_service.py: new detect_path_mapping_health_issues()
  method scans enabled path_mappings for the two failure modes from StudioNirin#136:
  cache_path set to the cache drive root, or cache_path pointing at the
  Unraid FUSE merged view (/mnt/user/...) instead of /mnt/cache/.
  Read-only — never auto-rewrites user config.
- web/main.py: startup runs the health check and logs any issues as
  WARNINGs. Non-fatal on exception.
- web/routers/api.py: new GET /api/config-health endpoint renders the
  detected issues as alert partials.
- web/templates/dashboard.html: inline #config-health-banner lazy-loads
  from the endpoint on page load.
- tests/test_config_migration.py: new file. Covers the subdir-mirroring
  helper (issue StudioNirin#136 scenario, already-specific cache_dir, multi-level
  subdir, non-/mnt/user/ paths, empty inputs) and end-to-end migration.
- tests/test_libraries_settings.py: new TestDetectPathMappingHealthIssues
  class covering cache_root, fuse_cache_path, /mnt/user0/ exemption,
  disabled-mappings skipping, and the exact reporter config.
…tudioNirin#136)

Advisory-only checks for the two cache_path patterns that caused the
dashboard stall in issue StudioNirin#136. Never blocks: some users legitimately
store media at the cache drive root, and some containers only have
/mnt/user mounted. The warnings surface in logs and via the dashboard
health banner added in the previous commit.

- SettingsService.warn_cache_path() returns a human-readable warning
  string for known-risky values (bare /mnt/cache root, /mnt/user/... FUSE
  paths) or None for everything else. Explicitly does not flag
  /mnt/user0/ (array-direct) or custom mount points. Warning text
  acknowledges the legitimate use cases.

- POST /settings/paths and PUT /settings/paths/{index} call the warner
  and log a WARNING but always persist the user's value.

- auto_fill_mapping() no longer trusts settings['cache_dir'] when it
  starts with /mnt/user/. A stale legacy cache_dir would otherwise leak
  into newly auto-created mappings. Users can still type /mnt/user/...
  manually and save.

- Softened detect_path_mapping_health_issues() messages to match the
  advisory tone, referencing the legitimate cases.

Adds 10 tests covering warn_cache_path (valid subdirs, empty, bare root
with/without slash, FUSE with suggestion, /mnt/user0/ exemption, custom
mounts) and the auto_fill tightening (stale cache_dir doesn't leak).
…in#136)

run_full_audit previously issued 3-4 os.path.exists probes per cache
file (backup check, duplicate check, second duplicates pass). On a
1.23M-file library that totalled ~4.9M probes and 26 minutes per cycle,
saturating the 5-minute background refresh thread.

Walk the array dirs exactly once inside _get_orphaned_plexcached, build
array_files_set and plexcached_set during that same walk, and answer all
backup/duplicate questions via O(1) set lookups. Collapse the old
separate duplicates pass into the main cache-file loop so cache files
are iterated once.

Also log run_full_audit total wall time at INFO (phase timings at
DEBUG) and warn in WebCacheService.refresh_all when a cycle exceeds
REFRESH_INTERVAL_SECONDS, surfacing the underlying symptom.
@StudioNirin StudioNirin marked this pull request as ready for review April 9, 2026 23:45
@StudioNirin StudioNirin self-requested a review as a code owner April 9, 2026 23:45

@StudioNirin StudioNirin left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't see any issues, I'll leave this for you to merge in case there was anything you wanted to double-check

@Brandon-Haney Brandon-Haney merged commit 29fcc51 into StudioNirin:main Apr 10, 2026
2 checks passed
@Brandon-Haney Brandon-Haney deleted the pr/issue-136 branch April 12, 2026 05:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dashboard Stuck Loading - Unraid Docker Version

2 participants