Fix #136: audit fan-out + legacy migration subdir + cache_path warnings#137
Merged
Merged
Conversation
…oNirin#136) The legacy→path_mappings migration copied cache_dir verbatim into the new mapping's cache_path. For users whose legacy cache_dir was the bare cache drive root ('/mnt/cache/') and real_source was a subdirectory ('/mnt/user/Media/'), this produced a mapping that made run_full_audit() walk the entire cache drive — appdata, docker, every other share. Fixes: - core/config.py: new _derive_migrated_cache_path() helper mirrors the real_source subpath under /mnt/cache/ when cache_dir is the bare root. migrate_path_settings() now uses the derived value. Configs that already have a specific cache_dir are preserved. - web/services/settings_service.py: new detect_path_mapping_health_issues() method scans enabled path_mappings for the two failure modes from StudioNirin#136: cache_path set to the cache drive root, or cache_path pointing at the Unraid FUSE merged view (/mnt/user/...) instead of /mnt/cache/. Read-only — never auto-rewrites user config. - web/main.py: startup runs the health check and logs any issues as WARNINGs. Non-fatal on exception. - web/routers/api.py: new GET /api/config-health endpoint renders the detected issues as alert partials. - web/templates/dashboard.html: inline #config-health-banner lazy-loads from the endpoint on page load. - tests/test_config_migration.py: new file. Covers the subdir-mirroring helper (issue StudioNirin#136 scenario, already-specific cache_dir, multi-level subdir, non-/mnt/user/ paths, empty inputs) and end-to-end migration. - tests/test_libraries_settings.py: new TestDetectPathMappingHealthIssues class covering cache_root, fuse_cache_path, /mnt/user0/ exemption, disabled-mappings skipping, and the exact reporter config.
…tudioNirin#136) Advisory-only checks for the two cache_path patterns that caused the dashboard stall in issue StudioNirin#136. Never blocks: some users legitimately store media at the cache drive root, and some containers only have /mnt/user mounted. The warnings surface in logs and via the dashboard health banner added in the previous commit. - SettingsService.warn_cache_path() returns a human-readable warning string for known-risky values (bare /mnt/cache root, /mnt/user/... FUSE paths) or None for everything else. Explicitly does not flag /mnt/user0/ (array-direct) or custom mount points. Warning text acknowledges the legitimate use cases. - POST /settings/paths and PUT /settings/paths/{index} call the warner and log a WARNING but always persist the user's value. - auto_fill_mapping() no longer trusts settings['cache_dir'] when it starts with /mnt/user/. A stale legacy cache_dir would otherwise leak into newly auto-created mappings. Users can still type /mnt/user/... manually and save. - Softened detect_path_mapping_health_issues() messages to match the advisory tone, referencing the legitimate cases. Adds 10 tests covering warn_cache_path (valid subdirs, empty, bare root with/without slash, FUSE with suggestion, /mnt/user0/ exemption, custom mounts) and the auto_fill tightening (stale cache_dir doesn't leak).
…in#136) run_full_audit previously issued 3-4 os.path.exists probes per cache file (backup check, duplicate check, second duplicates pass). On a 1.23M-file library that totalled ~4.9M probes and 26 minutes per cycle, saturating the 5-minute background refresh thread. Walk the array dirs exactly once inside _get_orphaned_plexcached, build array_files_set and plexcached_set during that same walk, and answer all backup/duplicate questions via O(1) set lookups. Collapse the old separate duplicates pass into the main cache-file loop so cache files are iterated once. Also log run_full_audit total wall time at INFO (phase timings at DEBUG) and warn in WebCacheService.refresh_all when a cycle exceeds REFRESH_INTERVAL_SECONDS, surfacing the underlying symptom.
StudioNirin
approved these changes
Apr 9, 2026
StudioNirin
left a comment
Owner
There was a problem hiding this comment.
Can't see any issues, I'll leave this for you to merge in case there was anything you wanted to double-check
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #136.
Summary
Three commits addressing the root cause and two contributing configuration issues behind the 26-minute audit cycles reported in #136. The reporter's server (1.23M cache files) was issuing ~4.9M
os.path.existsprobes per audit, saturating the 5-minuteWebCacheServicerefresh thread and keeping Unraid array disks spun up indefinitely.1. Audit fan-out fix (perf) — primary fix
MaintenanceService.run_full_audit()previously issued 3-4os.path.existsprobes per cache file:_check_plexcached_backup()— one probe on/mnt/user0/...plexcached_check_array_duplicate()— one probe on/mnt/user0/..._check_array_duplicate()on every cache file_get_orphaned_plexcached()separately walked every array dirOn cold Unraid arrays each probe blocks on disk spinup; even on a warm array the aggregate wall time on a 1.23M-file library was ~26 minutes per audit.
Fix: walk the array disks exactly once inside
_get_orphaned_plexcached, buildarray_files_setandplexcached_setduring that same walk, and answer all backup/duplicate questions via O(1) set lookups. The duplicates pass is collapsed into the main cache-file loop so cache files are iterated once instead of twice. Zero extra array I/O vs. the walk that was already happening.Also adds observability so this regression can't silently creep back:
run_full_auditlogs total wall time at INFO and per-phase timings (scan / array_walk / main_loop) at DEBUG.WebCacheService.refresh_all()logs a WARNING when a single refresh cycle exceedsREFRESH_INTERVAL_SECONDS, surfacing the exact symptom from Dashboard Stuck Loading - Unraid Docker Version #136.2. Legacy→
path_mappingsmigration subdir bugThe reporter's migrated config had:
{ "real_path": "/mnt/user/Media/", "cache_path": "/mnt/cache/" // ← should be /mnt/cache/Media/ }Because
cache_pathpointed at the cache drive root, the audit walked the entire SSD (appdata, docker, system, every container) — 1.23M files. The migration now mirrors thereal_pathsubdirectory intocache_path. Ships with a read-onlydetect_path_mapping_health_issuescheck and dashboard banner so existing bad configs get flagged loudly on startup without auto-rewriting user state.3.
cache_pathvalidation (warn, don't block)PathMapping.cache_pathvalues like/mnt/cache(bare root) or/mnt/user/...(FUSE) cause audit fan-out or make_check_array_duplicatesilently wrong. The final approach is warn-only, never blocking — some configs legitimately use/mnt/cache/as a flat root or/mnt/user/...in containers where/mnt/cacheisn't mounted.SettingsService.warn_cache_path()helper returns a human-readable warning for risky values, orNoneotherwise. Exempts/mnt/user0/and custom mount points.POST /settings/paths,PUT /settings/paths/{index}) call the warner, log a WARNING, and always persist the user's value.auto_fill_mapping()refuses to trustsettings['cache_dir']when it starts with/mnt/user/— falls back to/mnt/cacheto prevent stale legacy values from propagating into newly auto-generated mappings.Commits
Fix legacy migration cache_path subdir and add health detector (#136)Warn (non-blocking) on risky cache_path values and tighten auto-fill (#136)Fix audit fan-out: single array walk with O(1) set lookups (#136)Expected impact
Test plan
tests/test_config_migration.py, 12 tests) covering the exactreal_path=/mnt/user/Media/→cache_path=/mnt/cache/Media/shape.cache_pathwarnings +auto_fill_mappingtightening (tests/test_libraries_settings.py, 28 tests).TestRunFullAuditFanOuttests (tests/test_maintenance_service.py): one verifies unprotected/duplicate/orphaned detection is preserved against real tmp_path files, the other patchesos.path.existsand asserts it is not called per individual cache file — guards against reintroducing the fan-out._check_plexcached_backup/_check_array_duplicatehelper tests still pass (helpers are untouched; only the audit loop bypasses them with set lookups).os.path.existscall count and wall time before/after (reporter willing).Notes for reviewer
_get_orphaned_plexcached()signature changed to return a 4-tuple(orphaned, extensionless, array_files_set, plexcached_set)and accepts an optionalcache_filesarg sorun_full_auditdoesn't re-scan. All five in-repo callers updated.cache_pathwarnings are intentionally non-blocking per team discussion — blocking the save path would break legitimate flat-root and container configurations.run_full_audit().