fix: resolve hash mismatch and --indexer-path ignored in shallow index#85
Conversation
Two bugs caused the shallow index to be rebuilt on every server startup: 1. Hash length mismatch: shallow_index_manager.py and sqlite_index_manager.py used a 12-char truncated MD5 hash (hexdigest()[:12]) for the storage directory name, while project_settings.py used the full 32-char MD5 hash. This meant the index was written to e.g. /tmp/code_indexer/32697df368a6/ but looked for at /tmp/code_indexer/32697df368a617d19aac2f121d60624f/, so load_index() always failed and the index was always rebuilt from scratch. Fix: remove [:12] truncation from both files so all components use the same full 32-char MD5 hash. 2. --indexer-path flag ignored by shallow index: shallow_index_manager.py hardcoded tempfile.gettempdir() and ignored ProjectSettings.custom_index_root, so the --indexer-path CLI argument had no effect on the shallow index storage location (only the deep SQLite index respected it). Fix: check ProjectSettings.custom_index_root first, falling back to tempfile.gettempdir() when not set. Together these fixes ensure the shallow index is written to and loaded from the correct persistent location, eliminating unnecessary rebuilds on startup.
johnhuang316
left a comment
There was a problem hiding this comment.
Thanks for the fix — I agree there is a real inconsistency in how index paths are derived today.
That said, I could not confirm the main claim that the 12-char vs 32-char hash mismatch is what causes the shallow index to rebuild on every startup. From the current startup flow, shallow load/build seems to stay entirely within , and both paths there use the same 12-char hash.
What I could confirm is that is not fully wired into the actual index managers yet. This PR updates the shallow side, but still appears to use directly.
Could you please:
- narrow the PR description to the behavior that is definitely fixed,
- update the deep/SQLite path handling as well if full support is the goal, and
- add regression tests for the real shallow/deep index storage paths and load behavior?
johnhuang316
left a comment
There was a problem hiding this comment.
Thanks for the fix — I agree there is a real inconsistency in how index paths are derived today.
That said, I could not confirm the main claim that the 12-char vs 32-char hash mismatch is what causes the shallow index to rebuild on every startup. From the current startup flow, shallow load/build seems to stay entirely within ShallowIndexManager, and both paths there use the same 12-char hash.
What I could confirm is that --indexer-path is not fully wired into the actual index managers yet. This PR updates the shallow side, but SQLiteIndexManager still appears to use tempfile.gettempdir() directly.
Could you please:
- narrow the PR description to the behavior that is definitely fixed,
- update the deep/SQLite path handling as well if full
--indexer-pathsupport is the goal, and - add regression tests for the real shallow/deep index storage paths and load behavior?
Two bugs caused the shallow index to be rebuilt on every server startup:
Hash length mismatch: shallow_index_manager.py and sqlite_index_manager.py used a 12-char truncated MD5 hash (hexdigest()[:12]) for the storage directory name, while project_settings.py used the full 32-char MD5 hash. This meant the index was written to e.g. /tmp/code_indexer/32697df368a6/ but looked for at /tmp/code_indexer/32697df368a617d19aac2f121d60624f/, so load_index() always failed and the index was always rebuilt from scratch.
Fix: remove [:12] truncation from both files so all components use the same full 32-char MD5 hash.
--indexer-path flag ignored by shallow index: shallow_index_manager.py hardcoded tempfile.gettempdir() and ignored ProjectSettings.custom_index_root, so the --indexer-path CLI argument had no effect on the shallow index storage location (only the deep SQLite index respected it).
Fix: check ProjectSettings.custom_index_root first, falling back to tempfile.gettempdir() when not set.
Together these fixes ensure the shallow index is written to and loaded from the correct persistent location, eliminating unnecessary rebuilds on startup.