-
Notifications
You must be signed in to change notification settings - Fork 1
Open
Labels
enhancementNew feature or requestNew feature or request
Description
NOTE: Don't assign yourself unless you have have confirmed with Matthew you've got a working environment
🧠 Context
The last_synced_at field on a source is intended to track when it was most recently ingested. However, this field is currently not updated after ingestion runs, which makes it harder to determine whether chunks are stale or current.
This field needs to be updated immediately after a source is successfully ingested to ensure downstream logic (like stale chunk cleanup) works correctly.
🛠 Implementation Plan
-
In
src/ingestion/services/webpage_ingestion_service.py, after successfully processing and storing chunks:- Set
source.last_synced_at = datetime.utcnow(). - Save the updated
sourceto the database.
- Set
-
Ensure this happens before triggering the cleanup task.
-
Add a test to confirm that
last_synced_atis updated after ingestion.
✅ Acceptance Criteria
- After ingestion completes successfully for a source, update
source.last_synced_atto the current timestamp. - Save the updated source object to persist the change.
- This should happen in
WebpageIngestionServiceafter all chunks are created and saved.
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Type
Projects
Status
Ready