Skip to content

Closes #18287: Enable periodic synchronization for data sources #18747

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Mar 3, 2025

Conversation

jeremystretch
Copy link
Member

Closes: #18287

  • Add sync_interval field to DataSource
  • Automatically enqueue a SyncDataSourceJob when the configured interval for a DataSource changes

@jeremystretch jeremystretch added this to the v4.3 milestone Feb 26, 2025
@jeremystretch jeremystretch requested review from a team and jnovinger and removed request for a team February 26, 2025 21:03
@jeremystretch
Copy link
Member Author

I've noticed an issue that occurs when re-scheduling a job, but I don't think it's specific to this change. When enqueuing a new job after modifying the sync interval, the new job fill fail with an error like the following:

IntegrityError('duplicate key value violates unique constraint "core_job_job_id_key"\nDETAIL: Key (job_id)=(1f4fdacb-1825-451e-b324-05a2f7d2de66) already exists.')

I've seen this pop in other scenarios; I believe it's related to the background logic responsible for enqueuing Redis tasks. Will need to dig into this further.

@jeremystretch jeremystretch removed the request for review from jnovinger February 26, 2025 21:22
Copy link
Member

@jnovinger jnovinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues with the code, but ran in to what seem to be edge cases while testing the post-save signal handler.

Copy link
Member

@jnovinger jnovinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💥 all of my edge cases are resolved!

Except for the one other case I just now noticed, which may or may not be worth addressing. I'd call it non-blocking.

@jeremystretch jeremystretch merged commit 77b9820 into feature Mar 3, 2025
6 checks passed
@jeremystretch jeremystretch deleted the 18287-data-source-periodic-sync branch March 3, 2025 14:29
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 3, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants