Skip to content

[7/n] [installinator] fix reports getting delayed #8041

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

sunshowers
Copy link
Contributor

@sunshowers sunshowers commented Apr 24, 2025

There's been a long-standing issue with the installinator where reports to wicketd get delayed quite substantially. That happens because we use one task to send reports out to every peer, blocking until all of those peers come back.

I'd assumed in the past that if installinator reached out to an unreachable peer, it would receive a TCP connection refused message, but that isn't the case -- instead, it times out. This causes reports to only be sent out roughly every 15 seconds which isn't ideal.

To fix this issue, spin up separate report tasks for each peer. Introduce separate tasks for:

  • discovery (make this a persistent task that publishes updates to a watch channel)
  • peer reconciliation (new peers have new report tasks spun up, while removed peers' report tasks are cancelled)
  • reporting (each peer now has its own report loop)

Also simulate some kinds of network flakiness in our property-based tests.

I did a mupdate on dublin and saw that installinator reports started coming through every 2 seconds rather than every 15 or so, as expected.

Depends on:

Created using spr 1.3.6-beta.1
Created using spr 1.3.6-beta.1

[skip ci]
Created using spr 1.3.6-beta.1
Created using spr 1.3.6-beta.1

[skip ci]
Created using spr 1.3.6-beta.1
Copy link
Contributor

@jgallagher jgallagher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this!

Created using spr 1.3.6-beta.1

[skip ci]
Created using spr 1.3.6-beta.1
@sunshowers sunshowers changed the base branch from sunshowers/spr/main.7n-installinator-fix-reports-getting-delayed to main April 26, 2025 00:18
Created using spr 1.3.6-beta.1
@sunshowers sunshowers merged commit 4a0e9c1 into main Apr 26, 2025
17 checks passed
@sunshowers sunshowers deleted the sunshowers/spr/7n-installinator-fix-reports-getting-delayed branch April 26, 2025 20:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants