Skip to content

Conversation

@jakubno
Copy link
Member

@jakubno jakubno commented Dec 17, 2025

There's a race condition when the sandbox is added from sync and then the create fails


Note

Swallows ErrAlreadyExists on sandbox add and refactors insert flow into structured sync/async callbacks, updating orchestrator/metrics and adding comprehensive tests.

  • Sandbox Store:
    • Introduces Callbacks struct with AddSandboxToRoutingTable (sync), AsyncSandboxCounter, and AsyncNewlyCreatedSandbox (async) hooks.
    • Add now ignores ErrAlreadyExists (logs warning), calls routing-table + counter only on first insert, and triggers newly-created hook based on newlyCreated flag.
    • Updates InsertCallback signature (drops created arg); Sync uses Add(..., false).
  • Storage:
    • Memory storage Add returns sandbox.ErrAlreadyExists instead of string error.
    • Adds ErrAlreadyExists to sandbox/errors.go.
  • Orchestrator:
    • Replaces old insert/counter handlers with addSandboxToRoutingTable, sandboxCounterInsert, and handleNewlyCreatedSandbox (sends analytics, updates team metrics, increments created counter).
    • Cleans up analytics insert/remove handling and OTEL attribute usage.
  • Metrics:
    • TeamObserver.Add API simplified to Add(ctx, teamID) and always increments team_sandboxes_created with team_id.
  • Tests:
    • Adds store_test.go covering new callback behavior, already-existing cases, storage error propagation, and high-concurrency scenarios.

Written by Cursor Bugbot for commit 3a890d8. This will update automatically on new commits. Configure here.

cursor[bot]

This comment was marked as resolved.

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@jakubno jakubno requested a review from dobrac December 17, 2025 09:56
dobrac
dobrac previously approved these changes Dec 17, 2025
@jakubno jakubno marked this pull request as draft December 17, 2025 10:15
@dobrac dobrac self-requested a review December 17, 2025 10:30
@dobrac dobrac dismissed their stale review December 17, 2025 10:30

there are probably issues

@jakubno jakubno requested review from dobrac and removed request for dobrac December 17, 2025 15:21
@jakubno jakubno marked this pull request as ready for review December 17, 2025 15:21
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

so.teamSandboxesCreated.Add(ctx, 1, metric.WithAttributes(attributes...))
func (so *TeamObserver) Add(ctx context.Context, teamID uuid.UUID) {
attributes := []attribute.KeyValue{
attribute.String("team_id", teamID.String()),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

telemetry.WithTeamID

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't do it, because that changes it to team.id, that isn't compatible. Bless the tests 🙇🏻

Copy link
Contributor

@dobrac dobrac Dec 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, then one of the places is incorrect

callback(ctx, sandbox, newlyCreated)
}
// Add sandbox info to node for routing
s.callbacks.AddToSandboxRoutingTable(ctx, sandbox)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will be called twice for the race condition, is it idempotent correctly?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't make sense to call it multiple times. Changed it, so we're calling it only on the first insert

@jakubno jakubno force-pushed the fix-sandbox-already-added branch from 433ae5b to 2319697 Compare December 19, 2025 16:20
@jakubno jakubno merged commit 6dbf778 into main Dec 19, 2025
28 checks passed
@jakubno jakubno deleted the fix-sandbox-already-added branch December 19, 2025 17:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants