Add lock for NFS write

[jakubno]

Check that the lock works as expected (logs were removed afterwards)

Left is with locking (cache NFS was cleaned and orchestrator redeployed, so both caches were clean)

Without the lock the same load timed out.

---

Note

Introduce a file-based lock with TTL and integrate it into cache writes to prevent concurrent NFS writes; add comprehensive tests.

• Storage / Locking:
  • New packages/shared/pkg/storage/lock: Implement file-based lock with TTL cleanup (TryAcquireLock, ReleaseLock, ErrLockAlreadyHeld).
  • Tests: Add concurrency, staleness, and path consistency tests for the lock.
• Cache Providers:
  • CachedObjectProvider: Wrap full-file cache writes with lock acquisition; skip on ErrLockAlreadyHeld; add warnings.
  • CachedSeekableObjectProvider:
    • Lock around chunk writes in writeChunkToCache.
    • Lock around size writes in writeLocalSize.
    • Skip writes if lock already held; emit warnings on lock errors.

^{Written by Cursor Bugbot for commit cc765fc. This will update automatically on new commits. Configure here.}

[linear]

ENG-3293 Implement locking mechanism for NFS cache to prevent usage spike

[jakubno]

bugbot run

[ValentaTomas]

I assume this PR is intended to prevent the avalanche of requests when we start multiple orchestrators?

[ValentaTomas]

Was the problem with the NFS then throughput or IOPS?

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ValentaTomas]

Consideration—with the way the cleanup script picks files, can it delete the lock files?

[ValentaTomas]

Alternative—can we make the writes/reads to NFS timeout if they are taking too long, and just fetch from storage instead?

[ValentaTomas]

Alternative—if we limit the concurrency of writing things to cache per orchestrator, can the usage reasonably flatten (until we have too many orchestrators)?

[jakubno]

Consideration—with the way the cleanup script picks files, can it delete the lock files?

yes

[jakubno]

Alternative—can we make the writes/reads to NFS timeout if they are taking too long, and just fetch from storage instead?

yes, but that's a different problem, you want to prevent the requests in the first place

[jakubno]

Alternative—if we limit the concurrency of writing things to cache per orchestrator, can the usage reasonably flatten (until we have too many orchestrators)?

This is caused by the concurrency, the traffic from POV of one of the nodes is completely reasonable (and legit)