Feature or enhancement
Proposal:
The big question is: how we can ensure that Tachyon is trustworthy?
In particular: the profiler reports some frames, effectively: some distribution of gathered samples. How do we know if they actually match reality? How well do they match reality?
Do we even report the distribution of samples that's similar to how the actual app behaves? Do we report impossible frames?
These are not theoretical questions, as demonstrated by #151426, #151613 or #151022.
Especially in the non-blocking mode we don't have fully atomic reads. If we use too many remote calls, traverse the frames, coro or task-waiters in an inefficient way, or invalidate the cache incorrectly - there's a room for introducing the noise (bias? drift? whatever the name).
Unfortunately, the CI is not the right tool for the job, given that we should avoid flaky tests at all costs and the statistical nature of Tachyon means that we cannot aim for 100% perfection. Even in the blocking mode, we simply cannot guarantee that we block the target ideally.
With the pipeline of changes happening the Tachyon, I think we should introduce some "harness" for this purpose.
This was discussed offline with @pablogsal
Has this already been discussed elsewhere?
No response given
Links to previous discussion of this feature:
No response
Linked PRs
Feature or enhancement
Proposal:
The big question is: how we can ensure that Tachyon is trustworthy?
In particular: the profiler reports some frames, effectively: some distribution of gathered samples. How do we know if they actually match reality? How well do they match reality?
Do we even report the distribution of samples that's similar to how the actual app behaves? Do we report impossible frames?
These are not theoretical questions, as demonstrated by #151426, #151613 or #151022.
Especially in the non-blocking mode we don't have fully atomic reads. If we use too many remote calls, traverse the frames, coro or task-waiters in an inefficient way, or invalidate the cache incorrectly - there's a room for introducing the noise (bias? drift? whatever the name).
Unfortunately, the CI is not the right tool for the job, given that we should avoid flaky tests at all costs and the statistical nature of Tachyon means that we cannot aim for 100% perfection. Even in the blocking mode, we simply cannot guarantee that we block the target ideally.
With the pipeline of changes happening the Tachyon, I think we should introduce some "harness" for this purpose.
This was discussed offline with @pablogsal
Has this already been discussed elsewhere?
No response given
Links to previous discussion of this feature:
No response
Linked PRs
_remote_debugging: Tachyon Oracle #153806