Repository navigation
failOnPassedAfterRetry should not be a part of the cache key #113
Description
Many projects have a different value set for failOnPassedAfterRetry locally vs on CI systems. This will cause a remote cache miss for test tasks.
Since this property causes a test task to fail in more cases and thus not generate a cache entry it would mean that the successful outcomes of the test task are the same, its just a smaller set of cases where a cache entry is created.
Because of this, would it be possible to remove this flag as an input to the task and thus avoid this cache miss?
Activity
bigdaz commentedon Jul 22, 2021
The problem with removing the flag as an input would be the following case:
build1runs withfailOnPassedAfterRetry=false. The test task passes after some tests required a retry. Success is written to cache.build2runs withfailOnPassedAfterRetry=true. This task uses the cached output frombuild1and the test task passes when it should have failed.Consider a hypothetical test that always fails on the first invocation and always passes on retry. The test task on
build2should never pass, but with the proposed change it could pass when the result is taken from the cache.The real solution would be to consider a cache entry from
build2to satisfybuild1, but not vice versa.Something similar could be used for a task that runs a subset of tests: if there's a cache entry where all tests pass, then we could use it to avoid running a subset of tests. But we would never want to do the opposite.
runningcode commentedon Jul 23, 2021
That makes sense. For an ideal world we would only want caching in one direction. But in practice, I think the outcomes and successful task executions that are cached are the same. These are flaky tests. We are just caching the result when they're not flaky. This would be same as if the test happend to execute successfully the first time.
Another thing to consider is that the
maxRetriesproperty is not considered an input.runningcode commentedon Sep 29, 2021
For those who experience this miss but one of the two builds has
maxRetriesset to0settingfailOnPassedAfterRetry = trueon both CI and local builds will have the same effect. That is since there are no retries for local builds the pass / failure behavior is the same.Use same `failOnPassedAfterRetry` on CI and local.
Use same `failOnPassedAfterRetry` on CI and local.
Use same `failOnPassedAfterRetry` on CI and local.