Skip to content

Comments

chore: remove flaky settings since flaky=True is now the default for all tests#9016

Open
basvandijk wants to merge 1 commit intomasterfrom
basvandijk/rm-flaky
Open

chore: remove flaky settings since flaky=True is now the default for all tests#9016
basvandijk wants to merge 1 commit intomasterfrom
basvandijk/rm-flaky

Conversation

@basvandijk
Copy link
Collaborator

@basvandijk basvandijk commented Feb 23, 2026

Since 6cee23b all tests are automatically retried on failure (up to 3 times). When a test attempt passes after having failed in a previous attempt the test run will be considered flaky. This allows us to detect and fix flakiness ASAP without it being hidden.

This means the flaky = True settings are no longer necessary and can be removed.

@github-actions github-actions bot added the chore label Feb 23, 2026
@basvandijk basvandijk marked this pull request as ready for review February 23, 2026 21:25
@basvandijk basvandijk requested review from a team as code owners February 23, 2026 21:25
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pull request changes code owned by the Governance team. Therefore, make sure that
you have considered the following (for Governance-owned code):

  1. Update unreleased_changelog.md (if there are behavior changes, even if they are
    non-breaking).

  2. Are there BREAKING changes?

  3. Is a data migration needed?

  4. Security review?

How to Satisfy This Automatic Review

  1. Go to the bottom of the pull request page.

  2. Look for where it says this bot is requesting changes.

  3. Click the three dots to the right.

  4. Select "Dismiss review".

  5. In the text entry box, respond to each of the numbered items in the previous
    section, declare one of the following:

  • Done.

  • $REASON_WHY_NO_NEED. E.g. for unreleased_changelog.md, "No
    canister behavior changes.", or for item 2, "Existing APIs
    behave as before.".

Brief Guide to "Externally Visible" Changes

"Externally visible behavior change" is very often due to some NEW canister API.

Changes to EXISTING APIs are more likely to be "breaking".

If these changes are breaking, make sure that clients know how to migrate, how to
maintain their continuity of operations.

If your changes are behind a feature flag, then, do NOT add entrie(s) to
unreleased_changelog.md in this PR! But rather, add entrie(s) later, in the PR
that enables these changes in production.

Reference(s)

For a more comprehensive checklist, see here.

GOVERNANCE_CHECKLIST_REMINDER_DEDUP

Copy link
Contributor

@randombit randombit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks

@mbjorkqvist
Copy link
Contributor

Does the flaky=True default also apply to the tests in the Schedule Daily workflow, in particular, the FI and NNS nightly tests? For the FI nightly, these are the ICP, ICRC u64 and ICRC u256 tests:

  • //rs/ledger_suite/icp:ledger_canister_integration_golden_nns_state_test_tests/golden_nns_state_test
  • //rs/ledger_suite/icrc1:icrc_ledger_suite_integration_golden_state_upgrade_downgrade_test
  • //rs/ledger_suite/icrc1:icrc_ledger_suite_integration_golden_state_upgrade_downgrade_test_u256

@basvandijk
Copy link
Collaborator Author

@mbjorkqvist

Does the flaky=True default also apply to the tests in the Schedule Daily workflow ...

Yes it does. Is that something which you would rather not have for those tests?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants