Skip to content

run fuzz in CI#165

Merged
Zac-HD merged 1 commit intopython-trio:mainfrom
jakkdl:CI_tests
Mar 23, 2023
Merged

run fuzz in CI#165
Zac-HD merged 1 commit intopython-trio:mainfrom
jakkdl:CI_tests

Conversation

@jakkdl
Copy link
Copy Markdown
Member

@jakkdl jakkdl commented Mar 22, 2023

let's see how long they take to run in CI, I usually don't run them in my workflow locally since they're slow - but it's probably nice if they're run regularly somewhere

@jakkdl
Copy link
Copy Markdown
Member Author

jakkdl commented Mar 22, 2023

The check is once again fixed in #163, I kind of expect the fuzz tests to fail currently - so make sure to rerun after merging the others ... if the runtime looks the least bit reasonable.

@Zac-HD
Copy link
Copy Markdown
Member

Zac-HD commented Mar 22, 2023

~20 minutes to run the fuzztests seems fine to me, we could make that a non-required CI check. pytest-xdist would probably make that ~2x faster too.

If we're getting up above 20-30 minutes, I'd probably make this a scheduled job and look at the results occasionally. Fancier schemes can actually share a cache between them and use that to feed discovered failures back into the main job, but that's still pretty early for Hypothesis+HypoFuzz so I don't think I'd use it here yet.

@jakkdl
Copy link
Copy Markdown
Member Author

jakkdl commented Mar 23, 2023

Added a flag so slow_tests only runs the slow tests, added xdist, and rebased on top of main.
with "non-required" do you simple mean not listing it in needs for the release test?

Amusingly enough it actually runs faster in CI than on my personal computer now, due to me having way more installed packages ... _iter_python_files() gives 3900 files 😅 (inside tox environment it's a more reasonable 1561 ... which itself takes 11 mins on my laptop 😔 )

@Zac-HD Zac-HD merged commit e66e8a1 into python-trio:main Mar 23, 2023
@Zac-HD
Copy link
Copy Markdown
Member

Zac-HD commented Mar 23, 2023

with "non-required" do you simple mean not listing it in needs for the release test?

I mean "oops, let me go set up some branch protection rules so you can't merge a PR until the tests pass" 😆

@jakkdl jakkdl deleted the CI_tests branch March 24, 2023 10:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants