[NO MRG] Test CI #952

jakirkham · 2022-01-26T19:19:52Z

Opening just to test the current state of CI

jakirkham · 2022-01-27T02:45:49Z

Am seeing the same failure with commit ( e15001b ) on master (the latest change). Going to try reverting that to see what happens

Edit: Should add this seems potentially related as tests are hanging in test_sync.py as seen in this log and the latest change updated fasteners a library we use for locking.

Edit 2: Fortunately the changes in fasteners between 0.17.2 and 0.17.3 are quite small. So hopefully the issue is easy to narrow down once we have a reproducer.

Edit 3: Still seeing issues with 0.17.2, which came out a little over a week back. Trying 0.17.1.

We seem to be experiencing hangs in `test_sync.py` with this version. So exclude it from testing.

requirements_dev_minimal.txt

Co-authored-by: Josh Moore <[email protected]>

jakirkham · 2022-01-28T08:30:40Z

Here's a dependency diff between CI for commit ( 21739ad )'s run and commit ( e15001b )'s run where the issue first appeared

Diff:

27c27
< coverage==6.2
---
> coverage==6.3
38c38
< fasteners==0.17.2
---
> fasteners==0.17.3
62c62
< jsondiff==1.3.0
---
> jsondiff==1.3.1
68c68
< jupyter-client==7.1.1
---
> jupyter-client==7.1.2
78c78
< moto==2.3.2
---
> moto==3.0.1
82c82
< multidict==5.2.0
---
> multidict==6.0.2
90c90
< notebook==6.4.7
---
> notebook==6.4.8
103c103
< prometheus-client==0.12.0
---
> prometheus-client==0.13.0
107c107
< pure-eval==0.2.1
---
> pure-eval==0.2.2
115c115
< pyparsing==3.0.6
---
> pyparsing==3.0.7

jakirkham · 2022-01-28T08:50:34Z

Spotted issue ( harlowja/fasteners#86 ), which may also be related.

jakirkham · 2022-01-28T09:13:39Z

Looks like it was still hanging even with 0.16. Going to try running with more verbosity.

jakirkham · 2022-01-28T10:13:05Z

Canceled the tests and looked at the tests run. They were definitely using the ProcessSynchronizer. There wasn't any one test that was jamming things up. Each job had a different test running when canceled. This makes me think there are still some locking issues, but when they crop up is sporadic.

jakirkham · 2022-01-28T10:34:29Z

@joshmoore would be good to get your thoughts here on what we should do next? Should we debug further? Should we xfail the ProcessSynchronizer tests and go ahead with the release? Or should we do something else?

joshmoore · 2022-01-28T10:39:05Z

Hmmmm..... so we're sure it's just a change in fasteners and not (also) a change in the mainline that's leading to this?

It would definitely be good to get 2.11 out. We could move the synchronizer tests out to their own workflow (which only gets run periodically?)

jakirkham · 2022-01-28T11:20:19Z

To be completely honest, I'm not sure.

At the moment I'm torn between whether a new copy of fasteners is sneaking in somewhere despite the version constraints or whether there is some subtle locking issue introduced by not writing empty chunks (or maybe something else?).

That said, CI on master was passing until the most recent commit, which coincides with the recent fasteners release.

Davis' change ( 831e687 ) went in a couple months back (October) and issues only cropped up recently. Think we would have seen issues sooner if that was the cause.

All of the testing here is without the recent empty chunks change and it still has issues. So doesn't seem like that's the cause.

Finally where the tests slow down is when testing ProcessSynchronizer, which is the one place fasteners is used.

To summarize, there isn't an obvious cause outside of fasteners. Though it is disquieting that pinning doesn't seem to work.

joshmoore · 2022-01-28T11:37:20Z

Currently trying to reproduce locally with:

pytest -svvvx --count=1000 --random-order zarr/tests/test_sync.py

psarka · 2022-01-28T14:35:54Z

Hi, fasteners maintainer here, saw a note in our issue about the deadlock, got interested. I'm puzzled by this thing, looking at the commit log I see that your tests passed 8 days ago with fasteners 17.2, but didn't with 2 days ago with fasteners 17.3:

Which is weird, as you are using ProcessLock, which was not touched in 17.3. Maybe it would be helpful to rerun your CI on the commit 21739ad (the one that was fine 8 days ago) and see if it still passes?

joshmoore · 2022-01-28T14:38:43Z

Thanks, @psarka. Happy to give it a try. The previous action run was:

https://github.com/zarr-developers/zarr-python/runs/4878605616?check_suite_focus=true

I've just re-launched. Let's see what happens.

psarka · 2022-01-28T14:52:33Z

Looks like it is hanging 🤯

Is there any other explanation, other than that something changed with github actions infrastructure? I'll rerun fasteners CI just in case.

Edit: fasteners are passing fine. I'm out of ideas.

jakirkham · 2022-01-28T16:44:16Z

Thanks for taking a look Paulius! 😄

Yeah the hang seems to be occurring with the ProcessSynchronizer tests, which use ProcessLock. That was what we were seeing here as well.

That said, we probably need to turn this into a smaller reproducer so it is a bit easier to dig into.

joshmoore · 2022-02-03T15:49:35Z

I got side tracked by trying to reproduce locally with https://github.com/nektos/act (unsuccessfully). I will give up on that and try to find a way forward.

joshmoore · 2022-02-03T21:01:57Z

Plan discussed with @jakirkham:

I will try on bare-metal linux & linux/docker to try to reproduce.
John will try to tease apart whether or not the version of fasteners that we expect is actually being installed.
If we can't find a solution by tomorrow EOB, we will disable the synchronizer test in GHA and move forward with the 2.11 release.

joshmoore · 2022-02-04T08:58:50Z

Linux & Linux/docker don't show a hang locally when following the python-package.yml workflow instructions. Currently running with --count=1000 but I don't expect it will show anything.

jakirkham · 2022-02-04T09:04:17Z

More strange is we saw PR ( #955 ) pass. Trying to see if we can replicate that elsewhere. If so, maybe it was a GH Action's change?

joshmoore · 2022-02-04T09:26:06Z

Maybe it was a temporary issue in our segment of the GHA cloud?! (if so, reminder for next time: contact GH earlier)

jakirkham · 2022-02-04T09:27:01Z

Checking here as well ( #950 ) just to make sure this is consistently working

jakirkham · 2022-02-04T10:01:19Z

Looks like things are working 🎉

Sorry for the false alarm @psarka and thanks for the help 🙂

Going to go ahead and close

* Set write_empty_chunks to default to False * Add release entry for write_empty_chunks default * add Empty chunks section to tutorial.rst * add benchmarky example * proper formatting of code block * Fix abstore deprecated strings * Also catch ValueError in all_equal The call to `np.any(array)` in zarr.util.all_equal triggers the following ValueError: ``` > return ufunc.reduce(obj, axis, dtype, out, **passkwargs) E ValueError: invalid literal for int() with base 10: 'baz' ``` Extending the catch block allows test_array_with_categorize_filter to pass, but it's unclear if this points to a deeper issue. * Add --timeout argument to all uses of pytest * Pin fasteners to 0.16.3 (see #952) Co-authored-by: Davis Vann Bennett <[email protected]> Co-authored-by: Josh Moore <[email protected]> Co-authored-by: jmoore <[email protected]>

Test CI

323f9cd

jakirkham mentioned this pull request Jan 26, 2022

Default write empty chunks false #951

Closed

Exclude fasteners 0.17.3

5d90297

We seem to be experiencing hangs in `test_sync.py` with this version. So exclude it from testing.

jakirkham force-pushed the tst_ci branch from 62d100c to 5d90297 Compare January 27, 2022 02:55

Only use fasteners <0.17.2

262ac90

joshmoore reviewed Jan 27, 2022

View reviewed changes

requirements_dev_minimal.txt Outdated Show resolved Hide resolved

joshmoore added a commit to joshmoore/zarr-python that referenced this pull request Jan 27, 2022

Pin fasteners to 0.16.3 (see zarr-developers#952)

080a854

Fix remaining fasteners constraint

f5b3663

Co-authored-by: Josh Moore <[email protected]>

joshmoore mentioned this pull request Jan 28, 2022

add release note regarding introduction of BaseStore and KVStore #950

Merged

1 task

jakirkham added 2 commits January 28, 2022 00:43

Revert back to fasteners version 0.17.2

20f8115

Use fasteners pre-0.17

24039e8

Max pytest verbosity

ea83d93

jakirkham force-pushed the tst_ci branch from b88cbe7 to ea83d93 Compare January 28, 2022 09:26

Revert all changes and retest CI

d9d579c

jakirkham closed this Feb 4, 2022

jakirkham deleted the tst_ci branch February 4, 2022 10:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[NO MRG] Test CI #952

[NO MRG] Test CI #952

jakirkham commented Jan 26, 2022

jakirkham commented Jan 27, 2022 •

edited

Loading

jakirkham commented Jan 28, 2022 •

edited

Loading

jakirkham commented Jan 28, 2022

jakirkham commented Jan 28, 2022

jakirkham commented Jan 28, 2022

jakirkham commented Jan 28, 2022

joshmoore commented Jan 28, 2022

jakirkham commented Jan 28, 2022 •

edited

Loading

joshmoore commented Jan 28, 2022 •

edited

Loading

psarka commented Jan 28, 2022 •

edited

Loading

joshmoore commented Jan 28, 2022

psarka commented Jan 28, 2022 •

edited

Loading

jakirkham commented Jan 28, 2022

joshmoore commented Feb 3, 2022

joshmoore commented Feb 3, 2022

joshmoore commented Feb 4, 2022

jakirkham commented Feb 4, 2022

joshmoore commented Feb 4, 2022

jakirkham commented Feb 4, 2022

jakirkham commented Feb 4, 2022

[NO MRG] Test CI #952

[NO MRG] Test CI #952

Conversation

jakirkham commented Jan 26, 2022

jakirkham commented Jan 27, 2022 • edited Loading

jakirkham commented Jan 28, 2022 • edited Loading

jakirkham commented Jan 28, 2022

jakirkham commented Jan 28, 2022

jakirkham commented Jan 28, 2022

jakirkham commented Jan 28, 2022

joshmoore commented Jan 28, 2022

jakirkham commented Jan 28, 2022 • edited Loading

joshmoore commented Jan 28, 2022 • edited Loading

psarka commented Jan 28, 2022 • edited Loading

joshmoore commented Jan 28, 2022

psarka commented Jan 28, 2022 • edited Loading

jakirkham commented Jan 28, 2022

joshmoore commented Feb 3, 2022

joshmoore commented Feb 3, 2022

joshmoore commented Feb 4, 2022

jakirkham commented Feb 4, 2022

joshmoore commented Feb 4, 2022

jakirkham commented Feb 4, 2022

jakirkham commented Feb 4, 2022

jakirkham commented Jan 27, 2022 •

edited

Loading

jakirkham commented Jan 28, 2022 •

edited

Loading

jakirkham commented Jan 28, 2022 •

edited

Loading

joshmoore commented Jan 28, 2022 •

edited

Loading

psarka commented Jan 28, 2022 •

edited

Loading

psarka commented Jan 28, 2022 •

edited

Loading