Fix --trace for parametrized tests #6099

davidszotten · 2019-10-29T09:59:13Z

I'm not sure if this is the correct / best way to fix so please check
critically when reviewing

Without this, the second time it tries to stop in a parametrized
function it raises instead:

ValueError: --trace can't be used with a fixture named func!

Also lift the restriction that was disallowing --trace on tests with a parameter named func

RonnyPfannschmidt

great work in finding a temporary solution that sorts the issue for now

as its a necessary kludge, i proposed 2 changes to help make it more clear

src/_pytest/debugging.py

testing/test_pdb.py

src/_pytest/debugging.py

changelog/6099-bugfix.rst

davidszotten · 2019-10-29T19:31:04Z

ah, apologies i misread the instructions. while updating i also squashed the 3 commits, hope that's ok

RonnyPfannschmidt

Thanks for the swift follow ups,
great work

davidszotten · 2019-10-29T19:40:13Z

Thank you for the quick reviews!

blueyed

Thanks!
I've hardened the test a bit (and force-pushed, old head: b421aef).

blueyed · 2019-10-30T07:55:21Z

Also amended the changelog (punctuation).

davidszotten · 2019-10-30T09:06:28Z

Thanks @blueyed

blueyed · 2019-10-30T09:11:04Z

src/_pytest/debugging.py

+        # that needs to move to FunctionDefinition
+        new_list = list(pyfuncitem._fixtureinfo.argnames)
+        new_list.append("func")
+        pyfuncitem._fixtureinfo.argnames = tuple(new_list)


The false branch for "if "func" not in pyfuncitem._fixtureinfo.argnames:" is not covered.
@RonnyPfannschmidt
Is that something that can happen?
Can/should we cover it then?

blueyed · 2019-10-30T09:13:40Z

I've pushed a fixup to use functools.partial there.

dismissing for now, since it changed (using functools)

RonnyPfannschmidt · 2019-10-30T09:55:56Z

src/_pytest/debugging.py

-    new_list.append("func")
-    pyfuncitem._fixtureinfo.argnames = tuple(new_list)
+    pyfuncitem.obj = functools.partial(_pdb.runcall, testfunction)
+    pyfuncitem.obj.__name__ = testfunction.__name__


that solution checks out proper,
nonetheless this feels like a hijacking of the pr to me

nonetheless this feels like a hijacking of the pr to me

why?
If it wasn't wanted then you could uncheck the box that allows it.
It can always be reset also (I've explicitly mentioned the old head).
I'm just trying to help fixing the issue (and limitations that were in place due to unknown reasons).

Why shouldn't I push hardening/improvements when I have them locally already, so that they can be reviewed/tested on CI?

@davidszotten
What do you think?

Reset it to b421aef (from c45761c). The test was timing out on CI, and I do not want to leave the impression to hijack / being hostile.
Will try to fix this in my fork, and then leave a comment here.

the human interaction perception of such actions of merit can widely vary,
whiles some may be accepting of that, for me personally it would be the last time that i'd contribute to the project

we should perhaps clarify on what we hope to accomplish by enabling maintainers to push commits to other prs

for me the main reason is to help new contributors that struggle with the tooling,
for better solutions/code change suggestions outside of pre-commit,
i would always strongly suggest to use something like proposed edits for guidance

for me there is a very important difference between guiding/mentoring and accidentally taking over

and i perceive your actions as "taking over" instead of guiding/mentoring

for me personally it would be the last time that i'd contribute to the project

why?
It's not like anything has been merged here already, it's just still being worked on. References are/were provided to revert/reset it, and IMHO it is far better to have CI results and an actual commit rather than a diff in a comment. But I will stick to the latter then in the future here.

strongly suggest to use something like proposed edits for guidance

I'm doing that also, but here it was not fitting. Posting a diff then maybe instead.

actions demonstrate values and projects that brush over contributors/new contributors don't foster/respect them, and i don't want that kind of interaction

agreed. though as feedback to you @RonnyPfannschmidt , of course @blueyed is also a contributor and maybe we can think of a better way to explain the behaviours we want to promote and why than something as brief as "this feels like a hijacking of the pr"

davidszotten · 2019-10-30T10:39:11Z

i don't feel that strongly and can see both sides of this argument. i'm not sure i know enough about the internals to tell how "controversial" your new implementation is. but if that needs more discussion then that does feel like maybe it could be split out into a separate pr and leaving this pr to just tweak the existing workaround to work better

on the other hand, if this is just obviously better and fixes the problem, maybe it's a small enough change that it's better to just bundle this together

on the contributor management side, i'm relatively seasoned in open source and experienced enough to not mind, but someone more new to open source or this project might find it daunting and/or off-putting to have their pr "hijacked" (have someone else force-push a whole new / different implementation without talking it over first)

blueyed · 2019-10-30T10:48:35Z

Thanks for clarifying - it wasn't my intention to step on toes here.
It is reset, so we're back to where I can suggest to using functools here maybe.. :)

have their pr "hijacked" (have someone else force-push a whole new / different implementation without talking it over first)

FWIW, it was a separate commit on top. The previous force-push was for the test hardening and typo/changelog fix (I do not think that merits extra commits, and we cannot squash-merge (#4361)).

RonnyPfannschmidt · 2019-10-30T10:49:23Z

i also agree that the proposed/new solution by @blueyed is a better solution

however with the "maintainer/project hat on" i had to make a clear mention about the maintainer interaction behaviour here is potentially problematic

i'm happy that @davidszotten is on the accepting side and would like to see the proposed solution go into the final thing in a collaborative manner

davidszotten · 2019-10-30T11:05:45Z

ah, i missed that the impl change was a separate commit. however i think the comments about people feel like their prs are taken over would almost equally apply with new commits added to their pr.

as for multiple commits, i think it's easier to have multiple commits while in wip, and then (e.g.) ask the pr owner to squash once it's done

re: the actual impl (which i'm happy to force-push onto this myself). did you figure out why the tests hang?

davidszotten · 2019-10-30T11:20:16Z

@blueyed hm from what i can see, even with the impl based on partial, --trace is unhappy with a param named func (easier to see by running manually instead of inside pexpect)

RonnyPfannschmidt · 2019-10-30T11:29:59Z

ah, it needs a different wrapper, the partial has the positional argument named func already used up,

so it cant have that as fixture anymore

the solution is to create a wrapper with a complete own signature and invoke the running inside

davidszotten · 2019-10-30T20:22:12Z

actually, the issue is that the implementation of p(well, b)db.runcall "uses" a param named func for itself. i think the solution is to lift out its implementation. pr update coming soon

blueyed · 2019-10-30T21:44:12Z

src/_pytest/debugging.py

-    pyfuncitem._fixtureinfo.argnames = tuple(new_list)
+
+    # we need a copy of bdb.runcall because it takes `func` as a parameter
+    # which means we can't have a kwarg called func


Ah.. in 3.8 it is positional-only (https://github.com/python/cpython/blob/9e87d5d964f4f49c24dc3dd4bb7e84ec8db1876e/Lib/bdb.py#L621).

I'm not sure why we need to support kwargs named "func" for the param, when pytest does not allow for them to be kwargs in the first place? ("function already takes an argument 'param' with a default value")

import pytest @pytest.mark.parametrize("param", [1, 2]) def test_func_kw(param="foo"): pass

blueyed · 2019-10-30T21:46:40Z

@blueyed hm from what i can see, even with the impl based on partial, --trace is unhappy with a param named func (easier to see by running manually instead of inside pexpect)

This passes/passed? (on c45761c)

@pytest.mark.parametrize('func', [1,2])
def test_func(func, request):
    assert func in (1, 2)
    assert request.function.__name__ == "test_func"

Also, btw, please do not mention me in commit messages via @blueyed (triggers notifications, also later when being "merged around").

davidszotten · 2019-10-30T21:50:33Z

Also, btw, please do not mention me in commit messages via @blueyed (triggers notifications, also later when being "merged around").

ah, apologies. would you like me to edit that out? if so, can i keep your name without the @ for credit?

(not sure i understand what you mean by "merged around")

davidszotten · 2019-10-30T21:51:46Z

maybe the partial solution needs 3.8+ to work (sorry a little hard to tell the order of your two comments so maybe this is already what you were saying)

blueyed · 2019-10-30T21:53:04Z

ah, apologies. would you like me to edit that out? if so, can i keep your name without the @ for credit?

Use co-authored-by?

(not sure i understand what you mean by "merged around")

Every time the commit is used, also here already:

blueyed · 2019-10-30T21:53:45Z

maybe the partial solution needs 3.8+ to work (sorry a little hard to tell the order of your two comments so maybe this is already what you were saying)

No, I've tested that with 3.7.4. But might be different with older versions even more.

blueyed · 2019-10-30T21:54:52Z

Does #6099 (comment) not work for you (using partial with func as arg)? Which Python then?

davidszotten · 2019-10-30T21:56:15Z

python -V
Python 3.7.0
(pytest)david@odie:pytest (trace_parametrize)(2)$ cat p.py
import pdb

def foo(func):
    pass

pdb.runcall(foo, func=None)
(pytest)david@odie:pytest (trace_parametrize)(2)$ python p.py
Traceback (most recent call last):
  File "p.py", line 6, in <module>
    pdb.runcall(foo, func=None)
  File "/Users/david/.pyenv/versions/3.7.0/lib/python3.7/pdb.py", line 1600, in runcall
    return Pdb().runcall(*args, **kwds)
TypeError: runcall() got multiple values for argument 'func'

davidszotten · 2019-10-30T21:56:42Z

Use co-authored-by?

ah, hadn't heard of that before

blueyed · 2019-10-30T22:04:06Z

Use co-authored-by?

ah, hadn't heard of that before

You have it for ronny.. ;)
(GitHub might have added it)

pdb.runcall(foo, func=None)

Sure.
But the functools.partial solution does not use that, but pdb.runcall(func), doesn't it?
(pyfuncitem.obj = functools.partial(_pdb.runcall, testfunction), not func=testfunction)

It does not work when the original function as a kwarg "func", but like I've said pytest dues not support this for fixtures anyway.
All of these pass for me:

import pytest


@pytest.mark.parametrize('myparam', [1, 2])
def test_1(myparam, request):
    assert myparam in (1, 2)
    assert request.function.__name__ == "test_1"


@pytest.mark.parametrize('func', [1, 2])
def test_func(func, request):
    assert func in (1, 2)
    assert request.function.__name__ == "test_func"


@pytest.mark.parametrize('myparam', [1, 2])
def test_func_kw(request, myparam, func="foo"):
    assert myparam in (1, 2)
    assert func == "foo"
    assert request.function.__name__ == "test_func_kw"


# pytest fails here with:
# > function already takes an argument 'func' with a default value
# @pytest.mark.parametrize('func', [1, 2])
# def test_func_kw2(request, func="foo"):
#     pass

What combination am I missing?

davidszotten · 2019-10-30T22:09:24Z

for me, the partial implementation fails on your second test case (with --trace, raising TypeError: runcall() got multiple values for argument 'func') (it passes without --trace)

davidszotten · 2019-10-30T22:13:45Z

the signature of the original runcall is def runcall(self, func, *args, **kwds): which means it can't handle kwds containing func (as my pdb.runcall example shows)

blueyed · 2019-10-30T22:23:01Z

It's fixed here: python/cpython@a37f356#diff-5c870dcafc8a80ef16e789bb8e901de0.
But yeah, needs wrapping then.
You could use the same method though, instead of copying the internals / function body.

blueyed · 2019-10-30T22:24:37Z

But might be ok to just bail out for "func" on TypeErrors (like before).
I.e. it will be lean and clean, and will support "func" with 3.7.4+ automatically.

davidszotten · 2019-10-31T13:45:05Z

not sure i understand what you mean by

You could use the same method though, instead of copying the internals / function body

could you explain please?

davidszotten · 2019-10-31T13:46:44Z

@RonnyPfannschmidt do you have a preference between a more complicated implementation to support params named func on py < 3.7.4 vs just bailing out as we do now?

RonnyPfannschmidt · 2019-10-31T14:59:10Z

I'm on mobile, the trick is to make a new function that shares the original signature and invokes run all with a partial of the original function and it's arguments as the sole parameters

davidszotten · 2019-10-31T15:17:18Z

@RonnyPfannschmidt my understanding is that this is not possible (on py < 3.7.4), as per #6099 (comment)

because runcall takes the function to call as a param named func, func may not appear in the kwargs (regardless if how it's called). that's why we need to bypass/reimplement it .

(def runcall(self, func, *args, **kwds):)

RonnyPfannschmidt · 2019-10-31T16:55:59Z

This is why I am talking about creating a real wrapper that passes only a func into runcall

davidszotten · 2019-10-31T17:53:30Z

Oh I see what you mean. But won’t that mean pdb will start in the wrapper? That might be confusing

RonnyPfannschmidt · 2019-10-31T19:56:41Z

No, runcall would be passed a partial as func on test call

davidszotten · 2019-10-31T20:49:10Z

No, runcall would be passed a partial as func on test call

ok now i'm confused again. could you send some (pseudo/)code?

Without this, the second time it tries to stop in a parametrized function it raises instead: `ValueError: --trace can't be used with a fixture named func!` Implementation idea, test (and changelog tweaks) thanks to blueyed Co-Authored-By: Ronny Pfannschmidt <[email protected]> Co-Authored-By: Daniel Hahler <[email protected]>

davidszotten · 2019-10-31T21:42:06Z

ok think i finally understand. pushed an update. way better, thanks!

blueyed · 2019-11-01T04:06:33Z

Thanks! :)

76: Update pytest to 5.2.4 r=duckinator a=pyup-bot This PR updates [pytest](https://pypi.org/project/pytest) from **5.2.2** to **5.2.4**. <details> <summary>Changelog</summary> ### 5.2.4 ``` ========================= Bug Fixes --------- - `6194 <https://github.com/pytest-dev/pytest/issues/6194>`_: Fix incorrect discovery of non-test ``__init__.py`` files. - `6197 <https://github.com/pytest-dev/pytest/issues/6197>`_: Revert "The first test in a package (``__init__.py``) marked with ``pytest.mark.skip`` is now correctly skipped.". ``` ### 5.2.3 ``` ========================= Bug Fixes --------- - `5830 <https://github.com/pytest-dev/pytest/issues/5830>`_: The first test in a package (``__init__.py``) marked with ``pytest.mark.skip`` is now correctly skipped. - `6099 <https://github.com/pytest-dev/pytest/issues/6099>`_: Fix ``--trace`` when used with parametrized functions. - `6183 <https://github.com/pytest-dev/pytest/issues/6183>`_: Using ``request`` as a parameter name in ``pytest.mark.parametrize`` now produces a more user-friendly error. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pytest - Changelog: https://pyup.io/changelogs/pytest/ - Homepage: https://docs.pytest.org/en/latest/ </details> Co-authored-by: pyup-bot <[email protected]>

103: Update pytest to 5.2.4 r=rehandalal a=pyup-bot This PR updates [pytest](https://pypi.org/project/pytest) from **5.2.2** to **5.2.4**. <details> <summary>Changelog</summary> ### 5.2.4 ``` ========================= Bug Fixes --------- - `6194 <https://github.com/pytest-dev/pytest/issues/6194>`_: Fix incorrect discovery of non-test ``__init__.py`` files. - `6197 <https://github.com/pytest-dev/pytest/issues/6197>`_: Revert "The first test in a package (``__init__.py``) marked with ``pytest.mark.skip`` is now correctly skipped.". ``` ### 5.2.3 ``` ========================= Bug Fixes --------- - `5830 <https://github.com/pytest-dev/pytest/issues/5830>`_: The first test in a package (``__init__.py``) marked with ``pytest.mark.skip`` is now correctly skipped. - `6099 <https://github.com/pytest-dev/pytest/issues/6099>`_: Fix ``--trace`` when used with parametrized functions. - `6183 <https://github.com/pytest-dev/pytest/issues/6183>`_: Using ``request`` as a parameter name in ``pytest.mark.parametrize`` now produces a more user-friendly error. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pytest - Changelog: https://pyup.io/changelogs/pytest/ - Homepage: https://docs.pytest.org/en/latest/ </details> Co-authored-by: pyup-bot <[email protected]>

davidszotten force-pushed the trace_parametrize branch from d3372f4 to 96d8d0c Compare October 29, 2019 10:00

RonnyPfannschmidt requested changes Oct 29, 2019

View reviewed changes

src/_pytest/debugging.py Outdated Show resolved Hide resolved

testing/test_pdb.py Outdated Show resolved Hide resolved

src/_pytest/debugging.py Outdated Show resolved Hide resolved

RonnyPfannschmidt reviewed Oct 29, 2019

View reviewed changes

changelog/6099-bugfix.rst Outdated Show resolved Hide resolved

davidszotten force-pushed the trace_parametrize branch from c9a4d53 to b421aef Compare October 29, 2019 19:23

RonnyPfannschmidt approved these changes Oct 29, 2019

View reviewed changes

blueyed force-pushed the trace_parametrize branch from b421aef to e2b6ab0 Compare October 30, 2019 07:52

blueyed previously approved these changes Oct 30, 2019

View reviewed changes

blueyed force-pushed the trace_parametrize branch from e2b6ab0 to 3844cec Compare October 30, 2019 07:55

blueyed reviewed Oct 30, 2019

View reviewed changes

blueyed mentioned this pull request Oct 30, 2019

3610 add trace option #3647

Merged

blueyed force-pushed the trace_parametrize branch from 4912ed9 to 94ac6bf Compare October 30, 2019 09:28

blueyed mentioned this pull request Oct 30, 2019

Fix --trace for parametrized tests blueyed/pytest#24

Closed

blueyed requested a review from RonnyPfannschmidt October 30, 2019 09:34

RonnyPfannschmidt reviewed Oct 30, 2019

View reviewed changes

blueyed force-pushed the trace_parametrize branch 2 times, most recently from c45761c to b421aef Compare October 30, 2019 10:37

davidszotten force-pushed the trace_parametrize branch from b421aef to 844a957 Compare October 30, 2019 20:28

blueyed reviewed Oct 30, 2019

View reviewed changes

davidszotten force-pushed the trace_parametrize branch from 1463321 to 285524c Compare October 31, 2019 21:41

blueyed approved these changes Nov 1, 2019

View reviewed changes

blueyed merged commit dc5a4fb into pytest-dev:master Nov 1, 2019

Fix --trace for parametrized tests #6099

Fix --trace for parametrized tests #6099

Conversation

davidszotten commented Oct 29, 2019 • edited Loading

RonnyPfannschmidt left a comment

Choose a reason for hiding this comment

davidszotten commented Oct 29, 2019

RonnyPfannschmidt left a comment

Choose a reason for hiding this comment

davidszotten commented Oct 29, 2019

blueyed left a comment

Choose a reason for hiding this comment

blueyed commented Oct 30, 2019

davidszotten commented Oct 30, 2019

Choose a reason for hiding this comment

blueyed commented Oct 30, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidszotten commented Oct 30, 2019

blueyed commented Oct 30, 2019

RonnyPfannschmidt commented Oct 30, 2019

davidszotten commented Oct 30, 2019

davidszotten commented Oct 30, 2019

RonnyPfannschmidt commented Oct 30, 2019

davidszotten commented Oct 30, 2019

Choose a reason for hiding this comment

blueyed commented Oct 30, 2019 • edited Loading

davidszotten commented Oct 30, 2019

davidszotten commented Oct 30, 2019

blueyed commented Oct 30, 2019

blueyed commented Oct 30, 2019

blueyed commented Oct 30, 2019 • edited Loading

davidszotten commented Oct 30, 2019

davidszotten commented Oct 30, 2019 • edited Loading

blueyed commented Oct 30, 2019 • edited Loading

davidszotten commented Oct 30, 2019

davidszotten commented Oct 30, 2019

blueyed commented Oct 30, 2019

blueyed commented Oct 30, 2019 • edited Loading

davidszotten commented Oct 31, 2019

davidszotten commented Oct 31, 2019

RonnyPfannschmidt commented Oct 31, 2019

davidszotten commented Oct 31, 2019

RonnyPfannschmidt commented Oct 31, 2019

davidszotten commented Oct 31, 2019

RonnyPfannschmidt commented Oct 31, 2019

davidszotten commented Oct 31, 2019

davidszotten commented Oct 31, 2019

blueyed commented Nov 1, 2019

davidszotten commented Oct 29, 2019 •

edited

Loading

blueyed commented Oct 30, 2019 •

edited

Loading

blueyed commented Oct 30, 2019 •

edited

Loading

davidszotten commented Oct 30, 2019 •

edited

Loading

blueyed commented Oct 30, 2019 •

edited

Loading

blueyed commented Oct 30, 2019 •

edited

Loading