Skip to content

argparse.REMAINDER fails to parse remainder correctly #58382

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rr2do2 mannequin opened this issue Mar 2, 2012 · 11 comments
Open

argparse.REMAINDER fails to parse remainder correctly #58382

rr2do2 mannequin opened this issue Mar 2, 2012 · 11 comments
Labels
stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@rr2do2
Copy link
Mannequin

rr2do2 mannequin commented Mar 2, 2012

BPO 14174
Nosy @jaraco, @merwok, @cjerdonek
Files
  • bug_argparse.py: Reproduction case
  • test.py
  • issue14174_1.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2012-03-02.12:08:09.567>
    labels = ['type-bug', 'library']
    title = 'argparse.REMAINDER fails to parse remainder correctly'
    updated_at = <Date 2014-07-02.06:40:46.284>
    user = 'https://bugs.python.org/rr2do2'

    bugs.python.org fields:

    activity = <Date 2014-07-02.06:40:46.284>
    actor = 'paul.j3'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Library (Lib)']
    creation = <Date 2012-03-02.12:08:09.567>
    creator = 'rr2do2'
    dependencies = []
    files = ['24705', '28165', '35824']
    hgrepos = []
    issue_num = 14174
    keywords = ['patch']
    message_count = 10.0
    messages = ['154761', '154929', '170921', '172232', '172235', '176691', '180753', '187204', '187206', '222074']
    nosy_count = 7.0
    nosy_names = ['jaraco', 'eric.araujo', 'chris.jerdonek', 'idank', 'paul.j3', 'rr2do2', 'Michael.Edwards']
    pr_nums = []
    priority = 'normal'
    resolution = None
    stage = 'needs patch'
    status = 'open'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue14174'
    versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']

    @rr2do2
    Copy link
    Mannequin Author

    rr2do2 mannequin commented Mar 2, 2012

    Reproduction case is attached and should speak for itself, but the short brief is that the argparse.REMAINDER behaviour is very inconsistent based on what (potentially) defined argument handlers come before it.

    Tested this on Python 2.7 on OS X, but also grabbed the latest argparse.py from hg to verify against this.

    @rr2do2 rr2do2 mannequin added stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Mar 2, 2012
    @merwok
    Copy link
    Member

    merwok commented Mar 5, 2012

    Thanks for the report. Could you edit your script to add the expected results, for example Namespace(foo=..., command=[...])?

    @idank
    Copy link
    Mannequin

    idank mannequin commented Sep 21, 2012

    I just ran into this issue myself and worked around it by using parse_known_args*.

    @jaraco
    Copy link
    Member

    jaraco commented Oct 6, 2012

    I also ran into this problem. I put together this script to reproduce the issue:

    import argparse
    
    parser = argparse.ArgumentParser()
    parser.add_argument('app')
    parser.add_argument('--config')
    parser.add_argument('app_args', nargs=argparse.REMAINDER)
    args = parser.parse_args(['app', '--config', 'bar'])
    print vars(args)
    # actual: {'app': 'app', 'app_args': ['--config', 'bar'], 'config': None}
    # expected: {'app': 'app', 'app_args': [], 'config': 'bar'}

    I'll try using parse_known_args instead.

    @idank
    Copy link
    Mannequin

    idank mannequin commented Oct 6, 2012

    Unfortunately parse_known_args is buggy too: http://bugs.python.org/issue16142

    @MichaelEdwards
    Copy link
    Mannequin

    MichaelEdwards mannequin commented Nov 30, 2012

    I'm attaching my own bug repro script for Eric. Is this sufficient? I can demonstrate the entire resulting Namespace, but the problem is that argparse doesn't even produce a Namespace. The cases I show simply fail.

    @cjerdonek
    Copy link
    Member

    See also bpo-17050 for a reduced/simple case where argparse.REMAINDER doesn't work (the case of the first argument being argparse.REMAINDER).

    @paulj3
    Copy link
    Mannequin

    paulj3 mannequin commented Apr 17, 2013

    An alternative to Jason's example:

    parser = argparse.ArgumentParser()
    parser.add_argument('app')
    parser.add_argument('--config')
    parser.add_argument('app_args', nargs=argparse.REMAINDER)
    args = parser.parse_args(['--config', 'bar', 'app'])
    print vars(args)
    # as expected: {'app': 'app', 'app_args': [], 'config': 'bar'}

    When you have several positionals, one or more of which may have 0 arguments (*,?,...), it is best to put all of the optional arguments first.

    With 'app --config bar', parse_args identifies a 'AOA' pattern (argument, optional, argument). It then checks which positional arguments match. 'app' claims 1, 'app_args' claims 2 (REMAINDER means match everything that follows). That leaves nothing for '--config'.

    What you expected was that 'app' would match with the 1st string, '--config' would match the next 2, leaving nothing for 'app_args'.

    In http://bugs.python.org/issue14191 I wrote a patch that would give the results you want if 'app_args' uses '*'. That is makes it possible to interleave positional and optional argument strings. But it does not change the behavior of REMAINDER.

    parser.add_argument('app_args', nargs='*')

    Maybe the documentation example for REMAINDER needs to modified to show just how 'greedy' REMAINDER is. Adding a:

    parser.add_argument('--arg1',action='store_true')

    does not change the outcome. REMAINDER still grabs '--arg1' even though it is a defined argument.

    Namespace(arg1=False, args=['--arg1', 'XX', 'ZZ'], command='cmd', foo='B')

    @paulj3
    Copy link
    Mannequin

    paulj3 mannequin commented Apr 17, 2013

    By the way, parser.parse_args() uses parse_known_arg(). parse_known_args returns a Namespace and a list of unknown arguments. If that list is empty, parse_args returns the Namespace. If the list is not empty, parse_args raises an error.

    So parse_known_args does not change how arguments are parsed. It just changes how the unknowns are handled.

    @paulj3
    Copy link
    Mannequin

    paulj3 mannequin commented Jul 2, 2014

    Here's a possible solution to the problem (assuming there really is one):

    • redefine REMAINDER so it matches a '((?:A[AO]*)?)' pattern (where O is a string that looks like an optional flag, A an argument string). I've added the condition that the first match (if any) must be an A. It ends up being closer to the pattern for PARSER.

    I included a patch from bpo-15112, which delays the consumption of a positional that matches with 0 strings.

    In the sample case for this issue, results with this patch are:

        args = parser.parse_args(['app', '--config', 'bar'])
        # Namespace(app='app', app_args=[], config='bar')
    
        args = parser.parse_args(['--config', 'bar', 'app'])
        # Namespace(app='app', app_args=[], config='bar')
    
        args = parser.parse_args(['app', 'args', '--config', 'bar'])
        # Namespace(app='app', app_args=['args', '--config', 'bar'], config=None)

    In the last case, 'app_args' gets the rest of the strings because the first is a plain 'args'. I believe this is consistent with the intuition expressed in this issue.

    I've added one test case to test_argparse.TestNargsRemainder. This is a TestCase that is similar to the above example.

        argument_signatures = [Sig('x'), Sig('y', nargs='...'), Sig('-z')]
        failures = ['', '-z', '-z Z']
        successes = [
            ('X', NS(x='X', y=[], z=None)),
            ('-z Z X', NS(x='X', y=[], z='Z')),
            ('X A B -z Z', NS(x='X', y=['A', 'B', '-z', 'Z'], z=None)),
            ('X Y --foo', NS(x='X', y=['Y', '--foo'], z=None)),
            ('X -z Z A B', NS(x='X', y=['A', 'B'], z='Z')), # new case
        ]

    This patch runs test_argparse fine. But there is a slight possibility that this patch will cause backward compatibility problems. Some user might expect y=['-z','Z',...]. But that expectation has not been enshrined the test_argparse.

    It may require a slight change to the documentation as well.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @serhiy-storchaka
    Copy link
    Member

    I am not really sure that this is a bug. REMAINDER can be used to simulate a subparser if you cannot use standard subparsers for some reasons (for example if the syntax of the subparser is not known at that level). The original example is such a case. app_args can contain positional and optional arguments for app (you would use nargs='*' if it can contain only positional arguments). It can contain option --config. Therefore it should be considered a part of app_args if it is specified after apps. This looks reasonable.

    If redefine REMAINDER so that the first match must be not option, we could use "--" to pass an option as the first item in REMAINDER (except "--"): app -- --config bar. We cannot use this if REMAINDER is used with option, because arguments of options cannot be "--"). See #66419 which proposes to change this. Together this may work, but it will break some user code.

    Note also that REMAINDER is currently undocumented (see bpo-17050/gh-61252). It is now legacy feature. It adds additional pressure against significant changes of its behavior.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    Status: Bugs
    Development

    No branches or pull requests

    4 participants