Argparse incorrectly handles '--' as argument to option

[mmaker]

BPO	14364
Nosy	@rhettinger, @merwok, @bitdancer, @mmaker, @vadmium, @shihai1991, @horazont
PRs	gh-58572: Fix behavior when '--' as argument to option in argparse #15714
Files	foo.py issue14364.test.patch dbldash.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2012-03-18.14:29:12.929>
labels = ['3.7', '3.8', 'type-bug', 'library', '3.9']
title = "Argparse incorrectly handles '--' as argument to option"
updated_at = <Date 2021-03-23.01:07:11.301>
user = 'https://github.com/mmaker'

bugs.python.org fields:

activity = <Date 2021-03-23.01:07:11.301>
actor = 'rhettinger'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2012-03-18.14:29:12.929>
creator = 'maker'
dependencies = []
files = ['24926', '29938', '45944']
hgrepos = []
issue_num = 14364
keywords = ['patch', 'needs review']
message_count = 17.0
messages = ['156254', '156256', '156275', '156293', '187283', '187344', '187346', '187349', '262545', '262558', '283501', '351230', '353865', '354001', '354049', '354066', '389361']
nosy_count = 9.0
nosy_names = ['rhettinger', 'bethard', 'eric.araujo', 'r.david.murray', 'maker', 'martin.panter', 'paul.j3', 'shihai1991', 'jssfr']
pr_nums = ['15714']
priority = 'high'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue14364'
versions = ['Python 2.7', 'Python 3.7', 'Python 3.8', 'Python 3.9']

[mmaker]

http://docs.python.org/library/argparse.html#arguments-containing
The attached file shows different behaviours when using '--' immediately after an optional argument.

tumbolandia:cpython maker$ python foo.py --test=-- foo
[]
tumbolandia:cpython maker$ python foo.py --test -- foo
usage: foo.py [-h] [-t TEST] [yuri]
foo.py: error: argument -t/--test: expected 1 argument(s)

The same is for single-dash arguments.

tumbolandia:cpython maker$ python foo.py -t -- foo
usage: foo.py [-h] [-t TEST] [yuri]
foo.py: error: argument -t/--test: expected 1 argument(s)
tumbolandia:cpython maker$ python foo.py -t-- foo
[]

Obviously argparse should return an error in both cases.
The bug is probably due to Lib/argparser.py:2211

[bitdancer]

It does look like there's anomalous behavior here of some sort, but I'd expect --test=-- to result in test="--", myself, rather than an error. My intuition is that '--' would need to be preceded by a space to function as the 'end of options' marker. Because of that, I've never tried the above scenario with a unix command, so maybe my intuition is wrong :)

Just for reference (the code paths may be different), there is another open issue about -- parsing, bpo-13922.

[bethard]

I just tried this with grep's "-e" and "--regexp":

$ cat > temp.txt
a--b
cdef
$ grep -e-- -v temp.txt
cdef
$ grep --regexp=-- -v temp.txt
cdef
$ grep -e -- -v temp.txt
cdef
$ grep --regexp -- -v temp.txt
cdef

And with diff's "-I" and "--ignore-matching-lines":

$ cat > temp2.txt
cdef
a--b
$ diff temp.txt temp2.txt
1d0
< a--b
2a2
> a--b
$ diff -I-- temp.txt temp2.txt 
$ diff -I -- temp.txt temp2.txt 
$ diff --ignore-matching-lines -- temp.txt temp2.txt 
$ diff --ignore-matching-lines=-- temp.txt temp2.txt 
$

Note though that for options that don't take an argument, the "--" is just removed:

$ grep -v -- a temp.txt 
cdef
$ diff -i -- temp.txt temp2.txt
1d0
< a--b
2a2
> a--b

So I guess the unix rule is: if an option that takes an argument is followed by "--", use that as the option's argument and continue parsing as usual. If an option that takes no argument is followed by "--", then delete the "--" and treat all following flags as positional arguments.

Argparse can't follow this directly, because then people who are using "--" to signal the end of an option with nargs="*" would start getting "--" included in that list. (And I know people have used "--" this way for a while.)

I guess my preference is what R. David Murray suggests: "--" when part of an argument (i.e. not separated by spaces) is treated like any other characters, and only a lone "--" signals the end of options (and is ignored otherwise). That would mean that both "--test=--" and "-t--" in your example would give you ["--"], and the other errors would stay as you saw them.

[mmaker]

+1 also for me.
I will try to work for a patch in the next days. :)

[BreamoreBoy]

@michele could you provide a patch for this?

[paulj3]

The patch that I recently submitted for http://bugs.python.org/issue13922
appears to solve this issue. It only removes the '--' that marked the end of options.

With:

parser = argparse.ArgumentParser()
parser.add_argument('-f','--foo')
print(parser.parse_args(['-f--']))
print(parser.parse_args(['--foo=--']))
print(parser.parse_args(['-f', '--']))

I get:

Namespace(foo='--')
Namespace(foo='--')
usage: foodash.py [-h] [-f FOO]
foodash.py: error: argument -f/--foo: expected one argument

[mmaker]

wow, I was just writing the unittests, thanks paul.
Shall I continue? I don't see any test case on tip.

[mmaker]

Yes, http://bugs.python.org/file29845/dbldash.patch seems to fix this.
Attaching the unittests, and noisying on your issue.

[vadmium]

In Python 3.5, this does not seem fixed. bpo-13922 is marked as resolved and fixed, but Pauls’s patch has not actually been committed.

>>> ap = ArgumentParser()
>>> ap.add_argument("-t", "--test", type=str, nargs=1)
_StoreAction(option_strings=['-t', '--test'], dest='test', nargs=1, const=None, default=None, type=<class 'str'>, choices=None, help=None, metavar=None)
>>> ap.add_argument('yuri', nargs='?')
_StoreAction(option_strings=[], dest='yuri', nargs='?', const=None, default=None, type=None, choices=None, help=None, metavar=None)
>>> ap.parse_args(["--test=--", "foo"])
Namespace(test=[], yuri='foo')
>>> ap.parse_args(["--test", "--", "foo"])
usage: [-h] [-t TEST] [yuri]
: error: argument -t/--test: expected 1 argument
argparse.ArgumentError: argument -t/--test: expected 1 argument

During handling of the above exception, another exception occurred:

__main__.SystemExit: 2
>>> ap.parse_args(["-t--", "foo"])
Namespace(test=[], yuri='foo')
>>> ap.parse_args(["-t", "--", "foo"])
usage: [-h] [-t TEST] [yuri]
: error: argument -t/--test: expected 1 argument
argparse.ArgumentError: argument -t/--test: expected 1 argument

During handling of the above exception, another exception occurred:

__main__.SystemExit: 2

[paulj3]

I made a mistake of trying to add to or refine a closed patch. Maybe I need to move the dbldash.patch over here for more formal consideration.

[paulj3]

I've copied 'dbldash.patch' over from http://bugs.python.org/issue13922. I mistakenly added it to a closed issue.

The patch is old, and should be revised. It is also missing tests for this '--opt=--' case, even though it solves it.

I also reference this patch in http://bugs.python.org/issue9571, which tries to refine dbldash handling for PARSER, REMAINDER nargs.

I'm raising the priority on this because the issue has a come up a couple of times on Stackoverflow:

http://stackoverflow.com/questions/40685320/python-argparse-with-as-the-value

[shihai1991]

some test cases which paul provided looks doesn't keep compatible.

In TestDoubleDashRemoval:
# output in my env is Namespace(cmd='cmd', foo=None, rest=['--', '--foo'])
('-- cmd -- -- --foo', NS(cmd='cmd', foo=None, rest=['--', '--', '--foo']))

# output in my env is Namespace(cmd='--', foo='1', rest=['1', '2'])
('-f1 -- -- 1 -- 2', NS(cmd='--', foo='1', rest=['1', '--', '2']))

# output in my env is Namespace(cmd='--foo', foo=None, rest=['--bar', '2'])
('-- --foo -- --bar 2', NS(cmd='--foo', foo=None, rest=['--', '--bar', '2']))

# output in my env is Namespace(cmd='cmd', foo=None, rest=['--foo', '-f2'])
('cmd -- --foo -- -f2', NS(cmd='cmd', foo=None, rest=['--foo', '--', '-f2']))

In TestDoubleDashRemoval1:
# output in my env is Namespace(cmd='cmd', foo='1', rest=['-f2', '3'])
('-f1 -- cmd -- -f2 3', NS(cmd='cmd', foo='1', rest=['--', '-f2', '3']))

my python's version is: 2.7.5, 3.6.8

[horazont]

Since I have been adversely affected by this bug (1), I looked at the patches.

I combined bpo-14364.test.patch (which adds test cases for --foo=--) and dbldash.patch in my local working tree and that seems to resolve the issue (tests pass if and only if I apply dbldash.patch and my reproducer from 1) passes too).

The patches do not contain any type of metainformation, so I’m not comfortable with submitting this as a PR. I am also not at all familiar with the process of managing the changelog etc.

This should serve as a confirmation that this issue can be resolved with the patches. Someone more familiar than me with the process could take up the task to get this merged.

Note: As you can see in 1, this issue has already caused data loss.

[shihai1991]

Hi, Jonas. Thanks for your report.
Let us wait core team's discuss;)

[rhettinger]

Paul and David, is the current PR complete in your opinion? Are there known cases where the PR would break existing, working code?

Also there needs to be doc entry that is clear on both new and old strategy for distinguishing options from arguments when a dash or dashdash is present. The PR's Misc/NEWS entry, "Fix behavior of argparse when '--' as argument to option", is insufficient.

[shihai1991]

The PR's Misc/NEWS entry, "Fix behavior of argparse when '--' as argument to option", is insufficient.

Thanks for your comment, Raymond. I would continue update the desc.

[rhettinger]

Unassigning because I haven't been able to make time for this one.

[vadmium]

I expect this is now fixed for 3.11+ via #109475