programmatic install quiet option fails

[uchuugaka]

• Pip version: 9.0.1
• Python version: 2.7.10
• Operating system: macOS 10.12.6

Description:

calling

pip.main(['install',  '-q', '--user', 'doesnothing'])

fails with the following (note that I have replaced my home dir with ~ in the traceback for privacy)

Exception:
Traceback (most recent call last):
  File "~/Library/Python/2.7/lib/python/site-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "~/Library/Python/2.7/lib/python/site-packages/pip-9.0.1-py2.7.egg/pip/commands/install.py", line 324, in run
    requirement_set.prepare_files(finder)
  File "~/Library/Python/2.7/lib/python/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "~/Library/Python/2.7/lib/python/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line 666, in _prepare_file
    check_dist_requires_python(dist)
  File "~/Library/Python/2.7/lib/python/site-packages/pip-9.0.1-py2.7.egg/pip/utils/packaging.py", line 48, in check_dist_requires_python
    feed_parser.feed(metadata)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/email/feedparser.py", line 177, in feed
    self._input.push(data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/email/feedparser.py", line 99, in push
    parts = data.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'
2

What I've run:

Notably, this works fine from the CLI.
It fails with either -q or --quiet and regardless of which position the option is placed in.

pip uninstall works fine from both code and the CLI passing the quiet flag in any form.

Seems like a bug. But odd that it's freaking out on the email feedparser stuff.
Perhaps the pip install command is expecting a newer version of that lib under some circumstance?

[uchuugaka]

Adding -I before the quiet opt works! HMmm.
pip_sequence = ['install', '-I', '-q', '--user', package_name]

[benoit-pierre]

I would check for a difference in sys.path, does it work with python -m pip install -q ... when run from the directory containing your script?

Also, I would stay away from --ignore-installed, as it can break your installation (see pypa/setuptools#1104).

[pradyunsg]

Hi @uchuugaka!

Using pip in-process (i.e. using pip.main) is not a supported way of using pip. The only supported way is to use the CLI - you can use a subprocess to invoke pip.

So, unless you can reproduce this using a subprocess or on the CLI, this is not exactly a bug in pip - you're using it in a way that is not supported by pip developers, even though it is possible to use it like that. As you've said this doesn't reproduce when used via the CLI, I'm tagging this as invalid and closing it. We can still discuss if it's closed. :)

All that said, it would be difficult to help you out without some more context in this situation. It seems pip is tripping while trying to get the metadata for the distribution to be installed. Figuring out why that's happening would take some debugging - which feels unnecessary since this problem vanishes if you use pip in a supported manner.

I'd suggest you try doing the following instead of using pip.main:

subprocess.call(['/path/to/python', '-m', 'pip', 'install',  '-q', '--user', 'doesnothing'])

If the issue still persists, I'll happily reopen and retag this issue. :)

[uchuugaka]

Wow. Why would it be unsupported? The docs say this? That's really useless. None of pip's dependencies take that approach. Wouldn't you want to understand where a bug is? Why would you suggest starting a subprocess? That's great for one off quick and dirty but a bad idea for building software that needs to be used by others. I was literally willing to get help and see about finding and fixing the bug. Now I really feel more like writing a better package manager. Clearly pip really does need to be replaced. It's bad enough as it is and it's the best Python has ever had.

…

On Aug 27, 2017, at 2:31, Pradyun Gedam ***@***.***> wrote: Hi @uchuugaka! Using pip in-process (i.e. using pip.main) is not a supported way of using pip. The only supported way is to use the CLI - you can use a subprocess to invoke pip. So, unless you can reproduce this using a subprocess or on the CLI, this is not exactly a bug in pip - you're using it in a way that is not supported by pip developers, even though it is possible to use it like that. As you've said this doesn't reproduce when used via the CLI, I'm tagging this as invalid and closing it. We can still discuss if it's closed. :) All that said, it would be difficult to help you out without some more context in this situation. It seems pip is tripping while trying to get the metadata for the distribution to be installed. Figuring out why that's happening would take some debugging - which feels unnecessary since this problem vanishes if you use pip in a supported manner. I'd suggest you try doing the following instead of using pip.main: subprocess.call(['/path/to/python', '-m', 'pip', 'install', '-q', '--user', 'doesnothing']) If the issue still persists, I'll happily reopen and retag this issue. :) ― You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[pradyunsg]

I'm genuinely sorry for making you feel that way.

It's just that pip does not provide a Python API, ie. being used in process. That really comes down to the fact that it's mostly like 5 or so people volunteering their free time to maintain pip. There really isn't enough developer time to debug all the issues that can arise from that kind of usage - it comes with it's own set of issues that no one has come around to figuring out yet. Using pip via the command line is a much more specific case that is much easier to support given the limited developer time.

---

Why would you suggest starting a subprocess?

Because that would mean you use pip out-of-process which is supported.

It's bad enough as it is and it's the best Python has ever had.

It has its flaws and I'll let you know that there's a fair amount of work being done on that front to make life better for everyone. :)

---

Wouldn't you want to understand where a bug is?

I, personally, really want to. I'm doing it right now myself.

[pfmoore]

@uchuugaka, some questions:

1. Please confirm what which pip and pip --version state
2. Please confirm what python -m pip --version states
3. Please confirm what python -c "import sys; print(sys.path)" displays
4. Please confirm if you have any *.py files in your current directory, and if so what their names are.

Please also refrain from heaping scorn on all of the people, myself and @pradyunsg included, who have spent literally many hours of their free time developing and supporting software which you are able to use at completely no charge. Your aggressively negative comments ("That's really useless", "Now I really feel more like writing a better package manager", "Clearly pip really does need to be replaced", etc.) are neither constructive nor fair, and do nothing to motivate anyone to help you. In fact, the need to deal with abuse like this is the sort of thing that makes it harder for us to get anyone else involved in helping support pip, exacerbating the lack of manpower issues which are fundamentally at the root of most of the unresolved problems with pip.

[pradyunsg]

Off Topic: @pfmoore Is there some sort of a standardised response we can have for such issues? I always feel that I'd trigger a response like the one we got here when I write about this situation... Should I create a new issue for discussing that?

[pfmoore]

@pradyunsg First of all, I apologise for the fact that you had to deal with a response like that. It's pretty rare - most people are much more reasonable in their responses (and that's why thus far we've not really done anything more than deal with issues where the user's using the internal API on a case by case basis).

I assume you do mean the issue of people using pip's internal APIs? Not really. The pip internal API is deliberately not documented, so that people don't mistakenly believe it's supported. But people don't really seem to get that message. Maybe we need to add an explicit section to the documentation explaining that there is no support for using pip's internal API in your own code. Feel free to create an issue for that and I'll take a look at doing something.

We could also modify the issue template, to add a note along the lines of "If you're calling pip's internal APIs from your own program, please confirm that you can reproduce the issue using a command line invocation of pip. Pip is a command line program, not a library, and the pip developers don't document or support use of the internal API in user code."

Also, a quick search found #2450 - for people asking for a supported API, we can refer them to this issue (which is closed, BTW) and close theirs as a duplicate.

Finally, I'm generally inclined to be more sympathetic towards people using pip.main than other forms of internal API usage - actual problems with that (i.e, not user error) are more likely to indicate genuine issues than a misuse of the API. However, in my experience most such reports actually end up being user error - so the first step is always "does it work when used from the CLI?" and if it does then "what is the difference between the invocation in your code and what the CLI invokes?" I wouldn't normally close such issues immediately, although I would explain to the user that they are using pip in an unsupported way and we would expect the issue to be with how they are using pip, something we can only offer limited assistance with, rather than a bug in pip itself.

[pradyunsg]

I apologise for the fact that you had to deal with a response like that.

Well, it's not your fault in anyway. You have nothing to apologise for.

Feel free to create an issue for that and I'll take a look at doing something.

Making a documentation change sounds like a useful thing to me. I'll make an issue.

modify the issue template,

Sounds good. I'll shoot a PR for this and the documentation change.

I wouldn't normally close such issues immediately, although I would explain to the user that they are using pip in an unsupported way and we would expect the issue to be with how they are using pip, something we can only offer limited assistance with, rather than a bug in pip itself.

Which is a much better approach than the one I took here. I'll do something along these lines next time onwards.

quick search found #2450

You'd created #3121 about the API and I think that's a nicer discussion for linking on this topic. :)

[pfmoore]

You'd created #3121 about the API and I think that's a nicer discussion for linking on this topic. :)

Thanks for finding that - I thought there was a better one somewhere :-)

[uchuugaka]

Well, my apologies if too much scorn heaped upon folks, but raising an issue and seeing it nearly immediately closed for using something that was not documented to be forbidden or unsupported is kind of surprising and quite off-putting. I had not even a chance to follow up any response.
Both giving and receiving, everything sounds worse in text doesn't it?

I had really hoped to be able do an install-if-needed inline in my code, resolving dependencies lazily but in an easy to log and understand way if things go south. Installing local to the code, but if it is unsupported to use the api of a package, then that is fine. It will be good to be documented as such.

$ which pip
~/Library/Python/2.7/bin/pip
$ pip --version
pip 9.0.1 from ~/Library/Python/2.7/lib/python/site-packages/pip-9.0.1-py2.7.egg (python 2.7)
$ python -m pip --version
pip 9.0.1 from ~/Library/Python/2.7/lib/python/site-packages/pip-9.0.1-py2.7.egg (python 2.7)

$ python -c "import sys; print(sys.path)"
Is totally stock for a mac, the same you'll get on every mac on 2.7.x with the user site enabled.
Only delta ever is the current working directory.
I have tried moving the file to different locations and nothing.
Before importing sys to check the path in flight, the only thing directly imported is pip.

Looks more like the trace is accurate.
In packaging.py, in

def check_dist_requires_python(dist):
    metadata = get_metadata(dist)
    feed_parser = FeedParser()
    feed_parser.feed(metadata)
    ...

metadata is coming as None here and thus FeedParser is unhappy about that.

So the question would be why is that so, if calling pip.main() vs the command line?
I would have assumed the list of strings passed to pip.main() is essentially ARGV here.
Perhaps an off by one error? (random guess)
I couldn't really follow the path through all of the classes calling classes.
Happy to check additional bits if it is helpful in finding and resolving a bug whether it is public API or not.

[pfmoore]

pip 9.0.1 from ~/Library/Python/2.7/lib/python/site-packages/pip-9.0.1-py2.7.egg (python 2.7)

OK. You have pip installed as an egg. That's not a standard way to install pip, so I don't know how you got that or how it might affect the results you're seeing.

Is totally stock for a mac,
Which is no help to me as I don't have a mac.

I would have assumed the list of strings passed to pip.main() is essentially ARGV here.
I'd suggest you don't make assumptions when using undocumented APIs and then expect help from the developers.

To confirm:

pip install -q --user doesnothing

works? What precisely does it do? I don't know what the "doesnothing" package is.

python -c "import pip; pip.main(['install', '-q', '--user', 'doesnothing'])"

fails with the traceback described above? For me, they both "work":

>pip install -q --user doesnothing
(no output)
>python -c "import pip; pip.main(['install',  '-q', '--user', 'doesnothing'])"
(no output)

Looks like the issue is with your environment.