-
Notifications
You must be signed in to change notification settings - Fork 3.1k
programmatic install quiet option fails #4694
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Adding |
I would check for a difference in Also, I would stay away from |
Hi @uchuugaka! Using pip in-process (i.e. using So, unless you can reproduce this using a subprocess or on the CLI, this is not exactly a bug in pip - you're using it in a way that is not supported by pip developers, even though it is possible to use it like that. As you've said this doesn't reproduce when used via the CLI, I'm tagging this as invalid and closing it. We can still discuss if it's closed. :) All that said, it would be difficult to help you out without some more context in this situation. It seems pip is tripping while trying to get the metadata for the distribution to be installed. Figuring out why that's happening would take some debugging - which feels unnecessary since this problem vanishes if you use pip in a supported manner. I'd suggest you try doing the following instead of using
If the issue still persists, I'll happily reopen and retag this issue. :) |
Wow.
Why would it be unsupported?
The docs say this?
That's really useless.
None of pip's dependencies take that approach. Wouldn't you want to understand where a bug is?
Why would you suggest starting a subprocess?
That's great for one off quick and dirty but a bad idea for building software that needs to be used by others.
I was literally willing to get help and see about finding and fixing the bug.
Now I really feel more like writing a better package manager.
Clearly
pip really does need to be replaced.
It's bad enough as it is and it's the best Python has ever had.
… On Aug 27, 2017, at 2:31, Pradyun Gedam ***@***.***> wrote:
Hi @uchuugaka!
Using pip in-process (i.e. using pip.main) is not a supported way of using pip. The only supported way is to use the CLI - you can use a subprocess to invoke pip.
So, unless you can reproduce this using a subprocess or on the CLI, this is not exactly a bug in pip - you're using it in a way that is not supported by pip developers, even though it is possible to use it like that. As you've said this doesn't reproduce when used via the CLI, I'm tagging this as invalid and closing it. We can still discuss if it's closed. :)
All that said, it would be difficult to help you out without some more context in this situation. It seems pip is tripping while trying to get the metadata for the distribution to be installed. Figuring out why that's happening would take some debugging - which feels unnecessary since this problem vanishes if you use pip in a supported manner.
I'd suggest you try doing the following instead of using pip.main:
subprocess.call(['/path/to/python', '-m', 'pip', 'install', '-q', '--user', 'doesnothing'])
If the issue still persists, I'll happily reopen and retag this issue. :)
―
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
I'm genuinely sorry for making you feel that way. It's just that pip does not provide a Python API, ie. being used in process. That really comes down to the fact that it's mostly like 5 or so people volunteering their free time to maintain pip. There really isn't enough developer time to debug all the issues that can arise from that kind of usage - it comes with it's own set of issues that no one has come around to figuring out yet. Using pip via the command line is a much more specific case that is much easier to support given the limited developer time.
Because that would mean you use pip out-of-process which is supported.
It has its flaws and I'll let you know that there's a fair amount of work being done on that front to make life better for everyone. :)
I, personally, really want to. I'm doing it right now myself. |
@uchuugaka, some questions:
Please also refrain from heaping scorn on all of the people, myself and @pradyunsg included, who have spent literally many hours of their free time developing and supporting software which you are able to use at completely no charge. Your aggressively negative comments ("That's really useless", "Now I really feel more like writing a better package manager", "Clearly pip really does need to be replaced", etc.) are neither constructive nor fair, and do nothing to motivate anyone to help you. In fact, the need to deal with abuse like this is the sort of thing that makes it harder for us to get anyone else involved in helping support pip, exacerbating the lack of manpower issues which are fundamentally at the root of most of the unresolved problems with pip. |
Off Topic: @pfmoore Is there some sort of a standardised response we can have for such issues? I always feel that I'd trigger a response like the one we got here when I write about this situation... Should I create a new issue for discussing that? |
@pradyunsg First of all, I apologise for the fact that you had to deal with a response like that. It's pretty rare - most people are much more reasonable in their responses (and that's why thus far we've not really done anything more than deal with issues where the user's using the internal API on a case by case basis). I assume you do mean the issue of people using pip's internal APIs? Not really. The pip internal API is deliberately not documented, so that people don't mistakenly believe it's supported. But people don't really seem to get that message. Maybe we need to add an explicit section to the documentation explaining that there is no support for using pip's internal API in your own code. Feel free to create an issue for that and I'll take a look at doing something. We could also modify the issue template, to add a note along the lines of "If you're calling pip's internal APIs from your own program, please confirm that you can reproduce the issue using a command line invocation of pip. Pip is a command line program, not a library, and the pip developers don't document or support use of the internal API in user code." Also, a quick search found #2450 - for people asking for a supported API, we can refer them to this issue (which is closed, BTW) and close theirs as a duplicate. Finally, I'm generally inclined to be more sympathetic towards people using |
Well, it's not your fault in anyway. You have nothing to apologise for.
Making a documentation change sounds like a useful thing to me. I'll make an issue.
Sounds good. I'll shoot a PR for this and the documentation change.
Which is a much better approach than the one I took here. I'll do something along these lines next time onwards.
You'd created #3121 about the API and I think that's a nicer discussion for linking on this topic. :) |
Thanks for finding that - I thought there was a better one somewhere :-) |
Well, my apologies if too much scorn heaped upon folks, but raising an issue and seeing it nearly immediately closed for using something that was not documented to be forbidden or unsupported is kind of surprising and quite off-putting. I had not even a chance to follow up any response. I had really hoped to be able do an install-if-needed inline in my code, resolving dependencies lazily but in an easy to log and understand way if things go south. Installing local to the code, but if it is unsupported to use the api of a package, then that is fine. It will be good to be documented as such. $ which pip $ python -c "import sys; print(sys.path)" Looks more like the trace is accurate.
metadata is coming as None here and thus FeedParser is unhappy about that. So the question would be why is that so, if calling pip.main() vs the command line? |
OK. You have pip installed as an egg. That's not a standard way to install pip, so I don't know how you got that or how it might affect the results you're seeing.
To confirm:
works? What precisely does it do? I don't know what the "doesnothing" package is.
fails with the traceback described above? For me, they both "work":
Looks like the issue is with your environment. |
That part's on me. Sorry about that! |
doesnothing is a minimalistic dummy package that depends on a markdown package and has some pretty silly but benign content, which I came across, literally with the intent to test using pip.main() It may be my env. For reference, everything for every version of macOS is at: It is possible to run macOS in a VM for many versions now, so it may be useful in the future. Python 2.7.10 Bundled Packages PyObjC I digress. We install pip via the bundled easy_install to the user site, as this does not require sudo and does not tinker with the bundled stuff, so we do not need to worry about messing up any OS Python processes, because they would be very unlikely to run as our user accounts. Oh this gets interesting.
no output. If I do
no output. Both of these examples work smashingly perfect. Is pip architected such that it could or would fail if pip.main() is called more than once in the same process?
|
I even tried this cli invocation.
Works fine, of course. |
Absolutely. Actually, Python's import system isn't guaranteed to work "as expected" if you install new packages into your system while a Python process is running (depends on what you expect, of course). So this isn't even a pip-specific point - what you're doing has major risks that are outside of "documented correct usage", so you're very much on your own. Hopefully you are starting to see why supporting pip in-process is not trivially easy to do. Best of luck with modifying your processes to work around this behaviour (and I mean that genuinely - you have a workflow that you've found doesn't work the way you expect, and now you're going to have to address that, which is never a nice job). Hopefully what you've found out in the course of this discussion is of some use to you. |
It also fails without the quiet option.
Well, in the end, I do thank you all for you time and apologies we got off on the wrong foot. I would suggest that you save yourselves some future question answering on this not only by explicit docs but also hitting a few of the Stack Overflow posts on "python installing a package programmatically" or even posting your own question and answer on behalf to help it get bumped up. Long term that is as likely as anything to drive more awareness unless and until there is some API alternative you can recommend. Ultimately, I may simply have to forget doing this for a while. I know there is no dependency hell panacea. What I wanted to do was remove an external process. Ours are test machines that can be fully imaged on demand, but I wanted to add more flexibility to the test developer. We try to avoid packages that have external dependencies as much as possible because, they hurt sooner or later. (everything likes to version pin Requests to a different version.) Excellent support. I admit I initially felt the 3rd one, but I see that is not the case and happily so. You folks need some serious funding and/or hack fests at PyCons. Especially if there is ever the ability to import specific package/module versions ... :) Cheers |
Thanks for the update. We've raised an issue (#4696) to document the lack of support for using the internal API. How much good it'll do I'm not sure - people don't often read the docs, and it's hard to find any way to make an unwelcome message any better (and we do appreciate that the message is unwelcome). But at least it'll be an acknowledgement of the situation. Feel free to monitor that issue and provide comments when I get a PR up - it's always better to have feedback from the target audience. Personally, I don't have time to read or answer questions on StackOverflow, so I'll have to leave that to others. |
Sounds like a good idea. I'll might do it someday, when I get the time and motivation to do so. |
I just ran into the very same problem (on pip 9.0.1).
In my case this was to be blamed on a presumably broken package installation through conda (for SQLAlchemy 1.1.4 from conda defaults channel), with just broken links in the egg-info dir (as usual, red color in ls for broken symlinks): While in this instance this is clearly not a problem initially caused by pip, it's still not possible to tell what pip's problem is (broken package installation) and how to fix it from a user perspective, without hacking a debugger into the installed pip source code. So, I would strongly suggest the following diff, so that the user gets some info on the problem and has a chance to fix it without debugging into pip source code: --- packaging.py.orig 2017-11-06 11:53:41.292395270 +0100
+++ packaging.py 2017-11-06 11:50:33.132800936 +0100
@@ -44,6 +44,9 @@
def check_dist_requires_python(dist):
metadata = get_metadata(dist)
+ if not metadata:
+ msg = ("Failed to get distribution metadata for '%s'." % str(dist))
+ raise Exception(msg)
feed_parser = FeedParser()
feed_parser.feed(metadata)
pkg_info_dict = feed_parser.close() Which would make it (staged fail, removing the PKG-INFO file just to show the error message):
I can do a PR with this patch if you want me to (not sure what Exception type you'd prefer for the error and what wording you prefer.. distribution/package installation/...). Alternatively this exception could be placed directly inside |
This is intended to be a gentle note. I got here from guysoft/OctoPi#276, which suggests the workaround of pinning the environment to a pre-9 version of pip. That is the option I'm taking, because I really just need to get my deployment running. Regardless of whether this should be pip's responsibility, if the end result is that people are going to fix it by routing around pip, that would seem to make it pip's problem. |
..and the proposed patch is so simple and short |
@megies what patch are you talking about? If you mean the one in #4694 (comment) then you should submit it as a PR. I'm not sure how it addresses the comment by @adam-iris though, as it just provides a more informative error message. |
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Description:
calling
fails with the following (note that I have replaced my home dir with ~ in the traceback for privacy)
What I've run:
Notably, this works fine from the CLI.
It fails with either
-q
or--quiet
and regardless of which position the option is placed in.pip uninstall works fine from both code and the CLI passing the quiet flag in any form.
Seems like a bug. But odd that it's freaking out on the email feedparser stuff.
Perhaps the pip install command is expecting a newer version of that lib under some circumstance?
The text was updated successfully, but these errors were encountered: